End-to-end autonomous driving is a holistic approach to AV system development in which the system takes in raw sensor data from cameras, radar, and lidar and directly outputs vehicle controls. Unlike traditional systems that rely on modular designs with separate components—such as detection, tracking, prediction, planning and control—end-to-end driving aims to streamline the process to avoid the deep and cascaded path from perception to planning. NVIDIA’s end-to-end driving model—Hydra-MDP—combines the components into a single network with a minimalistic design. The planning input comes directly from a bird’s-eye view feature map generated from sensor data. The NVIDIA Research team’s progress in this area won the CVPR 2024 Autonomous Grand Challenge in the End-to-End Driving at Scale category. In addition to the real-world data provided in the competition, the model also generalizes to simulation environments with high-fidelity sensors, such as NVIDIA Omniverse, for further de...