PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Autonomous Driving: Challenges and Frontiers

Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, Hongyang Li

arXiv:2306.16927v3cs.ROcs.AIcs.CVcs.LG

TL;DR

End-to-end autonomous driving raises how raw sensor inputs can be transformed into vehicle plans while addressing the limitations of modular pipelines. This survey synthesizes more than 270 papers across methods, benchmarks, challenges, and emerging techniques, concluding that the field has substantial opportunities alongside unresolved practical and methodological challenges.

  • Problem

    Modular driving pipelines separately optimize perception, prediction, and planning, motivating learning-based frameworks that target driving decisions more directly.

  • Method

    The survey synthesizes end-to-end driving methodologies, simulation and benchmarking practices, critical challenges, and emerging directions including foundation models and visual pre-training.

  • Results

    The survey identifies broad progress and opportunities in end-to-end driving while systematically analyzing challenges including causal confusion, domain adaptation, simulation realism, and multimodal cost learning.

  • Takeaways & Limitations

    End-to-end autonomous driving is presented as a promising route toward generalist agents, supported by high-quality data, large-scale training, and reliable benchmarks.

  • Takeaways & Limitations

    Cost-learning approaches often require HD maps, auxiliary perception tasks, and multiple sensors, increasing the difficulty of learning and constructing multimodal multitask datasets.

Abstract

from arXiv · show

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework. we maintain an active repository that contains up-to-date literature and open-source projects at https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving.

1 INTRODUCTION

End-to-end autonomous driving replaces separately optimized modular components with jointly trained systems that map raw sensor data to planning or control outputs. This survey reviews the field’s methods, benchmarks, challenges, roadmap, and future directions across more than 270 papers.

  • Motivation: Modular systems develop perception, prediction, and planning separately, while rule-based planning struggles with the vast range of road situations.These limitations motivate learning-based planning from large-scale data.
  • Motivation: End-to-end systems combine perception, prediction, and planning in a jointly trained model optimized toward the ultimate planning or control task.Shared representations and global optimization replace direct handoff of task-specific outputs between components.
  • Scope and contributions: The survey reviews more than 270 papers, covering imitation learning, reinforcement learning, benchmarks, critical challenges, and future trends.Its scope includes closed- and open-loop evaluation, interpretability, generalization, world models, causal confusion, foundation models, and data engines.
  • Roadmap: The field’s roadmap spans early end-to-end systems, deep learning advances, policy distillation, multimodal Transformers, and newer benchmarks such as CARLA v2 and nuPlan.The survey presents these milestones chronologically and links them to performance developments in driving benchmarks.
  • Scope and contributions: The survey’s central contribution is a whole-framework analysis that connects methodologies, simulation and benchmarking, challenges, promising resolutions, and foundation-model opportunities.It also maintains an updated repository of literature and open-source projects.

2 METHODS

End-to-end driving methods are organized around imitation learning and reinforcement learning. Imitation learning includes behavior cloning and inverse optimal control, while reinforcement learning learns through interaction and is commonly studied in simulation.

  • Imitation Learning: Imitation learning trains a policy to imitate an expert using demonstrated state-action trajectories.The learned policy may output planned trajectories or control signals.
  • Behavior Cloning: Behavior cloning reduces imitation learning to supervised learning by minimizing the distance between predicted and expert actions.Its simplicity and efficiency avoid hand-crafted reward design, but covariate shift and causal confusion remain challenges.
  • Inverse Optimal Control: Inverse optimal control learns a reward or cost function from demonstrations and uses it to select preferred trajectories.Cost-learning methods typically sample candidate trajectories and encourage expert demonstrations to have lower cost than alternatives.
  • Inverse Optimal Control: Cost-learning approaches can improve decision safety and interpretability, but often require HD maps, auxiliary perception tasks, and multiple sensors.These additions increase the difficulty of learning and constructing multimodal multitask datasets.
  • Reinforcement Learning: Reinforcement learning learns by trial and error, yet driving applications are usually confined to simulation because exploration requires unsafe actions and substantially more data than imitation learning.Real-world deployment also requires environments that can support parallelized data collection without exposing vehicles to potentially unsafe behavior.
  • Reinforcement Learning: Reinforcement learning has achieved lane following on an empty real-world street but has not yet matched imitation learning for end-to-end driving.The survey suggests insufficient reinforcement-learning gradients for training the deep perception architectures required for driving as a likely reason.

3 BENCHMARKING

End-to-end driving is benchmarked through real-world, closed-loop simulation, and open-loop dataset evaluation, with simulation offering scalable testing but limited real-world transfer. Effective simulators must model environment initialization, traffic, sensors, and physically plausible vehicle dynamics.

  • Three benchmarking approaches are real-world evaluation, online or closed-loop simulation, and offline or open-loop evaluation on driving datasets.
  • Simulation enables rapid, low-cost testing across diverse scenarios and accurate measurement, but simulated results may not generalize to the real world.
  • Closed-loop simulators deploy the driving system in traffic toward a goal while modeling parameter initialization, traffic, sensors, and vehicle dynamics.
  • Parameter Initialization: Simulator initialization uses procedural generation for scalable variation or real-world log sampling for more realistic scenarios, although logs may omit rare situations.
  • Traffic Simulation: Traffic simulation can use predefined rules or data-driven models, trading implementation simplicity against realistic interactive behavior and labeled-data requirements.
  • Sensor and Vehicle Simulation: Sensor simulation generates camera or LiDAR observations with noise and occlusions using graphics-based or data-driven methods, while vehicle models must preserve physically plausible motion.

4 CHALLENGES

End-to-end driving faces challenges in integrating heterogeneous sensing modalities and selecting useful representations, while language-based interfaces introduce additional opportunities for grounding and control. Fusion must reconcile differing sensor perspectives, distributions, costs, and computational demands.

  • The survey organizes challenges around modalities, visual abstraction, world models, multi-task learning, policy distillation, interpretability, and other safety and reliability issues.
  • Sensing and Multi-sensor Fusion: Single-camera lane following cannot handle complex scenarios, motivating combinations of cameras, LiDAR, radar, event cameras, vehicle states, and navigation commands.
  • Sensing and Multi-sensor Fusion: Heterogeneous sensors differ in perspective, data distribution, and price, making effective sensory-layout and fusion design difficult.
  • Sensing and Multi-sensor Fusion: End-to-end systems explore early, mid, and late fusion, with early fusion combining sensory inputs before shared feature extraction.
  • Sensing and Multi-sensor Fusion: Unified spaces such as BEV can align modalities, while end-to-end driving must retain policy-relevant context and discard irrelevant details.
  • Sensing and Multi-sensor Fusion: Self-attention incurs substantial computational cost and does not guarantee useful information extraction, motivating more advanced fusion mechanisms.
  • Language and Driving: Language-grounded driving research uses advice, instructions, visual grounding, and LLM prompting for control prediction, explanations, and broader domain handling.

4.2 Dependence on Visual Abstraction

Visual abstraction is central because end-to-end driving must compress diverse, high-dimensional observations into representations aligned with policy decisions. Existing choices—including BEV, maps, semantic features, and self-supervised features—remain subject to bottlenecks and unresolved co-design questions.

  • End-to-end driving encodes a diverse, high-dimensional state before decoding a policy, creating risk that representations attend to areas misaligned with decision-making.
  • Representation Designs: CNNs remain efficient and translation-equivariant, while Transformers offer scalable perception and BEV unifies sensor and temporal information in 3D space.
  • Map Representation: Online map representations include BEV segmentation, vectorized lanes, centerlines with topology, and lane segments, but the best formulation for end-to-end systems remains unvalidated.
  • Representation Designs: Representation and policy design require co-design, and the necessity of explicit maps is uncertain as some simple approaches benefit from increased training resources.
  • Representation Risks: Inductive biases can create information bottlenecks, while removing redundant context may discard information unrelated to decisions or potentially useful for them.
  • Representation Learning: Semantic masks, affordances, pretrained intermediate features, and attention-augmented latent features have all been explored as policy-learning representations.
  • Representation Learning: Self-supervised representation learning from large-scale unlabeled data, including point-cloud forecasting pretraining, is described as promising for policy learning.

4.3 Complexity of World Modeling for Model-based RL

World models are proposed to improve model-based reinforcement learning efficiency by predicting future environment dynamics, but accurately modeling autonomous-driving scenes remains difficult. The survey also relates auxiliary-task learning and policy distillation to extracting useful supervision for end-to-end policies.

  • World Models: World models provide explicit future predictions that support safer maneuvering and address the high sample complexity of deep reinforcement learning.
  • World Models: Model-based reinforcement learning lets agents interact with learned transition dynamics and reward functions instead of the actual environment.
  • World Models: Researchers simplify or probabilistically model transition dynamics and estimate uncertainty to address inaccuracies in learned world models.
  • World Models: Raw-image world modeling can miss small but important details such as traffic lights, motivating diffusion methods and world models in BEV spaces.
  • World Models: World-model learning is promising for reducing reinforcement-learning sample complexity, but the required content and effective modeling strategy remain open questions.
  • Multi-task Learning: Multi-task learning shares representations across related tasks, but selecting auxiliary tasks and weighting their losses remains challenging for sparse policy supervision.
  • Policy Distillation: Policy distillation trains a privileged agent with ground-truth state information and transfers its behavior to a sensorimotor agent through feature and output imitation.

4.5 Inefficient Experts and Policy Distillation

Imitation learning commonly uses a teacher–student setup, but imperfect teachers and the student’s simultaneous perception-policy learning make distillation difficult. Privileged teachers can improve supervision, yet substantial gaps remain between privileged and sensorimotor agents.

  • Teacher–student imitation learning faces two core difficulties: teachers are imperfect, while students must learn perception and policy from sensor input simultaneously.Teachers may access ground-truth states unavailable to students.
  • Two-stage distillation trains a privileged teacher with environmental state access before transferring its policy to a sensorimotor student.Some methods use compact bird’s-eye-view representations for the privileged agent.
  • Feature-level distillation supplements planning supervision with auxiliary teachers, feature losses, BEV alignment, or model-based action-value supervision.Examples use segmentation, optical flow, and privileged reinforcement-learning experts.
  • Privileged-state information can be inefficient to distill, especially for small traffic lights, leaving visuomotor agents with large performance gaps relative to privileged agents.The passage also links this difficulty to possible causal confusion.

4.6 Lack of Interpretability

Interpretability is important for debugging, performance guarantees, and public acceptance, but end-to-end models make it especially challenging. The survey organizes design-based approaches around attention, interpretable tasks, rules and costs, language, and uncertainty.

  • Interpretability supports system debugging, societal performance guarantees, and public acceptance, yet remains especially difficult for end-to-end models.End-to-end driving models are often treated as black boxes.
  • Post-hoc saliency maps show which visual regions influence planning, but their information content, effectiveness, and validity are limited.The survey therefore emphasizes interpretability built directly into model design.
  • Attention Visualization: Attention-based methods aggregate important intermediate, object-region, or grid features using learned attention weights.Some methods iteratively predict and refine the aggregated features.
  • Interpretable Tasks: Interpretable-task methods decode latent representations into semantic segmentation, depth, detection, affordances, motion, or gaze predictions.These auxiliary outputs expose meaningful information beyond policy prediction.
  • Rules Integration and Cost Learning: Rules and cost learning explicitly incorporate perception, prediction, safety, comfort, traffic rules, and routes into trajectory scoring, improving reported robustness and safety.This design retains a connection to modular systems while optimizing decisions through learned costs.
  • Linguistic Explainability: Language-based methods produce control alongside explanations, narrations, reasoning, or decision justifications for human understanding.They use paired driving data or fuse trajectories with intermediate perception features.
  • Uncertainty Modeling: Uncertainty modeling quantifies output dependability and can help identify cases requiring improvement or intervention.Aleatoric and epistemic uncertainty represent task-inherent ambiguity and limitations from data or model capacity, respectively.

4.7 Lack of Safety Guarantees

End-to-end driving lacks the precise mathematical safety guarantees associated with traditional rule-based approaches. Existing modular safety constraints and interpretable intermediate predictions may nevertheless be adapted as checks or post-processing.

  • Learning-based end-to-end frameworks inherently lack precise mathematical safety guarantees compared with traditional rule-based approaches.This creates a central concern for real-world deployment.
  • Safety constraints from modular motion-planning and speed-prediction modules could be adapted into end-to-end systems as post-processing steps or safety checks.The passage presents this as a potential way to provide additional guarantees.
  • Causal confusion illustrates a safety-relevant failure risk because models may rely on correlations between actions and spurious features such as velocity or past trajectory.The figure identifies this reliance as a route to causal confusion.

4.8 Causal Confusion

Temporal smoothness makes past motion predictive of the next action, but multi-frame models can exploit this shortcut and fail catastrophically at deployment. Proposed remedies exist, although evidence in state-of-the-art settings remains incomplete.

  • Multi-frame models may over-rely on past motion, producing catastrophic deployment failures known as the copycat problem.The survey describes this as a manifestation of causal confusion, where more information can worsen performance.
  • Action is often correlated with velocity or past trajectory, but such correlations can break when the underlying situation changes, such as at a red light.A waiting car may repeatedly associate zero speed with braking until the light turns green.
  • Some approaches remove spurious temporal correlations through adversarial bottleneck training, single-frame predictions, action residuals, or aligned LiDAR histories.The adversarial approach works in MuJoCo but does not scale to complex vision-based driving according to the passage.
  • Existing studies often use environments simplified for causal-confusion analysis, so improvements in state-of-the-art settings remain an open problem.The limitation concerns the evaluation setting rather than the existence of proposed countermeasures.

4.9 Lack of Robustness

End-to-end autonomous driving remains vulnerable to long-tailed scenarios, covariate shift, and domain gaps across simulation, geography, weather, illumination, and sensors. Existing mitigation strategies include data rebalancing, scenario generation, iterative expert labeling, and domain-invariant representations, but realistic comprehensive testing and adaptation remain open challenges.

  • Long-tailed Data: Data processing methods address dataset imbalance through over-sampling, under-sampling, data augmentation, and weighting-based approaches.
  • Long-tailed Data: Safety-critical driving scenarios are rare, diverse, and difficult to reproduce in the real world, creating a severe long-tailed distribution problem.Most drives contain repetitive lane-following frames, while rare critical events are diverse and safety-relevant.
  • Scenario Generation: Adversarial and handcrafted simulation methods generate safety-critical scenarios, but efficiently covering realistic long-tailed behavior remains challenging.Approaches modify trajectories, optimize perturbations, or use risky scenario generation while attempting to preserve physical plausibility.
  • Covariate Shift: Covariate shift causes compounding errors when an agent encounters states outside the expert policy’s training distribution.DAgger addresses this by rolling out the current policy, labeling visited states with an expert, and retraining iteratively.
  • Covariate Shift: DAgger-based driving methods reduce or distribute expert-labeling costs through safety-policy querying and multi-environment data aggregation.SafeDAgger queries the expert only when estimated deviation is large, while MetaDAgger aggregates data across environments.
  • Domain Adaptation: Domain adaptation spans sim-to-real, geography-to-geography, weather-to-weather, day-to-night, and sensor-to-sensor shifts, which often overlap.Current methods emphasize visual sim-to-real adaptation, while LiDAR-specific techniques and traffic-behavior adaptation remain needed.
  • Domain Adaptation: Current domain adaptation focuses mainly on visual gaps, leaving LiDAR characteristics and simulator-to-real traffic-agent behavior as unresolved gaps.

5 FUTURE TRENDS

The survey identifies future directions centered on adapting driving models beyond training distributions, improving interpretable planning frameworks and data engines, and developing driving foundation models. It emphasizes world models and sophisticated future-oriented objectives rather than directly applying language-model generation to driving control.

  • Generalization: Adapting end-to-end driving models to unseen target domains with limited or no labels requires formalizing the task and incorporating zero-shot or few-shot learning.
  • Planning Frameworks: Modular end-to-end planning frameworks jointly optimize multiple modules while prioritizing the ultimate planning task and retaining interpretability.The survey highlights unresolved design questions concerning loss functions for differentiable perception modules.
  • Data Engines: Autonomous-driving data engines should automate high-quality perception labeling and streamline iterative development of data and models.Large perception models are identified as potential aids for automatic annotation in modular end-to-end planning systems.
  • Foundation Models: Foundation driving models may use world models that forecast plausible environmental futures in 2D, 3D, or latent space.The survey argues that downstream planning requires objectives beyond frame-level perception, while direct LLM adoption may be unsuitable for steady accurate measurements.

6 CONCLUSION AND OUTLOOK

The survey concludes that modular systems still face significant challenges in complex environments such as inner-city streets and intersections, motivating growing interest in end-to-end driving. It identifies high-quality data, large-scale training, and reliable benchmarks as central to realizing the approach’s potential while pursuing generalist agents.

  • Conclusion and Outlook: Modular-based systems face significant challenges in complex scenarios such as inner-city streets and intersections.
  • Conclusion and Outlook: Companies are increasingly exploring end-to-end autonomous driving for complex environments, where the approach is envisioned to offer substantial performance and effectiveness potential.The survey connects this outlook with extensive high-quality data collection, large-scale model training, and reliable benchmarks.
  • Conclusion and Outlook: End-to-end autonomous driving presents major opportunities and challenges while pursuing the broader goal of building generalist agents.
Loading 2306.16927v3…