PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey

Pranav Singh Chib, Pravendra Singh

arXiv:2307.04370v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Modular autonomous-driving systems face complexity and error-propagation drawbacks, motivating End-to-End alternatives. This survey synthesizes End-to-End deep-learning methods across the driving stack, modalities, learning approaches, evaluation, safety, and explainability, and compiles related datasets and simulators. It concludes by organizing recent advancements and identifying challenges and future research possibilities.

  • Problem

    Modular autonomous-driving pipelines face complexity and error-propagation drawbacks, motivating investigation of End-to-End alternatives.

  • Method

    The paper conducts a comprehensive survey and taxonomy of End-to-End deep-learning autonomous driving across modalities, learning methods, evaluation, safety, explainability, datasets, and simulators.

  • Results

    The survey organizes recent End-to-End advancements, summarizes evaluation resources, and discusses proposed solutions for safety and explainability.

  • Takeaways & Limitations

    The paper provides a structured account of End-to-End autonomous-driving research, including its methodologies, resources, safety considerations, explainability approaches, and open challenges.

  • Takeaways & Limitations

    Direct policy learning continuously requires expert access during training, which is costly and inefficient.

Abstract

from arXiv · show

End-to-End driving is a promising paradigm as it circumvents the drawbacks associated with modular systems, such as their overwhelming complexity and propensity for error propagation. Autonomous driving transcends conventional traffic patterns by proactively recognizing critical events in advance, ensuring passengers' safety and providing them with comfortable transportation, particularly in highly stochastic and variable traffic settings. This paper presents a comprehensive review of the End-to-End autonomous driving stack. It provides a taxonomy of automated driving tasks wherein neural networks have been employed in an End-to-End manner, encompassing the entire driving process from perception to control, while addressing key challenges encountered in real-world applications. Recent developments in End-to-End autonomous driving are analyzed, and research is categorized based on underlying principles, methodologies, and core functionality. These categories encompass sensorial input, main and auxiliary output, learning approaches ranging from imitation to reinforcement learning, and model evaluation techniques. The survey incorporates a detailed discussion of the explainability and safety aspects. Furthermore, it assesses the state-of-the-art, identifies challenges, and explores future possibilities. We maintained the latest advancements and their corresponding open-source implementations at https://github.com/Pranav-chib/Recent-Advancements-in-End-to-End-Autonomous-Driving-using-Deep-Learning.

1. Introduction

End-to-End autonomous driving is presented as an alternative to modular pipelines, aiming to reduce error propagation and computational complexity through direct sensory-to-control learning. The survey reviews recent methods, modalities, learning approaches, evaluation frameworks, safety, explainability, datasets, and simulators.

  • Motivation: Modular autonomous driving divides perception, planning, and control into interconnected subtasks, creating error-propagation, complexity, and computational-efficiency challenges.Perception errors can propagate into planning and control, while separate module processing introduces additional computation.
  • Motivation: End-to-End driving directly maps sensory input to control outputs, streamlining the pipeline and reducing task-specific computation.The approach uses a single learning task pipeline rather than separately trained modules.
  • Survey scope: The survey organizes End-to-End research by underlying principles, methodologies, modalities, learning approaches, and functionality.Its organization covers sensory inputs, outputs, imitation and reinforcement learning, domain adaptation, safety, and explainability.
  • Survey scope: It investigates recent state-of-the-art developments, provides quantitative analysis, and proposes open- and closed-loop evaluation coverage.The survey also compiles datasets and simulators for autonomous-driving research.
  • Modular pipeline: The modular pipeline processes raw sensory data through perception, localization, planning, and control before producing vehicle commands.Its standard components include obstacle detection, localization, trajectory planning, and motor control.
  • Modular pipeline: Trajectory-prediction approaches span physics-based, classical machine-learning, and deep-learning methods with different forecasting capabilities and computational costs.Physics-based methods offer low-cost simulation, classical methods use historical motion data, and deep learning supports broader prediction horizons.

3. End-to-End system architecture

Modular systems connect discrete perception, localization, mapping, planning, and control components, whereas End-to-End systems directly map sensory input to ego-motion or control. The survey frames End-to-End driving around direct sensorimotor learning through imitation or reinforcement learning.

  • Modular driving pipelines process raw sensory data through perception, localization, mapping, planning, and vehicle control before generating commands.
  • End-to-End driving bypasses perception and planning subtasks by directly generating ego-motion from sensory input.The approach is described as enabling continuous learning to sense and act.
  • End-to-End models can use camera, LiDAR, or other sensory modalities as inputs for driving outputs.
  • End-to-End driving is commonly trained either with reinforcement learning or supervised imitation learning from expert demonstrations.Imitation learning aims to resemble human driving behavior, while reinforcement learning explores driving policies.

4. Input modalities in End-to-End system

End-to-End systems use cameras, LiDAR, multimodal sensor combinations, and navigation information as inputs. Fusion may occur early, at intermediate feature levels, or through transformer-based integration of complementary views.

  • Camera: Camera-based End-to-End models use monocular or stereo images, sometimes combined with vehicle speed, to generate driving actions or instructions.Camera-only systems have been used for lane following, turning, stopping, and going straight through imitation learning.
  • LiDAR: LiDAR provides 3D point clouds, accurate distance estimates, and spatial information for BEV, HD-map, and SLAM representations.The sensor is described as resistant to lighting conditions and useful for feature mapping and driving-policy learning.
  • Multimodal inputs: Multimodality combines sensor data and is reported to outperform single-modality input in crucial perception tasks.
  • Fusion strategies: Sensor fusion can occur before the learnable system through early fusion or after preprocessing and feature extraction through mid-fusion.Transfuser uses self-attention layers and transformer modules to fuse image and LiDAR features at multiple resolutions.
  • Navigation inputs: Navigation inputs include route-planner endpoints, discrete commands, destination locations, and lane or turn directives.These inputs guide control actions or driving policies alongside sensory information.

5. Output modalities in End-to-End system

End-to-End systems primarily output control commands, waypoints, or trajectories, while auxiliary outputs add representations such as maps and occupancy estimates. Higher-level waypoint and trajectory outputs can be converted into low-level vehicle controls.

  • End-to-End systems output control commands, waypoints, or trajectories, and may additionally produce cost maps and auxiliary representations.
  • Waypoints and trajectories: Waypoint and trajectory predictions represent future vehicle motion and can be converted into steering and acceleration using MPC or PID controllers.Longitudinal control uses waypoint-vector magnitude, while lateral control uses waypoint direction.
  • Model examples: Representative models combine inputs and outputs differently, including waypoint prediction, PID-driven waypoints, control and trajectory branches, future trajectories, attention masks, and cost maps.ST-P3 uses occupancy probability and prior knowledge to balance safety considerations for the final trajectory.
  • Waypoints and trajectories: Waypoints are easier for controllers to analyze for steering and are not affected by vehicle geometry.Continuous waypoints can also be transformed into a specific trajectory.
  • Trajectory selection: Trajectory costs assign weights to candidate trajectories using factors such as safety, distance traveled, and comfort to select an option.The cost is used to choose among multiple safe maneuvering possibilities.
  • Auxiliary outputs: Auxiliary outputs include segmentation maps, BEV maps, future occupancy, and interpretable feature maps that support additional functionality and representation learning.

6. Learning approaches for End-to-End system

End-to-End driving is learned primarily through imitation or reinforcement learning, with extensions that address policy optimization, distribution shift, and expert supervision. The survey covers behavioral cloning, direct policy learning, inverse reinforcement learning, reinforcement learning, and human-in-the-loop methods.

  • Imitation learning: Imitation learning trains driving policies from expert demonstrations to mimic expert behavior across driving scenarios.Large-scale expert datasets can support models intended to perform at human-like standards.
  • Behavioral cloning: Imitation-learning policies map states to actions while behavioral cloning minimizes imitation loss over expert state-action examples.Behavioral cloning treats expert state-action combinations as independently and identically distributed examples.
  • Behavioral cloning: Behavioral cloning can suffer distribution shift when actual observations diverge from training observations, while latent variables influence real-world driving actions.
  • Direct policy learning: Direct policy learning augments an initial expert dataset with trajectories collected by the trained policy and expert evaluations during runtime.Its stated drawback is the continuous need for expert access during training.
  • Direct policy learning: DAGGER accumulates additional training instances and is described as providing robustness against cascading errors.
  • Inverse reinforcement learning: Inverse reinforcement learning infers an underlying reward function from expert state-action demonstrations to replicate expert behavior.Feature-based and maximum-entropy variants are discussed for learning driving styles and reward functions.
  • Reinforcement learning: Reinforcement learning maximizes cumulative rewards through environment interaction and is presented as a way to address distribution shift by exploring scenarios.The survey contrasts this exploration with imitation learning’s dependence on the training dataset.
  • Reinforcement learning: Reinforcement-learning families include DQN, DDPG, A3C, SAC, TRPO, and PPO.

7. Learning domain adaptation from simulator to real

Simulator-to-real adaptation addresses substantial visual and instance-level disparities between virtual and real-world driving data. The survey describes domain adaptation, simulator-real interaction, and policies learned in simulation before deployment in real settings.

  • Domain gap: Virtual engines enable large-scale scenario and data collection, but disparities between virtual and real-world data complicate dataset creation and deployment.Domain adaptation transfers features from a simulator source domain to accurate real-world target data.
  • Domain adaptation: The H-Divergence framework aligns visual and instance-level domains by adversarially learning a domain classifier and detector simultaneously.
  • Domain adaptation: Simulator-real interaction strategies align global and local differences while maintaining consistency between the aligned components.
  • Policy transfer: Driving policies can be learned in realistic simulated settings and transferred to real-world deployment, while perception and control may use differently sourced data.One approach trains segmentation on annotated real-world data and the controller on synthetic images and their associated inputs.

8. Safety

Safety in End-to-End driving requires diverse critical-scenario training, explicit safety constraints, monitoring and recovery mechanisms, adversarial testing, and quantitative safety metrics. The survey presents these elements as complementary ways to promote robust behavior and evaluate risks.

  • Safety-oriented training: Training on diverse, high-quality data that includes rare and critical situations helps systems learn robust and safe behaviors.Examples include unprotected turns, occluded pedestrians, aggressive lane changes, and other safety-critical scenarios.
  • Safety constraints: Safety constraints can be incorporated during learning or output processing through safety costs, unsafe-maneuver avoidance, and collision-avoidance strategies.
  • Safety mechanisms: Additional safety modules and testing mechanisms are used to enhance system safety.
  • Monitoring and recovery: Planners and monitoring mechanisms identify collision-free routes, reduce infractions, compensate for failures, and return vehicles to safe states.Approaches include rule-based expert planning, long-horizon safe-trajectory prediction, goal planning, and stop intentions.
  • Testing: Adversarial attacks and End-to-End testing oracles evaluate control-signal correctness and robustness, while metamorphic testing checks steering consistency across conditions.
  • Safety metrics: TTC, CI, CPI, and TTR provide quantitative measures for comparing safety performance and identifying areas for improvement.

9. Explainability

Explainability methods for End-to-End driving are organized into local explanations of individual predictions and global explanations of overall model behavior. Existing work emphasizes local methods, while global explanations remain unavailable in this domain according to the survey.

  • Foundations: Explainability concerns how users understand relationships between model inputs and outputs, including interpretability and completeness.
  • Local and global explanations: Local explanations describe the rationale for individual predictions, whereas global explanations characterize the model’s underlying knowledge and behavior.
  • Local and global explanations: The survey states that no research on global explanations for End-to-End autonomous driving was available at the time of review.
  • Local explanations: Post-hoc saliency methods answer where by identifying influential visual regions or objects, including transformer attention-based representations in PlanT.
  • Local explanations: Counterfactual explanations answer what by modifying input features or semantic regions to produce a different prediction and expose influential factors.Semantic representations are described as more interpretable than pixel-level representations.
  • Local explanations: Auxiliary outputs such as semantic and occupancy maps support counterfactual understanding by representing scene structure that can influence planning maneuvers.Higher occupancy probabilities discourage maneuvers, while lower probabilities encourage them.
  • Global explanations: Global explanation approaches include translating models into interpretable decision trees, rules, or causal models, and analyzing learned representations through neuron activations and coverage.

10. Evaluation

End-to-End systems are evaluated through open-loop comparison with expert behavior and closed-loop testing in interactive environments. Open-loop evaluation is faster and useful for isolated behaviors, while closed-loop evaluation tests route completion, infractions, collisions, and adaptation to unexpected scenarios.

  • Open-loop evaluation: Open-loop evaluation uses datasets such as KITTI and nuScenes to compare predicted driving behavior with expert actions.Reported metrics include MinADE, MinFDE, L2 error, and collision rate.
  • Open-loop evaluation: Open-loop evaluation enables faster initial assessment and testing of specific components or behaviors in isolation.
  • Open-loop limitations: Open-loop benchmarks cannot generalize to wider geographical distributions because they inherit limitations from their datasets.
  • Closed-loop evaluation: Closed-loop evaluation tests systems in unanticipated environments through route completion tasks containing scenarios such as crossing pedestrians and sudden lane changes.
  • Closed-loop metrics: Route Completion measures completed route distance, Infraction Score aggregates penalties, and Driving Score multiplies route completion by infraction penalty.
  • Closed-loop metrics: Closed-loop evaluation measures collisions, traffic-light and stop-sign infractions, off-road behavior, and other safety outcomes.It also supports customized configurations and sensor settings for dynamic testing.

11. Datasets and simulator

End-to-End driving depends heavily on rich, modality-diverse datasets and advanced simulation because real-world testing is expensive and restricted. The survey compares simulators used to create flexible virtual environments for training and evaluation.

  • Datasets: Training-data quality and richness strongly influence End-to-End model performance, with datasets organized by camera, LiDAR, GNSS, and vehicle-dynamics modalities.The summarized dataset content includes urban driving and traffic scenarios.
  • Simulators: Advanced software simulators support testing and training because End-to-End experiments are expensive and public-road testing is heavily restricted.Simulation environments allow algorithms or modules to be tested before road deployment.
  • Simulator platforms: MATLAB/Simulink supports varied settings, efficient plotting, and co-simulation with platforms such as CarSim.These capabilities simplify creation of different experimental settings.
  • Simulator platforms: CARLA and LGSVL provide high-quality simulation frameworks with customizable sensors, map-generation APIs, and driving-stack bridge connections.Both require GPU processing for decent operating speed and frame rate; CARLA uses Unreal Engine and LGSVL uses Unity.

12. Future research directions

Future End-to-End driving research centers on learning robustness, behavioral safety, interpretability, collaborative perception, and large vision models. The survey also emphasizes more effective evaluation and continued development of supporting datasets and simulators.

  • Learning principles: Distribution shift in imitation learning and computational instability in reinforcement learning remain central learning challenges.Reinforcement learning trains through simulated-environment interaction, whereas imitation learning learns from expert agents without extensive environmental interaction.
  • Safety: Behavioral safety and accurate prediction of uncertain behaviors are key priorities for reliable and comfortable End-to-End transportation.Future work should refine safety constraints and improve their effectiveness to support wider adoption.
  • Explainability: Interpretability research should develop global explanations for model actions leading to failures and improve feedback mechanisms for users.The survey reports promising joint improvements in performance and explainability from interpretable-feature approaches, while emphasizing that further exploration is required.
  • Collaborative perception: Collaborative perception lets vehicles jointly assess scenes beyond individual line of sight and field of view, addressing occlusion and limited receptive fields.Future work should improve transmission efficiency while safeguarding data privacy.
  • Large vision models: Future large vision models should be tailored to autonomous driving, with prompt fine-tuning for perception and control tasks.SAM-Track and VIMA are cited as examples showing potential for visual understanding and robot action manipulation.

13. Conclusion

The survey organizes recent End-to-End autonomous-driving advances across modalities, learning, training, evaluation, safety, and explainability. It compiles relevant datasets and simulators while concluding that safety and interpretability still require improvement for broader technology acceptance.

  • Survey scope: The survey develops a taxonomy spanning modalities, learning approaches, training methodology, and open- and closed-loop evaluation.It also investigates domain adaptation and compiles advancements, public datasets, and simulators.
  • Safety and explainability: The paper reviews proposed solutions for safety and explainability alongside recent End-to-End autonomous-driving developments.These topics are treated as part of the survey's broader assessment of the field.
  • Conclusion: Despite impressive End-to-End performance, continued work on safety and interpretability is needed for broader technology acceptance.
Loading 2307.04370v2…