PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of End-to-End Driving: Architectures and Training Methods

Ardi Tampuu, Maksym Semikin, Naveed Muhammad, Dmytro Fishman, Tambet Matiisen

arXiv:2003.06404v2cs.AIcs.RO

TL;DR

End-to-end autonomous driving replaces a conventional modular pipeline with a neural network, motivating a dedicated synthesis of its methods and design choices. The survey reviews learning, modalities, architectures, and evaluation, finding that interpretability, safety, and reliable evaluation remain open challenges.

  • Problem

    A dedicated review of end-to-end autonomous driving approaches was missing despite substantial research and reported promise.

  • Method

    The paper surveys end-to-end driving literature across learning methods, input and output modalities, architectures, and evaluation schemes.

  • Results

    Open-loop metrics only loosely predict closed-loop driving performance; mean squared error correlates with closed-loop success rate at r = 0.39.

  • Takeaways & Limitations

    The survey identifies design options for end-to-end systems while highlighting interpretability, safety, and evaluation as continuing challenges.

Abstract

from arXiv · show

Autonomous driving is of great interest to industry and academia alike. The use of machine learning approaches for autonomous driving has long been studied, but mostly in the context of perception. In this paper we take a deeper look on the so called end-to-end approaches for autonomous driving, where the entire driving pipeline is replaced with a single neural network. We review the learning methods, input and output modalities, network architectures and evaluation schemes in end-to-end driving literature. Interpretability and safety are discussed separately, as they remain challenging for this approach. Beyond providing a comprehensive overview of existing methods, we conclude the review with an architecture that combines the most promising elements of the end-to-end autonomous driving systems.

I. INTRODUCTION

Autonomous driving research contrasts modular pipelines with end-to-end systems that learn driving decisions as one task. This survey addresses the lack of a dedicated review while examining design choices, benefits, and unresolved interpretability and safety challenges.

  • Modular systems connect self-contained perception, localization, planning, and control modules, making faults easier to identify but costly to build and maintain.
  • End-to-end driving directly maps sensory inputs to steering and acceleration commands through one learned task, typically using imitation or reinforcement learning.
  • This work presents itself as the first survey focused solely on end-to-end autonomous driving.
  • The survey organizes end-to-end systems by learning methods, inputs, outputs, architectures, and evaluation schemes to expose design choices step by step.
  • End-to-end models avoid human-defined information bottlenecks and can learn task-specific intermediate representations from the driving objective.
  • End-to-end optimization makes driving decisions harder to explain and errors harder to trace, while neural-network susceptibility to adversarial attacks raises safety concerns.

III. LEARNING METHODS

End-to-end driving relies mainly on imitation learning, but training must address distribution shift and limited coverage of rare situations. Data augmentation, diversified collection, and trajectory perturbation expand recovery examples, while reinforcement learning offers online exposure to situations encountered during training.

  • Imitation learning trains a model to mimic human driving commands from recorded sensory inputs and works well for simple tasks such as lane following.
  • Rare or complicated traffic scenarios remain challenging for imitation learning because recorded expert demonstrations provide limited coverage.
  • Distribution shift occurs when the model encounters observations caused by its own actions rather than the expert’s driving trajectory.
  • 2) Data diversification:: Data augmentation generates shifted, rotated, blurred, cropped, brightness-adjusted, or noisy images paired with commands for recovering from deviations.
  • 2) Data diversification:: Injecting noise during data collection forces the expert to react to off-trajectory states, though this diversification can be dangerous with real vehicles.
  • 2) Data diversification:: ChauffeurNet uses top-down semantic images and synthetic trajectory perturbations, including collisions and road departures, to train a more robust policy.

3) On-policy learning:

On-policy and related training methods expose driving models to states produced by their own behavior, while dataset balancing and reinforcement learning address coverage, bias, and exploration challenges. These methods trade reduced distribution shift or human labeling against expense, instability, or data inefficiency.

  • 3) On-policy learning:: DAgger alternates model and expert control during data collection so the expert labels states produced by the model’s own errors.
  • 3) On-policy learning:: Keeping an expert in the loop is expensive because the possible ways for a driving policy to fail are unlimited.
  • 3) On-policy learning:: ChauffeurNet perturbs semantic trajectories by shifting an agent location and fitting a path back toward the lane center.
  • 3) On-policy learning:: Automatic controllers and privileged algorithmic experts can reduce or replace human annotations during on-policy imitation learning.
  • Dataset balancing counters overrepresented straight-driving examples and spurious correlations by upsampling rare steering angles, downsampling common ones, or weighting samples.
  • Reinforcement learning learns from rewards through online interaction, avoiding human-labeled recordings but generally requiring less data-efficient training than imitation learning.
  • Initializing reinforcement learning with an imitation-trained policy reduces long training times and helps address imitation learning’s off-policy problem.

1) Rewards:

End-to-end driving uses imitation or reinforcement learning, with reward design shaping reinforcement-learning behavior. Simulation and domain-adaptation methods address safety, transfer, and differences between simulated and real inputs.

  • Rewards: Reinforcement learning maximizes rewards, so positively and negatively rewarded events directly influence the learned driving behavior.Simpler reward schemes are easier to interpret, while more complicated schemes define desired behavior more explicitly.
  • Rewards: Common rewards encourage speed toward the goal or along the road, while penalties discourage deviation from a track or reference trajectory.Crashes may be implicitly avoided through discounted future reward or explicitly assigned a large negative reward.
  • Rewards: Real-world reinforcement-learning exploration can use a safety controller that takes over when the policy deviates excessively from the road.Riedmiller et al. used an analytically derived steering controller to prevent excessive track-center deviation.
  • Rewards: In CARLA, imitation-learning models and a classical modular pipeline outperformed the evaluated reinforcement-learning method.This comparison followed more extensive training of the reinforcement-learning model.
  • Rewards: Simulation-to-real transfer can suffer from input-distribution differences, motivating supervised fine-tuning, unsupervised input adaptation, or shared representations.Shared semantic-segmentation inputs transferred a CARLA-trained policy to a 1/5-sized real-world truck.
  • Rewards: Two variational autoencoders can map simulated and real images into a shared latent space, enabling a simulation-trained steering policy to operate on real images.The method required no paired simulation and real-world images and outperformed other tested domain-adaptation methods on a real vehicle.

IV. INPUT MODALITIES

End-to-end driving models commonly combine visual inputs with temporal information and vehicle-state measurements. Camera configuration and auxiliary representations affect capability, while speed inputs can introduce an inertia problem.

  • A. Camera vision: Vision is central to end-to-end driving, and multiple modalities are often combined because complementary inputs can improve generalization and accuracy.Monocular vision can achieve good performance, although stereo cameras can help networks implicitly extract depth.
  • A. Camera vision: Temporal modeling uses multiple past frames to capture aspects of driving that single images do not provide.Past inputs can be processed through recurrent networks or fixed-window convolutional models.
  • A. Camera vision: Surround-view cameras improve imitation accuracy on turns and intersections, but show no improvement on highways.They provide coverage useful for lane changes and yielding at intersections.
  • A. Camera vision: Models may use semantic segmentation, depth, surface normals, optical flow, and albedo alongside RGB images.These representations can be precise in simulation or produced by specialized networks from real images.
  • A. Camera vision: Vehicle-state inputs commonly include current speed and acceleration, with speed especially useful when consecutive frames are not modeled.Using current speed to predict next-step speed can cause inertia, making the model reluctant to resume motion after stopping.

D. Navigational inputs

Navigational inputs specify where or how the vehicle should drive, ranging from momentary commands to visual routes and map-based representations. LiDAR offers complementary spatial information but requires specialized processing.

  • 1) Navigational commands: Navigational commands such as “go left,” “go right,” “go straight,” and “follow the road” guide point-to-point driving.Separate output branches performed better than treating the command as an additional categorical input, though branch switching scales poorly.
  • 1) Navigational commands: Models can also receive categorical driving-style commands, including lane position, following behavior, speed, aggressiveness, or caution.Personalized driving styles have been investigated as an extension of these conditional inputs.
  • 2) Route planner: Visual route representations support longer-term path guidance than momentary left, right, or straight commands.Route inputs include navigation-app screens, GPS-coordinate lists, and binary top-down route maps.
  • 2) Route planner: Route-planner screens produced higher imitation accuracy than models without them, while GPS coordinates were less effective than a route-planner screen.ChauffeurNet represents the desired route as a binary top-down image aligned with other inputs.
  • 3) Textual commands: Textual advice can be goal-oriented or descriptive and can improve expert-trajectory prediction compared with camera-only approaches.Examples include “Drive slowly” and “There is a pedestrian.”
  • E. LiDAR: LiDAR provides illumination-insensitive distance estimates, but point clouds require processing such as occupancy grids, sparse convolutions, PointNet, or BEV projection.Three-dimensional occupancy grids can be costly because voxel counts increase rapidly with spatial precision.

F. High-definition maps

End-to-end systems can incorporate high-definition maps as structured top-down inputs, combining detailed scene information with learned driving. Multiple modalities may be fused early, midway, or late, while temporal context remains important.

  • F. High-definition maps: End-to-end policies may operate with only camera input, whereas high-definition maps can be incorporated when detailed scene information is needed.Using HD maps represents a compromise between modular and fully end-to-end software stacks, sometimes called mid-to-mid.
  • F. High-definition maps: Top-down HD maps encode static road information and dynamic traffic information, including roads, lanes, intersections, signs, speed limits, traffic lights, and other agents.ChauffeurNet uses a perception module to detect and draw other agents on the map.
  • F. High-definition maps: Some systems augment driving datasets with affordances extracted from detailed maps, including 15 measures in the cited Drive360 extension.The authors characterize detailed maps as required for fully autonomous cars in that discussion.
  • Multi-modal fusion: Early fusion combines modalities before the learnable system, while middle fusion combines learned features after partial extraction.Inputs may need a common reference frame, scale, or dimensionality before early fusion.
  • Multi-modal fusion: Late fusion combines modality-specific outputs, with ensembling as a representative approach; early fusion is often computationally most efficient.Concatenating RGB and depth channels was the best-performing RGB-depth fusion solution in the cited comparison.
  • Multi-modal fusion: Late fusion was more efficient than early fusion for navigational commands, which switch between output branches.Middle fusion was used for a similar behavior-mode switching problem.
  • Temporal modeling: Multiple past inputs help address physical characteristics such as speed and acceleration that are not directly observable from a single camera image.CNN+RNN and fixed-window CNN architectures are two ways to incorporate temporal context.

V. OUTPUT MODALITIES

End-to-end driving models commonly predict immediate controls, but higher-level outputs such as waypoints, trajectories, cost maps, and affordances support planning and interpretability. Output choices range from direct commands to structured representations processed by controllers.

  • A. Steering and speed: Most models predict next-timestep steering and speed, or acceleration and braking, usually as a regression task.Steering angle can be replaced by inverse turning radius to avoid dependence on vehicle geometry, while PID controllers convert outputs into actuator commands.
  • A. Steering and speed: Outputting acceleration and steering angular acceleration in addition to speed and steering improves smoothness and reported driving performance.Temporal consistency of commands is another strategy for smoothing behavior.
  • A. Steering and speed: Mean absolute error is increasingly preferred to mean squared error because it correlates better with actual driving performance.This choice has been reported across several end-to-end driving studies.
  • B. Waypoints and trajectories: Waypoints or trajectories force the model to plan ahead, remain independent of vehicle geometry, and provide outputs that are easier to interpret than momentary commands.Another trainable network or a controller such as PID can transform them into low-level controls.
  • C. Cost maps: Cost maps represent safe driving regions over future timesteps, after which trajectory selection and low-level control can be performed using MPC.The cited approach predicts top-down 2D cost maps from LiDAR and HD maps and trains them using human expert trajectories.
  • D. Affordances: Direct perception predicts human-understandable affordances, such as lane position, distances, speed signs, traffic lights, and unexpected agents, sometimes conditioned on navigational commands.Affordances may be discrete or continuous and conditional or non-conditional.

E. Multitask learning

End-to-end driving requires evaluation beyond offline prediction error because actual driving performance emerges when the model controls a vehicle. Closed-loop measures directly assess behavior, while selected open-loop metrics can help prioritize models for testing.

  • E. Multitask learning: Multitask learning jointly predicts driving outputs and side-tasks such as object detection or motion prediction from shared intermediate representations.Reported results indicate that optimizing these side-tasks produces more robust models and may help explain decisions and failures.
  • VI. Evaluation: Open-loop evaluation compares model predictions with recorded human decisions on held-out data without allowing the model to drive.Common metrics include mean absolute error and mean squared error, alongside other output-based measures.
  • VI. Evaluation: Closed-loop evaluation gives the model control in a real or simulated scenario and quantifies resulting behavior through driving metrics.Simulation makes repeated closed-loop testing easier and less costly than real-world deployment.
  • VI. Evaluation: Closed-loop metrics include successful-trial percentage, infractions, distance between infractions, lane or off-road time, autonomy percentage, and distance traveled toward the goal.These metrics directly measure autonomous driving ability rather than similarity to recorded actions.
  • VI. Evaluation: Good open-loop performance does not necessarily imply good closed-loop driving: MSE correlates weakly with success rate (r = 0.39), whereas MAE, quantized classification error, and thresholded relative error each achieve r > 0.6.Balanced-MAE has also been reported to correlate better with closed-loop performance than simple MAE and can guide model selection.
  • VI. Evaluation: Real-traffic collision-free driving is the most relevant quality measure, but safety comparisons also use publicly reported collisions and disengagements.The paper notes that real-world testing is costly and dangerous, motivating simulation and indirect evaluation practices.

VII. INTERPRETABILITY

Interpretability methods inspect which inputs influence end-to-end driving decisions and how attention is allocated. These tools include post-hoc visualization, built-in attention mechanisms, and input-specific saliency analysis.

  • VII. Interpretability: Interpretability is needed to understand why an end-to-end model drives as it does, especially when diagnosing failures and avoiding recurrence.Neural networks perform complex hierarchical computations, so specialized analysis methods are used to inspect them.
  • VII. Interpretability: Sensitivity analysis uses input gradients and their magnitudes to identify regions to which a model is most responsive.The passage describes this as a common approach for interpreting convolutional neural networks.
  • VII. Interpretability: VisualBackProp efficiently identifies image pixels influencing driving decisions, and shifting salient regions changes steering predictions nearly as much as shifting the whole image.It also applies to segmented images and 2D LiDAR projections.
  • VII. Interpretability: Textual commands can alter which input regions receive attention, linking user instructions to the model’s visual processing.The cited example uses textual commands as additional driving-policy input.
  • VII. Interpretability: Visual attention predicts spatial masks during learning and uses them to scale inputs or convolutional feature maps according to where the model should attend.The approach is demonstrated across visual input types, including RGB, semantic, and LiDAR representations.

B. Intermediate representations

Intermediate representations can improve interpretability, robustness, and task supervision, but end-to-end systems remain constrained by distribution shift, limited data diversity, adversarial vulnerability, and safety requirements.

  • B. Intermediate representations: Semantic inputs allow separate investigation of errors in scene prediction and errors in the driving model built on those representations.Semantic representations have also been associated with improved generalization.
  • B. Intermediate representations: Affordances and waypoints are human-understandable outputs that make questionable driving decisions easier to analyze and localize.Affordances can feed hard-coded controllers, while waypoints provide an intuitive description of intended motion.
  • B. Intermediate representations: Auxiliary outputs share intermediate representations with the main decision branch and can help both representation learning and comprehension of driving decisions.Examples include ego-speed, drivable area, and positions and speeds of other objects.
  • B. Intermediate representations: Safety testing is limited because proving rare accidents requires millions of driving hours and available test data rarely covers sufficient environmental diversity.Models can fail under changed lighting or combinations of unseen locations, weather, and dense traffic, including in simulation.
  • B. Intermediate representations: End-to-end models are not inherently protected against adversarial inputs such as stickers or projected images, so deployment requires specific countermeasures.The cited examples concern attacks on traffic signs and roads.
  • B. Intermediate representations: Simulation remains the main safety-testing environment because real-world testing is expensive, statistically difficult, and dangerous for other traffic participants.Simulation enables many short repetitions of critical situations.
  • B. Intermediate representations: A safety module or uncertainty estimate can detect situations the driving model may not handle and transfer control to a person or another algorithm.The paper mentions Bayesian approaches for estimating uncertainty in model outputs.
  • B. Intermediate representations: Passenger comfort is distinct from safety and can be improved by reducing longitudinal and lateral jerk through temporally smooth control losses.Comfort-oriented speed and steering should depend on each other rather than being treated as independent outputs.

A. Architectures

End-to-end driving spans diverse input, output, fusion, and auxiliary-task designs, with architecture choices shaped by affordability, planning horizon, and interpretability. The survey also highlights training and generalization considerations when selecting among these designs.

  • Design space: Architecture design spans promising choices for inputs, outputs, fusion techniques, and auxiliary tasks.The survey narrows the large space of possible end-to-end driving models by organizing these design dimensions.
  • Input modalities: Camera-only systems can improve affordability, whereas LiDAR and HD maps add sensor, mapping, repair, and insurance costs.These costs place LiDAR- and map-based end-to-end systems in the same price range as modular approaches.
  • Input modalities: A 360-degree vehicle view is useful for complicated maneuvers such as lane changing and giving way at intersections.Stereo vision for depth estimation is less commonly used because its usefulness is limited to close proximity, around 10m.
  • Input modalities: Map-image navigation instructions define intended routes more precisely over longer time scales than categorical commands such as “turn left.”Categorical commands can be confusing or arrive too late when multiple roads require the same turn direction.
  • Outputs: Predicting future waypoints explicitly asks models to plan ahead and produces outputs that are easier to visualize than instantaneous low-level commands.Instantaneous commands do not guarantee or directly measure long-term planning, whereas waypoints can be smoothed into trajectories.
  • Outputs: A series of future cost maps lets a planner select trajectories minimizing cost across multiple future time points.The approach supports motion planning many seconds ahead rather than estimating only an instantaneous cost map.
  • Training and evaluation: Imitation learning dominates end-to-end driving, but architecture and training-data effects are difficult to disentangle across studies.The survey also notes that network training and inference times are often not reported.
  • Training and evaluation: Online training can avoid imitation-learning distribution shift and has performed remarkably well in CARLA and NoCrash tasks.In real-world deployment, however, constructing an expert policy queryable in every state remains complicated.

C. Evaluation

Evaluation should measure closed-loop generalization across unseen environments, conditions, and traffic densities, while recognizing major limitations in safety testing and benchmark comparability. The survey proposes diverse testing conditions and presents an illustrative architecture combining multimodal inputs, temporal processing, and auxiliary tasks.

  • Evaluation metrics: Balanced-MAE correlates more strongly with driving ability than MAE or MSE, suggesting that training metrics should reflect closed-loop deployment.Off-policy imitation learning optimizes open-loop performance even though models are deployed closed-loop.
  • Generalization: Model performance in simulation does not degrade drastically in new cities or unseen weather, but generalization drops sharply as traffic density increases.
  • Benchmark comparability: Comparisons across models are unreliable when testing conditions and baselines differ, especially outside CARLA, so readers should inspect the evaluation setting closely.Reported safety measures such as miles per disengagement or miles per accident also omit the locations and conditions in which driving occurred.
  • Recommended evaluation: Future comparisons should use closed-loop evaluation across diverse locations, weather, lighting, and traffic densities, reporting CARLA and NoCrash results when applicable.
  • Candidate architecture: The candidate architecture combines multiple sensor inputs, optional costly modalities, navigational conditioning, feature fusion, recurrent temporal processing, and outputs such as actuation, waypoints, or cost maps.Auxiliary-task optimization is intended to shape camera features toward more semantic representations.
  • Rare situations: Rare traffic situations remain difficult because datasets overrepresent repeated common behaviors and naive average-risk training underweights atypical input-output combinations.The survey argues that rarity depends on the joint distribution of inputs and outputs, motivating broader sampling methods.
  • Safety testing: Real-world unusual situations are difficult to test reliably, and simulation success does not directly translate into safe real-world driving.End-to-end systems require black-box testing, while more realistic simulation and improved domain adaptation remain necessary.

APPENDIX A

The appendix organizes recent end-to-end driving contributions, CARLA benchmarks, NoCrash results, and publicly useful datasets. It highlights that standardized benchmarking is concentrated in CARLA, while many real-world datasets remain unavailable to the wider community.

  • Recent contributions: Recent contributions are selected mainly for real-world testing or closed-loop simulation evaluation, with cross-entropy and Gaussian mixture model abbreviations used in the summary table.
  • Benchmark scope: Only CARLA provides standardized benchmarking, whereas models trained in other simulators or on real-world data lack clear comparison baselines.
  • CARLA benchmarks: CARLA benchmark summaries evaluate generalization in an unseen town, across seen or unseen weather, and with or without dynamic objects.
  • NoCrash benchmark: The NoCrash results report performance in Town 2 because that town was not observed during training and therefore offers a fairer measure of generalization.
  • Datasets: Real-world datasets are rarely public, limiting researchers who cannot collect their own data; the appendix therefore lists datasets available to the wider community.
Loading 2003.06404v2…