PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform

Chengkai Xu, Yiming Cui, Jiaqi Liu, Yicheng Guo, Cheng Qin, Geyuan Zhang, Xinwei Dong, Shiyu Fang, Peng Hang, Jian Sun

arXiv:2610.00926v1cs.ROcs.CVcs.LG

TL;DR

E2E-AD performance depends on a training ecosystem, but its data, strategy, and platform requirements are interdependent. This survey reviews that ecosystem using a Data-Strategy-Platform taxonomy. It concludes that coordinated evolution across the layers is central to capability development and outlines shifts toward value-focused, foundation-driven, continuous training.

  • Problem

    E2E-AD effectiveness depends on whether its training ecosystem supplies sufficient data coverage and quality, aligned optimization strategies, and scalable platforms.

    Source: §I · pp. 1–2

  • Method

    The survey introduces a Data-Strategy-Platform taxonomy and reviews advances across data pipelines, learning paradigms, and training infrastructures.

    Source: §I · pp. 1–2

  • Results

    The review concludes that E2E-AD reliability depends on synchronized evolution of data quality, strategic alignment, and platform scalability rather than isolated advances in one layer.

    Source: §VII · p. 14

  • Takeaways & Limitations

    The paper’s vision shifts from data quantity to value, task-specific learners to foundation models, and static offline training to integrated training-testing.

    Source: §I · p. 2

    World-model policy training remains constrained by rollout fidelity, hallucination risks, and compounding latent-prediction errors over long horizons.

    Source: 3) World Models · p. 8

Abstract

from arXiv · show

Autonomous driving is a cornerstone technology for the future of intelligent transportation, where end-to-end learning has emerged as a transformative paradigm that directly maps multimodal sensory inputs to driving actions through unified differentiable models. While offering advantages, the effectiveness of end-to-end autonomous driving (E2E-AD) is ultimately determined by the quality of its training ecosystem. This paper provides a comprehensive review of training methods and ecosystem for E2E-AD. We introduce a Data-Strategy-Platform taxonomy that conceptualizes training as an interdependent system. The data layer defines what can be learned, the strategy layer governs how learning aligns with driving objectives, and the platform layer supports scalability and continuous evolution. Within this framework, we survey recent advances across data-centric pipelines, learning paradigms, and training infrastructures, and analyze their interplay in shaping model performance, robustness, and deployability. Finally, we reflect on current limitations and articulate a forward-looking vision that emphasizes a shift from data quantity to data value, from isolated optimization to foundation-driven generalization, and from static training to integrated training-testing loops, aiming toward robust, scalable, and trustworthy autonomous driving systems. We maintain a continuously updated repository tracking cutting-edge literature and works at \href{https://github.com/Jiaaqiliu/Awesome-Training-Ecosystem-for-E2E-AD}{Our Project Page}.

I. INTRODUCTION

The introduction positions E2E-AD training as an ecosystem challenge: model effectiveness depends on coordinated data, learning strategies, and platforms. The survey organizes its review around these layers and their interplay.

I. INTRODUCTION

  • Modular pipelines offer interpretability and engineering control but can accumulate errors across interfaces and struggle with global optimization in interactive mixed traffic.E2E-AD maps multimodal sensory inputs directly to driving actions through unified differentiable models, enabling joint optimization.Source: §I · p. 1
  • E2E-AD training effectiveness depends on data that cover safety-critical scenarios, strategies that align objectives, and platforms that support scale and lifecycle management.The paper says these dimensions jointly define the performance boundary of autonomous vehicles.Source: §I · p. 1
  • The survey distinguishes its focus on how data, learning strategies, and platforms jointly determine training effectiveness from surveys centered on datasets, modules, architectures, or particular learning paradigms.The introduction identifies existing surveys of datasets, perception, end-to-end architectures, imitation learning, and foundation models.Source: §I · p. 1
  • The survey introduces a Data-Strategy-Platform taxonomy to frame E2E-AD training as a system-level challenge, reviews advances across these dimensions, and analyzes their interplay.Its stated future vision shifts from data quantity to value, task-specific learners to foundation models, and static training to integrated training-testing.Source: §I · pp. 1–2

II. END-TO-END AUTONOMOUS DRIVING

The paper traces E2E-AD from direct control networks through structured differentiable approaches toward generative paradigms. It frames training challenges as systemic dependencies across data, strategy, and platform.

A. Development and Trends

  • E2E-AD has evolved from direct control networks to modular differentiable systems and generative approaches that model long-horizon interactions and counterfactual scenarios.The survey notes shifts in data priorities, training objectives, and evaluation toward scenario value, planning and safety consistency, and closed-loop benchmarks.Source: A. Development and Trends · pp. 2–3

B. Core Training Challenges

  • Training challenges span long-tail data scarcity, unstable optimization, and platform bottlenecks, whose dependencies require coordinated evolution across the ecosystem.Insufficient scenario coverage can amplify optimization instability, while fragile optimization increases infrastructure demands and platform inefficiencies restrict feasible data and strategies.Source: B. Core Training Challenges · p. 3
  • The three layers jointly determine the capability boundary: data sets what can be learned, strategy governs optimization, and platform supports reliable evolution.Fig. 2 depicts their systemic coupling.Source: Figure 2 · p. 3

III. DATA LAYER

The Data Layer defines learnable scenarios and signals through coverage, supervision, and value-guided governance. The review emphasizes prioritizing data and annotation according to their contribution to driving performance.

A. Sources and Scenario Coverage

  • Scenario coverage depends on planning-relevant interactions, not frame counts alone, and effective acquisition combines real-world grounding, controllable simulation, and curated synthetic data.World models can generate and replay counterfactual futures as data engines; virtual data requires calibration to preserve behavioral realism and limit distributional drift.Source: A. Sources and Scenario Coverage · pp. 3–4
  • Data sources include costly, stochastic real-world collections and virtual or synthetic sources that can expand rare or risky conditions but require attention to realism.Table II distinguishes earlier datasets emphasizing scale from more recent efforts prioritizing scenario value and rarity.Source: A. Sources and Scenario Coverage · pp. 3–4, Table II · p. 4

B. Supervision and Alignment of Labels

  • Manual labels offer high fidelity but are costly to scale, while automated labeling scales more readily but risks misalignment, temporal drift, and noise.The review proposes detailed multimodally aligned supervision for high-value scenarios and lower-fidelity supervision for common, low-impact cases.Source: B. Supervision and Alignment of Labels · pp. 4–5
  • Annotation should be value-guided because labeled scenarios contribute unequally to strategy improvement, especially when they expose decision ambiguity, complex interactions, or policy boundaries.Fig. 3 classifies scenarios by data value and highlights prioritizing high-value scenarios.Source: B. Supervision and Alignment of Labels · p. 4, Figure 3 · p. 5

C. Data Valuation and Governance

The review frames training as an ecosystem in which scenario value guides data allocation and complementary learning paradigms address driving objectives. It positions imitation learning as a useful policy bootstrap that benefits from further alignment and scenario imagination.

C. Data Valuation and Governance

  • Scenario value is defined as a scenario’s marginal contribution to the model’s current capability boundary, scored using risk, uncertainty, and distance from training data.The proposed value is dynamic: as model capability changes, difficult scenarios can lose utility and new edge cases can gain importance.Source: C. Data Valuation and Governance · p. 5
  • Data governance uses version trees to link data subsets to model checkpoints, making performance changes traceable to added or removed high-value scenarios.The review argues that prioritizing data by value can concentrate computational resources where they yield robustness benefits.Source: C. Data Valuation and Governance · p. 5

IV. STRATEGY LAYER

  • The strategy layer governs how E2E-AD maps multimodal perception to control, with diffusion policies, multimodal language models, and world models offering complementary strengths.The review presents these generative paradigms as a way to move beyond efficiency–robustness trade-offs in imitation and reinforcement learning.Source: §IV · p. 5

A. Imitation Learning

  • Imitation learning can bootstrap policies from expert demonstrations, but costly or noisy data, covariate shift, and underrepresented rare events constrain reliability and generalization.These limitations leave policies fragile in safety-critical conditions and can lead to compounding errors at deployment.Source: A. Imitation Learning · p. 5
  • Augmentation, expert corrections, distribution matching, and world-model rollouts extend imitation learning beyond passive trajectory replication, while the review positions it as a bootstrap rather than final alignment mechanism.The proposed ecosystem pairs imitation learning with reinforcement learning for safety alignment and generative paradigms for scenario imagination and rare-event robustness.Source: A. Imitation Learning · pp. 5–6

B. Reinforcement Learning

Reinforcement learning can refine driving policies toward long-horizon objectives, but direct application faces challenges in efficiency, safety, reward alignment, and generalization. The review describes mitigation strategies and a shift toward generative approaches that support reasoning, imagination, and adaptation.

B. Reinforcement Learning

  • Direct reinforcement learning faces sample inefficiency, unsafe exploration, reward misalignment, and cross-domain generalization challenges in autonomous driving.The review also notes that sim-to-real gaps amplify distribution shift.Source: B. Reinforcement Learning · p. 6
  • Offline and hybrid learning, world-model rollouts, safety constraints and shields, preference-based methods, and simulation-based generalization strategies target distinct reinforcement-learning challenges.The approaches address efficiency, safety, reward alignment, and transfer across environments.Source: B. Reinforcement Learning · p. 6
  • Reinforcement learning is positioned as a policy-alignment and refinement engine, often combined with world models or used to fine-tune imitation-learning and language-model policies.Direct application requires high-fidelity evaluation environments and guardrails for exploration and reward misspecification.Source: B. Reinforcement Learning · p. 6

C. Emerging Generative Paradigms

  • Generative paradigms move beyond reactive policy learning toward reasoning, imagination, and adaptation; the review’s diagram covers diffusion policies, multimodal language models, and world models.These approaches expand training toward reasoning, simulation, and scalable adaptation.Source: C. Emerging Generative Paradigms · pp. 6–7, Figure 4 · p. 7

1) Diffusion Policies:

Diffusion policies model diverse action distributions, making them useful for representing uncertainty and multiple safe futures, but sampling latency and safety integration constrain their deployment. Multimodal language models offer semantic reasoning and command-following capabilities, while temporal consistency, grounding, safety, and latency remain challenges.

1) Diffusion Policies:

  • Diffusion policies use denoising to generate trajectories or control sequences from noise, representing multiple feasible actions under uncertainty.Their distributional diversity can support plausible maneuvers in interactive mixed traffic where several safe behaviors may coexist.Source: 1) Diffusion Policies · p. 7
  • Iterative denoising raises training and sampling costs, while aligning generated trajectories with planning objectives, vehicle dynamics, and safety constraints remains difficult.Efficiency-oriented, guided, scenario-conditioned, and planner-oriented variants target sampling cost, safety-compliant generation, and kinematic constraints.Source: 1) Diffusion Policies · p. 7
  • Diffusion policies are best positioned as multimodal proposal and trajectory-planning components, especially when interaction uncertainty requires representing multiple safe futures.The review favors hybrid pipelines and notes constraints in latency, hard-constraint controllability, and safety certification.Source: 1) Diffusion Policies · p. 7

2) Multimodal Large Language Models:

  • Multimodal language models connect perception with language and vehicle actions, with pretraining supporting transfer, command reasoning, interpretability, and downstream task adaptation.The review presents them as semantic decision modules that can provide intermediate representations to planners and controllers.Source: 2) Multimodal Large Language Models · pp. 7–8
  • Direct multimodal-language-model deployment is constrained by temporal inconsistency, limited low-level grounding, hallucination and safety risks, and inference latency.The review notes that most pretrained models are optimized for static image–text pairs rather than sequential sensor data and long-horizon reasoning.Source: 2) Multimodal Large Language Models · p. 8
  • Paired perception–language–action pretraining, instruction tuning, preference optimization, and visual chain-of-thought methods target grounding, command following, alignment, and reasoning transparency.The review positions multimodal language models as complements that condition downstream planners and controllers rather than unsafeguarded direct controllers.Source: 2) Multimodal Large Language Models · p. 8

3) World Models:

World models support proactive policy learning by simulating latent driving dynamics, but their practical reliability is limited by rollout fidelity and long-horizon error accumulation. Training large E2E-AD models also requires distributed infrastructure, with data parallelism offering scalability at the cost of memory and bandwidth pressure.

3) World Models:

  • World models use latent rollouts to reduce real-world sampling while supporting long-horizon optimization and exploration of safety-critical scenarios scarce in naturalistic data.Counterfactual generation makes them relevant to the long-tail problem in E2E-AD.Source: 3) World Models · p. 8
  • Imperfect latent predictions can diverge from real dynamics, hallucinate implausible states, and accumulate errors over long horizons, undermining policy reliability.These fidelity and stability concerns constrain practical use.Source: 3) World Models · p. 8
  • Predictive modeling, imagination-augmented reinforcement or imitation learning, and closed-loop adaptation are strategies for refining dynamics and using imagined experience in policy optimization.The approaches aim to make world models central to policy training rather than auxiliary simulators.Source: 3) World Models · p. 8
  • World models are positioned as imagination and evaluation backbones that generate, replay, and diversify valuable scenarios, with effectiveness bounded by physical, causal, and motion fidelity.The passage also describes their use as a joint optimization tool and as data or closed-loop simulation support for other paradigms.Source: 3) World Models · p. 8

A. Distributed and Parallel Training Platform

  • Distributed training addresses the memory and throughput limits of a single device, targeting scalability, communication-computation efficiency, stability, and reproducibility.The platform discussion identifies five complementary parallel paradigms, often combined in hybrid systems.Source: A. Distributed and Parallel Training… · pp. 8–9

1) Data Parallelism (DP):

  • Data parallelism replicates full models and synchronizes gradients with all-reduce, enabling near-linear throughput scaling but requiring per-device model and optimizer state and depending on network bandwidth.Gradient accumulation, compression, and delayed synchronization are cited as ways to alleviate bandwidth pressure.Source: 1) Data Parallelism (DP) · p. 9

2) Tensor Parallelism (TP):

The platform section surveys parallelization choices for distributing model, sequence, and expert computation across devices. Hybrid strategies balance memory, computation, and bandwidth demands as resource requirements rise from classical imitation learning toward generative approaches.

2) Tensor Parallelism (TP):

  • Tensor parallelism partitions operators or weights across devices to reduce memory pressure and support ultra-large models, while requiring fine-grained communication and operator-level fusion and scheduling.Mismatched partitioning or topology can amplify communication overhead.Source: 2) Tensor Parallelism (TP) · p. 9

3) Pipeline Parallelism (PP):

  • Pipeline parallelism divides a network into stages and streams micro-batches to reduce peak memory and improve deep-network utilization, but idle bubbles and load imbalance constrain effectiveness.Finer stage partitioning, micro-batch scheduling, and recomputation can mitigate these costs.Source: 3) Pipeline Parallelism (PP) · p. 9

4) Sequence Parallelism (SP):

  • Sequence parallelism distributes activations and contexts across time or token dimensions, extending Transformer context length while reducing long-sequence memory peaks, with boundary exchanges adding communication.It can be combined with data, tensor, and pipeline parallelism.Source: 4) Sequence Parallelism (SP) · p. 9

5) Expert Parallelism (EP):

  • Expert parallelism distributes sparsely activated feedforward experts across devices, while systems such as GShard and DeepSpeed MoE illustrate approaches to scaling sparse models.The described DeepSpeed MoE approach combines ZeRO with all-to-all communication optimization to reduce memory, communication, and latency costs.Source: 5) Expert Parallelism (EP) · p. 9

6) Hybrid Parallelism and Strategy Selection:

  • Hybrid parallelism combines data and tensor parallelism for parameter scaling with pipeline and sequence parallelism for deep architectures and long-context streams, while expert parallelism expands capacity through sparsification.Communication-aware partitioning balances memory, computation, and bandwidth across the cluster.Source: 6) Hybrid Parallelism and Strategy… · p. 9
  • Reported computational requirements trend upward from classical imitation-learning models manageable on single nodes to generative paradigms requiring massive multi-node clusters, widening the gap with vehicle-edge resources.The passage identifies distillation and quantization as future platform needs for deployment under edge constraints.Source: 6) Hybrid Parallelism and Strategy… · p. 9

B. Testing and Evaluation Platform

Testing platforms combine simulation and real-world validation to assess autonomous-driving performance, safety, and reliability. They need broader driving-relevant metrics, while scalability, realism, and safety remain challenges for testing at scale.

1) Simulation-Based Testing and Closed-Loop Validation:

  • Simulation platforms enable controlled, repeatable testing across predefined scenarios, including costly or dangerous edge cases, but simulation-to-real discrepancies can affect performance, especially on rare events.The passage cites CARLA, LGSVL, and NVIDIA DRIVE Sim as examples of high-fidelity simulators.Source: 1) Simulation-Based Testing and… · p. 10

2) Real-World Testing and Real-Time Evaluation:

  • Real-world testing exposes systems to unpredictable driving conditions, while closed-loop testing feeds real-time sensor data into model decisions and requires monitoring and safety mechanisms.Those safeguards may mask true performance, motivating hybrid virtual-real testing as a possible balance between safety and evaluation accuracy.Source: 2) Real-World Testing and Real-Time… · p. 10
  • The figure depicts a vehicle-to-cloud continuous-training loop: vehicles collect high-value events, cloud systems train federatively and distributively, and asynchronous updates return improved models for redeployment.The caption describes this cycle as supporting sustainable, adaptive model evolution.Source: Figure 5 · p. 10

3) Testing Metrics and Performance Validation:

  • Testing should assess safety, comfort, and efficiency with measures such as collision rates, emergency maneuvers, traffic-law compliance, and acceleration smoothness, rather than relying only on perception accuracy.The passage frames traditional detection or classification accuracy as insufficient for autonomous-driving tasks.Source: 3) Testing Metrics and Performance… · p. 10
  • Testing platforms face scalability, realism, and safety challenges, and massive testing can waste resources; scenario value is suggested as a way to accelerate testing and training.The passage identifies this as an important future research topic.Source: 3) Testing Metrics and Performance… · p. 10

C. Cloud–Edge Collaboration and Continuous Training Platform

Cloud-edge collaboration links vehicle-side event collection with cloud-supported training and asynchronous model updates, while continuous deployment requires governance and controlled rollout. The future-facing data agenda emphasizes scenario value and addresses shortcomings in coverage, annotation, and traceability.

C. Cloud–Edge Collaboration and Continuous Training Platform

  • Cloud-edge collaboration links vehicle-side event collection and lightweight processing with cloud computing for asynchronous incremental updates and cross-regional adaptation.The review presents this closed loop as supporting dynamic adaptation and continuous optimization after deployment.Source: C. Cloud–Edge Collaboration and… · p. 11

1) Online Collection and Edge Preprocessing:

  • Event-driven collection prioritizes high-marginal-value safety and robustness samples, with local filtering, redundancy removal, preprocessing, and prioritized upload to supplement long-tail data.The passage identifies emergency braking, sharp turns, close encounters, abnormal yielding, and perception divergence as example events.Source: 1) Online Collection and Edge… · p. 11

2) Asynchronous Updates and Model Governance:

  • Asynchronous training pipelines trigger incremental jobs, version candidates with data, configurations, and checkpoints, evaluate them offline before online validation, and use scheduling to improve throughput and timeliness.Clear promotion criteria are proposed to keep update rhythms predictable and results interpretable.Source: 2) Asynchronous Updates and Model… · p. 11

3) Gray Update and Rollback Mechanisms:

  • Gray release limits new models to selected fleets or regions before staged expansion, while rollback channels restore the previous stable version when degradation alerts or thresholds are reached.The passage describes this pairing as making trial and error controllable and preserving failure samples for later training.Source: 3) Gray Update and Rollback Mechanisms · p. 11

VI. OPEN CHALLENGES AND FUTURE DIRECTIONS

  • The proposed future direction coordinates higher data value density, foundation-model capabilities, and continuous platform evolution to support robust and scalable autonomous systems.The passage presents these three axes as mutually reinforcing parts of ecosystem-wide development.Source: §VI · p. 11

A. Scenario Prioritization and Data Engine: From “Data Volume” to “Value Density”

  • Scaling automated scenario-value scoring is an engineering challenge because current collection has coverage gaps and redundancy, annotation is inefficient and misaligned, and fragmented governance limits reproducibility.The review calls for value systems that prioritize risk, rarity, learnability, and coverage gaps, alongside controlled synthesis, improved annotation, and data-version governance.Source: A. Scenario Prioritization and Data… · pp. 11–12
  • Version trees that record data sources, filtering rules, and scope, and map data versions to model checkpoints, support historical traceability and scenario-level reproduction.The passage links this governance approach to improved problem localization and reduced ineffective repeated training.Source: A. Scenario Prioritization and Data… · p. 12

B. Universal Driving Foundation Models and Regional Adaptation: Strong Transfer, Weak Customization

The section envisions foundation models that transfer across cities and driving conditions, but describes sharp domain degradation and costly, unstable adaptation as current barriers. It proposes unified representations, lightweight updates, and forgetting controls as directions for more efficient regional customization.

B. Universal Driving Foundation Models and Regional Adaptation: Strong Transfer, Weak Customization

  • Foundation models should provide consistent multimodal, long-sequence representations across cities, road types, and driving cultures while aligning training objectives with planning and safety metrics.The section also calls for lightweight regional customization, stable cross-domain transfer without catastrophic forgetting, and interpretable decision-making.Source: B. Universal Driving Foundation Models… · p. 12
  • Current foundation models degrade in new domains, where differences in signage, rules, and driving habits can cause planning drift and make adaptation costly and difficult to assess.Existing adaptation methods rely on costly retraining or extensive annotation; performance instability, catastrophic forgetting, and unclear retraining criteria also impede deployment.Source: B. Universal Driving Foundation Models… · p. 12
  • Unified cross-modal and cross-temporal representations, combined with contrastive learning, domain adaptation, or causal representation learning, are proposed to improve transfer across cities and road conditions.The proposed direction aims to reduce re-annotation and training needs as road rules, signage styles, and driving habits change.Source: B. Universal Driving Foundation Models… · p. 12
  • Lightweight adaptation layers, prompt tuning, low-rank updates, or sparse adaptation are proposed for rapid regional updates using small-scale incremental data.These approaches are intended to reduce computational and storage overhead while maintaining performance.Source: B. Universal Driving Foundation Models… · p. 12
  • Replay learning, distillation, and regularization are proposed to suppress forgetting, alongside criteria for choosing comprehensive retraining versus smaller parameter adjustments.The coupled-system schematic links real-world failures to targeted data generation, model distillation for deployment, and strategy evolution for new scenarios.Source: B. Universal Driving Foundation Models… · pp. 12–13, Figure 6 · p. 13

C. Continuous Learning and Training Operations: Closed-Loop Training Solutions with Training-Testing Integration

Static datasets and one-time evaluations can leave offline performance disconnected from unpredictable real-world conditions. The section proposes integrated training and testing operations, while the conclusion emphasizes coordinated evolution across data, strategy, and platform.

C. Continuous Learning and Training Operations: Closed-Loop Training Solutions with Training-Testing Integration

  • Static datasets and one-time evaluations cannot capture the frequency and diversity of real-world changes, contributing to a gap between offline performance and generalization in unpredictable situations.The section identifies persistent separation between training and testing as a challenge as driving expands from closed sites to urban environments.Source: C. Continuous Learning and Training… · p. 13
  • Training-testing integration combines virtual-real data fusion, foundation-model distillation for vehicle deployment, and strategy evolution with forgetting suppression.Real-world corner cases are augmented through controllable generation, and evolutionary updates aim to improve performance without degrading previously learned capabilities.Source: C. Continuous Learning and Training… · p. 13
  • Unified lineage across training samples, model versions, and evaluation results is proposed to make performance traceable and version comparisons reproducible.The lineage should record data inputs, optimization configurations, and validation outputs for each update, supporting scenario-level reproduction and audit traceability.Source: C. Continuous Learning and Training… · p. 13
  • Closed-loop evaluation should assess stability, decision reasonableness, and risk sensitivity using high-risk case replay, counterfactual perturbations, and simulation comparisons.The stated aim is to connect training results more closely to online performance than accuracy metrics alone can.Source: C. Continuous Learning and Training… · p. 14
  • A reproducible minimum loop is proposed: diagnose failure samples, retrain with minimal data increments, validate against standardized sets, then deploy in phases with automatic rollback.The proposed path is problem-driven, metric-constrained, and resource-sensitive, with controllable processes, resources, and benefits.Source: C. Continuous Learning and Training… · p. 14

VII. CONCLUSION

  • The conclusion frames data quality, strategic alignment, and platform scalability as interdependent challenges requiring synchronized ecosystem evolution.It argues that progress depends less on breakthroughs in a single layer than on synergy across all three.Source: §VII · p. 14
Loading 2610.00926v1…