PaperlayerView exact revision on arXiv

Source-linked AI summary

SimScale: Learning to Drive via Real-World Simulation at Scale

Haochen Tian, Tianyu Li, Haochen Liu, Jiazhi Yang, Yihang Qiu, Guang Li, Junli Wang, Yinfeng Gao, Zhang Zhang, Liang Wang, Hangjun Ye, Tieniu Tan, Long Chen, Hongyang Li

arXiv:2511.23369v3cs.CVcs.RO

TL;DR

Human driving logs underrepresent safety-critical and out-of-distribution situations, motivating SimScale’s scalable simulation framework. It perturbs trajectories, renders reactive environments, and supplies pseudo-expert supervision, while sim-real co-training improves planners by up to +8.6 EPDMS on navhard and +2.9 on navtest, with gains scaling as simulation data increases.

  • Problem

    Human expert driving data underrepresents safety-critical and out-of-distribution scenarios, limiting planners’ ability to generalize beyond the human driving distribution.

  • Method

    SimScale generates perturbed states with neural rendering and reactive environments, assigns pseudo-expert trajectories, and co-trains planners on real and simulated samples.

  • Results

    Sim-real co-training improves robustness and generalization across planners, reaching +8.6 EPDMS on navhard and +2.9 on navtest while scaling with simulation data at fixed real-data volume.

  • Takeaways & Limitations

    Scalable simulation can amplify the value of existing real-world datasets for end-to-end autonomy and supports predictable policy improvement without additional real-world data.

Abstract

from arXiv · show

Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, including safety-critical and out-of-distribution ones. However, such cases are underrepresented in real-world corpus collected by human experts. To complement for the lack of data diversity, we introduce a novel and scalable simulation framework capable of synthesizing massive unseen states upon existing driving logs. Our pipeline utilizes advanced neural rendering with a reactive environment to generate high-fidelity multi-view observations controlled by the perturbed ego trajectory. Furthermore, we develop a pseudo-expert trajectory generation mechanism for these newly simulated states to provide action supervision. Upon the synthesized data, we find that a simple co-training strategy on both real-world and simulated samples can lead to significant improvements in both robustness and generalization for various planning methods on challenging real-world benchmarks, up to +8.6 EPDMS on navhard and +2.9 on navtest. More importantly, such policy improvement scales smoothly by increasing simulation data only, even without extra real-world data streaming in. We further reveal several crucial findings of such a sim-real learning system, which we term SimScale, including the design of pseudo-experts and the scaling properties for different policy architectures. Simulation data and code have been released at https://github.com/OpenDriveLab/SimScale.

1. Introduction

SimScale addresses the lack of safety-critical and out-of-distribution driving data by generating diverse simulated states and pseudo-expert demonstrations from existing real-world logs. Its sim-real co-training improves robustness and generalization across planners while enabling predictable gains as simulation data scales.

  • Real-world human demonstrations underrepresent safety-critical and non-trivial scenarios, limiting planners’ generalization to rare or unseen situations.
  • SimScale uses neural rendering and reactive simulation to create high-fidelity observations for perturbed ego trajectories and generate feasible pseudo-expert demonstrations.
  • The framework compares recovery-based experts that remain human-like and cautious with planner-based experts that optimize trajectories but trade off realism.
  • Sim-real co-training mixes real and simulated samples to preserve human driving distributions while improving robustness and generalization across end-to-end planners.
  • SimScale identifies exploratory experts, reactive interaction environments, and multimodal planners as factors that improve simulation effectiveness and scaling.

2. Methodology

The methodology builds simulated driving data by perturbing trajectories, rolling out reactive environments, rendering multi-view observations, and assigning filtered pseudo-expert trajectories. Real and simulated samples are then combined through co-training, with reward-only supervision additionally explored for vocabulary-scoring planners.

  • Pseudo-Expert Scene Simulation: SimScale’s pipeline perturbs ego trajectories, simulates reactive interactions, renders observations with a 3DGS engine, and filters feasible samples into a simulation dataset.
  • Pseudo-Expert Scene Simulation: The pipeline performs one simulation to explore perturbed states and a second to generate pseudo-expert trajectories under physical, traffic, and kinematic constraints.
  • Pseudo-Expert Scene Simulation: Recovery-based experts retrieve human trajectories matching the perturbed state, producing human-like conservative fallback behavior that stabilizes under distributional drift.
  • Pseudo-Expert Scene Simulation: Planner-based experts use privileged ground-truth states to generate optimized reactive rollouts, offering diverse exploratory supervision at the cost of occasional realism.
  • Scalable Sim-Real Co-training: Sim-real co-training samples from real and simulated datasets to preserve human driving distributions and reduce visual-domain degradation from simulation artifacts.
  • Scalable Sim-Real Co-training: For vocabulary-scoring planners, reward-only co-training removes the need for a single expert trajectory when reward signals are sufficiently aligned.

3. Experiments

Experiments evaluate SimScale across NAVSIM benchmarks, planner paradigms, pseudo-expert strategies, and simulation-data scaling regimes. Results show improved leaderboard performance and distinct scaling behavior shaped by planner architecture, supervision, and reactive simulation.

  • 3.2. Leaderboard Results: Weaker LTF and DiffusionDrive baselines gain more than 20% on navhard with simulation data and sim-real co-training.The reported comparison identifies the largest relative benefits among weaker baseline models.
  • 3.2. Leaderboard Results: Up to +2.9 points on navtest: all evaluated models improve under large-scale, diverse conditions.The reported gains support improvements across planner paradigms rather than a single architecture.
  • 3.3. Ablation and Data Scaling Analysis: Planner-based pseudo-experts outperform recovery-based supervision at larger data scales because they explore broader feasible trajectory possibilities.Recovery-based experts retain an advantage only in small-data regimes, while planner-based experts produce feasible solutions in challenging situations.
  • 3.3. Ablation and Data Scaling Analysis: DiffusionDrive improves approximately linearly with simulation data, whereas LTF saturates and degrades at a 1:1 simulation-to-real ratio.The difference is attributed to multimodal supervision from diverse demonstrations and the contrasting capacities of diffusion and single-mode regression models.
  • 3.3. Ablation and Data Scaling Analysis: The experiments visualize four simulated OOD scenarios—off-center lane drift, near collision, departure, and cutting in—with synthetic front-view and auxiliary key-view observations.Each scenario pairs a pseudo-expert trajectory with a perturbed trajectory history and synthetic sensory input.

4. Conclusion

SimScale uses pseudo-expert demonstrations and high-fidelity simulation to improve end-to-end driving, while scaling predictably with additional simulation data on a fixed real-world corpus.

  • 4. Conclusion: +8.6 EPDMS on navhard and +2.9 on navtest are reported from sim–real co-training across various planners.The reported gains concern robustness and generalization on challenging real-world benchmarks.
  • 4. Conclusion: SimScale generates pseudo-expert demonstrations from perturbed OOD states in reactive environments and renders associated multi-view observations with 3DGS.The pipeline combines ego perturbation, reactive environments, and high-fidelity neural rendering.
  • 4. Conclusion: Performance scales clearly and predictably as simulation increases while the real-world corpus remains fixed.The paper also reports that exploration and interaction improve simulation effectiveness and that multi-modal planners strengthen scaling behavior.

Supplementary Material

The supplementary material extends the paper’s related-work discussion, implementation details, experimental analyses, limitations, broader impacts, and licensing information.

  • Supplementary Material: Supplementary Sec. A discusses related work, Sec. B gives implementation details, and Secs. C–D report extended experiments and qualitative results.These sections cover data curation, models, and additional analyses corresponding to the main experiments.
  • Supplementary Material: The related-work discussion describes end-to-end planning as mapping raw sensor inputs directly to planning and reviews regression, generative, and trajectory-scoring approaches.It notes that diffusion systems model trajectories through conditional denoising and that scoring methods rank candidate trajectories.
  • Supplementary Material: The supplement reviews traffic and sensor simulation, including rule-based or diffusion-based traffic models and graphics-based simulators affected by sim-to-real gaps.It contrasts these approaches with simulation aimed at generating OOD states with expert demonstrations.

A.3. Data Scaling for Driving

Driving-data scaling has improved end-to-end planning, but real-world collection remains costly and simulation-based scaling in raw sensory domains is comparatively underexplored.

  • A.3. Data Scaling for Driving: Increasing real-world driving logs improves end-to-end planner performance, although the gains diminish as data grows.Prior work increased datasets from thousands to millions of driving logs, while dense video supervision improved scaling efficiency.
  • A.3. Data Scaling for Driving: Existing scaling approaches often rely on costly real-world collection or abstract-state traffic simulation rather than raw sensory domains.3D rasterization reduces computational burden but loses information, while self-play scales reinforcement learning through simulation.
  • A.3. Data Scaling for Driving: SimScale is contrasted with 3DGS-based Online RL as a framework that generates OOD states with expert demonstrations for imitation and reinforcement learning.The comparison is presented as a learning-paradigm contrast in Figure 6.

B.1. Simulation Data Curation

The simulation-data curation process specifies configurations and filters pseudo-expert trajectories to retain feasible supervision under EPDMS-related constraints.

  • B.1. Simulation Data Curation: Table 6 lists the model and training hyperparameters used in the experiments.These hyperparameters complement the implementation details for the evaluated models.
  • B.1. Simulation Data Curation: Pseudo-expert candidates are filtered by requiring all EPDMS sub-metrics, with SEP relaxed, and enforcing EP ≥0.5.The filtering is intended to prevent biased driving styles while retaining valid supervision.

B.2. Models and Training

The study evaluates multiple end-to-end planners under standardized real and sim-real training settings, with detailed benchmark configurations and results reported for navhard and navtest.

  • B.2. Models and Training: All planners use identical settings across real and sim-real training and are trained to saturation for scaling analyses.The experiments use NAVSIM defaults and navtrain training logs.
  • B.2. Models and Training: LTF is a regression planner, while DiffusionDrive generates diverse trajectories through anchor-conditioned truncated diffusion.Both use pretrained ResNet34 image encoders.
  • B.2. Models and Training: The supplementary benchmark tables provide detailed navhard and navtest results, distinguishing recovery-based, planner-based, and reward-only supervision strategies.The main paper reports each planner’s best co-training result, while the supplements show full comparisons.

C.1. Detailed Leaderboard Results

SimScale’s detailed evaluations show that pseudo-expert choice and reactive simulation affect planner performance, with reactive data scaling better across model sizes.

  • C.1. Detailed Leaderboard Results: Planner-based experts generally outperform recovery-based experts, while scoring-based planners benefit most from reward-only supervision.The main tables report the best navhard and navtest co-training results for each planner.
  • C.1. Detailed Leaderboard Results: Qualitative visualizations show high-fidelity simulated scenes and feasible trajectories that converge to expert waypoints or cover varied traffic scenarios.Planner-based trajectories span intersections, dense urban avenues, and narrow local roads.
  • C.1. Detailed Leaderboard Results: Reactive simulation data consistently achieves superior scaling performance across GTRS-Dense model sizes.The full scaling curves extend the subset of simulation rounds reported in the main paper.

D.1. Multi-Expert Ensemble

The multi-expert ensemble combines complementary pseudo-expert behaviors, improving navhard performance while showing that simulation fidelity also matters for scoring-based planning.

  • D.1. Multi-Expert Ensemble: Recovery-based experts are conservative, whereas planner-based experts are exploratory, providing complementary behavioral characteristics for ensembling.GTRS-Dense averages predicted trajectory subscores across recovery-based, planner-based, and reward-only models.
  • D.1. Multi-Expert Ensemble: +2.9 EPDMS on V2-99 and +0.8 EPDMS on ResNet34 are achieved over reward-only scoring on navhard.Expanding the ensemble to all six models adds no further EPDMS gains.
  • D.1. Multi-Expert Ensemble: Simulation data has a lower EPDMS distribution than real data, indicating a more safety-critical simulated distribution and challenging scenario tags.EgoMLP evaluations expose the distribution shift between simulation and real-world data.
  • D.1. Multi-Expert Ensemble: Higher-fidelity simulation data with PSNR ≥27 consistently yields higher EPDMS than lower-fidelity data with PSNR <27.The comparison spans different data scales and co-training strategies.

D.4. Scaling with Varying Scales of Real Data

Simulation data continues to improve planning when the real-data corpus grows, with the largest gains appearing when real data is scarce and no noticeable narrowing through 100K samples.

  • D.4. Scaling with Varying Scales of Real Data: +22.4% EPDMS on ResNet34 and +12.1% EPDMS on V2-99 are achieved on navhard with 10K real-data samples.Simulation is generated from matching real-data scenarios at a fixed five-round sim-real ratio.
  • D.4. Scaling with Varying Scales of Real Data: Simulation gains remain consistently high as real data increases from 10K to 100K, without noticeable narrowing.The results indicate continued complementarity between simulated and real-world data.

E. Limitations and Broader Impact

SimScale's simulation approach remains constrained by the realism, diversity, and efficiency of its scene and pseudo-expert generation. These limitations create practical boundaries for deployment, especially in rare corner cases.

  • E.1. Pseudo-Expert: Static rule-based perturbations limit pseudo-expert exploration, while the rule-based privileged BEV planner degrades comfort metrics and fails in extreme corner cases.The authors suggest self-evolving exploration and learning-based BEV planners as possible improvements.
  • E.2. Scene Simulation: IDM-controlled surrounding agents enable interaction but constrain traffic-scenario diversity.The paper identifies diffusion-based traffic generators and unified world models as potential alternatives.
  • E.4. Societal Impact: Unrealistic simulation visuals and real-world distribution differences in corner cases may introduce risks during co-training and deployment.The authors also note that pseudo-expert scene simulation still needs quality and efficiency improvements for real-world use.
Loading 2511.23369v3…