Source-linked AI summary
Pseudo-Simulation for Autonomous Driving
Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, Kashyap Chitta
TL;DR
Existing AV evaluation is constrained by safety, reproducibility, realism, computational cost, and open-loop metrics that overlook compounding errors. The paper introduces pseudo-simulation, which augments real observations with pre-generated synthetic futures and proximity-based weighting; it achieves R2 = 0.8 correlation with closed-loop evaluation versus R2 = 0.7 for single-stage open-loop evaluation.
Problem
AV evaluation must assess safe, efficient, and comfortable navigation, but existing real-world, closed-loop, and open-loop strategies have important practical or methodological limitations.
Method
Pseudo-simulation evaluates recorded observations alongside pre-generated synthetic observations from 3D Gaussian Splatting, weighting plausible futures by proximity to the Stage 1 endpoint.
Results
R2 = 0.8 for pseudo-simulation versus R2 = 0.7 for the best existing open-loop approach in alignment with closed-loop evaluation.
Takeaways & Limitations
Pseudo-simulation supports more efficient experimentation and standardized benchmarking while revealing failure modes and testing error recovery beyond open-loop evaluation.
Takeaways & Limitations
Validation establishes correlation with simulation benchmarks but does not yet demonstrate direct correlation with real-world vehicle deployment.
Abstract
from arXiv · showhide
Existing evaluation paradigms for Autonomous Vehicles (AVs) face critical limitations. Real-world evaluation is often challenging due to safety concerns and a lack of reproducibility, whereas closed-loop simulation can face insufficient realism or high computational costs. Open-loop evaluation, while being efficient and data-driven, relies on metrics that generally overlook compounding errors. In this paper, we propose pseudo-simulation, a novel paradigm that addresses these limitations. Pseudo-simulation operates on real datasets, similar to open-loop evaluation, but augments them with synthetic observations generated prior to evaluation using 3D Gaussian Splatting. Our key idea is to approximate potential future states the AV might encounter by generating a diverse set of observations that vary in position, heading, and speed. Our method then assigns a higher importance to synthetic observations that best match the AV's likely behavior using a novel proximity-based weighting scheme. This enables evaluating error recovery and the mitigation of causal confusion, as in closed-loop benchmarks, without requiring sequential interactive simulation. We show that pseudo-simulation is better correlated with closed-loop simulations ($R^2=0.8$) than the best existing open-loop approach ($R^2=0.7$). We also establish a public leaderboard for the community to benchmark new methodologies with pseudo-simulation. Our code is available at https://github.com/autonomousvision/navsim.
1 Introduction
AV evaluation must balance realism, reproducibility, safety, and computational efficiency. Pseudo-simulation combines real observations with pre-generated synthetic views and proximity-weighted two-stage scoring to approximate likely futures and assess robustness.
- Existing evaluation limitations: Closed-loop evaluation provides interactive feedback but is costly, risky, and difficult to reproduce at benchmarking scale.Simulation improves reproducibility and scenario control but remains computationally intensive and sequential.
- Existing evaluation limitations: Open-loop evaluation efficiently compares predicted trajectories with expert demonstrations but does not fully assess behavior across future states.It operates on pre-recorded real-world observations and scores displacement or annotation-derived metrics.
- Pseudo-simulation: Pseudo-simulation generates synthetic observations from real-world frames before evaluation, avoiding sequential interactive simulation while retaining real-data grounding.Stage 1 evaluates original observations, while Stage 2 evaluates observations rendered using a driving-scene adaptation of 3D Gaussian Splatting.
- Pseudo-simulation: Stage 2 weights synthetic observations by proximity to the Stage 1 predicted endpoint, emphasizing likely futures and reducing penalties from improbable states.This weighting targets robustness to nearby variations and potential error recovery.
- Implications: Pseudo-simulation combines real and pre-rendered synthetic data to enable scalable, parallel evaluation.The framework is intended to support more efficient experimentation and broader robustness testing.
- Implications: 6× less environment interactions are required while pseudo-simulation achieves strong correlation with closed-loop results across 83 diverse nuPlan planners.NAVSIM v2 provides a standardized benchmarking framework and leaderboard that exposes previously unknown AV failure modes.
2 Related Work
Related work spans counterfactual augmentation and simulation-based benchmarking. Existing approaches improve scenario diversity or scalability, but important gaps remain for sensor-based, realistic, and consistent AV evaluation.
- Counterfactual Data Augmentation: Counterfactual augmentation exposes models to out-of-distribution data through structured perturbations, while AV studies commonly use viewpoint shifts during training.This paper instead adopts such augmentations primarily for evaluation.
- Closed-Loop Benchmarking: Graphics-based simulators support closed-loop evaluation but are computationally expensive and can introduce sensor-fidelity domain gaps.Data-driven planning simulators improve scalability but operate at the trajectory level and do not support sensor-based agents.
- Open-Loop Benchmarking: Open-loop benchmarks efficiently measure planning quality but often correlate poorly with real-world performance and may favor trivial or history-based baselines.nuScenes-based benchmarks also show implementation inconsistencies for metrics such as ADE and collisions.
3 Pseudo-Simulation
Pseudo-simulation evaluates planners in two stages: an initial real-world observation followed by pre-generated synthetic observations representing plausible future states. Stage 2 scores are aggregated with proximity-based weighting toward the Stage 1 endpoint.
- 3.1 Stage 1: Initial Observations: The AV generates a 4-second trajectory from multi-view cameras, ego status, and a discrete driving command.Commands specify left, straight, or right when the intended maneuver is ambiguous.
- 3.1 Stage 1: Initial Observations: Stage 1 executes the predicted trajectory in a BEV simulation using a kinematic bicycle model and LQR controller at 10Hz.The trajectory is committed for the full horizon without agent feedback, while background traffic is reactive.
- 3.1 Stage 1: Initial Observations: EPDMS combines multiplicative rule-violation penalties with a weighted average of traffic, progress, comfort, lane-keeping, and expert-compliance subscores.Human-driver filtering ignores violations also committed by the expert, including contextually justified maneuvers.
- 3.2 Stage 2: Synthetic Observations: Stage 2 repeats the scoring pipeline on pre-generated synthetic observations and weights their contributions by proximity to the Stage 1 endpoint.Synthetic observations are generated after the 4-second rollout so traffic can react and the planner receives a plausible history.
- 3.2 Stage 2: Synthetic Observations: Synthetic start points are sampled around the expert endpoint across lateral and longitudinal positions spanning physically plausible motion ranges.Lateral samples are spaced every 0.5 meters up to 2.0 meters per side, while longitudinal samples use 5.0-meter spacing and can produce up to 20 states.
- 3.2 Stage 2: Synthetic Observations: Each sampled state receives a plausible heading and motion history by matching it to human-driving trajectories, then filtering inconsistent or invalid candidates.Candidates are filtered by velocity, acceleration, and heading differences, followed by rejection of states violating multiplicative EPDMS constraints.
4 Results
The experiments show that pseudo-simulation aligns strongly with closed-loop evaluation across diverse planners, remains efficient under reduced synthetic-view density, and exposes performance degradation on synthetic observations. Ablations identify two-stage evaluation, proximity weighting, multiplicative aggregation, and neural rendering choices as important design factors.
- Alignment with closed-loop evaluation: Synthetic deviations expose a wider range of potential failures than standard reactive closed-loop evaluation, typically producing lower EPDMS than CLS.The added deviations reveal edge cases that may not appear during standard testing.
- Design choices: Smaller Gaussian weight variances perform better, with σ2 = 0.05 and the default σ2 = 0.1 achieving the highest correlations.Simple averaging, k-nearest neighbors, and hybrid k-NN/Gaussian weighting are less effective than the default configuration.
- Design choices: Multiplicative aggregation yields higher linear and rank correlation than arithmetic-mean and hybrid aggregation, likely because most subscores are binary.Multiplication combines scores across two 4-second segments into an 8-second interval.
- Efficiency and density: 25% synthetic-view density retains correlation above 0.85 while requiring approximately three Stage 2 observations per scene.At full density, pseudo-simulation uses 13 planner inferences per scenario versus 80 for nuPlan, a 6× difference.
- Leaderboard: NAVSIM v2’s navhard leaderboard standardizes evaluation on challenging scenarios using 450 Stage 1 and 5462 Stage 2 observations.The leaderboard addresses metric differences caused by implementation details and provides consistency and submission visibility.
- Leaderboard: Across navhard planners, Stage 2 subscores generally drop from Stage 1, indicating sensitivity to the distribution shifts introduced by synthetic observations.PDM-Closed achieves the highest overall EPDMS of 56.6 in the reported leaderboard analysis.
5 Conclusion
Pseudo-simulation offers a computationally efficient evaluation paradigm that correlates highly with closed-loop simulation and captures error recovery more effectively than open-loop evaluation. The paper also presents navhard and a public leaderboard to support standardized comparisons and broader adoption.
- 5 Conclusion: Pseudo-simulation demonstrates high correlation with computationally expensive closed-loop simulations while capturing error recovery more effectively than open-loop evaluation.The method is presented as an efficient evaluation paradigm for autonomous-driving development.
- 5 Conclusion: The public navhard benchmark provides pre-rendered data, standardized metrics, and an online leaderboard for community comparisons of autonomous-driving systems.The benchmark is intended to foster standardized comparisons and community adoption.
Limitations and Future Work
The authors identify limitations involving real-world validity, preprocessing cost, rendering fidelity, traffic realism, human-reference filtering, and metric design. They outline future work to address these boundaries while retaining pseudo-simulation’s scalability.
- Correlation with Real-World Deployment: Validation establishes correlation with simulation benchmarks, but not with real-world vehicle deployment metrics.The authors frame bridging simulation-based evaluation to real-world outcomes as future work rather than claiming pseudo-simulation replaces real-world validation.
- Pre-Processing Computational Cost: 1–2 hours per scene of preprocessing limits scalability to extremely large datasets.The current pipeline uses per-scene MTGS optimization; the authors suggest faster feedforward scene representations and rendering methods.
- Rendering Fidelity and Evaluation: Synthetic views may retain visual artifacts because evaluation primarily uses algorithmic metrics rather than human perceptual studies.The authors propose perceptual evaluation and combinations with generative diffusion models as possible improvements.
- Background Traffic Realism: Rule-based background traffic follows road-centerline paths, limiting the complexity and interactivity of Stage 2 scenarios.Future extensions include learned or adversarial traffic models that adapt to ego-agent actions.
- Human Flag Filtering: Filtering rule violations also committed by human experts reduces false positives but may overlook safety-relevant failures and edge cases.The authors note that human driving is not always a gold standard for safety.
- Metric Design Choices: Multiplicative aggregation captures compounding failures, but the aggregation and Gaussian weighting formulations remain design choices open to more principled alternatives.The authors selected Gaussian weighting for empirical performance with minimal assumptions.