Source-linked AI summary
AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles
Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, Raquel Urtasun
TL;DR
Self-driving testing must find safety-critical scenarios despite an exponentially large scenario space and limited coverage from conventional approaches. AdvSim perturbs actor trajectories physically plausibly, simulates matching LiDAR data, and evaluates the full autonomy stack. It generates failure cases at scale across modern systems, and the paper reports that training with these scenarios can further improve robustness and safety.
Problem
The central gap is scalable testing that exposes failures beyond planning with ground-truth actor states, including perception difficulties from occlusion and complex interactions.
Method
AdvSim searches physically plausible perturbations of actors’ trajectories in existing scenarios, updates LiDAR observations, and queries the autonomy system through black-box optimization.
Results
AdvSim generates physically plausible failure cases for full autonomy stacks at scale across a wide range of systems.
Takeaways & Limitations
AdvSim-generated scenarios can be used to train autonomy systems and further improve their robustness and safety.
Takeaways & Limitations
The formulation discusses a single perturbed actor for brevity, while experiments also apply AdvSim to multi-actor perturbations.
Abstract
from arXiv · showhide
As self-driving systems become better, simulating scenarios where the autonomy stack may fail becomes more important. Traditionally, those scenarios are generated for a few scenes with respect to the planning module that takes ground-truth actor states as input. This does not scale and cannot identify all possible autonomy failures, such as perception failures due to occlusion. In this paper, we propose AdvSim, an adversarial framework to generate safety-critical scenarios for any LiDAR-based autonomy system. Given an initial traffic scenario, AdvSim modifies the actors' trajectories in a physically plausible manner and updates the LiDAR sensor data to match the perturbed world. Importantly, by simulating directly from sensor data, we obtain adversarial scenarios that are safety-critical for the full autonomy stack. Our experiments show that our approach is general and can identify thousands of semantically meaningful safety-critical scenarios for a wide range of modern self-driving systems. Furthermore, we show that the robustness and safety of these systems can be further improved by training them with scenarios generated by AdvSim.
1. Introduction
AdvSim addresses the difficulty of finding safety-critical scenarios at scale by perturbing actor behavior in existing traffic scenes and updating LiDAR observations for the full autonomy stack. The framework targets physically plausible failures, including failures involving perception and complex multi-actor interactions.
- Exhaustive scenario search is computationally unfeasible because possible lane topologies, actor configurations, trajectories, velocities, appearances, and backgrounds combine exponentially.
- Human-designed testing usually varies one or two actors, limiting coverage of complex interactions and making the process time-consuming and difficult to scale.
- Prior automated methods often evaluate planning with ground-truth actor states, overlooking perception and forecasting difficulties caused by occlusion or hard-to-localize trajectories.
- AdvSim frames worst-case scenario generation as a black-box attack on any LiDAR-based autonomy system using physically feasible changes in actor behavior.
- AdvSim perturbs actors’ trajectories, updates sensor data with a high-fidelity LiDAR simulator accounting for occlusions, and evaluates the resulting autonomy system behavior.
- AdvSim-generated scenarios can be used in training to further improve the safety of autonomy systems.
2. Related Work
Related work spans scenario search for planning, physically realizable adversarial examples for perception, and black-box search methods. AdvSim combines fine-grained physically feasible actor trajectories with adversarial evaluation of full LiDAR-based autonomy systems at scale.
- Safety-critical scenario generation combines a parameterization space, a search algorithm, and an evaluation setting for the system under test.
- The comparison table characterizes AdvSim as producing physically plausible scenarios adversarial to full LiDAR-based autonomy systems at scale.
- Prior methods represent other agents with frames, initial states, routes, or controls; AdvSim uses kinematic bicycle-model trajectories for physical feasibility and fine-grained behavior control.
- AdvSim’s generation procedure requires sensory input, an initial perturbed-actor state, an adversarial objective, and a query budget, then iteratively selects perturbations using black-box search.
3. Generating Safety-Critical Scenarios
AdvSim searches for physically plausible actor-trajectory perturbations that increase autonomy failure risk, while regenerating realistic LiDAR observations for the perturbed scene. Its black-box framework evaluates the full LiDAR-based autonomy stack and supports multiple failure objectives.
- Adversarial Behavior Modeling: Actor behaviors are represented as kinematic bicycle-model trajectories, enabling physically feasible and fine-grained control through bounded acceleration and curvature changes.Initial position, heading, and velocity can also be perturbed within set bounds.
- Adversarial Behavior Modeling: AdvSim projects candidate perturbations onto a physically feasible trajectory set, rejecting trajectories that collide with other actors or the original expert trajectory.The search space remains low-dimensional for query-based black-box optimization.
- Realistic LiDAR Simulation: The method updates recorded LiDAR by removing original actor observations, filling their LiDAR shadows, and adding simulated observations at perturbed locations.Only modified regions are regenerated to accelerate the query function, while preserving sensor realism and occlusion effects.
- Problem Formulation: AdvSim frames safety-critical scenario generation as a black-box attack that perturbs actor behavior to increase the autonomy system’s planned-trajectory cost.The framework can test any LiDAR-based autonomy system through limited queries.
- Adversarial Objective: The adversarial objective combines costs that expose unnatural trajectories, collisions, lane violations, hard braking, and dangerous accelerations or jerk.Multiple costs allow the framework to identify different types of autonomy-system failures.
- Black-Box Search: AdvSim supports Bayesian optimization, genetic algorithms, random search, and gradient-estimation methods for maximizing the adversarial objective.Each search method queries the autonomy system with candidate perturbations and uses prior query history to select subsequent candidates.
4. Experiments
The experiments evaluate AdvSim on diverse autonomy systems and scenarios, showing that it generates challenging failures, transfers across systems, and improves robustness through training.
- Experimental Setup: UrbanScenarios provides 5,000 25-second driving logs collected across multiple North American cities with varied layouts and traffic densities.
- Experimental Setup: The evaluation uses IL, PLT, NMP, and P3 systems in an open-loop setting where 1 second of LiDAR history produces a 5-second trajectory plan.
- Evaluations on Modern Autonomy Systems: 32.8% average collision rate is achieved on AdvSim scenarios, over 200% higher than the original set across modern autonomy systems.
- Safer Planner with Challenging Scenarios: Robust training with AdvSim-generated scenarios further improves performance across most planning metrics on original and adversarial scenarios.
- Transferability of Adversarial Scenarios: Adversarial scenarios generated for the same target system usually perform best, while scenarios generated against P3 transfer more strongly across systems.
- Investigation on Attack Configurations: Increasing the number of perturbed actors worsens planning and prediction metrics, but collision rate decreases when m ≥3 because physically plausible candidates become harder to find.
- Investigation on Attack Configurations: Bayesian optimization is most efficient because the perturbation space is low dimensional and the cost measure is nonsmooth with respect to perturbations.
- Ablation Studies: The proposed hybrid adversarial objective outperforms alternative adversarial objectives on all planning metrics.
5. Conclusion
AdvSim is presented as a framework for generating physically plausible worst-case scenarios for modern autonomy systems by simulating sensor data after perturbing actor behavior. The generated scenarios scale across systems and can improve autonomy robustness and safety through training.
- AdvSim generates physically plausible failure cases for full autonomy stacks by simulating sensor data from perturbed actor behaviors.
- The framework generates failure cases at scale for a wide range of autonomy systems and uses them to improve robustness and safety.
Supplementary Material AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles
The supplementary material accompanies AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles and lists its authors and institutional affiliations.
- The paper is authored by Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun.
- The authors are affiliated with the University of Toronto, Uber ATG, and the University of Waterloo.
- The listed contact address uses the University of Toronto research group email domain.
A.1. Data Curation
AdvSim curates interactive traffic sequences, constructs physically feasible actor trajectories, and searches them with black-box optimization algorithms to generate adversarial scenarios.
- Data Curation: AdvSim selects nearby actors from curated traffic sequences and samples trajectories, pruning physically implausible behaviors before optimization.The experiments sample Nsample = 2e5 trajectories per perturbed actor and retain feasible trajectories.
- Trajectory Perturbation: Actor behaviors are represented as kinematic bicycle-model trajectories, enabling physically feasible and fine-grained control.Physical constraints are imposed through realistic bounds for the kinematic bicycle model.
- Black-box Search: AdvSim evaluates Bayesian optimization, genetic algorithms, random search, and gradient-estimation approaches for black-box scenario search.Bayesian optimization uses a Gaussian-process surrogate with an upper-confidence-bound acquisition function and a query budget of 75.
- Black-box Search: The genetic algorithm evolves candidate trajectories by fitness-weighted sampling while preserving the best candidate at each iteration.Adaptive parameter scaling is used to reduce sensitivity to genetic-algorithm hyperparameters.
A.4. Robust Training
Robust training augments a converged P3 model with challenging or AdvSim-generated scenarios and uses early stopping during retraining.
- A.4. Robust Training: Robust training retrains a converged P3 model on generated adversarial scenarios, while a comparison baseline uses selected challenging scenarios.The initial model is trained for 80k iterations, followed by 10k retraining iterations at learning rate 2e−4.
B.1. Generalization of Robust Training
Robustly trained PLT and P3 systems outperform standard systems consistently on adversarial scenarios generated for different autonomy systems.
- B.1. Generalization of Robust Training: Robustly trained PLT and P3 outperform standard systems consistently across scenarios generated with respect to other autonomy systems.The comparison reports cumulative collision rate up to 5s.
B.2. Supplementary Metrics for Table 3
Supplementary evaluations show realistic LiDAR updates, transferable scenario difficulty, and both collision and non-collision outcomes across generated scenarios.
- B.2. Supplementary Metrics for Table 3: Generating adversarial scenarios for the same target autonomy system usually produces the best 5-second L2 human prediction error.This evaluates transferability across different autonomy systems.
- C.1. Realistic LiDAR Simulation for AdvSim: AdvSim’s simulated LiDAR scenes match the perturbed actors’ changed motions while preserving realistic scene appearance.Original points are shown in blue and simulated points in red, with perturbed-actor bounding boxes highlighted.
- C.2. Collision Scenarios: Some generated scenarios cause P3 plans to collide with pedestrians, perturbed vehicles, or other vehicles.The examples visualize motion-forecasting occupancy predictions over the planning horizon.
- C.3. Non-collision Scenarios Generated by AdvSim: Other generated scenarios do not cause collisions because P3 reacts reasonably, although they may require plans with larger costs.Such cases remain useful for testing and analysis.
- C.4. Perturbing Multiple Actors with AdvSim: AdvSim generates multi-actor perturbations with physically plausible behaviors and difficulty for self-driving systems.These scenarios are evaluated in the generalization analysis associated with Table 4.