PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior

Davis Rempe, Jonah Philion, Leonidas J. Guibas, Sanja Fidler, Or Litany

arXiv:2112.05077v2cs.CVcs.LGcs.RO

TL;DR

Autonomous-vehicle planning needs scalable long-tail scenarios that are both challenging and realistically solvable, but such cases are scarce and costly to construct. STRIVE optimizes real-world scenes in the latent space of a learned traffic model, then verifies utility through solution optimization and collision-type analysis. It generates plausible challenging scenarios for two planners and uses them to expose limitations and tune a rule-based planner.

  • Problem

    Rare near-collision scenarios create data scarcity for developing and testing AV systems, while manual construction is costly at scale.

  • Method

    STRIVE optimizes perturbations of real-world scenes in a learned traffic model’s latent space, using a planner proxy and solution optimization to generate plausible, useful collision scenarios.

  • Results

    STRIVE generates plausible, diverse collision scenarios for two AV planners and uses them to identify rule-based planner limitations and tune its hyperparameters.

  • Takeaways & Limitations

    Generated scenarios can support planner improvement by exposing collision-specific weaknesses and guiding hyperparameter optimization.

  • Takeaways & Limitations

    STRIVE assumes perfect perception, attacks only the planner, and considers vehicle collisions from existing data rather than pedestrians, cyclists, or broader scene changes.

Abstract

from arXiv · show

Evaluating and improving planning for autonomous vehicles requires scalable generation of long-tail traffic scenarios. To be useful, these scenarios must be realistic and challenging, but not impossible to drive through safely. In this work, we introduce STRIVE, a method to automatically generate challenging scenarios that cause a given planner to produce undesirable behavior, like collisions. To maintain scenario plausibility, the key idea is to leverage a learned model of traffic motion in the form of a graph-based conditional VAE. Scenario generation is formulated as an optimization in the latent space of this traffic model, perturbing an initial real-world scene to produce trajectories that collide with a given planner. A subsequent optimization is used to find a "solution" to the scenario, ensuring it is useful to improve the given planner. Further analysis clusters generated scenarios based on collision type. We attack two planners and show that STRIVE successfully generates realistic, challenging scenarios in both cases. We additionally "close the loop" and use these scenarios to optimize hyperparameters of a rule-based planner.

1. Introduction

STRIVE addresses the scarcity and cost of generating realistic, challenging near-collision scenarios by optimizing perturbations of real-world scenes under a learned traffic prior. It verifies scenario utility through solution optimization and analyzes collision types while improving planners.

  • Near-collision scenarios are rare, creating data scarcity that hinders development and testing of AV perception, prediction, and planning models.
  • Manual scenario design is safer than relying on real-world data but becomes prohibitively expensive when large training or evaluation sets are required.
  • STRIVE optimizes scene-wide agent trajectories in the latent space of a learned traffic model to generate plausible collisions for a given planner.
  • A differentiable planner proxy enables gradient-based optimization even when the target planner is non-differentiable or inaccessible.
  • A solution optimization filters unsolvable scenarios, while collision-property clustering characterizes solvable scenarios by collision type.
  • STRIVE generates plausible, diverse collision scenarios for two AV planners and uses them to identify rule-based planner limitations and tune hyperparameters.

2. Related Work

Related work spans neural traffic-motion modeling, simulation-based scenario generation, and AV planning. STRIVE builds on these directions by combining a controllable learned traffic prior with scenario optimization and utility filtering.

  • Traffic Motion Modeling: Neural traffic models replace traditional dynamic, kinematic, or rule-based simulators to better capture traffic complexity.
  • Traffic Motion Modeling: Traffic prediction research varies in map encoding, multimodal future modeling, and explicit multi-agent interaction modeling.
  • Traffic Motion Modeling: The paper’s VAE-based traffic model provides a controllable latent-space prior, adds environment-collision penalties, and structures predictions with a bicycle model.
  • Challenging Scenario Generation: STRIVE uses a more expressive data-driven motion prior than AdvSim’s simplistic bicycle model, enabling plausible control of many adversarial agents.
  • Challenging Scenario Generation: Unlike approaches that constrain attack trajectories to avoid playback collisions, STRIVE uses solution optimization to filter worthwhile scenarios.
  • AV Planners: Rule-based planners remain common in practical AV systems, motivating evaluation on a lane-graph-based rule-based planner.

3. Challenging Scenario Generation

STRIVE generates collision-inducing scenarios by optimizing latent representations of all agents within a learned traffic prior, while querying a black-box planner. Its learned model promotes plausible, scene-consistent trajectories, and a subsequent solution process filters scenarios that cannot be safely resolved.

  • Scenario formulation: STRIVE perturbs real-world initial scenes in latent space to optimize non-ego trajectories toward an adversarial objective such as collision with a planner.The planner is treated as a black box, so optimization queries its rollout rather than differentiating through internal operations.
  • Learned traffic model: The traffic model is a graph-based CVAE whose low-dimensional latent space supports efficient optimization and whose prior scores scenario plausibility.The model learns joint future predictions conditioned on past motion and map context.
  • Learned traffic model: The decoder autoregressively predicts interacting agent trajectories and uses a kinematic bicycle model to maintain plausible vehicle dynamics during rollout.Message passing resolves interactions before predicting accelerations and the next agent states.
  • Adversarial optimization: Latent parameterization encourages realism through decoder behavior near the learned manifold and prior-based penalties on unlikely latent configurations.This prior enables joint optimization of all agents instead of selecting only a few adversaries in advance.
  • Adversarial optimization: At each optimization step, decoded planner and non-ego representations are rolled out through the planner, after which adversarial losses are computed.The planner trajectory is represented in the traffic model to preserve realistic interactions while the external planner supplies the actual rollout.

4. Analyzing and Using Generated Scenarios

STRIVE evaluates generated scenarios for usefulness by seeking an ego trajectory that avoids collision, discarding unsolvable cases, and clustering solvable cases by collision geometry. These labeled scenarios can reveal missing planner functionality and support parameter tuning.

  • Solution optimization: Solution optimization searches for an ego trajectory that avoids collision and discards scenarios when this optimization fails.This removes unavoidable-collision cases that are uninformative for evaluating or improving a planner.
  • Clustering and labeling: Collision scenarios are clustered using the adversary’s relative direction and heading in the planner’s reference frame.The resulting collision features provide labels describing scenario types.
  • Using generated scenarios: Repeated collision types can expose missing planner functionality, while generated scenarios also provide data for tuning hyperparameters or learned parameters.The paper gives lane-following planners’ failure to swerve as an example of a functionality gap.
  • Using generated scenarios: The rule-based planner serves as an interpretable proxy with few hyperparameters and known failure modes for demonstrating downstream scenario-based improvement.It relies on the lane graph to predict other vehicles and generate ego candidate trajectories.

5. Experiments

STRIVE is evaluated on Replay and Rule-based planners using nuScenes-initialized scenarios, showing that it generates accident-prone, solvable, plausible, and diverse collisions. The generated scenarios also expose planner limitations and support hyperparameter improvements for the Rule-based planner.

  • Experimental Setup: STRIVE is evaluated on Replay and Rule-based planners, with nuScenes providing both traffic-model training data and 8s scenario initializations.The Rule-based planner replans at 5 Hz, whereas Replay rolls out the ground-truth ego trajectory open-loop.
  • Scenario Generation Evaluation: 1200 initial nuScenes sequences are heuristically filtered to fewer than 500 scenarios before adversarial optimization.The filtering selects scenes likely to produce useful collisions.
  • Scenario Analysis: STRIVE produces diverse collision scenarios, including lane changes, turns in front of the planner, and pulls into oncoming traffic.Across collision-type clusters, solvable scenes are found in every cluster; Head On is most frequent, while Behind has the highest unsolvable rate.
  • Scenario Generation Evaluation: 27.4% of Rule-based scenarios cause collisions after adversarial optimization, compared with 1.2% in corresponding regular scenarios.Generated scenarios also produce higher collision velocities and accelerations, indicating less comfortable driving; the largest planner-trajectory position error is 1.23m.
  • Baseline Comparison: Compared with the Bicycle baseline, STRIVE generates more plausible and useful Replay scenarios by leveraging a learned traffic model.Bicycle trajectories exhibit higher accelerations, more collisions with non-drivable areas, and less typical nearest-neighbor trajectory distances; its optimization is also approximately 40× slower.
  • Improving Rule-Based Planner: Adding an accident mode reduces challenging-scenario collisions by 14.3% without hindering regular-scene performance, while exposing a lane-change limitation.A learned classifier selects between regular and accident-mode parameters; many Head On and Behind scenarios remain impossible because the planner cannot switch lanes.

6. Discussion

STRIVE generates plausible, accident-prone scenarios to improve planners, while the method remains bounded by assumptions about perception, data, incident types, and planner tuning.

  • STRIVE enables automatic and scalable generation of plausible, accident-prone scenarios to improve a given planner.
  • The method assumes perfect perception, attacks only the planner, generates scenarios from existing data, and considers only vehicle-vehicle collisions.Future directions include attacking detection and tracking, modeling pedestrians and cyclists, and changing assets or map topology.
  • Integrating generated scenarios into planner tuning requires balancing regular and challenging data and designing planners for variable driving conditions.
  • The paper acknowledges project support from a Vannevar Bush faculty fellowship and thanks Amlan Kar for discussion and feedback.

Appendices

The appendices provide omitted technical and experimental details, while the project page offers supplementary video results illustrating STRIVE's generated scenarios.

  • Appendices: Appendix A contains additional details on the main technical methods.
  • Appendices: Appendix B contains additional details for the experiments.
  • Appendices: Appendix C provides additional results supplementing those in the main paper, and the project page includes videos of STRIVE's generated scenarios.

A. Method Details

The learned traffic model represents multi-agent motion with graph-based conditional variational components and autoregressive decoding. Collision-aware optimization uses differentiable agent and environment penalties while retaining plausible dynamics and planner utility.

  • Feature Extraction: Trajectories encode position, heading, speed, and yaw rate, while context features combine past motion, local rasterized maps, semantic class, and bounding-box dimensions.Map crops cover drivable, carpark, road-divider, and lane-divider layers; missing trajectory frames receive validity flags.
  • Graph-Based Traffic Model: The prior, posterior, and decoder use graph neural networks with message passing to model interactions among scene agents.Edge features use relative position and heading, are max-pooled, and pass through node update networks.
  • Latent Modeling and Decoding: The prior and posterior output Gaussian distributions over 32-dimensional agent latents, while the decoder autoregressively predicts future accelerations and rolls them out with a kinematic bicycle model.The decoder updates past and map context after each step and can roll beyond the six-second training horizon.
  • Training: Training combines reconstruction, KL-divergence, and collision losses, with reconstruction supervising position and heading and collision losses using prior samples.Vehicle collisions are approximated by representing each vehicle with five discs and summing pairwise penalties over time.
  • Collision Penalties: Environment collision penalties detect overlap with non-drivable areas and apply only to partial ego-vehicle collisions during traffic-model training because fully embedded vehicles provide no useful gradient.The penalty depends on the distance between the vehicle position and collision point.
  • Adversarial Optimization: Adversarial optimization combines agent, environment, and planner losses to avoid unintended traffic collisions, keep non-ego vehicles drivable, and induce planner collisions.For masked agents near the planner, the planner loss discourages accidental rear-end collisions; closed-loop optimization requires gradients through the true planner and finite differences are inefficient.

B.1. Data and Metrics

Experiments use preprocessed nuScenes scenes, held-out evaluation, collision-focused metrics, clustering, and hyperparameter sweeps. The traffic model is competitive in prediction, while the bicycle-model ablation trades accuracy for less realistic dynamics.

  • Dataset: All experiments use nuScenes; preprocessing removes vehicle frames with more than 30% overlap with non-drivable or carpark areas.The learned traffic model trains on cars and trucks, while scenario generation starts from 1200 eight-second scenes from train and validation splits.
  • Metrics: Acceleration, environment collision rate, and nearest-neighbor distance are reported only until collision because subsequent motion is not physically simulated.Environment collision requires more than 5% bounding-box overlap with non-drivable area.
  • Scenario Evaluation: Scenario-generation evaluation reports collision rate over attacked scenes, solution rate over successful collisions, and planner metrics over scenarios with both collision and solution.The STRIVE comparison uses 139 shared initializations where both methods caused collisions.
  • Collision Analysis: Collision scenarios are clustered with k = 10 using collision properties, producing interpretable labels for analyzing the rule-based planner.The clustering model was built from over 400 scenarios generated across nuScenes subsets and planner versions.
  • Planner Improvement: Planner tuning sweeps 432 hyperparameter combinations and selects the lowest collision rate, breaking ties by lowest acceleration.The planner is first tuned on regular scenarios, then challenged with generated scenarios; “Behind” collisions are removed during challenging-scenario tuning.
  • Planner Results: On held-out nuScenes test scenarios, the best possible regular-scenario collision rate is 3.2%, while the regular-tuned planner reaches 4.6%.Nonzero collisions remain because log replay traffic is nonreactive and some ego vehicles begin off the lane graph.
  • Traffic-Model Evaluation: The full learned traffic model is competitive with current state-of-the-art prediction methods, while the no-bicycle-model ablation achieves better ADE/FDE but less realistic vehicle dynamics.The kinematic bicycle model is important for keeping optimized trajectories dynamically reasonable off-manifold.

C.2. Adversarial Optimization Ablation Study

The ablation study shows that objective components trade off scenario plausibility, collision generation, and solvability, while optimization mainly changes agents near the planner. The formulation also supports diverse collision mechanisms, including indirect second-order effects.

  • Ablation results: The full objective achieves the highest solution rate, generating useful scenarios at the highest frequency.
  • Ablation results: Removing Lprior produces less likely trajectories and more environment collisions, while removing Lcoll increases collisions and adversary accelerations.
  • Ablation results: Removing Linit lets trajectories stray farther from the realistic nuScenes initialization, reducing solution rates and adversary-motion plausibility.
  • Objective flexibility: Removing per-agent prior weighting produces collisions more aligned with usual traffic, demonstrating flexibility to tailor generated scenarios.
  • Scaling and locality: 21.2% is the collision rate for scenarios with ≥20 agents, comparable to 27.4% in Tab 1, indicating similar performance on larger scenes.Only a handful of agents near the planner strongly affect outcomes; latent changes occur most often within 10m.
  • Collision mechanisms: Adversarial optimization can create second-order collisions in which an adversary causes reactions that lead another agent to collide with the planner.

C.5. Solution Optimization Evaluation

The solution optimization filters generated collision scenarios by testing whether a planner can be made to avoid them through hyperparameter tuning. Scenarios without a found solution are substantially harder to resolve.

  • 29.8% of hyperparameter combinations avoid collisions on average for solution-found scenarios, versus 11.8% for solution-failed scenarios.The comparison uses the mean fraction of successful hyperparameter settings per scenario in each tuning set.

C.6. Additional Qualitative Results

Additional experiments show STRIVE can generate collisions involving pedestrians and cyclists, while qualitative results demonstrate substantial changes to real-world initial scenes for the Rule-based planner.

  • For the Rule-based planner, adversarial optimization can change an adversary’s intent or move stationary vehicles to create useful collision scenarios.
  • STRIVE generates collision scenarios for the Replay planner with both pedestrian and cyclist adversaries.The traffic model is trained on all nuScenes categories, and the adversary is selected before optimization.

C.8. Failure Cases and Limitations

Failure cases reveal that solution optimization may exploit future information, adversaries may use extreme non-drivable routes, and rare realistic motions remain difficult under the learned prior.

  • The iterative solution optimization accesses future planner trajectories, sometimes enabling evasive maneuvers before the attack becomes apparent.
  • Some adversaries cross non-drivable areas to collide with the planner, producing technically possible but extreme behavior that can be detected and discarded.
  • Behavior that is realistic but rare in training data, such as a parked car pulling out, can be difficult for the learned prior to generate.
Loading 2112.05077v2…