Source-linked AI summary
Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation
Anthony Corso, Peter Du, Katherine Driggs-Campbell, Mykel J. Kochenderfer
TL;DR
Autonomous-vehicle validation is costly, and AST can repeatedly find similar or unavoidable failures. The paper augments AST’s reward with driving-relevant information and reports a more diverse set of useful failure scenarios.
Problem
Real-world AV testing is costly, while AST can repeatedly find similar failures or scenarios where failure is unavoidable.
Source: §I · p. 1Method
The approach augments AST rewards with RSS to identify improper AV responses and trajectory dissimilarity to encourage distinct failures.
Source: §I · p. 1Results
Reward augmentation yielded a more diverse set of failures, including trajectories with improper vehicle responses using RSS and high failure diversity using trajectory dissimilarity.
Source: §V · p. 6Takeaways & Limitations
The authors conclude that augmented AST finds more useful failures that can aid in evaluating an AV policy.
Source: §V · p. 6
Abstract
from arXiv · showhide
Determining possible failure scenarios is a critical step in the evaluation of autonomous vehicle systems. Real-world vehicle testing is commonly employed for autonomous vehicle validation, but the costs and time requirements are high. Consequently, simulation-driven methods such as Adaptive Stress Testing (AST) have been proposed to aid in validation. AST formulates the problem of finding the most likely failure scenarios as a Markov decision process, which can be solved using reinforcement learning. In practice, AST tends to find scenarios where failure is unavoidable and tends to repeatedly discover the same types of failures of a system. This work addresses these issues by encoding domain relevant information into the search procedure. With this modification, the AST method discovers a larger and more expressive subset of the failure space when compared to the original AST formulation. We show that our approach is able to identify useful failure scenarios of an autonomous vehicle policy.
I. INTRODUCTION
Autonomous-vehicle validation is costly in the real world and difficult to cover through simulation because driving scenarios are expansive. This paper augments AST rewards to target more relevant and diverse failures.
I. INTRODUCTION
- Real-world validation is costly, while the expansive scenario space makes adequate coverage through naive simulation sampling difficult.The challenge matters for safety involving human drivers and vulnerable participants such as pedestrians.Source: §I · p. 1
- AST uses reinforcement learning to search an MDP for likely system failures, but can find unavoidable failures and repeatedly return similar ones.The reward depends on failure, such as a vehicle collision, and transition likelihood.Source: §I · p. 1
- The proposed reward augmentation uses RSS to target improper AV behavior before collisions and a dissimilarity metric to encourage distinct failures.The authors report that these modifications produce a wider range of failures than the existing AST reward.Source: §I · p. 1
- The paper applies augmented AST to an AV-and-pedestrian scenario and evaluates whether it finds relevant failure cases against existing AST.Its contributions include a framework and two reward-augmentation techniques for AV validation.Source: §I · p. 1
II. BACKGROUND
This section supplies background on applying AST to an autonomous-driving scenario and on the relevant RSS rules.
II. BACKGROUND
- The section introduces the AST application to an autonomous-driving scenario and the relevant RSS rules.Source: §II · p. 1
A. Adaptive Stress Testing
AST frames validation of an autonomous policy interacting with an environment as an MDP, then uses reinforcement learning to search for probable failures. In this application, the search acts on pedestrian acceleration and AV sensor noise in a crosswalk scenario.
A. Adaptive Stress Testing
- AST frames policy validation as an MDP and applies reinforcement learning to search for the system-under-test’s most probable failure modes.The simulator exposes initialization, stepping, and terminal-state checks; a reward function returns feedback to the solver.Source: A. Adaptive Stress Testing · p. 2
- The user defines critical states as failures; in the driving example, they can include the AV approaching too close to or colliding with another agent.AST requires a simulator and possible actions, and the simulator may be a black box if it exposes the required interface.Source: A. Adaptive Stress Testing · p. 2
- The example SUT is an IDM-controlled AV approaching a crosswalk, where it should detect crossing pedestrians and brake adequately.Each agent’s state comprises velocity and position.Source: A. Adaptive Stress Testing · p. 2
- AST controls pedestrian acceleration and AV sensor noise; sampled actions use acceleration values from -1 to 1 and noise values from 0 to 1.Source: A. Adaptive Stress Testing · p. 2
- The reward penalizes failure to find a collision using constants α = 10 000 and β = 1000, and includes vehicle-pedestrian distance and action improbability.The distance is Euclidean; the action term is Mahalanobis distance related to the action’s log probability.Source: A. Adaptive Stress Testing · p. 2
B. Responsibility-Sensitive Safety
Responsibility-Sensitive Safety (RSS) classifies driving behavior by whether agents follow rules for responding to unsafe situations. The section applies longitudinal and lateral collision-avoidance rules to judge danger and proper response over time.
B. Responsibility-Sensitive Safety
- RSS assigns responsibility when an agent fails to follow the rules for a proper response after a situation becomes unsafe.A response is proper when permitted by the RSS rules.Source: B. Responsibility-Sensitive Safety · p. 2
- Safe-distance thresholds define longitudinal and lateral danger, and proper responses specify how agents should react in each direction.The combined rule selects a longitudinal or lateral response according to which danger threshold is reached first.Source: B. Responsibility-Sensitive Safety · pp. 2–3
- In the example collision, danger shifts from lateral to longitudinal as the pedestrian approaches, while the car is classified as proper at every timestep.Figs. 2 and 3 show danger categories and the vehicle's response classification, respectively.Source: B. Responsibility-Sensitive Safety · p. 3, Figure 2 · p. 3, Figure 3 · p. 3
III. METHODS
The methods section introduces two AST reward augmentations: RSS-based behavior classification and trajectory dissimilarity for failures.
III. METHODS
- The methods use RSS classification of proper and improper behavior and a trajectory-dissimilarity metric for failure trajectories.These are the two reward-augmentation approaches outlined in the section.Source: §III · p. 3
- RSS classification supplies a behavior-based signal for the reward augmentation.Source: §III · p. 3
- Trajectory dissimilarity supplies a separate signal based on failure trajectories.Source: §III · p. 3
A. RSS Rewards
The RSS reward augmentation restricts critical events to trajectories with sufficiently improper AV behavior and adjusts the reward using the improper-timestep fraction. Its rationale assumes higher improper fractions are more likely to end in collision.
A. RSS Rewards
- Critical events are limited to trajectories in which the AV has behaved improperly, with the improper-timestep fraction defined as fimp.fimp is the fraction of trajectory timesteps classified as improper by RSS.Source: A. RSS Rewards · p. 3
- fcrit is a tunable cutoff governing how many improper timesteps are required before the vehicle is considered at fault.The threshold is specified as 0 ≤ fcrit < 1.Source: A. RSS Rewards · p. 4
- The method assumes trajectories with a higher fraction of improper AV behavior are more likely to end in collision.RSS classifies an action as improper only when collision is already imminent and the vehicle is not appropriately resolving the situation.Source: A. RSS Rewards · p. 4
B. Trajectory Dissimilarity (TD) Rewards
The trajectory-dissimilarity reward compares normalized spatial trajectories and rewards discovery of diverse failures. A tunable parameter controls the diversity incentive.
B. Trajectory Dissimilarity (TD) Rewards
- Trajectories are normalized into n segments, each represented by its center of mass, before dissimilarity is measured.The trajectories may have different lengths, and n is predefined.Source: B. Trajectory Dissimilarity (TD) Rewards · p. 4
- The reward is modified to encourage discovery of highly diverse failure scenarios.The parameter γ controls the extent of this diversity incentive.Source: B. Trajectory Dissimilarity (TD) Rewards · p. 4
- The reward uses up to k top failure trajectories, with the number considered also bounded by how many failures AST has already found.The equation's parameter μ is defined as min(k, k′).Source: B. Trajectory Dissimilarity (TD) Rewards · p. 4
IV. EXPERIMENTS
The experiments evaluate RSS and trajectory-dissimilarity reward augmentation on an autonomous-vehicle scenario.
IV. EXPERIMENTS
- The experiments assess RSS and trajectory-dissimilarity reward augmentation on an autonomous-vehicle scenario.The software setup is based on the Adaptive Stress Testing Toolbox.Source: §IV · p. 4
A. RSS Reward Experiments
In a single-pedestrian experiment, RSS reward augmentation produced collision trajectories with improper vehicle responses, including an example where sensor noise contributed to failure.
A. RSS Reward Experiments
- The RSS experiment used TRPO to train a policy for high-likelihood collision events and recorded 1,000 sampled action sequences that ended in collision.The RSS parameters were chosen to approximate plausible vehicle and pedestrian accelerations, with response time ρ = 0 because the simulation had no built-in delay.Source: A. RSS Reward Experiments · p. 4
- With RSS reward augmentation, every trajectory had some improper timesteps, and most had improper behavior for more than 25% of simulation timesteps.Without RSS augmentation, most trajectories involved failures where the vehicle was not at fault, with fewer trajectories blaming the AV policy.Source: A. RSS Reward Experiments · p. 4, Figure 4 · p. 4
- In the RSS-augmented example, sensor noise contributed to the vehicle failing to stop for a pedestrian crossing in front of it.The vehicle behaved improperly in the final quarter of the trajectory; the generic-AST example instead showed a side impact while the vehicle maintained zero lateral velocity, which the paper describes as proper behavior.Source: A. RSS Reward Experiments · pp. 4–5, Figure 6 · p. 5, Figure 5 · p. 5
B. TD Reward Experiments
In the two-vehicle, two-pedestrian scenario, generic AST converged on one pedestrian-induced failure mode, while trajectory-dissimilarity augmentation uncovered additional failure types.
B. TD Reward Experiments
- The experiment used two vehicles approaching a crosswalk on a single-lane road, with one pedestrian on each side attempting to cross.Failure states included a lead car within 0.5 meters of a pedestrian or the two cars within 0.5 meters longitudinally.Source: B. TD Reward Experiments · p. 5
- The starting-state table identifies the initial states of each pedestrian and vehicle.The text refers to Table II for the initial conditions.Source: Table II · p. 5
- Without reward augmentation, AST returned a single failure type, despite finding the requested 25 failures.The repeated mode involved a pedestrian approaching a lead vehicle stopped well ahead of the crosswalk, a situation the paper describes as unavoidable by the car.Source: B. TD Reward Experiments · p. 5, Figure 7 · p. 5
- With trajectory-dissimilarity augmentation, AST explored a larger failure subset and found vehicle-induced vehicle/pedestrian and vehicle/vehicle failures alongside pedestrian-induced cases.The vehicle/pedestrian cases involved inadequate braking near a pedestrian in the crosswalk; the vehicle/vehicle cases involved the second car following too closely.Source: B. TD Reward Experiments · pp. 5–6, Figure 8 · p. 6
V. CONCLUSIONS
The paper concludes that RSS and trajectory-dissimilarity reward augmentation help AST find more diverse and useful autonomous-vehicle failures.
V. CONCLUSIONS
- RSS augmentation found trajectories with improper vehicle responses, while trajectory-dissimilarity augmentation found trajectories with greater failure diversity.The authors state that these more useful failures can aid autonomous-vehicle validation and suggest future work combining both augmentations in one reward framework.Source: §V · p. 6