Source-linked AI summary
Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components
Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito, James Kapinski
TL;DR
Autonomous vehicles lack generally agreed testing methods, and their ML components are difficult to characterize. Sim-ATAV addresses this by combining virtual closed-loop simulation, covering-array test generation, and requirement falsification; the framework identifies critical behaviors and counterexamples, including boundary-case collisions.
Problem
Autonomous driving systems lack generally agreed methods for testing, debugging, and certification, while ML components are difficult to characterize across all circumstances.
Method
Sim-ATAV evaluates closed-loop autonomous-vehicle behavior in simulation, using covering arrays for discrete scenarios and falsification to search scenario parameters.
Results
The framework identifies problematic scenarios and critical vehicle behaviors, including glancing counterexamples near the boundary between satisfying and falsifying a property.
Takeaways & Limitations
Simulation-based adversarial testing can evaluate autonomous vehicles with ML components at the closed-loop system level and support reliability improvement.
Takeaways & Limitations
The framework’s scene rendering could be made more realistic, and its sensing could be extended beyond camera data.
Abstract
from arXiv · showhide
Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.
1 Introduction
Sim-ATAV addresses the lack of agreed testing methods for autonomous vehicles with difficult-to-characterize ML components by testing closed-loop behavior in simulation. It combines scenario-space adversarial testing, covering arrays, and requirement falsification to identify problematic behaviors.
- Motivation and framework: Sim-ATAV tests autonomous driving systems with ML components by evaluating their closed-loop properties in a virtual environment.The framework includes the perception system within the vehicle’s feedback cycle.
- ML components: Autonomous driving designs use DNNs and other neural networks for semantic segmentation, end-to-end control, and intermediate decisions such as risk assessment.These architectures may classify objects, estimate their positions, or directly produce actuator commands from images.
- ML components: ML components are difficult to test and verify because their high parameter counts and data-learned behavior make all circumstances hard to characterize.The paper illustrates this difficulty with AlexNet’s 60 million parameters and 1.2 million training images.
- Adversarial testing: Unlike image-space adversarial testing, the framework searches scenario-configuration perturbations that can produce misclassifications and ultimately collisions.The authors use virtual environments with advanced 3D models and rendering tools to reduce the gap between simulation and the real world.
- Contributions: The paper extends covering arrays to closed-loop CPS applications with ML and introduces falsification methods that search discrete and continuous scenario parameters.The contributions also characterize perception-system problems directly in configuration space rather than image space.
2 Related work
Prior work mainly verifies or tests neural networks at the component level, often through output-range verification or adversarial input perturbations. This paper instead searches for unsafe behavior at the autonomous-vehicle system level, directly over closed-loop simulations.
- NN-focused methods: NN verification and testing research includes output-range verification for bounded inputs and adversarial sample generation that changes classification decisions.These categories use techniques including SMT, LP, gradient-based search, and MILP.
- System-level testing: The paper targets NN testing at the system level, whereas much previous work evaluates ML components without considering closed-loop behavior.For autonomous driving, the stated evaluation goal is closed-loop performance.
- System-level testing: Related methods use static images to find candidate counterexamples and then simulate closed-loop behavior, while this work searches unsafe closed-loop behaviors directly.The proposed search uses a global optimizer guided by a cost function defined on closed-loop behaviors.
- Comparison with related frameworks: A comparable framework uses a game engine only to visualize results after testing simpler vehicle models that may not represent real behaviors well.That framework cannot simulate the autonomous vehicle system in a closed loop including perception.
- Broader context: The work also relates to robustness-guided falsification and automatic test generation for autonomous vehicles, but its methods and testing goals differ from those approaches.Its related context includes detecting boundary-case failures and generating tests for autonomous vehicles and driver-assistance systems.
3 Preliminaries
The framework models autonomous driving systems as simulated traces with continuous and discrete parameters, evaluates Signal Temporal Logic properties through robustness, and searches for falsifying or glancing behaviors.
- System Modeling: The framework represents a system as a model with states, inputs, parameters, and a simulation function that generates system behavior over time.Parameters may combine continuous-valued variables with discrete domains, such as Boolean values or finite lists of agent-car colors.
- Signal Temporal Logic: Signal Temporal Logic specifies real-time properties of observable simulation traces using predicates and Boolean or temporal operators such as eventually, always, and until.Predicates can constrain system states, inputs, and parameters, allowing reachability, safety, event-sequence, and infinite-behavior requirements.
- Robustness Semantics: Robust semantics assign each trace a real value, with positive values indicating satisfaction and negative values indicating violation of the specification.The robustness value also serves as a fitness measure for model checking and falsification.
- Robustness-Guided Model Checking: Sampled trajectories can be visualized against an STL specification, where trajectories entering forbidden regions produce negative values in the corresponding robustness landscape.The example samples initial conditions on a 0.05 grid over [−1,1]^2 and identifies red-box regions as falsifying behavior.
- Falsification and Critical System Behaviors: Falsification searches simulated traces minimizing robustness to find counterexamples, while glancing examples target violations near zero that correspond to low-speed collisions.These boundary cases are intended to help designers distinguish critical behaviors from high-velocity collisions or unavoidable failures.
- Covering Arrays: Covering arrays reduce exhaustive testing by guaranteeing t-way combinations, but faults caused by combinations involving more than t parameters may remain undetected.The number of possible input combinations can grow exponentially with the number of parameters.
4 Framework
Sim-ATAV provides a virtual simulation environment combining perception, control, environment modeling, and image rendering for autonomous-vehicle testing. It generates scenarios with covering arrays, then uses cost-function-guided falsification to search for problematic continuous configurations.
- Simulation Environment: Sim-ATAV integrates a vehicle perception system, controller, physical-environment model, and 2D image rendering in a virtual testing framework.The framework uses freely available, low-cost tools and can run on a standard desktop PC.
- Simulation Environment: The perception subsystem feeds front-camera images to SqueezeDet, which outputs object-detection boxes and corresponding class probabilities.SqueezeDet was trained on KITTI data and further trained on virtual images.
- Simulation Environment: The evaluations use only camera information, making the controller more sensitive to image-based perception than systems using additional sensors.LIDAR and radar are deliberately neglected for practical considerations.
- Simulation Environment: Webots models scenario physics and renders images corresponding to a virtual front-mounted vehicle camera.The environment framework supports autonomous-driving scenarios while modeling robotic and vehicle-related physical behavior.
- Test Generation and Falsification: Covering arrays generate discrete and discretized-continuous scenarios, after which the most promising case initializes S-TaLiRo falsification.Each simulation trace is evaluated with a cost function, which guides optimization of subsequent scenarios over continuous parameters.
5 Testing Application
The case study evaluates Sim-ATAV on a parameterized autonomous-driving scenario using covering arrays and falsification to find glancing behaviors. CA+SA combines discrete coverage with continuous-space optimization and identifies glancing cases more effectively than Global UR and CA+UR.
- 5.1 Scenario Setup: The evaluation uses a challenging one-way-road scenario with parked vehicles, stopped traffic, and a jaywalking pedestrian.The pedestrian crosses ahead of the ego car, reverses direction, and returns toward the right sidewalk.
- 5.1 Scenario Setup: Scenario parameters include vehicle position, vehicle colors and models, pedestrian clothing, walking speed, and initial longitudinal position.Vehicle colors and pedestrian clothing each have five possible values, as do the car models.
- 5.2 STL Specifications: The STL requirement specifies that the ego vehicle should not hit another vehicle or pedestrian.Collision predicates use whether an object is in the ego vehicle’s front corridor and its minimum Euclidean distance from the front bumper.
- 5.3 Testing Strategies: CA+UR uses covering arrays for discrete variables but explores continuous variables unguided, whereas CA+SA guides continuous search with simulated annealing and robustness-based cost.CA+SA uses the absolute robustness value as its cost function for identifying behaviors close to the safe–unsafe boundary.
- 5.3.3 Covering Arrays and Simulated Annealing (CA+SA): CA+SA can transform a non-failing, near-failure covering-array test into a rear-end collision while keeping discrete parameters unchanged.The collision follows perception detection failures for the white vehicle ahead.
- 5.4 Experiment Results: CA+UR outperforms Global UR for glancing-case identification and guarantees coverage of every 2-way parameter combination.CA+UR retains an unguided search over real-valued parameters.
- 5.4 Experiment Results: CA+SA outperforms both Global UR and CA+UR in identifying glancing behaviors while retaining covering-array coverage guarantees.Its robustness values are closer to zero on average, matching the target behavior.
6 Conclusions
The framework generates adversarial tests for autonomous vehicles in closed loop, including a DNN-based perception system. It combines covering arrays for discrete parameter combinations with simulated annealing for corner cases, while identifying future extensions for perception retraining and richer sensing or rendering.
- The framework generates simulation-based adversarial tests while the autonomous vehicle evolves in closed loop with its DNN-based perception system.
- Covering arrays test combinations of discrete parameters, while simulated annealing searches for corner-case behaviors.
- Identified counterexamples are intended for future retraining and improvement of the DNN-based perception system.
- Future extensions include additional sensor types and more realistic scene rendering tools.