Source-linked AI summary
Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World
Daniel J. Fremont, Edward Kim, Yash Vardhan Pant, Sanjit A. Seshia, Atul Acharya, Xantha Bruso, Paul Wells, Steve Lemke, Qiang Lu, Shalin Mehta
TL;DR
AI-based autonomous vehicles require stronger testing, while the relationship between simulation and real-world track testing remains uncertain. The paper combines formal scenario and safety specifications with simulation-based test generation, track-test selection, execution, and analysis. Experiments with a real AV indicate that formal simulation can identify useful track tests, while simulation-to-track differences can be measured but remain substantial.
Problem
The paper addresses the need for better testing of AI-based autonomous vehicles and asks how effectively simulation can guide track testing and match real-world behavior.
Method
The approach combines formal scenario and safety specifications with VERIFAI-based simulation, test selection, real track execution, and analysis using an LGSVL digital twin.
Results
Formal simulation identified relevant track tests, with 5 of 8 simulated failure-test runs violating the safety property in reality and 93.3% of marginally safe runs satisfying it on track.
Takeaways & Limitations
Simulation can efficiently identify real-world failures, but measured trajectory deviations show it is not yet accurate enough to guarantee real-system behavior purely through simulation.
Abstract
from arXiv · showhide
We present a new approach to automated scenario-based testing of the safety of autonomous vehicles, especially those using advanced artificial intelligence-based components, spanning both simulation-based evaluation as well as testing in the real world. Our approach is based on formal methods, combining formal specification of scenarios and safety properties, algorithmic test case generation using formal simulation, test case selection for track testing, executing test cases on the track, and analyzing the resulting data. Experiments with a real autonomous vehicle at an industrial testing facility support our hypotheses that (i) formal simulation can be effective at identifying test cases to run on the track, and (ii) the gap between simulated and real worlds can be systematically evaluated and bridged.
I. INTRODUCTION
The paper addresses the need for better testing and verification of AI-based autonomous vehicles by examining whether formal simulation can guide track tests and how closely simulation matches reality. It proposes a formal simulation-to-track workflow for evaluating safety and bridging the two settings.
- AI-based components are increasingly central to autonomous vehicles, but documented failures and vulnerability to adversarial examples motivate better testing and verification.
- Simulation can cover large numbers of miles, exercise difficult corner cases, and diagnose real-world issues, but its fidelity and value relative to physical testing remain open questions.
- Track testing runs the real vehicle and software in controlled environments, yet it is expensive, labor-intensive, and time-consuming to prepare.
- The paper asks whether formal simulation can identify effective road tests and whether simulated and track-testing outcomes correspond for formally specified scenarios.
- The approach combines SCENIC scenario specifications, Metric Temporal Logic safety properties, VERIFAI falsification, and an LGSVL digital twin of an industrial test track.
Related Work
Prior autonomous-vehicle testing uses crash data, naturalistic driving data, benchmarks, scenario languages, and simulation-based libraries. This paper differs by formally specifying scenarios and safety properties, automating test synthesis, and evaluating the approach with both simulation and real track testing.
- Existing scenario-based testing draws on crash reports, naturalistic driving data, and human-driving benchmarks to construct tests and characterize AV requirements.
- Scenario description languages include XML-based OpenSCENARIO and GeoScenario, while SCENIC offers a flexible high-level alternative with different support and specialization trade-offs.
- Test scenario library generation mathematically describes scenarios and metrics but commonly relies on surrogate AV models and reports only simulation results.
- Other work evaluates AV performance across closed-track, simulation, and mixed-reality settings, whereas this paper generates tests from formally specified scenarios and outcomes.
- The paper claims to be the first to combine formal scenario and safety-property specification, simulation-based test generation, track-test selection, and systematic simulation-to-track evaluation.
II. BACKGROUND
The paper builds its testing workflow on SCENIC, VERIFAI, and LGSVL: formal scenario models define environments, VERIFAI searches them for tests, and LGSVL executes full-stack AV simulations.
- A. SCENIC: A Scenario Specification Language: SCENIC is a probabilistic programming language for modeling cyberphysical-system environments through distributions over scenes, objects, and agents.
- A. SCENIC: A Scenario Specification Language: SCENIC supports concise scenario descriptions through geometry syntax and declarative constraints, including rare traffic situations for specialized test generation.
- B. VERIFAI: VERIFAI searches parameterized environment and system spaces to generate concrete tests, using methods ranging from random sampling to global optimization.
- C. The LGSVL Simulator: LGSVL is an open-source simulator that supports full-stack autonomous-vehicle testing with realistic 3D environments, multiple sensors, and configurable maps, weather, traffic, and pedestrians.
D. GoMentum Station Testing Facility
The methodology combines a secure, configurable test facility with a photorealistic simulation model and formal scenario-based test generation. It records simulation and track data to assess test selection, trace correspondence, and unsafe behavior while recognizing possible simulator nondeterminism.
- D. GoMentum Station Testing Facility: GoMentum Station provides a large secure AV test site with roads, intersections, testing zones, and an urban test area restricted to speeds below 30 mph.
- D. GoMentum Station Testing Facility: The workflow creates a photorealistic simulation environment, formalizes safety properties, generates and selects tests, executes them on the real AV, and analyzes resulting data.
- D. GoMentum Station Testing Facility: Each simulation test case is defined by a parameter vector configuring the AV, vehicle dynamics, environment, and other simulated agents.
- D. GoMentum Station Testing Facility: The method formally specifies a probability distribution over test cases as a scenario represented by a SCENIC program.
- D. GoMentum Station Testing Facility: Industrial simulators and AV stacks may be nondeterministic, so identical configurable parameters do not always define a unique simulation run.
IV. SIMULATION
The simulation uses a photorealistic digital twin and a formally specified intersection scenario in which an AV encounters a hesitant crossing pedestrian. SCENIC encodes scenario parameters and dynamics, while an MTL property monitors pedestrian clearance.
- Simulation Model Creation: The simulation environment is a photorealistic digital twin of GoMentum’s Urban A test area, built from LiDAR, imagery, a textured 3D mesh, annotations, and HD maps.The same HD maps were used in simulation and real-world testing.
- Test Scenario: The test scenario has the AV turning right at an intersection while a hesitant pedestrian crosses its path.The AV and pedestrian start from fixed positions and orientations, and the pedestrian follows a straight-line trajectory.
- Test Scenario: Scenario parameters control the pedestrian’s crossing delay tstart, walking distance dwalk before hesitation, and hesitation duration thesitate.The parameters are assigned uniform distributions over appropriate ranges in the SCENIC program.
- Scenario Encoding: SCENIC specifies the scenario distribution and behavior, with DriveTo and Hesitate implemented through simulator and vehicle-control APIs.The program is a simplified encoding of the test scenario.
- Safety Properties: Safety is specified as ϕsafe = G(dist > 2.5 m), requiring the AV–pedestrian distance to remain above 2.5 m at every time point.The threshold accounts for measuring distance between the centers of the AV and pedestrian.
C. Identifying Safe and Unsafe Test Cases
VERIFAI falsifies the formally specified scenario by sampling parameterized tests and monitoring the safety property. Satisfaction values identify both violations and marginal cases for further testing.
- Identification Procedure: VERIFAI sampled scenario parameters, ran corresponding tests in the LGSVL Simulator, and monitored ϕsafe for violations.The procedure used the SCENIC program’s parameter distribution and stored error cases.
- Simulation Results: 1294 simulated test cases were generated, and 2% violated ϕsafe.The error table retained parameter values for violating cases and their quantitative satisfaction values.
- Safety Metric: For ϕsafe, the satisfaction value ρ is the minimum AV–pedestrian distance over the test minus 2.5 m.Negative ρ indicates a violation, while larger values indicate greater clearance.
- Test-Candidate Identification: Plots of ρ over tstart and dwalk expose marginal regions that are candidates for testing, although collisions do not follow a simple parameter relationship.The visualization uses darker points for smaller satisfaction values and therefore closer approaches.
D. Test Case Selection
Simulation results were used to select track tests spanning unsafe, marginally safe, and robustly safe behavior. The selection relied on low-dimensional visualizations of satisfaction values and included practical track constraints.
- Test Case Selection: Seven parameterized track tests were selected from simulation as failure/unsafe, marginally safe, or robustly safe cases.The categories were chosen from the observed satisfaction-value plots.
- Test Case Selection: The main unsafe cluster appeared in the lower-left of the tstart–dwalk plot, with additional unsafe runs at large tstart values.The latter cases were harder to implement because of track-equipment limitations.
A. Experimental Setup
The experiments used a real Apollo-equipped autonomous vehicle and a pulley-based pedestrian dummy at GoMentum. Track execution produced repeated runs for simulation-selected cases, with safety violations classified by observed severity.
- Vehicle Setup: The test vehicle was a 2018 Lincoln MKZ Hybrid equipped with drive-by-wire, LiDAR, cameras, and high-accuracy GPS/IMU sensors.Apollo 3.56 ran on an industrial PC with an NVIDIA GTX-1080 GPU.
- Test Equipment: The pedestrian was implemented at GoMentum using a pulley-based 4Active surfboard platform.
- Track Testing: Track setup required about 8 people for 4 hours, followed by another half day for approximately 25 runs of 10–15 minutes each.The authors describe setup and execution as tedious and time-consuming.
- Track Testing: The experiment executed 7 selected test cases and produced 23 total runs at GoMentum.Runs violating ϕsafe were highlighted in the results table.
- Result Classification: Violations were classified as collisions, near-collisions, or unsafe runs with a larger margin, represented by red, orange, and yellow rows.A side approach can violate the center-distance property without producing a near-collision because the vehicle is longer than it is wide.
- Metrics: Minimum distance was measured from the centers of the AV and pedestrian in both simulation and track testing, and TTC was approximated from distance and relative speed.A center-based distance can remain positive even when a collision occurs.
VI. DATA ANALYSIS
The analysis shows that formal simulation identified several real-world failures, while track outcomes could differ substantially from simulation. Replay-based resimulation and trajectory metrics exposed both sim-to-real discrepancies and nondeterminism in the AV stack.
- Effectiveness of the methodology: 5 of 8 track runs selected from two simulated failure tests violated ϕsafe in reality, including one collision.In 93.3% of marginally safe runs, the AV satisfied ϕsafe in reality; no violations occurred in safe tests.
- Simulation versus track testing: Track outcomes sometimes diverged sharply from simulation, with M3 producing a 1.94 m minimum distance versus 5.85 m in simulation.M2 minimum distances ranged from 3–14 m across different track runs.
- Sources of discrepancy: Potential discrepancy sources included mismatched initial conditions, vehicle or pedestrian dynamics, sensory inputs, and hardware-dependent timing.Some setup errors could be reduced with improved equipment, but rendering details and perception responses remain difficult to quantify.
- Resimulation analysis: Resimulation used recorded track traces and five identical simulations per test to assess nondeterminism and reproduce observed trajectories.The trajectories overlapped considerably but differed, including sharper simulated AV turns and tightly clustered yet nonidentical simulations.
- Quantifying the sim-to-real gap: The Skorokhod metric quantified timed-trace discrepancies, with real-to-simulation deviations on the order of 5–11 meters or seconds.These deviations indicate qualitative similarity without sufficient accuracy for deriving real-system guarantees purely from simulation.
- Nondeterminism: Although the simulator was deterministic, resimulations still varied substantially, implicating Apollo’s asynchronous interface or internal nondeterminism.Lower-variance tests may be more reproducible on the track, whereas high variance may indicate undesirable sensitivity of the AV stack.
C. Why Did the Autonomous Vehicle Fail?
Track-data replay identified perception, planning, and prediction failures underlying unsafe behavior, including the crash caused by missed pedestrian detection.
- Failure analysis: Replay in Dreamview identified perception, planning, and prediction failures that led to unsafe behavior.Apollo failed to detect the pedestrian for at least 1 s in the crash case; unsafe planning was the most common failure.
- Failure analysis: Apollo alternated between yielding and overtaking the pedestrian, and once predicted that the pedestrian would walk around the AV.The latter prediction incorrectly made moving forward appear safe.
- Interpretation: The analysis illustrated how the methodology can help find and debug failure cases without assigning fault to Apollo.The experiments used Apollo version 3.5 from January 2019 on the available hardware.
VII. CONCLUSION
The paper presents formal-methods-based scenario testing for simulation and real-vehicle track testing, and reports that formal simulation effectively identified relevant track tests. Future work targets broader scenarios, automated failure analysis, and improved simulation–track alignment.
- Conclusion: The approach combines formal scenario-based test generation in simulation with test selection, execution, and analysis for a real AV.It leverages the SCENIC language and VERIFAI toolkit.
- Conclusion: Formal simulation effectively identified relevant tests for track testing with a real AV.The study also compared simulation and track time-series data quantitatively and qualitatively.
- Future work: Future work includes higher-dimensional scenarios, more automated analysis of perception, planning, and prediction failures, improved simulation, and better-matched track equipment.