Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

• AI-based AV components can fail in real-world perception, while simulation leaves questions about real-world fidelity and costly track testing makes effective test selection important.Track testing uses real hardware and software in controlled environments but can be expensive, labor-intensive, and time-consuming to set up.Source: §I · p. 1
• The approach combines SCENIC scenario models, Metric Temporal Logic safety specifications, VERIFAI falsification, and an LGSVL digital twin of GoMentum Station to select track tests.Selected test cases are systematically mapped to hardware controlling agents on the track.Source: §I · p. 2
• 62.5% of unsafe simulated test cases led to unsafe track behavior, including a collision, while 93.3% of safe simulated cases led to safe behavior with no collisions.The results also shed light on potential causes of AV failure in perception, prediction, and planning.Source: §I · p. 2
• AV and pedestrian trajectories were qualitatively similar across simulation and track tests, but time-series and minimum-distance metrics showed differences; repeated simulations also varied less.The repeated-run variations were attributed to non-deterministic behavior of the AV stack.Source: §I · p. 2

Related Work

• Unlike tests derived from crash and naturalistic-driving data (,,,) and PEGASUS benchmarking and requirements work, this approach uses formal specifications and automated test-case synthesis.Source: Related Work · p. 2
• SCENIC [2] complements XML-based OpenSCENARIO and GeoScenario, while offering more probabilistic and geometric modeling than M-SDL and open-source verification, debugging, and synthesis support through VERIFAI.Source: Related Work · p. 2
• Prior TSLG methods ([17], [18]) rely on human-data surrogate models and report simulation only, while Abbas et al. search unsafe scenarios in simulation but lack a formal language for scalable ODD representation.Source: Related Work · p. 2
• Closed-track and mixed-reality work, compares AV performance using standard tests rather than computational test generation from formal scenarios and outcomes, and the SAE EDGE report discusses modeling fidelity without formal evaluation methods.Source: Related Work · p. 2
• The authors claim this is the first formal-methods approach to evaluating ML-based AV safety across formal scenario and safety-property specification, simulation-based test generation and track-test selection, and systematic simulation-to-track gap measurement.The evaluation spans simulation and the real world.Source: Related Work · p. 2

A. SCENIC: A Scenario Specification Language

• SCENIC is a probabilistic programming language that defines distributions over cyberphysical scenes, including object and agent configurations.Source: A. SCENIC: A Scenario Specification… · p. 2
• Its geometric syntax and declarative constraints support concise scenario descriptions, while its formal semantics provide a precise environment model.Source: A. SCENIC: A Scenario Specification… · p. 3
• SCENIC programs can sample rare scenarios, such as a disabled car blocking the road, to generate specialized training data that augments existing datasets.Source: A. SCENIC: A Scenario Specification… · p. 3

B. The VERIFAI Toolkit

• VERIFAI parameterizes environments and system configurations, then searches them with methods from random sampling to global optimization to generate simulation tests.Source: B. The VERIFAI Toolkit · p. 3
• Each VERIFAI test checks a system-level specification, records violations, and uses results to guide further search; supported applications include falsification, fuzz testing, debugging, data augmentation, and parameter synthesis, [23].Source: B. The VERIFAI Toolkit · p. 3

C. The LGSVL Simulator

• LGSVL [5] supports ROS, ROS2, and CyberRT platforms including Apollo and Autoware, enabling full-vehicle simulation with multiple sensor outputs in a safe, deterministic, realistic 3D environment.Source: C. The LGSVL Simulator · p. 3
• LGSVL exposes map, weather, time, traffic, and pedestrian settings and allows simulation control through a Python API.Source: C. The LGSVL Simulator · p. 3

D. GoMentum Station Testing Facility

• GoMentum Station is currently the largest secure AV test site in the United States, located in Concord, CA, with 19 miles of roadways, 48 intersections, and 8 distinct testing zones over 2,100 acres.Source: D. GoMentum Station Testing Facility · p. 3
• Experiments used GoMentum’s urban or downtown zone: mostly flat streets with several intersection types, visible lane markings, urban and suburban features, and speed limits under 30 mph.The approximately 20-foot-wide roads had one-way, stop, yield, and speed-limit signs.Source: D. GoMentum Station Testing Facility · p. 3

III. METHODOLOGY

• A SCENIC program defines a probability distribution over test cases; a parameter valuation defines a test, with one simulation run assumed per test but potentially multiple track runs.Industrial simulators or AV stacks may be nondeterministic, so configurable parameters may not uniquely define a simulation run.Source: §III · p. 3
• The workflow combines a detailed simulation model, formal scenario and safety specifications, and VERIFAI falsification to classify tests as safe or violating.VERIFAI assigns each trace a quantitative robustness value: positive values indicate satisfaction, and larger values mean greater modifications would be needed to falsify the specification.Source: §III · pp. 3–4
• VERIFAI safe and error tables are analyzed using PCA, clustering, or visualization to identify behavior modes and select representative tests for track execution.Direct visualization is an option for low-dimensional spaces.Source: §III · p. 4
• Track implementation requires controlling dynamic agents according to SCENIC test parameters and matching tests to the capabilities and limitations of track hardware.Agent parameters can include starting locations, motion start times, and velocities.Source: §III · p. 4
• Track testing records AV videos, sensor and software logs, and hardware data to assess test selection, simulation-track correspondence, and causes of unsafe or interesting behavior.Recorded hardware data includes GPS beacons and equipment used to control dynamic agents.Source: §III · p. 4

A. Simulation Model Creation

• A unified 3D mesh of Urban A was built from LiDAR and location data, textured with tens of thousands of processed images, and loaded into LGSVL Simulator.The mesh represents road-surface bumps and cracks as well as surrounding objects such as curbs, buildings, signs, and sidewalks.Source: A. Simulation Model Creation · p. 4
• The mesh was annotated with drivable-area details and compiled as a simulation environment with HD maps used both in simulation and in the AV for real-world testing.Annotations include lane lines, driving directions, road speeds, crosswalks, intersections, and traffic signs.Source: A. Simulation Model Creation · p. 4

B. Test Scenario and Safety Properties

• The test scenario has the AV turn right through an intersection while a hesitant pedestrian crosses, parameterized by crossing start delay, distance before hesitation, and hesitation duration.The pedestrian walks at a fixed 1 m/s; the SCENIC program assigns uniform distributions to parameters, including tstart ∈ (7, 15).Source: B. Test Scenario and Safety Properties · p. 4, Figure 4 · p. 5
• Apollo and LGSVL API calls command the AV through the intersection and the pedestrian along the specified crossing behavior.These dynamic behaviors are encoded in the SCENIC program.Source: B. Test Scenario and Safety Properties · p. 5, Figure 4 · p. 5
• The monitored safety property requires the center-to-center AV–pedestrian distance to remain greater than 2.5 m at every time point, formalized as ϕsafe = G(dist > 2.5 m).The threshold accounts for the AV’s 2.1 m center-to-front-bumper and 0.95 m center-to-side distances.Source: B. Test Scenario and Safety Properties · p. 5

C. Identifying Safe and Unsafe Test Cases

• VERIFAI generated 1,294 test cases, of which 2% violated the safety specification ϕsafe.It sampled SCENIC parameters, ran tests in the LGSVL Simulator, and monitored for specification violations.Source: C. Identifying Safe and Unsafe Test… · p. 5
• A prototype SCENIC extension with dynamics defines DriveTo and Hesitate using the behavior property, with its description deferred to future work.Source: C. Identifying Safe and Unsafe Test… · p. 5
• The simulator interface did not measure distance between the AV and pedestrian surfaces; improving that measurement was left for future work.Source: C. Identifying Safe and Unsafe Test… · p. 5

D. Test Case Selection

• The ρ plot shows unsafe runs in a main bottom-left cluster and toward the right at large tstart, where track equipment made tests harder to implement.The latter unsafe cases were harder to implement because of track-equipment limitations.Source: Figure 5 · p. 5, D. Test Case Selection · p. 5

C. Identifying Safe and Unsafe Test Cases

• The satisfaction value ρ measures minimum AV–pedestrian distance minus 2.5 m, and saved safe runs help identify marginal test regions.Runs near zero ρ are close to collision; marginal candidates satisfy ϕsafe but lie near failures.Source: C. Identifying Safe and Unsafe Test… · p. 5
• The irregular relationship between parameters and collisions made systematic falsification crucial; a few manually selected tests would not suffice.The plotted ρ values vary with start delay tstart and walk distance dwalk.Source: C. Identifying Safe and Unsafe Test… · p. 5, Figure 5 · p. 5

D. Test Case Selection

• The authors selected seven track tests spanning failure/unsafe, marginally safe, and robustly safe/success cases.Success cases came from a neighborhood of safe tests; marginal cases satisfied ϕsafe but were close to failure and could fail under real-hardware imprecision.Source: D. Test Case Selection · p. 5

A. Experimental Setup

• The test vehicle was a 2018 Lincoln MKZ Hybrid equipped with drive-by-wire, LiDAR, cameras, and GPS/IMU sensors.Apollo 3.56 ran on an industrial PC with a GTX-1080 GPU; GPS/IMU RTK correction provided approximately 2 cm position accuracy, and Apollo 3.56 was the most recent version supported by the hardware.Source: A. Experimental Setup · pp. 5–6, B. Test Results · p. 6
• Because the pedestrian platform could not trigger automatically by distance or delay, testers manually triggered it using a countdown timer keyed to when the AV began moving.The scenario was reparameterized from start distance to elapsed start delay because manual distance estimates were inaccurate; the timer-based manual trigger was more accurate.Source: A. Experimental Setup · p. 6
• The pedestrian was an articulated soft target mounted on a powered surfboard platform programmed to reproduce the scenario’s hesitating-pedestrian motion.The dummy was designed to have a sensor signature similar to real pedestrians.Source: A. Experimental Setup · p. 6
• Track setup and calibration took about eight people four hours, followed by another four hours for around 25 runs of 10–15 minutes each.The paper characterizes the track-testing effort as tedious and time-consuming.Source: A. Experimental Setup · p. 6

B. Test Results

• Across seven simulation-selected test cases, 23 GoMentum runs were performed, with safety-property violations classified as collisions, near-collisions, or unsafe runs with larger margins.Table II summarizes the runs; highlighted rows violate ϕsafe.Source: Table II · p. 6, B. Test Results · p. 6
• A ϕsafe violation from a side approach is not always a near-collision because the car is much longer than it is wide.The authors note that this geometry accounts for yellow unsafe-but-not-near-collision rows in Table II.Source: B. Test Results · p. 6, Table II · p. 6
• Minimum distance was measured from the AV center to the pedestrian, while TTC was approximated from distance divided by relative speed until lane crossing or path intersection.Thus, minimum distance can remain positive even when a collision occurs, as in F1 Run1.Source: B. Test Results · p. 6

A. Effectiveness of our Methodology

• 5 of 8 runs of the two failure tests identified in simulation violated ϕsafe on track, including one collision.The paper reports these results in Table II.Source: A. Effectiveness of our Methodology · p. 7
• 93.3% of marginally safe runs satisfied ϕsafe in reality.A violation occurred in a marginally safe test, but none occurred in safe tests.Source: A. Effectiveness of our Methodology · p. 7
• A violation occurred in a marginally safe test, but none occurred in safe tests.Source: A. Effectiveness of our Methodology · p. 7
• The authors note that the number of runs was small because of limited time and resources.They describe the results as demonstrating that simulation can efficiently identify real-world failures.Source: A. Effectiveness of our Methodology · p. 7
• Test M3, safe in simulation, produced a 1.94 m minimum distance on its first track run, versus 5.85 m in simulation.Source: A. Effectiveness of our Methodology · p. 7
• Test M2 showed track-run minimum distances ranging from 3–14 m.Source: A. Effectiveness of our Methodology · p. 7

B. The Gap Between Simulation and Reality

• Possible discrepancies include mismatched initial conditions, vehicle or pedestrian dynamics, sensory inputs, and Apollo timing across different hardware.Automated triggering could reduce some implementation error, but rendering details that affect Apollo perception are difficult to quantify.Source: B. The Gap Between Simulation and… · p. 7
• To measure sim-to-real gap from hard-to-quantify variation, traces from track tests were used to extract implemented policy, then five identical simulations per test assessed nondeterminism of hardware and AV stack.Source: B. The Gap Between Simulation and… · p. 7
• The example trajectories overlap considerably, yet the simulated AV needed a slight starting-position adjustment and turned more sharply than the real AV, possibly because uphill driving was modeled imprecisely.The five resimulations were tightly clustered but not identical.Source: B. The Gap Between Simulation and… · p. 7, Figure 6 · p. 7
• Across four logged track tests, real-to-simulation Skorokhod distances were 5–11 meters or seconds, too large to derive real-system guarantees purely from simulation.The runs were qualitatively similar, but the simulation was not yet accurate enough for that transfer.Source: B. The Gap Between Simulation and… · p. 7, Table III · p. 7
• Identical test cases still produced substantially varying simulations, implicating Apollo or its asynchronous interface and potentially signaling sensitivity that affects track reproducibility.The LGSVL Simulator is deterministic; tests with lower variance may be more reproducible on the track, while high variance may indicate undesirable AV-stack sensitivity.Source: B. The Gap Between Simulation and… · pp. 7–8

C. Why Did the Autonomous Vehicle Fail?

• Apollo failed to detect a pedestrian for at least 1 s in a perception failure responsible for the crash shown in Fig. 1.Source: C. Why Did the Autonomous Vehicle Fail? · p. 8
• The most common failure was unsafe planning, as Apollo alternated between yielding to and overtaking the pedestrian.Source: C. Why Did the Autonomous Vehicle Fail? · p. 8
• In a prediction failure, Apollo incorrectly predicted that the pedestrian would walk around the AV and that moving forward was safe.Source: C. Why Did the Autonomous Vehicle Fail? · p. 8

VII. CONCLUSION

• The authors combined SCENIC and VERIFAI for formal scenario-based test generation in simulation and for selecting and executing road tests, demonstrating that simulation can identify relevant tests for a real AV.Source: §VII · p. 8
• They compared time-series data recorded in simulation and on the track quantitatively and qualitatively, and made their data and analysis scripts available online.Source: §VII · p. 8
• Future work includes testing more complex, higher-dimensional scenarios and automating more detailed analysis of perception, planning, and prediction failures.Source: §VII · p. 8
• The authors also propose improving simulation to bridge the sim-to-real gap and developing track test equipment that better matches simulation.Source: §VII · p. 8