Source-linked AI summary
Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles, Part II: Assessment Results
Henry X. Liu, Tinghan Wang, Xintao Yan, Haowei Sun, Zhijie Qiao, Kenneth Boyd, Shuo Feng, Greg Stevens, Greg McGuire
TL;DR
Existing vehicle safety programs do not adequately assess Level 4–5 AV behavior in complex, non-deterministic traffic. This paper applies a third-party framework to Autoware.Universe in simulation and on physical test tracks, finding 6 of 14 competencies and a crash rate about 1,000 times the average human-driver rate. The tests also uncover unknown unsafe events, including roundabout failures.
Problem
Existing vehicle testing programs and standards are inadequate for Level 4–5 AVs because they focus on vehicle features or system malfunctions and deterministic scenarios rather than holistic behavioral safety.
Source: §I · p. 1Method
The paper applies a third-party framework combining predefined-scenario BCT and environment-level DIT to Autoware.Universe in simulation, with physical-track testing as well.
Source: §I · pp. 1–2Results
Autoware.Universe passed 6 of 14 behavioral competencies and had a crash rate of 3.01 × 10^-3 crashes per mile, about 1,000 times the average human-driver rate.
Source: §I · p. 2Takeaways & Limitations
The framework identified unknown unsafe events, including roundabout navigation failures that caused unexpected stops and were resolved following developer feedback.
Source: §I · p. 2The BCT demonstrates traffic-law compliance with only one case—stopping at red lights—so additional scenarios are needed for local traffic laws and regulations.
Source: §V · p. 11
Abstract
from arXiv · showhide
Third-party evaluations of autonomous vehicle (AV) safety can play a vital role in improving public acceptance, building consumer confidence, and establishing effective safety standards. In Part I of this study, we propose a dedicated third-party testing initiative for systematically evaluating AV behavioral safety. In this paper, we validate our proposed framework using Autoware.Universe, an open-source Level 4 Automated Driving System (ADS), tested both in simulated environments and on the physical test track at the University of Michigan's Mcity Testing Facility. The results indicate that Autoware.Universe possesses 6 out of 14 behavioral competencies and exhibited a crash rate of 3.01x10^-3 crashes per mile, approximately 1,000 times higher than the average human driver crash rate. During the tests, we also uncovered a number of unknown unsafe scenarios for Autoware.Universe. These findings underscore the necessity of behavioral safety evaluations for improving AV safety performance prior to widespread public deployment.
I. INTRODUCTION
The paper applies a third-party behavioral-safety framework to Autoware.Universe in simulation and on physical test tracks. The evaluation finds limited competency and a high estimated crash rate, while exposing previously unknown unsafe behavior.
I. INTRODUCTION
- Conventional vehicle testing programs and standards are inadequate for Level 4–5 AVs because they emphasize vehicle features or malfunctions and deterministic scenarios rather than holistic, probabilistic behavior.AVs’ deep-learning-based behavior is non-deterministic, while safety-critical events are too rare for practical brute-force mileage testing.Source: §I · p. 1
- The evaluation framework pairs the Behavioral Competency Test for predefined scenarios with the Driving Intelligence Test for unknown failure modes and statistically measured event rates.The DIT uses a Naturalistic and Adversarial Driving Environment and importance sampling to expose long-tail risks without bias.Source: §I · p. 1
- The study evaluates Autoware.Universe across 14 urban-driving competencies spanning major collision configurations, using high-fidelity scenarios informed by real-world data.The selected collision types account for 63.3% of reported crashes in the study area.Source: §I · p. 2
- Autoware.Universe passed 6 of 14 behavioral competencies and had an estimated crash rate of 3.01 × 10^-3 crashes per mile, about 1,000 times the average human-driver rate.The cited average human-driver rate is 3.00 × 10^-6 crashes per mile.Source: §I · p. 2
- Testing uncovered unknown unsafe events, including roundabout navigation failures that caused unexpected stops and were resolved after developer feedback.The findings point to software vulnerabilities that can be addressed through a closed-loop development process.Source: §I · p. 2
- Physical tests at Mcity demonstrated the framework beyond simulation, with two Tesla BCT competencies passed and Autoware-equipped Lincoln MKZ DIT testing indicating room to reduce risk exposure.The physical tests were limited in scope and intended to demonstrate feasibility rather than comprehensive testing.Source: §I · p. 2
II. EXPERIMENTAL SETUP OF THE BEHAVIORAL SAFETY ASSESSMENT
The experimental setup develops BCT competencies from urban driving tasks and creates parameterized, risk-stratified test cases. The section’s process is illustrated with a cut-in scenario.
A. Development process of the Behavioral Competency Test
- The BCT selects 14 urban-ODD competencies covering single- and multi-agent situations and all local multi-vehicle collision configurations, which represent 63.3% of reported Washtenaw County crashes in 2021.The set demonstrates the methodology within resource constraints; it is not intended to represent the full urban ODD.Source: A. Development process of the… · p. 2, Figure 1 · p. 3
- Each competency is developed through scenario modeling, calibration of parameter distributions, and generation of NADE test cases; the cut-in example is modeled using relative speed and longitudinal distance.Because the AV is treated as a black box, its speed cannot be controlled; the BV changes lanes at constant speed along a sine-function trajectory.Source: A. Development process of the… · p. 2, Figure 2 · p. 3
- For cut-in calibration, 11,701 Argoverse 2 segments inform the NDE distribution, while cases are sampled across five risk levels with greater density in high-risk regions.Risk is based on the AV’s minimum required deceleration to avoid collision, benchmarked against human-driver deceleration distributions.Source: A. Development process of the… · pp. 2–3
- Each test case runs three times; any collision run counts as failure, and one failed scenario makes the AV fail the BCT.The repeated runs account for stochastic variability in the AV and testing systems.Source: A. Development process of the… · p. 3
B. Development process of the Driving Intelligence Test
The DIT builds complete simulated traffic environments, then amplifies rare safety-critical events to assess AV performance over entire trips. Its Mcity simulation reproduces real-world crash rates and distributions with reported fidelity.
B. Development process of the Driving Intelligence Test
- DIT uses TeraSim to construct complete urban trips: the NDE reproduces statistically realistic traffic, while the NADE amplifies rare critical events.This contrasts with BCT’s independent modeling of individual scenarios.Source: B. Development process of the Driving… · p. 3
- The Mcity simulation yielded 2.93 × 10^-6 crashes per mile versus 3.00 × 10^-6 in five-year Michigan ground-truth data, and replicated crash composition across road geometries and types.The simulated environment includes a 1,000-foot arterial, six signalized intersections, three roundabouts, and other urban roads; traffic-pattern variety lacked direct real-world comparison data.Source: B. Development process of the Driving… · p. 4, Figure 3 · p. 4
- NADE testing uses adversarial background vehicles to accelerate safety-critical event exposure by orders of magnitude while retaining unbiased assessment and can reveal failures missed by static scenario tests.DIT estimates crash rates, types, and severity across many runs in an interactive, dynamic environment.Source: B. Development process of the Driving… · p. 4
C. AV under test
The study tests Autoware.Universe, an open-source Level 4 AV system, in a defined urban operating domain at Mcity. Evaluation begins in simulation and extends to Tesla and Autoware-equipped physical vehicles.
C. AV under test
- The AV under test is Autoware.Universe, an open-source Level 4 system evaluated in typical U.S. urban conditions at 30 mph, during daytime, in sunny or cloudy weather.The paper notes that the platform is accessible and widely used in research, though its performance may not yet match commercial systems such as Waymo.Source: C. AV under test · pp. 4–5
- Testing took place at the University of Michigan’s Mcity facility, first in simulation and then with a Tesla Model 3 using FSD v12.5.5 and an Autoware-equipped Lincoln MKZ.Physical testing incorporates vehicle dynamics and other effects that are difficult to reproduce in simulation.Source: C. AV under test · p. 5
A. Results of the Behavioral Competency Test
Autoware.Universe passed 6 of 14 behavioral competencies in the BCT. The detailed results show conservative crosswalk responses and braking behavior that spans under- and over-braking.
A. Results of the Behavioral Competency Test
- Autoware.Universe passed 6 of 14 behavioral competencies; Table I reports case counts, pass/fail outcomes, collision rates, and phase-specific behavior categories.The AV failed eight competencies: (d)–(i), (k), and (l).Source: A. Results of the Behavioral… · p. 5, Table I · p. 5
- Among six passed competencies, Autoware.Universe was conservative in 81% of cut-in and 67% of partially blocked-lane cases; 51% of same-direction lane-encroachment cases were aggressive pre-encroachment.It waited several seconds after a VRU crossed at a crosswalk in 71% of cases. After cut-in or lane encroachment, it resumed normal driving within a human-like recovery time in almost all cases and made the correct stop-or-go decision at the traffic light.Source: A. Results of the Behavioral… · p. 5
- 58 of 150 crosswalk cases (39%) had minimum clearance above 8 m in the Conservative zone; the rest were in the Normal zone (3–8 m).Fig. 4 plots minimum longitudinal clearance against distance available at case start.Source: A. Results of the Behavioral… · p. 6, Figure 4 · p. 6
- 140 of 150 crosswalk cases (93%) had Conservative or Ultra-Conservative PET; six had PET below 1 s in the Aggressive zone.The conservative cases reflect waiting several seconds after the VRU left the conflict zone before entering it.Source: A. Results of the Behavioral… · p. 6, Figure 5 · p. 6
- In the braking diagnosis, 100 of 150 cases (67%) were Normal; the highest-required-deceleration case under-braked, while the lowest-required-deceleration case over-braked.R1-24 had 4.48 m minimum clearance; R2-41 had 10.03 m.Source: A. Results of the Behavioral… · p. 6, Figure 6 · p. 6
B. Results of the Driving Intelligence Test
The Driving Intelligence Test estimated Autoware.Universe’s crash rate and uncovered unsafe behaviors, including errors that were difficult to find through scenario-based testing.
B. Results of the Driving Intelligence Test
- 3.01 × 10^-3 crashes per mile for Autoware.Universe, versus 3.00 × 10^-6 crash per mile for an average human driver.The estimate is about 1,000 times higher; Fig. 7 presents crash-rate, crash-type, and crash-severity results.Source: Figure 7 · p. 7, B. Results of the Driving Intelligence… · p. 8
- Around 3,500 episodes over 936 wall-time hours exposed head-on, sideswipe, angle, and rear-end crashes across the testing route.The route included urban arterials, signalized intersections, and single- and double-lane roundabouts, with randomized background-vehicle behavior and initialization.Source: B. Results of the Driving Intelligence… · p. 8
- A subtle relative position at a roundabout triggered an obstacle-avoidance replanning error that left the AV stuck; developers later proposed a fix that resolved it.The issue did not occur every time a background vehicle merged, making it difficult to detect through scenario-based approaches.Source: B. Results of the Driving Intelligence… · p. 8
- The AV failed to yield proactively to aggressive vehicles despite having right-of-way, while a separate prediction error caused it to stop for a vehicle facing a red light.The right-of-way behavior led to multiple crashes; the prediction error arose because the module did not incorporate traffic-light information.Source: B. Results of the Driving Intelligence… · p. 8
IV. DEMONSTRATION OF BEHAVIORAL SAFETY ASSESSMENT ON PHYSICAL TEST TRACK
Physical-track experiments with a Tesla Model 3 and a Lincoln MKZ were conducted to validate the framework’s applicability in closed testing facilities, not to produce comprehensive results.
IV. DEMONSTRATION OF BEHAVIORAL SAFETY ASSESSMENT ON PHYSICAL TEST TRACK
- Physical tests with a Tesla Model 3 and a Lincoln MKZ aimed to validate framework applicability in closed testing facilities, not generate comprehensive results.The authors cite resource constraints as the reason for this limited objective.Source: §IV · p. 8
- The physical experiments followed simulation testing of Autoware.Universe and demonstrated application of the framework to real vehicles.The stated purpose was to show how third-party evaluation could be applied in actual testing procedures.Source: §IV · p. 8
- The testing took place on Mcity physical test tracks, with demonstrations of events provided in Supplementary Movie 5.Source: §IV · p. 8
A. Demonstration of the Behavioral Competency Test
The Behavioral Competency Test used robotic proxies and RTK state measurement to test two safety-critical scenarios; Tesla’s FSD passed all ten sampled cases.
A. Demonstration of the Behavioral Competency Test
- Robotic platforms carrying a dummy vehicle and dummy child substituted for real vehicles and pedestrians in the two scenarios.The UFO Pro and UFO Nano platforms were equipped with RTK GPS; the AV also had RTK GPS for real-time state collection and scenario triggering.Source: A. Demonstration of the Behavioral… · p. 9, Figure 8 · p. 9
- The test system used RTK state information to set scenario conditions but transmitted no information to the AV, which relied on its own sensors and algorithms.The AV was treated as a black box during testing.Source: A. Demonstration of the Behavioral… · p. 9
- All ten sampled Tesla FSD test cases passed across the two tested scenarios.The scenarios were detecting and responding to a left-turning vehicle and to a VRU crossing without a crosswalk.Source: A. Demonstration of the Behavioral… · p. 9
- The tests demonstrate that BCT can evaluate commercial vehicles on physical test tracks without relying on data APIs.The illustrated cases show the AV stopping for an aggressive vehicle and for a jaywalking VRU.Source: A. Demonstration of the Behavioral… · p. 9, Figure 9 · p. 10
B. Demonstration of the Driving Intelligence Test
The field Driving Intelligence Test used Mcity’s augmented-reality platform to test Autoware.Universe with a real vehicle and demonstrated a collision with a red-light-running vehicle.
B. Demonstration of the Driving Intelligence Test
- For the field DIT, Autoware.Universe ran on a retrofitted Lincoln MKZ, and ground-truth object states were fed directly into the stack after bypassing perception.The vehicle’s planned trajectory was executed using Preview Control.Source: B. Demonstration of the Driving… · p. 10
- Mcity’s augmented-reality platform synchronized simulated traffic with the physical world, enabling virtual background vehicles to interact with the real AV.This allowed testing without putting large numbers of real vehicles on the track, reducing costs and safety risks.Source: B. Demonstration of the Driving… · p. 10
- Field tests used the same route and conditions as simulation tests and produced crash events consistent with the simulation results.The presented event is one example from the physical test-track experiment.Source: B. Demonstration of the Driving… · p. 10
- At a signalized intersection, Autoware.Universe decelerated but collided with a vehicle that ran a red light despite the AV having right-of-way.The authors note that a more proactive safety maneuver could have avoided the collision.Source: B. Demonstration of the Driving… · p. 10
V. CONCLUSION
The paper presents a standardized, black-box framework for assessing AV behavioral safety. It combines behavioral-competency checks with safety-event assessment and identifies simulation fidelity and evaluation scope as remaining challenges.
V. CONCLUSION
- The framework pairs minimum behavioral-competency requirements with assessment of measurable safety-related events, such as crash rates.It uses naturalistic and adversarial scenarios and is designed for simulation and physical test tracks.Source: §V · pp. 10–11
- In the Autoware.Universe case studies, simulation bypassed perception and supplied object ground-truth states, leaving the sim-to-real gap unresolved.The passage identifies sensor-input fidelity and realistic human-agent behavior over extended periods as sources of that gap.Source: §V · p. 11
- The study demonstrates traffic-law compliance assessment with only a red-light case, while mobility and congestion effects remain outside its scope.The passage calls for more scenarios aligned with local laws and dedicated frameworks for those broader traffic effects.Source: §V · p. 11