Source-linked AI summary
Accelerated Evaluation of Automated Vehicles in Car-Following Maneuvers
Ding Zhao, Xianan Huang, Huei Peng, Henry Lam, David J. LeBlanc
TL;DR
AV evaluation must provide realistic safety evidence without the excessive duration of naturalistic testing or the limited real-world relevance of predefined test matrices. The paper proposes Accelerated Evaluation for car-following by intensifying stochastic HV–AV interactions and reweighting accelerated-test outcomes with Importance Sampling. Simulations show evaluation-time reductions of 300 to 100,000 times while accelerated crash-rate estimates converge to naturalistic-test estimates.
Problem
Existing AV evaluation must reconcile realistic but slow naturalistic testing with faster approaches that may not rigorously estimate naturalistic crash rates or safety benefits.
Method
AE models stochastic lead-HV motion from naturalistic data, modifies its statistics to intensify safety-critical interactions, and applies Importance Sampling to estimate naturalistic safety performance.
Results
Evaluation time for crash, injury, or conflict events is reduced by 300 to 100,000 times, while accelerated crash-rate estimates converge to naturalistic-test estimates.
Takeaways & Limitations
The technique has potential to reduce AV development and validation time and to support government assessment and automotive product improvement.
Abstract
from arXiv · showhide
The safety of Automated Vehicles (AVs) must be assured before their release and deployment. The current approach to evaluation relies primarily on (i) testing AVs on public roads or (ii) track testing with scenarios defined in a test matrix. These two methods have completely opposing drawbacks: the former, while offering realistic scenarios, takes too much time to execute; the latter, though it can be completed in a short amount of time, has no clear correlation to safety benefits in the real world. To avoid the aforementioned problems, we propose Accelerated Evaluation, focusing on the car-following scenario. The stochastic human-controlled vehicle (HV) motions are modeled based on 1.3 million miles of naturalistic driving data collected by the University of Michigan Safety Pilot Model Deployment Program. The statistics of the HV behaviors are then modified to generate more intense interactions between HVs and AVs to accelerate the evaluation procedure. The Importance Sampling theory was used to ensure that the safety benefits of AVs are accurately assessed under accelerated tests. Crash, injury and conflict rates for a simulated AV are simulated to demonstrate the proposed approach. Results show that test duration is reduced by a factor of 300 to 100,000 compared with the non-accelerated (naturalistic) evaluation. In other words, the proposed techniques have great potential for accelerating the AV evaluation process.
I. INTRODUCTION
AV evaluation must balance realistic but slow naturalistic testing against fast test-matrix evaluation that may not estimate real-world safety. The proposed Accelerated Evaluation approach intensifies safety-critical interactions and uses Importance Sampling to assess naturalistic safety benefits.
- AV evaluation asks how to choose test scenarios and interpret test results for real-world performance.
- Test matrices are efficient and comparable but may let driverless AVs optimize for predefined scenarios while neglecting others.
- N-FOT provides unpredicted scenarios and directly reflects real-world performance, but its low exposure to crashes makes it prohibitively time-consuming.
- Crude Monte Carlo simulations reduce field-test time and cost but cannot accelerate testing when safety-critical scenarios have low exposure probabilities.
- AE models correlated HV maneuvers, intensifies safety-critical interactions, and analyzes crash, injury, and conflict events.
II. MODEL OF THE CAR-FOLLOWING SCENARIOS
The paper focuses on car-following, a fundamental driving task, and replaces predefined lead-vehicle motions with a stochastic, dynamic lead-HV environment.
- Car-following is a fundamental driving task commonly tested with predefined static, constant-speed, or constant-deceleration lead-vehicle motions.
- This research models the lead HV stochastically and tests the AV in a dynamic environment.
A. Extraction of Naturalistic Car-following Events
Naturalistic car-following events are extracted from the SPMD database using instrumented vehicles and consistency criteria, producing a large event set for modeling lead-vehicle motion.
- The SPMD database recorded naturalistic driving from 2,842 equipped vehicles in Ann Arbor for over two years and logged 34.9 million miles by April 2016.
- SPMD instrumentation measured relative position to the lead vehicle and lane-tracking measures from painted boundaries and road edges.
- The extraction criteria exclude cut-ins and lane changes by either vehicle.
- Car-following events had durations exceeding 50 seconds, yielding 163,332 detected events.
B. Lead Human Controlled Vehicle Model
The lead HV is modeled as a stochastic process learned from naturalistic car-following data, with next-step acceleration predicted from current acceleration and velocity.
- A stochastic driver model is adopted because it is suitable for estimating AV safety benefits such as crash counts.
- The next-step lead-vehicle acceleration is predicted from current acceleration and velocity.
- Lead-vehicle velocity is calculated and acceleration is estimated by forward Euler differentiation followed by moving-average smoothing.
- The driver-model parameter vector is estimated with weighted least squares, which is less influenced by outliers than standard least squares.
- Acceleration sequences are represented across indexed car-following samples, with each sequence containing lead-HV acceleration steps.
C. Automated Vehicle Model
The AV model combines longitudinal vehicle dynamics with a longitudinal controller that follows the lead HV velocity and maintains distance. The dynamics are linearized around equilibrium and represented as a first-order lag system, while the controller uses range and range-rate feedback.
- Model structure: The AV model consists of longitudinal vehicle dynamics and a longitudinal control system.The control system is designed to follow the lead HV velocity and maintain a proper distance.
- Vehicle dynamics: The longitudinal dynamics account for mass, longitudinal force, road grade, gravity, rolling resistance, aerodynamics, and wind speed.
- Vehicle dynamics: The dynamics are linearized around an equilibrium point using a Taylor series expansion.Velocity deviation is defined relative to the equilibrium velocity.
- Vehicle dynamics: Assuming zero wind speed and zero road grade, Laplace transformation yields a first-order lag system for longitudinal dynamics.
- Longitudinal control: The controller regulates range error with a discretized PI controller and range rate with a proportional controller.The initial range equals the desired range to start the test from equilibrium.
III. ACCELERATED EVALUATION
The accelerated-evaluation formulation discretizes the car-following model into state-space form and modifies lead-HV acceleration statistics to create more intense maneuvers. Stochastic optimization identifies shifts that target critical events while preserving likely realizations.
- Accelerated distribution: The accelerated model adds stepwise biases to the mean of the lead-HV acceleration distribution.This modifies the acceleration distribution to generate more intense maneuvers.
- State-space model: The section discretizes the car-following model and rewrites it in state-space form.The lead-HV velocity and vehicle dynamics are discretized using zero-order hold and transformed into discrete equations.
- Optimization: Stochastic optimization calculates the optimal shift sequence to make a critical event occur at a selected time while maximizing its likelihood.The resulting optimization can be rewritten in quadratic-programming form.
- State-space model: The lead-HV velocity is updated in discrete time using velocity deviation and discretized vehicle-force dynamics.
- State-space model: The resulting discrete state equations include range, lead-vehicle velocity deviation, AV velocity deviation, and force-related state terms with physical constraints.
B. Accelerated Evaluation
The Accelerated Evaluation procedure intensifies lead-HV maneuvers, randomizes termination time, simulates critical events, and uses Importance Sampling to recover naturalistic safety probabilities. The method targets crash, injury, and conflict metrics while controlling estimation accuracy.
- Procedure: The procedure calculates optimal mean shifts, randomizes termination time, runs accelerated simulations, and estimates real-world safety benefits.The same procedure is applied to crash, injury, and conflict metrics.
- Critical events: Crash and injury use a critical distance of 0, whereas conflict uses a critical distance of 30 feet.
- Mean-shift optimization: The optimal shift targets a critical event at a selected time while maximizing the likelihood of the shifted maneuver.The shift is computed offline for each possible event time.
- Termination time: Termination time is randomized from the minimum feasible event time through the maximum test horizon to represent events occurring at different moments.The real termination time is distinct from the randomized optimization time.
- Importance Sampling: Importance Sampling compensates for bias introduced by replacing the naturalistic HV distribution with an accelerated distribution.The likelihood ratio compares naturalistic and accelerated event likelihoods.
- Estimation accuracy: The estimator is accepted when the relative half-width falls below a threshold, while increasing test counts drive convergence toward the target probability.The relative half-width measures confidence-interval half-width relative to the estimated probability.
IV. RESULTS AND ANALYSIS
The study compares accelerated and naturalistic simulations for crash, injury, and conflict rates. In the reported crash-rate example, a uniform acceleration distribution did not effectively accelerate evaluation, and convergence remained weak after one million simulations.
- Evaluation setup: Crash, injury, and conflict rates are evaluated using both accelerated and naturalistic simulations.The naturalistic baseline uses Monte Carlo simulation.
A. Simulation Results with uniform distribution
The uniform-distribution baseline generates oscillatory but generally followable lead-vehicle motion, so crashes rarely occur and crash-rate convergence remains weak even after extensive simulation.
- Uniform-distribution baseline: The baseline uniform distribution generates lead-vehicle acceleration by sampling from a modified uniform model with parameter ϑ_ud = 6σ_ud.The likelihood ratio is used to account for the modified sampling distribution.
- Uniform-distribution baseline: A sample baseline maneuver shows lead-vehicle acceleration oscillating up and down, while the overall motion remains followable.The maneuver is presented as an example generated by the baseline accelerated evaluation approach.
- Crash-rate estimation: A million simulations produced weak convergence, so the uniform distribution did not effectively accelerate crash-rate evaluation.The stated reason is that the method does not consider correlations between samples in a dynamic system.
B. Results of the Proposed Accelerated Evaluation Process
The proposed Accelerated Evaluation intensifies human-driven lead-vehicle maneuvers and uses Importance Sampling to estimate crash, injury, and conflict rates against naturalistic simulations.
- Simulation behavior: Accelerated simulations produce stronger lead-vehicle acceleration and deceleration than naturalistic-driving simulations.Lead-vehicle speed profiles are compared in Fig. 9, and an accelerated maneuver leading to a crash is shown in Fig. 10.
- Scenario coverage: High-risk maneuvers occur frequently in accelerated simulations but not in current Euro-NCAP or ISO standard tests.The authors suggest that some such maneuvers could be considered in future government certification processes.
- Safety metrics: The method models crash and conflict as binary events and injury probabilistically, focusing on MAIS2+ injuries related to relative velocity.The injury probability is represented by a nonlinear model.
- Rate estimation: Accelerated crash-rate estimates converge to naturalistic estimates, demonstrating that the proposed Accelerated Evaluation is unbiased.Convergence is evaluated for crash, injury, and conflict rates using an 80% confidence level and β = 0.2.
- Acceleration performance: Conflict evaluation accelerates by 300 times, likely because Importance Sampling provides a larger acceleration rate for rarer target events.The reported comparison concerns the conflict case.
V. CONCLUSION
The paper proposes Accelerated Evaluation for car-following safety assessment by modifying lead-HV motion statistics and correcting accelerated-test results with Importance Sampling. Simulations indicate substantial reductions in evaluation time.
- Conclusion: The method modifies lead human-controlled vehicle statistics to intensify HV–AV interactions, then uses Importance Sampling to estimate naturalistic safety performance.A stochastic optimization method is used to minimize evaluation duration.
- Conclusion: 300 to 100,000 times: accelerated tests reduce evaluation time for crash, injury, or conflict events.The paper states that 1,000 simulated miles can expose an AV to scenarios requiring 300 thousand to 100 million real-world miles to encounter.