Source-linked AI summary
Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques
Ding Zhao, Henry Lam, Huei Peng, Shan Bao, David J. LeBlanc, Kazutoshi Nobukawa, Christopher S. Pan
TL;DR
The paper addresses the high time and cost of evaluating AVs when naturalistic driving rarely exposes them to safety-critical lane changes. It models human-controlled cut-ins from SPMD data, accelerates testing by skewing behavior statistics, and uses Cross Entropy to search for the skewing parameters. The resulting conflict, crash, and injury rates are estimated 2,000 to 20,000 times faster than naturalistic driving tests in simulation, with potential to reduce development and validation time.
Problem
Naturalistic-Field Operational Tests expose AVs infrequently to safety-critical events, making thorough evaluation time-consuming and expensive.
Method
The paper models human-controlled lane changes from SPMD data, skews stochastic-variable distributions toward riskier behavior, and iteratively searches for parameters using Cross Entropy.
Results
2,000 to 20,000 times faster than naturalistic driving tests in simulation, the approach accurately estimates lane-change conflict, crash, and injury rates for a modeled AV.
Takeaways & Limitations
The approach can provide statistical conclusions and objectively selected critical scenarios, with potential to reduce AV development and validation time.
Takeaways & Limitations
The authors identify the need for a more comprehensive human-controlled vehicle model as additional Safety Pilot and other project data become available.
Abstract
from arXiv · showhide
Automated vehicles (AVs) must be evaluated thoroughly before their release and deployment. A widely-used evaluation approach is the Naturalistic-Field Operational Test (N-FOT), which tests prototype vehicles directly on the public roads. Due to the low exposure to safety-critical scenarios, N-FOTs are time-consuming and expensive to conduct. In this paper, we propose an accelerated evaluation approach for AVs. The results can be used to generate motions of the primary other vehicles to accelerate the verification of AVs in simulations and controlled experiments. Frontal collision due to unsafe cut-ins is the target crash type of this paper. Human-controlled vehicles making unsafe lane changes are modeled as the primary disturbance to AVs based on data collected by the University of Michigan Safety Pilot Model Deployment Program. The cut-in scenarios are generated based on skewed statistics of collected human driver behaviors, which generate risky testing scenarios while preserving the statistical information so that the safety benefits of AVs in non-accelerated cases can be accurately estimated. The Cross Entropy method is used to recursively search for the optimal skewing parameters. The frequencies of occurrence of conflicts, crashes and injuries are estimated for a modeled automated vehicle, and the achieved accelerated rate is around 2,000 to 20,000. In other words, in the accelerated simulations, driving for 1,000 miles will expose the AV with challenging scenarios that will take about 2 to 20 million miles of real-world driving to encounter. This technique thus has the potential to reduce greatly the development and validation time for AVs.
I. INTRODUCTION
AV evaluation must capture interactions with human-controlled vehicles, but naturalistic testing rarely encounters safety-critical events and can require decades of driving. The paper proposes an accelerated lane-change evaluation procedure that reshapes human-driving statistics while retaining real-world statistical information.
- Evaluation motivation: AVs will coexist and primarily interact with human-controlled vehicles for decades, making imperfect human driving a central evaluation disturbance.The paper also notes that 70-90% of motor vehicle crashes are attributed to human errors.
- Evaluation motivation: N-FOTs collect data from vehicles driven naturally on public roads, but dangerous-event exposure is sufficiently low to make evaluation time-consuming.The paper reports approximately 0.53 million miles per police-reported crash and 99 million miles per fatal crash in the U.S. during 2013.
- Existing approaches: Existing evaluation approaches include N-FOTs, simulations using naturalistic data, test matrices, and worst-case scenario evaluation.Worst-case methods identify challenging control situations but do not account for their probability of occurrence.
- Existing approaches: Worst-case evaluation can reveal control-system weaknesses, but its omission of scenario probability limits real-world risk interpretation and design comparisons.The limitation follows directly from the method’s focus on the most challenging scenarios rather than their occurrence frequencies.
- Proposed approach: The proposed procedure models human vehicles from N-FOT data, skews lane-change probability densities toward riskier behavior, searches for optimal parameters, and estimates real-world host-vehicle performance.The approach uses accelerated results together with the accelerated model’s statistics to recover performance estimates for real-world driving.
- Study scope: Unsafe lane changes are studied as a cut-in scenario because a short time gap can cause a frontal collision for the following vehicle.The paper uses SPMD naturalistic driving data, including measurements from equipped vehicles, to develop the lane-change model.
B. Lane Change Models
The lane-change model represents gap acceptance using lane-changing-vehicle speed, range, and TTC at the lane-change time. Reciprocal range and TTC variables capture rare, less-safe events, while fitted distributions generate complete scenarios.
- Model variables: Gap acceptance is modeled through v_L(t_LC), R_L(t_LC), and the AV’s Time To Collision.These variables describe lane-changing speed, range, and collision timing at the lane-change instant.
- Data selection: 403,581 lane-change events were observed, with 173,692 having negative range rate and retained for model construction.The lane-changing vehicle speed is assumed constant during the lane change.
- Variable transformation: Reciprocal range and TTC place rarer, less-safe scenarios in the distribution tails for fitting and accelerated sampling.Larger range and TTC indicate safer scenarios, whereas smaller values indicate rarer, less-safe cases.
- Range model: The range distribution is modeled across vehicle-speed conditions, with vehicle speed having little influence on reciprocal range.The fitted reciprocal-range distributions use candidate distributions evaluated by goodness-of-fit procedures.
- TTC model: Reciprocal TTC is approximated by Pareto and exponential distributions, differing by 0.23% in relative BIC; the exponential form is used.Its parameters vary with lane-changing vehicle speed and are smoothed by linear interpolation and extrapolation.
- Scenario generation: Scenario generation samples v_L, generates R_L, samples reciprocal TTC, and calculates host-vehicle speed using the model equations.Range and TTC can be modeled independently given the same lane-changing speed because range has limited effect on TTC.
III. ACCELERATED EVALUATION
Accelerated evaluation addresses the slow execution of crude Monte Carlo for rare safety-critical events. It uses importance sampling and Cross Entropy optimization to skew samples while preserving statistical correctness.
- Motivation: Crude Monte Carlo is limited for rare-event simulation because estimating low-probability safety events requires many samples.The section introduces importance sampling for rare events and Cross Entropy to optimize its use.
- Approach: The accelerated-evaluation workflow applies importance sampling to lane-change AV evaluation and tunes the sampling distribution with Cross Entropy.The method is presented as a sequence from crude Monte Carlo limitations to importance sampling and parameter optimization.
A. Monte Carlo Estimation
Monte Carlo estimation uses unbiased random samples to estimate rare-event probabilities, but its required sample size grows sharply as event probability decreases. Confidence-interval relative half-width quantifies estimation accuracy.
- Estimator: Monte Carlo estimates a stochastic process expectation from unbiased statistical samples.The lane-change motion vector x describes possible events, including crashes as rare events.
- Crude Monte Carlo: The crude Monte Carlo estimator averages independent and identically distributed samples of the event indicator.The target is the probability that the rare event E occurs.
- Accuracy: Relative half-width measures confidence-interval width relative to the probability being estimated at confidence level 100(1 − α)%.The required sample count is determined by requiring this relative half-width to remain below a specified constant β.
- Rare-event limitation: When the event probability γ is rare, the required crude-Monte-Carlo test number n tends to infinity, making the method slow.This follows from the sample-size relationship needed to maintain a satisfactory relative half-width.
B. Importance Sampling (IS)
Importance sampling accelerates rare-event estimation by sampling more likely safety-critical events under a modified density and correcting the resulting bias with likelihood ratios. The resulting estimator remains unbiased under an absolute-continuity condition.
- Core idea: Importance sampling reduces variance for rare events by replacing the original density f(x) with a density f*(x) that increases rare-event likelihood.Using a different density introduces sampling bias, which must be corrected when computing crash rates.
- Validity condition: The modified density must be absolutely continuous with respect to the original density within the rare-event set.This condition guarantees validity of the likelihood ratio.
- Bias correction: The likelihood ratio compensates for the modified sampling density so the importance-sampling estimator remains unbiased.Samples are generated under f*(x) and weighted by the likelihood ratio within the event estimator.
- Scope: Although the paper uses a continuous model, similar importance-sampling approaches can also be applied to discrete models.This is stated as a scope extension of the presented formulation.
- Evaluation effort: The importance-sampling formulation also supports relative-half-width and minimum-test-number calculations for accelerated evaluation.The paper uses these quantities to characterize the required evaluation effort.
C. Accelerated Evaluation of Automated Vehicles in Lane Change Scenarios
The lane-change evaluation defines rare events around unsafe cut-ins by modeling the lane-changing vehicle's speed, time-to-collision, and range. Importance sampling modifies these motion distributions to accelerate evaluation while retaining the variables describing the original scenarios.
- Unsafe cut-ins are defined as events in which a slower lane-changing vehicle cuts in front of the AV.
- A lane change belongs to the event set when its minimum range falls below the critical range at any time during the lane-change duration.
- The lane-changing vehicle is represented by speed, time-to-collision, and inverse range variables.
- Importance sampling uses modified probability density functions for the scenario variables to accelerate evaluation.
D. Searching for optimal IS distributions with the Cross Entropy approach
The Cross Entropy approach searches for an effective importance-sampling distribution by iteratively reducing its divergence from the ideal rare-event distribution. The paper uses exponential change-of-measure families and updates parameters across velocity conditions.
- The Cross Entropy method searches iteratively for a good importance-sampling distribution within a prescribed parametric family.
- The theoretical optimal importance-sampling distribution is the conditional distribution given occurrence of the rare event.
- The zero-variance distribution would make every estimate have zero variance, but it cannot be implemented because the target probability is unknown.
- Cross Entropy selects a member of the candidate family by minimizing its Kullback–Leibler divergence from the zero-variance distribution.
- At each iteration, Monte Carlo samples from the current distribution and likelihood ratios are used to derive the next distribution parameters.
- The implementation uses exponential change-of-measure distributions, including an exponential approximation for inverse range, across low-, medium-, and high-velocity conditions.
IV. SIMULATION ANALYSIS
An automated vehicle model was designed to demonstrate the proposed accelerated evaluation approach in lane-change scenarios.
- An automated vehicle model demonstrates the proposed accelerated evaluation approach in lane-change scenarios.
A. Design of Test Automated Vehicle
The test AV combines Adaptive Cruise Control and Autonomous Emergency Braking, with switching governed by time-to-collision. Its AEB model is based on a 2011 Volvo V60 but is explicitly an approximation.
- The AV model includes both Adaptive Cruise Control and Autonomous Emergency Braking.
- When TTC_L is at least TTC_AEB, the vehicle uses ACC; otherwise, the AEB algorithm becomes active.
- ACC is approximated by a discrete Proportional-Integral controller targeting a desired time headway.
- The modeled AEB is based on a 2011 Volvo V60 and uses speed-dependent TTC_AEB thresholds with a deceleration build-up rate limit.
- The AEB model is an approximation and may not accurately represent production-vehicle AEB systems.
- A first-order lag models the transfer from commanded to actual acceleration, while more accurate vehicle simulations such as CarSim can also be used.
B. Simulation Analysis
The simulations estimate conflict, crash, and moderate-to-fatal injury rates for lane-change scenarios using accelerated sampling and compare convergence with naturalistic simulations. Accelerated evaluation reaches the target confidence substantially sooner while retaining agreement with non-accelerated estimates.
- Event analysis: The analysis estimates conflict, crash, and moderate-to-fatal injury events in lane-change scenarios for an automated vehicle model.Injury events are defined as MAIS2+ outcomes, representing moderate-to-fatal injuries.
- Parameter optimization: Cross Entropy searches optimize skewing parameters for conflict and crash events, with 100 lane changes per conflict iteration and 500 per crash iteration.The crash search emphasizes TTC because the AEB function is mainly affected by time-to-collision in the crash analysis.
- Conflict-rate estimation: The accelerated conflict-rate estimate is unbiased because it converges to the estimate from non-accelerated crude Monte Carlo simulations.Both accelerated and non-accelerated simulations were conducted to assess performance and credibility.
- Conflict-rate estimation: 364 accelerated simulations reached relative half-width below β = 0.2 with 80% confidence, compared with 5.90e3 naturalistic simulations.The relative half-width is the confidence-interval half-width relative to the estimated probability.
- Accelerated rates: Crash and injury events achieve higher accelerated rates than conflicts because their occurrence probabilities are lower and Importance Sampling performs better for rarer events.The injury-rate estimate uses the modified crash-event statistics and a nonlinear moderate-to-fatal injury model.
V. CONCLUSION
The paper presents an accelerated lane-change evaluation approach based on naturalistic driving data and estimates AV conflict, crash, and injury rates 2,000 to 20,000 times faster than naturalistic simulations. The approach is intended to reduce development and validation time while retaining statistical conclusions and objectively selected critical scenarios.
- Conclusion: The proposed approach models lane changes from the Safety Pilot Model Deployment database to accelerate AV evaluation.The paper estimates conflict, crash, and injury rates for a given AV model.
- Conclusion: 2,000 to 20,000 times faster than naturalistic driving tests, the method accurately estimates lane-change conflict, crash, and injury rates in simulation.The conclusion presents the accelerated rate across the three event types.
- Conclusion: The approach has potential to reduce development and validation time by providing statistical conclusions and objectively selected critical scenarios.Future extensions include other human-controlled models, Importance Sampling distribution families, scenarios, and testing platforms.