PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Automated Driving System Testing: Landscapes and Trends

Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Yan Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, Yinxing Xue, Yang Liu

arXiv:2206.05961v2cs.SE

TL;DR

ADS testing must address complex module-specific behaviors, oracle difficulties, and the gap between simulation and real-world testing. This survey synthesizes module-level and system-level studies, analyzes their methodologies, oracles, and adequacy measures, and identifies challenges and research opportunities.

  • Problem

    ADS testing is difficult because planning lacks a correct-trajectory oracle, parameter combinations are effectively infinite, and module-specific features complicate effective test design.

  • Method

    The survey comprehensively reviews module-level and system-level ADS testing from the perspectives of test methodology, test oracle, and test adequacy.

  • Results

    The reviewed literature covers adversarial, search-based, fault-injection, metamorphic, differential, and coverage-based testing across ADS modules and system-level settings.

  • Takeaways & Limitations

    ADS testing research should further develop generalized metrics, realistic test cases, mixed-reality approaches, and solutions to simulator–real-world gaps.

  • Takeaways & Limitations

    Simulation-based testing remains common but differs from real-world testing; scenario-in-the-loop testing is still theoretical and not widely adopted.

Abstract

from arXiv · show

Automated Driving Systems (ADS) have made great achievements in recent years thanks to the efforts from both academia and industry. A typical ADS is composed of multiple modules, including sensing, perception, planning, and control, which brings together the latest advances in different domains. Despite these achievements, safety assurance of ADS is of great significance, since unsafe behavior of ADS can bring catastrophic consequences. Testing has been recognized as an important system validation approach that aims to expose unsafe system behavior; however, in the context of ADS, it is extremely challenging to devise effective testing techniques, due to the high complexity and multidisciplinarity of the systems. There has been great much literature that focuses on the testing of ADS, and a number of surveys have also emerged to summarize the technical advances. Most of the surveys focus on the system-level testing performed within software simulators, and they thereby ignore the distinct features of different modules. In this paper, we provide a comprehensive survey on the existing ADS testing literature, which takes into account both module-level and system-level testing. Specifically, we make the following contributions: (1) we survey the module-level testing techniques for ADS and highlight the technical differences affected by the features of different modules; (2) we also survey the system-level testing techniques, with focuses on the empirical studies that summarize the issues occurring in system development or deployment, the problems due to the collaborations between different modules, and the gap between ADS testing in simulators and the real world; (3) we identify the challenges and opportunities in ADS testing, which pave the path to the future research in this field.

1 INTRODUCTION

ADS testing is important for safety assurance but remains difficult because ADS are complex, multidisciplinary systems whose modules have distinct testing needs. This survey addresses those gaps by covering module-level and system-level testing, including simulation–real-world differences and future challenges.

  • ADS remain vulnerable to safety and security threats from complex environments and deliberate attacks, potentially causing catastrophic system failures.
  • Testing ADS is challenging because the systems combine complex, multidisciplinary components and span research across transportation, software engineering, artificial intelligence, and security.
  • Existing surveys mainly study ADS at the system level and rely on software simulators, leaving module-specific features, inter-module collaboration, and simulation–real-world gaps insufficiently covered.
  • The survey covers module-level techniques by distinguishing how different module characteristics affect testing.
  • At the system level, the survey examines empirical development and deployment issues, inter-module safety problems, and differences between simulation-based and real-world testing.
  • The survey identifies challenges and research opportunities while emphasizing the intrinsic differences among module-testing techniques and the comparison between simulation-based and real-world testing.

2 PRELIMINARIES

ADS are organized into automation levels and functional modules that process environmental data into vehicle-control commands. The section also introduces end-to-end designs, open-source systems, and the software and hardware context surrounding ADS testing.

  • Automation Levels: SAE J3016 classifies driving automation from L0, with no driving automation, to L5, with full driving automation.
  • Automation Levels: L0–L2 systems are ADAS requiring driver responsibility for the complete dynamic driving task, whereas L3–L5 systems perform all dynamic driving tasks without expected driver intervention.
  • ADS Architecture: A typical ADS contains sensing, perception, planning, and control modules that sequentially collect context, interpret the environment, generate trajectories, and issue vehicle commands.
  • ADS Architecture: The perception module supports localization, detection, and prediction using sensor data such as images and 3D point clouds.
  • ADS Architecture: The planning module predicts other agents’ trajectories and produces behavior and motion plans based on safety, efficiency, and comfort.
  • ADS Architecture: An end-to-end design combines perception, planning, and control in deep learning models that map sensor information directly to control commands.
  • Open-Source Systems and Tool Stacks: The survey introduces Apollo, Autoware, OpenPilot, and Pylot, alongside BeamNG as a simulation platform, with differences in automation level, communication stack, and supported functions.

3 PAPER COLLECTION METHODOLOGY AND RESULT

The survey collects and screens ADS-testing literature through database search, manual analysis, snowballing, and data extraction, then analyzes the resulting studies by venues, targets, and publication years.

  • Collection process: The five-step collection process comprises database search, abstract analysis, full-text analysis, backward and forward snowballing, and data extraction.The process and intermediate results are documented through supplementary materials and Zenodo.
  • Database search: The DBLP search retrieved 1,185 papers, and removing 144 duplicates left 1,041 papers for abstract analysis.The search combined ADS-related terms with testing and quality-assessment terms.
  • Screening criteria: The inclusion criteria covered ADS module or system testing methods, test oracles, and adequacy criteria, while excluding non-peer-reviewed, non-ADS, survey, and unrelated quality-assessment studies.Test oracles measure ADS or component misbehavior, whereas test adequacy judges whether a test suite is sufficient.
  • Screening results: Abstract screening produced agreement on 101 papers and disagreements on 54, with papers selected by either assessor retained in a tentative set of 155.The tentative set was subsequently examined through full-text analysis and conflict resolution.
  • Collection results: The final literature spans transportation, software engineering, security, and artificial-intelligence venues, with 38%, 25%, 10%, and 7% respectively.System-level testing accounts for 54% of targets and perception testing for 22%, while publications generally increased from 2015 to 2021.

4 LITERATURE OF EMPIRICAL STUDY ON ADS TESTING

Empirical ADS-testing studies analyze system architectures, repository bugs and issues, and public disengagement or crash reports to understand system behavior and real-world safety problems.

  • Scope and categories: Empirical studies analyze existing databases rather than executing ADS in simulated or real-world environments.The reviewed evidence comes from project repositories and public crash reports.
  • Scope and categories: The literature is organized into system studies, bug/issue studies, and public report studies.These categories respectively address system architecture, repository-reported problems, and public-road failures.
  • System studies: System studies of Apollo expose connections among modules and interactions among its 28 DNN models, while also identifying the role of system code.The observed DNN interactions include using one model’s output as another’s input and combining multiple outputs.
  • Bug and issue studies: Repository analyses identify concrete failure taxonomies: Apollo and Autoware bugs yield 13 root causes, 20 symptoms, and 18 related components.OpenPilot analysis collected 235 bugs from 1,293 pull requests and 694 issues, with car-interface bugs comprising 31.48%.
  • Public report studies: Public-report studies examine disengagement and crash records to identify causes, correlations, impacts, and failure types relevant to real-world ADS safety.The survey highlights both disengagement analyses and crash-report analyses, including recurring crash types and contributing factors.

5 LITERATURE OF TECHNIQUES ON MODULE-LEVEL ADS TESTING

Module-level ADS testing is reviewed across sensing, perception, planning, control, and end-to-end modules, using module-specific methodologies, oracles, and adequacy criteria.

  • Scope: The module-level survey covers sensing, perception, planning, control, and end-to-end modules.These modules are treated as distinct testing targets within the module-level literature.
  • Analytical perspectives: The review compares module-level work through test methodology, test oracle, and test adequacy.Methodology describes testing techniques, oracles judge behavioral correctness, and adequacy criteria assess test-suite sufficiency.

5.1 Sensing Module

Sensing-module testing addresses physical sensor performance under environmental conditions and deliberate attacks that interfere with sensor inputs or functionality.

  • Testing categories: Physical sensing tests assess sensor capabilities under environmental conditions, while deliberate attacks intentionally interfere with sensor operation.The review separates these methods into physical testing and deliberate attack.
  • Physical testing: LiDAR detection distance decreases by 20−40m under foggy and snowy conditions.The cited physical-testing studies focus mainly on LiDAR performance in harsh weather.
  • Deliberate attacks: Deliberate sensor attacks include jamming with noise or intense signals and spoofing with injected or modified data.Examples target LiDAR, cameras, ultrasonic sensors, GPS receivers, and mmWave radar.
  • Literature pattern: Existing sensing studies mainly test LiDAR in foggy or snowy weather, while deliberate attacks more often target other physical sensors.Reported attack methods include jamming and spoofing using devices such as lasers and LEDs.

5.2 Perception Module

Perception-module testing centers on adversarial attacks against DNNs, while test-oracle and adequacy techniques address correctness and coverage challenges specific to neural inference.

  • Testing Methodology: Adversarial attacks generate examples that trigger wrong perception in DNN models, using optimization, GAN-based, or training-data-poisoning methods.Attacks may be white-box or black-box and targeted or non-targeted.
  • Testing Methodology: Optimization-based attacks seek minimum perturbations that cause a DNN to output a wrong label, with applications to camera- and LiDAR-based perception.Physical attacks additionally account for conditions such as viewpoint and distance.
  • Testing Methodology: GAN-based attacks train generators to create perturbations while discriminators distinguish generated images from originals, including attacks spanning camera and LiDAR models.The loss balances generator and discriminator objectives through a weighting parameter.
  • Test Adequacy: Testing adequacy is difficult because DNN inference is non-interpretable, motivating structural coverage and combinatorial coverage criteria.Neuron coverage and its extensions measure activation-based coverage, while t-way coverage measures covered parameter combinations.
  • Test Oracle: Perception test oracles include ground-truth labeling, metamorphic testing, and formal specifications, with the latter two suited to runtime evaluation across continuous frames.These approaches address the challenge of judging whether perception outputs are correct.
  • Discussion: The literature widely adopts adversarial attacks, metamorphic testing, neuron coverage, and combinatorial testing, but neuron-coverage effectiveness remains contested.Physical robustness is also addressed because perturbations may fail in noisy environments.

5.3 Planning Module

Planning-module testing evaluates trajectories against safety and related properties, but independent testing is difficult because planning is tightly coupled to perception and control and lacks a definitive correct trajectory.

  • Test Methodology: Planning testing supplies traffic scenarios and checks whether generated trajectories satisfy properties such as safety, comfort, and low cost.The trajectory is a reference for control rather than the directly observable vehicle trajectory.
  • Test Methodology: Independent planning-module testing is challenging because its inputs come from perception and its outputs guide control.Consequently, relatively few studies test planning as a dedicated module, often assuming perfect perception and control.
  • Test Methodology: Search-based testing is the dominant planning-testing technique and generates scenarios intended to reveal faults or constrain the drivable area.Scenario descriptions range from functional and logical abstractions to concrete parameterized cases.
  • Test Oracle: Evaluating planned-trajectory correctness is difficult because no oracle represents the single correct trajectory.Avoidable collisions distinguish failures that can be prevented through a different ADS configuration from unavoidable collisions.
  • Test Adequacy: Weight coverage and route coverage measure diversity in planning decisions or routes, helping explore the large space of scenario and internal parameters.Weight coverage targets six parameters affecting decisions such as safety and comfort, while route coverage uses junction topology and route features.
  • Discussion: Existing planning metrics are effective for particular path-planning systems, but their generalization to other systems remains open.The survey identifies avoidable collision, weight coverage, and route coverage as representative metrics.

5.4 Control Module

Control-module testing addresses vulnerabilities in longitudinal and lateral control, using fault injection, sampling, and falsification while also tackling the oracle problem. The survey finds limited work on test adequacy because continuous dynamics make adequacy criteria difficult to define.

  • Control-module testing targets vulnerabilities in the ADS’s longitudinal and lateral control mechanisms.
  • Test methodology: Fault injection deliberately introduces faults, such as faulty GPS signals, to assess control-module fault tolerance.
  • Test methodology: Sampling generates scenarios with varying challenge levels by drawing parameters such as non-player-character and ego-vehicle speeds.
  • Test methodology: Falsification searches for critical scenarios using temporal-logic quantitative semantics as guidance.
  • Test oracle: An oracle-area framework judges control decisions by whether they drive the vehicle close to the closest safe position ahead.
  • Test adequacy: Few studies address test adequacy for control modules, because their continuous dynamics make adequacy criteria challenging to define.

5.5 End-to-End Module

End-to-end modules integrate perception, planning, and control in a single DNN, so testing adapts perception-oriented attacks and coverage methods to continuously changing driving-image sequences. The survey also identifies oracle strategies for judging control decisions whose correctness is difficult to evaluate directly.

  • An end-to-end module integrates perception, planning, and control in a single DNN-based model trained on realistic driving data.
  • Test methodology: End-to-end testing uses search-based testing, optimization-based adversarial attacks, and GAN-based attacks.
  • Test methodology: Search-based testing can target coverage and the seriousness of unsafe behavior, including steering-angle deviation from an expected angle.
  • Test methodology: Optimization-based and GAN-based attacks generate physical or visually realistic perturbations that can mislead end-to-end driving models across image sequences.
  • Test oracle: Metamorphic, differential, and model-based oracles address the difficulty of determining whether an output steering angle is correct.
  • Test adequacy: Unlike perception testing on single images, end-to-end testing commonly uses continuously changing camera frames in a driving context.

5.6 Answer to RQ1

The survey covers more than 80 papers and finds that testing techniques differ across ADS modules. It also identifies open challenges, including uncertainty about whether neuron coverage effectively tests perception and end-to-end modules.

  • The survey covers over 80 papers on testing different ADS modules.
  • Physical testing and deliberate sensor attacks can reveal abnormal sensing-module behavior.
  • Adversarial attack is the most widely used approach for perception and end-to-end modules, which rely mainly on DNNs.
  • Planning-module testing prominently uses search-based techniques, while control-module testing mainly uses fault injection, sampling, and falsification.
  • Neuron coverage may not be effective for testing perception and end-to-end modules, leaving an open challenge.

6 LITERATURE OF TECHNIQUES ON SYSTEM-LEVEL ADS TESTING

System-level ADS testing studies failures arising from module collaboration, using simulation-based methods such as search, adaptive stress testing, sampling, and adversarial attack. Search-based testing is most prevalent, while simulation cost motivates acceleration techniques and comparison with real-world testing.

  • System-level testing focuses on safety-threatening failures caused by collaborations between ADS modules.
  • Test methodology: The literature uses search-based testing, adaptive stress testing, sampling-based methods, and adversarial attack for system-level ADS testing.
  • Search-based testing: Search-based testing searches parameter spaces for values that achieve testing objectives and is widely adopted in ADS testing.
  • Search-based testing: Compositional search reduces counterexample-search effort by cooperatively constraining perception-input and whole-system input spaces.
  • Search-based testing: Lane-keeping studies show that road curvature and orientation affect system behavior, while generated virtual roads can induce lane deviation.
  • Search-based testing: Surrogate models accelerate expensive simulations by mapping scenario parameters to fitness functions and reducing the search space.
  • Sampling: Importance sampling accelerates rare-unsafe-scenario testing by aiming to detect more system failures with fewer simulations.
  • Discussion: Search-based testing is the most widely used system-level technique, while adaptive stress testing and sampling help address costly, repetitive simulation.

6.2 Mixed-Reality Testing

Mixed-reality testing combines simulation with real components to reduce the gap between software simulators and real-world ADS testing. The survey covers HiL, ViL, and SciL approaches, realism of test cases and simulators, and evidence about how simulation results transfer to reality.

  • Mixed-Reality Testing: Mixed-reality testing addresses the limited realism of software simulators by combining simulation-based testing with real-world testing.The surveyed schemes replace certain components of the testing setup with real components.
  • HiL and ViL: HiL introduces real ECU hardware into the testing loop, while ViL integrates a synchronized virtual scenario into a real vehicle.HiL platforms target high-fidelity vehicle dynamics and sensor outputs; ViL platforms can reconstruct scenarios from HD maps and traffic simulators.
  • SciL: SciL narrows the simulator–real-world gap by integrating additional real components, such as pedestrian dummies, into the testing loop.SciL was implemented in a platform based on SUMO and Unity and compared with ViL testing.
  • Simulation–Reality Gap: Simulation-based testing is evaluated along two dimensions: the realism of generated test cases and the realism of the simulators themselves.The survey notes that virtual scenarios may include traffic participants or failures that cannot occur in the real world.
  • Realistic Test Cases: Crash-report reconstruction can accurately recreate reported crashes, and the generated cases expose faults in open-source ADS.Methods use distance metrics, computer vision, and natural-language processing to reconstruct scenarios from public crash reports.
  • Simulation Validity: 62.5% of unsafe simulator scenarios translated to real collisions, while 93.3% of simulator-safe scenarios were also safe in the real world.Other experiments found an average prediction-error difference below 0.1 between simulator-generated and real-world datasets.
  • Survey Findings: The survey covers more than 90 system-level testing papers and more than 30 papers addressing the simulation–real-world gap.Open challenges include accelerating expensive, time-consuming system executions during testing.

7 STATISTICS AND ANALYSIS OF LITERATURE

The survey analyzes ADS testing literature through threat models, datasets, simulators, and programming languages. It finds diverse module-specific threats and resources, while noting continuing adoption, realism, accessibility, and efficiency challenges.

  • Statistical Analysis: The statistical analysis summarizes ADS threats, datasets, tool stacks, and programming languages based on the surveyed literature.It constructs a general ADS threat model and collects popular datasets, simulators, and programming languages.
  • Threat Model: Perception is the most investigated module, with 23 dedicated testing techniques, while planning has 8 collected techniques.Perception threats include adversarial examples, Trojan attacks, and attacks on cloud-provided HD maps.
  • Threat Model: Control is relatively hard to attack because it mainly uses mature techniques, but injected faults, emergency situations, and interface mismatches remain threats.Interface mismatches can occur in software–hardware transmission through the CAN bus.
  • Datasets: The survey collects 27 ADS testing datasets released from 2004 to 2022, spanning varied physical conditions and application scenarios.Examples include different times of day, weather conditions, traffic densities, urban streets, highways, and intersections.
  • Datasets: Dataset reuse is uneven: KITTI and Udacity support diverse tasks, whereas some datasets are limited to particular scenarios or use synthetic images that may lack realism.roundD and openDD target roundabouts, while SYNTHIA and GTA5 contain virtual-environment images.
  • Simulators: Simulator comparisons organize tools by accessibility, physical modeling, mixed-reality support, co-simulation, and research adoption.The survey notes that commercial simulators such as CarMaker and PreScan are not open-source, while BeamNG supports dynamic-model customization and realistic collision effects.
  • Programming Languages: The survey reviews 8 representative programming languages, whose features range from probabilistic sampling and bio-inspired operations to graphical scenario definition.Despite their capabilities, programming languages for ADS test generation have not yet been widely adopted in practice.

8 CHALLENGES AND OPPORTUNITIES

The survey identifies seven major ADS-testing challenges, spanning efficiency, realism, oracles, coverage, monitoring, fault analysis, and simulation–real-world gaps. It reports that some issues may improve in the short run, while others require sustained research.

  • Efficient test generation: Test-generation efficiency remains important because ADS executions are expensive, while surrogate-model preparation can itself be time-consuming.Suggested directions include test selection and prioritization alongside surrogate models and sampling-based methods.
  • Realism of test cases: Realistic test cases are necessary because scenarios that cannot occur in the real world are meaningless for safety testing.Examples include physically grounded perception tests and filtering avoidable collisions in planning tests.
  • Oracle problem for different modules: ADS testing still needs more accurate and reliable module-specific oracles, since existing automated labels, ad hoc criteria, and metamorphic relations have limitations.The survey highlights gaps in high-fidelity labels for perception and generalizable criteria for planning.
  • Effective coverage criteria: Coverage research should address temporal behavior and the control module, rather than focusing mainly on spatial coverage or momentary DNN activations.Persistent incorrect predictions over time are more relevant to system-level consequences such as collisions.
  • Online monitoring: Online monitoring can detect unsafe behavior during execution, but monitoring other modules and expressing real-world requirements remain open challenges.The survey points to richer specification languages as a future direction.
  • Fault analysis of system failure: Fault analysis must distinguish module failures from failures caused by interactions between modules, yet current work emphasizes detection more than causal analysis.The survey calls for effective fault-analysis techniques and validation methods.
  • Simulators vs. real world: The simulation–real-world gap persists despite high-fidelity simulators, motivating mixed-reality methods and approaches that use simulation results to reduce real-world testing costs.HiL and ViL have advanced, whereas SciL remains largely theoretical and not widely adopted.
  • Overview: Seven major ADS-testing challenges are identified, with efficiency improvements considered nearer-term and simulation–real-world gaps requiring longer research.The survey distinguishes challenges that could improve in the short run from those requiring a long period of research.

9 CONCLUSION

The survey reviews ADS testing across module-level and system-level settings, including empirical, mixed-reality, and real-world studies. It emphasizes differences among module-testing techniques, the simulation–real-world gap, and research directions identified from the literature.

  • Scope: The survey covers both module-level and system-level ADS testing, including empirical system studies, mixed-reality testing, and real-world testing.Module introductions organize the literature by test methodology, test oracle, and test adequacy.
  • Key findings: Its analysis emphasizes technical differences among modules and the gap between simulation-based and real-world testing.These issues are presented as central features of the surveyed testing landscape.
  • Future directions: The survey’s discussion of challenges and opportunities identifies future research directions intended to support ADS safety assurance.The authors express the goal of motivating further work toward more reliable ADS adoption.
Loading 2206.05961v2…