PaperlayerView exact revision on arXiv

Source-linked AI summary

In the Driver's Seat: A Multi-Company Study on the Reality of Autonomous Driving System Testing

Qunying Song, Yuan Gao, Johannes Betz, Dietmar Pfahl, Mohammad Reza Mousavi, Federica Sarro

arXiv:2607.15820v1cs.SE

TL;DR

ADS testing lacks established standards amid challenges in scenario realism, coverage, and acceptance criteria; this study interviews industry experts across nine companies and finds scenario-based and X-in-the-loop testing dominate current practice, informing a closed-loop testing framework.

  • Problem

    ADS testing faces unresolved challenges in simulation fidelity, scenario realism, and comprehensive coverage, complicating reliable evaluation across real-world conditions.

  • Method

    The study analyzes interviews with nine ADS experts from nine companies across six countries to characterize industrial testing practices and challenges.

  • Results

    Current ADS testing primarily centers on scenario-based approaches and X-in-the-loop activities, while participants report concerns about scenario realism, acceptance criteria, and other unresolved issues.

  • Takeaways & Limitations

    The study provides an industry-grounded overview and an evidence-centered closed-loop framework offering actionable guidance for ADS testing.

  • Takeaways & Limitations

    The findings are based on interviews with participants from nine companies, with later interviews showing saturation but limited generalizability beyond the sampled organizations.

Abstract

from arXiv · show

Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications. This creates growing demands for effective testing to ensure system functionality and safety. However, ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria. To better understand current ADS testing practices and challenges, we conducted an interview study with experts working on ADS development and testing in nine companies from six different countries. Through thematic analysis, we synthesized industrial testing practices, challenges, potential solutions, future trends, and proposed an evidence-centered closed-loop testing framework for ADS testing. Our findings show that current practices primarily focus on scenario-based and X-in-the-loop testing approaches, supported by diverse tools, metrics, benchmarks, and testing strategies. The participants highlighted major challenges related to scenario realism, scenario coverage, simulation fidelity, and acceptance criteria, while also discussing potential solutions such as the use of AI, world models, and end-to-end approaches. Furthermore, participants envisioned future ADS testing to become more automated, data-driven, and transparent across the industry. Overall, this study provides a comprehensive industry-grounded overview of ADS testing, proposes an evidence-centered closed-loop testing framework to provide actionable guidance for ADS testing, and outlines important directions for future research and practice.

1 Introduction

ADS deployment is advancing faster than standardized testing processes, leaving scenario selection, performance indicators, and acceptance criteria insufficiently defined. An interview study with nine experts from nine companies across six countries examined industrial practices, challenges, solutions, and future directions.

  • ADS testing must co-evolve with advancing deployment to validate system functionality and safety, but remains non-trivial across approaches, environments, tools, and evaluation criteria.
  • The study used interviews and thematic analysis to examine industrial testing practices, challenges and potential solutions, and future outlooks.The researchers interviewed nine experts with diverse roles and experiences from nine ADS companies across six countries.
  • Reported industrial practices primarily center on scenario-based approaches and X-in-the-loop testing activities, alongside varied strategies, pipelines, metrics, benchmarks, acceptance criteria, and tools.

2 Related Work

Prior ADS-testing research has examined challenges, scenario-based practices, critical scenarios, and available testing resources using interviews and literature reviews. Recent work also addresses generative AI, while this study adopts a broader scope than these approaches.

  • Related work: Beringhoff et al. identified 31 ADS-testing challenges through expert interviews, while Song et al. studied scenario-based testing practices and challenges through practitioner interviews.These studies use interview-based methods to investigate industrial testing concerns.
  • Related work: Literature reviews have examined ADS testing from different scopes, including critical scenarios and other focused aspects.Zhang et al. and Ding et al. are cited as examples of reviews focused on critical scenarios.
  • Related work: Other studies have aggregated ADS-testing tools, datasets, simulators, and platforms, while recent work has investigated generative AI for ADS testing.The literature varies in methodology, relying mainly on academic literature or public sources; the present study adopts a broader scope.

3 Research Method

The study used semi-structured interviews with ADS testing experts across multiple companies and countries, recruited through several sampling strategies. Interview data were participant-validated, thematically analyzed, and supported by explicit validity mitigations.

  • Study design and participants: Nine experts from nine companies across six countries participated in interviews exploring contemporary ADS testing practices.The researchers invited 21 companies across nine countries; participating companies were predominantly large motor-vehicle manufacturers, alongside one small and one medium-sized company.
  • Interview design: The researchers used semi-structured interviews guided by predefined research-question-centered questions while allowing questions to be adapted during interviews.This design supported flexible discussion of participants’ responses and reactions.
  • Participant recruitment: Participants were recruited through convenience, purposive, snowball, and social sampling using professional networks, public contact channels, and social media.These approaches included reaching out to existing contacts, requesting recommendations, searching LinkedIn and company websites, and posting open calls on LinkedIn and X.
  • Interview procedure: Interviews lasted between 45 and 75 minutes, with an average duration of approximately 60 minutes, and were conducted through Microsoft Teams with recording permission.Researchers guided discussions using nine interview questions and expanded topics based on participant responses.
  • Data analysis and validity: Transcripts were reviewed by participants, checked against recordings, clarified when necessary, and synthesized through iterative thematic coding into a unified hierarchical model.Relevant segments were coded, similar codes grouped into themes, and related themes organized into higher-level themes following qualitative thematic-analysis guidelines.

4 ADS

Participants worked on diverse ADS spanning parking, urban, highway, and full-stack applications, with automation levels from SAE Level 2 to Level 5. Most reported modular architectures, while some were exploring end-to-end or hybrid designs.

  • Participants reported autonomous parking, urban-driving, highway-driving, and full-stack ADS operating across urban, highway, and rural roads.Autonomous parking systems were frequently discussed and provided parking without human intervention.
  • Several participants worked on Level 2 ADAS providing driver assistance while human drivers remained responsible for driving.Reported functions included lane keeping, lane changing, adaptive cruise control, blind spot detection, forward collision warning, and intelligent speed assistance.
  • The systems spanned SAE Level 2 to Level 5, including Level 4 highway and parking systems and a Level 5 system primarily supporting urban driving.One Level 4 parking system was officially classified as Level 2 ADAS, while participants associated higher automation levels with greater liability and stricter regulatory requirements.
  • Most participants who described architectures used modular designs dividing perception, planning, and control into separate components.P1 and P4 were gradually moving toward and experimenting with end-to-end deep-neural-network architectures.
  • Modular architectures were viewed as more deterministic and easier to develop in fixed environments, whereas end-to-end architectures reflected broader industry adoption and promising performance.P8 used a hybrid architecture, with each approach serving as a fallback for the other in uncertain or difficult situations to reduce failures.

5 Testing Practices

This section characterizes participant practices for testing autonomous driving systems from process and approach perspectives. It covers testing pipelines, activities, transitions, satisfaction criteria, metrics, tools, and benchmarks.

  • Section scope: The study reports ADS testing practices through both process and approach perspectives.The process perspective covers pipelines and activities, while the approaches perspective covers supporting evaluation and testing resources.
  • Process perspective: The process perspective examines testing pipelines, involved activities, each activity’s focus, transitions between activities, and satisfaction criteria.These elements describe how testing is organized and how progression between activities is assessed.
  • Approaches perspective: The approaches perspective reports the metrics, tools, and benchmarks used for ADS testing.These resources support the evaluation and execution of the reported testing practices.

5.1 Processes and Activities

ADS testing strategies vary from function-, requirement-, simulation-, development-, data-, and objective-driven approaches, while pipelines center on X-in-the-loop testing and culminate in on-road validation and certification. Progression and acceptance remain weakly standardized, relying on engineering experience, performance indicators, and data-saturation criteria.

  • Strategies: Testing strategies include function-driven planning tied to the target function and real-world conditions, requirement-driven testing across system levels, and objective-driven evidence generation.Objective-driven plans derive test cases from assurance cases and organize safety and quality evidence through Goal Structuring Notation (GSN).
  • Strategies: Shift-to-left, simulation-driven testing reduces physical testing through multiple purpose-specific simulators, while iterative development-driven testing evolves with vehicle-system maturation.Required simulation fidelity depends on functionality: perception and vision need photorealism, whereas decision-making and control may use lower-fidelity environments.
  • Pipeline: Testing is increasingly data-driven because traditional model-, software-, and hardware-in-the-loop methods poorly fit AI-driven ADS relying on implicit representations and large-scale sensor data.The pipeline centers on model-, software-, hardware-, and vehicle-in-the-loop testing, with possible vehicle integration and certification stages.
  • Activities: On-road testing supports continuous validation alongside software updates, with real-world feedback, shadow-mode testing, and large-scale data collection addressing simulation gaps.Certification testing is required for European-market delivery but may emphasize technical compliance over real-world user experience and can be costly when failures occur.
  • Transitions and acceptance: Transition boundaries and acceptance criteria remain insufficiently defined, so teams rely on engineering experience, collision avoidance, behavioral consistency, and target-mileage data.Some release processes advance only after observed behavior remains consistent with expectations over collected mileage, supported by automated data collection and analysis.

5.2 Approaches, Tools, and Metrics

Participants primarily test ADS with scenario-based approaches, supplemented by data-driven and regression testing, while using diverse metrics and relatively few benchmarks. Generative AI remains exploratory, and simulation fidelity—especially realistic camera simulation—continues to challenge evaluation.

  • Testing approaches: Scenario-based testing is the dominant approach, with scenario selection shaped by target functions, requirements, tools, standards, and effectively infinite possible scenarios.Participants use scenarios to evaluate ADS under diverse environmental and driving conditions, while regulatory, standard-based, and common driving scenarios remain necessary.
  • Testing approaches: Data-driven testing incorporates uncovered or post-deployment scenarios into data platforms for future testing and development, while regression testing retests failures and previously passed scenarios after updates.Failure scenarios may be uploaded daily, and rerunning passed scenarios helps detect unexpected issues introduced by new versions.
  • Metrics: Scenario coverage is the most important metric, complemented by collision, comfort, accuracy, success-rate, and fault-rate measures tailored to ADS functions and modules.Collision-related metrics are critical for safety, while comfort metrics can reveal overly aggressive behavior that causes customer takeovers or unsuccessful tests.
  • Benchmarks: Few participants explicitly discussed benchmarks, including human drivers and common certification-testing scenarios used as baseline performance references.The study reports limited disclosure and discussion of benchmark use across companies.
  • Tools and emerging approaches: Generative AI models are being explored for ADS testing but remain largely pre-production because their instability makes direct real-vehicle or on-road use risky.Simulation platforms are common, yet realistic camera simulation remains difficult because rendered environments do not fully match real sensor data.

6 Testing Challenges … 6.3 Lack of Public Benchmarks

ADS testing faces intertwined challenges involving sim-to-real gaps, uncertain scenario realism, incomplete coverage, resource constraints, unclear acceptance criteria, and a lack of public benchmarks. Participants proposed technical and organizational responses including improved physical simulation, broader representations, end-to-end systems, world models, transparency, regulation, leadership, and standards.

  • 6 Testing Challenges: Participants identified sim-to-real gaps, missing benchmarks, incomplete coverage, resource constraints, and unclear acceptance criteria as broad ADS testing challenges.These challenges and proposed response directions were summarized across the study’s interviews.
  • 6.1 Sim-to-Real Gaps: Sim-to-real gaps concern differences between simulation and reality, including simulation fidelity, element representation, and simulation-quality assessment.Several participants described these gaps as a major challenge for ADS testing.
  • 6.1 Sim-to-Real Gaps: Limited fidelity in simulated traffic environments and sensor effects makes early-stage testing difficult despite significant improvements in physical sensors.Participants questioned whether simulation can faithfully represent real-world scenarios, including road surfaces, buildings, glass, and trees.
  • 6.1.1 Simulation Fidelity: 3D Gaussian Splatting was explored to improve simulation quality, but physical realism still requires modeling materials, textures, sensor reflections, and environmental deformations.The proposed technique should improve more than visual appearance alone.
  • 6.1.1 Simulation Fidelity: Perception-planning interfaces can restrict simulation to represented outputs, excluding objects such as traffic cones, workers, and temporary barriers.End-to-end architectures and world models may reduce these representation limits, but end-to-end systems can reduce interpretability.
  • 6.1.2 Simulation Representation: Simulation-platform quality assessment remains challenging because it is unclear whether widely used platforms such as CARLA provide sufficient fidelity for particular ADS tests.Different platforms and technologies leave open research questions before their suitability can be determined.
  • 6.2 Uncertain Scenario Realism: Scenario-generation methods such as optimization and AI can produce unrealistic or unrepresentative scenarios because sufficient real critical data is lacking.Open questions remain about scientifically defining and evaluating realism and confidence at pixel, image, and point-cloud levels.
  • 6.3 Lack of Public Benchmarks: Public ADS benchmarks are scarce because companies rarely share performance results, testing data, sensor configurations, or evaluation methods, hindering cross-company comparison.P8 contrasted ADS with large language models, which have common public benchmarks even for closed-source systems; proposed responses included regulations, transparency, industry leadership, and standards.

6.4 Incomplete Scenario Coverage … 6.7 Resource Constraints

The study identifies incomplete scenario coverage, deployment gaps, data-transfer limitations, and resource constraints as interconnected obstacles to reliable and scalable ADS testing. Proposed remedies emphasize AI-generated scenarios, cloud-based continuous improvement, operational-data feedback, and iterative system updates, while remaining dependent on substantial infrastructure and usage.

  • 6.4 Incomplete Scenario Coverage: AI-supported scenario generation could use accumulated real-world data to produce situations beyond human-designed scenarios and human imagination.The proposed approach relies on training large AI models on continuously accumulated data and scenarios.
  • 6.5 Deployment Discrepancies: Simulation-developed models can diverge from vehicle deployment because latency, model size, inference time, and implementation environments differ.Models developed in frameworks such as PyTorch may require conversion for optimized C++ production environments.
  • 6.6 Data Transfer Challenge: Missing or incomplete vehicle logs make faults difficult to reproduce and analyze, leaving development teams with insufficient data for debugging and resolution.P6 described vehicle log retrieval and transfer as a particularly critical ADS-testing challenge.
  • 6.6 Data Transfer Challenge: Cloud-based, real-time data access could accelerate analysis, model training, and software updates, but many manufacturers still store testing data locally in vehicles.P6 presented Tesla as an example of this cloud-based continuous-improvement approach.
  • 6.7 Resource Constraints: Limited time, equipment, and testing resources constrain ADS testing, while costly hardware replacements or upgrades can make technically solvable issues impractical to fix later.P3 considered resource constraints the most critical issue; late-stage limitations may remain unresolved at product release.
  • 6.7 Resource Constraints: Operational driving data, continuous scenario updates, and OTA upgrades can iteratively expand scenario libraries and improve vehicles as they are driven.This mitigation requires massive driving mileage, a large active user base, and continuous system iteration.

6.8 Unstable Customer Systems … 6.13 Unclear Acceptance Criteria

ADS testing is hindered by unstable systems, unresolved safety argumentation for AI components, non-reproducible failures, incomplete scenario databases, complex perception testing, and unclear top-level acceptance criteria. Participants therefore emphasized challenges spanning system reliability, scenario coverage, simulation fidelity, safety justification, and defining whether a system is “safe enough.”

  • 6.8 Unstable Customer Systems: Unstable customer software, hardware faults, and integration issues can repeatedly interrupt testing and force engineers to wait for recovery or restart systems.These failures may emerge continuously during testing and substantially delay the testing process.
  • 6.9 Safety Argumentation Gaps: Safety argumentation remains inadequate for machine-learning and AI components in safety-critical ADS, including unresolved questions about scenarios, environments, and simulation fidelity.Participants also identified a lack of safety processes and methodologies specifically designed for AI-driven systems.
  • 6.10 Non-Reproducible Issues: Some critical ADS testing issues occur intermittently with very low reproduction frequency, making them difficult to reproduce under seemingly identical conditions.Resolving these issues can require substantial time and collaboration among testing teams, system owners, suppliers, and other stakeholders.
  • 6.11 Incomplete Scenario Database: Scenario databases should be continuously enriched with real-world road-testing data to build more complete libraries and address gaps between limited regulatory requirements and variable practical performance.Existing libraries may be regularly updated, with frequently occurring scenarios expected to meet higher performance targets than rare scenarios.
  • 6.12 Perception Testing Complexity: Perception testing is complex because natural environments contain numerous factors and corner cases, while perception errors propagate to downstream modules.Testing cases such as shadows misclassified as lane markings requires modeling visual effects and camera capture behavior in simulation.
  • 6.13 Unclear Acceptance Criteria: Unclear acceptance criteria was identified as the most urgent challenge because ADS testing needs a top-level criterion for deciding whether a system is “safe enough.”P1 emphasized Goal Structuring Notation for constructing, justifying, and visualizing safety arguments.

7 Outlook and Future Trends

Participants envisioned ADS testing becoming more efficient, automated, transparent, and data-driven, with wider adoption of AI, end-to-end approaches, and world models. They also identified fidelity, reliability, explainability, and safety-argumentation challenges that could constrain these developments.

  • Continuous real-world log-data feedback could make model-, software-, and hardware-in-the-loop testing workflows shorter and faster before deployment.P9 linked this evolution to software-defined vehicles.
  • AI is expected to support test-scenario creation, but adoption depends on realistic physics, sensor behavior, powertrain interactions, reliability, and explainability.Suggested applications include Agentic AI for critical scenarios and generative models for VLM testing under varied weather conditions.
  • Greater transparency and sharing may bring more open metrics, data, and benchmarks as ADS development shifts from demonstrations toward measurable real-world performance.Participants cited Nvidia’s open-sourcing of Alpamayo as an example of this direction.
  • End-to-end ecosystems are emerging to integrate ADS simulation, testing, modelling, and validation, with OEMs and autonomous-driving companies beginning to adopt them.Nvidia’s Alpamayo and related components were identified as examples intended to become common infrastructure.
  • World models may learn realistic driving environments and sensor interactions from data, enabling more integrated ADS simulation, testing, and training.Waymo and Wayve were identified as companies investing heavily in this direction.
  • Testing may become increasingly automated, with vehicles autonomously selecting tests, collecting data, identifying issues, and sending feedback while reducing human testers’ involvement.This vision integrates testing and data-collection tools directly into vehicles.

8 Evidence-centered Closed-loop ADS Testing Framework

The paper proposes a six-stage evidence-centered closed-loop framework that translates ADS testing practices, challenges, and future directions into actionable guidance. It links safety claims and evidence requirements to scenario portfolios, environment selection, evidence evaluation, safety arguments, and operational feedback.

  • Stage 1: Stage 1 starts with safety claims and testing intent, then specifies evidence requirements based on the target function, ODD, automation level, and system architecture.Evidence requirements include relevant scenarios, test environments, evaluation metrics, and acceptance criteria.
  • Stage 2: Stage 2 constructs and continuously enriches a metadata-linked scenario portfolio using standards, regulations, expert knowledge, naturalistic data, optimization, and AI-assisted generation.New scenarios can emerge from testing and operational feedback.
  • Stage 3: Stage 3 routes scenarios to simulation, proving grounds, or real-world and X-in-the-loop environments according to required evidence, realism, efficiency, cost, and safety.Test readiness requires correct platform or vehicle configuration, sensors and logging, data synchronization, access, and ground truth.
  • Stage 4: Stage 4 applies evidence gates to determine whether testing results sufficiently support safety claims and goals before advancing testing or release activities.The proposed gates include assessments of scenario quality and simulation fitness.
  • Stages 5–6: Stages 5–6 assemble evidence into safety arguments for progression decisions and feed operational feedback into scenario and regression testing for continuous lifecycle refinement.Feedback includes logs, safety-driver interventions, disengagements, non-reproducible failures, deployment discrepancies, and newly encountered scenarios.

9 Discussion

The study finds that ADS testing is centered on scenario-based, X-in-the-loop, and data-driven practices but remains constrained by realism, acceptance, and operational challenges. It proposes an evidence-centered closed-loop framework linking safety claims, scenarios, environments, evidence, and feedback while supporting more automated and transparent testing.

  • Current Testing Practices: Current ADS testing primarily combines scenario-based testing with X-in-the-loop activities that progress from simulation toward real-world validation.Testing decisions also depend jointly on target functions, requirements, development status, operational data, tools, and objectives.
  • Current Testing Practices: Participants also emphasized data-driven testing based on collected driving data, log replay, operational feedback, and tools for processing, visualization, scenario manipulation, and evaluation.These findings provide an industry perspective covering multiple aspects of ADS testing.
  • Challenges: Major unresolved challenges concern scenario realism, simulation’s sim-to-real gap, acceptance criteria, safety arguments, and the absence of publicly shared benchmarks.These issues hinder reliable results and objective evaluation or comparison of ADS performance across companies.
  • Challenges: Practitioners also reported deployment discrepancies, vehicle-log transfer difficulties, resource constraints, unstable customer systems, and non-reproducible issues during real-world testing.These operational problems identify additional areas for future improvement.
  • Framework and Future Outlook: The proposed six-stage evidence-centered closed-loop framework connects safety claims, testing intent, scenario portfolios, environments, evidence production, evaluation, and operational feedback.It offers structured guidance for systematic planning, execution, evaluation, and continuous improvement while accommodating increased automation, AI, data sharing, and world models.

10 Conclusion

The study uses interviews with experts from nine ADS companies to characterize current testing practices and challenges. It synthesizes these findings into an evidence-centered closed-loop framework offering actionable guidance for safer ADS development and deployment.

  • 10 Conclusion: Experts from nine companies helped characterize current ADS testing practices and related challenges.The study aimed to understand industry practices and facilitate responses to unresolved testing issues.
  • 10 Conclusion: The study translates industry insights into an evidence-centered closed-loop testing framework and outlines directions for future ADS testing.The framework is presented as actionable guidance grounded in the study’s industry perspective.
  • 10 Conclusion: Industry testing generally follows scenario-based and X-in-the-Loop processes.These approaches form the prevailing testing processes identified by the study.
  • 10 Conclusion: Scenario realism, acceptance criteria, and other unresolved issues continue to hinder progress toward safe and accelerated ADS deployment.The conclusion highlights continued efforts as necessary to address these concerns and support further deployment.
Loading 2607.15820v1…