Source-linked AI summary
Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry
Markus Borg, Cristofer Englund, Krzysztof Wnuk, Boris Duran, Christoffer Levandowski, Shenjian Gao, Yanwen Tan, Henrik Kaijser, Henrik Lönn, Jonas Törnqvist
TL;DR
Safety assurance for automotive DNNs is difficult because their data-driven nature challenges ISO 26262, and certification for public-road use remains unclear. The paper reviews safety-critical ML verification and validation and combines literature findings with automotive-industry input. It reports differing academic and industry emphases on challenges and recommends further work on safety cages and simulated test cases.
Problem
DNNs’ data-driven behavior challenges ISO 26262 safety assurance, and how to certify safety-critical DNN systems for public roads remains unclear.
Source: §1 · p. 2, §6 · p. 22Method
The paper reviews safety-critical ML verification and validation, elicits automotive challenges through workshops, and validates literature findings with an industrial survey.
Source: §1 · p. 2Results
Industry stresses robustness, whereas academia most often addresses state-space explosion and the lack of ML transparency.
Source: §6 · p. 22Takeaways & Limitations
The authors recommend exploring safety-cage architectures and simulated test cases as promising directions for safety-critical automotive DNN systems.
Source: §4.4.1 · p. 17, §6 · p. 23The discussion mostly concerns fixed DNNs trained before deployment; effects of continued learning remain future work.
Source: §6 · p. 23
Abstract
from arXiv · showhide
Deep Neural Networks (DNN) will emerge as a cornerstone in automotive software engineering. However, developing systems with DNNs introduces novel challenges for safety assessments. This paper reviews the state-of-the-art in verification and validation of safety-critical systems that rely on machine learning. Furthermore, we report from a workshop series on DNNs for perception with automotive experts in Sweden, confirming that ISO 26262 largely contravenes the nature of DNNs. We recommend aerospace-to-automotive knowledge transfer and systems-based safety approaches, e.g., safety cage architectures and simulated system test cases.
1. Introduction
DNNs support automotive environmental perception, but their data-driven, black-box behavior challenges conventional ISO 26262 assurance. The paper reviews safety-critical ML verification and validation and gathers automotive experts’ challenges through workshops, validating literature findings with an industrial survey.
1. Introduction
- DNN-based environmental perception supports autonomous-driving functions, but successful perception is a prerequisite for those capabilities.The paper identifies front-facing cameras as the most common sensor approach and notes DNNs’ usefulness in classifying camera data.Source: §1 · p. 2
- DNNs learn behavior from historical data, making their development a major shift from conventional systems under ISO 26262.Their black-box nature makes it currently impossible to analyze when they may fail to generalize, while source-code reviews and exhaustive coverage testing do not apply.Source: §1 · p. 2
- The paper combines a review of safety-critical ML verification and validation with automotive-expert workshops and an industrial survey.The survey validates findings from the literature review.Source: §1 · p. 2
2. Background
The background introduces automotive safety engineering and ISO 26262, then describes DNNs and their use in perception. It highlights why existing software-level safety practices may need special treatment for ML-based software.
2.1. Safety Engineering in the Automotive Domain: ISO 26262
- Safety engineering integrates hazard analysis, risk management, mitigation, and verification throughout system development rather than adding safety at the end.ISO 26262 is the automotive derivative of IEC 61508 and covers development, production, and maintenance of safety-related systems.Source: §2.1 · p. 3
- ISO 26262 assigns an ASIL from A to D using exposure, driver controllability, and outcome severity, with more rigorous practices for higher levels.The standard’s software-level inspection and conventional code-coverage requirements cannot directly apply to ML-based software units.Source: §2.1 · p. 3
2.2. Deep Learning for Perception: Approaches and Challenges
- DNNs learn features from input data, reducing the need for manual feature engineering in perception tasks such as computer vision.Convolutional neural networks are described as the state of the art in computer vision; GANs can generate synthetic samples or adversarial examples, while transfer learning adapts learned knowledge across datasets.Source: §2.2 · p. 4
3. Research method
The SMILE project combines literature review and practitioner input to address the state of the art and automotive engineering challenges in safety-critical ML. Its evidence collection spans related work, workshops, snowballed literature, and a questionnaire survey.
3. Research method
- The research combines academic and practitioner-led project activities to address the state of the art and automotive DNN engineering challenges.The project collected evidence across related work, workshops, snowballed literature, and survey responses.Source: §3 · pp. 4–5, Figure 2 · p. 6
3.1. The systematic review
- The review broadened its scope beyond automotive DNNs to ML in autonomous cyber-physical systems addressing verification, validation, or safety analysis.Snowballing was chosen as an alternative to database searches because terminology in emerging research areas can be diverse.Source: §3.1.1 · p. 6, §3.1 · p. 6
3.1.1. Paper selection
- Researchers started with 14 papers and iteratively used backward and forward snowballing, with separate researchers collecting and validating candidates.The authors note that citation patterns may cause relevant papers to be missed from the start set.Source: §3.1.1 · p. 7
3.1.2. Data extraction and analysis
- The review extracted publication metadata, assessed rigor and relevance, and used thematic analysis to address its first research question.Because existing classification schemes did not span the selected papers, the researchers inductively developed categories for challenges and solution proposals.Source: §3.1.2 · p. 7
3.2. The questionnaire-based survey
- A web-based questionnaire gathered practitioner ratings of challenge importance and solution promise, plus open-ended comments.It used convenience sampling to reach safety-critical engineering and managerial practitioners beyond the SMILE project; responses were summarized and open answers coded and validated by co-authors.Source: §3.2 · pp. 7, 9
4. Results and Discussion
The section organizes findings around four evidence sources, distinguishing practice-oriented evidence from sources aimed at academic rigor.
4. Results and Discussion
- The findings are organized as related work, workshop findings, snowballed literature, and survey responses.The first two focus on industrial relevance; the latter two aim at academic rigor.Source: §4 · p. 9
4.1. Related work
The related work spans aerospace, automotive, and general ML safety research, with the aerospace study [P1] especially close in aims and method. It covers challenges in assuring adaptive systems and proposed approaches including safety cages, formal methods, and monitoring.
4.1.1. The snowballing start set
- [P1] identifies state-space explosion, unpredictable environments, emergent behavior, and human-machine communication as enduring challenges for autonomous systems.Its proposed research goals include lifecycle safety evidence, formal methods, verifiable requirements, run-time assurance, and compositional safety cases.Source: §4.1.1 · p. 10
4.1.2. Non-snowballed related work
- Other reviewed work reports testing challenges involving simulation, safety, sensors, test-case scale, and driver hand-off, alongside concerns about model insufficiencies and growing system complexity.Formally verified AI also requires environment models, formal specifications, ML component models, systematic data generation, and scalable verification.Source: §4.1.2 · pp. 10–11
- Related proposals include rejecting uncertain classifications, visualizing DNN decisions, and assessing robustness through individual neurons as units of failure.A reject option must also account for decisions based on too few training examples, not only proximity to a decision boundary.Source: §4.1.2 · pp. 11–12
- Safety cages and supervisors monitor autonomous functions and can invoke safe actions, while ML safety frameworks also discuss run-time monitoring and procedural safeguards.A safety cage may support ASIL decomposition as a functionally redundant system.Source: §4.1.2 · pp. 11–12
- Aerospace certification work highlights difficulties specifying complete, verifiable requirements and documenting or transparently explaining adaptive AI designs.The reviewed report also notes uncertainty about interpreting source-code coverage testing for AI algorithms.Source: §4.1.2 · p. 12
4.2. The Workshop Series
Six industry workshops explored DNN robustness, integration with conventional software, and verification and validation. Discussions emphasized representative training data, system-level safeguards, and new V&V methods, including simulation.
4.2.1. Robustness of DNN Components
- Robustness requires avoiding false positives and false negatives under low-quality inputs or changing conditions, while DNN architectures are often problem-specific and developed through trial and error.The workshops stressed that training data must adequately represent the task and operating environment, including sensor quality, position, orientation, and geography.Source: §4.2.1 · p. 13
- Synthetic data can complement training data, but the workshops identified open questions about the best training mix and measuring its match to the planned operating environment.One proposed sequence combines simulator-generated data, public datasets, and data from the vehicle's target region.Source: §4.2.1 · p. 13
4.2.2. Complementing DNNs with Conventional Components
- Because safety is a systems issue, a proposed safety cage monitors DNN inputs and redirects execution to a non-ML safe track when inputs appear outside the training region.The workshops also noted that domain-specific malfunctions may require different safety cages for different systems.Source: §4.2.2 · pp. 13–14
4.2.3. V&V Approaches for Systems with DNN Components
- Workshop participants found ISO 26262 applicability to DNNs unclear and called for formal or semiformal requirements, lifecycle tools, scalable test generation, and simulation strategies.Simulation was proposed for normal and rare dangerous traffic situations and for assessing sensitivity to adversarial examples.Source: §4.2.3 · pp. 14–15
4.3. The systematic snowballing
The review's snowballing identified 64 papers, spanning two publication peaks and varied challenge and solution categories. The mapping highlights unaddressed requirements-specification and systems-engineering challenges, while aerospace experience offers relevant V&V precedents.
4.3. The systematic snowballing
- 64 papers were identified; 29 appeared in 2002–2007, dominated by adaptive flight controllers, and 25 appeared in 2013–2016, predominantly addressing automotive applications.The authors suggest automotive ML developers could learn from aerospace V&V, while noting that 19 papers addressed ML systems without a specific domain.Source: §4.3 · p. 15
- State-space explosion and robustness were the most frequent challenges, while formal methods, control theory, and probabilistic approaches were common solution categories.Discussion of state-space explosion was concentrated in recent years; research specifically addressing ML test-case design appeared after 2012.Source: §4.3 · p. 15, Table 2 · p. 16
- The mapping shows no proposed solutions for requirements-specification or systems-engineering challenges; transparency was addressed most, followed by state-space explosion.Connections in the figure represent instances linking solution-proposal categories to challenge categories.Source: §4.3 · p. 15, Figure 3 · p. 15
- Earlier aerospace ANN work argued that V&V must evolve, covering iterative lifecycles, design and training-data choices, and testing against both architecture and knowledge requirements.A later book's seven solution groupings include architecture analysis, training-data assessment, opacity, and online monitoring; the authors describe the schemes as complementary.Source: §4.3 · pp. 15–16
4.4. The survey
The survey found substantial concern about the difficulty of V&V for ML-based systems, with robustness and simulated test cases standing out. Respondents’ ratings and open-ended comments also reveal connections among challenges and differing views about which issues matter most.
4.4.1. Closed questions
- 49 practitioners responded; 20 primarily worked in automotive and 14 in aerospace, alongside respondents from other domains and roles.Most respondents worked primarily in Europe (38 of 49, 77.6%).Source: §4.4.1 · p. 17
- 29 of 49 respondents (59.2%) said ML-based safety-critical applications will be important to their organizations, while 29 (59.2%) said their organizations find V&V particularly difficult; only 10 (20.4%) considered their organizations well-prepared.20 respondents strongly agreed that V&V of ML-based features is particularly difficult.Source: §4.4.1 · p. 17
- Robustness was rated extremely important by 29 of 49 respondents (59.2%), and simulated test cases were the most promising proposal, rated extremely promising by 18 (36.7%) and moderately promising by 12 (24.5%).Probabilistic methods were rated least promising, followed by process guidelines.Source: §4.4.1 · p. 17
4.4.2. Correlation analysis
- ML proficiency correlated with perceived importance (ρ = 0.53), transparency challenges (ρ = 0.61), and state-space explosion (ρ = 0.54); perceived organizational preparedness correlated with systems-engineering challenges (ρ = 0.57).The strongest reported challenge correlation was between state-space explosion and transparency (ρ = 0.91); test specification and formal methods were the only strongly correlated challenge–solution pair (ρ = 0.71).Source: §4.4.2 · pp. 17, 19
4.4.3. Open questions
- 16 of 49 respondents (32.7%) answered the open question, offering contrasting views on the findings and citing robustness, requirements specification, sensor-data uncertainty, regulatory knowledge gaps, and non-determinism.Some respondents connected state-space explosion with rigid testing and limited transparency, while another suggested safety-cage architectures as guard rails.Source: §4.4.3 · p. 19
- Seven respondents added concluding thoughts, including support for aerospace-to-automotive knowledge transfer and the need to specify actions for hazardous situations when a classifier is wrong.Two respondents also emphasized conservatism and a paradigm shift associated with ML’s “end of predictability.”Source: §4.4.3 · p. 19
5. Revisiting the RQs
The review finds growing research interest in safety for DNNs, but no agreed development approach and a substantial mismatch with automotive safety certification. Its literature and practitioner evidence point to robustness, transparency, and state-space challenges, while suggesting both cross-domain learning and system-level safety approaches.
5.1. RQ1: State-of-the-art in V&V of safety-critical ML
- Research on DNN safety is growing, but there is no agreement on how to develop safety-critical DNNs, and much contemporary work addresses their opacity to support analyzability and interpretability.The paper describes DNNs as a paradigm shift for functional-safety certification and argues that the gap with ISO 26262 is unlikely to be resolved through incremental updates alone (Salay et al.78).Source: §5.1 · p. 20
- Aerospace research on neural networks for adaptive flight controllers preceded automotive DNN research by roughly a decade, motivating the authors’ hypothesis that knowledge transfer between domains is possible.The literature review found aerospace research peaking in 2002–2007 and automotive research dominating from 2013 onward.Source: §5.1 · p. 20
- The literature most often addresses state-space explosion and robustness, while practitioners find simulated test cases more promising than formal, control-theoretic, or probabilistic proposals.The paper reports shared views on important challenges but less agreement on the best way forward.Source: §5.1 · p. 20
5.2. RQ2: Main challenges for safe automotive DNNs
- Practitioners across organizations do not know how to approach regulators and certification agencies with DNN-based systems, and the authors judge incremental science unlikely to bridge the current gap soon.Existing safety cases for impressive applications and market features rely on a human driver taking control in critical situations.Source: §5.2 · pp. 20–21
- Workshops and survey responses emphasized robustness, but how to engineer robust components for the complexity of public traffic remains open; the authors call for empirical study of what automotive industry means by robustness.The paper also notes that practitioners confirmed transparency and state-space explosion as challenges highlighted in academic work.Source: §5.2 · p. 21
- Workshop participants viewed continuous DNN-input monitoring with a safety cage and simulated test cases as promising approaches for safety-critical DNNs.The safety cage can redirect execution to a safe track without DNN involvement when uncertainty grows too large (Adler et al.1).Source: §5.2 · p. 21
6. Conclusion and future work
The paper concludes that DNN-based automotive safety creates a mismatch with ISO 26262 and that certifying such systems for public roads remains unclear. Its findings point to aerospace knowledge transfer and systems-based V&V, while future work must address limitations of the current discussion.
Findings
- Most ML research showcases applications while V&V development lags; road certification remains unclear, and industry and academia emphasize different challenges.Industry stresses robustness, while academia more often addresses state-space explosion and limited ML transparency; the elicited challenges corroborate NASA and USAF work.Source: §6 · p. 22
Implications for research and practice
- The gap between ML development and ISO 26262 calls for novel standards rather than incremental updates.The authors state that some ISO 26262 process requirements contravene ML-based development, particularly for V&V.Source: §6 · pp. 21–23
- Aerospace-to-automotive transfer appears promising, including adapting configuration management, architecture specifications, requirements, testing, and lifecycle alignment for DNNs.The paper recommends aligning DNNs’ highly iterative lifecycle with the traditional automotive V-model.Source: §6 · pp. 22–23
- Empirical studies are needed to clarify what robustness means for DNN-based autonomous vehicles.The authors also note that industry respondents stressed robustness.Source: §6 · p. 22
- The authors favor systems-based safety approaches, including safety cages that redirect execution to a non-ML safe track and V&V that makes heavy use of simulation.The proposed safety cage monitors operational input and redirects execution when uncertainty grows too large.Source: §6 · pp. 22–23
Future work
- The discussion mostly concerns fixed DNN systems trained before deployment; future work includes transfer learning, dynamic DNNs, ethical and legal issues, and the forthcoming ISO/PAS 21448.The authors say they were not aware of ISO/PAS 21448’s contents at the time of writing.Source: §6 · p. 23