Source-linked AI summary
An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software
Rick Salay, Rodrigo Queiroz, Krzysztof Czarnecki
TL;DR
ML introduces safety-assurance challenges for ISO 26262 because trained models can be non-transparent, error-prone, dependent on incomplete data, and unstable. The paper analyzes these impacts across the standard, identifies five affected areas, and proposes adaptations, while noting that incomplete training data violates key V-model assumptions.
Problem
ISO 26262 must address ML-based automotive software even though ML models rely on incomplete data and have properties that complicate safety assessment.
Method
The paper analyzes how ML affects hazard analysis and software-development phases across ISO 26262 and develops recommendations for standard changes and further research.
Results
The analysis identifies five impacted areas, while concluding that existing ISO 26262 failure-analysis recommendations apply because ML faults increase error rates and produce incorrect outputs.
Takeaways & Limitations
Accommodating ML in automotive safety certification requires adapting ISO 26262 to address ML-specific hazards, training-data assumptions, and usage levels.
Takeaways & Limitations
Incomplete ML training sets violate the V-model assumption that component functionality is fully specified and refinements are verifiable.
Abstract
from arXiv · showhide
Machine learning (ML) plays an ever-increasing role in advanced automotive functionality for driver assistance and autonomous operation; however, its adequacy from the perspective of safety certification remains controversial. In this paper, we analyze the impacts that the use of ML as an implementation approach has on ISO 26262 safety lifecycle and ask what could be done to address them. We then provide a set of recommendations on how to adapt the standard to accommodate ML.
I. INTRODUCTION
ML is increasingly used in ADAS and autonomous vehicles, but ISO 26262 was not designed for it. This paper analyzes the resulting impacts and recommends changes to the standard and further research.
- ISO 26262 supports systematic automotive safety practices, but its design creates tension with innovation involving ML.
- Prior research addresses ML safety, neural-network verification and validation, safety cases, and certification, but the current analysis is complementary.
- The paper identifies five problems created by ML in hazard analysis and software development, then recommends changes to ISO 26262 and additional research.
A. ISO 26262
ISO 26262 regulates road-vehicle functional safety and uses a V-model-based software-development process. Part 6 specifies software techniques and ASIL-dependent architectural error-handling recommendations.
- ISO 26262 regulates functional safety for road vehicles, while Part 6 addresses product development at the software level using the V model.
- ASIL classifies automotive risk and determines the rigor required for testing, documentation, and other safety activities, with ASIL D highest and ASIL A lowest.
- Part 6 specifies software-development compliance requirements and recommends architectural error-handling mechanisms according to ASIL.
B. Machine learning
The paper distinguishes programmed components from trained ML components and highlights four safety-relevant ML characteristics: non-transparency, error rates, incomplete training coverage, and instability.
- ML components are trained models using supervised, unsupervised, or reinforcement learning, unlike programmed components implemented with programming languages.
- Non-transparency makes it harder for assessors to determine whether an ML model operates as intended, especially as expressive power increases.
- ML models periodically produce errors, while estimated true error rates provide only statistical guarantees and may not match operational error rates.
- Incomplete or unrepresentative training sets, overfitting, and under-represented safety-critical cases create uncertainty about operational behavior.
- Local optimization and multiple optima can make retraining produce structurally different models, complicating debugging and reuse of safety assessments.
III. ANALYSIS OF ISO 26262
The analysis identifies five areas where ML changes how ISO 26262 should treat software components, focusing on hazard analysis and software development in Parts 3 and 6.
- Applying its impact criterion to all ten ISO 26262 parts, the paper identifies five impacted areas in concept-phase hazard analysis and software development.
A. Identifying hazards
ML introduces hazards beyond conventional malfunctioning behavior, including operator complacency and reinforcement-learning failures that exploit reward functions or the environment. The paper recommends broadening ISO 26262’s hazard definition and identification methods.
- A. Identifying hazards: ML can create hazards through operator complacency, such as reduced monitoring when drivers overestimate automated assistance.
- A. Identifying hazards: Reinforcement-learning faults can produce surprising hazards, including law-breaking behavior or exploitation of sensor vulnerabilities to game the reward function.
- A. Identifying hazards: ISO 26262 should broaden hazard definitions to include harmful human–vehicle interactions not caused by system malfunction.The paper recommends informing identification methods with ADAS behavioral-impact and human–robot-interaction research.
B. Faults and failure modes
ML introduces distinctive fault types, especially in network topology, learning algorithms, and training data, but its resulting failure mode remains an incorrect output for an input. The paper therefore recommends ML-aware fault-detection techniques while retaining existing failure-analysis guidance.
- B. Faults and failure modes: ML faults may be unique to network topology, learning algorithms, or training sets, creating opportunities for domain-independent fault-detection techniques.
- B. Faults and failure modes: ML faults increase deployed error rates, but the resulting failure mode—an incorrect output for an input—is not different from ordinary software failure.
- B. Faults and failure modes: Existing ISO 26262 failure-analysis recommendations therefore apply to ML components, despite their distinctive fault characteristics.
- B. Faults and failure modes: ISO 26262 should require fault-detection tools that account for ML-specific features, including bad training data and interactions among linked ML components.
C. The use of training sets
Training sets violate ISO 26262’s assumption that behavior is completely specified and verifiable, while some perception tasks may be inherently difficult to specify. The paper recommends partial specifications, coverage measures, improved training data, and fault tolerance.
- C. The use of training sets: Training sets are incomplete substitutes for specifications, so trained models cannot ensure that hazards are always mitigated beyond the represented examples.
- C. The use of training sets: Advanced perception functionality, such as recognizing pedestrians, may be inherently unspecifiable through rules alone and require examples.
- C. The use of training sets: ISO 26262 should distinguish functionality types: require conventional programming for fully specifiable functions, but permit ML with partial specifications for perception tasks.The paper suggests incorporating partial properties into training or checking them after training.
- C. The use of training sets: Training-set specifications and coverage metrics should be required, while high-ASIL systems should also use fault tolerance because better data alone may not reduce error sufficiently.Examples include redundant pedestrian recognizers and safety envelopes restricting possible behaviors.
D. Level of ML usage
End-to-end ML can eliminate stable, independently developed software units and undermine ISO 26262’s hierarchical-architecture assumptions. It may also require exponentially larger training sets, while opaque component-level models complicate modularity assessment; the paper therefore advises against encouraging end-to-end use.
- D. Level of ML usage: End-to-end ML can replace conventional architectural components with a model trained directly from sensor data to steering commands.
- D. Level of ML usage: Training instability means an end-to-end model’s functional structure can change arbitrarily after retraining, eliminating conventional independently developed units.
- D. Level of ML usage: In some cases, end-to-end approaches require exponentially larger training sets than programmed architectures, intensifying safety-critical data challenges.
- D. Level of ML usage: Opaque ML components can hamper assessment of complexity and application of ISO 26262 modularity principles even when ML is used below the end-to-end level.
- D. Level of ML usage: End-to-end ML therefore conflicts with ISO 26262 assumptions about stable hierarchical architectures and is not recommended by the paper.
E. Required software techniques
The assessment finds that many ISO 26262 unit-level techniques apply to ML components, but about 40% do not apply and highly recommended techniques are concentrated at lower ASILs. The paper recommends specifying techniques by intent and maturity rather than implementation-specific details.
- E. Required software techniques: Black-box techniques are generally directly applicable, while some white-box techniques, such as fault injection, can also apply after adaptation.Fault injection can target neural networks by breaking links or randomly changing weights.
- E. Required software techniques: About 40% of unit-level techniques do not apply to ML components, regardless of recommendation level.The assessment classifies techniques as directly applicable, adaptable, or not applicable.
- E. Required software techniques: A significant portion of the standard remains directly applicable to ML components, with emphasis on highly recommended techniques.Chart (a) aggregates applicability across recommendation groups and ASIL levels.
- E. Required software techniques: Most highly recommended techniques apply at lower ASIL values, making them less relevant from a safety-critical perspective.The high standard deviation in the highly recommended group motivates the per-ASIL view in Chart (b).
- E. Required software techniques: The paper recommends expressing requirements through technique intent and maturity to reduce imperative-language bias without compromising safety.The example replaces a specific variable-naming rule with accepted clarity-increasing techniques.
IV. SUMMARY AND CONCLUSION
The paper identifies five ways ML affects ISO 26262, spanning hazard analysis and software development, and recommends adapting the standard rather than excluding ML. Its recommendations address ML-specific hazards, lifecycle faults, training-data limits, usage level, and technique requirements.
- IV. SUMMARY AND CONCLUSION: The paper recommends five ISO 26262 adaptations covering ML-related hazards, lifecycle faults, training sets, usage level, and required software techniques.The identified impacts occur in hazard analysis and the software development phase.
- IV. SUMMARY AND CONCLUSION: ML can create hazards through complex human interactions with advanced functionality, even without component malfunction; the standard should expand its hazard definition.The recommendation places these situations within system design mitigation.
- IV. SUMMARY AND CONCLUSION: ML-specific development lifecycles produce distinct fault types, so ISO 26262 should address that lifecycle and require customized fault-detection techniques.The recommendation targets tools and techniques tailored to ML development stages.
- IV. SUMMARY AND CONCLUSION: Incomplete training data and potentially unspecifiable perception functionality mean ML components may have known error rates and periodic failures.The paper recommends different safety requirements according to whether functionality is specifiable.
- IV. SUMMARY AND CONCLUSION: End-to-end ML challenges hierarchical component decomposition, so the paper recommends limiting ML use to the component level.The paper links this restriction to the limited applicability of system-safety techniques.
- IV. SUMMARY AND CONCLUSION: Because many mandated techniques assume imperative programming, requirements should instead specify technique intent and maturity.This change is intended to remove the language-specific bias while retaining safety requirements.