Source-linked AI summary
Open Problems in Engineering Machine Learning Systems and the Quality Model
Hiroshi Kuwajima, Hirotoshi Yasuoka, Toshihiro Nakae
TL;DR
Safety-critical machine-learning engineering lacks widely accepted frameworks for organizing open problems and demonstrating system safety and security. The paper classifies problems across requirements, design, and verification for models and systems, then identifies requirements specification and robustness as the most impactful gaps in SQuARE. It concludes that much of the engineering work and quality-model development remains future work.
Problem
Method
The authors review model- and system-level problems across requirements, design, and verification and analyze machine-learning characteristics against SQuARE.
Source: §6 · p. 13Results
The SQuARE analysis found that missing requirements specifications and robustness gaps have the greatest impact on the system quality model.
Source: §6 · p. 13Takeaways & Limitations
The paper presents a standard quality model for machine-learning systems as one approach toward broadly accepted engineering, while identifying further work on engineering technologies and quality characteristics.
Source: §6 · p. 13The authors state that much of the proposed engineering work remains future work, including developing requirements techniques and discussing quality characteristics beyond SQuARE.
Source: §6 · p. 13
Abstract
from arXiv · showhide
Fatal accidents are a major issue hindering the wide acceptance of safety-critical systems that use machine learning and deep learning models, such as automated driving vehicles. To use machine learning in a safety-critical system, it is necessary to demonstrate the safety and security of the system to society through the engineering process. However, there have been no such total concepts or frameworks established for these systems that have been widely accepted, and needs or open problems are not organized in a way researchers can select a theme and work on. The key to using a machine learning model in a deductively engineered system, developed in a rigorous development lifecycle consisting of requirement, design, and verification, cf. V-Model, is decomposing the data-driven training of machine-learning models into requirement, design, and verification, especially for machine learning models used in safety-critical systems. In this study, we identify, classify, and explore the open problems in engineering (safety-critical) machine learning systems, i.e., requirement, design, and verification of machine learning models and systems, as well as related works and research directions, using automated driving vehicles as an example. We also discuss the introduction of machine-learning models into a conventional system quality model such as SQuARE to study the quality model for machine learning systems.
1 Introduction
Safety-critical machine-learning systems need engineering frameworks and quality measures to demonstrate safety and security and support social acceptance. This study reviews open engineering problems and examines how SQuARE might be adapted for machine-learning systems.
1 Introduction
- Common engineering frameworks and quality measures are needed to manage risks and help demonstrate system safety and security.The passage connects quality assurance based on such frameworks with social receptivity.Source: §1 · p. 1
- Experimental automated-vehicle accidents have highlighted the need for quality assurance based on engineering frameworks.The text cites accidents during the use of experimental vehicles as evidence of this need.Source: §1 · p. 2
- The paper reviews open engineering problems in safety-critical machine-learning systems and their related work and research directions.It also examines system quality for machine-learning systems as a basis for future requirements and evaluations.Source: §1 · p. 2
2 Background
The paper uses in-vehicle automated driving systems as an example and examines safety-critical machine-learning engineering at both system and model levels. It organizes the open problems around requirements, design, and verification.
2 Background
- The paper focuses on in-vehicle automated-driving systems, whose perception, planning, and control subsystems may include machine-learning models.The described vehicle system can also cooperate with cloud and roadside systems.Source: §2 · p. 2
- The analysis considers open engineering problems at the system and model levels through requirements, design, and verification.Automated driving is the example, and the paper notes that many of the problems also apply to safety-critical systems generally.Source: §2 · p. 2
- Figure 1 presents the engineering process for machine-learning systems.The text identifies the figure as showing the two analysis levels and three development steps.Source: Figure 1 · p. 2, §2 · p. 2
3 Engineering Machine-Learning Models
The paper proposes treating machine-learning model development as an engineering process of requirements, design, and verification. It reviews open issues in specifying data, designing training, and verifying properties that test performance alone may not establish.
3 Engineering Machine-Learning Models
- The proposed model-engineering process maps test-data specification to requirements, training-data construction to design, and property checks to verification.The paper identifies robustness and interpretability as properties not captured by test-data performance metrics alone.Source: §3 · p. 3
3.1 Requirements of Machine Learning Models
- Test data should be specified as part of model requirements to connect deductive requirements with data-driven training.The paper distinguishes test data, which should reflect operational conditions, from training data, which is used to improve model performance; same-source data splits can implicitly assume test and operational data are close.Source: §3.1 · p. 3
- Operational data may drift from test data over time, potentially deteriorating performance and motivating consistency checks, maintenance, or deviation detection.The text names covariance shift, distributional shift [3], and concept drift for this phenomenon.Source: §3.1 · pp. 3–4
Related Works and Research Directions of Machine Learning Model Requirements
- Requirements research should describe test environments and distributions, associate conditions with desired performance, and account for condition-specific metrics.The cited discussion notes that prior coverage work addressed combinatory environments, while environment importance and required performance may vary.Source: Related Works and Research Directions… · p. 4, Table 1 · p. 4
3.2 Design of Machine Learning Models
- Training specifications and data can be designed to meet requirements, but the authors report no widely accepted process for designing training data.The paper describes iteratively identifying model weak points and collecting or generating additional training data as a research direction.Source: §3.2 · p. 5, Related Works and Research Directions… · p. 5
3.3 Verification of Machine Learning Models
- Robustness verification can quantify the largest perturbation radius around a required data point for which the model still answers correctly.The paper describes generating adversarial examples by perturbing data points and treats robustness against disturbance as a key verification concern.Source: §3.3 · p. 5
Research Directions of Machine Learning Model Verification
- Formal neural-network verification research distinguishes property quantification from deciding whether properties hold, including proof and counterexample approaches.Examples include global safety, local safety, robustness properties, and falsification tools such as CNN Analyzer and Reluplex.Source: Research Directions of Machine… · pp. 5–6
4 Engineering Machine Learning Systems
The paper reviews system-level engineering problems for safety-critical machine-learning systems, using automated driving as an example. It highlights challenges in defining operational environments, accommodating model changes, and verifying system safety within practical testing limits.
4 Engineering Machine Learning Systems
- System-level problems arise from model characteristics and open operating environments, including low modularity, missing design specifications, and difficult field-operation testing.The review uses in-vehicle automated driving as its example and treats open environments as a common challenge for machine-learning systems.Source: §4 · p. 6
4.1 Requirements of Machine Learning Systems
- Machine-learning systems need suitable system quality characteristics, but the authors found no standard quality model accounting for model-specific characteristics.The paper analyzes gaps in SQuARE because models are embedded in systems rather than used alone.Source: §4.1 · p. 6
- Defining an operational design domain is difficult, and the paper identifies the lack of a standard method for specifying the operating environment as an open requirements problem.The domain may be described through geography, road type, traffic conditions, and maximum vehicle speed.Source: §4.1 · p. 6
Related Works and Research Directions of Machine Learning System Requirments
- PEGASUS uses scenario-based verification to clarify expected performance and evaluation criteria for automated-driving systems.Its scenarios are drawn from test drives and the market to assess whether systems match or exceed human-driver performance.Source: Related Works and Research Directions… · pp. 6–7
4.2 Design of Machine Learning Systems
- System architectures should accommodate newly trained models and additional requirements without requiring model modification, addressing the rework risk associated with CACE.Because data-driven training makes changes difficult to localize, retraining after a small change can alter the entire model and require system changes.Source: §4.2 · p. 7
Related Works and Research Directions of Machine Learning System Design
- The paper identifies test stubs, rule-based safeguards, and redundant diverse architectures as possible system-design directions for managing model uncertainty and low robustness.It notes that SOTIF addresses uncertain functions through scenario identification and system modification, a process potentially applicable to machine-learning system design.Source: Related Works and Research Directions… · p. 7
4.3 Verification of Machine Learning Systems
- System verification can reduce time demands by excluding unrealistic scenarios or accelerating simulation, while reproducing rare corner cases on test courses.The paper contrasts these options with accumulating large volumes of real-world driving data.Source: §4.3 · p. 7
Related Works and Research Directions of Machine Learning System Verification
- Statistically significant field-operation testing may require hundreds of thousands to hundreds of billions of miles, although the cited estimate uses simple hypothesis testing and is not optimized to actual situations.Source: Related Works and Research Directions… · p. 7
5 Quality of Machine-Learning Systems
The paper evaluates machine-learning system quality by combining model characteristics with the conventional SQuARE model. Its gap analysis identifies affected quality measures and proposes extensions for requirements, uncertain observations, rare-case discovery, and development data.
5.1 Conventional System Quality Model
- SQuARE organizes quality measures and measure elements beneath quality models, characteristics, and sub-characteristics; service product quality has no defined measures or measure elements.The top-level models are product quality, data quality, service product quality, and quality in use; data quality has no defined sub-characteristics.Source: §5.1 · pp. 7–8
5.2 Gap Analysis
- 21 of 1,464 examined QME–characteristic combinations were affected; product quality had the highest impact, while data quality and quality in use had low impacts.The impact summaries cover system and software product quality, data quality, and quality in use.Source: §5.2 · pp. 8–9, Table 3 · p. 10, Table 4 · p. 11
- Missing preconditions and ambiguous function specifications make related quality measures difficult to measure, affecting functional suitability and product portability.Machine-learning preconditions are encoded in training and test data rather than explicitly described; varied inputs such as pedestrians complicate precise function definitions.Source: §5.2 · p. 9
- Low robustness makes behavior uncertain under slight input changes, leaving correctness ratios unmeasurable when both incorrect-function counts and functions considered cannot be established.These affected measures overlap with those impacted by missing requirements specifications.Source: §5.2 · pp. 10–11
- Rare cases can be missed or neglected during limited observation, causing related measures of outliers, failures, and failure avoidance to be underestimated.The paper notes that a long field operational test is required to capture rare events.Source: §5.2 · pp. 11–12
- Lack of design specifications and transparency has a small impact, including on modification effort, component coupling, and redundancy measures; diagnostic and monitoring measures are also not measurable.The paper attributes the latter gap to the absence of an established diagnostic and monitoring method for machine-learning models.Source: §5.2 · p. 12
5.3 Toward Machine Learning System Quality Model
- Proposed requirements measures combine deductive specifications with sample-data coverage, while observation measures add trial counts and variance and rare-case measures track discovery effort and cases found per unit time.For structured requirements arguments, sample-data coverage can be measured as the ratio of leaf nodes with corresponding sample data.Source: §5.3 · pp. 12–13
- The proposed quality model should include training-data and testing-data quality, which SQuARE’s existing data-quality perspective does not cover as development data.Source: §5.3 · p. 13
6 Conclusion
The conclusion frames safety-critical machine-learning assurance as open engineering work spanning requirements, designs, and verifications for models and systems. It connects model training to deductive lifecycle concepts and reports that missing requirements specification and robustness most affect the SQuARE-based quality model.
6 Conclusion
- The paper presents open engineering problems, related work, and research directions for requirements, designs, and verifications of machine-learning models and systems.Automated driving is the example, though many problems apply more broadly to safety-critical systems.Source: §6 · p. 13
- At the model level, the authors hypothesize a training process linking deductive requirements to data-driven training, treating test data as requirements specifications and training data as design specifications.They identify missing requirements and design specifications, interpretability, and robustness as model characteristics.Source: §6 · p. 13
- Missing requirements specifications and robustness have the greatest impact on the SQuARE-based system quality model.The cited missing specifications concern preconditions and function detail; robustness concerns observation uncertainty and extremely rare cases.Source: §6 · p. 13
- A widely accepted engineering process remains unsettled; future work includes a standard quality model, specific QMs and QMEs, and guidelines for requirements, design, and verification.The paper also proposes exploring quality characteristics beyond SQuARE and multiple verification tiers.Source: §6 · p. 13