PaperlayerView exact revision on arXiv

Source-linked AI summary

Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges

Rob Ashmore, Radu Calinescu, Colin Paterson

arXiv:1905.04223v1cs.LGcs.SEstat.ML

TL;DR

Safety-critical applications need assurance that ML is sufficiently safe for its intended use, but evidence must span the complex lifecycle from data collection through deployment. The paper surveys assurance methods across that lifecycle, defines stage-specific desiderata, and identifies open challenges. It finds uneven support across stages and desiderata, with important gaps in context-sensitive verification, adaptable deployment, and lifecycle integration.

  • Problem

    Safety-critical ML applications require assurance evidence beyond that needed for current applications, but existing support must address the full ML lifecycle.

  • Method

    The paper surveys assurance methods across four ML lifecycle stages, defining desiderata for each stage and reviewing methods, assumptions, limitations, and open challenges.

  • Results

    The survey finds uneven assurance coverage: Model Learning methods emphasize performance and robustness, Model Verification methods mainly support comprehensive testing, and Model Deployment has only two methods for model updating.

  • Takeaways & Limitations

    Assurance of ML for safety-critical systems must cover all lifecycle stages, while remaining challenges require further research to satisfy stage-specific desiderata.

  • Takeaways & Limitations

    Open challenges include integrating operating context into Model Learning, mapping verification requirements to model features, and managing fleet-wide diversity during deployment.

Abstract

from arXiv · show

Machine learning has evolved into an enabling technology for a wide range of highly successful applications. The potential for this success to continue and accelerate has placed machine learning (ML) at the top of research, economic and political agendas. Such unprecedented interest is fuelled by a vision of ML applicability extending to healthcare, transportation, defence and other domains of great societal importance. Achieving this vision requires the use of ML in safety-critical applications that demand levels of assurance beyond those needed for current ML applications. Our paper provides a comprehensive survey of the state-of-the-art in the assurance of ML, i.e. in the generation of evidence that ML is sufficiently safe for its intended use. The survey covers the methods capable of providing such evidence at different stages of the machine learning lifecycle, i.e. of the complex, iterative process that starts with the collection of the data used to train an ML component for a system, and ends with the deployment of that component within the system. The paper begins with a systematic presentation of the ML lifecycle and its stages. We then define assurance desiderata for each stage, review existing methods that contribute to achieving these desiderata, and identify open challenges that require further research.

1 INTRODUCTION

Machine learning is moving toward safety-critical applications, where errors can cause irreversible harm and therefore require stronger assurance. The paper surveys assurance methods across the ML lifecycle, organizing them by stage-specific requirements and identifying remaining challenges.

  • Motivation: Safety-critical ML applications require stronger assurance because errors may cause failures that cannot be ignored or reverted.The motivation contrasts tolerable errors in consumer applications with unacceptable errors in medical diagnosis and self-driving systems.
  • Scope and contribution: The paper surveys methods for obtaining assurance evidence for ML components across their development and use.It frames assurance as evidence that ML is sufficiently safe for its intended use.
  • Survey organization: The survey organizes assurance around four lifecycle stages: Data Management, Model Learning, Model Verification, and Model Deployment.The first three stages produce ML models, while deployment integrates them into operational systems.
  • Survey organization: For each stage, the paper identifies assurance desiderata, reviews methods with their assumptions and limitations, and highlights open challenges.The survey is structured around stage-specific assurance considerations and the artefacts produced at each stage.
  • Scope and contribution: The lifecycle-based survey applies broadly to supervised, unsupervised, and reinforcement learning and serves practitioners, researchers, and regulators.Methods are marked when they apply only to particular ML types.

2 THE MACHINE LEARNING LIFECYCLE

The ML lifecycle is a four-stage process that begins with data and model production, verifies generalization, and ends with deployment in an operational system. Deployment integrates verified models with conventional software while supporting monitoring, execution, and updating in dynamic environments.

  • Lifecycle overview: The ML lifecycle consists of Data Management, Model Learning, Model Verification, and Model Deployment.The first three stages form the machine learning workflow; deployment places models within an operational system.
  • Data Management: Data Management acquires, preprocesses, labels, and augments data for the training and verification data sets.These data sets support subsequent model synthesis and verification activities.
  • Model Learning: Model Learning selects a model and training configuration, then minimizes a loss function using the training data.Hyperparameter selection controls characteristics including overfitting, underfitting, and model complexity.
  • Model Verification: Model Verification provides evidence that a trained model generalizes to unseen inputs, including through testing against an independently produced verification set.The verification data may contain deliberately selected examples that would be inappropriate for training.
  • Model Deployment: Model Deployment integrates verified ML models with traditionally engineered components, monitors operation, and supports offline maintenance or online updating.New operational data can feed data-management activities and, where appropriate, enable new models to be learned and deployed.
  • Operational control: In dynamic systems, ML models can support the MAPE loop by analyzing sensed data or helping select actions under system constraints.The loop repeatedly monitors, analyzes, plans, and executes until objectives are achieved or a stopping criterion is reached.

3 RELATED SURVEYS

Existing ML surveys usually address particular ML types, lifecycle stages, or assurance properties rather than assurance across the full lifecycle. The paper positions its survey as broader by covering assurance evidence across stages and ML types.

  • Coverage gaps: Most ML surveys focus narrowly on a particular ML type and do not explicitly address assurance.Examples include surveys of deep, reinforcement, transfer, and ensemble learning.
  • Coverage gaps: Some broader surveys examine one lifecycle stage or activity, such as data management, feature selection, or dimensionality reduction.Their scope remains limited to the specific stage or activity studied.
  • Assurance-specific surveys: Interpretability surveys address one assurance-related model property but do not cover other desiderata or artefacts from other lifecycle stages.Their focus differs from a lifecycle-wide assurance treatment.
  • Assurance-specific surveys: Few recent surveys address assurance evidence or safety, and those that do generally specialize in a technique, ML type, or lifecycle stage.The cited surveys include neural-network verification, safe reinforcement learning, deep-neural-network safety, and automotive supervised learning.

4 DATA MANAGEMENT

Data is foundational to ML because it encodes the desired relationship between inputs and outputs, and therefore also encodes requirements that assurance arguments must explicitly consider.

  • Role of data: Data underpins every ML approach and describes the relationship that the model should learn between inputs and outputs.For unsupervised learning, the desired outputs may be implicit.
  • Role of data: Because data encode the requirements intended for the model, assurance arguments must explicitly consider the data.This connects data quality and content directly to the requirements embodied in the ML model.

4.1 Inputs and Outputs

The Data Management stage takes model requirements as input and produces separate development and verification data sets as outputs.

  • Model requirements are the key input to Data Management and may be informed by verification artefacts from earlier lifecycle iterations.
  • The stage produces a combined data set for training and validation and a separate verification data set for independent verification.

4.2 Activities

Data Management collects, preprocesses, augments, and analyzes data to create assured data-management artefacts. These activities support data preparation while addressing practical constraints such as collection cost, ethics, and class balance.

  • Collection: Data collection obtains samples from new or reused sources, including controlled or uncontrolled processes in real-world or synthetic environments.
  • Preprocessing: Preprocessing applies a one-to-one mapping to adjust each raw sample, including standardization and manual label addition.
  • Augmentation: Augmentation generally applies a one-to-many mapping that derives new samples from existing ones, often to address collection difficulty or improve model robustness.
  • Analysis: Exploratory analysis guides collection and augmentation and provides assurance that Data Management artefacts exhibit the required desiderata.

4.3 Desiderata

Assured data sets should be relevant, complete, balanced, and accurate. Relevance concerns the intended operational domain, while the other properties address distribution, representation, and measurement quality.

  • Relevant data sets overlap with the behaviour required in the intended operational domain.A data set containing only German road signs would not be relevant for a system intended for UK roads.
  • Complete data sets contain suitable distributions and combinations of features across the input domain.Completeness considers whether appropriate samples and feature combinations are present, including avoiding inappropriate correlations.
  • Balanced data sets appropriately distribute included features, including sample counts across classes in classification problems.Balance takes an internal perspective on the data set as an abstract entity.
  • Accurate data sets reflect the intended operational domain despite measurement issues such as sensor inaccuracy and labeling errors.The property also covers data collection and preprocessing software correctness and configuration management.
  • Relevance is associated with validation, whereas completeness, balance, and accuracy concern verification.

4.4 Methods

Data Management methods address the four data-set desiderata through collection, analysis, augmentation, and controls for leakage, bias, failures, and labeling. The survey also identifies unresolved challenges, especially in demonstrating completeness and resolving labeling discrepancies.

  • Methods for Data Management are organized around each desideratum and its applicable activities.
  • Relevant: Operational-use collection guarantees relevance in principle but is unlikely to provide all required data practically.
  • Relevant: Reusing existing data sets requires appropriate sources because malicious entries can introduce backdoors that control model behavior on specific inputs.
  • Relevant: Controlled trials require justified feature choices, while real-world observations and simulations benefit from corresponding experimental-design techniques.
  • Relevant: Synthetic data should be shown representative of the real world, with simulation suitably verified and validated because simulation bugs can affect ML approaches.
  • Relevant: Data leakage makes a data set irrelevant when training includes information unavailable during operational use; exploratory analysis can help detect it.
  • Relevant: Adversarial inputs labeled correctly in training data can help reduce misclassification and increase relevance.
  • Relevant: Bias can be addressed through preprocessing and augmentation, including feature de-predictability, relabeling, reweighting, and resampling.

4.5 Summary and Open Challenges

The Data Management stage has relatively few assurance methods for preprocessing and accuracy, while its open challenges span security, fitness, coverage, balance, collection integrity, and simulation realism.

  • Few methods are associated with preprocessing, likely because preprocessing is problem-specific.
  • Few methods address the Accurate desideratum, partly reflecting reliance on commonly available datasets such as ImageNet and MNIST.
  • Data Management challenges largely concern evidence about data and security rather than the core process of learning a model.
  • Open challenges include demonstrating cybersecurity, fitness for purpose, operational and adversarial coverage, class balance, uncompromised collection, and representative simulations.

5 MODEL LEARNING

Model Learning creates models from training data and should produce models that are performant, robust, reusable, and interpretable, supported by methods with different assumptions and trade-offs.

  • Model Learning creates a model from data while targeting output quality, timing, and acceptable computational-resource use.
  • The stage takes a training dataset as input and produces a model for verification plus a performance deficit report for remedial data management.
  • Model types, parameter optimization, hyperparameter selection, and transfer learning are activities used to construct or improve models.
  • Desiderata: The four assurance desiderata are performance, robustness to changed inputs, reusability across systems, and interpretability of model decisions.
  • Robust: Regularization reduces generalization error without reducing training error by penalizing model complexity or modifying training behavior.
  • Reusable: Transfer learning can significantly reduce training time by reusing parts of models trained for related tasks.

5.5 Summary and Open Challenges

Model Learning methods mainly support performance and robustness, whereas reuse and interpretability have fewer methods and remain less mature; major challenges involve operating context and global interpretability.

  • Most Model Learning assurance methods focus on performance and robustness.
  • Reuse and interpretability are supported by fewer methods that typically restrict the available model types.
  • This imbalance reflects different research maturity and the greater prominence of reuse and interpretability after advances in deep learning and model complexity.
  • Open challenges include integrating operating context, evaluating multiple objectives, relating performance to hyperparameters, and checking transferred models for errors.
  • Global interpretability methods for complex models remain lacking, and local explanations do not yet systematically yield global model properties.

6 MODEL VERIFICATION

Model Verification provides auditable evidence that trained models satisfy requirements on unseen inputs. It combines requirements encoding, testing, and formal verification, but remains limited by difficult coverage, context mapping, and actionable interpretation.

  • Verification activities: Model Verification transforms requirements into tests and mathematical properties, then checks trained models against them to produce auditable evidence.The stage outputs a verified model and verification result indicating suitability for intended applications.
  • Verification activities: Testing supplies specially formed inputs and compares model outputs with expected results, while also examining test completeness and robustness properties.Test-based verification can use simulations, fuzzing, and path-oriented input generation.
  • Verification activities: Formal verification provides mathematical evidence for specified properties and returns counterexamples when those properties are violated.Counterexamples can inform further iterations of Data Management and Model Learning activities.
  • Verification desiderata: Comprehensive verification must cover requirements, operating conditions, lifecycle desiderata, and faults introduced during data management and model learning.ML models typically have a larger testing space than traditional software, while conventional code-coverage measures do not adequately capture trained-model behavior.
  • Verification desiderata: Model coverage methods exist for data and model structure, but high coverage is generally unfeasible for large models and its empirical utility remains un demonstrated.Consequently, appropriate coverage thresholds cannot currently be defined, and it remains unclear whether they should be generic or application-specific.
  • Open challenges: Verification methods struggle to map formal properties to real-world context, and no general method links verification results to improved model-learning choices.Formal methods may produce counterexamples that are difficult to comprehend and cannot guide performance-improvement actions.

7 MODEL DEPLOYMENT

Model Deployment integrates verified ML models into operational systems, monitors their operation, and manages updates. Assurance focuses on fit-for-purpose, tolerated, and adaptable deployments, while distribution shift, incorrect outputs, and fleet-wide updating remain challenges.

  • Deployment activities: Model Deployment integrates verified ML models with traditionally engineered system components and produces a fully deployed, operating system.The stage also includes operational monitoring and offline or online updates.
  • Integration and monitoring: Integration links sensors and model outputs to the wider architecture while protecting the system against occasional incorrect model outputs.Monitoring can cover inputs, the operating environment, model internals, and model outputs.
  • Desiderata: Deployment assurance requires models to be fit-for-purpose, tolerated by the wider system, and adaptable to change.The same model may be suitable in one system context but unsuitable in another, while occasional incorrect outputs must be tolerated.
  • Fit-for-purpose: Operational inputs can differ from training and verification data because of inadequate representation, subsystem failures, adversaries, or environmental and behavioral changes.This distribution shift can produce deployed performance unlike verification performance, and timely detection remains an open challenge, especially for high-dimensional data.
  • Adaptability: Updating deployed models requires fleet-level management when multiple system instances are operating, including controlled rollout and monitoring of fleet-wide diversity.The survey identifies understanding how to manage this diversity as an open challenge.
  • Open challenges: Only two methods support ML-model updating in the surveyed deployment methods, helping explain the lack of assurance methods for adaptable deployments.The survey relates this scarcity to the limited current market of autonomous systems with many operational instances.

8 CONCLUSION

The survey examines assurance across the ML lifecycle because extending ML to safety-critical domains requires evidence that deployed models are sufficiently safe. It finds substantial method coverage alongside important gaps and open challenges at every lifecycle stage.

  • Scope: Extending ML from non-safety-critical applications to healthcare, transportation, and manufacturing requires assurance of models deployed in those systems.The paper frames assurance as necessary for applying ML in safety-critical domains.
  • Contribution: The survey covers all ML lifecycle stages, defines stage-specific assurance desiderata, reviews methods, and identifies remaining open challenges.Its lifecycle-wide organisation is intended to structure assurance evidence across the development and deployment process.
  • Data Management: Data-management methods support Relevant, Complete, Balanced, and Accurate training and verification data, but data security, fitness-for-purpose, and realistic simulation remain research needs.The survey reports broad coverage of collection, preprocessing, augmentation, and analysis methods.
  • Model Learning: Model-learning research provides many methods for Performant and Robust models, while Reusable and Interpretable models and improved hyperparameter and transfer-learning methods remain needed.The paper particularly highlights the need for context-relevant explanations of model behaviour.
  • Model Verification: Model verification has numerous test-based methods and a smaller formal-verification repertoire, but encoding requirements into verifiable tests and making results Comprehensible remain difficult.Verification results may be Comprehensive and Contextually Relevant, yet these properties are not consistently achieved.
  • Model Deployment: Deployment methods support Fit-for-Purpose and Tolerated models, whereas updating methods are scarce and generally insufficient to assure Adaptable safety-critical deployments.Integration and monitoring draw substantially on methods analogous to traditional software engineering.
  • Conclusion: Assurance concerns remain at every ML lifecycle stage, although future work can build on advances in ML, software engineering, and systems engineering.The conclusion presents safety-critical ML as involving numerous open challenges.
Loading 1905.04223v1…