PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products

Kush R. Varshney, Homa Alemzadeh

arXiv:1610.01256v2cs.CYstat.ML

TL;DR

The paper defines safety in machine learning and develops this definition in the machine learning context. It finds that standard statistical risk minimization omits epistemic uncertainty, which should be included when considering safety.

  • Problem

    The paper addresses how to define safety in machine learning.

  • Method

    The paper builds a machine learning safety discussion from a basic definition involving harm, risk, and uncertainty.

  • Results

    Standard statistical risk minimization omits epistemic uncertainty, which needs inclusion when considering machine learning safety.

  • Takeaways & Limitations

    Safety considerations should extend beyond risk minimization to include epistemic uncertainty.

Abstract

from arXiv · show

Machine learning algorithms increasingly influence our decisions and interact with us in all parts of our daily lives. Therefore, just as we consider the safety of power plants, highways, and a variety of other engineered socio-technical systems, we must also take into account the safety of systems involving machine learning. Heretofore, the definition of safety has not been formalized in a machine learning context. In this paper, we do so by defining machine learning safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. We then use this definition to examine safety in all sorts of applications in cyber-physical systems, decision sciences, and data products. We find that the foundational principle of modern statistical machine learning, empirical risk minimization, is not always a sufficient objective. Finally, we discuss how four different categories of strategies for achieving safety in engineering, including inherently safe design, safety reserves, safe fail, and procedural safeguards can be mapped to a machine learning context. We then discuss example techniques that can be adopted in each category, such as considering interpretability and causality of predictive models, objective functions beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software and open data.

I. INTRODUCTION

The paper formalizes machine learning safety as minimizing the risk and epistemic uncertainty of harmful unwanted outcomes. It examines this framework across applications and maps established engineering safety strategies to machine learning systems.

  • Definition of Safety: Machine learning safety is defined as minimizing both the risk and uncertainty of harms.The definition distinguishes safety from ordinary predictive performance by including uncertainty about harmful outcomes.
  • Safety and Statistical Learning: Empirical risk minimization and structural risk minimization do not address epistemic uncertainty and use abstract losses rather than harm-specific costs.The paper questions whether average-loss objectives and laws of large numbers adequately capture outcomes such as loss of life or quality of life.
  • Safety Strategies: Four engineering safety strategies are mapped to machine learning: inherently safe design, safety reserves, safe fail, and procedural safeguards.The discussion emphasizes adapting these approaches especially to mitigate epistemic uncertainty.
  • Application Domains: The framework is applied to cyber-physical systems, decision sciences, and data products, whose interactions, operating timescales, and consequence costs vary widely.Examples include surgical robots, self-driving cars, parole and medical decisions, loan approval, advertising, recommendation, and spam filtering.
  • Overall Framework: The paper proposes a common language for understanding, evaluating, and designing machine learning systems embedded in society and technology.It also discusses characterizing and quantifying outcome desirability, undesirability, and costs, including data-driven approaches for large outcome spaces.

II. SAFETY IN MACHINE LEARNING

This section introduces statistical machine learning notation and examines how machine learning applications fit the paper’s conception of safety.

  • II. SAFETY IN MACHINE LEARNING: The section frames machine learning applications using the paper’s conception of safety after introducing standard statistical machine learning notation.The stated focus is the relationship between machine learning applications and the preceding safety framework.

A. Notation

The section formalizes risk through expected loss and empirical risk, then connects loss to harmful outcomes and safety across application domains. It emphasizes that harm and acceptable safety levels depend on outcomes, perspective, and context.

  • Risk and empirical risk: Risk is the expected loss over the joint distribution of features and labels, with the learned function chosen to minimize that risk.The loss measures prediction discrepancy, such as squared error in regression.
  • Risk and empirical risk: Because the underlying distribution is unavailable, machine learning instead uses an empirical risk computed from an i.i.d. training set.The training samples are represented as paired features and labels, and empirical risk minimization seeks a function minimizing observed loss.
  • Harmful costs: Loss can incorporate human cost, and sufficiently severe costs make prediction outcomes a safety issue.The loss domain is prediction-oriented, but real-world costs may also depend on features and become harmful.
  • Application domains: Decision-science outcomes are often few and clearly undesirable, while cyber-physical systems have vast outcome spaces that make enumeration and cost elicitation nearly impossible.Accident-producing outcomes can carry high real-time human costs, requiring explicit safety consideration.
  • Application domains: Acceptable safety levels vary by society and application domain, as illustrated by differing accident rates and costs for motor vehicles and commercial aircraft.Data-product errors are generally not immediate safety hazards because their severe human consequences are usually only hypothetical.

C. Risk and Epistemic Uncertainty

Empirical risk minimization can reduce expected prediction loss, but safety also depends on epistemic uncertainty and harms arising outside observed or representative data. The section motivates safety strategies that extend standard risk minimization.

  • Risk and epistemic uncertainty: Empirical risk minimization can reduce the probability of harms, but its standard formulation does not capture uncertainty relevant to safety.The paper identifies uncertainty as an additional safety concern beyond expected risk.
  • Distributional uncertainty: Training data may come from a different distribution than deployment data, and this epistemic uncertainty can cause substantial harm.The paper treats unknown distribution mismatch as directly relevant to safety.
  • Coverage uncertainty: Even representative samples can leave large regions of feature-label space unobserved, making predictions depend on inductive bias and potentially introducing safety hazards.This issue is especially relevant when probability density is small across parts of the domain.
  • Operational uncertainty: Operational risk is empirical because systems encounter finite test samples, and it may exceed actual risk for small test sets even when the model is risk-optimal.Test-set instantiation uncertainty can have large safety implications for individual samples.
  • Application scope: Large datasets and feature-space exploration reduce epistemic uncertainty, whereas limited representation and large outcome spaces leave uncertainty about whether testing reflects deployment.Cyber-physical systems are especially difficult because their outcome spaces prevent even mild training coverage.
  • Safety strategies: The paper frames safety engineering through four categories: inherently safe design, safety reserves, safe fail, and procedural safeguards.These categories motivate extending machine-learning formulations beyond risk minimization.

1) Inherently Safe Design:

Inherently safe machine-learning design removes or constrains hazards through interpretability, causality, robustness, and fairness-aware objectives. These safeguards may require constraints beyond standard risk minimization and can trade accuracy for reduced uncertainty and bias.

  • Robustness: Unknown biases and harmful patterns can yield high accuracy yet unsafe predictions under domain shifts, especially when complex models make their reactions difficult to understand.Extreme gradient boosting and deep neural networks are identified as examples of models that may exploit such biases.
  • Model structure: Interpretable models can expose data quirks for exclusion, while causally related features can remove patterns outside the system’s true physics.Both approaches are presented as ways to avoid associated harm through model design.
  • Model structure: Post hoc interpretation and repair of complex models do not assure inherently safe design because the interpretation is not the decision rule actually used.The safety principle requires the deployed decision rule itself to embody the relevant safeguards.
  • Beyond risk minimization: Interpretability and causality are not properly captured by standard risk minimization, so inherently safe models require extra regularization or constraints beyond structural risk minimization.Such constraints may reduce standard accuracy while reducing epistemic uncertainty and undesired bias.
  • Fairness: Fairness constraints can limit the relative harm risk for protected groups compared with unprotected groups, including through the 80% rule.The resulting constraint aims to prevent protected-group predictions from being much more harmful.

3) Safe Fail:

Safe-fail strategies defer, stop, or transfer decisions when predictions are unreliable or hazardous, but their triggers and human involvement must match the application's time scale and uncertainty.

  • The reject option lets a model decline an unreliable prediction so a human operator can examine the sample and provide a manual prediction.
  • Distance from the decision boundary is an unreliable confidence trigger in low-density regions with sparse or absent training data.
  • For rare feature combinations, safe fail can require manual examination rather than automated prediction.
  • Manual intervention suits applications with sufficiently long time scales, whereas millisecond operations require mechanisms such as dead man’s switches.
  • Procedural safeguards include simulating rare hazardous events and using user experience design to guide practitioners toward safer system setup.
  • Opening data supports safety by enabling use, reuse, and redistribution, while open source software alone is insufficient because behavior also depends on data.

IV. EXAMPLE APPLICATIONS

The paper examines machine-learning safety across cyber-physical systems, using surgical robots as an example where uncertainty, rare events, and interacting human and physical factors complicate safe operation.

  • Cyber-physical systems combine real-time interaction with physical environments and human operators, creating challenges for reliability, safety, and expected performance.
  • Surgical robots estimate tissue states from sensor measurements, generate action plans, and learn surgical skills from their actions or expert demonstrations.
  • Uncertainty in operator behavior, tissue dynamics, and incidental failures makes it difficult to predict all system states and assign costs to outcomes.
  • Surgical incidents, including interruptions or minor injuries, can arise from interacting failures involving instruments, surgeons, embedded systems, and patient history.
  • Training data from selected tasks and well-trained surgeons may not represent the action and task variety of real procedures.
  • Robustness assessment, historical adverse-event analysis, supervisory control, interpretable models, and robust planning illustrate procedural, safe-fail, inherent-design, and reserve strategies.

2) Self-Driving Cars:

Self-driving cars face safety hazards from noisy sensing, perception limits, and unexpected environments, including rare circumstances that can defeat autonomous control.

  • Self-driving cars combine sensor data to map the environment, estimate position, plan routes, and control steering, braking, and throttle.
  • Noisy sensors, computer-vision limitations, and unexpected road changes can impair environmental understanding and safe decision-making.
  • A fatal autopilot collision occurred after the system failed to brake; it was the first known fatality in over 130 million testing miles.
  • The crash occurred under an extremely rare combination involving the truck’s height and white color under a bright sky and vehicle positioning.
  • Research assesses robustness to rare or unseen events, with recognition of such events triggering safe-fail mechanisms.
  • No self-driving car system is known to use an inherently safe design based on interpretable models; fallback options include backup systems or driver control.

B. Decision Sciences

Decision-science systems make decisions about people with people in the loop, but training-label construction and unusual data patterns introduce epistemic uncertainty that can affect safe use.

  • Data-driven decision applications use machine-learning tools to support human decisions about people and present safety-related challenges.
  • Predicting employee resignation requires constructing labels from historical resignations and workforce status, but future resignations can be mislabeled as negative samples.
  • Predictive quirks in acquired-company contract stipulations may not generalize to future data.
  • A unique Australian employee profile creates substantial uncertainty because no similar employees occupy that region of feature space.
  • Interpretability revealed the acquired-company pattern, while causality mattered because interventions such as salary increases were intended to retain at-risk employees.
  • Rare samples can trigger safe fail through manual inspection, but manual intervention is suitable only when the application has sufficiently long time scales.

2) Loan Approval:

Loan approval illustrates that machine-learning safety depends on stakeholder-specific costs, fairness, and epistemic uncertainty rather than prediction correctness alone. The example also shows that feature removal may not ensure fairness, while safety margins can improve it.

  • Epistemic uncertainty: The loan-approval example also involves epistemic uncertainty when the training set is not representative of the true test distribution.The paper states that similar safety strategies can address this uncertainty.
  • Stakeholder perspectives: Loan-approval loss functions are not always monotonic in prediction correctness and depend on the stakeholder’s perspective.Applicants, lenders, and society can assign different costs to the same approval decision.
  • Fairness: Using surname as a feature improved classification accuracy by a couple of percentage points but worsened fairness.The surname indicated religion and caste in the application setting.
  • Fairness: Dropping a sensitive attribute does not ensure fairness because other features may be correlated with it.The paper instead describes a safety margin on group accuracy as making the system fairer.
  • Data products: The paper does not treat delayed, hypothetical consequences from informational products as safety issues.Examples include biased advertisements or spam filtering that might eventually affect quality of life.
  • Data products: In data products, billions of data points and large-scale A/B testing can make uncertainty less prominent in safety considerations.The paper contrasts these applications with settings where uncertainty is more foregrounded.

V. CONCLUSION

The conclusion frames machine-learning safety around harm, risk, and epistemic uncertainty, and argues that epistemic-uncertainty minimization is missing from standard statistical risk minimization. It presents application analyses and safety strategies as foundations for further research, while acknowledging that the strategy list is incomplete and underdeveloped.

  • Conclusion: The paper defines machine-learning safety in terms of harm, risk, and uncertainty, building on a basic engineering definition.It applies this framing to machine-learning contexts and example applications.
  • Conclusion: Standard statistical machine learning emphasizes risk minimization but misses the minimization of epistemic uncertainty.The conclusion says epistemic uncertainty needs to be included when considering safety.
  • Research agenda: The discussed safety strategies are not comprehensive and remain far from fully developed.The paper presents itself as laying foundations for a research agenda rather than a finished strategy catalogue.
  • Research agenda: Safety, especially epistemic-uncertainty minimization, is presented as a direction for new research problems.The conclusion connects this agenda to machine learning’s broadening role in society.
  • Legal context: The paper relates non-harm to ethical duty and observes that safety concerns are also entering legal frameworks.The conclusion invokes the phrase “ahim. s¯a paramo dharmah.” in this context.
  • Legal context: The conclusion notes that machine-learning safety issues are beginning to enter legal obligations.It cites European data-protection regulations concerning decisions based solely on automated processing.
  • Applications: The paper examines epistemic uncertainty, harmful outcomes, and potential safety strategies across increasingly common machine-learning applications.These include cyber-physical systems, decision sciences, and other settings with differing criticality and direct harm.
Loading 1610.01256v2…