Source-linked AI summary
Engineering Safety in Machine Learning
Kush R. Varshney
TL;DR
As machine learning becomes embedded in sociotechnical systems, its safety must be considered, yet standard risk minimization omits epistemic uncertainty and may not represent human harms. The paper defines safety, examines risk minimization and application types, and concludes that epistemic uncertainty is more prevalent in Type A than Type B applications.
Problem
Standard risk minimization captures risk but not epistemic uncertainty or application-specific human harms, complicating safety analysis as machine learning becomes embedded in sociotechnical systems.
Source: §I · p. 1Method
The paper defines safety through harm, risk, and epistemic uncertainty, examines risk-minimization principles, and discusses engineering safety strategies for machine-learning systems.
Source: §I · p. 1, §VI · p. 5Results
The paper concludes that risk minimization omits epistemic uncertainty, which is more prevalent in Type A than Type B applications.
Source: B. Epistemic Uncertainty · p. 3, §VI · p. 5Takeaways & Limitations
Opening data can serve as a procedural safeguard for increasing safety, particularly in Type A applications.
Source: D. Procedural Safeguards · p. 4The paper focuses on safety related to predictions and their outcomes, leaving other parts of machine-learning systems, such as privacy and disclosure risk, for further study.
Source: §VI · p. 5
Abstract
from arXiv · showhide
Machine learning algorithms are increasingly influencing our decisions and interacting with us in all parts of our daily lives. Therefore, just like for power plants, highways, and myriad other engineered sociotechnical systems, we must consider the safety of systems involving machine learning. In this paper, we first discuss the definition of safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. Then we examine dimensions, such as the choice of cost function and the appropriateness of minimizing the empirical average training cost, along which certain real-world applications may not be completely amenable to the foundational principle of modern statistical machine learning: empirical risk minimization. In particular, we note an emerging dichotomy of applications: ones in which safety is important and risk minimization is not the complete story (we name these Type A applications), and ones in which safety is not so critical and risk minimization is sufficient (we name these Type B applications). Finally, we discuss how four different strategies for achieving safety in engineering (inherently safe design, safety reserves, safe fail, and procedural safeguards) can be mapped to the machine learning context through interpretability and causality of predictive models, objectives beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software.
I. INTRODUCTION
The paper frames machine-learning safety as minimizing risk and epistemic uncertainty tied to harmful outcomes, then examines where standard risk minimization falls short. It distinguishes high-consequence Type A applications from lower-consequence Type B applications and discusses four engineering strategies for safer ML systems.
I. INTRODUCTION
- Safety is defined through risk and epistemic uncertainty associated with unwanted outcomes severe enough to be harmful.The paper notes that the magnitude of harm matters, not simply the frequency of undesirable outcomes.Source: §I · p. 1
- Empirical and structural risk minimization do not address epistemic uncertainty and rely on average losses that may not capture application-specific human harms.Their abstract prediction-error losses may not represent quantities such as loss of life or quality of life.Source: §I · p. 1
- Type A applications support consequential decisions with profound effects on people, whereas Type B applications operate at low consequence and large scale.The paper identifies safety as paramount for Type A applications.Source: §I · p. 1
- The paper discusses four safety-engineering strategies for ML: inherently safe design, safety reserves, safe fail, and procedural safeguards.It presents examples of these strategies for mitigating epistemic uncertainty and proposes them as a basis for safer ML methods and further research.Source: §I · pp. 1–2
II. DEFINITION OF SAFETY
The paper defines safety in terms of harmful outcomes, their costs, and both risk and epistemic uncertainty. Safety therefore concerns reducing expected harm as well as the possibility of unexpected harm.
II. DEFINITION OF SAFETY
- An unwanted outcome counts as harm only when its cost exceeds a threshold; events with small severity are not safety issues.Costs can be quantified socially, though sometimes with difficulty.Source: §II · p. 2
- Risk is expected harm cost when the outcome distribution is known, while uncertainty means that distribution is unknown or only partly known.Epistemic uncertainty reflects a lack of knowledge that could in principle be obtained, though gathering it may be practically intractable.Source: §II · p. 2
- Safety involves reducing both the probability of expected harms and the possibility of unexpected harms.The paper emphasizes that an event must carry sufficiently high human cost to count as harmful.Source: §II · p. 2
III. SAFETY AND MACHINE LEARNING
The paper argues that standard risk-minimization principles do not capture all safety-relevant uncertainty or human costs. It identifies hazards from distributional uncertainty, finite deployment samples, and loss functions that fail to represent application-specific harm.
III. SAFETY AND MACHINE LEARNING
- Empirical risk minimization selects a model by minimizing average loss on sampled training data, with convergence to risk as sample size grows.Structural risk minimization addresses small-sample complexity by restricting the hypothesis class or adding regularization based on an inductive bias.Source: §III · p. 2
- Risk minimization captures the risk component of safety but does not capture relevant uncertainty or application-specific loss functions.Training data may come from a different distribution than the one of interest, and that mismatch can cause harm.Source: §III · p. 2
- When training samples are absent from low-density regions of the feature-label space, the learned model can depend entirely on inductive bias rather than the uncertain true distribution.The paper identifies this dependence as a potential safety hazard.Source: §III · p. 2
- Finite deployment test sets can produce operational risk much larger than true risk, even when the model is risk-optimal.The realized test-set risk can have safety implications for individual samples.Source: §III · p. 2
- Application-specific human costs can make loss depend on features as well as predictions, with safety concerns arising only in parts of the domain where costs are severe enough.Ordinary risk-minimization losses instead represent abstract prediction error over the prediction and label spaces.Source: §III · pp. 2–3
IV. TYPE A AND TYPE B APPLICATIONS
The section turns from general machine-learning safety considerations to specific applications.
IV. TYPE A AND TYPE B APPLICATIONS
- The section examines safety considerations in specific applications after discussing general machine-learning safety.Source: §IV · p. 3
A. Harmful Costs
Whether an unwanted prediction is a safety hazard depends on the human severity of its consequences and whose perspective the loss function represents.
A. Harmful Costs
- Predictions in medical diagnosis, loan approval, and prison sentencing can have profoundly harmful human consequences, unlike quality-of-service errors in lower-consequence applications.The passage contrasts these with video compression, news ranking, and phoneme classification.Source: A. Harmful Costs · p. 3
- Loss functions are not always monotonic in prediction correctness, and their costs depend on whether the applicant, lender, or society's perspective is represented.For loan approval, applicants prefer approval, lenders prefer approval when repayment is likely, and society values fairness for protected groups.Source: A. Harmful Costs · p. 3
- The lender's perspective matches the typical loss function, while the applicant's and society's perspectives do not.Source: A. Harmful Costs · p. 3
B. Epistemic Uncertainty
Type B applications tend to combine lower-consequence errors with large-scale data and testing, while Type A applications more often face epistemic uncertainty and consequential decisions.
B. Epistemic Uncertainty
- Type B applications can use large training and testing sets, feature-space exploration, and large-scale A/B testing, making epistemic uncertainties less prevalent.The news-portal example uses billions of training points and evaluates average performance across millions or billions of clicks.Source: B. Epistemic Uncertainty · p. 3
- Type A applications more often face uncertainty about training samples representing test samples, with uncertainty common and only a few predictions made.Source: B. Epistemic Uncertainty · p. 3
- Type B applications have less costly errors and less system uncertainty, so risk minimization can be their main focus, unlike safety-critical Type A applications.Source: B. Epistemic Uncertainty · p. 3
V. STRATEGIES FOR ACHIEVING SAFETY
The paper maps four general safety strategies to machine learning and develops inherently safe design through interpretable and causally grounded models.
V. STRATEGIES FOR ACHIEVING SAFETY
- The paper discusses four safety strategies—inherently safe design, safety reserves, safe fail, and procedural safeguards—and their machine-learning specializations.Source: §V · p. 3
A. Inherently Safe Design
- Inherently safe design excludes hazards rather than merely controlling them; in machine learning, the target is robustness to training–test distribution shifts.Training data may contain unknown quirks or biases that do not occur at test time, which complex models may learn and later fail on under domain shift.Source: A. Inherently Safe Design · p. 3
- Interpretable models can reveal data quirks to exclude, while excluding non-causal features can remove variables unrelated to the system's underlying phenomena.Source: A. Inherently Safe Design · pp. 3–4
- Interpretability and causality require extra regularization or constraints beyond standard risk minimization, potentially sacrificing accuracy on a common training and testing distribution while reducing epistemic uncertainty.Source: A. Inherently Safe Design · p. 4
B. Safety Reserves
Safety reserves extend risk minimization by accounting for uncertainty and by constraining harm across protected and unprotected groups.
B. Safety Reserves
- Safety factors and safety margins represent multiplicative and additive reserves, respectively, relative to a system's designed load.Source: B. Safety Reserves · p. 4
- Robust formulations constrain or minimize worst-case risk across unknown conditions, including uncertainty in class priors, label noise, and which feature-space regions appear in a small test set.For medical diagnosis with few patients, maximum test error can matter more than average test error.Source: B. Safety Reserves · p. 4
- Disparate impact is a safety factor that constrains protected-group harm relative to unprotected-group harm, so the former is not much greater than the latter.The passage gives a minimum value such as 4/5 for the constraint.Source: B. Safety Reserves · p. 4
C. Safe Fail
Safe fail keeps a system safe when it cannot operate as intended; in machine learning, a reject option can withhold uncertain predictions, but rare feature combinations call for manual examination.
C. Safe Fail
- Safe fail means a system remains safe when it fails in its intended operation.Engineering examples include fuses, train dead man’s switches, and boiler safety valves.Source: C. Safe Fail · p. 4
- The reject option lets a model withhold a prediction it cannot give confidently, typically leaving a human operator to examine the sample and predict manually.The paper presents this as a way for a model to fail safely.Source: C. Safe Fail · p. 4
- Rare feature combinations in low-density regions call for manual examination because distance from the decision boundary is not a reliable trigger for rejection there.The usual assumption that distance from the boundary tracks confidence is considered reasonable in high-density regions with many training samples, but not in sparse regions.Source: C. Safe Fail · p. 4
D. Procedural Safeguards
Procedural safeguards supplement a system’s core functionality; for machine learning, the paper discusses user experience design and openness as ways to increase safety.
D. Procedural Safeguards
- Procedural safeguards are measures beyond those built into a system’s core functionality, including audits, training, and posted warnings.The paper identifies user experience design and openness as two machine-learning directions in this category.Source: D. Procedural Safeguards · p. 4
- User experience design can guide and warn practitioners, including non-specialist operators in Type A applications, about subtleties in training-data setup and evaluation that can cause harm if handled incorrectly.The paper identifies proper system setup as a way to increase safety.Source: D. Procedural Safeguards · p. 4
- Opening source code permits public audits, but safety review also needs to account for data because system behavior is driven by both data and software.The paper describes open data as increasingly adopted in Type A applications, while data is often a key value proposition in Type B applications.Source: D. Procedural Safeguards · p. 4
VI. CONCLUSION
The paper argues that machine-learning safety requires attention to epistemic uncertainty alongside risk minimization, especially in Type A applications. It presents strategies for safety while noting limits in the discussion’s scope and development.
VI. CONCLUSION
- The paper notes that machine-learning systems are already embedded in many societal functions and anticipates broader adoption across areas of life.This trend motivates its safety discussion.Source: §VI · p. 4
- The paper identifies epistemic uncertainty minimization as missing from standard risk-minimizing approaches and says safety considerations should include it, alongside strategies especially pertinent to Type A applications.It distinguishes Type A applications, where safety is important, from Type B applications, where it is not so.Source: §VI · p. 5
- The paper does not distinguish nominal safety, meeting regulations and design criteria, from substantive safety, the long-term performance a system actually exhibits.It identifies making this distinction as worthwhile future work.Source: §VI · p. 5
- The paper’s safety discussion covers predictions and outcomes based on predictions, while leaving other parts of machine-learning systems, including privacy and disclosure risk in microdata release, for further study.The paper calls for examining these other parts from first principles of cost, risk, and uncertainty.Source: §VI · p. 5
- The proposed safety strategies are neither comprehensive nor fully developed; the paper frames them as foundations for further research, particularly for Type A applications.The paper also identifies epistemic uncertainty minimization as a direction for further work.Source: §VI · p. 5