Source-linked AI summary
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Harini Suresh, John V. Guttag
TL;DR
ML systems can introduce harm throughout their life cycle, while broad labels such as “data bias” obscure the processes and choices involved. The paper develops a framework and vocabulary for distinguishing these sources of harm and connecting them to application-grounded analysis and mitigation.
Problem
Existing broad descriptions such as “data bias” do not precisely capture the complex historical processes, human choices, and pipeline stages through which ML harms can arise.
Method
The paper develops a framework and vocabulary that formalize distinct sources of downstream harm across data generation, model building, evaluation, and deployment.
Results
The framework identifies distinct sources of downstream harm at different ML life-cycle stages and provides corresponding terminology for analyzing them.
Takeaways & Limitations
Application knowledge and engagement with stakeholders should inform the identification of harms and the selection of appropriate mitigations rather than relying on one-size-fits-all notions of fairness.
Abstract
from arXiv · showhide
As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this paper, we provide a framework that identifies seven distinct potential sources of downstream harm in machine learning, spanning data collection, development, and deployment. In doing so, we aim to facilitate more productive and precise communication around these issues, as well as more direct, application-grounded ways to mitigate them.
1 INTRODUCTION
The paper argues that downstream harm can enter at many stages of the ML life cycle, not only through data, and presents a framework for identifying distinct sources and application-grounded mitigations.
- ML systems can produce harmful downstream consequences through data collection, model development, and deployment.
- Treating data as a static artifact obscures the historical context, human choices, and norms shaping how data is produced.
- The ML pipeline contains choices from model definition through deployment interfaces, and each stage can introduce undesirable effects.
- The framework provides vocabulary for distinguishing downstream harms by life-cycle stage and avoiding overly broad or overloaded terms.
- Application- and data-specific analysis can inform mitigation, while practitioners and external stakeholders can use the framework to probe systems.
2 MACHINE LEARNING OVERVIEW
Machine learning learns patterns or functions from existing data and generalizes them to unseen inputs. The paper introduces the ML life cycle as a sequence whose stages can introduce problems and are analyzed in later sections.
- Machine learning learns a function from existing data and applies it to make predictions about new, unseen data points.
- The ML life cycle is described through a running loan-approval example, with each step presented as typically practiced rather than ideally designed.
- The paper analyzes the implications of each life-cycle step and later formalizes the steps and their mitigations.
Data Collection
Data collection defines the target population, features, and labels, then samples a development subset because the full population is usually infeasible to include. Existing datasets are often used instead.
- Data collection identifies a target population and defines and measures its features and labels.
- Because including the entire target population is typically infeasible, practitioners sample a subset called the development sample.
- Practitioners often use existing datasets instead of collecting new data.
- The sampled loan-approval dataset depends on choices about population, sampling method, and which attributes to measure.
Data Preparation
Before model use, datasets undergo task-dependent preprocessing and are divided into training, validation, and test data for development and evaluation.
- Preprocessing varies with the data modality and task before the dataset is used.Examples include imputing missing values, simplifying features, and normalizing continuous measurements.
- Datasets are usually split into training data for model development and test data for model evaluation.Part of the training data may serve as validation data.
- Loan-approval preprocessing can impute missing credit history, group occupations, and scale income to a 0-to-1 range.
Model Development
Model development defines and optimizes a model using training data, while comparing configurations on validation data to select a final model.
- Model definition and optimization: Models are built using training data and optimized against a specified objective function.The example uses a dense feedforward neural network and cross-entropy loss to map inputs such as income and occupation to loan repayment outcomes.
- Model selection: Different model types, hyperparameters, and optimization methods may be tested during development.These configurations are compared using validation-data performance, and the best-performing configuration is chosen.
- Data use: The development process uses training data while excluding held-out validation data from model training.
Model Evaluation
Model evaluation reports performance on held-out test data and may use benchmark datasets for robustness demonstrations or comparisons with existing methods.
- Test evaluation: Test-data performance is reported only after the final model is chosen.Because test data is not used earlier, its performance is intended to represent behavior on unseen data.
- Benchmark evaluation: Benchmark datasets may demonstrate model robustness or enable comparison with existing methods.
- Metric selection: Evaluation metrics are selected according to the task and data characteristics.For loan approval, applicants may prioritize false negatives, whereas lenders may prioritize false positives.
Model Postprocessing
Post-processing converts model outputs into formats suitable for users, requiring choices such as thresholds for turning probabilities into categorical decisions.
- Output transformation: Binary-classification probabilities may be converted into hard classifications by choosing output thresholds.
- Output transformation: A loan-approval model’s continuous score between 0 and 1 may become risk buckets or a binary recommendation.
Model Deployment
Deployment integrates a trained model into a real-world setting, where interface, explainability, consistency, feedback, and recourse requirements may shape its use.
- Deployment requirements: Deployment may require changes for explainability, apparent consistency, or integration of real-time feedback.
- Deployment requirements: The post-deployment use population is not guaranteed to match the population in the development sample.
- User interaction and recourse: A loan-approval deployment may provide stakeholder-specific explanations and mechanisms for applicants to seek recourse.
3 SEVEN SOURCES OF HARM IN ML
The framework organizes harms by the ML life-cycle stages where they arise, replacing broad labels with distinct, more actionable sources. These sources include population and measurement choices, modeling, evaluation, and deployment practices.
- The taxonomy uses ML life-cycle stages to distinguish sources of harm and clarify how they arise.The paper chooses life-cycle stages as its organizing principle and treats “bias” as a set of granular sources of harm.
- 3.1 Historical Bias: Historical bias occurs when accurately measured and sampled data still reflects a harmful world or historical conditions.It can produce representational harm, such as reinforcing stereotypes.
- 3.2 Representation Bias: Representation bias occurs when the development sample underrepresents part of the population and generalizes poorly to some users.It can result from target-population definitions, underrepresented groups, or limited sampling methods.
- 3.2 Representation Bias: ImageNet contains approximately 45% U.S. images, while only 1% come from China and 2.1% from India.Classifiers trained on it perform significantly worse for some images from under-sampled countries such as Pakistan or India.
- 3.3 Measurement Bias: Measurement bias arises when features or labels poorly represent the constructs they are intended to measure.Problems include oversimplified proxies, uneven measurement, and inaccurate measurements shaped by structural discrimination.
- 3.5 Learning Bias: Learning bias arises when modeling choices amplify performance disparities across examples.Prioritizing one objective, such as overall accuracy, can damage another objective, such as disparate impact.
- 3.7 Deployment Bias: Deployment bias occurs when a model’s intended problem differs from its actual use in a sociotechnical system.Human decision-makers and institutional structures can introduce automation or confirmation bias despite good isolated model performance.
4 FORMALIZATION AND MITIGATIONS
The paper formalizes the ML pipeline as linked data transformations and uses that representation to connect each harm source to targeted mitigation points. It emphasizes application-specific interventions rather than a universal checklist.
- 4.1 Formalizing the framework: The formalization abstracts the ML pipeline into data transformations and provides a context for targeted mitigations.Figure 2 presents the pipeline as mapping functions spanning data collection, model building, evaluation, and deployment.
- 4.1 Formalizing the framework: The framework distinguishes target-population constructs, development samples, measured proxies, learned functions, evaluation metrics, and real-world decisions.The functions s, r, t, f_actual, k, and h represent sampling, measurement, learning, evaluation, and decision processes.
- 4.1 Formalizing the framework: The learned function f_actual is evaluated on measured data, while h combines it with a new input and environmental information to govern decision d.This makes deployment behavior part of the formal pipeline rather than treating the model as autonomous.
- 4.2 Designing Mitigations: The paper avoids a comprehensive mitigation checklist because solutions may not be generalizable across problems and applications.The framework instead organizes problem analysis, assesses suitable interventions, and motivates new mitigation techniques.
- 4.2 Designing Mitigations: Measurement and historical bias are not generally addressed by collecting more data through the same transformations.Measurement bias concerns feature and label generation, while historical bias concerns harmful population-level distributions.
- 4.2 Designing Mitigations: Representation-bias interventions should target population definition or sampling, assuming the feature and label transformations are acceptable.Changing features, labels, or objectives may be misguided when the problem originates in s rather than r, t, or f.
- 4.2 Designing Mitigations: Learning-bias mitigations should target objectives and optimization, although improving sample representation can also help underrepresented groups.The paper explicitly connects learning bias with disparities affecting underrepresented groups.
- 4.2 Designing Mitigations: Evaluation-bias mitigation can redefine evaluation metrics or data and should use granular, application-specific, intersectional analysis.Metric and subgroup choices should involve domain specialists and affected populations because stakeholders may value different error rates.
5 CONCLUSION
The framework locates downstream harm across the broader ML pipeline rather than attributing problems solely to data or model training. It supports precise communication and application-grounded analysis, mitigation, and anticipation of risks.
- Its vocabulary helps stakeholders communicate which type of bias they are addressing and clarify assumptions about the data and domain.The framework is intended for people who build, evaluate, use, or are affected by ML systems.
- Framing harm by pipeline stage encourages application-appropriate solutions instead of relying on broad, one-size-fits-all notions of fairness.Application knowledge and stakeholder engagement should inform how sources of harm are identified.
- The framework identifies distinct sources of downstream harm across data generation, model building, evaluation, and deployment.This broader view includes choices extending beyond model training and considers the ML pipeline’s iterative feedback loop.