Source-linked AI summary
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Shira Mitchell, Eric Potash, Solon Barocas, Alexander D'Amour, Kristian Lum
TL;DR
The paper addresses inconsistent terminology and notation in quantitative fairness research on prediction-based decisions, while questioning the choices and assumptions behind such systems. It organizes those choices, presents a consistent catalogue of fairness definitions and results, and concludes that metrics can reveal when prediction changes are insufficient and broader interventions are needed.
Problem
Quantitative fairness research uses inconsistent terminology and notation, while prediction-based systems embed choices and assumptions that can affect their fairness.
Method
The paper examines design choices and assumptions, develops a common setup and notation, and catalogues mathematical fairness definitions, tensions, and impossibility results.
Results
The paper shows that fairness metrics can reveal when changing prediction-based decision-making is insufficient to achieve different outcomes.
Takeaways & Limitations
Fairness analysis should support critical reflection on policy goals and procedures, while quantitative methods are used with care and humility rather than treated as sufficient on their own.
Takeaways & Limitations
Prediction-based analyses are bounded by the populations selected for decisions and by systematic measurement error that may differ across groups.
Abstract
from arXiv · showhide
A recent flurry of research activity has attempted to quantitatively define "fairness" for decisions based on statistical and machine learning (ML) predictions. The rapid growth of this new field has led to wildly inconsistent terminology and notation, presenting a serious challenge for cataloguing and comparing definitions. This paper attempts to bring much-needed order. First, we explicate the various choices and assumptions made---often implicitly---to justify the use of prediction-based decisions. Next, we show how such choices and assumptions can raise concerns about fairness and we present a notationally consistent catalogue of fairness definitions from the ML literature. In doing so, we offer a concise reference for thinking through the choices, assumptions, and fairness considerations of prediction-based decision systems.
1 Introduction
Prediction-based decisions are spreading across private and public domains, intensifying the need to clarify what fairness means in systems built from statistical and ML predictions. This paper organizes the field by examining its underlying choices and assumptions and cataloguing fairness definitions consistently.
- Prediction-based decisions are used in lending, hiring, advertising, pretrial detention, immigration detention, child maltreatment screening, public health, and welfare eligibility.
- Fairness in ML has developed across books, reports, surveys, textbooks, software, and earlier psychometric research on predictive educational tests.
- The paper treats fairness definitions as socially contextual mathematical formalisms rather than universally meaningful axioms.
- It aims to catalogue fairness-relevant design choices, model assumptions, evaluation metrics, and methods while identifying gaps between convenient formalism and broader social goals.
- The article grounds its conceptual and theoretical discussion in pretrial risk assessment and lending models, then develops notation and reviews fairness notions, causal frameworks, and impossibility results.
2 Choices, assumptions, and considerations
Prediction-based fairness depends on choices about goals, populations, decisions, data, models, and evaluation assumptions. These choices can encode or overlook social and statistical sources of unfairness.
- Policy design: Fairness analysis begins with the social objective, affected population, and available decision space, each of which shapes whether a model advances fairness.Conflicting goals may not be resolved by more data, and a model can advance an unacceptable overarching goal even when it satisfies mathematical fairness criteria.
- Policy design: Prediction systems often reduce broad social goals to a few predicted outcomes, focus on one decision, and represent progress through a scalar utility function.Pretrial systems may predict crime or court appearance while omitting outcomes such as wellbeing or caretaker status.
- Population: Evaluating only the model’s target population can overlook unfairness in how people entered that population, such as through targeted policing or discriminatory loan screening.The populations used in pretrial assessment and lending are selected from larger populations by mechanisms that may reflect objectionable social structures.
- Data: Training data raise distinct statistical and societal concerns: sampling and measurement errors can misrepresent outcomes, while data can encode objectionable social structures even when representative and accurate.Sampling bias can produce selective labels and biased utility estimates; differential measurement error can systematically disadvantage groups, while societal bias may require changes to data collection or lack a technical solution.
- Model: Predictive models reproduce unfair patterns in their data or problem formulation, while model form and covariate choices can change individual predictions and group disparities.A person’s predicted value may reverse relative to another person’s when the conditioned covariates change.
- Evaluation: Performance evaluations assume decisions can be assessed separately, including no interference, but decisions such as denying one family member’s loan may affect another’s outcome.This dependence conflicts with evaluating individual decisions independently.
2.3 Axes of fairness and protected groups
Fairness can be measured along one or more axes, but selecting protected groups is context-specific and may overlook discrimination affecting people at intersecting identities.
- Axes of fairness and protected groups: Fairness definitions choose the axis or axes along which fairness is measured, often using a sensitive attribute to distinguish advantaged and disadvantaged groups.Which attributes define these groups depends on legal, historical, and cultural context.
- Axes of fairness and protected groups: Most Fairness in ML work considers one sensitive attribute at a time, although discrimination can affect people at the intersection of two groups.Intersectional discrimination may remain hidden when each group is evaluated separately.
- Axes of fairness and protected groups: Commercial gender-classification systems found darker-skinned females were the most misclassified group, illustrating the importance of considering combinations of attributes.
3 Setup and notation
The paper formalizes prediction-based binary decisions by defining population features, decisions, outcomes, and scores that estimate unknown conditional probabilities.
- Setup and notation: The setup focuses on binary decisions made from predictions of binary outcomes in a finite population approximating a larger superpopulation.
- Setup and notation: Each person has decision-time covariates, potentially separated into sensitive variables and other variables, and receives a binary decision function of those covariates.The decision rule is δ: V → {0, 1}, with d_i = δ(v_i).
- Setup and notation: Prediction-based decisions first estimate an outcome that is unknown at decision time, then use a score intended to approximate its conditional probability.
- Setup and notation: The decision depends on the score alone: δ(v) = f(ψ(v)), where the score is learned from sample pairs of observed features and outcomes.
- Setup and notation: In pretrial assessment, predicted rearrest informs detention or release; in lending, predicted repayment informs loan decisions.
4 Flavors of fairness definitions from data alone
Oblivious fairness definitions evaluate parity using observed features, outcomes, scores, and decisions without invoking additional causal or contextual structure.
- Flavors of fairness definitions from data alone: Oblivious fairness definitions depend only on observed data and equate fairness with parities derived from distributions of features, outcomes, scores, and decisions.They are contrasted with non-oblivious definitions that use additional structure or context.
4.1 Unconstrained utility maximization and single-threshold fairness
Single-threshold fairness treats people with the same score equally and can coincide with utility maximization under specified score and utility assumptions, but its desirability is sensitive to those choices.
- Unconstrained utility maximization and single-threshold fairness: Single-threshold fairness treats individuals with the same score equally regardless of group membership.
- Unconstrained utility maximization and single-threshold fairness: Utility-maximizing rules for certain scores and utility functions use one threshold, applying different decisions below and above it.
- Unconstrained utility maximization and single-threshold fairness: A score must closely approximate the true conditional probability for single-threshold optimality to hold; for loan repayment, higher scores can receive loans while lower scores are denied.
- Unconstrained utility maximization and single-threshold fairness: Because the optimal threshold depends only on the utility function, the rule maximizes utility within each subgroup under the stated assumptions.
- Unconstrained utility maximization and single-threshold fairness: Single-threshold desirability depends on scoring, utility, data, measurement, and modeling choices, and the optimality results require separate, symmetric, and simultaneous utility assumptions.
- Unconstrained utility maximization and single-threshold fairness: These sensitivities motivate fairness notions external to utility maximization that can be evaluated without taking scoring models or utility functions for granted.
4.2 Equal prediction measures
Equal prediction measures compare how predictions or decisions relate to outcomes across sensitive groups. These definitions can condition on outcomes or decisions, and score-based measures connect to confusion-matrix notions.
- Equal accuracy requires P[D = Y |A = a] = P[D = Y |A = a′] when false positives and false negatives have equal cost.
- Confusion matrices summarize matches and mismatches between outcomes Y and decisions D, with margins representing conditioning subsets.
- Conditional on outcome: Equality of false positive rates requires equal P[D = 1|Y = 0, A = a] across groups; analogous definitions apply to other confusion-matrix cells.
- Conditional on outcome: Conditioning on Y = 0 yields equality of false positive or true negative rates, while conditioning on Y = 1 yields equality of true positive or false negative rates.
- Outcome-conditioned definitions treat people with the same outcome similarly, whereas decision-conditioned definitions group people by observed predictions or decisions.
- Score-based measures include AUC parity, balance for each outcome class, and calibration within groups, with calibration closely related to predictive-value equality and sufficiency.
4.3 Equal decision measures
Equal decision measures assess fairness using decisions alone or within strata, motivated by allocation concerns and possible measurement error in outcomes. Demographic parity requires equal decision rates across groups.
- Decision-only fairness can examine benefit and harm allocation when one decision is preferred regardless of outcome Y.
- Measurement error, especially differential error across groups, can motivate fairness constraints that do not rely on Y.
- Demographic parity requires equal decision rates across groups, expressed as unconditional independence D ⊥A or S ⊥A.
- In lending, the consequences of decisions can depend on repayment ability and loan terms, so decision-only assessments do not exhaust relevant effects.
- Conditional demographic parity requires D ⊥A | Data; choosing Data = Y yields separation, while choosing insensitive variables X yields unawareness or treatment parity.
4.4 Impossibilities
Fairness definitions are generally incompatible when outcomes differ across groups and prediction is imperfect. The COMPAS debate illustrates how different valid measures can support opposing assessments of the same system.
- Fairness definitions formalize intuitive goals but are not mathematically or morally compatible in general.
- The COMPAS debate: ProPublica reported that COMPAS violated equal false positive rates by race, while Northpointe reported equal positive predictive values and calibration within groups.
- Separation and sufficiency: Separation and sufficiency together require either equal outcome and score distributions across groups or a zero-probability event.
- Separation and sufficiency: In the COMPAS context, equal predictive values and equal false positive rates can coexist only with equal recidivism rates across races or groups that never experience some outcomes.
- Separation and sufficiency: Balance for both classes and calibration require either equal base rates or perfect prediction, conditions not met in ProPublica’s COMPAS data.
- Incompatibilities with demographic parity: Demographic parity is likewise incompatible with separation or sufficiency unless base rates are equal, and utility-maximizing constrained rules generally use sensitive variables.
5 Flavors of fairness definitions incorporating additional context
Context-sensitive fairness definitions add metrics, construct-space assumptions, or causal models to connect mathematical criteria with social goals. These approaches expand what can be represented but introduce metric, counterfactual, and identification challenges.
- Metric fairness requires similar individuals, according to a metric m, to receive similar decisions: |δ(v) − δ(v′)| ≤m(v, v′).
- The metric may be defined over individuals or classifier inputs, and is intended to capture the ground-truth construct on which decisions should be based.
- WYSIWYG assumes observed-space distances approximate construct-space distances, while WAE assumes groups are close on average in construct space.
- Metric fairness is difficult to implement because defining a similarity metric is challenging, especially in high dimensions.
- Causal definitions: Causal fairness uses counterfactual decisions under different sensitive attributes, ranging from individual counterfactual fairness to conditional counterfactual fairness and counterfactual parity.
- Causal definitions: Causal graphs distinguish direct, indirect, and back-door paths; causal fairness definitions may permit back-door association even though it can contribute to observed disparity.
- Causal definitions: Some causal fairness claims require ignorability, and no direct effect fairness cannot be confirmed without direct access to model internals.
- Causal definitions: Causal reasoning can also guide earlier interventions to reduce disparities and improve outcomes, rather than only compensating at decision time.
6 Ways Forward
The paper sketches ways to address the choices and assumptions underlying prediction-based decisions directly, rather than only constraining decision functions with fairness definitions. It recommends clarifying goals, populations, interventions, outcomes, causal effects, data collection, and model evaluation.
- The paper does not prescribe what to do with any particular fairness definition; its dominant literature instead treats fair decision-making as constrained optimization.
- Starting with explicit goals, intervention levels, eligible populations, and broader intervention options can improve accountability and mitigate fairness concerns.
- Prior information can help specify realistic utility functions rather than assuming benefits and harms are constant across decisions.
- Causal methods can estimate decision effects and examine interference and heterogeneous effects instead of assuming one potential outcome is known or effects are constant.
- Documenting data collection and model performance, including subgroup performance, supports modeling under data-collection conditions and more effective evaluation.
7 Conclusion
The paper concludes that fairness metrics should be used carefully: they can reveal problems and clarify assumptions, but cannot by themselves resolve the social conditions or goals underlying prediction-based decisions. This reflection can motivate interventions and broader policy choices beyond prediction.
- Fairness concerns also apply to human decisions when those decisions rely on predictions, not only to automated predictive models.
- Formal fairness metrics can show when changing prediction-based decisions is insufficient and interventions are needed to produce different outcomes.
- Quantitative fairness measures can narrow attention toward adjusting decisions instead of addressing structural conditions sustaining inequality, despite also clarifying assumptions.
- Used with care and humility, fairness research can reveal problems and support progress toward policy goals, although mathematical formalism cannot solve them alone.
- The reviewed literature supports critical reflection on how social goals are chosen and on the procedures used to realize them.