Source-linked AI summary
Equality of Opportunity in Supervised Learning
Moritz Hardt, Eric Price, Nathan Srebro
TL;DR
The paper asks how supervised-learning systems can measure and address discrimination when black-box tests may not identify the underlying fairness interpretation. It proposes an oblivious fairness criterion with optimal post-processing, showing that fair classifiers can retain meaningful utility and create incentives for more accurate prediction.
Problem
The paper examines how to measure and ensure fairness when machine-learning decisions may perpetuate existing biases, including what black-box tests can identify.
Method
The paper develops an oblivious fairness criterion and shows how to obtain optimal non-discriminating classifiers by post-processing learned predictors.
Results
The framework is presented as more meaningful than demographic parity while achieving higher utility, and Bayes-optimal fair classifiers can be derived through post-processing.
Takeaways & Limitations
Equalized odds aligns fairness incentives with improving prediction accuracy across protected groups by encouraging features that directly capture the target.
Takeaways & Limitations
Satisfying or violating an oblivious fairness condition is not conclusive proof of fairness, which ultimately requires substantial domain-specific investigation.
Abstract
from arXiv · showhide
We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. In line with other studies, our notion is oblivious: it depends only on the joint statistics of the predictor, the target and the protected attribute, but not on interpretation of individualfeatures. We study the inherent limits of defining and identifying biases based on such oblivious measures, outlining what can and cannot be inferred from different oblivious tests. We illustrate our notion using a case study of FICO credit scores.
1 Introduction
The paper proposes an oblivious, actionable criterion for non-discrimination in supervised learning that preserves perfect accuracy and becomes easier to satisfy as prediction accuracy improves. It also provides post-processing, optimality, incentive, and impossibility results defining what this framework can achieve.
- Motivation and problem: Machine learning can encode discrimination in automated decisions, while existing methodology for avoiding discrimination against protected attributes remains insufficient.Ignoring protected attributes fails because other features can redundantly encode them.
- Framework: Unlike demographic parity, the criterion permits the perfectly accurate predictor bY = Y and becomes easier to achieve as prediction accuracy improves.This aligns fairness with the supervised-learning goal of constructing accurate predictors.
- Framework: The proposed criterion evaluates predictions using labeled samples from the joint distribution of features, protected attributes, and true outcomes.The protected attribute is observed in the labeled training data.
- Framework: The framework is oblivious, depending only on joint statistics of the target, predictions, and protected attribute rather than feature content or predictor form.This matches other oblivious tests, including demographic parity and analyses of common risk scores.
- Contributions: The framework shifts classification uncertainty from protected groups to decision makers, encouraging better features and data, but oblivious tests cannot distinguish different underlying dependency structures.The non-identifiability result limits what can be inferred about intuitive fairness notions from oblivious information alone.
- Contributions: A simple post-processing step converts any learned predictor into a criterion-satisfying classifier while minimizing utility loss.The paper also shows that post-processing a Bayes-optimal regressor yields the Bayes-optimal non-discriminating classifier and quantifies losses for non-optimal starting scores.
2 Equalized odds and equal opportunity
The section defines equalized odds as conditional independence of predictions and protected attributes given outcomes, requiring equal true- and false-positive rates across groups. It then introduces equal opportunity as a weaker advantaged-outcome-only constraint and frames both criteria as oblivious properties of joint distributions.
- Equalized odds: Equalized odds requires a predictor to be independent of the protected attribute conditional on the outcome.The criterion applies to targets and protected attributes in binary, multi-class, continuous, or structured spaces.
- Equalized odds: Equalized odds equalizes true positive rates for Y = 1 and false positive rates for Y = 0 across demographics.The perfect predictor bY = Y is always acceptable, while models performing well only for the majority are penalized.
- Equal opportunity: Equal opportunity relaxes equalized odds by requiring non-discrimination only within the advantaged outcome group.It is weaker than equalized odds and typically permits stronger utility.
- Scores and thresholds: For real-valued scores, equalized odds requires the score to be independent of the protected attribute given the outcome, and every thresholding of such a score also satisfies it.Scores can be converted into binary classifiers by thresholding, which changes the sensitivity–specificity trade-off.
- Obliviousness: Both non-discrimination notions are oblivious because they depend only on the joint distribution of the outcome, protected attribute, and predictor or score.In the binary case with reasonably balanced protected attributes and outcomes, this joint distribution is determined by 8 parameters estimable from samples.
3 Comparison with related work
The paper situates its fairness criterion against demographic parity and task-specific similarity-based notions, emphasizing demographic parity’s conceptual and utility limitations. It also contrasts its approach with calibration, logical rule-based methods, and representation-learning strategies for fairness.
- Demographic parity: Demographic parity is widely studied, including approaches that learn representations independent of the protected attribute while preserving feature information.Deep variational auto-encoders extend this representation-learning approach.
- Demographic parity: The “80% rule” differs from demographic parity by comparing probabilities as a ratio rather than additively and by permitting one-sided constraint violations.This passage describes the distinction between the two notions without introducing a different fairness criterion.
- Demographic parity: Demographic parity has serious conceptual limitations and can also fall short on utility; related work argues that fairness should be task-specific.Dwork et al. formalize fairness using a hypothetical similarity measure requiring similar individuals to receive similar outcome distributions.
- Calibration and equalized odds: Calibration within each group does not generally imply non-discrimination under a criterion equivalent to equalized odds, while equalized odds may compromise other desirable score properties.This comparison highlights that calibration alone is insufficient for the paper’s fairness measure.
- Rule-based approaches: Logical rule-based approaches to non-discrimination do not easily relate to the paper’s statistical approach.Pedreshi et al. and follow-up works are identified as early contributors to this line of research.
4 Achieving equalized odds and equality of opportunity
The section constructs equalized-odds and equal-opportunity predictors as post-learning, possibly randomized functions of a learned predictor or score and the protected attribute. For binary predictors, the optimal adjustment is obtained by a linear program, while score-based feasibility and performance are characterized geometrically through group-conditional ROC curves.
- Optimization: The adjustment is a post-learning step that avoids changing the existing training process and minimizes expected loss subject to equalized odds or equal opportunity.The objective is to design derived predictors minimizing Eℓ(eY,Y) under the chosen fairness condition.
- Derived predictors: A derived predictor depends only on the learned score or predictor and protected attribute, may add randomness, and is conditionally independent of features given them.Its construction uses the joint distribution of (R,A,Y) at training time, while prediction requires only (R,A).
- Fairness constraints: Equalized odds requires equal false-positive and true-positive rates across groups, whereas equality of opportunity requires equality only of true-positive rates.For binary protected attributes, these conditions are γ0(bY)=γ1(bY) and γ0(bY)2=γ1(bY)2, respectively.
- Binary predictors: For a binary learned predictor, every derived classifier corresponds exactly to group-specific rate pairs inside convex polytopes, enabling geometric optimization.Lemma 4.3 states that eY is derived iff γa(eY) belongs to Pa(bY) for each group.
- Binary predictors: The optimal equalized-odds adjustment is a four-variable linear program whose coefficients come from the joint distribution of the learned predictor, protected attribute, and target.The analogous equality-of-opportunity program uses the weaker constraint γ0(eY)2=γ1(eY)2.
- Real-valued scores: Equalized odds from a real-valued score requires selecting group-specific ROC points that coincide, but ROC curves may intersect only at trivial endpoints.The feasible equalized-odds rate region is the intersection of areas under the group-conditional ROC curves and above the main diagonal.
5 Bayes optimal predictors
The section characterizes optimal non-discriminating predictors through the Bayes optimal regressor and protected attribute, including equalized odds and equal opportunity. It also shows that approximating the unconstrained regressor can yield nearly optimal fair classifiers, with the loss controlled by conditional Kolmogorov distance.
- Bayes optimal predictors: Optimal equalized odds and equal opportunity predictors are therefore obtained by transforming the Bayes optimal regressor rather than using the full feature vector directly.The section frames Bayes optimal classifiers as threshold predictors of the regressor and extends the result to oblivious fairness properties.
- Bayes optimal predictors: An optimal predictor satisfying any oblivious property can be derived from the Bayes optimal regressor R and protected attribute A.Proposition 5.2 establishes optimality among all predictors satisfying the property; Corollary 5.3 applies this characterization to equalized odds and equal opportunity.
- Near optimality: A nearly optimal regressor can produce an equalized odds or equal opportunity predictor that remains nearly optimal among predictors satisfying the same fairness constraint.Theorem 5.6 states this for an arbitrary regressor and bounded loss, with the comparison made to a predictor derived from the Bayes optimal regressor.
- Near optimality: Conditional Kolmogorov distance must be small under each conditioning on A and Y, because this captures distance between the corresponding restricted ROC curves.Unconditional closeness is weaker; the relevant requirement concerns all four conditionings on protected-group membership and target value.
- Near optimality: The proof bounds the fair predictor’s loss gap by combining restricted-ROC proximity with the bounded-loss representation and Cauchy-Schwarz.The construction compares feasible points from the restricted ROC regions of the approximate and Bayes regressors before translating their distance into loss.
6 Oblivious identifiability of discrimination
The two scenarios can have fundamentally different fairness interpretations yet be indistinguishable by any oblivious test based on (Y,A,R). The examples also show that equalized-odds and conditional-frequency properties constrain correction and information differently.
- Scenario I: In Scenario I, R*=X1+X2 uses A or its surrogate, whereas eR=X2 uses the predictive feature independently of A given Y.R* is statistically more powerful but does not satisfy equalized odds; eR satisfies equalized odds.
- Scenario II: In Scenario II, R*=X3 uses a feature directly predictive of Y, while eR=X3−A is the equalized-odds score.Here X3 is correlated with A but predicts Y directly, making its use not naturally appear discriminatory.
- Oblivious identifiability: Any oblivious test based only on (Y,A,R) cannot distinguish the two scenarios, because their joint distributions over (Y,A,R*,eR) can be identical.Even access to features may not resolve the ambiguity without assigning feature meaning or making causal assumptions.
- Optimality: R* and eR are optimal unconstrained and equalized-odds scores, respectively, with optimal ROC curves and threshold-based classifiers for any loss function.The proposition establishes this jointly for both scenarios.
- Oblivious measures: eR satisfies equalized odds and has identical ROC curves, whereas R* violates equalized odds but has matching, non-identical ROC curves.Matching conditional ROC curves permit deterministic correction using the score and A, but do not imply that the original score is nondiscriminatory.
- Oblivious measures: R* has matching conditional frequencies, while eR does not; matching conditional frequencies are equivalent to Y being independent of A conditioned on R.This condition means the score contains all information about Y that A can provide.
7 Case study: FICO scores
The FICO case study compares fairness criteria for race-aware credit-loan decisions, showing that equal opportunity substantially preserves profit while addressing unequal access for qualified borrowers. It also argues that equal opportunity shifts the cost of poor classification toward the lender, whereas race blindness offers weak fairness guarantees.
- Observed disparities: At the common 620 cutoff, black non-defaulters are much less likely to qualify for loans than white or Asian non-defaulters.The cutoff corresponds to an 82% non-default rate, and the race-blind threshold violates the study’s fairness definitions.
- Profit and fairness trade-offs: At 82% non-default profitability, race blind achieves 99.3% of maximal profit, equal opportunity 92.8%, equalized odds 80.2%, and demographic parity 69.8%.The authors conclude that equal opportunity fairness costs less than a quarter what demographic parity costs.
- Profit and fairness trade-offs: Equal opportunity can exploit better accuracy among whites, while equalized odds requires classifying every group as poorly as the hardest group.This makes equalized odds cost over twice as much as equal opportunity in this case.
- Incentives and limitations: Equal opportunity shifts the cost of poorly classified groups to the company, strengthening incentives to improve accuracy, while race blindness can treat small groups poorly and even degenerate into max profit.Race blindness therefore provides a weak fairness guarantee and little incentive for the company to improve classification accuracy.
8 Conclusions
The proposed fairness measure addresses conceptual shortcomings of demographic parity while aligning fairness with the supervised-learning goal of building higher-accuracy classifiers. The conclusions emphasize reliable targets, cautious interpretation of oblivious measures, aligned incentives, efficient post-processing, and predictive affirmative action.
- 8 Conclusions: The fairness measure remedies demographic parity’s main conceptual shortcomings while supporting higher-accuracy classifiers.These are presented as the framework’s two central desiderata.
- Choose reliable target variables: Reliable target variables are essential because the framework requires observed outcomes, whose measurement may be unreliable or biased.Domain-specific scrutiny is required when defining and collecting labeled data.
- Measuring unfairness, rather than proving fairness: Satisfying or violating an oblivious fairness measure is not conclusive proof of fairness or unfairness, but a way to identify potential concerns for further scrutiny.The framework treats fairness assessment as measuring potential concerns rather than proving fairness.
- Incentives: Equalized odds creates incentives to collect features that more directly capture the target rather than its correlation with the protected attribute.The framework links better prediction under equalized odds to features more directly related to Y.
- When to use our post-processing step: The fairness notion can be implemented through simple, efficient post-processing using only aggregate information, potentially in a privacy-preserving manner.This contrasts with approaches requiring changes to complex training pipelines or access to raw data.
- Predictive affirmative action: Equalized odds can act as predictive affirmative action by shifting the optimally predictive score based on the protected attribute to compensate for score bias.The shift addresses the score’s stronger correlation with the protected attribute than the target label has, due to uncertainty.