Source-linked AI summary
On Fairness and Calibration
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, Kilian Q. Weinberger
TL;DR
Fairness constraints seek to reduce error disparities across groups, but the paper studies their tension with calibrated probability estimates. It characterizes when relaxed parity is feasible, shows that the optimal solution uses randomized information withholding, and evaluates these findings empirically.
Problem
Calibration and Equalized Odds are generally incompatible, leaving limited understanding of feasible relaxed fairness guarantees that preserve calibrated probabilities.
Method
The paper characterizes calibrated error-rate constraints geometrically and derives a post-processing algorithm that equalizes feasible group costs while preserving calibration.
Results
Calibration and relaxed error-rate constraints are often mutually incompatible; when feasible, the unique optimal solution randomly withholds predictive information from some inputs.
Takeaways & Limitations
Under required calibration, no lower-error solution can be achieved than the paper’s information-withholding solution, making joint fairness and calibration difficult in practice.
Takeaways & Limitations
The proposed solution can make a classifier strictly worse for one group and withhold information randomly, creating inequity within that group.
Abstract
from arXiv · showhide
The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be "fair." In this paper, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.
1 Introduction
The paper examines how calibration interacts with fairness constraints designed to prevent group-disparate errors. It argues that even relaxed error-rate parity can conflict with calibration, and reports empirical support for choosing between these goals in practice.
- Motivation: Errors in sensitive domains can systematically discriminate against particular groups, motivating formal frameworks for fairness in classification.The introduction discusses criminal justice, online advertising, and medical testing as examples of sensitive applications.
- Calibration: Calibration requires predicted probability p to correspond to a p fraction of positive instances, including within each protected group.Groupwise calibration gives probability estimates the same meaning across groups.
- Research problem: Prior work shows that calibration and Equalized Odds are generally incompatible except in highly constrained cases.The paper frames its contribution as investigating feasible relaxations of these guarantees.
- Contributions: The paper characterizes when weighted group error costs can match under calibration and gives a post-processing algorithm based on withholding predictive information for randomly chosen inputs.When feasible, the calibrated relaxation has a unique optimal solution obtainable from existing classifiers.
- Empirical findings: Experiments on several datasets support the conclusion that calibration and error-rate constraints are often mutually incompatible, making it potentially advisable to choose one goal.The paper compares calibrated non-discrimination with uncalibrated Equalized Odds.
3 Problem Setup
The paper formalizes probabilistic Equalized Odds and calibration for two groups, then uses geometric relationships between false-positive and false-negative rates to characterize their conflict. With different base rates, both conditions can hold only for perfect predictors.
- Problem setup: The framework considers two disjoint groups with potentially different base rates and group-specific probabilistic classifiers outputting positive-class probabilities.Each classifier maps inputs to values in [0,1].
- Problem setup: Probabilistic Equalized Odds requires equal generalized false-positive and false-negative rates across the two groups.The generalized rates extend standard error rates to probabilistic classifiers.
- Problem setup: Groupwise calibration requires predicted probabilities to carry the same frequency interpretation within each group.For example, among people assigned probability 0.6, the expected positive fraction is 60%.
- Geometric characterization: Trivial classifiers output a constant c and lie on cfp(h) + cfn(h) = 1, while classifiers better than random lie below this diagonal.The geometric representation uses generalized false-positive and false-negative rates as its axes.
- Geometric characterization: Calibrated classifiers for a group lie on a line determined by its base rate, with perfect and trivial calibrated classifiers at opposite endpoints.The trivial calibrated classifier outputs the group base rate, and better calibrated classifiers lie closer to the origin.
- Impossibility result: When group base rates differ, calibration and Equalized Odds hold simultaneously if and only if both classifiers are perfect predictors.Geometrically, the calibrated group lines intersect at the origin, corresponding to perfect classification.
4 Relaxing Equalized Odds to Preserve Calibration
The paper relaxes Equalized Odds to a single equal-cost constraint while preserving calibration, characterizes when this is feasible, and gives an optimal post-processing solution. With unequal base rates, multiple distinct equal-cost constraints remain compatible with calibration only for perfect classifiers.
- Relaxed Equalized Odds: The relaxed framework equalizes a weighted combination of generalized false-positive and false-negative rates while requiring calibration for each group.The cost is g_t(h_t) = a_t cfp(h_t) + b_t cfn(h_t), with nonnegative group-specific weights.
- Relaxed Equalized Odds: For calibrated classifiers, increasing cost strictly increases both generalized false-positive and false-negative rates, so cost can proxy either error rate.The cost curves move farther from the origin as cost increases in the generalized error plane.
- Feasibility and Optimality: A calibrated equal-cost solution is feasible if and only if group G1’s cost does not exceed group G2’s trivial calibrated classifier cost, g1(h1) ≤ g2(hµ2).Under the paper’s assumptions, the trivial classifier outputs the group base rate and achieves maximum cost.
- Implications: The resulting solution is optimal under the assumptions, but it can worsen one group and withhold information randomly from a population subset.The paper identifies these as primary objections and notes that the resulting performance loss may be significant.
- Multiple Constraints: With unequal base rates, satisfying two distinct equal-cost constraints under calibration requires both classifiers to be perfect.The approximate version likewise forces approximately zero generalized false-positive and false-negative rates.
5 Experiments
Experiments on income, health, and criminal-recidivism datasets compare calibrated equal-cost relaxations with Equalized Odds. The results show that feasibility and performance costs vary by task, while calibration can trade one disparity for another.
- Experimental Design: Across income, health, and criminal-recidivism tasks, the experiments evaluate calibrated equal-cost constraints against Equalized Odds.The study chooses task-appropriate cost functions and compares calibrated and uncalibrated frameworks.
- Income Prediction: Income prediction feasibly equalizes generalized false-negative rates while maintaining calibration, but increases disparity in false-positive rates and overall errors.The calibrated framework trades one notion of disparity for another in this experiment.
- Health Prediction: The health experiment is the only case with little additional cost under the calibrated framework because the original classifiers already lie approximately on the same cost curve.The cost weights are rfp = 1 and rfn = 3.
- Criminal Recidivism Prediction: In criminal recidivism, Equalized Odds matches error rates while sacrificing calibration, whereas the calibrated relaxation is infeasible under the assumption that the original classifiers cannot be improved.The experiment modifies COMPAS predictions and also evaluates training-time Equalized Odds constraints.
6 Discussion and Conclusion
The discussion finds that calibrated relaxed Equalized Odds is sometimes feasible and sometimes incompatible, with costs amplified by base-rate differences and by limited room to interpolate toward trivial classifiers. The conclusion cautions that the feasible solution relies on randomly withholding predictive information.
- Discussion: Calibration and relaxed Equalized Odds are compatible in some cases but incompatible in others.Infeasibility occurs when the best discriminatory classifiers are not far from the trivial classifiers.
- Discussion: When feasible, differing base rates amplify the penalty of equalizing cost, and equalizing one error rate necessarily increases disparity in the other.The paper notes that this trade-off may be tolerable in income prediction but problematic in criminal justice.
- Conclusion: The paper concludes that maintaining calibration and cost parity is desirable but often difficult in practice.Its algorithm finds the unique feasible solution when one exists.
- Conclusion: The feasible algorithm randomly replaces predictions from the better classifier with the trivial base rate, which may be difficult to accept for consequential individual decisions.The conclusion emphasizes that fairness in expectation can still involve random information withholding for an individual case.
Supplementary Information for: On Fairness and
The supplementary material introduces approximate calibration for subsequent proofs and supplies supporting notation, extended approximate results, and experiment details.
- Approximate Calibration: The supplement introduces an approximate calibration notion for use in later proofs.It defines the calibration gap ε(h_t) with respect to group G_t.
- Approximate Results: The supplementary results degrade smoothly as the calibration condition is relaxed.The supplement focuses largely on approximate versions of the paper’s major findings.
- Supporting Material: The supplementary material also provides extended experiment details and clarifies notation for group probabilities and expectations.It uses P_Gt and E_Gt as shorthand for distributions over (x,y) sampled from G_t.
S1 Linearity of Calibrated Classifiers
The section characterizes perfectly calibrated classifiers for each group geometrically: they lie on a line in the generalized false-positive/false-negative plane, with slope determined by the group’s base rate.
- Proof: The proof derives lower bounds for generalized false-positive and false-negative rates from calibration inequalities and combines them to complete the result.The derivation uses (S3) and (S4), applies analogous steps to cfp(ht) and cfn(ht), and multiplies the resulting inequalities by µt.
- Perfect calibration: Perfectly calibrated classifiers for group Gt are collected in the set Ht.The corollary introduces Ht as the set of perfectly calibrated classifiers for Gt.
- Geometric characterization: For group Gt, all perfectly calibrated classifiers lie on a line in the generalized false-positive/false-negative plane.This is stated as the geometric consequence of the preceding calibration conditions.
- Geometric characterization: The line’s slope is uniquely determined by group Gt’s base rate µt.The base rate appears alongside the generalized false-positive and false-negative rates in the characterization.
S2 Cost Functions
The paper analyzes weighted error costs over calibrated classifiers. The calibrated classifier that outputs a group’s base rate for every sample maximizes any cost function of the specified form, and randomized interpolation makes cost linear.
- For cost functions gt(ht) = atcfp(ht) + btcfn(ht), the trivial calibrated classifier hµt has maximum cost.The coefficients at and bt are non-negative constants.
- The maximum-cost calibrated classifier outputs the same probability for every sample, with that constant equal to the group base rate µt.
- Randomized interpolation between a classifier and hµt yields a cost equal to the corresponding weighted average of their costs.The interpolation parameter α determines the mixture weights 1 − α and α.
S3 Relationship Between Cost and Error
For approximately calibrated classifiers, reducing a weighted error cost is tightly connected to reducing generalized false-positive and false-negative rates. For perfectly calibrated classifiers, cost ordering implies a trade-off between these two error rates.
- Reducing any weighted cost gt(ht) approximately improves generalized false-positive and false-negative rates for approximately calibrated classifiers.
- The cost comparison is analyzed by considering whether the alternative classifier has a lower generalized false-positive rate or a lower generalized false-negative rate.
- Among perfectly calibrated classifiers, lower weighted cost cannot be achieved without worsening at least one generalized error rate.The result compares classifiers through their costs and generalized false-positive and false-negative rates.
S4 Proof of Algorithm 1 Optimality and Approximate Optimality
Algorithm 1 produces optimal non-discriminatory classifiers under exact calibration and near-optimal classifiers under approximate calibration. Its interpolation step preserves or improves calibration while achieving the equal-cost constraint.
- Algorithm 1 is near-optimal with respect to both generalized error rates among approximately calibrated classifiers satisfying the equal-cost constraint.
- The interpolated classifier’s calibration error satisfies ϵ(˜h2) ≤ (1 − α)ϵ(h2).Here α ∈ [0, 1] is the interpolation parameter.
- Algorithm 1 produces classifiers satisfying perfect calibration and the equal-cost constraint with the lowest possible generalized false-positive and false-negative rates.This is the theorem’s exact optimality claim.
S5 Proof of Impossibility and Approximate Impossibility
The paper proves that calibrated classifiers cannot satisfy multiple distinct equal-cost constraints unless they are perfect, and shows that approximate satisfaction forces approximately perfect classification.
- Exact Impossibility Theorem: Multiple distinct equal-cost constraints plus calibration are impossible unless both group classifiers are perfect.The result assumes unequal group base rates and distinct cost functions.
- Exact Impossibility Theorem: Four linearly independent constraints on four error-rate variables force the classifiers to be perfect.Two constraints encode calibration and at least two encode distinct equal-cost requirements.
- Approximate Impossibility Theorem: The approximate impossibility proof represents calibration and equal-cost requirements as a linear system A⃗q = 0, or as bounded residuals under approximate constraints.The vector q contains the two groups’ generalized false-positive and false-negative rates.
- Approximate Impossibility Theorem: Approximate calibration and multiple distinct equal-cost constraints are possible only when generalized false-positive and false-negative rates are approximately zero.The theorem assumes rational constraint coefficients with common denominator D and bounded entries M.
- Approximate Impossibility Theorem: Theorem 9 provides a smooth degradation result rather than a tight quantitative bound.The bound depends on D and M through a constant L.
S6 Details on Experiments
The experiments use post-processing and constrained learning to construct Equalized Odds and calibrated-relaxation baselines across income, health, and COMPAS-related settings.
- Post-processing for Equalized Odds: Equalized Odds post-processing randomly flips some original predictions to satisfy false-positive and false-negative constraints.The flipping probabilities are group-specific and can be selected through a linear program.
- Post-processing for Equalized Odds: The post-processed classifiers use Bernoulli random variables whose expectations determine positive-to-negative and negative-to-positive flipping rates.When original outputs are binary, this probabilistic formulation reduces to the earlier post-processing method.
- Constrained-learning for Equalized Odds: Constrained learning achieves Equalized Odds during training by optimizing a logistic classifier under non-convex fairness constraints.The constraints can be formulated as a disciplined convex-concave program and solved in many instances.
- Training Procedure for Income Prediction: Income experiments train random forest, multilayer perceptron, and RBF-kernel SVM models, using Platt scaling to calibrate the SVM.The data allocation reserves 10% for tuning and post-processing and another 10% for final evaluation.
- Training Procedure for Health Prediction: Health experiments train random forest and linear SVM models, while the Equalized Odds baseline uses constrained logistic learning.Both Equalized Odds and calibrated-relaxation classifiers are post-processed from the original COMPAS classifier.