PaperlayerView exact revision on arXiv

Source-linked AI summary

The Measure and Mismeasure of Fairness

Sam Corbett-Davies, Johann D. Gaebler, Hamed Nilforoshan, Ravi Shroff, Sharad Goel

arXiv:1808.00023v3cs.CY

TL;DR

Fair machine learning has accumulated formal fairness criteria, but their statistical and policy consequences require closer examination. The paper categorizes these criteria, analyzes their implications, and finds that they often produce dominated policies, motivating context-specific algorithm design aligned with policy goals.

  • Problem

    As algorithmic decisions become widespread, formal fairness criteria have proliferated amid concerns that algorithms may encode and entrench human biases.

  • Method

    The paper surveys and taxonomizes fairness definitions, develops a consequentialist framework, and applies formal analysis to equitable decision-making in settings including college admissions.

  • Results

    Both major families of fairness definitions typically produce strongly Pareto dominated policies, including admissions policies with lower diversity and academic preparedness than tailored alternatives.

  • Takeaways & Limitations

    Equitable algorithm design should address substantive trade-offs and real-world consequences, recognizing algorithms as encoding policy choices and separating prediction from intervention design.

  • Takeaways & Limitations

    The paper’s counterfactual fairness results rely on conditions that do not hold in every setting, including assumptions about how race affects test scores.

Abstract

from arXiv · show

The field of fair machine learning aims to ensure that decisions guided by algorithms are equitable. Over the last decade, several formal, mathematical definitions of fairness have gained prominence. Here we first assemble and categorize these definitions into two broad families: (1) those that constrain the effects of decisions on disparities; and (2) those that constrain the effects of legally protected characteristics, like race and gender, on decisions. We then show, analytically and empirically, that both families of definitions typically result in strongly Pareto dominated decision policies. For example, in the case of college admissions, adhering to popular formal conceptions of fairness would simultaneously result in lower student-body diversity and a less academically prepared class, relative to what one could achieve by explicitly tailoring admissions policies to achieve desired outcomes. In this sense, requiring that these fairness definitions hold can, perversely, harm the very groups they were designed to protect. In contrast to axiomatic notions of fairness, we argue that the equitable design of algorithms requires grappling with their context-specific consequences, akin to the equitable design of policy. We conclude by listing several open challenges in fair machine learning and offering strategies to ensure algorithms are better aligned with policy goals.

1. Introduction

The paper organizes formal fairness definitions into two families and argues that both can produce decision policies that conflict with substantive policy goals. It advocates treating algorithm design as consequential policy design rather than relying solely on fairness axioms.

  • Common fairness criteria almost always lead to strongly Pareto dominated policies for natural utility families valuing both academic preparedness and student-body diversity.A policy is strongly Pareto dominated when another feasible policy is preferred under every utility function in the relevant family.
  • In college admissions, several popular fairness conceptions would yield both lower diversity and a less academically prepared class than policies tailored to desired outcomes.
  • One prominent fairness definition would require admitting all applicants with equal probability, regardless of academic qualifications or group membership.
  • Formal fairness definitions are organized into two broad families: those constraining decisions’ effects on disparities and those constraining protected attributes’ effects on decisions.
  • The paper’s consequentialist framework views algorithmic fairness as a policy objective and exposes statistical and normative limitations in popular formal definitions.

2. Mathematical Definitions of Fairness

The paper organizes formal fairness criteria into definitions limiting decisions’ effects on disparities and definitions limiting protected attributes’ effects on decisions. It then introduces these criteria within mathematical decision settings involving outcomes, causal effects, and domain-specific trade-offs.

  • Formal fairness definitions fall into two families: those constraining decisions’ effects on disparities and those constraining protected attributes’ effects on decisions.The second family includes attributes such as race and gender.
  • 2.1 Formal setting: The formal setting represents individuals by observed covariates X, protected attributes A, randomized binary decisions D, a decision budget b, and binary outcomes Y.When decisions causally affect outcomes, the framework uses potential outcomes Y(0) and Y(1).
  • 2.1 Formal setting: The paper studies diabetes screening and college admissions, whose policy evaluations differ because stakeholders may agree less about competing priorities in admissions.Admissions may require balancing academic preparedness and class diversity under a limited number of places.
  • 2.2 Limiting the Effect of Decisions on Disparities: Classification parity requires a chosen confusion-matrix measure, such as false-positive or false-negative rates, to be equal across protected groups.The paper also includes positive-decision proportions and AUC among the measures considered under classification parity.
  • 2.2 Limiting the Effect of Decisions on Disparities: Causal fairness definitions constrain relationships among potential outcomes, protected attributes, and decisions, including counterfactual predictive parity, counterfactual equalized odds, and conditional principal fairness.In admissions, counterfactual predictive parity compares degree attainment under acceptance among rejected applicants across race groups.
  • 2.3 Limiting the Effect of Attributes on Decisions: Counterfactual and path-specific fairness assess direct and indirect effects of protected attributes on decisions, with path-specific fairness permitting influence along selected causal paths.The framework requires a causal DAG; the admissions example considers race’s influence through educational opportunities and test scores.

3. Equitable Decisions in the Absence of Externalities

Threshold policies can optimize individual and aggregate utility, yet generally conflict with classification parity when groups have different risk distributions. Blinding protected attributes can also reduce utility and miscalibrate risk, while some fairness constraints impose highly restrictive policies.

  • An optimal threshold policy selects individuals with risk above t, maximizing both individual and aggregate utility.The policy admits or screens individuals when r(x) > t and rejects them when r(x) ≤ t.
  • Demographic-parity adjustments can harm both groups by missing relatively high-risk Asian Americans while screening relatively low-risk White Americans.The threshold alteration departs from selecting patients by highest risk.
  • 79% versus 67%: Asian Americans have the higher false positive rate than White Americans at the 1.5% diabetes-screening threshold.Equalizing the rates would require screening some relatively low-risk people and not screening others at relatively high risk.
  • Classification parity generally conflicts with utility-maximizing thresholds, because small differences in group risk distributions can violate demographic parity or equalized false positive rates.Theorem 9 states this occurs for almost every collection of group-specific risk distributions with densities on [0, 1].
  • Causally fair policies can impose severe structural restrictions: under mild conditions, college admissions must give every applicant the same admission probability.The authors characterize this as a potentially unintended consequence of the formalism.
  • Blinding and protected-attribute-specific tools each involve context-dependent trade-offs, including possible harms to individuals, communities, trust, or social legitimacy.The paper notes that legal, political, and social benefits of blinding may outweigh its costs in some settings, while race- or gender-specific tools can create separate harms.
  • Removing protected attributes can reduce utility when race or gender adds predictive value, affecting both individuals and aggregate outcomes.Theorem 10 formalizes this under positive conditional variance of protected-attribute-informed risk given blinded risk.

4. Equitable Decisions in the Presence of Externalities

The paper examines how formal fairness constraints interact with competing objectives in decision problems such as college admissions. It shows that several prominent constraints can exclude policies on the Pareto frontier, sometimes producing outcomes that are worse across multiple dimensions.

  • Competing objectives: The analysis models admissions as balancing academic preparedness against student-body diversity under a limited admissions budget.The policy determines each applicant’s probability of admission, while the committee values both academic preparedness and representation of a target group.
  • Geometric analysis: For any feasible policy not on the frontier, another feasible policy exists that achieves both greater diversity and a higher academic index.Such an alternative is preferred by every committee member under the assumed objectives.
  • Formal results: Under counterfactual or path-specific fairness in the admissions setting, the only feasible policies may admit all students with a constant probability regardless of covariates.These policies form a single line segment in the illustrative geometry and can be strongly Pareto dominated.
  • Formal results: For almost every distribution under stated regularity conditions, policies satisfying counterfactual equalized odds, conditional principal fairness, or path-specific fairness are Pareto dominated.The corresponding results do not extend analogously to every fairness definition; counterfactual predictive parity, equalized false positive rates, and demographic parity are treated separately.
  • Implications and scope: The paper argues that equitable algorithm design should explicitly confront context-specific trade-offs rather than rely solely on formal fairness axioms.It notes that policies satisfying counterfactual predictive parity are also Pareto dominated in some common cases, including admissions when the target group has lower average graduation rates.

5. A Path Forward

The paper argues that equitable algorithm design should evaluate context-specific consequences rather than rely primarily on formal fairness axioms. It highlights trade-offs involving calibration, prediction targets, data quality, and resource allocation, while advocating consequentialist policy analysis.

  • Formal fairness criteria can produce policies that are both less diverse and less academically prepared than feasible alternatives in college admissions.
  • Consequentialist design weighs the costs and benefits of alternative decision policies against context-specific objectives rather than automatically enforcing fairness constraints.
  • Assessing calibration: Calibration is necessary for comparable risk meanings across groups, but calibrated scores can still encode intentional discrimination or reflect ignored predictive information.
  • Designing data collection strategies: Risk-assessment algorithms require suitable, representative training data, yet collecting more complete data can be costly and imperfect data may remain preferable to available alternatives.

6. Conclusion

The paper argues that fairness should be evaluated through real-world consequences rather than axiomatic criteria alone. It recommends separating risk estimation from intervention design while addressing measurement, preference, and equilibrium challenges.

  • Statistical risk assessments can improve human decisions but may also exacerbate historical inequities if developed without care.
  • A consequentialist approach grounds fairness discussions in real-world quantities such as public safety, incarceration, and other policy outcomes.
  • Risk assessment should be decoupled from intervention design because predictions estimate risks but do not determine policies or thresholds.
  • Fairness criteria can constrain algorithms away from the best available risk estimates, implicitly conflating statistical and policy problems.
  • Open challenges include measurement error, sample bias, externalities, equilibrium effects, and aggregating preferences across competing algorithms.

B. Constructing Causally Fair Policies

The paper formulates utility-maximizing decision policies under causal fairness constraints and shows that the resulting problems are tractable linear programs or collections of linear programs. These constructions characterize feasible policies and outcomes for several causal fairness definitions.

  • Utility maximization under causal fairness, outcome, and budget constraints is posed over decision policies satisfying a selected fairness definition.
  • The optimization is efficiently solvable as one linear program for four causal fairness definitions and as a series of linear programs for counterfactual predictive parity.
  • O(n) variables and constraints suffice for utility-maximizing policies under counterfactual equalized odds or conditional principal fairness on finite support of size n.
  • O(n) variables and constraints also suffice for path-specific fairness, including counterfactual fairness, when the relevant finite-support distribution has size n.
  • Counterfactual predictive parity is handled by a family of linear programs indexed by feasible conditional outcome distributions.

D. Proof of Theorem 10

The proof establishes that certain threshold events are not measurable with respect to the available information. It uses conditional-variance bounds and positive-probability neighborhoods around a threshold to derive the contradiction.

  • Lemma 22 bounds conditional second moments of a bounded nonnegative variable and, via Jensen’s inequality, its conditional variance.
  • The lemma implies that a bounded random variable’s minimum cannot be too close to its conditional mean, enabling the theorem’s proof.
  • The proof seeks to show that no event lying between {r(X) ≥ t} and {r(X) > t} is measurable with respect to π(X).
  • For covariates with ρ(X) just above t, both r(X) < t and r(X) > t occur with positive conditional probability.
  • Therefore, the conditional expectation of the threshold indicator given π(X) is neither 0 nor 1 with positive probability, contradicting measurability.

E. Proof of Proposition 15

The proof shows that, under utilities consistent modulo protected attributes, feasible policies outside the multiple-threshold class are strongly Pareto dominated. It constructs utility-improving alternatives while preserving feasibility and extends the result to non-negative-threshold and budget-exhaustion conditions.

  • A multiple threshold policy is defined by group-specific thresholds, assigning higher decision probabilities above each group’s threshold and lower probabilities below it.
  • Any feasible policy that is not almost surely a multiple threshold policy with non-negative thresholds is strongly Pareto dominated.
  • For a policy that is not a multiple threshold policy, reallocating decision probability from below a utility threshold to above it yields a feasible policy with greater utility.
  • Consistency modulo the protected attribute transfers the utility improvement across the entire utility collection, producing strong Pareto dominance.
  • A multiple threshold policy lacking a non-negative-threshold representation is dominated by removing decisions assigned where utility is negative.
  • A feasible policy that is not budget-exhausting and is a multiple threshold policy with non-negative thresholds is also strongly Pareto dominated.

F. Prevalence and the Proof of Theorem 17

This section generalizes null-set reasoning to infinite-dimensional spaces through shy and prevalent sets, then applies relative shyness to distributions relevant to fairness. Theorem 17’s proof identifies fairness-compatible, non-dominated policies as a shy subset.

  • Prevalence and shyness: Prevalence generalizes full-measure sets, while shy sets generalize null sets in infinite-dimensional spaces where no suitable Lebesgue measure exists.In finite-dimensional spaces, prevalence is equivalent to having a complement of Lebesgue measure zero.
  • Prevalence and shyness: A set is prevalent when its complement is shy; prevalent sets are dense, closed under countable intersections, and translation invariant.These properties also extend to suitable convex subsets through relative prevalence.
  • Relative shyness: Relative shyness defines smallness within a convex, completely metrizable subset, preserving the finite-dimensional relationship between prevalence and Lebesgue measure.When the convex subset has non-empty interior in R^n, relative prevalence is equivalent to a complement of Lebesgue measure zero.
  • Probes: A k-dimensional probe witnesses shyness when every translate of the target set is Lebesgue-null in the probe, while a translate of the ambient convex set has positive measure.The probe provides finite-dimensional cross-sections for detecting small sets in the surrounding space.
  • Application to Theorem 17: The proof applies relative shyness to U-fine probability distributions, defining E as those distributions admitting a relevant fairness-satisfying policy that is not strongly Pareto dominated.The ambient space consists of totally bounded Borel measures, and the proof constructs a probe within that framework.

F.3.2 Convexity, complete metrizability, and universal measurability

This section establishes the regular U-fine probability-measure set as a convex, completely metrizable space and proves that the fairness-compatible, non-dominated set is universally measurable.

  • Convexity and complete metrizability: The regular U-fine probability-measure set Q is convex and completely metrizable.Convexity follows by direct calculation, while complete metrizability follows from Q being closed in a complete Banach space.
  • Universal measurability: The proof also establishes universal measurability using the fact that closed sets are Borel and Borel sets are universally measurable.The argument first proves closure, then invokes these standard measurability implications.
  • Universal measurability: The closure proof handles conditional expectations across converging measures by bounding differences between conditional-expectation versions.A technical lemma supplies a coarse bound in terms of total variation, with boundedness controlling the resulting error.
  • Universal measurability: The set E of distributions admitting a relevant fairness-satisfying policy that is not strongly Pareto dominated is closed and therefore universally measurable.The statement covers counterfactual equalized odds, conditional principal fairness, and path-specific fairness.
  • Universal measurability: For counterfactual equalized odds, convergent sequences of qualifying distributions yield convergent threshold policies whose limit still satisfies the fairness condition.Compactness of the quantile parameters produces a limiting multiple-threshold policy, and iterated expectations establish the condition at the limit.

F.5 Proof of Theorem 17

The proof of Theorem 17 constructs a finite-dimensional probe that preserves feasibility while generically disrupting fairness constraints. It concludes that fairness-compatible, non-dominated policies occur only on a shy set of distributions.

  • Shyness proof: Theorem 17’s argument shows that the relevant set E has zero probe measure while translated feasible distributions have positive probe measure.This is the condition needed to apply the probe characterization of relative shyness.
  • Extensions: The same analysis extends to conditional principal fairness and path-specific fairness through a minor modification of the probe.The paper also states that the thrust applies to non-causal equalized odds, which is generically strongly Pareto dominated.
  • Probe construction: The probe W is designed as WUp + WLo, with perturbations that preserve probability and budget feasibility while moving mass across outcome and utility regions.The components separately perturb outcome labels and utility-scale mass to target different fairness constraints.
  • Probe construction: Any policy that is not strongly Pareto dominated must be a budget-exhausting multiple-threshold policy with non-negative thresholds.This structural restriction is the starting point for constructing the probe.
  • Shyness proof: For the policy 1u(x)>0, WLo perturbations break the balance requirements implied by counterfactual equalized odds for typical perturbations.The policy remains the unique budget-exhausting multiple-threshold policy, but fairness holds only on a null subset of probe perturbations.
  • Extensions: For almost every U-fine distribution, any decision policy satisfying equalized odds is strongly Pareto dominated under the stated utility and support assumptions.The result is stated in Corollary 70 for distributions of X and Y on X × Y.

F.6 Proof of Theorem 9

This section proves that utility-maximizing policies satisfying demographic or equalized fairness constraints are confined to shy sets of risk distributions. The proof uses a one-dimensional perturbation that makes the fairness equality hold only on a measure-zero parameter set.

  • Risk-scale setup: On the risk scale, a policy is utility maximizing exactly when it selects every individual with risk above t and rejects every individual below t, almost surely.Absolute continuity removes the ambiguity at the threshold itself.
  • Risk-scale setup: EDP denotes risk distributions where demographic parity holds for a utility-maximizing policy, while EFP denotes distributions satisfying the corresponding fairness condition.The paper characterizes these sets through conditional and unconditional probabilities of exceeding the utility threshold.
  • Probe construction: The probe shifts probability mass above and below the threshold across two demographic groups, producing opposite changes in their exceedance probabilities.The perturbation is constructed so one group gains mass above t while the other loses it.
  • Shyness conclusion: For both fairness sets, the defining equality holds only on a measure-zero subset of the perturbation parameter.The proof derives this after expressing the perturbed probabilities using baseline quantities e0, e1, p0, and p1.
  • Shyness conclusion: Both EFP and EDP are therefore shy.The result follows after showing that the fairness-compatible sets have zero measure along the constructed probe.

F.7 Proof of Corollary 18

Under the theorem’s assumptions, utility-maximizing policies are Pareto efficient but generically fail several causal fairness definitions. The proof uses convexity and separating hyperplanes to construct a utility for which the maximizing policy is efficient, then shows fairness violations hold almost everywhere.

  • Under Theorem 17’s hypotheses, the constructed utility-maximizing policy is Pareto efficient but does not satisfy counterfactual equalized odds, conditional principal fairness, or path-specific fairness for almost every joint distribution.
  • For almost every fine distribution, no utility-maximizing policy satisfies counterfactual equalized odds.
  • For almost every fine distribution satisfying the stated support conditions, no utility-maximizing policy satisfies conditional principal fairness or path-specific fairness.
  • The feasible set of utility-outcome pairs is convex because convex combinations of feasible policies remain feasible.
  • A policy maximizing a utility in the admissible family is Pareto efficient.

F.8 General Measures on K

This section studies when Pareto-efficient policies can satisfy counterfactual equalized odds and characterizes path-specific fairness under general covariates. It shows both that fairness-compatible efficient policies can exist on an open set and that, under continuity conditions, path-specific fairness can force constant decision probabilities.

  • Theorem 17’s restriction to fine distributions excludes atoms on the utility scale, which can create threshold-policy degrees of freedom that locally support causal fairness.
  • There exist settings in which an open ball of distributions admits a Pareto-efficient policy satisfying counterfactual equalized odds.
  • A threshold policy can be budget-exhausting, utility-maximizing, Pareto efficient, and satisfy counterfactual equalized odds in the constructed example.
  • Path-specific fairness with W = X is characterized by the corresponding counterfactual decision condition.
  • Under the stated continuity and nonsingularity assumptions, any Π-fair policy with W = X is constant almost surely.

G.2 A Markov Chain Perspective

The Markov-chain perspective rewrites Π-fairness as an invariance condition for transition matrices. This yields a structural characterization: fair policies are constant on recurrent classes, recovering the earlier theorem when those classes correspond to strata.

  • The finite-state restriction reflects technical difficulties in characterizing long-run behavior for arbitrary infinite Markov chains.
  • For finite covariate spaces, Π-fairness is equivalent to Pa′d = d for every protected-attribute intervention a′.
  • A policy that is a 1-eigenvector of a finite Markov chain is constant on each recurrent class.
  • Therefore, any Π-fair policy is constant on the recurrent classes of the averaged transition matrix P.
  • The recurrent-class characterization recovers Theorem 11 when the averaged transition matrix decomposes into irreducible blocks corresponding to strata.

H.2 Proof of Proposition 19

The proof of Proposition 19 shows that counterfactual predictive parity imposes smooth constraints on beta-distribution parameters and group-specific thresholds. Sard’s theorem then implies that parameter settings admitting a Pareto-efficient policy satisfying the constraint form a measure-zero set.

  • Counterfactual predictive parity imposes three smooth constraints on α0, β0, α1, β1, t0, and t1.
  • Sard’s theorem implies that projecting this three-dimensional solution manifold onto the four beta-distribution parameters yields a measure-zero set.
  • Conditional on Y (1), the risk distributions are beta distributions with parameters shifted according to the outcome.
  • A Pareto-efficient policy satisfying counterfactual equalized odds must be a multiple-threshold policy with group-specific thresholds.
  • If the constraint map is smooth and has full-rank Jacobian, its zero set is a smooth 3-manifold.

H.3 Proof of Proposition 20

Under beta-distributed conditional scores with ordered parameters, the proof shows that every counterfactual predictive-parity policy is strongly Pareto dominated. It establishes this by reducing Pareto-efficient candidates to threshold policies and deriving a contradiction.

  • After accounting for the parameterization change, Proposition 20 follows as a corollary.
  • The proof assumes beta-distributed conditional distributions of r(X) given A, with αa1 < αa0 and βa0 < βa1.
  • Any policy satisfying counterfactual predictive parity is strongly Pareto dominated under the stated beta-parameter ordering.
  • For Pareto-efficient policies, Proposition 15 permits restricting attention to threshold policies when λ = 0.
  • The threshold ordering ta1 ≤ ta0 implies a comparison of the conditional untreated-group expectations under the beta-distribution assumptions.
  • Counterfactual predictive parity requires equal untreated-group expectations, contradicting the inequality derived from the threshold policy; therefore, no such Pareto-efficient threshold policy exists.
Loading 1808.00023v3…