PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Unfairness in Fair Machine Learning from Prejudiced Data

Nathan Kallus, Angela Zhou

arXiv:1806.02887v1stat.MLcs.LG

TL;DR

The paper asks when fairness adjustments based on historically censored training data remain unfair on the target population. It characterizes residual unfairness and shows that, under specified conditions, adjustments can disadvantage the same groups as the biased policy; the SQF case study exhibits this pattern.

  • Problem

    When biased policies censor training outcomes, fairness adjustments assessed on the available data may not be fair on the population where the resulting policy is applied.

    Source: §1 · p. 2

  • Method

    The paper characterizes conditions for residual unfairness and uses sample reweighting, when benchmark data is available, to estimate target-population fairness metrics.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Under certain conditions, fairness-adjusted policies retain unfairness against the same groups disadvantaged by the original policy, a pattern also found in the SQF case study.

    Source: §4 · p. 3, §6 · p. 8, §7 · p. 8

  • Takeaways & Limitations

    The paper concludes that fairness evaluation must account for the sampling process when the goal is fairness on the target population.

    Source: §1 · p. 2, §7 · p. 8

    Target-population evaluation relies on a missing-at-random assumption, which can fail when outcomes influence whether observations are censored.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Recent work in fairness in machine learning has proposed adjusting for fairness by equalizing accuracy metrics across groups and has also studied how datasets affected by historical prejudices may lead to unfair decision policies. We connect these lines of work and study the residual unfairness that arises when a fairness-adjusted predictor is not actually fair on the target population due to systematic censoring of training data by existing biased policies. This scenario is particularly common in the same applications where fairness is a concern. We characterize theoretically the impact of such censoring on standard fairness metrics for binary classifiers and provide criteria for when residual unfairness may or may not appear. We prove that, under certain conditions, fairness-adjusted classifiers will in fact induce residual unfairness that perpetuates the same injustices, against the same groups, that biased the data to begin with, thus showing that even state-of-the-art fair machine learning can have a "bias in, bias out" property. When certain benchmark data is available, we show how sample reweighting can estimate and adjust fairness metrics while accounting for censoring. We use this to study the case of Stop, Question, and Frisk (SQF) and demonstrate that attempting to adjust for fairness perpetuates the same injustices that the policy is infamous for.

1. Introduction

The paper studies how historically biased selection into training data can leave fairness-adjusted policies unfair on the population where they are applied. It characterizes this problem and describes methods for evaluating and adjusting fairness under censoring.

1. Introduction

  • Systematic censoring can make training data unrepresentative of the target population, so fairness adjustments measured on the training data may leave residual unfairness.The paper identifies this mismatch in applications such as loan approval, policing, child welfare, bail, and recidivism prediction.Source: §1 · pp. 1–2
  • The paper characterizes when censoring induces residual unfairness through group differences in conditional Bayes-optimal risk-score distributions.When benchmark data is available, sample reweighting can estimate target-population accuracy metrics and indicate which groups remain disadvantaged.Source: §1 · p. 2
  • The paper studies how historical inclusion policies affect fairness adjustments using data from New York City’s Stop, Question, and Frisk policy.It argues that sampling processes must be accounted for when fairness is required on the general population.Source: §1 · p. 2

2. Problem Setting

The paper distinguishes the population represented by labeled training data from the population where a learned policy must be fair. It frames equal-opportunity and equalized-odds adjustments around that distinction.

2. Problem Setting

  • The historical inclusion indicator Z determines which instances have labeled outcomes in the training data, while T marks membership in the policy’s target population.The target population is not simply a held-out test set, and a random target-population sample is generally unavailable.Source: §2 · p. 2
  • The learned policy assigns labels using observed covariates X and protected class A, while labeled training data describe only the Z = 1 population.Unlabeled X and A data from the T = 1 population may or may not be available.Source: §2 · p. 2
  • Equalized odds requires a policy’s prediction to be independent of protected class A conditional on the label, whereas equal opportunity applies this requirement for positive labels.The paper frames both criteria relative to an event, distinguishing fairness on the training population from fairness on the target population.Source: §2 · pp. 2–3
  • Hardt et al.’s adjustment post-processes a score with group-specific, potentially randomized thresholds to equalize error rates while optimizing classification loss.For equal opportunity, the threshold for each group is derived from its conditional score distribution among positive-label cases.Source: §2 · p. 3

3. Related Work

Related work covers fairness criteria, biased-data effects, selective labels, and population-level sampling adjustment. The paper connects these strands by studying fairness impacts when training data are censored.

3. Related Work

  • Prior work formalizes equal opportunity and equalized odds, while studies of predictive policing document how training on police records can perpetuate disparate enforcement.Other work models how arrest observations shape beliefs about crime rates and considers online rejection sampling.Source: §3 · p. 3
  • Selective-label research learns from decisions that determine whether outcomes are observed, but the cited work does not consider the subsequent fairness of the learned policy.That work uses differences among decision makers’ thresholds to identify comparison subpopulations.Source: §3 · p. 3
  • Sampling adjustment and reweighting support population-level inference under population mismatch, and the Horvitz-Thompson estimator uses inverse sampling-probability weights.Earlier sample-selection-bias work studied classifier evaluation without considering fairness impacts.Source: §3 · p. 3

4. Residual Unfairness Under Disparate Benefit of the Doubt: Bias In, Bias Out

The section shows that fairness adjustment using censored training data can leave the same historically disadvantaged group worse off. It characterizes score-distribution conditions for residual unfairness, illustrates the effect in a loan example, and states how the interpretation depends on missingness assumptions.

4. Residual Unfairness Under Disparate Benefit of the Doubt: Bias In, Bias Out

  • Fairness adjustment based on censored-data error parity can leave residual unfairness against the same group prejudiced by the training policy.Source: §4 · p. 3
  • In the loan simulation, naive adjustment sets group 1's threshold higher than full-data or reweighted policies at every group-0 threshold, remaining systematically harsher.Source: §4 · p. 4

4.1. Disparate Benefit of the Doubt

  • Under strong disparate-benefit conditions, every derived equal-opportunity classifier has nonnegative inequity against group b, and at least one has strictly positive inequity.The conditions concern opposing score-distribution orderings for positive members of groups a and b between training and target populations.Source: §4.1 · p. 5
  • If the score-distribution ordering is strict throughout (0,1), every nontrivial derived equal-opportunity classifier strictly disadvantages group b.Source: §4.1 · p. 5
  • Even when both groups' score distributions shift in the same direction, a larger shift for one group over a threshold interval makes classifiers using thresholds there disadvantage group b.In the loan example, equal true-positive rates of 0.58–0.95 on training data fall in such a range and disadvantage the underprivileged class.Source: §4.1 · p. 6
  • For groups with disparate score endowments, the condition can be relaxed: thresholds in the specified interval still yield strictly positive inequity against group b.Source: §4.1 · p. 6
  • Disparate suspicion, expressed through true-negative rates, also leads derived equalized-odds classifiers to violate equalized odds against the same disadvantaged group.Source: §4.1 · p. 6

4.2. Interpretation of Scores Under MAR

  • Under MAR, training and target Bayes scores coincide and the unrestricted loss-minimizing decision function is the same, but outcome-related self-censoring can violate MAR.Source: §4.2 · pp. 6–7

5. Fairness Assessment and Adjustment with Biased Data via Sample Reweighting

When unlabeled target-population data are available under MAR, sample reweighting can estimate accuracy metrics on that population and support fairness adjustment there.

5. Fairness Assessment and Adjustment with Biased Data via Sample Reweighting

  • Unlabeled target-population data under MAR allow evaluation of accuracy metrics on the target population and assessment of residual unfairness.The paper uses this approach to assess SQF and adjust fairly in an example.Source: §5 · p. 7
  • The propensity score ratio compares the probabilities of inclusion in the target and training populations conditional on X and A.A weight equal to this ratio up to a factor depending on A can evaluate true positive rates.Source: §5 · p. 7
  • Under Proposition 6, reweighted training-data averages estimate target-population rates, so equal opportunity or equalized odds can be adjusted using those rates.The procedure computes target-adjusted TPRs and/or FPRs for a black-box predictor, then applies the adjustment as usual.Source: §5 · p. 7
  • When only labeled examples are available, an unlabeled target-population dataset can support weights estimated from the target and training distributions of X and A.When labeled and unlabeled target examples are both available, inverse labeling probabilities provide an alternative weight.Source: §5 · p. 7

6. Case Study: Stop, Question, and Frisk

The SQF case study adjusts stop-and-search predictors using estimated target-population rates, yet the adjusted policies remain unfair across racial groups in New York City.

6. Case Study: Stop, Question, and Frisk

  • SQF records contain only stops, and the stop and NYC population distributions differ geographically, motivating the target-population correction.The study treats race as the protected class and includes precinct encoding among the covariates.Source: §6 · pp. 7–8
  • 11% of white non-Hispanic innocents, up to 20% of white Hispanic, 16% of other, and 14–15% of Black innocents are wrongly targeted under equal opportunity adjustment.Equalized-odds adjustment also leaves disparities, though they are smaller; Hispanics remain particularly burdened.Source: Table 2 · p. 8, §6 · p. 8
  • The study trains a logistic-regression predictor on SQF stops, while defining the target population through NYC precinct demographics and assuming ancillary covariates are not disproportionate.The weights use SQF data and 2010 American Community Survey estimates matched from census blocks to precincts, with ratios clipped at 10.Source: §6 · p. 8
  • The case study derives equal-opportunity and equalized-odds stop policies, balancing false negatives and false positives at an exchange rate of 25:1.The predicted outcome is whether a search would not recover a weapon from a person suspected of possessing one.Source: §6 · p. 8
  • The adjusted policies continue to disadvantage non-white individuals, perpetuating the same biases associated with SQF.The study frames predictive targeting as a policy for whom to stop and search, rather than as a secondary filter for officer-targeted individuals.Source: §6 · p. 8

7. Conclusion

The conclusion frames residual unfairness as a practical consequence of fairness adjustment on biased datasets. It argues that historical prejudices can persist in the adjusted policy.

7. Conclusion

  • Residual unfairness occurs when policies learned from biased datasets remain unfair in practice after fairness adjustment.The conclusion characterizes this as a general setting in which datasets arise under a prejudiced historical policy.Source: §7 · p. 8
  • The paper studies datasets generated under prejudiced historical policies, a structure it says captures many settings where fairness has been considered.Source: §7 · p. 8
  • The paper proves that the same prejudices can be reflected in a supposedly fairness-adjusted policy.Source: §7 · p. 8

A. Omitted Proofs

The omitted proofs establish inequalities for derived equal-opportunity classifiers and give an algebraic derivation supporting the reweighting result.

A. Omitted Proofs

  • The proof of Proposition 2 uses the signs of group-specific inequity terms to establish positive residual inequity for a derived equal-opportunity classifier.The proof states ε_a,b = Δ_a(θ_a) − Δ_b(θ_b), with Δ_a(θ_a) ≥ 0 and Δ_b(θ_b) ≤ 0.Source: A. Omitted Proofs · p. 11
  • The proof of Proposition 3 applies strict opposite signs for the two groups’ inequity terms to a nontrivial derived equal-opportunity classifier.It likewise expresses residual inequity as the difference between the groups’ Δ terms.Source: A. Omitted Proofs · p. 11
  • The Proposition 6 proof rewrites a target-population joint probability by conditioning on X and A and then applying Bayes’ law.Source: A. Omitted Proofs · p. 11

B. Weak Disparate Benefit of the Doubt in Credit Card Data

In the credit-card example, censoring accepted applicants by income or income per dependent yields different directions of residual unfairness after fairness adjustment. The section also introduces residual unfairness under MAR and the role of reweighting disparities.

B. Weak Disparate Benefit of the Doubt in Credit Card Data

  • The credit-card example defines the protected group as applicants below median age and compares censoring accepted cardholders by income or income per dependent.The target population is all accepted applicants; the age cutoff is 31.67.Source: B. Weak Disparate Benefit of the Doubt… · p. 11
  • Income correlates with young-group membership at -0.32, while income per dependent correlates at 0.03.The passage attributes the near-zero correlation for income per dependent to income and household-size correlations with age canceling out.Source: B. Weak Disparate Benefit of the Doubt… · pp. 11–12
  • Income censoring yields threshold and FNR ranges where fairness-adjusted classifiers disadvantage younger applicants.The relaxed condition for disparately endowed groups allows a wider threshold range than the standard condition.Source: B. Weak Disparate Benefit of the Doubt… · p. 12, Figure 6 · p. 12
  • Income-per-dependent censoring reverses the direction: the identified FNR ranges disadvantage older applicants, and the standard condition applies.Here the ordering of the group discrepancies is opposite, so the alternative condition provides no relaxation.Source: B. Weak Disparate Benefit of the Doubt… · p. 12, Figure 6 · p. 12

C. Residual Unfairness Under MAR

  • Under MAR, the section considers cases where protected-attribute-only prejudice can be corrected and disparities in importance weights characterize residual unfairness.The passage also states that reweighting can help characterize which groups remain disadvantaged.Source: C. Residual Unfairness Under MAR · p. 12

C.1. No Residual Unfairness if Inclusion Depends Only on Protected Attributes

When inclusion and target-population membership depend only on the protected attribute, training and target inequity of opportunity are equal. Residual unfairness therefore requires inclusion to vary with covariates.

C.1. No Residual Unfairness if Inclusion Depends Only on Protected Attributes

  • With protected-attribute-only inclusion, true positive rates remain the same between training and target populations.The passage states that this setting has no residual unfairness.Source: §C.1 · p. 12
  • If inclusion and target-population membership each depend only on protected attribute A, training and target inequity of opportunity are equal.This result holds under Assumption 1.Source: §C.1 · p. 12
  • Residual unfairness occurs only when biased inclusion varies across covariates, as in SQF stops that differ by precinct and race.The passage notes that censoring often operates through proxies for protected attributes.Source: §C.1 · p. 12

C.2. Characterizing Residual Unfairness in Terms of Propensity Ratios Disparities

Under Assumption 1, reweighting estimates characterize residual unfairness. The section relates target-population TPR shifts to the average ratio weights of included true positives and false negatives.

C.2. Characterizing Residual Unfairness in Terms of Propensity Ratios Disparities

  • Under Assumption 1, residual unfairness can be characterized using reweighting estimates.Proposition 8 supplies a condition relating target and training TPRs.Source: §C.2 · pp. 12–13
  • The target TPR is higher when the average ratio weight among included true positives exceeds that among included false negatives.The passage explains that this occurs when positive examples the predictor gets right were less likely to enter training data.Source: §C.2 · p. 13
  • For a classifier satisfying equal opportunity on training data, Corollary 9 characterizes when target inequity of opportunity is positive.The characterization follows from Proposition 8 and equal training TPRs under equal opportunity.Source: §C.2 · p. 13

C.3. Proofs

The proof section derives the protected-attribute-only result by applying Proposition 6 and iterating expectations. The following SQF background describes the program and the demographic imbalance associated with its implementation.

C.3. Proofs

  • The proof of Proposition 7 obtains its result by applying Proposition 6 under Pr(Z = 1 | X, A) = Pr(Z = 1 | A) and iterating the expectation over X.Source: §C.3 · p. 13
  • The Proposition 8 proof writes the TPR as a conditional probability for true positives in group A=a and develops the comparison using reweighting expressions.The supplied proof fragments include conditional expectations for predicted-positive and predicted-negative true positives.Source: §C.3 · p. 13

D. Information on Stop, Question and Frisk

  • SQF permits stops, questioning, and possible searches on reasonable suspicion short of probable cause; about 90% of 2011 stops found no evidence of crime.Officers file individual reports documenting characteristics, location, and reasons for stops.Source: D. Information on Stop, Question and… · p. 14
  • SQF stop data misrepresent New York City's population amid disparate patrol levels, precinct-level implementation correlated with demographics, and potential individual racial bias.The passage also notes that recorded reasons for stops may act as proxies for discrimination.Source: D. Information on Stop, Question and… · p. 14
Loading 1806.02887v1…