Why Is My Classifier Discriminatory?

Irene Chen; Fredrik D. Johansson; David Sontag

Source-linked AI summary

Why Is My Classifier Discriminatory?

Irene Chen, Fredrik D. Johansson, David Sontag

arXiv:1805.12002v2May 30, 2018stat.MLcs.LG

TL;DR

The paper asks how predictive discrimination should be addressed when fairness constraints may reduce accuracy, particularly in high-stakes settings. It decomposes cost-based discrimination into bias, variance, and noise, then studies model and data-collection interventions. Across case studies, additional data often improves fairness without sacrificing accuracy, while the analysis remains subject to scope and distributional limitations.

Problem
Fairness interventions can trade predictive accuracy for fairness, a difficult trade-off in high-stakes applications where errors may have serious consequences.
Method
The paper decomposes cost-based group discrimination into bias, variance, and noise and proposes model, sample-collection, and variable-collection procedures for reducing each component.
Results
Across three real-world tasks, additional samples often improve fairness without sacrificing accuracy; mortality prediction accuracy varies by as much as 20% between protected groups.
Takeaways & Limitations
Data collection can be an alternative to constraining models when inadequate samples or unmeasured predictive variables contribute to discrimination.
Takeaways & Limitations
The analysis assumes observed predictive differences are discriminatory and does not address data provenance or historical bias in labels; additional data may also cause delayed impacts or feedback loops.

Abstract

from arXiv · show

Recent attempts to achieve fairness in predictive models focus on the balance between fairness and accuracy. In sensitive applications such as healthcare or criminal justice, this trade-off is often undesirable as any increase in prediction error could have devastating consequences. In this work, we argue that the fairness of predictions should be evaluated in context of the data, and that unfairness induced by inadequate samples sizes or unmeasured predictive variables should be addressed through data collection, rather than by constraining the model. We decompose cost-based metrics of discrimination into bias, variance, and noise, and propose actions aimed at estimating and reducing each term. Finally, we perform case-studies on prediction of income, mortality, and review ratings, confirming the value of this analysis. We find that data collection is often a means to reduce discrimination without sacrificing accuracy.

1 Introduction

The paper argues that fairness should be analyzed in the context of data and model choice, especially when accuracy is critical in high-stakes applications. It proposes decomposing discrimination and addressing its sources through data collection and model development rather than automatically constraining models.

Data quality and model choice can encode unintentional discrimination and produce systematic disparate impact.
High-stakes applications make accuracy-for-fairness trade-offs difficult to justify because prediction errors can have serious consequences.
The paper assumes observed predictive differences between protected groups are considered discriminatory, while excluding data provenance and historical label bias from scope.
The paper analyzes discrimination through bias-variance-noise decompositions and proposes estimating additional samples’ value and identifying disadvantaged subpopulations.
Experiments cover income, critical-care mortality, and book-review rating prediction, with mortality accuracy varying by as much as 20% between protected groups.

2 Background

The background defines fairness through similar prediction costs across protected groups, including equalized false-positive and false-negative rates. It also notes that exact fairness constraints and simultaneous calibration are generally difficult to achieve.

A model is fair when its errors are distributed similarly across protected groups according to a cost function.
Equalized odds requires equal false-positive and false-negative rates across protected groups.
The paper measures discrimination as differences in group-specific costs, including FPR, FNR, zero-one loss, and regression MSE.
Exact equality is difficult to verify or enforce, so the analysis studies the degree to which fairness constraints are violated.
Calibration and multiple cost-based fairness criteria cannot generally all hold simultaneously unless accuracy is perfect or outcome base rates are equal.

3 Sources of perceived discrimination

The paper attributes perceived discrimination to group differences in bias, variance, and outcome noise, separating effects of model flexibility, training-data randomness, and unobserved predictive factors. This decomposition identifies which interventions may reduce disparities without sacrificing accuracy.

Data collection and model choice jointly affect fairness through the predictive variables, sampling distribution, training-set size, and hypothesis class.
Bias is loss from the main prediction relative to the optimal prediction, variance is loss across training datasets, and noise is algorithm-independent Bayes error.
Theorem 1 decomposes group-specific cost disparities for zero-one, class-conditional, and squared losses into differences in bias, variance, and noise.
Large bias differences can indicate an insufficiently flexible model class, making increased capacity or alternative model classes potential remedies.
Variance differences may arise from unequal group sample sizes or feature variability combined with high-capacity models, motivating targeted sample collection.
When group noise differs, model choice and training-set size alone cannot achieve zero expected discrimination without additional information or increased bias or variance.

4 Reducing discrimination through data collection

The paper proposes data-collection strategies that use learning curves and targeted subgroup analysis to reduce discrimination while preserving predictive accuracy. These strategies estimate how more samples or variables could change group-specific errors.

Additional training samples or predictive variables can improve both accuracy and fairness when predictive performance is limited by expensive or impractical data collection.
4.1 Increasing training set size: Discrimination learning curves quantify the absolute difference between group-specific costs as training-set size changes.
4.1 Increasing training set size: The study fits inverse power-law curves to group-conditional and class-conditional errors to extrapolate discrimination from subsampled data.
4.2 Measuring additional variables: When noise differences dominate, clustering individuals with high predictive disparity can guide collection of additional variables.
4.2 Measuring additional variables: Clusters with large protected-group cost differences identify subpopulations where targeted variables or samples may be most useful.

5 Experiments

Experiments on income, mortality, and review-rating prediction examine discrimination using standard classifiers, varying training data and identifying subpopulations with unequal errors. Across tasks, additional samples or targeted variables often offer ways to improve fairness without sacrificing predictive accuracy.

Experimental setup: The study evaluates fairness in income, mortality, and book-review rating prediction using cost-based discrimination measures.The experiments assess standard machine-learning models across three real-world datasets.
Income prediction: The Adult income experiment finds zero-one loss differing between genders, with a 95%-confidence interval of .085±.069.The random forest is well-calibrated, with Brier scores of 0.13 for men and 0.06 for women.
Income prediction: Men have consistently higher estimated noise than women, and non-overlapping nearest-neighbor intervals suggest noise may substantially contribute to discrimination.Noise is estimated using Mahalanobis, Bhattacharyya, and k-nearest-neighbor methods.
Income prediction: Among executive or managerial workers, false negatives occur more than twice as often for women as men: 0.412 versus 0.157.This subgroup contains 12% of the sample; further occupation sub-categorization may help reduce the disparity.
Mortality prediction: In mortality prediction, generalized zero-one loss differs significantly across five ethnic groups, with p < 0.001.Other and Hispanic patients have the highest and lowest loss, respectively, while the largest group, White patients, does not have the best accuracy.
Mortality prediction: Some mortality-loss differences decrease with additional training data, while topic-based clusters such as cardiac or cancer patients show large group disparities.Loss curves average 50 trials, and topic clusters are identified using a 50-topic latent Dirichlet allocation model.
Book-review ratings: For book-review ratings, the gender gap in mean-squared error is 0.136 ± 0.048, with MSEM = 0.224 and MSEF = 0.358.The reported findings suggest targeted sampling of the less represented gender may eliminate the gap.

6 Discussion

The discussion argues that discrimination should be analyzed through its data- and model-related sources before imposing fairness constraints. It presents statistical testing and targeted data collection as alternatives that can preserve accuracy, while noting broader scope and distributional risks.

Discussion: The paper decomposes cost-based discrimination into differences in bias, variance, and outcome noise.This separates effects associated with model choice, data collection, and inherent outcome unpredictability.
Discussion: Additional samples often improve fairness across three real-world case studies, whereas post-hoc correction may unnecessarily sacrifice predictive accuracy.The authors frame data collection as a potential cost of fairness rather than an unavoidable loss of accuracy.
Limitations and future work: Future work should address local and intersectional unfairness, while recognizing that additional data collection may cause delayed impacts, feedback loops, or distributional shifts.The discussion specifically mentions combining gender and race and studying fairness in non-stationary populations.
Statistical testing: The paper estimates discrimination from samples because group-conditional expected losses cannot generally be computed exactly.Monte Carlo estimates and large-sample tests support significance testing, while small binary-sample settings require more appropriate tests.
Statistical testing: Comparing two predictors requires testing both signed differences because the absolute difference of estimated discrimination levels is not normally distributed.The procedure uses folded-normal structure and evaluates both Z−1 and Z1 under the null hypothesis.

B.1 Datasets

The experiments use three datasets covering census income, clinical-note mortality, and book-review ratings, each paired with a protected attribute and task-specific outcome.

Adult Income Dataset: The Adult Income Dataset contains 32,561 instances, with income above $50,000 as the target and Gender as the sensitive feature.The processed data contains 105 predictive features after quantizing categorical attributes.
Goodreads reviews: The Goodreads dataset contains 13,244 reviews about the 100 most popular authors, with rating as the target and author gender as the sensitive feature.Gender was gathered using Wikipedia queries and pronoun inference.
MIMIC-III dataset: The MIMIC-III dataset includes 25,879 adult intensive-care patients, using clinical notes from the first 48 hours to predict hospital mortality.The protected attribute is self-reported ethnicity, and 13.8% of adult patients died in hospital.

B.2 Synthetic experiments

The synthetic experiment varies training-set size and model choice to examine learning curves and the difficulty of extrapolating Bayes error.

The outcome is generated as a quadratic function with heteroskedastic noise, using group-dependent Gaussian covariates.The protected-group probability is p(A = 1) = 0.3, with µ0 = 0, µ1 = 1, σ0 = 1, and σ1 = 2.
Figure 4 fits inverse power-laws to generalization error as a function of training-set size and extrapolates from selected sample sizes.The dotted extrapolations illustrate the difficulty of estimating the Bayes error; the caption gives N 0 = 3 · 10−4.
The experiment varies training-set sizes exponentially from 25 to 2^17 samples and fits decision-tree, random-forest, and ridge regressors.Trees are limited to depth T ≤4, and each training size is fit 200 times.

B.3 Book review ratings

Goodreads review-rating prediction shows group disparities in MSE, while increasing data—especially for the under-represented group—can reduce discrimination without simply changing model features.

Increasing the number of features reduces MSE but does not reduce Γ, whereas adding training data to one group has a larger effect on Γ.
The dataset contains 13,244 reviews, with 18% about female authors and 82% about male authors, and uses Tf-Idf features from the 5000 most frequent words.The target is review rating on a 1–5 scale, and author gender is the protected attribute.
ΓMSE( ˆY ) is 0.136±0.048, with MSEM = 0.224 for male-author reviews and MSEF = 0.358 for female-author reviews.This result uses a random forest with an 80/20 split, hyperparameter tuning, and averages over 50 trials.
Additional training data lowers MSE for both groups, while extrapolated noise differences are near zero: δM = 0.0011, δF = 0.0013, and |δF −δM| ≈0.The authors therefore expect discrimination to reflect differences in bias and variance more than noise.
Increasing nF/nM from 0.1 to 1.0 decreases MSEF far below MSEM and the best full-dataset random-forest MSE.The experiment fixes nM = 1939 and varies female-author training data while reserving 20% for testing.

B.4 Clinical notes

Clinical-note experiments identify group disparities in mortality-prediction errors and use topic-based subpopulations to guide whether additional data or features are more promising.

Subpopulation analysis: Topic 48, corresponding to cancer patients, is a subpopulation with large differences in errors between protected groups.Topics are inferred from representative words after modeling 50 clinical-note topics.
Subpopulation analysis: For the cancer subpopulation, increasing training size is estimated not to reduce error, so the authors recommend collecting more features such as lab results or detailed patient history.
Error analysis: Figure 6 reports confidence intervals for false positive and false negative rates and examines training-size effects for cancer patients.
Bias–variance–noise decomposition: Bias is generally not identifiable without estimating or bounding noise components, or assuming equal or negligible noise.Differences in bias across models can nevertheless be identified when variance is estimated.
Bias–variance–noise decomposition: Under squared loss, homoskedastic noise does not contribute to discrimination Γ.The result assumes N(x, a) = N for every x and protected group a.
Bias–variance–noise decomposition: For zero-one loss and class-specific variants, expected noise terms do not cancel because they depend on cn(x, a).

E Bias-variance decomposition. Proof of Theorem 1.

The proof decomposes expected prediction loss into bias, variance, and noise terms, establishing the result for squared and zero-one losses through auxiliary lemmas. It also analyzes the critical points and intersections of related power-law curves.

Bias-variance decomposition: Lemma A1 states the decomposition for both zero-one and squared loss, with loss-specific coefficients c1 and c2.For squared loss, both coefficients equal 1; for zero-one loss, c1 depends on agreement with the optimal predictor and c2 on agreement between optimal and majority predictors.
Bias-variance decomposition: For class-specific zero-one loss, expected loss at each x and a is expressed using noise, bias, and variance contributions.The stated identity holds for every y ∈ {0, 1}.
Bias-variance decomposition: The proof introduces c0 to relate loss against the data-generating prediction to loss against the optimal prediction and the optimal-prediction error.For zero-one loss, c0 equals +1 when the data-dependent and optimal predictions agree and −1 otherwise.
Bias-variance decomposition: Averaging conditional loss over datasets yields γa(Ŷ) as the expected sum of bias, signed variance, and scaled noise for protected group a.The derivation uses the assumption that datasets are drawn independently of the protected attribute A.
Difference between power law curves: For f(x) = ax^−b + c and g(x) = dx^−e + h, their difference has at most two local minima because the derivative has a unique positive root.The proof further states that f has one critical point for x > 0 and can switch signs at most twice.