Source-linked AI summary
Identifying and Correcting Label Bias in Machine Learning
Heinrich Jiang, Ofir Nachum
TL;DR
Datasets can contain complex group-related biases that classifiers inherit, while existing model-adjustment methods may create unfavorable accuracy or training trade-offs. The paper models observed labels as biased versions of unknown unbiased labels, derives example weights without changing labels or features, and reports fair classifiers with reasonable or comparative predictive error across fairness benchmarks and notions.
Problem
Training data can encode complex biases that classifiers reinforce, and existing fairness methods adjust models even though the data itself may be biased.
Method
The paper assumes observed labels come from an accuracy-seeking but potentially biased agent, derives bias-related weights, and trains classifiers on reweighted examples without modifying labels or features.
Results
Across standard fairness benchmarks and multiple fairness notions, the method often achieves the lowest test violation while often providing better or comparative predictive error than other fair-classification methods.
Takeaways & Limitations
The resulting procedure is general, practical, and compatible with off-the-shelf classification algorithms for several fairness notions.
Takeaways & Limitations
The framework assumes an unknown unbiased ground-truth label function and models observed labels as generated by a biased process that remains otherwise accurate.
Abstract
from arXiv · showhide
Datasets often contain biases which unfairly disadvantage certain groups, and classifiers trained on such datasets can inherit these biases. In this paper, we provide a mathematical formulation of how this bias can arise. We do so by assuming the existence of underlying, unknown, and unbiased labels which are overwritten by an agent who intends to provide accurate labels but may have biases against certain groups. Despite the fact that we only observe the biased labels, we are able to show that the bias may nevertheless be corrected by re-weighting the data points without changing the labels. We show, with theoretical guarantees, that training on the re-weighted dataset corresponds to training on the unobserved but unbiased labels, thus leading to an unbiased machine learning classifier. Our procedure is fast and robust and can be used with virtually any learning algorithm. We evaluate on a number of standard machine learning fairness datasets and a variety of fairness notions, finding that our method outperforms standard approaches in achieving fair classification.
1. Introduction
The paper frames unfair classification as a data-bias problem and introduces a fairness approach that models biased labels as deviations from unknown unbiased labels. It corrects this bias by reweighting examples without modifying observed labels or features.
- Motivation: Training data can encode complex biases that classifiers reinforce, while removing protected-group features is ineffective because of redundant encodings.These limitations make fairness difficult to achieve through simple feature-based remedies.
- Motivation: Existing approaches either post-process an already trained classifier or incorporate fairness constraints into training, potentially weakening accuracy trade-offs or complicating optimization.Post-processing decouples learning from fairness enforcement, while constrained methods may produce difficult or unstable training.
- Approach: The paper assumes an unknown unbiased ground-truth label function and observed labels assigned by an accuracy-seeking but potentially biased agent.The framework also covers settings where both features and labels are biased.
- Approach: A closed-form expression identifies training-data bias and suggests correcting it by assigning appropriate weights to examples.The paper derives this correction from its mathematical formulation rather than changing the observed labels or features.
- Scope: The method applies to demographic parity, equal opportunity, equalized odds, and disparate impact, and can use off-the-shelf classification procedures on the weighted data.The paper reports experiments on standard fairness benchmarks under varied fairness notions.
2. Background
The framework formalizes binary classification with an unknown unbiased ground-truth label function and observed labels generated by a biased function. It defines fairness through constraint functions over protected groups and identifies recovering the ground truth as the central learning problem.
- Setup: The framework considers a data domain X with distribution P and binary labels Y := {0, 1}, while noting that the method can generalize to other settings.A finite sample D[n] is drawn from the data distribution.
- Biased and Unbiased Labels: The observed dataset uses labels generated by ybias, whereas ytrue is an inaccessible unbiased ground-truth function from X to [0, 1].The paper explicitly assumes access only to data sampled from the biased labeling process.
- Biased and Unbiased Labels: The learning objective is to recover ytrue from D, but the relationship between ytrue and ybias is generally unknown without additional assumptions.The paper proposes a minimal relationship assumption in subsequent sections.
- Notions of Bias: Fairness is expressed using constraint functions c : X × Y → R, with a label function unbiased when its expected constraint value is zero.The signed expectation also represents the degree of positive or negative bias.
- Notions of Bias: Protected-group fairness uses an indicator g(x), group probability ZG, and positive-label proportions PX and PG to define group-based quantities.The framework allows multiple fairness constraints and protected groups, including overlapping groups.
3. Modeling How Bias Arises in Data
The paper models observed labels as the closest biased labeling to an unbiased ground truth under a constrained optimization assumption. Using KL-divergence, it derives closed-form relationships that enable recovery of the unbiased labels and motivate example reweighting.
- Modeling Bias: The framework uses a constrained optimization assumption to relate observed biased labels ybias to an unbiased ground truth ytrue.The ground truth satisfies the paper’s fairness constraints, while ybias achieves specified amounts of bias.
- Modeling Bias: Under this assumption, ybias is the label function closest to ytrue while achieving a specified bias level, with closeness measured by KL-divergence.The interpretation reflects accuracy-seeking labeling affected by human or process-level biases.
- Modeling Bias: The choice of KL-divergence yields a closed-form expression for ybias and supports the subsequent derivation.The paper notes that other divergences could be chosen, but KL-divergence enables this result here.
- Modeling Bias: Unlike prior constrained-optimization approaches that use constraints to form a machine-learning objective, this framework uses them to express the relationship between biased and unbiased labels.That relationship is then used to motivate a weighting procedure for learning from biased data.
4. Learning Unbiased Labels
The section derives a weighting procedure that uses biased-label data to train classifiers equivalent to those trained on unbiased labels. Coefficients are learned iteratively to satisfy fairness constraints, while the approach extends beyond demographic parity under additional assumptions.
- Learning h Given λ1, . . ., λK: The procedure computes example weights from the closed-form coefficients λ1, ..., λK and trains a classifier without changing observed labels.The method is proposed because only samples from the biased label function are available in practice.
- Learning h Given λ1, . . ., λK: Theorem 1 shows that weighting examples with biased labels makes training under any loss equivalent to training on unbiased labels over another feature distribution.The reweighted objective uses the biased-label distribution, while the equivalent objective uses the true-label distribution over some distribution ˜P.
- Learning h Given λ1, . . ., λK: The feature-distribution change introduced by reweighting does not affect the learned classifier’s bias under the mild conditions analyzed later.Theorem 1 trades training on unbiased labels for training over a slightly different feature distribution ˜P.
- Learning h Given λ1, . . ., λK: Coefficients are learned iteratively by evaluating fairness violations, updating multipliers, recomputing weights, and retraining the classifier.The classification procedure H may be any procedure that minimizes a weighted loss over a parametric function class.
- Learning h Given λ1, . . ., λK: The algorithm directly minimizes weighted loss while maximizing fairness, which the authors characterize as a non-zero-sum two-player game.This interpretation concerns the competing classifier and fairness objectives.
- Learning h Given λ1, . . ., λK: Demographic parity and disparate impact are directly supported, whereas equal opportunity and equalized odds require handling constraints that depend on unknown true labels.For the latter notions, the method uses an iterative reweighting approach with weights determined by protected attributes and observed labels.
5. Theoretical Analysis
The theoretical analysis establishes finite-sample consistency for the weighted classifier under regularity, hypothesis-class, and coefficient conditions. On data supported by a lower-dimensional manifold, the convergence rate depends on intrinsic rather than ambient dimension.
- Theoretical Analysis: The guarantee requires compact support for X and Lipschitz regularity of both ybias and ytrue.The stated regularity assumption places X in RD and bounds label-function variation by L.
- Theoretical Analysis: Under Assumptions 1 and 2, bounded coefficients satisfying Proposition 1, and a 2L-Lipschitz hypothesis class, the weighted optimum has a finite-sample error bound to ytrue.The bound applies for sufficiently large n and uses the weighted mean square error objective.
- Theoretical Analysis: The error bound is ||h∗−ytrue||2 ≤ C0 · log(2/δ)^(1/(2+D)) · n^(-1/(4+2D)).The norm is defined as the expected squared difference under x sampled from P.
- Theoretical Analysis: If X lies on a d-dimensional Riemannian submanifold, the rate depends on d rather than ambient dimension D.Theorem 3 assumes finite volume and condition number and gives probability at least 1 − δ for sufficiently large n.
6. Related Work
Related work enforces fairness through post-processing, Lagrangian objectives, or preprocessing. The paper distinguishes its approach by retaining the original labels and features while using mathematically derived reweighting with theoretical guarantees.
- Post-processing: Post-processing calibrates an already trained classifier, but it has limited flexibility and can produce poor accuracy trade-offs or provably suboptimal results.The cited discussion states that deterministic solutions may be incompatible with multiple fairness constraints, including equalized odds.
- Lagrangian Approach: Lagrangian methods transform fairness constraints into penalties, but constraints may require relaxation, and equality constraints can create feasibility and slack-selection challenges.The paper states that its method does not use the constraints directly in the model loss.
- Lagrangian Approach: Some cost-sensitive Lagrangian formulations use non-negative weights with modified labels, which the paper contrasts with reweighting the original loss.The modified-label interpretation may have legal implications because it can be viewed as training on falsified data.
- Pre-processing: Preprocessing methods may change labels or features, have legal implications, and have generally offered few theoretical guarantees.The paper instead only reweights the importance of sensitive groups and grounds the procedure in a mathematical model of bias.
7. Experiments
The experiments compare the proposed method with unconstrained, post-processing, and Lagrangian baselines across benchmark fairness tasks. The method generally achieves lower fairness violations while maintaining comparable or better predictive error.
- Datasets: The benchmark experiments use Bank Marketing, Communities and Crime, COMPAS, German Credit, and Adult datasets.The datasets cover banking, crime, recidivism, credit, and income-prediction tasks with protected attributes based on age, race, or gender.
- Results: Our method consistently produces fairer classifiers and often achieves the lowest test violation among the compared methods.The results also report reasonable fairness–accuracy trade-offs rather than optimizing fairness alone.
- Results: Our method often has better or comparable predictive error than other fair-classification methods.Figure 2 examines the trade-off between test error and fairness violations as the weighting parameter changes.
- Baseline comparison: Post-processing calibration improves over an unconstrained model but is often unable to find the classifier with the lowest bias.The calibration procedure separates model training from fairness enforcement.
- Experimental setup: Table 1 evaluates accuracy and fairness violation across datasets and fairness notions for four training approaches.The approaches are unconstrained training, post-processing calibration, the Lagrangian method, and the proposed method.
- Baseline comparison: The Lagrangian approach does not consistently provide fair classifiers and may suffer from instability, overfitting, or poor fairness generalization.These issues are associated with relaxed constraints and the method’s added optimization complexity.
8. MNIST with Label Bias
The MNIST experiment tests whether reweighting can recover performance when 20% of training labels are deliberately changed. The proposed method most closely approaches training on the true labels and outperforms the comparison methods.
- Experimental setup: 20% of MNIST training labels are randomly changed to create a biased-label dataset.The classifier is also encouraged to predict digit 2 at a rate of 10% through a demographic-parity-like constraint.
- Results: The proposed method is the only approach that approaches the accuracy of a classifier trained on the true labels.Test accuracy is computed against the unmodified labels.
- Results: The proposed method reduces error by over half compared with the Lagrangian approach or calibration.The comparison uses the true labels for evaluating test-set accuracy.
- Results: The proposed method improves error by roughly 30% compared with the next-best method, the Lagrangian approach.These results support the method’s ability to train toward underlying true labels despite observing biased labels.
9. Conclusion
The paper concludes that biased labels can be modeled through an unknown unbiased ground truth and corrected by reweighting training examples. Under the framework’s assumptions, the resulting classifier is approximately unbiased, and the approach extends across several fairness settings.
- Conclusion: The framework assumes an unbiased ground-truth label function underlying the observed biased labels.The paper presents this as a model for how dataset bias can arise.
- Conclusion: Reweighting training examples can correct label bias without changing the observed labels.The paper states finite-sample guarantees for approximate unbiasedness of the learned classifier.
- Correction procedures: The sampling alternative retains examples only when sampled labels agree with the observed labels, corresponding to training on unbiased labels.The weighting technique is presented as more practical when sampling would discard many examples.
- Fairness constraints: The weighting strategy extends to demographic parity, equal opportunity, equalized odds, disparate impact, and more general rate-based constraints.Equal opportunity and equalized odds are handled by up-weighting examples associated with violated true- or false-positive-rate constraints.
- Theory: The theoretical analysis shows that weighted empirical optimization approaches the unbiased target under stated smoothness, partition, and concentration conditions.The proof selects a radius R that depends on sample size, confidence, and dimension.
E. Proof of Theorem 3
Theorem 3’s proof transfers the earlier finite-sample argument to data supported on a compact low-dimensional manifold. A manifold volume bound replaces the ambient-dimensional partition bound, yielding rates in terms of intrinsic dimension.
- Geometric bound: Lemma 2 lower-bounds the volume of sufficiently small intersections between partition regions and the manifold.The bound uses the volume v_d of a unit ball in R^d and the manifold’s uniform volume measure.
- Geometric bound: Lemma 3 derives the manifold analogue of the earlier partition bound from the lower volume estimate.For sufficiently small R, each region has volume at least C2 · R^d.
- Proof strategy: Theorem 3 repeats Theorem 2’s proof while replacing the ambient-dimensional lemma with the manifold-specific Lemma 3.This replacement produces rates in terms of d rather than D.
- Experimental context: The experimental baselines include unconstrained logistic regression, post-calibration, and a hinge-relaxed Lagrangian method.The proposed algorithms use fixed learning-rate and loop settings across experiments.
F.2. MNIST Experiment Details
The MNIST experiments use a fixed fully connected neural-network architecture and optimizer settings across the calibration, Lagrangian, and proposed procedures.
- The classifier is a three-hidden-layer fully connected neural network with ReLU activations and 1024 hidden units per layer.
- Training uses TensorFlow’s ADAM optimizer under default settings for 10000 iterations with batchsize 50.
- MNIST experiment procedures: The calibration procedure swaps low-softmax-probability labels among predictions of 2 to make the prediction rate for 2 approach 10%.
- MNIST experiment procedures: The Lagrangian approach uses the same settings as before but matches the other procedures’ learning rate, while the proposed method retains its prior settings.
G. Additional Experimental Results
The experiment varies the weighting scale relative to Algorithm 1’s optimal λ and compares test error with demographic-parity violations. The optimal scale minimizes fairness violations, while no weighting minimizes error.
- At c = 1, using λ = λ∗ from Algorithm 1 produces the lowest demographic-parity fairness violation.The x-axis scale sets λ = c · λ∗ for classifier training.
- At c = 0, training on the unweighted dataset produces the lowest test error.