Source-linked AI summary
A Confidence-Based Approach for Balancing Fairness and Accuracy
Benjamin Fish, Jeremy Kun, Ádám D. Lelkes
TL;DR
The paper asks how fair learning can reduce discriminatory bias while retaining high accuracy, especially when training data are biased. It proposes shifting the protected group’s decision boundary using confidence and margins, and evaluates fairness with RRB alongside bias and accuracy. SDB performs comparably to or better than prior methods, while RRB distinguishes naive baselines from more sensible algorithms.
Problem
Biased data prevent evaluation against unbiased ground truth, while accuracy and statistical parity can conflict and statistical parity alone may not capture fairness.
Method
The paper applies Shifted Decision Boundary to confidence-producing classifiers and proposes Resilience to Random Bias to evaluate robustness to synthetic random bias.
Results
On Census Income, both SDB algorithms achieve statistical parity with about 18% error, while SDB has the highest RRB; across studied datasets, RRB distinguishes less reasonable baselines from SDB.
Takeaways & Limitations
Bias and accuracy should be supplemented with RRB because naive label modifications can achieve low bias and high accuracy without being reasonably fair.
Takeaways & Limitations
RRB models a comparatively mild form of bias and is presented as minimally necessary, not sufficient, for individual fairness.
Abstract
from arXiv · showhide
We study three classical machine learning algorithms in the context of algorithmic fairness: adaptive boosting, support vector machines, and logistic regression. Our goal is to maintain the high accuracy of these learning algorithms while reducing the degree to which they discriminate against individuals because of their membership in a protected group. Our first contribution is a method for achieving fairness by shifting the decision boundary for the protected group. The method is based on the theory of margins for boosting. Our method performs comparably to or outperforms previous algorithms in the fairness literature in terms of accuracy and low discrimination, while simultaneously allowing for a fast and transparent quantification of the trade-off between bias and error. Our second contribution addresses the shortcomings of the bias-error trade-off studied in most of the algorithmic fairness literature. We demonstrate that even hopelessly naive modifications of a biased algorithm, which cannot be reasonably said to be fair, can still achieve low bias and high accuracy. To help to distinguish between these naive algorithms and more sensible algorithms we propose a new measure of fairness, called resilience to random bias (RRB). We demonstrate that RRB distinguishes well between our naive and sensible fairness algorithms. RRB together with bias and accuracy provides a more complete picture of the fairness of an algorithm.
1 Background and Motivation
Machine-learning systems can perpetuate discriminatory outcomes from biased data, creating a need to balance accuracy with reduced bias. The paper reviews fairness notions and introduces SDB alongside RRB to address limitations of statistical parity and the bias-error trade-off.
- Motivation: Biased training data can cause algorithms to perpetuate discriminatory decisions against protected groups, even without discriminatory intent.
- Motivation: The central question is how to maintain high accuracy while reducing discriminatory bias.
- Motivation: In the biased-data setting, statistical parity is studied as a goal while recognizing that achieving it may require reducing accuracy.
- Contributions: The paper proposes SDB, a confidence-based method that optimizes the bias-error trade-off, and RRB, a measure intended to distinguish naive from more sensible fairness algorithms.
- Existing notions of fairness: Statistical parity measures population-wide differences in positive-label probabilities between protected and complementary groups.
- Existing notions of fairness: Statistical parity can be achieved by arbitrarily flipping labels in the protected group, without regard to the classification task.
logistic regression.
The paper situates SDB among massaging, regularization, metric-based, and representation-based fairness methods, then develops confidence measures for boosting, SVMs, and logistic regression. Margin-based confidence motivates changing predictions near the decision boundary, where errors are more likely.
- Related approaches: Massaging changes biased data before training, while regularization adds a discrimination penalty to the learning objective.
- Related approaches: SDB can be viewed as post-processing regularization that makes the bias-accuracy trade-off transparent and controllable.
- Related approaches: Unlike metric-based individual-fairness approaches, SDB does not require a similarity metric or ranking of individuals.
- Margins: Margins combine correctness and confidence: positive margins indicate correct labels, while larger magnitudes indicate greater prediction confidence.
- AdaBoost: For boosting, signed confidence measures normalized agreement among weighted voters, whose magnitude reflects voter agreement.
- AdaBoost: Low-confidence examples are more likely to be misclassified, motivating SDB's strategy of flipping negative protected-group labels near the decision boundary.
- SVM and logistic regression: For SVMs, confidence is the kernel-defined distance from the separating hyperplane, while logistic regression uses its pre-rounding output as confidence.
2 Methods and Technical Solutions
The paper introduces SDB, which shifts the protected group's decision boundary to achieve statistical parity with minimal error, and theoretically bounds the resulting accuracy loss. It also defines naive baselines and RRB to expose why low bias and error alone are insufficient measures of fairness.
- Shifted decision boundary: SDB shifts the protected group's decision boundary to achieve statistical parity up to bias ε while minimizing error.It takes classifier confidences as input and returns the minimal-error shift satisfying the parity constraint.
- Naive baseline algorithms: The naive baselines can match much prior fairness literature on bias and error, showing that statistical parity and accuracy do not establish why an algorithm is fair.They therefore provide comparisons for SDB and tests for proposed fairness measures.
- Naive baseline algorithms: Random relabeling flips protected-group negative predictions with probability p chosen so bias is zero in expectation.The resulting classifier is randomized and otherwise follows the original hypothesis.
- Naive baseline algorithms: Random massaging flips protected-group negative training labels with the probability needed to achieve statistical parity in expectation before retraining.This adapts massaging strategies by randomizing the label modifications.
- Shifted decision boundary: SDB changes labels only for protected-group examples with small signed confidence, limiting its potential error increase.Margin theory provides the basis for this guarantee.
- Theoretical properties of SDB: The SDB error bound remains tied to sample margins for λ ≤ θ, then increases with protected examples having margins in [−λ, −θ].The theorem describes how the bound changes as the shift grows.
- Resilience to random bias: RRB evaluates whether a learning algorithm recovers original labels after independently adding random bias against a protected group.The construction introduces an uncorrelated random feature and flips protected-group positive labels with probability η.
3 Empirical Evaluation
Across three datasets, SDB generally achieves strong bias–error trade-offs and often outperforms baseline fairness methods, while RRB distinguishes more sensible methods from naive alternatives. The German results are less conclusive because the dataset is small and sensitive to kernel choice.
- Trade-offs: SDB achieves substantial bias reductions before significant accuracy losses in many experiments.The trade-off can therefore be inspected as bias approaches zero without an immediate substantial drop in accuracy.
- Dataset results: On Census Income, the three SDB techniques outperform the baselines and prior literature except DADT.Both SDB algorithms achieve statistical parity with about 18% error, and the two have the highest RRB.
- Dataset results: On Singles, SDB combined with logistic regression outperforms all other baselines.When baselines perform comparably, SDB tends to have much larger resilience to random bias.
- Dataset results: On German, two SDB techniques moderately outperform prior literature but not random relabeling or random massaging by a significant margin.Label error remains constant as the decision boundary shifts.
- Dataset limitations: The German dataset is too small to support a significant conclusion, so it is included mainly for completeness and comparison with prior literature.It contains only about a thousand records.
- Baseline comparisons: FWL reduces bias but fails to achieve statistical parity in two of three datasets and performs worse on either label error or bias in each dataset.The bias–error trade-off is also difficult to control with FWL, random massaging, or random relabeling.
- Trade-offs: SDB controls the bias–error trade-off after training, making selection of a desired operating point computationally inexpensive and transparent.Unlike most other fair learning methods, it does not require choosing the relevant hyper-parameter before training.
- Fairness evaluation: RRB separates random relabeling and random massaging from SDB when their accuracy and bias are comparable.The results indicate that fairness evaluation should not rely solely on accuracy and bias.
4 Significance and Impact
The paper introduces SDB to balance discrimination and label error, and RRB to evaluate fairness beyond the bias-error trade-off. Together, these contributions support transparent fairness analysis and distinguish naive from more sensible fairness methods.
- SDB is a general method for balancing discrimination and label error across learning algorithms with efficiently computable confidence measures.The authors apply it to AdaBoost, support vector machines, and linear regression.
- SDB allows practitioners to choose a point on the error-bias trade-off after training, while AdaBoost applications have theoretical generalization-error bounds.This makes the method fast and transparent for studying algorithmic fairness.
- RRB is proposed as a novel fairness measure intended to distinguish naive fairness modifications from more sensible algorithms.The measure is motivated by the requirement that a fair algorithm remain fair under random bias.
Acknowledments
The authors acknowledge Lev Reyzin for helpful discussions.
- Lev Reyzin is thanked for helpful discussions.
- The acknowledgment credits discussions that contributed to the paper’s development.
- The paper closes by recognizing Lev Reyzin’s intellectual assistance.