PaperlayerView exact revision on arXiv

Source-linked AI summary

Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness

Sumon Biswas, Hridesh Rajan

arXiv:2005.12379v2cs.LGcs.SEstat.ML

TL;DR

The paper asks how much unfairness appears in practical machine-learning models and how mitigation affects fairness and performance. It evaluates 40 Kaggle models across five datasets with fairness metrics and seven mitigation techniques, finding that performance-oriented optimization can induce unfairness and that preprocessing mitigation can retain performance.

  • Problem

    The study addresses limited evidence about how much fairness issues exist in machine-learning models used in practice.

    Source: §1 · p. 1

  • Method

    The authors evaluate 40 Kaggle models across five datasets with fairness metrics, apply seven mitigation techniques, and assess performance impacts.

    Source: §9 · p. 11

  • Results

    Performance-oriented optimization goals can induce unfairness, while effective preprocessing mitigation is reported as more reliable and as retaining performance.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The study suggests further software-engineering research and library enhancements to make fairness concerns more accessible to developers.

    Source: §9 · p. 11

Abstract

from arXiv · show

Machine learning models are increasingly being used in important decision-making software such as approving bank loans, recommending criminal sentencing, hiring employees, and so on. It is important to ensure the fairness of these models so that no discrimination is made based on protected attribute (e.g., race, sex, age) while decision making. Algorithms have been developed to measure unfairness and mitigate them to a certain extent. In this paper, we have focused on the empirical evaluation of fairness and mitigations on real-world machine learning models. We have created a benchmark of 40 top-rated models from Kaggle used for 5 different tasks, and then using a comprehensive set of fairness metrics, evaluated their fairness. Then, we have applied 7 mitigation techniques on these models and analyzed the fairness, mitigation results, and impacts on performance. We have found that some model optimization techniques result in inducing unfairness in the models. On the other hand, although there are some fairness control mechanisms in machine learning libraries, they are not documented. The mitigation algorithm also exhibit common patterns such as mitigation in the post-processing is often costly (in terms of performance) and mitigation in the pre-processing stage is preferred in most cases. We have also presented different trade-off choices of fairness mitigation decisions. Our study suggests future research directions to reduce the gap between theoretical fairness aware algorithms and the software engineering methods to leverage them in practice.

1 INTRODUCTION

The paper examines whether real-world machine-learning models exhibit unfairness, what factors relate to bias, and how mitigation affects fairness and performance. Its study reports that performance-oriented optimization can induce unfairness and that mitigation involves trade-offs.

1 INTRODUCTION

  • The study evaluates fairness in 40 Kaggle models and asks which models and measures are more prone to bias, what relates to bias, and how mitigation affects performance.Source: §1 · p. 1
  • Performance-oriented optimization goals can cause unfairness, and some model constructs are directly related to fairness.Source: §1 · p. 1
  • Machine-learning libraries do not explicitly mention fairness in their documentation, while effective preprocessing mitigation is reported as more reliable and as retaining performance.Source: §1 · p. 1
  • The study reports trade-off concerns when mitigating bias in the models.Source: §1 · p. 2

2 BACKGROUND

The background frames fairness as avoiding discrimination based solely on protected attributes, and distinguishes group-level from individual-level fairness.

2 BACKGROUND

  • Removing a protected attribute alone may not prevent bias because correlated attributes can redundantly encode it.Source: §2 · p. 2
  • Group fairness assesses discrimination between protected groups, whereas individual fairness asks whether similar individuals receive similar predictions.Source: §2 · p. 2
  • The paper considers both fairness metrics and mitigation techniques for addressing bias in model predictions.Source: §2 · p. 2

3 METHODOLOGY

The methodology builds a benchmark of Kaggle models, standardizes their evaluation, measures performance and fairness, and applies mitigation algorithms across three stages.

3.1 Benchmark Collection

  • The benchmark contains 40 top-rated Kaggle models from five datasets, selected using predictive-model, upvote, and accuracy criteria.Source: §3.1 · p. 2

3.2 Experiment Design

  • For comparable fairness analysis, datasets receive consistent preprocessing, protected-group and favorable-outcome specifications, and a shared shuffled 70%-30% train-test split.Source: §3.2 · pp. 2–3
  • The experiment extracts each kernel's best-performing model, evaluates accuracy and F1 score, then measures seven fairness metrics before and after seven mitigation algorithms across ten runs.Source: §3.2 · p. 3

3.3.1 Accuracy Measure.

  • Performance is measured with accuracy and F1 score before fairness is assessed.Source: §3.3.1 · p. 4

3.3.2 Fairness Measure.

  • The fairness measures include base-rate, group-conditioned, and individual metrics, including DI, SPD, EOD, AOD, ERD, CNT, and TI.Source: §3.3.2 · p. 4

3.4 Bias Mitigation Techniques

  • Mitigation methods are grouped as preprocessing, in-processing, and post-processing, which respectively alter data, the model, or prediction outputs.Source: §3.4 · p. 4

4 UNFAIRNESS IN ML MODELS

The benchmark models exhibit bias across fairness measures, with fairness patterns varying by model, metric, and protected attribute. The analyses connect some differences to model choices and show that repeated runs and multiple fairness measures matter for interpretation.

4 UNFAIRNESS IN ML MODELS

  • All 40 models exhibit bias on most fairness metrics, although models from a dataset show similar patterns and some are fairer than others.Figure 3 presents metric-level bias, while Figure 4 sums absolute measures to compare cumulative bias.Source: §4 · p. 5, Figure 3 · p. 6, Figure 4 · p. 6
  • GC1 has lower cumulative bias than GC6, but its recall-oriented optimization is associated with greater error-rate-difference bias as false negatives rise.The paper links this pattern to the German Credit dataset's 69% male and 31% female composition.Source: §4 · p. 5
  • HC1's balanced class_weight affects group fairness, yet LightGBM documentation describes the parameter for performance rather than bias control.The authors report that class weighting mitigates male-female imbalance in HC1's predictions.Source: §4 · p. 6
  • BM5's disparate impact is 0.14, versus a 0.74 mean for the other seven Bank Marketing models, which the authors link to standard scaling.They identify age as a potentially dominating feature because older individuals are more numerous in that dataset.Source: §4 · p. 6
  • AC5's consistency bias is 3.61 times AC6's; the authors link the difference to AC5 excluding years of education, not to tree count or learning rate.Consistency measures individual fairness, including whether similar individuals receive different outcomes.Source: §4 · pp. 6–7
  • Metric patterns can disagree: BM5's DI is less than half other Bank Marketing models' while its ERD is more than twice theirs; CNT–TI correlation also reverses by dataset.The authors argue that a comprehensive set of fairness measures is needed because metric correlations and rankings vary.Source: §4 · p. 7, Figure 5 · p. 7
  • Across 10 runs, average standard deviations were 0.12 for DI and 0.04 each for EOD and AOD; Titanic's 891-case dataset varied most.The authors recommend representative training samples and repeated fairness evaluation for smaller datasets.Source: §4 · p. 7, Figure 6 · p. 7
  • Total bias differs by protected attribute: German Credit is 4.82 for sex and 7.72 for age; Adult Census is 15.15 for sex and 8.56 for race.GC1 and GC6 have cumulative bias of 0.12 and 0.60 for sex, but 0.85 and 0.88 for age, respectively.Source: §4 · p. 8, Figure 7 · p. 8

5 MITIGATION

The study compares seven mitigation algorithms across 40 models and finds that the strongest approach depends on how biased a model is initially. Preprocessing is preferred when effective, while post-processing performs better on more biased models.

5 MITIGATION

  • The study identifies preprocessing as preferable when effective, since Reweighing retained accuracy and F1 in several debiased models.When preprocessing was ineffective, the authors had to change the model or alter its predictions, suggesting bias was induced or propagated by the model.Source: §5 · p. 9
  • Reweighing effectively debiased GC1, GC6, AC3, AC5, AC8, BM1, and BM4 while retaining accuracy and F1 score.For AC1, AC2, BM2, and BM5, Reweighing also retained both measures when the most successful algorithm lost at least 22% of accuracy or F1.Source: §5 · p. 9
  • Post-processing algorithms mitigated bias in 90% of models, though they were not necessarily the most successful choice for each model.The three methods considered were Equalized Odds, Calibrated Equalized Odds, and Reject Option Classifier.Source: §5 · p. 9
  • Post-processing mitigated bias more effectively in more-biased models, but all three tested post-processing methods increased bias in the five least-biased models.Reject option classification and Equalized odds ranked more successfully for the most-biased models; preprocessing and Adversarial Debiasing were effective for the least-biased models.Source: §5 · p. 9

6 IMPACT

Mitigation often reduced model performance, and the size of the cost varied across algorithms and models. Comparing the top two algorithms reveals that some alternatives preserve performance with nearly comparable bias.

6 IMPACT

  • In-processing performance was uncertain: Adversarial Debiasing worked without performance loss on German Credit, but accuracy fell at least 21.4% in AC1, AC7, HC1, and HC7.The authors report good performance on German Credit but not Adult Census or Home Credit, and caution that these methods replace the original model.Source: §6 · pp. 9–10
  • Many models lost performance while mitigation reduced bias, although the Reweighing cases highlighted for GC1, GC6, AC3, AC5, AC8, BM1, and BM3 retained performance.Figure 9 presents changes in accuracy, F1 score, and total bias for the most effective algorithms.Source: §6 · p. 9, Figure 9 · p. 10
  • Post-processing was most successful for 21 of 40 models, but every such case lost accuracy and F1, averaging 7.49% and 10.07% reductions.The authors suggest considering the next-best algorithm; some post-processing cases had much smaller losses, such as HC8.Source: §6 · p. 10
  • 7.49% average accuracy reduction and 10.07% average F1 decrease accompanied the most successful post-processing algorithms across 21 models.Losses varied: AC1 lost 26.1% accuracy and 40% F1, whereas HC8 lost 1.6% accuracy and 1.3% F1.Source: §6 · p. 10
  • 18 of 40 models had a second-ranked mitigation algorithm with performance the same as or better than the top-ranked algorithm.For AC4, AC6, BM5, and HC8, the second-ranked algorithm's bias was within 0.1 of the first-ranked one; the first-ranked method was post-processing in all four cases.Source: Figure 10 · p. 10, §6 · p. 10

7 THREATS TO VALIDITY

The benchmark and evaluation are limited to selected, runnable Kaggle models and fairness-focused datasets. The authors repeated metric evaluation to reduce variability and used consistent data processing for comparisons.

7 THREATS TO VALIDITY

  • The benchmark included top-voted runnable kernels with more than five votes and accuracy of at least 65%, narrowing its scope to selected Kaggle models.The authors also screened for datasets and competitions with protected attributes and prediction outcomes they considered suitable for fairness analysis.Source: §7 · p. 10
  • Fairness metrics were evaluated 10 times and averaged, while model performance was checked against the corresponding Kaggle kernels.The study used IBM's AIF 360 toolkit and validated its results against a prior Adult Census setup.Source: §7 · pp. 10–11
  • For fairness comparisons, the authors used the same training and test preprocessing method for each dataset.Source: §7 · p. 11

8 RELATED WORKS

Prior work developed fairness metrics, mitigation methods, and software testing approaches, but the paper distinguishes its focus as empirical evaluation of practitioner-written models and mitigation impacts.

8 RELATED WORKS

  • Earlier fairness-testing tools detect feature associations, discriminatory inputs, or individual discrimination, whereas this study empirically evaluates practitioner-written models and mitigation impacts.The authors identify the absence of prior empirical measurement and comparison of fairness in models used in practice as the gap they address.Source: §8 · p. 11
  • Fairness-measure and mitigation research is described as primarily theoretical, while this work focuses on software-engineering aspects of models used in practice.Source: §8 · p. 11
  • Prior ML testing studies focused on robustness, a property the authors distinguish from fairness.The cited examples include DeepCheck, DeepXplore, and DeepTest.Source: §8 · p. 11

9 CONCLUSION

The paper empirically evaluates fairness in ML models and finds that ML libraries do not provide enough support to address fairness in practice. Its findings point to further software-engineering research and library enhancements to make fairness concerns more accessible to developers.

9 CONCLUSION

  • The authors conclude that ML libraries do not provide enough support to address fairness in practice.Source: §9 · p. 11
  • The study evaluates 40 models across five problem domains using fairness metrics, seven mitigation techniques, and performance-impact measurements.The authors created a benchmark of Kaggle models and assessed fairness before and after mitigation.Source: §9 · p. 11
  • The study calls for further software-engineering research and library enhancements to make fairness concerns more accessible to developers.Source: §9 · p. 11

Related papers

Loading 2005.12379v2…