PaperlayerView exact revision on arXiv

Source-linked AI summary

Fair-SSL: Building fair ML Software with less data

Joymallya Chakraborty, Suvodeep Majumder, Huy Tu

arXiv:2111.02038v5cs.SEcs.LG

TL;DR

Prior software-engineering fairness work largely used supervised methods requiring labeled data, even though obtaining trustworthy labels is challenging and labels may contain human bias. Fair-SSL uses four semi-supervised pseudo-labelers with 10% labeled data, and its reported fairness and performance comparisons are similar to existing mitigation methods. Its authors describe the approach as cheaper than Fair-SMOTE, while noting that it cannot address every source of bias.

  • Problem

    Prior bias mitigation in software engineering mainly used supervised approaches requiring labeled data, which is challenging to obtain and may contain human bias.

    Source: §1 · p. 1

  • Method

    Fair-SSL uses four semi-supervised techniques to pseudo-label unlabeled data, then balances the combined training data by class and protected attribute.

    Source: §1 · p. 2, RQ1. Can Fair-SSL reduce bias? · p. 5

  • Results

    Fair-SSL’s bias scores are as good as those of three state-of-the-art mitigation algorithms, while its performance metrics sometimes lag but not by much.

    Source: RQ1. Can Fair-SSL reduce bias? · p. 5

  • Takeaways & Limitations

    Fair-SSL is cheaper than Fair-SMOTE because it uses 10% labeled data, and its data-preprocessing approach is model agnostic.

    Source: §5 · p. 7

    The evaluation assumes unbiased test data, and SSL cannot address objective-function, homogenization, or feature-correlation bias.

    Source: §5 · p. 7

Abstract

from arXiv · show

Ethical bias in machine learning models has become a matter of concern in the software engineering community. Most of the prior software engineering works concentrated on finding ethical bias in models rather than fixing it. After finding bias, the next step is mitigation. Prior researchers mainly tried to use supervised approaches to achieve fairness. However, in the real world, getting data with trustworthy ground truth is challenging and also ground truth can contain human bias. Semi-supervised learning is a machine learning technique where, incrementally, labeled data is used to generate pseudo-labels for the rest of the data (and then all that data is used for model training). In this work, we apply four popular semi-supervised techniques as pseudo-labelers to create fair classification models. Our framework, Fair-SSL, takes a very small amount (10%) of labeled data as input and generates pseudo-labels for the unlabeled data. We then synthetically generate new data points to balance the training data based on class and protected attribute as proposed by Chakraborty et al. in FSE 2021. Finally, the classification model is trained on the balanced pseudo-labeled data and validated on test data. After experimenting on ten datasets and three learners, we find that Fair-SSL achieves similar performance as three state-of-the-art bias mitigation algorithms. That said, the clear advantage of Fair-SSL is that it requires only 10% of the labeled training data. To the best of our knowledge, this is the first SE work where semi-supervised techniques are used to fight against ethical bias in SE ML models.

1 INTRODUCTION

The introduction frames fair machine-learning software as a quality concern and asks whether the labeling effort required to build fair models can be reduced. Fair-SSL applies semi-supervised pseudo-labeling and reports performance comparable to state-of-the-art fairness algorithms.

1 INTRODUCTION

  • The introduction motivates fairness as a software-quality concern because machine-learning systems make consequential decisions and can show bias across protected attributes.Examples include translation, speech captioning, recruiting, and face recognition.Source: §1 · p. 1
  • Prior work links biased predictions to improper labels and imbalanced training data, and reports that Fair-SMOTE improved fairness and performance.Fair-SMOTE is presented as breaking the premise that fairness and high performance cannot be achieved together except in trivial cases.Source: §1 · p. 1
  • The central question is whether the labeling effort associated with building fair models can be reduced.The introduction notes that trustworthy labels are challenging to obtain and may contain human bias.Source: §1 · pp. 1–2
  • Fair-SSL uses four semi-supervised techniques as pseudo-labelers and performs as well as three state-of-the-art fairness algorithms.The framework uses a small labeled set alongside unlabeled data.Source: §1 · p. 2
  • Fair-SSL uses 10% labeled training data, and the authors present it as the first SE work using semi-supervised learning to generate fair classification models.The contributions also include a situation-testing technique for creating fairly labeled data without human intervention.Source: §1 · p. 2

2 BACKGROUND

The background situates fairness work in software engineering alongside the costs and risks of labeled data. It introduces semi-supervised learning, dataset and fairness terminology, and the study’s performance and fairness metrics.

Software Fairness

  • The background describes bias testing and mitigation as increasingly important in the software life cycle, alongside industry tools and software-engineering initiatives.It names tools from IBM and Microsoft and events including Fairware and EXPLAIN.Source: §2 · p. 2
  • Within software engineering, prior work includes fairness testing tools, but the authors identify only Fairway and Fair-SMOTE as bias-mitigation frameworks, both supervised and label-intensive.The paper contrasts this with its semi-supervised approach using 10% labeled training data.Source: §2 · p. 2

Semi-supervised Learning

  • Outside software engineering, the authors identify one SSL-and-fairness study, which used three datasets and self-training, while noting SSL’s use in several SE tasks.SSL incrementally pseudo-labels unlabeled data and combines it with labeled data for model training.Source: §2 · pp. 2–3

Fairness Terminology & Metrics

  • The study uses ten binary-classification datasets and defines favorable labels and protected attributes in terms of benefits and privileged or unprivileged groups.Table 1 gives details of the datasets used.Source: Table 1 · p. 2, §2 · p. 3
  • The study uses five performance and four fairness metrics; it reports abs(1 - DI) so that all four fairness metrics are lower-is-better.For recall, precision, accuracy, and F1, larger values are better; for false alarm, AOD, EOD, and SPD, smaller values are better.Source: Table 2 · p. 2, §2 · p. 3

3 METHODOLOGY

Fair-SSL selects a small fairly labeled seed, pseudo-labels the remaining training data, then balances it by class and protected attribute before training and testing.

3.1 Prepare the Fairly Labeled data

  • Logistic regression models trained separately by protected-attribute subgroup keep a label only when both models’ predictions match the ground truth; otherwise, the point becomes unlabeled.The authors use this situation-testing procedure to prepare the fairly labeled set.Source: §3.1 · p. 3
  • The seed uses 10% of the fairly labeled set, selected to give equal proportions to favorable-privileged, favorable-unprivileged, unfavorable-privileged, and unfavorable-unprivileged cases.The selection is not random; it balances both class and protected attribute.Source: §3.1 · p. 3

3.2 Pseudo Labeling the Unlabeled data

  • Four semi-supervised techniques pseudo-label unlabeled data by iteratively adding predictions above 0.7 confidence, stopping when the maximum iteration limit is reached.The techniques are self-training, label propagation, label spreading, and co-training.Source: §3.2 · pp. 3–4
  • Self-training uses logistic regression; label propagation and label spreading infer labels over graphs, while co-training adds labels by majority vote across feature-specific models.Label spreading modifies the original labels and normalizes graph edge weights; label propagation uses the raw similarity matrix.Source: §3.2 · p. 4

3.3 Synthetic Oversampling & Balancing

  • Fair-SMOTE synthetically oversamples four class-by-protected-attribute groups until they have equal sizes in the training data.The groups are favorable-privileged, favorable-unprivileged, unfavorable-privileged, and unfavorable-unprivileged.Source: §3.3 · pp. 4–5

4 RESULTS

Across ten datasets and three learners, Fair-SSL with 10% labeled data matches state-of-the-art methods on bias scores and sometimes improves recall and F1 over two comparators. Its pseudo-labelers differ by dataset, and results improve as the initial labeled set grows.

RQ1

  • All four Fair-SSL versions significantly reduce AOD, EOD, SPD, and DI across the datasets, while recall and F1 improve and false alarm or accuracy sometimes worsen.The results reflect a fairness-performance trade-off also reported in prior fairness work.Source: RQ1. Can Fair-SSL reduce bias? · p. 5, Table 3 · p. 6

RQ2

  • 10% labeled data lets Fair-SSL match three bias-mitigation methods on bias scores across 10 datasets and three learners, while outperforming Optimized Pre-processing and Fairway on recall and F1.The comparison uses the best Fair-SSL version selected on validation data; the other methods use full training data.Source: RQ1. Can Fair-SSL reduce bias? · p. 5, Table 4 · p. 6

RQ4

  • No pseudo-labeler wins universally: performance choice depends on the dataset, while self-training is fastest and label propagation slowest.The four Fair-SSL versions have similar performance metrics with minor differences.Source: RQ4. Which semi-supervised approach is… · pp. 6–7

RQ3

5 DISCUSSION & LIMITATION

Fair-SSL is reported to perform similarly to Fair-SMOTE at lower labeling cost, with benefits and trade-offs across performance measures. Its evaluation and applicability are bounded by the study’s scope, an unbiased-test-data assumption, and the types of bias SSL can address.

5 DISCUSSION & LIMITATION

  • Fair-SSL reduces bias and improves recall and F1, sometimes at the cost of accuracy and false alarms, while performing similarly to Fair-SMOTE.The authors describe Fair-SSL as cheaper because it uses 10% of labeled data.Source: §5 · p. 7
  • As a data pre-processor, Fair-SSL can be followed by any supervised model for final prediction.Source: §5 · p. 7
  • The evaluation covers ten datasets, three classification models, and four fairness metrics, and assumes unbiased test data.The authors identify exploring more datasets and learners as future work.Source: §5 · p. 7
  • SSL may help with bias from improper labels or sampling, but cannot solve objective-function, homogenization, or feature-correlation bias.Source: §5 · p. 7

6 CONCLUSION

The conclusion presents semi-supervised techniques as a way to build fair machine-learning software with only 10% labeled training data. The authors hope the work will encourage further software-fairness research.

6 CONCLUSION

  • Prior software-engineering fairness work mainly used supervised approaches requiring ground-truth labels, which are not always available in real life.Source: §6 · p. 7
  • The paper shows that semi-supervised techniques can be used to achieve fairness with only 10% labeled training data.Source: §6 · p. 7
  • The authors hope this work will encourage future software researchers to work on software fairness.Source: §6 · p. 7

Related papers

Loading 2111.02038v5…