Source-linked AI summary
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models
Nicholas Meade, Elinor Poole-Dayan, Siva Reddy
TL;DR
Pre-trained language models encode social biases, but existing mitigation techniques have not been thoroughly compared across bias types and model capabilities. This paper empirically evaluates five techniques with intrinsic bias benchmarks, language modeling, and downstream NLU tests, finding Self-Debias strongest overall while debiasing often worsens language modeling.
Problem
Existing debiasing research has insufficiently examined non-gender biases and the effects of mitigation on language modeling and downstream NLU performance.
Method
The paper empirically surveys CDA, Dropout, INLP, Self-Debias, and SentenceDebias across three intrinsic bias benchmarks, language modeling, and downstream NLU tasks.
Results
Self-Debias was strongest across gender, racial, and religious bias, while most techniques tended to worsen language modeling and did not damage downstream NLU performance.
Takeaways & Limitations
Debiasing methods that leverage a model’s internal knowledge, such as Self-Debias, are identified as a promising direction for future research.
Takeaways & Limitations
The evaluation is skewed toward North American social biases, and many techniques make simplifying assumptions such as binary gender.
Abstract
from arXiv · showhide
Recent work has shown pre-trained language models capture social biases from the large amounts of text they are trained on. This has attracted attention to developing techniques that mitigate such biases. In this work, we perform an empirical survey of five recently proposed bias mitigation techniques: Counterfactual Data Augmentation (CDA), Dropout, Iterative Nullspace Projection, Self-Debias, and SentenceDebias. We quantify the effectiveness of each technique using three intrinsic bias benchmarks while also measuring the impact of these techniques on a model's language modeling ability, as well as its performance on downstream NLU tasks. We experimentally find that: (1) Self-Debias is the strongest debiasing technique, obtaining improved scores on all bias benchmarks; (2) Current debiasing techniques perform less consistently when mitigating non-gender biases; And (3) improvements on bias benchmarks such as StereoSet and CrowS-Pairs by using debiasing strategies are often accompanied by a decrease in language modeling ability, making it difficult to determine whether the bias mitigation was effective.
1 Introduction
Pre-trained language models can encode social biases from large, unmoderated training corpora, motivating empirical evaluation of debiasing methods across bias types and model capabilities.
- Motivation: Proposed debiasing techniques have often been insufficiently investigated beyond gender bias and their effects on language modeling and downstream NLU remain underexplored.The paper identifies racial and religious bias as additional targets requiring evaluation.
- Scope: The study empirically surveys five debiasing techniques: CDA, Dropout, INLP, Self-Debias, and SentenceDebias.These methods are evaluated for representational bias mitigation.
- Scope: The experiments examine gender, racial, and religious biases in four pre-trained language models.The models include BERT, ALBERT, RoBERTa, and an autoregressive model.
- Evaluation: Bias mitigation is evaluated with SEAT, StereoSet, and CrowS-Pairs, while language modeling and downstream NLU are measured with WikiText-2 and GLUE.Self-Debias is reported as strongest overall, debiasing tends to worsen language modeling, and downstream NLU is little affected.
2 Techniques for Measuring Bias
The paper evaluates bias through sentence associations, stereotypical versus anti-stereotypical preferences, and minimally differing sentence pairs, using scores that quantify biased preferences.
- Benchmark scope: The selected benchmarks can measure gender, racial, and religious bias in language models.StereoSet evaluations use the split corresponding to each debiasing domain.
- SEAT: SEAT extends WEAT to sentence representations by placing attribute and target words into synthetic sentence templates.The resulting sentence representations are evaluated using the WEAT test statistic and effect size.
- SEAT: WEAT compares associations between two attribute-word sets and two target-word sets using differences in mean cosine similarity.The test statistic uses A and B as attributes and X and Y as targets.
- SEAT: A SEAT effect size closer to zero indicates a smaller degree of representational bias.The effect size is computed using means and standard deviations.
- StereoSet: StereoSet measures the percentage of examples where a model prefers a stereotypical association over an anti-stereotypical association.Its language modeling score separately measures preference for either meaningful association over an unrelated one.
- CrowS-Pairs: CrowS-Pairs uses minimally different sentence pairs and measures how often masked token probabilities favor the stereotypical sentence.The study uses masked-token scoring rather than pseudolikelihood-based scoring because of reported calibration concerns.
3 Debiasing Techniques
The paper compares five debiasing approaches spanning data augmentation, regularization, projection, and post-hoc generation, with important differences in what model components they alter.
- CDA: CDA re-balances training data by swapping bias-attribute words and then uses the augmented sentences for additional pre-training.The experiments apply CDA to sentences from English Wikipedia.
- Dropout: Dropout mitigates bias by increasing dropout in BERT and ALBERT attention weights and hidden activations during additional pre-training.Prior experiments reported reduced gender bias from increased dropout regularization.
- Self-Debias: Self-Debias uses prompts to elicit toxic continuations and then scales down probabilities of tokens likely under those continuations.It is a post-hoc text-generation procedure that leverages a model’s internal knowledge.
- Self-Debias: Self-Debias does not alter internal representations or parameters, so it cannot be used for downstream NLU and cannot be evaluated with representation-based SEAT.These constraints follow from its post-hoc generation design.
- Projection methods: SentenceDebias estimates a bias subspace and removes its projection from sentence representations.Its subspace construction contextualizes attribute words and applies CDA to paired sentences.
- Projection methods: INLP trains classifiers to predict a protected property and projects representations into the classifiers’ nullspace.The experiments construct classification data from gendered-word occurrences in English Wikipedia.
4 Which Technique is Most Effective in Mitigating Bias?
Self-Debias was the most consistently effective technique across the reported bias evaluations, while other methods varied by model and bias domain. Benchmark results also raise caution because lower stereotype scores may coincide with degraded language-model quality.
- On SEAT, two of four debiased BERT models achieved lower average absolute effect sizes than baseline, with INLP performing best across six tests.INLP and SentenceDebias improved the average effect size, whereas CDA and Dropout did not.
- All debiased GPT-2 models had higher average absolute SEAT effect sizes than baseline, although SEAT detected no significant baseline bias in any test.This result led the authors to question SEAT’s reliability for GPT-2.
- Four of five gender-debiased BERT models lowered stereotype scores, compared with only two of five race-debiased models.Four of five religion-debiased BERT models also reduced stereotype scores, with similar trends for ALBERT and RoBERTa.
- For GPT-2, gender debiasing was less consistent, and CDA produced a higher stereotype score than the baseline.
- Self-Debias consistently reduced stereotype scores across gender, racial, and religious bias domains for BERT and GPT-2.The same pattern held for most Self-Debias ALBERT and RoBERTa models.
- CrowS-Pairs results varied substantially, including a GPT-2 religion SentenceDebias score of 35.24 versus a 27.62-point baseline difference.The authors attribute this variability partly to the dataset’s small size and caution against relying on StereoSet and CrowS-Pairs alone.
- A random model can obtain a 50% StereoSet stereotype score, so improved benchmark scores may reflect worsened language modeling rather than effective bias mitigation.
5 How Does Debiasing Impact Language Modeling?
The study evaluates debiased models’ language modeling ability using WikiText-2 perplexities and StereoSet language modeling scores. Most debiasing methods worsen language modeling, although CDA and Dropout improve BERT perplexity in some cases.
- The evaluation measures perplexity on WikiText-2 and language modeling scores on StereoSet before and after debiasing.Masked language models use pseudo-perplexity, while GPT-2 uses standard perplexity.
- Most debiased models obtain higher perplexities and lower language modeling scores than their respective baselines.The study reports a strong negative correlation between WikiText-2 perplexity and StereoSet language modeling score.
- 65.493 perplexity: SentenceDebias GPT-2 reaches twice the baseline GPT-2 perplexity.
- CDA and Dropout BERT obtain lower perplexities than the baseline BERT model.The authors hypothesize that additional training on English Wikipedia may explain these exceptions.
6 How Does Debiasing Impact Downstream Task Performance?
The study evaluates gender-debiased models on GLUE after fine-tuning. Debiasing largely preserves downstream NLU performance, with some GPT-2 methods improving and several BERT methods slightly declining.
- The study evaluates gender-debiased models on the GLUE validation set after fine-tuning.Table 6 reports average GLUE scores for BERT and GPT-2 models.
- CDA, Dropout, and INLP GPT-2 models obtain higher average GLUE scores than the baseline model.
- Three of four debiased BERT models obtain slightly lower GLUE scores than the baseline model.
- Most ALBERT and RoBERTa models are relatively unaffected by debiasing.The authors hypothesize that fine-tuning helps models relearn essential task information.
7 Discussion and Limitations
The discussion identifies Self-Debias as the strongest method, while emphasizing evaluation difficulties, language and cultural scope limits, simplifying assumptions, and the distinction between language modeling and downstream-task effects.
- Discussion: Self-Debias consistently reduces gender bias and appears effective against racial and religious bias across all four studied models.It also has minimal impact on language modeling ability.
- Discussion: Most debiasing techniques worsen language modeling, making bias-mitigation effectiveness difficult to determine with noisy bias benchmarks.The authors argue that reliable evaluation requires measuring how debiasing affects language modeling.
- Discussion: Debiasing does not damage downstream NLU learning ability, which the authors conjecture may reflect the benefits of fine-tuning.
- Limitations: The study evaluates only English models, and some techniques cannot easily address gender bias in languages with grammatical gender.French is given as an example.
- Limitations: The benchmarks may reflect North American social biases, have only positive predictive power, and the techniques often assume binary gender.The study focuses on representational biases rather than all forms of bias.
8 Conclusion
The paper presents a large-scale evaluation of five debiasing techniques across four pre-trained language models, three bias domains, language modeling, and downstream NLU performance.
- The study evaluates multiple debiasing techniques across gender, racial, and religious bias in BERT, ALBERT, RoBERTa, and GPT-2.It uses three intrinsic bias benchmarks and measures language modeling and downstream NLU performance.
10 Further Ethical Considerations
The survey uses narrow binary or category definitions for the studied biases and does not assess real-world harms. Its intrinsic benchmarks can identify bias but cannot establish that a model is unbiased, so benchmark results alone do not support definitive claims.
- Gender bias is evaluated with a binary definition, despite the authors recognizing that gender is non-binary.The authors follow the original methodologies but identify more fluid gender definitions as important for future research.
- The study does not investigate whether debiasing reduces the extrinsic harm caused by language-model biases.
- The intrinsic bias benchmarks have positive predictive power: they can identify biased models but cannot verify unbiasedness.
- A 50% StereoSet or CrowS-Pairs stereotype score does not indicate an unbiased model, and benchmark unreliability warrants caution about definitive claims.
A SEAT Test Specifications
The appendix specifies the word sets, model resources, and training configurations used for bias testing and debiasing. It covers gender, race, and religion attributes alongside positive, negative, pleasant, and unpleasant target terms.
- SEAT word sets: Religion SEAT evaluations use Christian, Islamic, or Jewish attribute word sets paired with good and bad target terms.
- SEAT word sets: Additional religion evaluations pair Christian, Islamic, or Jewish attributes with pleasant and unpleasant target terms.
- Debiasing attributes: CDA, SentenceDebias, and INLP use paired gender terms, three-way race terms, and three-way religion terms as bias attributes.
- Debiasing attributes: The listed race and religion attributes include black, Caucasian, Asian, African, American, Chinese, Jewish, Christian, and Muslim terms.
- Implementation: Experiments use Hugging Face Transformers and Datasets libraries with checkpoints listed in Table 7.
- Training setup: INLP forms sentence representations by averaging last-hidden-state token representations and trains classifiers on sampled bias-attribute sentences.
- Training setup: Self-Debias uses the implementation of Schick et al., while the listed models are trained for three epochs with maximum sequence length 128, batch size 32, and learning rate 2e-5.
E Additional Results
The appendix provides additional benchmark, GLUE, implementation, and robustness results for debiased models. It also documents an instability affecting INLP with GPT-2 and the evaluation layouts used across bias types and tasks.
- Implementation caveat: Using many classifiers for GPT-2 in INLP was unstable, so the authors refer readers to another INLP-based strategy.
- Debiasing details: Table 8 lists dropout parameters for BERT, ALBERT, and RoBERTa, while Table 9 lists the Self-Debias prompts used in the experiments.
- Additional results: Tables 10–12 report SEAT effect sizes for gender, race, and religion across BERT, ALBERT, RoBERTa, and GPT-2.Effect sizes closer to 0 indicate less biased representations, with averages reported across each bias category’s tests.
- Additional results: Tables 13–15 report StereoSet stereotype and language-modeling scores for gender, race, and religion debiasing.Stereotype scores closer to 50% indicate less biased behavior, while a random model obtains 50% in expectation.
- Additional results: Tables 16–18 report CrowS-Pairs stereotype scores for gender, race, and religion debiasing.A random model obtains a stereotype score of 50%.
- Additional results: Table 19 reports GLUE validation results for debiased models, using task-specific metrics and means over three training runs.
- Additional results: Table 20 reports StereoSet results over three random seeds for gender, race, and religion debiased models.