Source-linked AI summary
Sustainable Modular Debiasing of Language Models
Anne Lauscher, Tobias Lüken, Goran Glavaš
TL;DR
Pretrained language models encode stereotypical biases, while existing debiasing methods modify all parameters at high computational cost and with a risk of forgetting pretrained knowledge. ADELE trains dedicated adapters on counterfactually augmented data while freezing the original model parameters. It attenuates gender bias across intrinsic and extrinsic evaluations, retains fairness through task adapters after downstream training, and transfers debiasing zero-shot to six languages.
Problem
Pretrained language models encode stereotypical societal biases, while existing debiasing methods modify all parameters, creating high computational cost and risking forgetting of pretrained knowledge.
Method
ADELE injects debiasing adapters into pretrained model layers, freezes the original parameters, and trains only the adapters with masked language modeling on a counterfactually augmented corpus.
Results
ADELE attenuates gender bias across three intrinsic and two extrinsic benchmarks, retains most debiasing effects with task adapters after downstream training, and transfers debiasing from mBERT to German without German training data.
Takeaways & Limitations
The modular design makes debiasing more efficient while preserving pretrained knowledge and supporting fairness during downstream training and zero-shot cross-lingual transfer.
Takeaways & Limitations
The study uses a binary conceptualization of gender and relies on existing evaluation datasets rather than creating resources for the full gender spectrum.
Abstract
from arXiv · showhide
Unfair stereotypical biases (e.g., gender, racial, or religious biases) encoded in modern pretrained language models (PLMs) have negative ethical implications for widespread adoption of state-of-the-art language technology. To remedy for this, a wide range of debiasing techniques have recently been introduced to remove such stereotypical biases from PLMs. Existing debiasing methods, however, directly modify all of the PLMs parameters, which -- besides being computationally expensive -- comes with the inherent risk of (catastrophic) forgetting of useful language knowledge acquired in pretraining. In this work, we propose a more sustainable modular debiasing approach based on dedicated debiasing adapters, dubbed ADELE. Concretely, we (1) inject adapter modules into the original PLM layers and (2) update only the adapters (i.e., we keep the original PLM parameters frozen) via language modeling training on a counterfactually augmented corpus. We showcase ADELE, in gender debiasing of BERT: our extensive evaluation, encompassing three intrinsic and two extrinsic bias measures, renders ADELE, very effective in bias mitigation. We further show that -- due to its modular nature -- ADELE, coupled with task adapters, retains fairness even after large-scale downstream training. Finally, by means of multilingual BERT, we successfully transfer ADELE, to six target languages.
1 Introduction
Pretrained language models encode stereotypical societal biases from biased associations in human-produced corpora, while existing debiasing methods modify all PLM parameters. ADELE addresses these costs by isolating debiasing knowledge in modular adapters while preserving the pretrained model parameters.
- Motivation: Pretrained language models encode stereotypical associations because their training corpora contain biased concept cooccurrences.These associations include links between male terms and career terms that occur more often than corresponding female-term pairings.
- Limitations of prior work: Existing debiasing approaches modify all PLM parameters, incurring high computational cost and risking catastrophic forgetting of pretrained knowledge.For giant models such as GPT-3 or GShard, full-parameter debiasing is described as prohibitively expensive.
- Sustainability: Current debiasing procedures have a large carbon footprint that jeopardizes sustainable fair representation learning in NLP.
- ADELE: ADELE injects adapter layers into PLM layers and incorporates debiasing knowledge only in those additional parameters without changing pretrained knowledge.The approach is presented as more efficient than existing state-of-the-art debiasing approaches while remaining similarly effective in bias attenuation.
- Contributions: ADELE combines adapter-based debiasing with counterfactual data augmentation, evaluates intrinsic and extrinsic bias measures, addresses fairness forgetting with task adapters, and transfers debiasing zero-shot to six languages.
2 ADELE: Adapter-Based Debiasing
ADELE uses parameter-efficient adapter modules for debiasing while keeping the pretrained Transformer frozen. It trains these adapters with counterfactual data augmentation, preserving the original model parameters and isolating debiasing functionality in a modular component.
- Adapter-based debiasing: Adapters are lightweight components injected into PLM layers; downstream fine-tuning freezes original PLM parameters and trains only the adapters.Because adapters contain fewer parameters than the original PLM, adapter-based fine-tuning is more computationally efficient.
- Adapter architecture: The Pfeiffer architecture adds one adapter after each Transformer feed-forward sub-layer, unlike the Houlsby architecture, which adds two adapters per layer.The paper chooses Pfeiffer because it is more parameter-efficient and has shown slightly better downstream performance.
- Adapter architecture: Each adapter projects hidden representations down to a bottleneck dimension, applies a nonlinear transformation, and projects them back before adding the residual.The ratio h/m captures the parameter-efficiency factor relative to full Transformer fine-tuning.
- ADELE training: ADELE trains BERT’s injected adapters with language modeling on counterfactually augmented data while freezing the original BERT parameters.After training, the debiasing functionality is isolated in the adapter parameters.
- Counterfactual augmentation: Counterfactual data augmentation replaces terms from one target group with corresponding terms from the other to create counterfactual sentences.Two-sided CDA retains both original and counterfactually created sentences, then trains the adapter with masked language modeling on S ∪ S′.
3 Experiments
ADELE is evaluated for gender debiasing in BERT using intrinsic and downstream benchmarks, including BEC-Pro, DisCo, WEAT, Bias-STS-B, and Bias-NLI. The evaluation also covers zero-shot transfer to German and six target languages, with measures designed to quantify gender-associated differences in probabilities, candidate predictions, and embedding associations.
- Evaluation scope: ADELE is tested on three intrinsic measures—BEC-Pro, DisCo, and WEAT—and two downstream benchmarks, Bias-STS-B and Bias-NLI.The downstream evaluations assess gender-biased correlations in semantic similarity and natural-language inference.
- Intrinsic measures: BEC-Pro contains 2,700 male–female sentence pairs linked to profession terms and measures gender–profession association differences.Bias is averaged across instances and also reported as the percentage below thresholds 0.1 and 0.7.
- Intrinsic measures: DisCo compares masked-token candidate distributions for male and female person terms, including the average fraction of shared candidates.Higher average candidate overlap indicates lower bias.
- Intrinsic measures: WEAT measures differential associations between male and female target terms and science-versus-art attribute terms using mean cosine similarity.The test uses permutation comparisons and reports a normalized effect size based on the separation of association distributions.
- Downstream measures: Bias-NLI evaluates whether models predict NEUTRAL for gender–occupation instances and pairs fraction-neutral and net-neutral scores with MNLI accuracy.The synthetic evaluation contains N = 1,936,512 instances, and higher FN and NN scores indicate lower bias.
- Data and transfer: The evaluation uses English Wikipedia and BookCorpus for adapter training, while zero-shot transfer is assessed with German BEC-Pro and WEAT translations in six target languages.The target languages are German, Spanish, Italian, Croatian, Russian, and Turkish.
4 Results and Discussion
ADELE substantially reduces gender bias across monolingual and cross-lingual evaluations while preserving a modular path for fairness during downstream training. Its debiasing effects can weaken under large-scale fine-tuning, but task adapters largely preserve them.
- Monolingual Evaluation: ADELE attenuates BERT’s gender bias across three intrinsic and two extrinsic benchmarks, often outperforming the more computationally intensive Zari models.On BEC-Pro and DisCo, ADELE substantially outperforms both Zari variants.
- Monolingual Evaluation: WEAT heatmaps show that ADELE’s most gender-neutral BERT representations come from lower-layer aggregations, whereas higher layers can reverse the bias direction.The original BERT and Zari models show different layer-region patterns.
- Zero-Shot Cross-Lingual Transfer: ADELE reduces mBERT’s gender bias in English and transfers debiasing to German without German training data.On BEC-Pro, bias decreases from 0.81 to 0.3 for English and from 1.1 to 0.67 for German.
- Zero-Shot Cross-Lingual Transfer: Across six target languages, ADELE reduces all significantly biased mBERT XWEAT combinations; Italian [0:12] drops from 1.02 to -0.25.For already insignificant biases, effects generally remain insignificant or decrease further.
- Fairness Forgetting: After 10K MNLI examples, ADELE retains much of its debiasing advantage, but by 25K much of the effect vanishes while remaining slightly better than BERT.NN estimates are unstable across runs, with wide confidence intervals for both BERT and ADELE.
- Preventing Fairness Forgetting: ADELE-TA preserves most debiasing effects by freezing debiasing adapters and training task adapters, with a 0.557 FN score versus ADELE’s 0.127.This fairness preservation costs 3 points in NLI accuracy relative to fully fine-tuning BERT and ADELE.
5 Related Work
The paper situates ADELE at the intersection of pretrained-language-model debiasing and parameter-efficient adapter methods. Prior work includes projection-based mitigation, counterfactual data augmentation, dropout, and language- or task-specific adapters.
- Adapter Layers in NLP: Adapters have been applied in NLP for general language understanding, generation, machine translation quality estimation, and multilingual parsing.
- Adapter Layers in NLP: Related work has explored language adapters that encode language-specific knowledge for machine translation and multilingual parsing.
- Debiasing Methods: Prior work on mitigating stereotypical biases in pretrained language models includes projection-based debiasing methods.
- Debiasing Methods: Counterfactual data augmentation and dropout have been used to mitigate bias in BERT during pretraining.
6 Conclusion
The conclusion presents ADELE as a modular alternative to full-model debiasing: it trains dedicated adapters while leaving the pretrained model intact. Experiments show effective gender-bias attenuation for BERT, multilingual transfer to six languages, and preserved fairness during downstream training with task adapters.
- 6 Conclusion: ADELE performs parameter-efficient debiasing by training dedicated adapters instead of fully fine-tuning the pretrained language model.
- 6 Conclusion: ADELE attenuated gender bias in BERT across three intrinsic and two extrinsic debiasing benchmarks.
- 6 Conclusion: ADELE’s debiasing effects transferred from multilingual BERT to six target languages.
- 6 Conclusion: Combining ADELE’s debiasing adapters with task adapters preserved representational fairness after large-scale downstream training.
Further Ethical Considerations
The ethical evaluation is limited by a binary treatment of gender and by the absence of intersectional analysis. The authors identify broader gender identities and intersectional stereotyping as important directions for future work.
- Further Ethical Considerations: The evaluation uses a binary conceptualization of gender because available bias tests largely impose that restriction.
- Further Ethical Considerations: The study does not address intersectionality because suitable training and evaluation data were unavailable.
- Further Ethical Considerations: The authors state that ADELE’s modular design may suit future work on complex intersectional biases.
A Code Base
The paper directs readers to Table 5 for implementation resources used in the study. These resources include frameworks, code bases, and model checkpoints.
- A Code Base: Table 5 provides further information and links for the frameworks used in the study.
- A Code Base: The paper provides links to the code bases used in the work.
- A Code Base: Model checkpoints used in the experiments are also listed through Table 5.
B Word Pairs
This section lists the word pairs used to evaluate gender-related associations, including names, general gendered nouns, and additional gendered terms.
- The study lists paired first names drawn from U.S. Social Security name statistics.
- General noun pairs include gendered occupational, familial, relational, and pronoun forms.
- An extra word list adds further gendered terms, including occupations, titles, and kinship-related words.
C BEC-Pro.
BEC-Pro constructs occupation templates that pair gendered person terms with jobs grouped by female, balanced, or male participation.
- BEC-Pro starts from templates such as “PERSON is a OCCUPATION” and “PERSON wants to become a OCCUPATION.”
- Person slots use nine gendered term pairs, including pronouns, family terms, and romantic relationships.
- Occupations are grouped by female and male participation using 2019 U.S. Bureau of Labor Statistics data.
D DisCo
DisCo uses templates connecting person names with academic subjects, personal attributes, occupations, and preferences to test gendered associations.
- DisCo is built around 14 templates involving interests, courses, majors, and school subjects.
- Additional templates connect people with attributes, occupations, happiness, and likes or dislikes.
- The person slots are filled with names from Section B.
E WEAT Test Specification
The WEAT gender-bias specification uses term sets representing the stereotype that males are more associated with math and females with art.
- WEAT gender bias test 7 uses term sets reflecting male–math and female–art associations.