Source-linked AI summary
Identifying and Reducing Gender Bias in Word-Level Language Models
Shikha Bordia, Samuel R. Bowman
TL;DR
Gender bias in corpora can be propagated or amplified by language models, motivating methods that identify and reduce its effects. The paper defines corpus and word-level bias measures, regularizes embeddings by penalizing their gender-subspace projections, and evaluates the approach across three corpora. Debiasing is effective up to an optimal regularization weight, beyond which instability and increasing perplexity emerge.
Problem
The paper addresses how to identify gender bias in text corpora and reduce its effect on word-level language-model behavior.
Method
The authors define gender-bias metrics and regularize language-model embeddings by penalizing their projections onto a gender subspace.
Results
Across Penn Treebank, WikiText-2, and CNN/Daily Mail, bias decreases as λ increases until model instability; word-level β values increase with higher λ but remain below 1.
Takeaways & Limitations
Word-level β complements corpus-level measures by evaluating changes in individual-word bias between generated text and the training corpus.
Takeaways & Limitations
The debiased models may retain substantial bias that the proposed metric cannot detect, because different metrics capture different kinds of bias.
Abstract
from arXiv · showhide
Many text corpora exhibit socially problematic biases, which can be propagated or amplified in the models trained on such data. For example, doctor cooccurs more frequently with male pronouns than female pronouns. In this study we (i) propose a metric to measure gender bias; (ii) measure bias in a text corpus and the text generated from a recurrent neural network language model trained on the text corpus; (iii) propose a regularization loss term for the language model that minimizes the projection of encoder-trained embeddings onto an embedding subspace that encodes gender; (iv) finally, evaluate efficacy of our proposed method on reducing gender bias. We find this regularization method to be effective in reducing gender bias up to an optimal weight assigned to the loss term, beyond which the model becomes unstable as the perplexity increases. We replicate this study on three training corpora---Penn Treebank, WikiText-2, and CNN/Daily Mail---resulting in similar conclusions.
1 Introduction
Gender bias in training data can be amplified by language models and influence downstream outputs. This study evaluates gender bias in word-level language models and develops a method to reduce its effect on model behavior.
- Biased training data can be amplified by models, reinforcing stereotypes, distorting truth, and enabling problematic profiling decisions.
- Language modeling is vulnerable to gender bias because word prediction systems have practical applications such as onscreen keyboards.
- The study analyzes gender bias in public language-modeling datasets and recurrent neural network outputs.
- The authors propose and evaluate a method for reducing bias learned by word-level language models.
2 Related Work
Prior work measures and mitigates bias in text, embeddings, classifiers, and structured prediction systems. This paper builds on gender-subspace debiasing while applying it to word-level language models trained from scratch.
- Bias research distinguishes explicit framing bias from implicit epistemological bias in language.
- Bolukbasi et al. construct a gender subspace and remove or reduce non-gendered embeddings’ projections onto it.
- Other work reduces bias amplification through optimization constraints, while debiasing methods have also been effective in machine translation.
- WEAT and SEAT use embedding associations to provide evidence for social bias in word and sentence representations.
- Winogender evaluations found that coreference systems’ predictions strongly preferred one gender over another for occupations.
- This paper minimizes cosine similarity between word embeddings and the gender direction, while noting that gender-direction methods may leave other bias cues intact.
3 Methods
The study measures gender bias in corpora and generated text, then trains word-level LSTM models with embedding debiasing regularization and evaluates its effects across three datasets.
- 3 Methods: The experiments compare Penn Treebank, WikiText-2, and CNN/Daily Mail using word-level LSTM language models and generated outputs.The models use three-layer AWD-LSTMs; baseline perplexities are 62.56 for PTB, 67.67 for WikiText-2, and 118.01 for CNN/Daily Mail.
- 3 Methods: Gender bias is measured from word cooccurrence with male or female gendered words within fixed or exponentially weighted context windows.The fixed-window experiments use an optimal context parameter k = 10, while infinite-context weights diminish exponentially with distance.
- 3 Methods: The study evaluates generated text using mean absolute bias, bias-score deviation, and regression slope relative to the training corpus.A lower regression slope β indicates reduced bias, while mean absolute bias summarizes bias magnitude in either direction.
- 3 Methods: The gender subspace is derived by stacking difference vectors for gender-opposing word pairs and applying singular value decomposition.The subspace B uses the first k right-singular vectors, with k chosen to capture 50% of the variation.
- 3 Methods: A regularization term minimizes the squared Frobenius norm of target embeddings projected onto the gender subspace, with λ controlling its loss weight.The input and output embeddings can be debiased separately or simultaneously; simultaneous debiasing produces a much larger perplexity increase, so results report input-only debiasing.
4 Experiments
Experiments across three corpora show that increasing the debiasing weight initially reduces several gender-bias measures, but larger weights can destabilize the model. Word-level examples and regression analysis further indicate that moderate debiasing changes gendered word distributions, while corpus-level measures may miss word-level improvements.
- Effect of λ: Increasing λ initially decreases µ and σ, but beyond an optimal point the model becomes unstable.The same pattern appears for fixed context windows and exponential cooccurrence weightings.
- Generated-text examples: Generated text with λ = 0 reflects gender stereotypes, whereas λ = 0.5 or 1.0 increases mentions of the less probable gender in subsequent text.For fragile at λ = 1.0 and prisoners at λ = 0.5, generated text contains fewer or no gendered words and more neutral content.
- Evaluation measures: The qualitative examples are small snippets, so supplementary corpus-wide bias scores quantify gender-word distributions around target words across the entire corpus.The selected targets were chosen because they are commonly perceived gender biases and showed prominent debiasing effects.
- Corpus-level bias: 0.83, 0.80, and 0.72 are the aggregated absolute bias scores µ for Penn Treebank, WikiText-2, and CNN/Daily Mail, respectively.Higher µ indicates greater average gender bias across words in the corpus.
- Evaluation measures: The authors argue that corpus-level µ and σ are less effective for tracking word-level debiasing than the regression slope β.β compares word-level bias scores in generated text with those in the training corpus; lower β indicates debiasing.
- Effect of λ: For smaller λ, σ, µ, and β decrease relative to training data before increasing with λ toward the original corpus variance.This trend holds under both fixed context windows and exponential weightings.
5 Conclusion
The paper quantifies gender bias in word-level language models and reduces embedding projections onto a gender subspace. The method lowers measured bias at smaller regularization weights, but bias reduction trades off against perplexity and remains metric-specific.
- The method defines a gender subspace and penalizes word-embedding projections onto it to quantify and reduce gender bias.
- Corpus-level bias uses absolute mean (µ) and standard deviation (σ), while regression slope (β) measures word-level changes between generated and training text.
- The method targets one measurable bias distribution, so substantial bias may remain undetected because different metrics capture different kinds of bias.
- Bias reduction incurs a perplexity compromise because the additional regularization term increases perplexity as bias decreases.
A Defining sets
The gender-pair defining sets are constructed separately for each corpus and include only pairs occurring in that corpus.
- Each corpus receives a separately designed gender-pair list restricted to gender pairs present in its training corpus.
- The defining-set procedure uses corpus occurrence as the criterion for including gender pairs.
- The lists provide the gender-pair resources used for the corpus-specific analysis.
A.1 Penn Treebank
The Penn Treebank gender-word list includes paired male and female terms used in the corpus-specific analysis.
- The male list includes terms such as actor, boy, father, he, man, son, spokesman, wife, king, and brother.
- The female list includes paired terms such as actress, girl, mother, she, woman, daughter, spokeswoman, husband, queen, and sister.
- The lists pair gendered nouns and pronouns, including actor–actress, boy–girl, father–mother, and he–she.
A.2 WikiText-2
The WikiText-2 gender-word lists include capitalization and morphological variants of paired male and female terms.
- The male list includes capitalization and plural variants such as Actor, Boy, boys, Fathers, gentlemen, Men, males, and Kings.
- Both lists contain paired gender terms and pronouns, including he–she, him–her, husband–wife, king–queen, and son–daughter.
- The female list includes corresponding variants such as Actress, Girl, girls, Mothers, Ladies, Women, females, and Queens.
A.3 CNN/Daily Mail
The study uses explicit male- and female-associated word lists to analyze gender bias in language-model text.
- Male Words lists gender-associated terms including actor, boy, father, he, man, men, prince, son, and uncle.
- Female Words lists corresponding terms including actress, girl, mother, she, woman, women, princess, daughter, and aunt.
- The lists include both gendered nouns and gendered pronouns for word-level bias analysis.
B Word Level Bias Examples
The paper reports individual-word bias scores for WikiText-2 and CNN/Daily Mail, separating words biased toward male or female gender and varying λ.
- The reported scores compare training text with language-model-generated text.
- WikiText-2 tables report individual-word bias scores for selected male-biased and female-biased words across different λ values.
- CNN/Daily Mail tables report individual-word bias scores for selected male-biased and female-biased words across different λ values.