Source-linked AI summary
Gender Bias in Contextualized Word Embeddings
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, Kai-Wei Chang
TL;DR
The paper investigates how gender bias in ELMo’s contextualized embeddings is represented and transferred to coreference resolution. It combines intrinsic analyses with WinoBias evaluation and tests data augmentation and embedding neutralization, finding that augmentation largely mitigates the downstream bias while neutralization is only partially effective.
Problem
The paper addresses limited understanding of how contextualized ELMo embeddings encode gender bias and whether that bias transfers to downstream coreference resolution.
Method
The paper analyzes ELMo’s training-data skew, embedding geometry, and gender predictability, then evaluates an ELMo-based coreference system on WinoBias and tests two mitigation strategies.
Results
The analyses find unequal gender encoding, with male entities predicted 14% more accurately, while data augmentation largely mitigates WinoBias bias and neutralization is only partially effective.
Takeaways & Limitations
ELMo’s contextualized representations can transfer gender bias to coreference systems, so their downstream use requires evaluating and mitigating that bias.
Abstract
from arXiv · showhide
In this paper, we quantify, analyze and mitigate gender bias exhibited in ELMo's contextualized word vectors. First, we conduct several intrinsic analyses and find that (1) training data for ELMo contains significantly more male than female entities, (2) the trained ELMo embeddings systematically encode gender information and (3) ELMo unequally encodes gender information about male and female entities. Then, we show that a state-of-the-art coreference system that depends on ELMo inherits its bias and demonstrates significant bias on the WinoBias probing corpus. Finally, we explore two methods to mitigate such gender bias and show that the bias demonstrated on WinoBias can be eliminated.
1 Introduction
This paper extends bias analysis from static word embeddings to ELMo’s contextualized representations, examining training-data skew, encoded gender information, downstream effects, and mitigation.
- ELMo’s training corpus contains nearly three times more male than female entities, contributing to gender bias in pretrained embeddings.
- PCA reveals a low-dimensional subspace that captures much of the gender information in ELMo’s contextualized embeddings.
- Male entities are predicted from occupation words 14% more accurately than female entities, showing unequal gender encoding.
- An ELMo-based coreference system exhibits WinoBias disparity between pro- and anti-stereotypical predictions nearly 30% higher than a comparable GloVe system.
- Gender-swapped training-data augmentation and test-time embedding neutralization are evaluated as mitigation strategies, with neutralization only partially effective.
2 Related Work
Prior work shows that word embeddings encode societal gender biases and proposes post-processing and training-based mitigation, while also questioning whether bias can be entirely removed.
- Word embeddings trained on large human-produced corpora can encode societal biases, including gender bias.
- Bias-subspace projection reduces gender information in gender-neutral words while maintaining downstream task performance in prior evaluations.
- Other work proposes separating gender information from other factors during training, whereas later analysis argues complete bias removal may be difficult or impossible.
3 Gender Bias in ELMo
The intrinsic analyses show that ELMo is trained on gender-skewed data, systematically represents gender in its geometry, and encodes male and female information unequally.
- 3.1 Training Data Bias: Male pronouns occur three times more often than female pronouns in ELMo’s training corpus.Male pronouns also co-occur more frequently with occupation words, regardless of whether occupations are stereotypically male or female.
- 3.2 Geometry of Gender: PCA of gender-swapped sentence-pair differences identifies two gender-related components in ELMo, representing contextual and occupational gender.
- 3.2 Geometry of Gender: Figure 1 compares occupation-word projections under male and female contexts across the first two principal components.
- 3.3 Unequal Treatment of Gender: ELMo propagates gender information to occupation words, with development accuracy of 95.1% for male contexts and 80.6% for female contexts.
- 3.3 Unequal Treatment of Gender: The gender-prediction classifier is substantially above chance for both genders, but represents male information more accurately than female information.
4 Bias in Coreference Resolution
The ELMo-based coreference system transfers ELMo’s gender bias to WinoBias, while data augmentation largely mitigates it and neutralization works only in simpler cases.
- 4.1 Setup: WinoBias measures differences between pro- and anti-stereotypical coreference performance across semantic-only and syntactic-cue examples.
- 4.2 Bias Mitigation: Test-time neutralization averages representations from original and gender-swapped sentences without retraining the coreference model.
- 4.3 Results: ELMo improves OntoNotes coreference F1 but produces large pro-/anti-stereotype disparities on both semantic and syntactic WinoBias examples.
- 4.3 Results: Data augmentation reduces coreference bias to insignificant levels, but requires retraining the system.
- 4.3 Results: Neutralization is less effective than augmentation and cannot fully remove bias on WinoBias’s Semantics Only subset.
5 Conclusion and Future Work
The paper concludes that contextualized embeddings can inherit gender bias, which transfers to downstream coreference systems. Data augmentation and embedding neutralization effectively eliminate this bias in the evaluated ELMo-based system.
- ELMo inherits implicit gender bias from its training corpus and transfers that bias to downstream tasks such as coreference resolution.
- The study explores two mitigation strategies: gender-swapped data augmentation and contextualized embedding neutralization.
- The evaluated mitigation strategies effectively eliminate bias from ELMo in a state-of-the-art coreference system.