Source-linked AI summary
Gender Bias in Neural Natural Language Processing
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, Anupam Datta
TL;DR
The paper investigates whether neural NLP systems reflect historical gender biases in training data and develops methods to measure and mitigate them. It proposes a general benchmark and CDA, finding gender bias across coreference resolution and language modeling while CDA reduces bias and preserves accuracy. The paper also reports that bias grows during original-data training and that CDA mitigates this behavior.
Problem
The paper asks whether neural NLP systems trained on human language exhibit gender bias and how such bias should be measured and mitigated.
Method
The paper defines a matched-pair benchmark for measuring gender-related score disparities and introduces CDA, which augments training data with intervened copies retaining the original ground truth.
Results
Evaluations find gender bias in neural coreference resolution and RNN language models, while CDA decreases bias and preserves accuracy.
Takeaways & Limitations
Bias can increase as gradient-descent training reduces loss, whereas CDA mitigates this behavior.
Takeaways & Limitations
CDA assumes generated counterfactual sentences remain semantically and grammatically sound and preserve coreference labels.
Abstract
from arXiv · showhide
We examine whether neural natural language processing (NLP) systems reflect historical biases in training data. We define a general benchmark to quantify gender bias in a variety of neural NLP tasks. Our empirical evaluation with state-of-the-art neural coreference resolution and textbook RNN-based language models trained on benchmark datasets finds significant gender bias in how models view occupations. We then mitigate bias with CDA: a generic methodology for corpus augmentation via causal interventions that breaks associations between gendered and gender-neutral words. We empirically show that CDA effectively decreases gender bias while preserving accuracy. We also explore the space of mitigation strategies with CDA, a prior approach to word embedding debiasing (WED), and their compositions. We show that CDA outperforms WED, drastically so when word embeddings are trained. For pre-trained embeddings, the two methods can be effectively composed. We also find that as training proceeds on the original data set with gradient descent the gender bias grows as the loss reduces, indicating that the optimization encourages bias; CDA mitigates this behavior.
1 Introduction
The paper asks whether neural NLP systems reproduce gender bias and introduces a general benchmark and counterfactual data augmentation to measure and mitigate it. Evaluations find bias in coreference and language modeling, while CDA reduces bias while preserving accuracy.
- Neural NLP models trained on human language may exhibit gender bias, motivating questions about how to measure and mitigate it.
- The paper introduces a general benchmark that interprets outcome differences between matched instances differing in a target concept as that concept’s influence.The definition is parameterized by the scoring function and target concept.
- The benchmark is instantiated for coreference resolution and language modeling, measuring disparities in coreference scores and conditional word-emission log-likelihoods.
- Evaluations of state-of-the-art neural coreference systems and textbook RNN language models trained on benchmark datasets find gender bias.
- Counterfactual data augmentation adds matched training instances with targeted gendered words replaced by their partners while preserving the original ground truth.This makes ground truth independent of the targeted distinction and discourages models from using it.
- CDA decreases gender bias while preserving accuracy, and outperforms word embedding debiasing, especially when embeddings are co-trained.For pre-trained embeddings, CDA and word embedding debiasing can be effectively composed.
- As original-data training reduces loss, gender bias grows; CDA mitigates this behavior.
2 Background
The paper reviews neural coreference resolution, language modeling, word embeddings, and debiasing, establishing the concepts and evaluation context for its experiments.
- Coreference Resolution: Neural coreference systems score mention pairs for coreference likelihood, then process those scores into entity clusters.
- Coreference Resolution: Coreference performance compares predicted clusters with ground-truth clusters using MUC, B3, and CEAFφ metrics and their average F1 score.
- Language Modeling: Language models estimate the likelihood of each word given a sentence prefix and are trained with cross-entropy loss to predict words in unseen text.
- Word Embeddings: Word embeddings transform vocabulary words into real-valued vectors that capture syntactic and semantic relationships from corpus contexts.
- Word Embeddings: Prior work finds that word embeddings encode stereotypical gender associations, and debiasing removes gender components from gender-neutral words.The described procedure affects only gender-neutral words.
- Experimental Setup: The paper studies embedding configurations that are fixed, jointly trained, or trained from scratch, with debiasing applied at different training stages.
3 Measuring Bias
The paper measures gender bias by comparing model scores on matched instances that differ only in gender, focusing on gender–occupation associations in coreference resolution and language modeling.
- Bias is defined as the expected score difference between matched pairs that differ in a targeted concept.The framework can also measure expected absolute bias across a class of matched-pair sets.
- Intervention matching creates paired instances by transforming an input under a specified concept intervention.The core naive intervention swaps every gendered word with its opposite-gender counterpart.
- Gender–occupation bias is tested by instantiating occupation templates and comparing scores for male- and female-pronoun variants.Coreference templates score the occupation and pronoun mentions, while language-model templates condition occupation likelihoods on gendered sentence prefixes.
- The evaluation aggregates occupation bias across occupations using matched pairs generated from the template set.For coreference, the paired examples compare occupation–pronoun coreference scores across gender-swapped sentences.
4 Counterfactual Data Augmentation (CDA)
Counterfactual Data Augmentation augments training data with intervention-matched examples while preserving ground truth, aiming to reduce gender associations without sacrificing predictive performance.
- CDA adds an intervention-transformed copy of each training instance while retaining the original ground truth.The augmented dataset is formed by adding (c(x), y) for each original pair (x, y).
- An unbiased model should produce the same outcome for matched instances, so interventions must change the target concept while preserving meaning and grammar.The method assumes counterfactual sentences remain semantically and grammatically sound; coreference labels therefore remain unchanged.
- Naive augmentation can reduce performance when models are evaluated on the original, non-augmented corpora.This creates a trade-off between eliminating measured bias and preserving performance on the original data distribution.
- Gendered words referring to proper nouns are excluded from flipping to avoid semantically incorrect counterfactual sentences.For coreference data, cluster labels identify proper-noun contexts where gendered words should not be changed.
- Grammatical augmentation relaxes naive gender swapping when grammatical structures require adjustments.The paper also adjusts articles and handles pronoun forms using part-of-speech information.
5 Evaluation
The evaluation compares CDA with word embedding debiasing across neural coreference resolution and language modeling, measuring occupation bias alongside predictive performance. CDA generally reduces bias while preserving accuracy, and training dynamics show bias increasing as loss decreases unless CDA is applied.
- Evaluation setup: The experiments evaluate CDA across three models from coreference resolution and language modeling, reporting aggregate occupation bias and performance on original test sets.Most experiments use grammatical augmentation, with naive intervention also examined for language modeling.
- Neural Coreference Resolution: The original coreference model is biased, while each debiasing method reduces bias to some extent; combining CDA and WED yields the largest reduction with negligible performance impact.This model uses pretrained embeddings, so WED is applied to the pretrained embedding.
- Neural Coreference Resolution: CDA reduces occupation bias aligned with gender stereotypes, including negative bias for female-dominant occupations and positive bias for male-dominant occupations.The comparison concerns Models 1.1 and 1.2.
- Neural Coreference Resolution: CDA has better or comparable debiasing strength than WED while having lower impact on accuracy, although some combined configurations overcorrect and flip aggregate bias toward females.The overcorrection appears in configurations 2.7 and 2.8.
- RNN Language Modeling: In the RNN language model, word embedding debiasing has a very detrimental performance effect, whereas naive augmentation nearly eliminates bias and incurs a lower perplexity hit.The authors speculate that the lower perplexity hit may be a small random effect from the relatively small dataset.
- Learning Bias: During training, bias increases as loss decreases in both tasks, while CDA greatly bounds bias growth and naive augmentation limits bias to almost 0 after an initial growth stage.Figure 3 compares evolving performance and aggregate occupation bias for coreference resolution and language modeling.
- Overall Results: Overall, word embedding debiasing alone can preserve accuracy while doing little to reduce bias or reduce bias at the cost of predictive performance, whereas CDA reduces bias while preserving predictive power.The original results also show bias in downstream NLP tasks, and CDA configurations reduce it across tasks.
6 Future Work
The paper identifies neural machine translation as a challenging next step and proposes studying model internals to explain and prevent bias.
- Future Directions: Future work will explore gender bias in neural machine translation as a concrete challenging next step.The authors also plan to continue exploring bias in neural NLP.
- Future Directions: Studying neural network inner workings could support encoding bias constraints in models or training data before bias is introduced.The paper frames such explanations as a way to help prevent bias from entering the system.
Supplemental Material
The supplemental material documents the templates, occupation vocabulary, gender-pair interventions, and an additional figure used in the experiments.
- Coreference Templates: Coreference experiments use sentence templates containing an occupation placeholder and gendered pronouns in contexts such as hunger, lateness, and emotion.The occupation placeholder marks where listed occupation words are inserted.
- Language-Modeling Templates: Language-modeling templates condition occupation completions on prefixes such as “He is a” and “The man is a.”Both capitalization variants are included.
- Occupation Vocabulary: The occupation category is a hand-picked list of occupations, and multi-word occupations are excluded from the language-modeling experiments.The supplemental material provides the occupation list.
- Gender Interventions: Gender interventions swap hand-picked masculine and feminine word pairs, including father–mother, nephew–niece, and prince–princess.The supplemental material lists the intervention pairs.