Source-linked AI summary
Mitigating Gender Bias in Natural Language Processing: Literature Review
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, William Yang Wang
TL;DR
NLP systems can propagate or amplify gender bias, while methods for mitigating it remain relatively nascent. This paper reviews how gender bias is recognized and mitigated across four representation-bias categories, then discusses method limitations and future directions. It concludes that current approaches are not sufficient for end-to-end debiasing across applications.
Problem
Gender bias can propagate through NLP systems and reinforce damaging stereotypes, while existing mitigation methods remain relatively nascent.
Method
The paper provides a comprehensive literature review organized around four forms of representation bias and methods for recognizing and mitigating gender bias.
Results
The review finds that existing gender-debiasing methods are insufficient for end-to-end debiasing in many applications.
Takeaways & Limitations
Future research should address coordination across system components, broader generalization, languages beyond English, and non-binary gender bias.
Takeaways & Limitations
Most debiasing methods have been empirically verified only in limited applications, so their generalization to other tasks or models remains unclear.
Abstract
from arXiv · showhide
As Natural Language Processing (NLP) and Machine Learning (ML) tools rise in popularity, it becomes increasingly vital to recognize the role they play in shaping societal biases and stereotypes. Although NLP models have shown success in modeling various applications, they propagate and may even amplify gender bias found in text corpora. While the study of bias in artificial intelligence is not new, methods to mitigate gender bias in NLP are relatively nascent. In this paper, we review contemporary studies on recognizing and mitigating gender bias in NLP. We discuss gender bias based on four forms of representation bias and analyze methods recognizing gender bias. Furthermore, we discuss the advantages and drawbacks of existing gender debiasing methods. Finally, we discuss future studies for recognizing and mitigating gender bias in NLP.
1 Introduction
Gender bias can appear throughout NLP systems, from training data and resources to pretrained models and algorithms, potentially reinforcing stereotypes in downstream applications. This review organizes representation bias, surveys recognition and mitigation methods, and identifies limitations and future research directions.
- Gender bias can occur in NLP training data, resources, pretrained models, and algorithms, producing biased predictions and sometimes amplifying training-set biases.
- 1 Introduction: Bias propagation can reinforce damaging stereotypes in downstream applications, including resume filtering that favors male applicants when gender is the only distinguishing factor.
- Representation bias is categorized as denigration, stereotyping, recognition, or under-representation, while allocation bias concerns unequal resource distribution.
- Statistical patterns in training corpora can become embedded in semantic representations and models, producing both allocative and representational harms.
- 1 Introduction: The paper reviews recent gender-bias recognition and mitigation studies under a unified framework and critically examines current methods, knowledge gaps, and future directions.
2 Observing Gender Bias
The paper surveys methods for observing gender bias in NLP through psychological associations, embedding geometry, and gender-related performance differences. It emphasizes carefully designed gender-swapped evaluations because standard datasets may conceal bias.
- Evaluation approaches: Recent analyses quantify gender bias through psychological tests, performance differences across genders, and vector-space geometry.These approaches include adapting the Implicit Association Test to word and sentence embeddings and comparing task performance across gendered inputs.
- Embedding-based evaluation: WEAT adapts the IAT’s association-based measurement to embeddings, while SEAT extends this approach to sentence encoders.Caliskan et al. report that human biases identified by IAT tests occur in GloVe and Word2Vec embeddings; May et al. extend the test to sentence encoders.
- Embedding-based evaluation: Removing neutral-word projections onto a gender subspace may leave similarly biased words clustered together, motivating cluster-bias analysis.Cluster bias measures the proportion of stereotypically male or female words among a word’s k nearest neighbors.
- Performance-difference evaluation: Gender-swapping compares model behavior on otherwise matched male- and female-coded inputs; unequal evaluation scores indicate gender bias.The paper notes that global accuracy averages can hide occupation-specific disparities, so performance differences should be examined with respect to each occupation.
- Performance-difference evaluation: FPED and FNED measure differences in false-positive and false-negative rates between original and gender-swapped inputs.The paper notes that these measures, introduced for abusive-language detection, can generalize beyond that task.
- Gender bias evaluation tests: GBETs use gender-swapped data points to compare probing concepts or prediction accuracies, exposing recognition, stereotyping, and under-representation bias.Standard NLP datasets may contain unequal gender references and confounding factors, so test sets must isolate the effect of gender.
3 Debiasing Methods Using Data Manipulation
Data-manipulation methods mitigate gender bias by balancing or transforming training data, transferring from less-biased data, tagging gender information, or altering embeddings. These approaches can improve fairness but involve task-specific trade-offs, retraining costs, semantic concerns, and language limitations.
- Method categories: Debiasing methods are categorized as retraining or inference, with retraining addressing bias earlier but requiring additional resources and time.Inference methods patch existing models by adjusting outputs or representations without retraining.
- Data augmentation: Data augmentation creates gender-swapped, name-anonymized training examples and trains on the combined original and augmented data to reduce gender associations.Name anonymization replaces named entities with labels such as “E1” to remove gender associations with specific entities.
- Data augmentation: Data augmentation reduced gender-performance disparities across coreference resolution and hate speech detection, including lower pro-/anti-stereotypical F1 gaps and smaller FNED and FPED differences.The method was also applied to knowledge graphs, although success metrics were not provided there.
- Data augmentation: Data augmentation can require costly annotation and increased training time, while indiscriminate swapping may produce nonsensical sentences.The training set doubles in size, and examples such as “she gave birth” becoming “he gave birth” illustrate the semantic risk.
- Bias fine-tuning: Bias fine-tuning transfers from an unbiased related-task dataset before fine-tuning on biased target data, but gender-swapping outperformed it in one abusive-versus-sexist-language comparison.The authors caution that task differences may have limited transfer learning’s effectiveness and call for testing on more similar datasets.
- Word embeddings: Embedding debiasing removes or reduces gender components, yet subtle cluster bias can remain, semantic gender information may be altered, and cosine-based methods may not extend to non-Euclidean spaces or other languages.The review notes particular uncertainty for languages with grammatical gender.
4 Debiasing by Adjusting Algorithms
Algorithm-adjustment methods mitigate gender bias by modifying model objectives or learning dynamics rather than changing the underlying data. The reviewed approaches constrain predictions or use adversarial learning to reduce gender information in task representations.
- Algorithm adjustment: Algorithm-adjustment methods modify predictions in NLP systems to mitigate gender bias.The review discusses two approaches in this family.
- Reducing Bias Amplification: Reducing Bias Amplification constrains an existing optimization function so predictions satisfy defined gender-related conditions.In visual semantic role labeling, it restricted predicted male-to-female activity ratios to prevent amplification.
- Adversarial learning: Adversarial learning trains a generator against a discriminator that attempts to identify gender, reducing gender information in tasks such as analogy completion.The approach is potentially generalizable to models trained with gradient-based learning.
5 Conclusion and Future Directions
The review identifies limits in current gender-debiasing research and outlines future directions spanning broader coverage, stronger evaluation, and interdisciplinary collaboration.
- The survey excludes extensive work on gender stereotypes and algorithmic bias outside NLP, as well as transparency and privacy preservation.
- Gender-bias research lacks unified metrics and benchmarks, while different applications may require different metrics and involve trade-offs between bias notions.
- Current debiasing techniques often target one modular process, have limited empirical verification, may not generalize across tasks or models, and can degrade performance by introducing noise.
- Mitigating Gender Bias in Languages Beyond English: Future work should address gender bias beyond English, where gender-swapping may require changing nouns and their modifiers to preserve linguistic agreement.
- Non-Binary Gender Bias: Future research should consider non-binary genders and racial biases, which have largely been ignored in NLP debiasing studies.
- Interdisciplinary Collaboration: Interdisciplinary collaboration may improve understanding of latent gender bias in machine-learning datasets and model predictions.