Source-linked AI summary
Towards Debiasing Sentence Representations
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
TL;DR
Social biases are documented in word embeddings, while increasingly adopted contextualized sentence representations require sentence-level debiasing. The paper proposes SENT-DEBIAS, which contextualizes bias words with diverse natural templates, estimates a bias subspace, and removes its projection. Experiments report reduced bias in BERT and ELMo while preserving downstream-task performance.
Problem
Sentence-level representations such as ELMo and BERT are increasingly used, but existing debiasing work does not adequately address biases in pretrained sentence representations.
Method
SENT-DEBIAS contextualizes bias-attribute words with diverse natural sentence templates, estimates a sentence-representation bias subspace, and removes projections onto it.
Results
SENT-DEBIAS reduces biases in BERT and ELMo while preserving performance on downstream sequence tasks.
Takeaways & Limitations
Diverse naturally occurring sentence templates are important for estimating bias subspaces and support post-hoc debiasing of sentence representations.
Takeaways & Limitations
Neutral sentences are difficult to identify, so removing bias from all potentially relevant sentences may harm downstream performance when many sentences are non-neutral.
Abstract
from arXiv · showhide
As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes. Previous work has revealed the presence of social biases in widely used word embeddings involving gender, race, religion, and other social constructs. While some methods were proposed to debias these word-level embeddings, there is a need to perform debiasing at the sentence-level given the recent shift towards new contextualized sentence representations such as ELMo and BERT. In this paper, we investigate the presence of social biases in sentence-level representations and propose a new method, Sent-Debias, to reduce these biases. We show that Sent-Debias is effective in removing biases, and at the same time, preserves performance on sentence-level downstream tasks such as sentiment analysis, linguistic acceptability, and natural language understanding. We hope that our work will inspire future research on characterizing and removing social biases from widely adopted sentence representations for fairer NLP.
1 Introduction
Sentence representations are increasingly used in real-world NLP, but their social biases remain insufficiently addressed. SENT-DEBIAS extends post-hoc debiasing to sentence representations and reduces bias while preserving downstream performance.
- Motivation: Word-level embeddings can reflect and propagate social biases, while systems using them may amplify bias and unfairly discriminate against disadvantaged groups.
- Motivation: Sentence-level encoders such as ELMo, BERT, and GPT increasingly support NLP and real-world applications, making their potential role in shaping social biases important to examine.
- Problem: Post-hoc debiasing is needed because retraining state-of-the-art sentence encoders requires massive data, hundreds of machines, and several weeks.
- Related Work: Prior work measured bias in sentence representations but did not successfully remove it from pretrained sentence representations through post-hoc methods.
- Approach: SENT-DEBIAS contextualizes bias-attribute words with diverse sentence templates and extends HARD-DEBIAS to binary gender and multiclass religious attributes.
- Results: Experiments on BERT and ELMo show reduced bias while preserving downstream sequence-task performance, with diverse templates important for estimating bias subspaces.
2 Debiasing Sentence Representations
SENT-DEBIAS constructs a sentence-level bias subspace from contextualized bias-attribute words, then removes that subspace's projection from new sentence representations. The method uses diverse natural templates and skips equalization because identifying suitable sentence pairs is difficult.
- Method: SENT-DEBIAS defines bias attributes, contextualizes their words into sentences, estimates a bias subspace, and debiases new representations by removing its projection.
- Defining Bias Attributes: Bias attributes are represented by binary word pairs or multiclass word tuples whose members share meaning except for the targeted attribute.
- Contextualizing Words into Sentences: Naturally occurring templates replace one bias-attribute term with corresponding terms, producing contextualized sentence tuples across varied syntax, topics, settings, and spoken or written text.
- Estimating the Bias Subspace: The encoder maps contextualized sentences to representation sets, and PCA selects the top-k orthogonal directions that represent the bias subspace.
- Neutralize: For neutral sentences, debiasing subtracts the representation's projection onto the estimated bias subspace, yielding a vector orthogonal to that subspace.
- Equalize: The method skips HARD-DEBIAS equalization because the complexity of natural sentences makes it difficult to identify all or most sentence pairs requiring equalization.
3 Experiments
SENT-DEBIAS reduces measured binary gender and multiclass religion bias in BERT and ELMo sentence representations, while largely preserving downstream performance. Experiments also show that diverse, numerous templates improve debiasing effectiveness and stability.
- 3.3 Debiasing Results: SENT-DEBIAS reduces bias across binary gender and multiclass religion tests for both BERT and ELMo representations.The strongest reductions occur when debiasing pretrained BERT; fine-tuning alone produces inconsistent changes and can amplify multiclass bias.
- 3.4 Comparison with Baselines: SENT-DEBIAS outperforms word-level averaging and simple-template baselines on average absolute effect size across Caliskan tests.The comparison supports using diverse naturally occurring sentence templates rather than debiasing words alone or relying on simple templates.
- 3.5 Effect of Templates: Using more templates decreases effect sizes and variance for BERT fine-tuned on SST-2 and QNLI, improving algorithm stability.The QNLI trend tapers off quickly, whereas SST-2 shows a continued decreasing trend.
- 3.5 Effect of Templates: Using templates from more domains similarly decreases effect sizes and, for QNLI, reduces variance and improves stability.The domains include Reddit, SST, POM, and WikiText-2.
- 3.6 Visualization: After debiasing, non-gender-specific concepts become more equidistant from the average representations of man and woman.Before debiasing, science and technology are closer to man, while literature and art are closer to woman.
- 3.7 Performance on Downstream Tasks: Downstream performance decreases by 1−3% after debiasing in the reported BERT and ELMo SST-2 and CoLA experiments, while ELMo SST-2 rises from 89.6 to 90.0.Debiased BERT on QNLI achieves 90.6% versus 91.3% without debiasing.
4 Discussion
The discussion identifies limitations in bias evaluation and in removing bias selectively from neutral sentences, while noting downstream performance is generally preserved. It also emphasizes that debiasing does not eliminate all biases or replace broader scrutiny.
- WEAT, SEAT, and MAC detect the presence of bias but cannot establish its absence.
- Debiasing should target sentences neutral to the relevant attribute, but identifying neutral sentences is difficult.
- Applying debiasing to all potentially relevant sentences may harm downstream performance when datasets contain many non-neutral sentences.
- The approach remains constrained by large sentence encoders that are infeasible to fully retrain, motivating better post-hoc techniques.
- Debiased models can still reproduce or amplify biases introduced during later fine-tuning and are not a one-size-fits-all solution.
5 Conclusion
The paper presents SENT-DEBIAS as a post-hoc method for removing social biases from pretrained sentence representations. Using diverse naturally occurring templates, it reduces biases in BERT and ELMo while preserving downstream performance.
- SENT-DEBIAS estimates sentence-representation bias subspaces from diverse templates drawn from naturally occurring text corpora.
- Experiments show that SENT-DEBIAS removes biases in BERT and ELMo while preserving downstream-task performance.
- Using many diverse sentence templates improves bias-subspace estimation.
A Debiasing Details
The appendix provides implementation details for estimating bias subspaces and contextualizing bias words with templates from multiple text sources. It lists the bias-word sets and datasets used for these procedures.
- The appendix describes the bias-subspace estimation and sentence-debiasing steps.
- Bias attribute words are specified for binary gender bias and multiclass religious bias.
- WikiText-2 supplies formal-text templates, while Hacker News and Reddit provide discussion and news templates across varied topics.
B.1 BERT
The BERT appendix identifies the model variants, sentence-embedding representation, fine-tuning tasks, and bias-word pairs used in the experiments. It covers pretrained BERT and versions fine-tuned on SST-2 or CoLA.
- The evaluated BERT variants are pretrained BERT, BERT fine-tuned on SST-2, and BERT fine-tuned on CoLA.
- BERT’s second output, the pooled output, is treated as the sentence embedding.
- SST-2 is a binary single-sentence sentiment-classification task, while CoLA is a binary single-sentence linguistic-acceptability task.
- The appendix lists word pairs for estimating binary gender and three-class religion bias subspaces.
B.2 ELMo
For ELMo, SENT-DEBIAS removes the gender direction from each timestep of sentence representations before downstream processing.
- ELMo sentence representations are formed by summing aggregated layer outputs into 1024-dimensional time-sequence vectors.Mean pooling produces a 1024-dimensional vector for each definitional sentence when computing the gender direction.
- SENT-DEBIAS removes the gender direction from every timestep of each sentence representation before feeding the debiased representation into an LSTM.
C Additional Results
Additional BERT experiments on CoLA evaluate how template quantity and domain diversity affect debiasing. Debiasing performance improves and stabilizes as both increase.
- The BERT-on-CoLA experiments vary either the percentage of data from one domain or the number of domains at fixed total size.
- Debiasing performance improves and stabilizes with increasing numbers of sentence templates and domains.