Source-linked AI summary
Measuring Bias in Contextualized Word Representations
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, Yulia Tsvetkov
TL;DR
Contextual embeddings such as BERT can encode social biases, but existing tests do not reliably expose them because contextual representations depend on context. The paper introduces a template-based masked-language-model method, finds it more consistent with human biases and more sensitive than WEAT on BERT, and examines downstream gender-bias effects.
Problem
Existing bias-exposure methods are difficult to apply reliably to contextual embeddings, while biases learned by these models may affect downstream tasks.
Method
The paper queries BERT with template sentences, masking target and attribute tokens to compute normalized target–attribute associations and bias scores.
Results
The method exposes statistically significant associations across all tested bias categories, whereas WEAT on BERT finds none at p < 0.01, and the paper reports negative downstream effects.
Takeaways & Limitations
Querying BERT’s underlying language model can measure multiple embedded stereotypes and reveal gender bias relevant to downstream applications.
Takeaways & Limitations
The study highlights potential negative impacts in downstream applications and notes that permutation-test effect size does not guarantee statistical significance.
Abstract
from arXiv · showhide
Contextual word embeddings such as BERT have achieved state of the art performance in numerous NLP tasks. Since they are optimized to capture the statistical properties of training data, they tend to pick up on and amplify social stereotypes present in the data as well. In this study, we (1)~propose a template-based method to quantify bias in BERT; (2)~show that this method obtains more consistent results in capturing social biases than the traditional cosine based method; and (3)~conduct a case study, evaluating gender bias in a downstream task of Gender Pronoun Resolution. Although our case study focuses on gender bias, the proposed technique is generalizable to unveiling other biases, including in multiclass settings, such as racial and religious biases.
1 Introduction
Contextual embeddings such as BERT adapt representations to context, making traditional analogy- and cosine-based bias tests unsuitable or inconsistent. The paper therefore proposes a template-based method to quantify bias in BERT and examines its downstream effects.
- Contextual models assign each word an embedding that depends on its surrounding context and language-model state.
- Analogy-based methods developed for type-level embeddings are not applicable when representations vary with context.
- Cosine-based methods for exposing bias in contextual embeddings produce inconsistent results, including counterintuitive gender-related p-values.
- The proposed method uses simple templates containing an attribute and a target, masking them sequentially to compare bias across target classes.The introduction gives “programmer” and “she” as example attribute and target tokens.
- The study compares the template method with cosine similarity and evaluates gender bias in Gendered Pronoun Resolution.
- The paper highlights potential negative impacts of using BERT in downstream real-world applications.
2 Quantifying Bias in BERT
The method queries BERT’s masked-language-model predictions in template sentences to measure target–attribute associations. It normalizes target probabilities against a prior obtained without the attribute, producing a relative bias score.
- BERT’s masked language-model objective predicts masked tokens from the entire sentence context, and these predictions are used to measure encoded bias.
- It masks both target and attribute to compute a prior probability representing the model’s target preference from sentence structure alone.
- The method compares normalized target predictions, such as “he” and “she,” to quantify bias toward an attribute such as “programmer.”
- The procedure prepares a template sentence containing a target and an attribute, then masks the target to compute its conditional probability.
4. Compute the association as log
The paper defines normalized log-probability measures from template-based target–attribute associations. It argues that flexible templates support more consistent bias exposure and finer-grained measurements than cosine-based measures.
- The normalized target–attribute association is called the increased log probability score.
- The difference between two targets’ increased log probability scores is the log probability bias score used to measure bias.
- Template sentences can be shared across multiple targets and attributes, while their flexibility may support finer-grained bias measurement.
- The paper reports that the proposed log probability bias score is more effective than traditional cosine-based measures at exposing bias.
3 Correlation with Human Biases
The study applies WEAT-style stimuli to compare cosine-based BERT embeddings with a template-based log probability bias measure. WEAT finds no significant BERT biases at p < 0.01, whereas querying BERT’s language model exposes significant associations across all tested categories.
- The WEAT uses cosine similarity, a permutation test over target sets, and an effect-size statistic for quantifying associations.The statistical test is a permutation test, so a large effect size does not guarantee higher statistical significance.
- For BERT, target and attribute embeddings are computed from multiple templates by masking each token in turn.Out-of-vocabulary targets are removed, and target sets are balanced by randomly removing words from the smaller set.
- Grammatically incorrect target templates are replaced with common category indicators such as flower, he, and she to reduce variance in predicted probabilities.The resulting effect size uses the standard deviation of mean log probability bias scores.
- The experiments cover flower/insect, racial-name, gender-career/family, gender-math/arts, and gender-science/arts stimuli.
- WEAT on BERT finds no statistically significant biases at p < 0.01, while the proposed language-model-querying method finds significant associations across all categories.The comparison indicates that WEAT is ineffective for detecting these biases in BERT embeddings, whereas the proposed method is more sensitive.
4 Case Study: Effects of Gender Bias on Gendered Pronoun Resolution
The study evaluates gender bias in BERT through Gendered Pronoun Resolution and finds that BERT-based models handle female pronouns differently, especially in male-biased topics.
- Dataset: The GAP dataset contains 8,908 human-labeled ambiguous pronoun-name pairs from Wikipedia, classified as referring to entity A, entity B, or neither.
- Model: The pronoun-resolution model uses contextual BERT embeddings for the pronoun and candidate entities, followed by a one-hidden-layer MLP classifier.
- Results: The model assigned female pronouns to no entity with significantly higher probability than male pronouns, despite a balanced training set (p = 0.007).
- Results: A Spearman correlation of 0.207 linked male-gender topic bias scores with the model’s probability of assigning female pronouns to no entity.
- Results: The authors conclude that BERT-based systems struggle with female-pronoun coreference when sentence topics are biased toward men.
5 Real World Implications
The paper examines gender bias in BERT for employment-related attributes and reports stronger male associations, raising concerns about downstream systems such as resume filtering.
- Motivation: The study motivates employment-focused analysis because gender associations in models may affect recruiting and job recommendation systems.
- Datasets: It analyzes high-paying jobs from the Employee Salary Dataset, positive and negative adjectives, and technology skills from O*NET.
- Templates: The researchers use “TARGET is ATTRIBUTE” for occupations and traits, and “TARGET can do ATTRIBUTE” for technology skills.
- Results: Table 5 reports the percentage of employment-related attributes more strongly associated with male than female gender.
- Results: The results indicate strong male-pronoun preferences in BERT and raise concerns about using it in resume-filtering systems.
6 Related Work
Prior work established social biases in embeddings and downstream applications, but existing methods did not consistently expose bias in contextualized representations.
- Type-level embeddings: Earlier research found social biases in type-level embeddings and linked them to stereotypes in human-generated training data.
- Contextualized embeddings: Contextualized models such as BERT and ELMo increasingly replaced type-level embeddings while achieving state-of-the-art results across many NLP tasks.
- Measurement gap: WEAT and related cosine-based approaches did not provide clear or consistent evidence of bias for contextualized embeddings.
- Prior findings: Prior studies nevertheless reported gender and domain-related biases in contextualized embeddings and their downstream applications.
- This work: This work introduces a BERT-specific bias measure intended to align more consistently with human biases and examine downstream effects.
7 Conclusion
The paper concludes that querying BERT’s underlying language model measures bias and exposes stereotypes, while these biases can produce negative downstream effects.
- Querying BERT’s underlying language model effectively measures bias and exposes multiple stereotypes embedded in the model.
- The proposed measure is reported to be more consistent with human biases and to outperform traditional WEAT on BERT.
- The paper finds that BERT’s biases can have negative effects in downstream tasks.
- Future work includes studying other downstream tasks and developing methods to debias contextualized word embeddings.
Appendix
The appendix includes Table 6, which presents extracted topics for the GPR dataset.
- Table 6 lists extracted topics for the GPR dataset.
- The appendix identifies the displayed material as topics extracted from the GPR dataset.
- The table is presented as an appendix resource for the GPR dataset’s extracted topics.