Source-linked AI summary
Assessing Social and Intersectional Biases in Contextualized Word Representations
Yi Chern Tan, L. Elisa Celis
TL;DR
Existing evidence of social bias in context-free embeddings leaves open how state-of-the-art contextual models encode gender, racial, and intersectional biases. The paper adapts bias testing to contextual word representations and finds corpus- and model-level bias, including strong racial and amplified intersectional effects. It concludes that contextual word and sentence encodings capture different bias evidence and should both be evaluated.
Problem
The paper asks how state-of-the-art contextual word models encode gender, racial, and intersectional biases beyond evidence from context-free embeddings and sentence-level encodings.
Method
The paper adapts embedding association tests to assess social and intersectional bias at the contextual word level alongside sentence encodings.
Results
The study finds corpus-level gender imbalances, strong racial bias in contextual models, and intersectional bias effects exceeding those of constituent minority identities.
Takeaways & Limitations
Bias evaluation should use both contextual word and sentence encodings because they detect bias at different levels and instances.
Takeaways & Limitations
The study assumes binary gender and provides only a preliminary investigation of interactions among multiple minority identities.
Abstract
from arXiv · showhide
Social bias in machine learning has drawn significant attention, with work ranging from demonstrations of bias in a multitude of applications, curating definitions of fairness for different contexts, to developing algorithms to mitigate bias. In natural language processing, gender bias has been shown to exist in context-free word embeddings. Recently, contextual word representations have outperformed word embeddings in several downstream NLP tasks. These word representations are conditioned on their context within a sentence, and can also be used to encode the entire sentence. In this paper, we analyze the extent to which state-of-the-art models for contextual word representations, such as BERT and GPT-2, encode biases with respect to gender, race, and intersectional identities. Towards this, we propose assessing bias at the contextual word level. This novel approach captures the contextual effects of bias missing in context-free word embeddings, yet avoids confounding effects that underestimate bias at the sentence encoding level. We demonstrate evidence of bias at the corpus level, find varying evidence of bias in embedding association tests, show in particular that racial bias is strongly encoded in contextual word models, and observe that bias effects for intersectional minorities are exacerbated beyond their constituent minority identities. Further, evaluating bias effects at the contextual word level captures biases that are not captured at the sentence level, confirming the need for our novel approach.
1 Introduction
Contextual word models such as BERT and GPT-2 are state-of-the-art systems whose gender, racial, and intersectional biases require evaluation. The paper extends bias analysis to contextual word representations and finds substantial social bias, especially racial and intersectional bias.
- Contextual word representations condition token embeddings on sentence context and can also encode entire sentences.
- Prior work established social bias in context-free embeddings and preliminary sentence-level bias in BERT, but contextual word-level analysis remained limited.
- The paper evaluates gender, racial, and intersectional bias across state-of-the-art contextual word models using contextualized adaptations of embedding association tests.
- Contextual word models show evidence of social and intersectional bias, with racial bias strongly encoded and intersectional effects exceeding those of constituent minority identities.
- Bias appears at different levels and instances in contextual and sentence representations, so both encoding types should be used when measuring or debiasing models.
2 Related Work
Related work documents social bias in language technologies and word embeddings, motivating analyses that test stereotypical associations in contextual models. Existing methods compare associations between social concepts and attributes such as occupations.
- Social bias has been demonstrated in NLP applications including sentiment analysis, abusive language detection, image captioning, and text classification.
- WEAT-based studies show that word embeddings associate occupations with stereotypical gender roles.
- WEATs evaluate differences in word-vector distances when pairing concept groups with similar or contrasting attribute groups.
- Table 1 reports gendered and neutral pronoun counts and occupation co-occurrences across four pre-training datasets.
3 Gender Bias in Datasets
The paper measures gender representation and stereotypical occupation associations in the corpora used to pre-train contextual models. Male pronouns consistently outnumber female pronouns, and several datasets show pro-stereotypical co-occurrence patterns.
- BERT, ELMo, GPT, and GPT-2 were trained on Wikipedia, BooksCorpus, the 1 Billion Word Benchmark, or WebText, which the study analyzes for gendered co-occurrences.
- 3x: male pronouns occur at roughly three times the female-pronoun frequency in the 1 Billion Word Benchmark, Wikipedia, and WebText.
- 1.3x: male pronouns occur at this factor over female pronouns in BooksCorpus.
- On the 1 Billion Word Benchmark and WebText, pro-stereotypical associations are proportionally more frequent than anti-stereotypical associations.
- BooksCorpus contains proportionally more co-occurrences between gendered pronouns and stereotypically male occupations, regardless of pronoun gender.
- These dataset patterns reflect social bias captured in language, while dataset size and construction costs constrain replacement with representationally balanced corpora.
4 Social and Intersectional Bias using Embedding Association Tests
The paper adapts embedding association tests to contextual word representations and extends them to gender, race, and intersectional identities. It compares contextual word-level tests with sentence-level encodings to expose biases that sentence pooling can obscure.
- 4.1 Embedding Association Tests: WEATs measure associations between two target concepts and two attributes by comparing cross-concept attribute similarities.The paper uses equal-size target sets X and Y and attribute sets A and B.
- 4.1 Embedding Association Tests: Permutation tests assess whether observed associations are significant, while larger effect sizes indicate more severe pro-stereotypical representations when significance is controlled.Significance is computed over equal-size partitions of the combined target sets.
- 4.1 Embedding Association Tests: SEATs apply these association tests to sentence encodings produced from semantically bleached templates such as “This is a [doctor].”Sentence encodings may pool token representations or use the first or last token.
- 4.1 Embedding Association Tests: Table 2 reports proportions of significant positive effects across 92 tests, including 34 gender, 31 race, and 21 intersectional tests.The table breaks results down by identity type and model; c-word tests are invalid for CBoW.
- 4.2 Extension of Embedding Association Tests to Contextual Word Representations: The proposed contextual-word test replaces the sentence encoding with the contextual representation of the token of interest before pooling.The selected token representation differs by model from the sentence-level representation, such as BERT’s [CLS] token or GPT’s last token.
- 4.3 New Embedding Association Tests for Social and Intersectional Biases: The new tests target race, gender, and intersectional identities using attributes including pleasantness, work, discipline, competence, and likability.Intersectional tests primarily examine African American and female identity, including comparisons against each constituent minority identity.
5 Empirical Analysis
The empirical analysis finds social, racial, gender, and intersectional bias across contextual word models, with bias measurements varying by encoding type and model.
- Experimental Setup: The analysis evaluates GPT-2 117M and 345M alongside CBoW-GLoVe, ELMo, BERT, and GPT using p = 0.01 significance testing.Table 4 organizes race tests by word, sentence, and contextual-word encodings; Table 5 does so for intersectional tests.
- Overall Analysis: BERT bbc exhibits the highest bias proportion on race and intersectional tests and the highest overall proportion among contextual word models.Larger models showed smaller proportions in the BERT and GPT families, but the authors caution against a definitive conclusion without more model sizes.
- Overall Analysis: 93 significant effects were found across tests, with 36.6% detected only by contextual-word encoding, 25.8% only by sentence encoding, and 37.6% by both.Across 17 tests, nine had more positive significant associations with contextual-word than sentence encoding across models.
- Gender and Race: Gender tests vary across models; negative effect sizes suggest anti-stereotypical associations, but none are significant at p = 0.01.Work, discipline, competence, and likability attributes display gender bias, while P/U attributes are not stereotypically associated with gender.
- Gender and Race: Race tests reveal extensive positive bias associations, especially for pleasantness, competence, and likability attributes.Large BERT and GPT models showed few significant positive gender associations but many on race.
- Intersectional Analysis: Intersectional tests generally produce larger significant effects for African American women than for singular race or gender minority comparisons.Race comparisons produced nine significant positive associations versus two for gender, and the Angry Black Woman Stereotype test also produced nine.
6 Discussion and Limitations
The discussion frames the work as evidence that contextual models encode social and intersectional bias, while identifying limits in measurement scope and identity coverage.
- Discussion: The paper contributes evidence that social bias exists in state-of-the-art contextual models and that both sentence and contextual-word encodings are needed for measurement.It also presents racial bias as potentially stronger than gender bias and reports greater bias for intersectional identities than constituent minorities.
- Limitations: The study’s intersectional analysis is preliminary, covering specific identity interactions and leaving their interactive nature for further investigation.The authors attempted to isolate different identity dimensions but state that more work is needed.
- Future Directions: Larger contextual word models seem to encode less social bias, but the authors call for more model sizes and layer-level analysis before drawing firm conclusions.Future work would investigate how bias differs across model size and model layers.
7 Conclusion
The conclusion argues that sentence-level techniques do not sufficiently detect social and intersectional bias and proposes contextual-word evaluation alongside sentence encoding.
- Conclusion: Contextual-word evaluation is proposed to assess primary and intersectional social biases that previous sentence-encoding techniques did not sufficiently detect.The conclusion also positions social-bias detection as a first step toward evaluating and improving debiasing methods.
- Conclusion: Debiasing techniques should consider contextual associations because removing a defined gender direction may leave implicit associations intact.The paper gives “receptionist” and “captain” as an example of associations that can persist after projection-based debiasing.
A Details for Models
Table 6 summarizes the models, their encoding methods, and representation dimensions.
- Table 6 summarizes the models evaluated in the paper.
- The table identifies the encoding method used for each representation type.
- The table reports the representation dimension for each model and encoding type.
A.1 CBoW (GloVe)
The CBoW model represents sentences by averaging 300-dimensional GloVe word embeddings trained on Common Crawl.
- CBoW encodes sentences as the average of word embeddings.
- The model uses 300-dimensional GloVe vectors.
- The GloVe vectors were trained on the Common Crawl corpus.
A.2 ELMo
The paper constructs sentence and contextual-word encodings for ELMo, BERT, GPT, and GPT-2, then evaluates gender, race, and intersectional associations with templated tests.
- A.2 ELMo: ELMo sentence encodings mean-pool token vectors, while contextual-word encodings select the token of interest before summing layer outputs.Both ELMo encodings are 1024-dimensional.
- A.2 ELMo: BERT uses the top-most [CLS] hidden state for sentence encoding and the top-most hidden state of the token of interest for contextual encoding.Subword-tokenized targets use the representation corresponding to the token start.
- A.2 ELMo: GPT uses the last sequence token for sentence encoding and the token of interest for contextual-word encoding.Both encoding types are 768-dimensional, with subword targets represented from their start.
- A.2 ELMo: GPT-2 applies the same sentence and contextual-word encoding procedure as GPT in 117M and 345M versions.
- A.2 ELMo: The association-test suite includes Caliskan, double-bind, additional, and intersectional tests using gender, race, and stereotype attributes.Templates can be semantically bleached or unbleached, and targets include M/F and EA/AA names.
C Full Results for Embedding Association Tests
Tables 7 and 8 report association-test results and effect sizes for word, sentence, and contextual-word encodings, with significance and encoding-specific outcomes marked separately.
- The tables compare word, sentence, and contextual-word encodings using embedding association tests and effect sizes.
- M/F denotes Male/Female, P/U denotes Pleasant/Unpleasant, and EA/AA denotes European American/African American.
- Gray shading indicates significance at p = 0.01.
- The tables distinguish tests significant only for sentence encoding, only for contextual-word encoding, or for both.
- ABW denotes the Angry Black Woman stereotype in the reported intersectional tests.