PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases

Wei Guo, Aylin Caliskan

arXiv:2006.03955v5cs.CYcs.AIcs.CL

TL;DR

Bias measurement in contextualized language models is limited by template-based contexts and incomplete treatment of intersectional groups. The paper introduces CEAT, IBD, and EIBD to measure contextual distributions and automatically detect intersectional and emergent biases, finding biased representations across studied models and especially strong associations for intersectional groups.

  • Problem

    Prior methods measure bias in limited template-defined contexts and do not comprehensively identify intersectional and emergent biases.

  • Method

    The paper combines CEAT’s random-effects analysis of contextualized embeddings with IBD and EIBD for automatically detecting intersectional and emergent biases in static embeddings.

  • Results

    All studied models contain biased representations, while intersectional groups such as African American and Mexican American females show the strongest measured associations.

  • Takeaways & Limitations

    Intersectional and emergent biases can be automatically discovered in static embeddings and measured across contextualized language-model representations.

  • Takeaways & Limitations

    Evaluation uses closed-world validation sets, and the gender categories in those sets are binary.

Abstract

from arXiv · show

With the starting point that implicit human biases are reflected in the statistical regularities of language, it is possible to measure biases in English static word embeddings. State-of-the-art neural language models generate dynamic word embeddings dependent on the context in which the word appears. Current methods measure pre-defined social and intersectional biases that appear in particular contexts defined by sentence templates. Dispensing with templates, we introduce the Contextualized Embedding Association Test (CEAT), that can summarize the magnitude of overall bias in neural language models by incorporating a random-effects model. Experiments on social and intersectional biases show that CEAT finds evidence of all tested biases and provides comprehensive information on the variance of effect magnitudes of the same bias in different contexts. All the models trained on English corpora that we study contain biased representations. Furthermore, we develop two methods, Intersectional Bias Detection (IBD) and Emergent Intersectional Bias Detection (EIBD), to automatically identify the intersectional biases and emergent intersectional biases from static word embeddings in addition to measuring them in contextualized word embeddings. We present the first algorithmic bias detection findings on how intersectional group members are strongly associated with unique emergent biases that do not overlap with the biases of their constituent minority identities. IBD and EIBD achieve high accuracy when detecting the intersectional and emergent biases of African American females and Mexican American females. Our results indicate that biases at the intersection of race and gender associated with members of multiple minority groups, such as African American females and Mexican American females, have the highest magnitude across all neural language models.

Introduction

The paper addresses incomplete bias measurement by developing methods that detect intersectional and emergent biases and quantify contextualized bias without relying solely on templates.

  • Motivation: Existing bias measurements often focus on single social categories, leaving intersectional biases underidentified.This gap can produce incomplete measurement of social biases.
  • Contributions: IBD and EIBD automatically identify intersectional and emergent intersectional biases in static word embeddings.EIBD targets associations unique to intersectional groups and absent from their constituent identities.
  • Contributions: CEAT measures distributions of bias effect magnitudes across naturally occurring contexts rather than relying on sentence templates.It uses a random-effects model to summarize overall contextualized bias.
  • Findings: All studied models contain biased representations, with the strongest associations linked to intersectional group members such as African American women.The introduction reports ELMo as most biased, followed by BERT and GPT, with GPT-2 least biased.

Related Work

Prior work established that embeddings contain human-like biases and that contextualized representations vary with context, but existing measurements remain limited in intersectional coverage and contextual breadth.

  • Static embeddings: Static word embeddings encode human-like biases documented by the Implicit Association Test.These embeddings derive representations from word co-occurrence statistics.
  • Intersectional bias: Research on intersectional bias has examined groups such as African American females and documented accuracy disparities in computer vision.Prior embedding studies measured emergent biases for African American females using predefined attributes.
  • Contextualized embeddings: Contextualized word embeddings vary with context, unlike static embeddings, and prior methods measure them using sentence templates.Template examples include “This is a [TARGET]” and “[TARGET] is [MASK].”
  • Research gap: Most prior methods examine limited contexts and report unweighted mean bias magnitudes, which does not reflect contextualized bias distributions.This motivates measuring bias across a broader range of naturally occurring contexts.

Data

The study combines static and contextualized embeddings with corpora, stimuli, and validation sets to detect and measure social and intersectional biases.

  • Embedding sources: Static word embeddings provide word-level signals for automatically retrieving biased associations through IBD and EIBD.The study uses GloVe trained on Common Crawl word co-occurrence statistics.
  • Embedding sources: Contextualized embeddings are extracted from ELMo, BERT, GPT, and GPT-2 implementations for context-dependent bias measurement.The models differ in architecture, tokenization, layers, and training corpora.
  • Context corpus: CEAT simulates contextual distributions by randomly sampling sentences containing target words from a Reddit corpus of 500 million comments.The comments span January 1 through December 31, 2014.
  • Stimuli and validation: WEAT stimuli represent racial categories with frequent given names, while human-validated intersectional attributes provide evaluation ground truth.The validation sets represent gender as a binary category, a stated limitation for future work.

Approach

The approach combines WEAT, random-effects meta-analysis, and two automated detectors to measure social, intersectional, and emergent biases in static and contextualized embeddings.

  • Approach: The framework combines WEAT, random-effects modeling, IBD, and EIBD to summarize and detect biases across static and contextualized word embeddings.WEAT provides the foundation for contextualized-bias summaries; IBD detects intersectional associations, while EIBD detects associations unique to intersectional groups.
  • Approach: WEAT measures effect sizes by comparing relative associations between two target-word sets and two polar-attribute sets.Examples include career and family targets paired with woman and man attributes.
  • Intersectional Bias Detection (IBD): IBD identifies attributes highly associated with intersectional groups using multiple social-category comparisons and thresholded WEFAT pairs.The method represents groups defined by two categories and detects associated attributes from all intersectional-group comparisons.
  • Emergent Intersectional Bias Detection (EIBD): EIBD detects words associated with an intersectional group but not with its constituent categories.For example, it targets attributes associated with African American females but not African Americans or females separately.
  • Contextualized Embedding Association Test (CEAT): CEAT avoids sentence templates by sampling contextualized embeddings from a control corpus and summarizing their varying bias effects.Because contextualized embeddings vary by context, CEAT treats effect sizes as random effects rather than assuming one fixed value.
  • Random-Effects Model: The random-effects model computes a combined effect size as a variance-weighted mean across sampled effects, incorporating within-sample and between-sample variance.The approach combines 10,000 WEAT samples for contextualized embeddings.

Results and Evaluation

The evaluation combines IBD and EIBD for detecting intersectional and emergent biases with CEAT for measuring bias distributions across contextualized embeddings. Intersectional biases show the strongest magnitudes across models, while CEAT reveals model- and context-dependent variation.

  • Detection setup: IBD and EIBD evaluate four intersectional bias tests involving African American, European American, and Mexican American men and women.The tests extend bias measurement beyond single social categories.
  • Detection performance: 84.7% and 65.3% are EIBD accuracies for detecting emergent biases unique to African American and Mexican American females, respectively.The corresponding random correct-identification rates are 9.2% and 6.1%.
  • Detection performance: 81.6% and 82.7% are IBD accuracies for detecting intersectional biases of African American and Mexican American females, respectively.The random correct-identification rates are 14.3% and 13.3%.
  • Context variation: GPT-2 has the highest variance in bias magnitudes, followed by GPT, BERT, and ELMo, despite having the lowest overall bias magnitude.The same social bias can therefore vary in intensity across contexts and models.
  • CEAT measurement: CEAT pools 10,000 effect-size samples with a random-effects model to report overall CES magnitudes and combined p-values.Table 1 distinguishes completely random context samples from shared sentences across models.

Discussion

The discussion presents IBD and EIBD as generalizable methods for identifying biases in static embeddings, while CEAT measures their contextualized manifestations. It also identifies boundaries involving validation coverage, language availability, corpus assumptions, lexicon induction, and potential misuse.

  • Discussion: Highly contextualized representations coincide with higher bias variance, lower overall bias, and lower self-similarity across the studied models.The discussion relates this pattern to more context-specific representations in upper layers.
  • Discussion: IBD and EIBD are presented as generalizable methods for identifying biases associated with social or intersectional group members.The methods detect and measure biases for Mexican American and African American females in static and contextualized embeddings.
  • Discussion: The methods can complement psychologist-predefined stimuli through unsupervised lexicon induction, but principled and robust induction methods remain future work.The current approach uses conventional frequent given names to represent group membership.
  • Scope: The methods are language agnostic in principle, but the study leaves other-language generalization for future work because suitable state-of-the-art models were not widely or freely available.The paper notes that stimuli for representing concepts would also be required.
  • Assumptions: Context simulation assumes that Reddit represents naturally occurring sentences, although the corpus also reflects biases of its contributing population.The paper leaves evaluation of common context distributions for future work.
  • Ethical concerns: Using IBD and EIBD carries dual-use risks because retrieved stereotypes could support offensive content, influence operations, or an AI bias feedback cycle.The concern applies to automated exploitation of biased model outputs and their reuse in later training corpora.

Conclusion

The paper introduces IBD, EIBD, and CEAT to detect and measure intersectional biases in word embeddings and language models. It finds strong intersectional effects, high detection accuracy, and a negative relationship between bias magnitude and contextualization.

  • IBD and EIBD automatically detect intersectional and emergent intersectional biases associated with members of multiple minority groups.
  • 81.6% and 82.7% detection accuracy were achieved for African American female and Mexican American female intersectional biases, respectively.
  • 84.7% and 65.3% detection accuracy were achieved for emergent intersectional biases of African American females and Mexican American females, respectively.
  • Intersectional biases associated with African American and Mexican American females had higher effect sizes than other investigated social biases.
  • CEAT uses a random-effects model to measure context-dependent social biases, finding ELMo most biased, followed by BERT and GPT, with GPT-2 least biased.
  • Overall bias magnitude negatively correlates with language-model contextualization, while understanding this relationship may help mitigate harmful downstream effects.

Appendices

The appendices formalize WEAT and the random-effects calculations underlying CEAT, then describe procedures for detecting emergent biases and validating results across sample sizes and group comparisons.

  • Formal Definition of WEAT: WEAT compares cosine-based associations between two equally sized target-word sets and two attribute-word sets.
  • Formal Definition of WEAT: Permutation testing assesses significance by treating random partitions of the target words as a null hypothesis without biased associations.
  • Formal Definition of EIBD: IBD first detects an intersectional group’s biases, then identifies attributes associated with its constituent race- and gender-based subcategories.
  • Formal Definition of EIBD: EIBD constructs WEFAT pairs across all constituent subcategories and identifies attributes highly associated with the intersectional group’s constituent categories.
  • Random-Effects Model Details: Each WEAT receives an inverse-variance weight based on estimated in-sample and between-sample variance, and CEAT combines the weighted effects into CES.
  • Random-Effects Model Details: The standard error of CES is derived from the inverse total weight, and CES divided by its standard error is evaluated with a two-tailed normal-theory p-value.
  • Appendix Experiments: Using N = 1,000 reproduces valid CEAT results for comparison with N = 10,000, while supplementary tests compare six intersectional groups pairwise.

Stimuli

The paper constructs CEAT stimuli from names and attributes representing intersectional groups, using validation-set attributes and specified exclusions. It also reports detection stimuli and random-chance baselines for intersectional and emergent intersectional bias classification.

  • CEAT stimuli: Intersection-related CEAT uses frequent given names to represent target intersectional groups and attributes associated with their biases.African American female and Mexican American female tests use these group representations, with European American males used as a comparison group in specified tests.
  • Validation and detection: Human-subject validation supplies ground-truth intersectional attributes for evaluating the intersection-related CEAT, IBD, and EIBD experiments.The validation set is used to evaluate intersectional attributes, while stereotype-congruity determines the comparison ordering in one CEAT test.
  • Stimulus exclusions: Attributes shared across tested groups are excluded from some comparisons, including ‘assertive’ for African American females and European American males and ‘attractive’ for Mexican American females and European American males.The exclusions prevent an attribute associated with both groups in a comparison from being included in that test.
  • Validation and detection: IBD and EIBD detection stimuli include group names and bias attributes, with African American and Mexican American female tests using distinct intersectional and emergent attribute sets.The reported sets include 14.3% random-chance classification for African American female intersectional biases, 9.2% for their emergent biases, 13.3% for Mexican American female intersectional biases, and 6.1% for their emergent biases.

Open Source Code, Data, and Documentation

The project provides an open-source repository containing code, dataset links, and documentation for extracting contextualized word embeddings. The extracted stimulus embeddings require approximately 50GB of memory.

  • Repository and documentation: The open-source repository provides code, dataset links, and documentation answering frequently asked questions about extracting contextualized word embeddings.The repository is available at https://github.com/weiguowilliam/CEAT.
  • Repository and documentation: Extracted embeddings for the stimuli occupy approximately ∼50GB of memory.This memory estimate applies to the extracted stimulus embeddings described with the repository materials.
Loading 2006.03955v5…