Source-linked AI summary
StereoSet: Measuring stereotypical bias in pretrained language models
Moin Nadeem, Anna Bethke, Siva Reddy
TL;DR
Pretrained language models can encode stereotypical biases, while prior evaluations relied on small sets of artificial sentences. The paper introduces StereoSet and CATs to measure bias and language modeling across four domains, finding strong stereotypical bias in current models.
Problem
Existing evaluations of pretrained-language-model bias use small sets of artificially constructed sentences, limiting natural-context assessment.
Method
The paper introduces Context Association Tests and the StereoSet dataset, using stereotype, anti-stereotype, and unrelated associations across four domains.
Results
Current pretrained language models exhibit strong stereotypical biases, and the best model is 27.0 ICAT points behind the idealistic language model.
Takeaways & Limitations
GPT2 models exhibit relatively more idealistic behavior than BERT, ROBERTA, and XLNET on StereoSet.
Takeaways & Limitations
Some dataset stereotypes and anti-stereotypes remain objectively false, and treating both as equally preferable can be problematic when stereotypes may reflect likely outcomes.
Abstract
from arXiv · showhide
A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. In order to assess the adverse effects of these models, it is important to quantify the bias captured in them. Existing literature on quantifying bias evaluates pretrained language models on a small set of artificially constructed bias-assessing sentences. We present StereoSet, a large-scale natural dataset in English to measure stereotypical biases in four domains: gender, profession, race, and religion. We evaluate popular models like BERT, GPT-2, RoBERTa, and XLNet on our dataset and show that these models exhibit strong stereotypical biases. We also present a leaderboard with a hidden test set to track the bias of future language models at https://stereoset.mit.edu
1 Introduction
Pretrained language models can reflect stereotypical biases from the real-world corpora used to train them. StereoSet addresses limitations of prior evaluations by testing bias and language modeling with natural contexts across four domains.
- A race-domain example pairs a natural context about an Arab person with stereotypical, anti-stereotypical, and unrelated continuations.
- Pretrained language models may reflect stereotypical biases present in the massive text corpora used for training.
- Prior contextual-bias evaluations used artificial contexts, predefined stereotypical attributes, and single-word targets, excluding multiword terms.
- StereoSet introduces association tests for sentence-level and discourse-level bias and language-modeling evaluation.
- StereoSet contains 4 target domains, 321 target terms, and 16,995 English test instances organized as triplets.
2 Task Formulation
Context Association Tests evaluate whether models distinguish meaningful from meaningless associations while separately measuring preference for stereotypes over anti-stereotypes at sentence and discourse levels.
- Task Formulation: An ideal model ranks meaningful contexts above meaningless ones and gives equal preference to stereotypical and anti-stereotypical contexts.
- Task Formulation: CATs determine bias from the relative likelihoods of stereotypical and anti-stereotypical associations, while unrelated associations measure language modeling ability.
- Task Formulation: CATs assess both language-modeling ability and stereotypical bias using three stereotype, anti-stereotype, and unrelated instantiations.
- Intrasentence: The intrasentence CAT uses a fill-in-the-blank sentence and measures which attribute has the greatest likelihood of filling the blank.
- Intrasentence / Intersentence: Intrasentence CATs target sentence-level reasoning, whereas intersentence CATs target discourse-level reasoning.
- Intersentence: The intersentence CAT uses a target-containing sentence followed by stereotype, anti-stereotype, or unrelated attribute sentences.
3 Related Work
Prior work measured stereotypical bias in word embeddings and contextual representations using analogies, association tests, artificial contexts, and downstream tasks. StereoSet builds on these approaches with natural contexts and discourse-level evaluation.
- Word analogy and association tests established that word embeddings capture syntactic, semantic, and stereotypical biases.
- WEAT compares associations between complementary word classes and pleasant or unpleasant attribute classes to quantify bias.
- SEAT extended WEAT to sentence encoders by embedding target and attribute terms in generic artificial sentences.
- Kurita et al. used masked-token prediction probabilities and found observations similar to earlier embedding-bias results.
- StereoSet uses natural context for intrasentence evaluation and adds intersentence CATs because language modeling extends beyond individual sentences.
- Extrinsic evaluations estimate bias through downstream applications such as coreference resolution and sentiment analysis.
4 Dataset Creation
StereoSet was constructed from diverse social-group terms and crowdsourced associative contexts across four domains. Validation retained CATs with majority agreement on stereotype, anti-stereotype, and unrelated labels.
- The dataset covers 4 domains—gender, profession, race, and religion—and uses U.S. crowdworkers because stereotypes may vary by country.
- Profession, race, and religion target terms were collected from Wikidata relations, while gender terms came from prior work.
- Terms were manually filtered for infrequency or excessive specificity, and multiword targets such as software developer were retained.
- Intrasentence CAT: Intrasentence CATs pair a target-containing fill-in-the-blank context with stereotype, anti-stereotype, and unrelated attribute terms.
- Intersentence CAT: Intersentence CATs pair a target-containing sentence with associative sentences, where only stereotype and anti-stereotype continuations can naturally follow.
- Annotators were instructed to provide realistic stereotype and anti-stereotype associations rather than implausible statements.
- Validation: Five validators classified each association, and CATs were retained only when at least three agreed; this kept 83% of collected CATs.
5 Dataset Analysis
StereoSet analysis examines the sentiment and domain-specific vocabulary of stereotypical associations. Stereotypes are not uniformly negative, but they are more often negative than anti-stereotypes, while extracted keywords reflect behavioral and religious attributes.
- Sentiment analysis: 41% of stereotypical associations have negative sentiment, compared with 33% of anti-stereotypical associations.Stereotypical associations are therefore relatively more negative, although stereotypes also include positive associations.
- Sentiment analysis: Stereotypes are not always negative; “Asians are good at math” is given as a positive-sentiment stereotype.
- Keyword analysis: StereoSet keywords identify attribute terms associated with target groups by comparing their frequency against the TenTen web corpus.Target terms themselves are removed before extracting the keywords.
- Keyword analysis: Behavioral keywords include pushy, greedy, and hardwork, whereas religious keywords include diety, forgiving, and reborn.
6 Experimental Setup
The experimental setup evaluates whether pretrained language models can rank meaningful associations above unrelated ones without preferring stereotypical over anti-stereotypical associations. It formalizes these goals with language-modeling, stereotype, and combined idealized CAT scores, alongside reference baselines.
- Data splits: StereoSet uses disjoint development and hidden test target terms, with 25% assigned to development and 75% to testing.There is no training set because the dataset measures pretrained models rather than models fine-tuned on StereoSet.
- Evaluation metrics: Language Modeling Score measures the percentage of instances where a model ranks a meaningful stereotype or anti-stereotype above the unrelated association.
- Evaluation metrics: Stereotype Score measures how often a model prefers a stereotypical association over an anti-stereotypical association, with 50 as the unbiased target.
- Evaluation metrics: Idealized CAT combines language-modeling and stereotype scores so ideal, fully biased, and random models receive icat scores of 100, 0, and 50 respectively.The metric is designed to reward language modeling while penalizing stereotypical bias.
- Baselines: IDEALLM, STEREOTYPEDLM, RANDOMLM, and SENTIMENTLM provide reference behaviors based on correct ranking, stereotypical preference, random choice, and negative sentiment.IDEALLM has lms, ss, and icat scores of 100, 50, and 100, while RANDOMLM has 50, 50, and 50.
7 Main Experiments
The main experiments apply CAT to BERT, RoBERTa, XLNet, and GPT-2 using model-specific scoring procedures for intrasentence and intersentence association ranking. These procedures adapt masked-language, next-sentence, bidirectional, or autoregressive modeling capabilities to the test.
- BERT: BERT evaluates intrasentence CAT by ranking attribute terms according to their average log probabilities in a masked context.For intersentence CAT, BERT uses its pretrained next-sentence prediction head to rank follow-up attribute sentences.
- RoBERTa: RoBERTa uses a BERT-like scoring mechanism but requires a separately trained next-sentence classification head because it lacks a pretrained NSP head.
- XLNet: XLNet is evaluated bidirectionally to match the BERT and RoBERTa setting, using the pretrained model for intrasentence CAT and a trained NSP head for intersentence CAT.
- GPT-2: GPT-2 scores intrasentence CAT autoregressively using average subword log probabilities for full sentences with inserted attribute terms.The average avoids penalizing attribute terms that contain multiple subword units.
- GPT-2: GPT-2 uses a separately trained next-sentence classifier for intersentence CAT because its analogous method performed poorly.The classifier reaches 92.5%, 94.2%, and 96.1% accuracy for GPT2-small, GPT2-medium, and GPT2-large, respectively.
8 Results and discussion
Pretrained models generally outperform baselines on language modeling and idealized-bias performance, but stronger language modeling correlates with stronger stereotypical bias. Results vary across domains, model sizes, architectures, and pretraining corpora.
- Baselines vs. Models: All pretrained models outperform baselines on icat, while GPT2-large reaches 88.9 lms and the ENSEMBLE reaches 90.7 lms on development.GPT2-small has the highest pretrained-model icat score at 71.9, whereas XLNET-base has the lowest at 61.6.
- Relation between lms and ss: A strong correlation between lms and ss means that stronger language models also tend to exhibit stronger stereotypical bias.GPT2 variants balance lms and ss relatively well, producing high icat scores.
- Impact of model size: As model size increases, lms and stereotypical score generally increase, but icat does not follow the same pattern across architectures.Roberta and XLNET icat increase with size, whereas BERT and GPT2 do not.
- Impact of pretraining corpora: Corpus size does not correlate with lms or icat across BERT, ROBERTA, XLNET, and GPT2, although the authors conjecture that GPT2 benefits from its Reddit-linked training data.The corpus-size comparison covers 16GB, 160GB, 158GB, and 40GB respectively; the authors note that controlled verification requires training one model on increasing corpus sizes.
- Domain-wise bias: 69.7 icat: the ENSEMBLE is relatively less biased on race than on the other evaluated domains.High-bias terms include mother, software developer, and Africa, while producer and Crimean are among the low-bias terms.
- Intrasentence vs Intersentence CATs: Intersentence language modeling is harder than intrasentence language modeling for most models because intersentence instances contain more words.BERT is the noted exception.
9 Limitations
The authors identify limitations concerning annotator representativeness, objective factual content in examples, and cases where preferring stereotypes may be useful.
- StereoSet may not reflect stereotypes held by the wider US population because its Mechanical Turk worker pool is predominantly under age 50.The dataset was crowdsourced from a US-based pool, with 80% below age 50.
- Some dataset statements remain objectively false despite instructions to avoid objective claims and validator filtering.The paper gives examples where stereotypes or anti-stereotypes conflict with demographic or linguistic facts.
- The idealized CAT assumption of equal preference between stereotypes and anti-stereotypes can be problematic when stereotypes reflect likely outcomes.The authors illustrate this with a chef being more likely to make delicious than disgusting food and leave the issue for future work.
10 Conclusion
The paper introduces CAT and ICAT alongside StereoSet, a 16,995-example benchmark spanning four domains. Its evaluations find strong stereotypical biases in pretrained language models, with the best model 27.0 ICAT points behind an idealistic model and GPT-2 relatively more idealistic than BERT, RoBERTa, and XLNet.
- StereoSet contains 16,995 CATs covering gender, race, religion, and professions, and includes a hidden-test leaderboard for future models.
- 27.0 ICAT points separate the best evaluated model from the idealistic language model, while current pretrained models exhibit strong stereotypical biases.
- GPT-2 models exhibit relatively more idealistic behavior than BERT, RoBERTa, and XLNet in the paper’s evaluation.
A Appendix
The appendix documents StereoSet’s data collection, CAT evaluation setup, fine-tuned sentence-ordering classifiers, sentiment fine-tuning, and per-domain performance tables. It also includes interface screenshots and the target-term inventory.
- Data collection: 475 intrasentence and 803 intersentence annotators completed the US-based crowdsourcing tasks, with a 95% HIT acceptance requirement.
- Intersentence evaluation: The intersentence task computes p(s|c), the likelihood of sentence s given context sentence c.
- Intersentence evaluation: The authors train a Next Sentence Prediction classifier because models other than BERT lack BERT’s pretrained classification head for p(s|c).
- Intersentence evaluation: The sentence-ordering corpus uses adjacent Wikipedia sentences and randomly sampled negative sentences, with a 256-token maximum and 9.5 million training examples.
- Classifier results: 93–98% accuracy was achieved on the 3.5-million-example test set, while additional classifiers reached 94.6% for roberta-base, 97.1% for roberta-large, 93.4% for xlnet-base, and 94.1% for xlnet-large.
- Classifier results: GPT-2 classifiers achieved 92.5% accuracy on gpt2-small, 94.2% on gpt2-medium, and 96.1% on gpt2-large.
- Sentiment evaluation: BERT fine-tuned on the Large Movie Review dataset achieved 92% test accuracy after seven epochs.
- Supplementary materials: Tables report per-domain development and test performance, while Figures 2 and 3 show the intrasentence and intersentence collection interfaces and Table 9 lists target terms by dataset frequency.