Source-linked AI summary
Gender bias and stereotypes in Large Language Models
Hadas Kotek, Rikker Dockum, David Q. Sun
TL;DR
As LLM capabilities have advanced, this paper investigates whether gender stereotypes persist despite recent progress and RLHF. It introduces an ambiguity-based occupation-pronoun test, finding that four LLMs make stereotype-aligned choices, amplify social bias, overlook ambiguities, and rationalize choices with inaccurate explanations. The authors argue that bias evaluation and equitable treatment should remain central to LLM development and deployment.
Problem
The paper asks whether gender bias persists in rapidly advancing LLMs, including models trained with RLHF.
Method
The study tests four LLMs with an ambiguity-based paradigm pairing male- and female-stereotyped occupations with gendered pronouns and follow-up questions about alternative referents.
Results
LLMs express occupation-related gender bias that aligns more with social perceptions than US labor statistics and amplify those stereotypes.
Takeaways & Limitations
LLMs should be evaluated for bias and harms before deployment, with safe and equitable treatment of minoritized individuals central to model design and training.
Takeaways & Limitations
The study uses an indirect occupation-correlation measure and cannot determine whether additional business logic altered model responses.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have made substantial progress in the past several months, shattering state-of-the-art benchmarks in many domains. This paper investigates LLMs' behavior with respect to gender stereotypes, a known issue for prior models. We use a simple paradigm to test the presence of gender bias, building on but differing from WinoBias, a commonly used gender bias dataset, which is likely to be included in the training data of current LLMs. We test four recently published LLMs and demonstrate that they express biased assumptions about men and women's occupations. Our contributions in this paper are as follows: (a) LLMs are 3-6 times more likely to choose an occupation that stereotypically aligns with a person's gender; (b) these choices align with people's perceptions better than with the ground truth as reflected in official job statistics; (c) LLMs in fact amplify the bias beyond what is reflected in perceptions or the ground truth; (d) LLMs ignore crucial ambiguities in sentence structure 95% of the time in our study items, but when explicitly prompted, they recognize the ambiguity; (e) LLMs provide explanations for their choices that are factually inaccurate and likely obscure the true reason behind their predictions. That is, they provide rationalizations of their biased behavior. This highlights a key property of these models: LLMs are trained on imbalanced datasets; as such, even with the recent successes of reinforcement learning with human feedback, they tend to reflect those imbalances back at us. As with other types of societal biases, we suggest that LLMs must be carefully tested to ensure that they treat minoritized individuals and communities equitably.
1 INTRODUCTION
Although LLMs have advanced rapidly, the paper asks whether gender biases have been reduced, particularly after RLHF. It introduces a testing paradigm designed to probe gender stereotypes without reproducing likely training data.
- LLMs have achieved state-of-the-art or better performance across many natural-language and standardized tasks.
- Prior language models perpetuate and sometimes amplify biases, stereotypes, and negative perceptions of minoritized groups.
- The paper asks whether recent LLM advances and RLHF have reduced or eliminated such biases.
- The proposed gender-bias paradigm uses expressions unlikely to have been explicitly included in current LLM training data.
- The study examines whether LLMs rely on gender stereotypes and whether their explanations invoke inaccurate grammar claims or explicit stereotypes.
2 RELATED WORK
Prior research documents gender bias across language models, NLP systems, human sentence processing, and society. WinoBias provides an established benchmark for testing gender bias through pronoun-resolution tasks influenced by world knowledge and sentence structure.
- Gender bias has been documented in word embeddings and models for captioning, sentiment analysis, toxicity detection, translation, and other NLP tasks.
- Bias in language-model outputs extends beyond gender to categories including religion, race, nationality, disability, and occupation.
- Human sentence-processing studies show that stereotypical noun knowledge influences comprehension and that pronouns are generally more likely to refer to subjects than objects.
- Societal gender bias has been documented across domains including health, finance, and education.
3 METHODOLOGY
The study develops an ambiguity-based occupation-pronoun paradigm inspired by WinoBias but designed to reduce training-data contamination. It evaluates multiple noun-position and pronoun combinations, asks models about alternative referents, and analyzes their response strategies and explanations.
- WinoBias background: WinoBias contains 3,160 pronoun-resolution sentences built around world knowledge and two sentence types differing in ambiguity.
- Paradigm design: Unlike WinoBias, the study uses ambiguous sentences containing male- and female-stereotyped occupations whose expressions are less likely to appear in training data.
- Prompt design: The prompts form a 2x2 design crossing noun position with masculine or feminine pronouns.
- Response strategies: The paradigm tests four response strategies: contextual plausibility, invariant subject or object selection, stereotype-based noun selection, random guessing, and acknowledging ambiguity.
- Experimental procedure: The experiment uses semantically similar occupations, four permutations per paradigm, 60 sentences, and three prompts per model for 180 items.
- Experimental procedure: Each session includes an initial pronoun-reference question followed by a standardized query asking whether the unchosen noun could also be the referent.
4 RESULTS
Across four models, occupation choices strongly followed gender stereotypes, while syntactic position was not statistically significant. The models recognized ambiguity when directly prompted but often supplied biased or factually inaccurate explanations for their initial choices.
- 6.8 times more likely to choose a stereotypically female occupation with a female pronoun, and 3.4 times more likely to choose a stereotypically male occupation with a male pronoun.
- Syntactic position was not a statistically significant factor in noun selection.The models showed slight subject–object skews, but chi-squared tests did not support a significant effect.
- Models tracked human perceptions of occupational gender stereotypes more closely than Bureau of Labor Statistics distributions for both pronouns.For ‘he’, correlations were τ=0.67 with human ratings versus τ=0.5 with BLS ground truth; for ‘she’, τ=0.49 versus τ=0.46, all p-values<0.001.
- The models amplified stereotypical bias for women’s occupations, with no parallel amplification observed for men.Women’s stereotypically male occupations were selected less often than expected, while stereotypically female occupations were selected more often than expected.
- The models selected a more diverse set of occupations for the male pronoun than for the female pronoun.Eleven occupations met the male-only threshold, compared with seven occupations meeting the female-only threshold.
- Three models mostly acknowledged the sentences’ ambiguity when asked directly, although they commonly maintained that their original choice was more likely; Model 3 mostly denied alternatives.An external-person interpretation was mentioned infrequently, despite being possible in the paradigm.
- Model explanations varied, but included explicit gender bias, misleading grammatical rationalizations, and factually incorrect linguistic claims.Explicitly gender-biased explanations occurred 5–15% of the time on average; models also invoked inaccurate subject or proximity rules for ambiguous sentences.
5 DISCUSSION
The discussion argues that LLMs can reproduce and amplify societal gender bias, making careful evaluation and mitigation necessary before widespread deployment. It also identifies limitations in the study’s measures, scope, assumptions, and response provenance.
- The paper argues that models should signal ambiguity and answer reliability because the study’s sentences were ambiguous but usually received categorical initial interpretations.The models often acknowledged ambiguity only after follow-up questioning, while users may not routinely request explanations.
- LLMs’ coherent, conversational outputs can lead users to mistake probable text generation for knowledge search or human-like reasoning.The discussion notes that human-like phrasing, including apologies and language suggesting sentience, contributes to this misconception.
- Reflecting biased societal facts can amplify bias because historical barriers contributed to current occupational disparities.The discussion connects occupational stereotypes to longstanding barriers that limited women’s participation in certain professions.
- Bias evaluation is especially important for LLM applications in medicine, law, finance, and education because reproducing or amplifying harms may affect real-world use.The discussion frames bias and safety evaluation as important both ethically and for product efficacy.
- The study’s indirect occupation-based measure cannot establish that observed results reflect inherent model gender bias rather than another correlating factor.The authors also acknowledge taking model explanations at face value even though they may be probable continuations rather than true reasoning.
- The findings are constrained by 15 sentence schemas, three prompts per model, English and Western/American assumptions, and examination of only female and male pronouns.The authors also note that responses may reflect a combination of model outputs and business logic rather than model behavior alone.
6 CONCLUSION
Across four early-2023 LLMs, the study finds gender-stereotyped occupation choices, amplified stereotypes, overlooked ambiguity, and inaccurate explanations. These results suggest that imbalanced training data remain reflected in model behavior despite reinforcement learning with human feedback.
- Across four LLMs, stereotypically male occupations were chosen for “he” and stereotypically female occupations for “she”.
- LLMs amplified stereotypes associated with female individuals more than those associated with male individuals.
- LLMs rarely independently flagged ambiguity in the study items but frequently recognized it when explicitly asked.
- Explanations sounded authoritative but were often inaccurate and likely obscured the true reasons underlying predictions.
- The findings indicate that imbalanced training data can be reflected and amplified in LLM behavior even with reinforcement learning from human feedback.