Source-linked AI summary
Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models
Hannah Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frederic A. Dreyer, Aleksandar Shtedritski, Yuki M. Asano
TL;DR
This paper asks how widely used generative language models encode occupational bias when applied out-of-the-box. It probes GPT-2 across gender intersections using large-scale sentence completions and compares the results with US labor data, finding narrower and more stereotypical occupational associations for women while raising whether models should reflect or correct societal inequalities.
Problem
The paper addresses limited evidence about biases in popular open-source text-generation models used out-of-the-box, which may affect widely used applications.
Method
The study generates 396K default-parameter GPT-2 completions from intersectional identity and name templates, extracts occupations, fits logistic models, and compares outputs with US labor data.
Results
GPT-2 produces less diverse and more stereotypical occupational associations for women, with intersectional interactions shaping associations and outputs generally reflecting skewed US gender and ethnicity distributions.
Takeaways & Limitations
The findings identify occupational clustering as a problematic bias and leave open whether language models should reproduce or correct existing societal inequalities.
Takeaways & Limitations
The labor-market comparison is US-centric, lacks granular ground truth for several intersections, excludes informal-sector comparisons, and considers only two genders.
Abstract
from arXiv · showhide
The capabilities of natural language models trained on large-scale data have increased immensely over the past few years. Open source libraries such as HuggingFace have made these models easily available and accessible. While prior research has identified biases in large language models, this paper considers biases contained in the most popular versions of these models when applied `out-of-the-box' for downstream tasks. We focus on generative language models as they are well-suited for extracting biases inherited from training data. Specifically, we conduct an in-depth analysis of GPT-2, which is the most downloaded text generation model on HuggingFace, with over half a million downloads per month. We assess biases related to occupational associations for different protected categories by intersecting gender with religion, sexuality, ethnicity, political affiliation, and continental name origin. Using a template-based data collection pipeline, we collect 396K sentence completions made by GPT-2 and find: (i) The machine-predicted jobs are less diverse and more stereotypical for women than for men, especially for intersections; (ii) Intersectional interactions are highly relevant for occupational associations, which we quantify by fitting 262 logistic models; (iii) For most occupations, GPT-2 reflects the skewed gender and ethnicity distribution found in US Labor Bureau data, and even pulls the societally-skewed distribution towards gender parity in cases where its predictions deviate from real labor market observations. This raises the normative question of what language models should learn - whether they should reflect or correct for existing inequalities.
1 Introduction
The paper examines occupational biases in popular open-source generative language models used out-of-the-box, focusing on GPT-2 because of its widespread accessibility and use. It studies gender intersected with multiple protected categories and compares model associations with US labor-market data.
- Open-source language models can perpetuate or exacerbate stereotypical associations affecting marginalized communities.
- The study targets biases in popular, downloadable text-generation models applied with default settings.
- Over half a million monthly downloads motivate studying GPT-2 because biased outputs may affect widely used hiring-related applications.
- The analysis considers representational harms from occupation-related stereotypes, including associations between women and care-oriented jobs.
- 396K GPT-2 completions are analyzed across gender intersections with ethnicity, religion, sexuality, political affiliation, and continental name origin, alongside US labor data.
2 Related Work
Prior work documents bias in language models and develops probes for eliciting it. This paper extends that line by using fixed templates, larger sampling, direct distributional analysis, and comparison with US labor data.
- Generative language models can learn and sometimes amplify negative generalizations, stereotypes, and misrepresentations of social groups.
- Bias probes use batches of prompts to elicit model outputs whose demographic attributes can be analyzed.
- The paper uses one fixed sentence template, directly analyzes occupational distributions, generates an order of magnitude more samples, and compares outputs with US labor data.
- The probing protocol is chosen to study generative models through sentence completion rather than seed-word-sensitive embedding association tests.
- Intersectional analysis treats combined identities as necessary because separate gender and race analyses can distort marginalized groups’ experiences.
3 Methods
The methods probe default GPT-2 with identity- and name-based templates, extract occupational entities, model intersectional effects, and compare predictions with US labor statistics. The pipeline combines large-scale completions with distributional and regression analyses.
- 3.1 Model selection: GPT-2’s 124M-parameter version is selected as the most downloaded text-generation model, and inference hyperparameters remain fixed to represent out-of-the-box use.
- 3.2 Data collection: The study collects 396,000 total calls using prefix templates that vary protected identities, gender, and geographic name origin.
- 3.2 Data collection: Identity-based templates combine ethnicity, religion, sexuality, or political affiliation with man/woman baselines to probe intersectional associations.
- 3.2 Data collection: Name-based templates sample first names by gender across Africa, the Americas, Asia, Europe, and Oceania to examine inferred identity associations.
- 3.2 Data collection: Stanford CoreNLP NER extracts job titles from generated sentences, while sentences without detected titles are removed before frequency analysis.
- 3.3 Empirical Analysis: A 0.25% frequency threshold removes rare job tokens while preserving approximately 80% of the sample.
- 3.3 Empirical Analysis: Logistic regressions with gender, intersectional categories, and interaction terms estimate job probabilities and test effects beyond isolated main effects.
- 3.4 Comparison with US Labor Market Data: Model outputs are compared with 2019 US labor statistics covering gender and ethnicity shares across 567 occupational categories.
4 Results
GPT-2 produces narrower and more stereotypical occupational associations for women than men, with intersectional attributes significantly shaping returned jobs. Its gender-ethnicity predictions generally track US labor-market distributions, while some occupational outputs remain substantially more clustered or stereotypical.
- 4.1 Gender differences in distributions: 16 jobs account for 50% of men’s outputs versus 8 for women, while 66 versus 43 jobs reach 90%, indicating narrower occupational diversity for women.The Gini coefficients likewise confirm a more unequal distribution for women.
- 4.2 Gender differences in occupations: Men are associated with manual and professional occupations, whereas women are associated with domestic and caregiving roles.Over 90% of ‘prostitute’ returns were women and over 90% of ‘software engineer’ returns were men; only writer, reporter, and sales representative appeared gender-neutral.
- 4.3 Intersectional analysis: Across 262 logistic regressions, the woman dummy was significant in 71% of ethnicity regressions and 59% of political regressions, increasing pseudo-R2 by 3.3 percentage points on average.The regressions assessed whether gender and intersectional terms added explainability to job predictions.
- 4.3 Intersectional analysis: Gender and intersectional attributes significantly determine returned job tokens across a wide range of occupations, with some intersections anchoring GPT-2 to stereotypical occupation sets.The analysis found greater clustering of women into fewer jobs across intersections, especially sexuality, religion, and ethnicity.
- 4.4 Comparison to Labor Market Ground Truth: GPT-2’s gender-ethnicity predictions closely align with US population data, with the baseline woman group attaining Kendall-Tau 0.628 and all gender-ethnicity coefficients significant at the 1% level.Low MSEs further corroborate similarity between predicted and ground-truth distributions.
- 4.4 Comparison to Labor Market Ground Truth: GPT-2 over-predicts occupational concentration for women: the top five jobs comprise 41% of outputs versus 17% in US data, compared with 24% versus 10% for men.The most popular occupation accounts for 14% of women’s outputs and 8% of men’s outputs.
5 Discussion
GPT-2 reflects some occupational distributions found in US labor data but intensifies occupational clustering for women, associating them with fewer and less-diverse jobs. Intersectional prompts can substantially alter returned occupations and create both allocational and representational concerns.
- Intersectional associations: Religious intersections considerably change returned occupations, including 4% of Buddhist men predicted as monks despite an estimated requirement exceeding 70 monks per US Buddhist center.The authors caution that communicative intent may contribute because textual references to being Buddhist need not mean practicing a religion professionally.
- Societal consequences: Intersectional demographic associations can reduce allocational harms by weakening name-based stereotypes, while simultaneously introducing representational harms through blanket group-level occupational stereotypes.The paper frames this as a double-edged societal consequence for downstream applications such as automated CV screening.
- Occupation distribution per demographic: GPT-2 predicts women with a smaller and less-diverse set of occupations than men, and its top-five distributions are more unequal than US labor-force data.The Gini coefficients confirm greater inequality for women, even when comparing the same occupation categories in model predictions and labor data.
- Occupation distribution per demographic: GPT-2 reflects gender-ethnicity proportions for real-world occupations, yet its occupational clustering for women exceeds the observed US labor-force clustering.The paper distinguishes accurate demographic proportions within occupations from excessive concentration across the occupations associated with women.
- Interpretation: GPT-2’s bias lies less in which jobs are associated with women than in repeatedly associating women with a narrow set of stereotypically female-suited jobs.The authors characterize this concentration as potentially propagating damaging and self-fulfilling stereotypes.
- Limitations: The labor-market comparison is US-centric, lacks granular ground truth for several protected intersections, excludes informal-sector comparisons, and covers only binary gender identities.The authors recommend broader country coverage, richer intersectional ground truth, and analysis of more gender identities.
6 Conclusion
The paper asks whether widely used generative models should reflect or correct skewed societal distributions, focusing on occupational associations in GPT-2. It emphasizes that the model may reproduce societal occupational skews while also repeatedly returning a narrow set of stereotypical jobs, especially for women.
- 6 Conclusion: GPT-2 can reflect skewed societal occupational distributions for particular jobs while sometimes shifting those distributions toward gender parity.The paper presents this as a normative question rather than resolving whether models should mirror or correct existing inequalities.
- A Note on language used in this paper: The study focuses on occupational associations with binary gender identities, “man” and “woman,” rather than explicitly comparing sex through prompts such as “female worker is a...”.The authors state that non-binary gender identities warrant further study.
- A Note on language used in this paper: Preliminary prompts using “womxn” produced stereotypical job associations and outputs that sometimes failed to recognize “womxn” as a person noun.The authors present these observations as motivation for extending the same data-collection pipeline to non-binary identities.
- 6 Conclusion: The analysis intentionally uses the small GPT-2 model because it was the most downloaded text-generation model on HuggingFace and represents an out-of-the-box tool for non-expert users.The paper’s focus is practical exposure to inherited downstream bias rather than optimized prediction from tuned hyperparameters.
C GPT-2 Hyperparameter Ablation
GPT-2’s default decoding settings produce gender-skewed occupational diversity, while increasing top k expands job variety at the cost of coherence.
- The analysis varies top k and temperature independently while holding the other parameter at its default value.
- Temperature 1.0 yields the highest number of unique jobs, while both lower and higher temperatures reduce diversity.
- Increasing top k increases unique jobs but produces less coherent output, so diversity is not the sole evaluation target.
- At top k=10, men’s outputs contain 51 unique jobs versus 30 for women, with 19 versus 11 jobs comprising 95% of mentions.
D.1 Named Entity Recognition
The pipeline extracts occupations with named-entity recognition and adjusts GPT-2 and US labor data to make their distributions comparable despite differing group structure and job taxonomies.
- Stanford CoreNLP NER loses 10.6% of gender-occupation sentences and 19.6% of name-occupation sentences, with losses broadly balanced across groups.
- The US comparison assumes equal female proportions across ethnicities and uses 7,000 prompts per gender-ethnicity pair, creating equal predicted group representation.
- The adjustment rescales each predicted job share using the US gender and ethnicity shares relative to an artificial 12.5% population share.
- GPT-2 job titles are matched to the lowest comparable US occupational category, with broader matches for salesperson and manager.
E Comparison with XLNet
XLNet reproduces GPT-2’s central gender pattern: women receive narrower and more stereotypical occupational distributions, while intersectional effects vary across demographic categories.
- Distributional Analysis: XLNet assigns 9 jobs 50% of men’s outputs but only 6 jobs for women, and 30 versus 23 jobs account for 90% of outputs.
- Top occupations: XLNet’s top occupation is maid for women at 27%, compared with carpenter for men at 11%.
- Top occupations: Men and women are associated with stereotypical but model-specific jobs, with men linked especially to manual occupations.
- Intersectional effects: XLNet’s intersectional Gini patterns are less clearly divided by gender than GPT-2’s, but woman-sexuality combinations remain the most unequal.
- Intersectional effects: Across GPT-2 regressions, Buddhist membership is significant in 78% and Hindu membership in 55% of regressions.
- Intersectional effects: Religion, political affiliation, ethnicity, and sexuality main effects are significant in 66%, 60%, 53%, and 48% of regressions, respectively.
F.2 Full Regression Results
The full regressions show that gender and intersectional terms significantly structure GPT-2’s occupational associations, but gender generally explains more variation than interactions.
- Some occupations are segmented simultaneously by gender, ethnicity, and their interaction, whereas non-significant rows can reflect insufficient variation for model convergence.
- Interaction terms are significant in approximately one-third of GPT-2 occupation regressions but explain less variation than gender.
- The regressions encode significance using p < 0.05, with black cells indicating significant terms and white cells indicating non-significant terms.
- Coefficient visualizations distinguish negative, positive, and insignificant effects using red, blue, and white, respectively.
- Adding the woman dummy improves model fit more than adding interaction terms across GPT-2 occupation regressions.
G Comparison to Equi-Proportion Baseline for Intersectional Occupational Associations
GPT-2’s occupational associations vary across gender intersections, with religion, sexuality, and ethnicity producing especially strong disparities, while continental name origin has a weaker effect.
- Ethnicity: GPT-2 associates plumber, guard, contractor, and police officer with men, while secretary, prostitute, model, and babysitter are associated with women across ethnic intersections.Only Black men, rather than men of every ethnicity, are associated with prostitute; most occupations remain distributed across several ethnicities.
- Religion: GPT-2 generates large gender disparities for several religious occupations, including Buddhist-associated nuns, Jewish-associated rabbis, and Buddhist-associated monks, pastors, and priests.Hindu men and women are associated only with non-religious professions in the reported examples.
- Sexuality: Male-dominated occupations are almost exclusively associated with straight men, whereas massage therapist, counselor, and graphic designer are nearly unique to lesbian women.Detective, plumber, guard, and coach are among the occupations dominated by straight men.
- Political affiliation: Political intersections largely preserve the baseline gender split, with liberal men associated with critic and banker and conservative men with driver and host.Some occupations have equal representation across political affiliations.
- Name origin: Continental name origin has less effect on GPT-2’s returned occupation than explicit categorical intersections, while gender remains the stronger determinant.Jobs lie more tightly around the equi-proportion line for continent-based names, including mechanic and leader for men and women-associated occupations.
H Further Analysis for Intersectional Breakdowns
Across distributional and occupational breakdowns, women’s predicted jobs are generally less diverse and more concentrated than men’s, especially for ethnicity, religion, and sexuality intersections. Continental name-origin prompts are the main exception, producing more similar occupational sets across genders.
- Distributional Analysis: Ethnicity, religion, and sexuality intersections produce less diverse and more stereotypical occupational distributions for women than for men.The disparity is less apparent for political and continent-based name intersections, where predicted job distributions are more similar.
- Lorenz Curve Analysis: Women are concentrated in fewer job titles than men for all intersections except continent-based names, with the largest differences in religion and sexuality.The distributional difference is smaller for political intersections.
- Occupations by intersections: Most occupations remain split across ethnicities, while religion is also relatively evenly distributed except for a few occupations such as Buddhist-associated monks and nuns.The figures cover occupations exceeding the specified mention thresholds for each intersection.
- Occupations by intersections: Sexuality strongly concentrates some occupations among straight men or lesbian women, while political affiliation produces narrower occupational associations such as liberal men with banker and critic.Conservative men are associated with driver and host, and liberal women with barista and translator.
- Occupations by intersections: Continent-based name-origin prompts yield no occupations dominated by one origin and less gender disparity than explicit man/woman prompts.Top jobs by name are similar across genders, including writer, consultant, journalist, and lawyer.
- Top occupations: Women’s top-five occupations remain close to the baseline woman case across intersections except continent-based names, with waitress consistently among the top five.Men’s top-five occupations show more variety, and security guard is absent from some intersectional male lists.
I Further Analysis for US Comparison
The paper compares GPT-2’s occupational gender predictions with US labor data using correlation and error analyses. GPT-2 broadly tracks observed occupational gender and gender-ethnicity distributions while shifting skewed gender shares toward parity.
- Methods: Mean-square error and Kendall-Tau measure GPT-2’s deviation from US occupational gender shares, with all Kendall-Tau coefficients showing strong positive monotonic relationships significant at 1%.Kendall-Tau coefficients are reported in Table 14, while MSE is reported in Figure 4 of the main paper.
- Gender-share comparison: GPT-2 pulls skewed real-life occupational gender distributions toward parity, predicting more women in mechanics, carpentry, taxi driving, and policing than observed.It predicts fewer women in occupations including secretarial, maid, and nursing roles in the reported comparison.
- Gender-ethnicity comparison: GPT-2 accurately predicts gender-ethnicity splits for frequently mentioned occupations, especially for Asian and Black workers.Figure 29 compares predicted and observed proportions for the 50 most frequently mentioned jobs matched with US occupational categories.
- Motivation: AI hiring systems are already used or proposed for talent acquisition and HR virtual-assistant applications.The passage names Gartner-identified use cases and companies automating parts of hiring.