Source-linked AI summary
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
TL;DR
Gender bias in occupation classification matters because automated recruiting and hiring can affect professional opportunities. The paper analyzes this bias using occupation classification on online biographies across semantic representations, comparing explicit gender indicators with versions in which they are scrubbed. Scrubbing reduces TPR gender gaps while preserving roughly similar overall accuracy, but significant gaps remain and correlate with existing occupational gender imbalances.
Problem
Occupation classification in automated recruiting and hiring may reproduce gender bias linked to existing occupational gender imbalances, with potentially negative consequences.
Method
The paper studies occupation classification on hundreds of thousands of online biographies using three semantic representations, with and without explicit gender indicators.
Results
Scrubbing explicit gender indicators reduces TPR gender gaps while overall accuracy remains roughly the same, but significant gaps and correlations with occupational gender imbalances remain.
Takeaways & Limitations
Scrubbing sensitive attributes alone is insufficient to remove gender bias, and occupation classifiers may compound existing gender imbalances.
Takeaways & Limitations
The binary gender model simplifies gender and erases people who do not fit its assumptions, while the dataset does not contain all Internet biographies.
Abstract
from arXiv · showhide
We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators---such as first names and pronouns---in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are "scrubbed," and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.
1 INTRODUCTION
The paper examines gender bias in occupation classification, motivated by the high stakes of automated hiring and recruiting. It studies how explicit gender indicators and their removal affect classifier bias and potential compounding of occupational gender imbalances.
- Automated decision-making systems increasingly shape access to information, products, job postings, and professional opportunities.
- Occupation classification is susceptible to gender bias stemming from existing gender imbalances in occupations.
- The study uses hundreds of thousands of English online biographies to analyze gender bias in occupation classification.
- The analysis compares bag-of-words, word embeddings, and deep recurrent neural networks with explicit gender indicators available or scrubbed.
- The paper finds that scrubbing reduces TPR gender gaps while preserving overall accuracy, but significant gaps remain and may compound existing imbalances.
2 RELATED WORK
Related work situates the study within research on bias in semantic representations, hiring, gender inference, proxy behavior, and fairness metrics. The paper connects TPR gender gaps with existing gender imbalances and compounding injustices.
- Prior work documents gender bias in word embeddings, coreference resolution, and other language technologies.
- Research on structured data examines proxy behavior and differential subgroup validity when sensitive attributes are omitted or unevenly predictive.
- The paper measures bias using occupation-level TPR gender gaps, related to equality of opportunity, and studies their correlation with occupational gender imbalances.
- The study also relates to hiring experiments and research on gendered self-presentation in professional online profiles.
3 DATA COLLECTION PROCESS
The authors construct a large English-language biography dataset from Common Crawl, organized around twenty-eight occupations. They describe its occupational coverage, biography lengths, sampling boundaries, and stratified evaluation splits.
- Online biographies were identified in Common Crawl by matching name-like openings followed by occupation statements from the BLS classification system.
- The collection retained biographies for the twenty-eight most frequent occupations, with some related occupations merged and duplicates removed.
- 397,340 biographies span twenty-eight occupations, ranging from 118,400 professor biographies to 1,406 rapper biographies.
- Biography lengths range from eighteen to 194 tokens, with a median length of seventy-two tokens.
- The dataset uses stratified-by-occupation splits of 65% training, 10% validation, and 25% testing.
4 METHODOLOGY
The methodology predicts occupations from the remainder of each biography using three semantic representations, under conditions with or without explicit gender indicators. It operationalizes scrubbing by deleting names and gendered terms before representation.
- The task predicts the occupation stated in a biography’s first sentence from the remainder of its text.
- 4.1 Semantic Representations: The study compares bag-of-words, averaged word embeddings, and deep recurrent neural-network representations.
- Gender-indicator conditions: Each representation is evaluated with all word types or after deleting first names and gendered pronouns as explicit gender indicators.
- Bag-of-words: Bag-of-words represents each biography as a sparse vocabulary vector and uses one-versus-all L2-regularized logistic regression.
- Word embeddings: Word embeddings represent each biography by averaging fastText embeddings for its words, followed by one-versus-all L2-regularized logistic regression.
- Deep recurrent neural networks: The deep recurrent representation uses fastText inputs, a bidirectional GRU, attention, and a weighted-sum representation for occupation prediction.
5 ANALYSIS AND RESULTS
The analysis measures occupation-classification bias through gender differences in true positive rates, comparing three semantic representations with and without explicit gender indicators. It finds that scrubbing reduces some gaps but leaves proxy information and correlations that can compound existing occupational gender imbalances.
- Bias metric: The study measures bias as the occupation-specific difference in true positive rates between binary genders.TPR is the proportion of people with a given gender and occupation who are correctly classified into that occupation.
- With Explicit Gender Indicators: Figure 3 compares the female TPR gender gap with the female share of each occupation for BOW representations using explicit indicators.The corresponding quantities are Gapfemale,y and πfemale,y.
- Cross-representation pattern: The TPR gender gap is positively correlated with occupational gender imbalance across BOW, word-embedding, and DNN representations, with and without explicit indicators.Figure 4 reports correlations of 0.85, 0.74, 0.86, 0.71, 0.82, and 0.74 for BOW-w, BOW-wo, WE-w, WE-wo, DNN-w, and DNN-wo, respectively.
- Compounding imbalance: 14.6% of surgeons in the testing split were women, but BOW correctly classified 71% of male surgeons versus 54.5% of female surgeons.Women consequently constituted 11.6% of the true positives, compounding the occupation’s gender imbalance.
- Counterfactuals: Swapping explicit gender indicators changed predictions for 5.5% of BOW, 12.2% of word-embedding, and 4.6% of DNN biographies.Occupation-pair analyses identify gender-indicator-dependent changes in otherwise incorrect or correct predictions.
- Without Explicit Gender Indicators: Gender information remains recoverable after scrubbing, with gender-classifier accuracies of 65.5% for BOW and 68.2% for DNN.The authors also report that scrubbing leaves some very large occupation-specific TPR gaps and does not prevent compounding gender imbalances.
- Proxy behavior: DNN attention weights identify words that act as gender proxies even when explicit gender indicators are absent.Non-zero attention to “women” can occur even for occupations with small or no TPR gender gaps, so gender information does not necessarily produce a gap.
6 DISCUSSION AND FUTURE WORK
The study finds persistent gender bias across representations, with TPR gaps correlated with occupational gender imbalances and potentially compounding them, especially under repeated classification. Scrubbing reduces gaps without substantially changing overall accuracy but does not eliminate proxy-based bias.
- Significant TPR gender gaps appear across bag-of-words, word embeddings, and deep recurrent neural network representations.
- TPR gender gaps correlate with existing occupational gender imbalances, which may compound those imbalances.
- Repeated encounters with occupation classifiers can make compounding imbalances especially problematic by further underrepresenting the already underrepresented gender.
- The paper notes that fairness may fail under composition, paralleling the possibility that sequential classifier use compounds unfairness.
- Scrubbing explicit gender indicators reduces TPR gender gaps while leaving overall classifier accuracy roughly unchanged.
- Even after scrubbing, gender remains predictable above random, significant gaps remain for some occupations, and non-explicit differences persist in biographies.
- The analysis focuses on gender, while race- and socioeconomic-status biases may also occur in occupation classification and recruiting systems.
A TRUE POSITIVE RATE GENDER GAPS ACROSS REPRESENTATIONS
The figures compare occupation-level TPR gender gaps with the percentage of females in each occupation across three semantic representations and indicator settings. The series cover BOW without indicators, WE with and without indicators, and DNN with and without indicators.
- Figure 9 plots gender gap per occupation against the percentage of females in the occupation for BOW without gender indicators.
- Figures 10 and 11 show the corresponding occupation-level comparison for word embeddings with and without gender indicators.
- Figures 12 and 13 show the corresponding comparison for deep neural networks with and without gender indicators.
B.1 Attention to gender proxies
The DNN’s attention to the proxy words “wife” and “husband” shifts when explicit gender indicators are removed. The shift is smaller than for “women,” consistent with lower aggregate attention in the gender-prediction model.
- Figure 14 compares aggregated DNN attention to “wife” and “husband” with and without gender indicators.
- Without gender indicators, the model places more attention on “wife” and “husband,” although the shift is smaller than for “women.”
B.2 Attention to gender indicators
The DNN continues to use gender-related signals during occupation prediction even when some indicators were absent during training. Word embeddings allow the model to infer information from an unseen gendered word.
- Figure 15 compares attention to “she” during occupation prediction for models trained with and without gender indicators.
- The model trained without gender indicators attends much more to “she” at prediction time, despite not seeing it during training.
- Word embeddings enable the model to learn about words it has not explicitly seen, allowing “she” to provide a stronger gender signal during prediction.