Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes

Nikhil Garg; Londa Schiebinger; Dan Jurafsky; James Zou

Source-linked AI summary

Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes

Nikhil Garg, Londa Schiebinger, Dan Jurafsky, James Zou

arXiv:1711.08412v1Nov 22, 2017cs.CLcs.CY

TL;DR

The paper addresses the difficulty of measuring how gender and ethnic stereotypes change across time and social groups. It develops a temporal word-embedding framework linked to U.S. Census data, finding that embedding changes track demographic and occupation shifts while revealing evolving stereotypes. The analysis is exploratory and bounded by limitations in historical text and category definitions.

Problem
Existing stereotype studies often require manual surveys or qualitative analysis, motivating a scalable way to quantify gender and ethnic stereotype trends over time.
Method
The authors analyze word embeddings trained on 100 years of text, quantify associations for groups, adjectives, and occupations, and compare them with census data.
Results
Embedding dynamics track changes in gender and ethnic stereotypes and significantly correspond with demographic and occupation shifts in the United States.
Takeaways & Limitations
Word embeddings provide quantitative measures and visualizations that complement qualitative linguistic and sociological analyses of historical bias.
Takeaways & Limitations
Historical embeddings may reflect written text imperfectly, and the analysis is limited to two major binary gender categories.

Abstract

from arXiv · show

Word embeddings use vectors to represent words such that the geometry between vectors captures semantic relationship between the words. In this paper, we develop a framework to demonstrate how the temporal dynamics of the embedding can be leveraged to quantify changes in stereotypes and attitudes toward women and ethnic minorities in the 20th and 21st centuries in the United States. We integrate word embeddings trained on 100 years of text data with the U.S. Census to show that changes in the embedding track closely with demographic and occupation shifts over time. The embedding captures global social shifts -- e.g., the women's movement in the 1960s and Asian immigration into the U.S -- and also illuminates how specific adjectives and occupations became more closely associated with certain populations over time. Our framework for temporal analysis of word embedding opens up a powerful new intersection between machine learning and quantitative social science.

1 Introduction

The paper introduces word embeddings as a scalable quantitative framework for measuring temporal changes in gender and ethnic stereotypes. Applied to a century of U.S. text, the framework tracks social and demographic shifts while revealing changing associations.

Existing stereotype research relies heavily on surveys, qualitative analysis, or specialized linguistic knowledge that can be costly and difficult to scale.
Word embeddings represent words as vectors whose geometry captures semantic relationships and can encode stereotypes present in training text.
The framework analyzes embeddings trained over 100 years to quantify how gender and ethnic stereotypes change over time.
Embedding dynamics strongly correlate with demographic and occupation shifts in U.S. society, including the women’s movement and Asian American population growth.
The authors validate findings against external metrics and across training algorithms, showing that embeddings quantify evolving stereotypes toward women and ethnic groups.

2 Overview of the embedding framework and validations

The framework combines temporal word embeddings, demographic word lists, and census occupation data, then validates embedding bias against observed participation patterns. Across gender and ethnic analyses, embedding associations track occupation proportions and their historical changes.

Data and methods: The study uses contemporary Google News, decade-level Google Books/COHA, and yearly New York Times embeddings, alongside word lists for genders, ethnicities, adjectives, occupations, and neutral terms.
Data and methods: Occupation participation from U.S. Census data provides an external benchmark for comparing demographic proportions with embedding bias.
Data and methods: Embedding bias is quantified by comparing average distances from representative group vectors to vectors in a neutral word list.
Gender validation: p < 10^-9 with r2 = .46: gender occupation bias significantly correlates with women’s occupation proportions in Google News embeddings.
Gender validation: Across 1910–1990 decade embeddings, gender bias significantly correlates with occupation frequency, with similar relationships over time.
Gender validation: From the 1950s to the 1990s, average occupation bias moves toward zero alongside increasing women’s participation.
Ethnic validation: Ethnic occupation bias significantly predicts census occupation proportions for Hispanics at p < 10^-5 and Asians at p < .05.
Ethnic validation: For Asian Americans relative to Whites, increasing occupation proportions across time are well tracked by embedding bias.

3 Using embeddings to quantify historical gender stereotypes

Word embeddings quantify historical gender stereotypes by relating occupation and adjective associations to demographic and human-rated benchmarks. Across decades, gender bias decreased but remained significant, with major shifts around the women’s movement.

Occupation stereotypes: Crowd-sourced stereotype scores remain associated with embedding bias after controlling for occupation proportions, whereas occupation proportions do not.The reported associations are r2 = .66 for crowd-sourced scores and r2 = .41 for occupation proportions; in the joint regression, p < 10^-5 versus p > .2.
Adjective stereotypes: Adjective embedding biases correlate significantly with human-assigned gender stereotype scores for both the 1970s and 1990s.Both correlations have p < .0002, and human-rated gender-neutral adjectives tend to be unbiased in the embeddings.
Adjective stereotypes: Intelligence-related adjectives became less male-associated over time, especially after the 1960s, while adjective associations changed substantially across historical periods.The intelligence adjective group shows a positive trend with p < .005, and decade-pair correlations show a sharp divide between the 1960s and 1970s.
Adjective stereotypes: Individual word associations also shifted: hysterical fell from a top-five woman-biased word in 1920 to outside the top 100 in 1990, while emotional became more woman-associated.The analysis emphasizes large shifts rather than isolated rankings because embeddings are noisy.
Overall findings: Overall, the framework depicts decreasing but still significant gender bias and provides quantitative evidence of systemic change in women’s portrayals.The authors frame the method as quantitative exploratory analysis rather than a causal model of how stereotypes arise.

4 Using embeddings to quantify historical ethnic stereotypes

Word embeddings also track changes in ethnic stereotypes across time, linking shifts in Asian-American associations and portrayals of Islam to broader historical and media trends. These analyses extend the framework beyond gender to other ethnic and cultural groups.

Asian stereotypes: Asian-American adjective associations show phase shifts over the twentieth century, indicating that external historical events changed attitudes.The analysis uses common and distinctly Asian last names and compares adjective-bias correlations across embeddings over time.
Asian stereotypes: Words describing outsiders show a steadier change in Asian association than the broader adjective set, enabling more precise measurement of stereotype evolution.The broader adjective analysis contains two distinct phase shifts, whereas outsider-related words change steadily.
Religious portrayals: In New York Times data, Islam remains more associated with terrorism-related words than Christianity across the analyzed period.The measure compares words such as terror, bomb, and violence with religion-related words such as mosque and church.
Other ethnic groups: Russian-name embeddings show a dramatic shift in the 1950s and a smaller shift during the initial years of the Russian Revolution, while Hispanic names provide a steadier control pattern.These comparisons illustrate how the framework can examine ethnic attitudes around significant global events.

5 Discussion

The study uses temporal word-embedding geometry to quantify how gender and ethnic stereotypes evolve and to compare those associations with demographic changes. Its conclusions are exploratory, with robustness checks supporting generalizability but limitations from data, metric, word-list, and black-box choices.

Findings: Embedding gender and ethnic occupation bias significantly tracks actual U.S. occupation frequencies over time.Occupation associations provide a validation comparison against empirical participation rates.
Findings: The framework quantifies changing stereotypes toward women and ethnic groups, including biases that decrease and others that increase over time.It also examines how adjectives and occupations become more closely associated with particular populations.
Limitations: The selected relative norm difference metric and word lists may affect robustness and recall, although alternate choices reproduce similar results.The appendix repeats occupation analysis with professional occupations and obtains an identical figure.
Robustness: Results are checked against external metrics and remain robust across different word-embedding training algorithms.Replication across embeddings and measured bias types suggests generalizability.
Limitations: Historical interpretation is constrained because written text may not fully reflect popular attitudes, while census proportions imperfectly capture social associations.The authors treat written text as a window into attitudes expressed in popular culture rather than a complete measure of them.
Scope: Because the embeddings are black boxes, the framework supports quantitative exploratory analysis rather than specific causal explanations of how stereotypes arise.Future work could use embeddings with interpretable dimensions and finer-grained temporal modeling.

6 Data & Methods

The paper combines multiple pretrained and historical word-embedding datasets with group, occupation, and adjective word lists to measure stereotype associations. It represents groups by averaged vectors and quantifies relative association using norm-based distances, with census and human ratings providing external comparisons.

Embeddings: The study uses publicly available embeddings spanning Google News, decade-specific Google Books/COHA, and New York Times corpora.Google Books/COHA supplies separate decade embeddings; the main text uses SGNS, while SVD results are qualitatively similar.
Related Work: The paper contrasts temporal stereotype analysis with prior work that studied static embedding bias or aimed to debias embeddings.Its stated focus is historical change, including attitudes toward women and ethnic minorities through adjective associations.
Word Lists: Group words represent genders and ethnicities, while neutral words include occupations and adjectives used to measure relative associations.Gender uses noun and pronoun pairs; ethnicity is represented through last-name lists differentiated by U.S. ethnic distributions.
External Metrics: Occupation associations are compared with census participation data, while adjective associations are compared with human stereotype scores or analyzed over time.The adjective lists include a 1977/1990 human-labeled set and a larger list primarily drawn from Gunkel (1987).
Bias Metric: Relative norm difference averages distances from each neutral-word vector to two representative group vectors and subtracts those average distances.The representative vectors are averages of the words in each group; the sign indicates which group has the stronger association.
Bias Metric: Cosine similarity provides an alternative to the 2-norm, and the paper reports high agreement between the resulting metrics.The main text uses relative norm distance because the vectors have norm 1 and the denominators can be omitted.

Appendix A Data

The appendix documents the supplementary embeddings, group and neutral-word lists, and occupation subsets used to support reproducible analyses. It also records hand-coding choices and a limitation that professional occupations were not systematically defined.

Embeddings: Supplementary embeddings include Wikipedia 2014+Gigaword and Common Crawl vectors trained with GloVe.The Wikipedia vectors combine Wikipedia and newswire data; the Common Crawl vectors are trained on Common Crawl.
Group Lists: Gender and ethnicity group lists contain gendered words and surname sets for White, Hispanic, Asian, and Russian groups.The appendix lists the component words and surnames, including separate lists for women and men.
Group Lists: The ethnicity procedure identifies 20 last names per group using frequency and ethnicity-conditioned surname information.The selection begins from top surnames by ethnic percentage and total ethnic-group count.
Neutral Lists: Neutral-word resources include broad occupations, professional occupations, stereotype-scored occupations, and multiple adjective categories.Adjective categories cover intellectual, physical-appearance, terrorism-related, and other descriptors, alongside gender-stereotype adjectives.
Data Construction: Professional occupations were hand-coded from the overall occupation list, and the authors note that follow-on work should study this classification more systematically.This is an explicit limitation of the appendix data construction.

A.4 Embedding Quality

The average vectors’ quality remains stable over time, so changing vector quality does not explain the observed temporal trends.

Average vector quality does not appreciably change over time and therefore cannot explain the overtime trends.Word counts increase as datasets grow, while average variance across embedding dimensions remains relatively steady.

A.5 Similarity metrics

The study primarily uses relative norm difference, but relative cosine similarity produces highly consistent results.

Pearson correlation exceeds .95 in general between relative norm difference and relative cosine similarity metrics.The two metrics show nearly identical relationships across embedding, neutral-word, and group combinations.
The analysis primarily reports results using the relative norm difference bias metric.

B.1 Additional Validation Analysis

Additional analyses validate that embedding bias tracks occupation demographics consistently over time and remains comparable across modeling choices. Supplementary tables, figures, and robustness checks document these relationships for gender analyses.

B.1 Additional Validation Analysis: A relative woman bias of −.05 corresponds to approximately 12% women in an occupation regardless of embedding decade.The mapping remains consistent even as occupation proportions shift considerably over time.
B.1 Additional Validation Analysis: Embedding bias alone significantly predicts occupation proportion across all years, with p < 10−37.
B.1 Additional Validation Analysis: The aggregate model’s decade-specific Mean Squared Error is within about 10% of individually trained models.R-squared values and coefficient confidence intervals are also similar across aggregate and decade-specific models.
B.1 Additional Validation Analysis: The supplementary analysis includes occupation-proportion figures, regression tables, model comparisons, and Pearson correlations for embedding-bias analyses.
B.1 Additional Validation Analysis: Supplementary materials report gender-associated occupations and adjectives by decade, alongside regression tables and plots for robustness analyses.The authors caution that noisy embedding associations should be interpreted in aggregate rather than individually.

Supplementary for gender time dynamical analysis, SVD embeddings

The supplementary gender time-dynamics analysis repeats key analyses with SVD embeddings and provides additional occupation-bias figures, regression results, and correlations.

Supplementary for gender time dynamical analysis, SVD embeddings: The full gender time-dynamics analysis is repeated using SVD embeddings as a robustness check.
Supplementary for gender time dynamical analysis, SVD embeddings: Supplementary figures report occupation-proportion versus embedding-bias plots and regression information for SVD embeddings.
Supplementary for gender time dynamical analysis, SVD embeddings: Pearson correlations are reported for SVD occupation and adjective embedding-bias scores across decades.

C.1 Snapshot Analysis

The snapshot analysis examines ethnic bias in occupations and adjectives, comparing embedding-based associations with occupational representation and tracking bias similarity across decades.

Occupation proportions: Hispanic occupational representation is positively associated with relative embedding distance, with p < 10−5 and r-squared= .277.More positive values indicate stronger Hispanic association in both occupational proportion and relative distance.
Occupation proportions: Asian occupational representation is positively associated with relative embedding distance, with p < .05 and r-squared= .069.More positive values indicate stronger Asian association in both occupational proportion and relative distance.
Occupation proportions: Figure 32 compares relative Hispanic bias in SGNS embeddings with the average conditional log proportion of Hispanic workers across occupations.The figure presents embedding bias in blue and occupational representation in green.
Adjective bias: Figures 33 and 34 report Pearson correlations between adjective bias scores in SGNS embeddings from different decades for Russian and Hispanic bias, respectively.The supplementary material also includes a table of top Asian-versus-White adjectives over time by relative norm difference.