Source-linked AI summary
Semantics derived automatically from language corpora contain human-like biases
Aylin Caliskan, Joanna J. Bryson, Arvind Narayanan
TL;DR
AI systems trained on ordinary language may inherit human prejudice, challenging assumptions that computation is neutral. This paper tests that possibility with word embeddings and finds that language-derived representations reproduce a broad range of human-like biases.
Problem
It was unclear whether standard machine-learning models trained on ordinary language reproduce the human-like biases documented in psychological studies.
Method
The authors apply implicit-association-style tests to GloVe word embeddings trained on ordinary Web text, introducing WEAT and WEFAT to measure semantic and factual associations.
Results
The embeddings reproduce every linguistic bias examined from psychology, along with prejudicial hiring patterns and factual employment and naming information.
Takeaways & Limitations
AI systems learning the properties of language from human culture will inherit human-like prejudices, which must be compensated for rather than entirely eliminated.
Takeaways & Limitations
The same gender associations can be harmful for evaluating job applicants but useful for inferring historical employment patterns, so their value depends on context.
Abstract
from arXiv · showhide
Artificial intelligence and machine learning are in a period of astounding growth. However, there are concerns that these technologies may be used, either with or without intention, to perpetuate the prejudice and unfairness that unfortunately characterizes many human institutions. Here we show for the first time that human-like semantic biases result from the application of standard machine learning to ordinary language---the same sort of language humans are exposed to every day. We replicate a spectrum of standard human biases as exposed by the Implicit Association Test and other well-known psychological studies. We replicate these using a widely used, purely statistical machine-learning model---namely, the GloVe word embedding---trained on a corpus of text from the Web. Our results indicate that language itself contains recoverable and accurate imprints of our historic biases, whether these are morally neutral as towards insects or flowers, problematic as towards race or gender, or even simply veridical, reflecting the {\em status quo} for the distribution of gender with respect to careers or first names. These regularities are captured by machine learning along with the rest of semantics. In addition to our empirical findings concerning language, we also contribute new methods for evaluating bias in text, the Word Embedding Association Test (WEAT) and the Word Embedding Factual Association Test (WEFAT). Our results have implications not only for AI and machine learning, but also for the fields of psychology, sociology, and human ethics, since they raise the possibility that mere exposure to everyday language can account for the biases we replicate here.
Introduction
The paper shows that standard machine-learning systems trained on human language reproduce human-like semantic biases because language reflects cultural regularities, including prejudice. These findings challenge machine neutrality and suggest implications for explaining and addressing prejudice in both humans and machines.
- Machine learning captures knowledge transmitted through human culture, undermining the assumption that its outputs are neutral.
- Standard natural-language-processing tools reproduce biases documented in human psychological studies because word meanings reflect cultural regularities.
- Culturally derived data can transmit prejudice, motivating a new null hypothesis for how prejudice spreads between humans.
- Because prejudice is rooted in language, it is difficult but not impossible to address in humans or machines.
- The study replicates documented biases in attitudes toward ordinary objects, animals, and humans.
Meaning and Bias in Humans and Machines
The paper adapts the human Implicit Association Test to word embeddings to detect semantic bias in text. Because embeddings summarize uncontrolled aggregate web writing, their results cannot be directly numerically compared with individual-level IAT measurements.
- Prejudice is defined as harmful bias, identifiable by negative consequences and therefore requiring deliberate action rather than purely algorithmic elimination.
- The study documents human bias primarily with the Implicit Association Test, which detects differing reaction times when subjects pair similar versus dissimilar concepts.
- Its text-based method applies an implicit-association procedure to word embeddings, representing words as high-dimensional vectors derived from their textual contexts.
- To reduce noise from words’ multiple meanings, the analysis uses small concept baskets drawn from the psychological studies being replicated.
- Unlike IAT results from individual subjects, embeddings reflect uncontrolled aggregate human writing on the web and are not representative of a single population.
- These differences preclude direct numerical comparison because IATs provide p-values and effect sizes from statistically significant subject samples, whereas word-embedding measures do not.
Results
The study reports that standard word embeddings trained on ordinary Web language recover a broad range of documented human-like linguistic biases. Using established psychological benchmarks and WEAT, the analysis evaluates semantic associations through p-values and effect sizes.
- Results: The authors report finding every documented psychological linguistic bias they examined, with persuasive examples and uniformly high effect sizes rather than cherry-picked results.They present these findings as evidence that implicit human biases can be recovered from language regularities as part of linguistic meaning.
- Results: GloVe trained on ordinary World Wide Web language reproduced the reported biases, with similar results from other standard tools and corpora.The approach uses benchmarks and keywords established in well-known, heavily cited human-science, psychology, and sociology studies.
- Results: For each result, WEAT compares two target-concept sets with two attribute-word sets and reports a p-value and an effect size based on standard-deviation separation.The p-value quantifies how likely the observed similarity scores are under no semantic association between the target concepts and attributes.
Baseline: Replication of Associations That Are Universally Accepted
Using WEAT on GloVe embeddings, the study replicates universally accepted, morally neutral IAT associations: flowers over insects and musical instruments over weapons as more pleasant. These associations emerge from word co-occurrence statistics without GloVe’s direct experience of the world.
- Baseline: Replication of Associations That Are Universally Accepted: WEAT finds flowers closer to pleasant than unpleasant than insects, with effect size 1.50 and p-value < 10^-7.The analysis uses the same stimuli as Greenwald et al. (1998).
- Baseline: Replication of Associations That Are Universally Accepted: GloVe captures the flower–insect association solely from implicit word co-occurrence statistics, without direct experience of the world.
- Baseline: Replication of Associations That Are Universally Accepted: WEAT finds musical instruments closer to pleasant than unpleasant than weapons, with effect size 1.53 and p-value < 10^-7.The analysis uses the same stimuli as Greenwald et al. (1998).
Racial Biases
GloVe reproduces racial biases documented by implicit-association and résumé studies, showing that names alone carry recoverable associations relevant to pleasantness and interview invitations. The replications required adapting low-frequency stimuli while preserving balanced comparisons.
- Racial Biases: The study reproduced the finding that candidate names affect interview prospects, using semantic nearness to pleasantness as the correlate of an invitation.The analysis evaluated two pleasant/unpleasant attribute sets: the original IAT stimuli and a shorter revised set from Nosek et al. (2002a).
- Racial Biases: GloVe replicated attitudes toward European American and African American names by measuring their semantic nearness to pleasantness.Some original African American names were too infrequent in the corpus, so stimuli were altered and the same number of European American names were randomly removed for balance.
- Racial Biases: The original résumé experiment sent nearly 5,000 identical résumés to 1,300 job advertisements and found European American candidates 50% more likely to receive interviews.Only the candidates’ names differed across the résumés.
Gender Biases
Word embeddings reproduce established gender biases, associating women with family and arts while associating men with careers and science. They also recover veridical gender distributions for occupations and androgynous names with strong correlations.
- Gender Biases: Female names were more associated with family than career, while male names were more associated with career, with an effect size of 1.81 and p-value < 10−3.This replicated the reported human IAT pattern.
- Gender Biases: Female terms were more associated with arts than mathematics compared to male terms, with an effect size of 1.06 and a p-value of 10−2.The result matched the expected human association.
- Gender Biases: Female terms were associated more with arts and male terms with science, with an effect size of 1.24 and a p-value of 10−2.This reproduced the expected associations in a separate arts-and-sciences analysis.
- Gender Biases: ρ = 0.90 with p-value < 10−18: WEFAT predicted the percentage of women in the 50 most relevant occupations from occupation-word gender associations.The comparison used 2015 U.S. Bureau of Labor Statistics occupational participation data.
- Gender Biases: ρ = 0.84 with p-value < 10−13: WEFAT predicted the percentage of women among people with androgynous names.The analysis used 1990 U.S. census name and gender statistics, with a weaker correlation than for occupations but still striking significance.
Methods
The study uses pre-trained GloVe word embeddings from a large web corpus and introduces WEAT and WEFAT to quantify semantic associations and their relationships to factual properties. Statistical tests assess differential associations between word sets, while analyses examine whether occupation and name vectors encode real-world gender information.
- Embedding model: All experiments use pre-trained GloVe embeddings, where vector similarity reflects word co-occurrence in text and dimensionality reduction amplifies co-occurrence signals.The embeddings were distributed by their authors rather than trained for this study.
- Embedding model: The selected Common Crawl model contains 840 billion tokens, 2.2 million case-sensitive tokens, and 300-dimensional vectors.The corpus was the largest of four corpora with author-provided trained embeddings.
- WEAT: WEAT uses permutation tests to compare two target-word sets’ relative cosine similarities to two attribute-word sets under a no-difference null hypothesis.Its effect size is a normalized measure of separation between association distributions.
- WEAT: WEAT statistics and p-values concern words rather than people and compare sets of target concepts, so they do not have the same interpretation as IAT results.The method measures differential association between two target sets and an attribute.
- WEFAT: WEFAT tests whether normalized word–attribute associations predict factual properties using linear regression, including occupations’ female-worker percentages and names’ gender frequencies.Occupation data come from the Bureau of Labor Statistics, while name frequencies use popular names from 1990 U.S. Census data.
Discussion
The discussion argues that language-trained AI can inherit human prejudices because statistical learning reproduces group regularities embedded in language. It emphasizes that such bias is consequential in applications, difficult to define or remove algorithmically, and requires explicit human oversight and interdisciplinary safeguards.
- Origins of prejudice: The findings support a parsimonious hypothesis that unthinking reproduction of linguistic group regularities and ingroup preference can contribute to prejudicial behavior.The authors stress that explaining prejudice does not justify prejudiced behavior, and implicit IAT bias does not necessarily predict cooperative performance.
- AI consequences: AI can inherit substantially the same biases as humans, but its consequences differ because machine-learning systems increasingly receive agency in societal tasks.Unlike humans, machine learning can be shut off after deployment, while its outputs may affect applications ranging from predictive text to criminal sentencing.
- Application risks: Word embeddings can transmit racial and gender stereotypes into downstream applications, including sentiment analysis and machine translation.Gender associations in word vectors almost perfectly predict whether translations use “he” or “she” for gender-neutral source-language sentences.
- Limits of correction: The authors argue that embeddings capture the entire spectrum of human biases because bias is inseparable from meaningful language use.This makes merely removing specific, enumerable stereotypes insufficient for addressing the broader problem.
- Limits of correction: Algorithmically correcting prejudice is difficult because understandings of prejudice, humanity, and human rights evolve and vary across cultures.The discussion also distinguishes harmful stereotypes in hiring from potentially informative stereotypes in tasks that analyze the real world.
- Proposed response: The authors propose combining statistical information with explicit rules, instructions, and context-sensitive evidence rather than relying on implicit bias alone.They recommend interdisciplinary research and selecting training corpora with as little prejudice as possible, using their tools to identify it.