Source-linked AI summary
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai
TL;DR
Word embeddings can encode and amplify gender stereotypes, yet their geometry and practical implications were not adequately recognized. This paper identifies the gender subspace, debiases embeddings while preserving useful gender-specific associations, and finds that the resulting representations reduce stereotypical bias.
Problem
Prior work had not recognized that widely used word embeddings encode gender stereotypes, creating a risk of bias amplification in machine-learning applications.
Method
The paper identifies a gender direction or subspace, measures direct and indirect bias, and neutralizes or equalizes gender-neutral representations while retaining gender-specific structure.
Results
The embedding’s biases closely align with crowd-worker-judged gender stereotypes, while debiasing makes neutral associations more equal and preserves analogies such as she:grandmother :: he:grandfather.
Takeaways & Limitations
Debiased embeddings can help machine-learning systems avoid inadvertently amplifying gender stereotypes while retaining useful semantic relationships.
Takeaways & Limitations
The paper focuses on gender bias, while other stereotypes, including racial stereotypes, require separate quantification and removal methods.
Abstract
from arXiv · showhide
The blind application of machine learning runs the risk of amplifying biases present in data. Such a danger is facing us with word embedding, a popular framework to represent text data as vectors which has been used in many machine learning and natural language processing tasks. We show that even word embeddings trained on Google News articles exhibit female/male gender stereotypes to a disturbing extent. This raises concerns because their widespread use, as we describe, often tends to amplify these biases. Geometrically, gender bias is first shown to be captured by a direction in the word embedding. Second, gender neutral words are shown to be linearly separable from gender definition words in the word embedding. Using these properties, we provide a methodology for modifying an embedding to remove gender stereotypes, such as the association between between the words receptionist and female, while maintaining desired associations such as between the words queen and female. We define metrics to quantify both direct and indirect gender biases in embeddings, and develop algorithms to "debias" the embedding. Using crowd-worker evaluation as well as standard benchmarks, we empirically demonstrate that our algorithms significantly reduce gender bias in embeddings while preserving the its useful properties such as the ability to cluster related concepts and to solve analogy tasks. The resulting embeddings can be used in applications without amplifying gender bias.
1 Introduction
Word embeddings encode useful semantic relationships but also capture and amplify gender stereotypes present in training text, creating risks for machine-learning applications. The paper quantifies these biases, distinguishes direct from indirect forms, and develops debiasing methods intended to reduce stereotypes while preserving embedding utility.
- Problem: The w2vNEWS embedding produces gender-stereotypic analogies, including “man is to computer programmer as woman is to homemaker” and “father is to doctor as mother is to nurse.”These outputs expose sexism implicit in the training data rather than merely neutral semantic relationships.
- Motivation: Word embeddings both reflect gender stereotypes in broader society and can amplify them when used as basic features in machine-learning systems.The introduction illustrates this risk through downstream applications such as web-page retrieval.
- Bias definitions: The paper defines direct bias as unequal association between a gender-neutral word and a gender-specific pair, and indirect bias as gender-related association between gender-neutral words.For example, receptionist being closer to softball than football may reflect female associations with both words, although some same-gender associations are legitimate.
- Empirical alignment: Crowd workers judged the embedding’s vector locations and analogies to align closely with common gender stereotypes.Examples include doctor being closer to man than woman and “he:coward :: she:whore.”
- Debiasing: The debiasing goal is to reduce gender bias by exploiting a low-dimensional gender subspace while preserving the embedding’s useful properties.The stated objectives include making gender-neutral words equidistant between gender pairs and reducing gender associations among neutral words.
2 Related work
Prior work examines gender bias in language, online systems, and algorithms, including implicit associations, stereotypes, and discriminatory effects. Related research also identifies bias in word embeddings and algorithmic fairness, but embedding debiasing lacks a predefined classification task and must address direct and indirect bias.
- Bias in language: Gender bias in language can reinforce psychological group status and appear as implicit gender-word associations that people may not report or recognize.Implicit Association Tests have uncovered such biases, and researchers have studied gender bias across multiple contexts.
- Bias in language: Benevolent sexism can associate women with positive traits while reducing their association with professional or other valued concepts.The literature characterizes this seemingly positive framing as potentially harmful, insulting, and discriminatory.
- Bias in online systems: Online news, Web search, Wikipedia, advertising, and predictive systems have exhibited gender, racial, or other demographic biases.Reported examples include emphasizing women’s gender and family relationships in Wikipedia and showing gender or racial discrimination in online systems.
- Word-embedding debiasing: Schmidt independently identified bias in word embeddings and proposed removing multiple gender dimensions to eliminate gender from the embedding.The paper describes that approach as similar to, but simpler than, its own.
- Algorithmic fairness: Algorithmic-fairness research modifies data or classifiers, but word embeddings lack clear individuals, sensitive features, and a predefined classification problem while still exhibiting direct and indirect bias.The authors’ approach is more similar to methods that modify the underlying data, yet prior methods are difficult to apply without a classification task, especially under indirect bias.
3 Preliminaries
The paper formalizes word embeddings as normalized vectors, distinguishes gender-neutral words from gender-definition pairs, and measures similarity using cosine or inner products. Its experiments use a normalized 300-dimensional Google News word2vec embedding and U.S.-based Mechanical Turk workers to assess stereotypes.
- Embedding and terminology: Each word is represented by a unit vector in R^d, with gender-neutral words collected in N and F-M gender pairs collected in P.Examples include flight attendant and shoes for N, and she-he or mother-father for P.
- Similarity: Word similarity is measured by the inner product, which equals cosine similarity when word vectors are normalized.For arbitrary vectors, cosine similarity is cos(u, v) = u · v / (∥u∥∥v∥).
- Embedding: Unless otherwise stated, the experiments use w2vNEWS, a d = 300-dimensional word2vec embedding pretrained on the Google News corpus and normalized to unit length.The embedding was downloaded pretrained and selected from the 50,000 most frequent words.
- Crowd experiments: Human experiments were conducted with U.S.-based Amazon Mechanical Turk workers using word-solicitation and rating tasks.The tasks examined whether embedding and crowd biases contain one another, using words or analogies generated from the embedding for ratings.
- Crowd experiments: Because gender associations vary by culture and person, the study asks workers to rate stereotypes rather than their own gender biases.The paper states that stereotype ratings provide greater consistency and may be more comfortable for participants.
4 Gender stereotypes in word embeddings
The section evaluates gender stereotypes in word embeddings through occupational associations and analogy judgments, finding that these biases are consistent across embedding methods. It also introduces an approach for examining indirect bias among gender-neutral words.
- Occupational stereotypes: Gender bias in occupations is highly consistent across w2vNEWS and GloVe web-crawl embeddings, with Spearman ρ = 0.81.The consistency suggests the observed stereotypes are not artifacts of one training corpus or word2vec methodology.
- Occupational stereotypes: Occupational words are projected onto the she–he direction and evaluated by ten crowd-workers on a 0–10 stereotype scale.This measures how geometric occupation bias agrees with human judgments of female-, male-, or neutral-stereotypic associations.
- Analogies exhibiting stereotypes: The modified analogy task generates word pairs x and y such that a seed pair, such as he and she, is analogous to x and y.The method systematically produces pairs associated with the seed direction for stereotype evaluation.
- Analogies exhibiting stereotypes: Of 150 generated analogies, 72 were rated gender-appropriate and 29 were rated as exhibiting gender stereotypes by at least five crowd-workers.Each analogy was judged by ten U.S.-based crowd-workers for analogy coherence and gender stereotyping.
- Indirect gender bias: Indirect gender bias is tested by projecting occupation words onto directions defined by pairs of gender-neutral words, such as softball and football.The analysis examines whether gender bias affects the geometry among gender-neutral words themselves.
5 Geometry of Gender and Bias
The section characterizes embedding gender bias geometrically by identifying a dominant gender direction and defining metrics for direct and indirect bias. It shows that gender-related associations can distort similarities involving ostensibly neutral words, such as receptionist, waitress, and homemaker.
- Gender direction: Gender bias is represented by a direction g estimated by aggregating multiple gender-pair difference vectors rather than relying on individual word pairs.The approach combines directions such as she–he and woman–man to robustly estimate gender despite polysemy, differing pairwise biases, and sampling noise.
- Gender direction: The first principal component of ten gender-pair difference vectors explains substantially more variance than the remaining components, indicating a dominant gender direction.The comparison against random unit vectors is used to assess whether the observed eigenvalue decrease is greater than expected from sampling noise.
- Direct bias: Direct gender bias measures the overlap of gender-neutral word vectors with g, with parameter c controlling whether bias is measured strictly or more gradually.When c = 0, any nonzero overlap is treated as biased; c = 1 provides a more gradual measure.
- Indirect bias: Indirect bias measures how removing the gender component changes the similarity between two word vectors after projecting out and renormalizing the gender subspace.The metric is based on decomposing each vector into gender-aligned and orthogonal components, and it satisfies β(w, w) = 0.
- Indirect bias: Indirect gender bias explains substantial similarity patterns: receptionist, waitress, and homemaker are closer to softball than football, with β values of 67%, 35%, and 38%, respectively.Businessman and maestro are similarly closer to football, suggesting these associations can largely be attributed to gender bias in the embedding.
6 Debiasing algorithms
The debiasing framework identifies a gender subspace and then either removes gender components from neutral words while equalizing designated sets, or softly reduces bias while preserving embedding similarities. Hard debiasing eliminates pair bias, whereas soft correction trades off bias reduction against preservation of original inner products.
- Framework: The algorithms operate on word sets, including words to neutralize and equality sets, enabling generalization beyond gender to racial or religious biases.The neutralization set may be specified externally or derived from the embedding.
- Framework: The method first identifies a direction or subspace capturing bias, then applies Neutralize and Equalize or Soften.The bias subspace is obtained from the defining word sets using SVD.
- Hard debiasing: Neutralize projects gender-neutral words out of the gender subspace, while Equalize makes words in each equality set equidistant from every neutral word.Equalize removes pair distinctions outside the bias subspace, which can erase useful semantic differences.
- Hard debiasing: After hard debiasing, every neutral word has equal inner products and distances to words within an equality set, and PairBias = 0.These guarantees follow because neutral words have zero projection onto the bias subspace and equality-set differences lie within it.
- Soft debiasing: Soften learns a linear transformation that minimizes neutral-word projections onto the gender subspace while preserving pairwise inner products, with λ = 0.2 in the experiment.Large λ recovers the hard-debiasing behavior; the optimization is a semidefinite program and outputs unit-normalized vectors.
7 Determining gender neutral words
The paper identifies gender-specific words from dictionary definitions and expands this set across the Google News embedding with a linear classifier. This procedure yields 6,449 gender-specific words, with an F-score of .627 ± .102 under stratified 10-fold cross-validation.
- Base vocabulary: Dictionary definitions produce a base set S0 of 218 gender-specific words from the 26,377-word w2vNEWS subset.Gender-neutral words are defined as the complement N = W \ S.
- Classifier-based expansion: A linear SVM with default regularization C = 1.0 extends the base list across the full 3 million-word Google News embedding, producing S with 6,449 gender-specific words.The final set is S = S0 ∪ S1, where S1 contains newly classified gender-specific words.
- Evaluation: F-score: .627 ± .102 under stratified 10-fold cross-validation evaluates the accuracy of the gender-specific-word classification process.Binary accuracy exceeds 99%, but the classes are imbalanced.
8 Debiasing results
Debiasing was evaluated for reducing direct and indirect gender bias while preserving embedding quality. Results show no degradation on standard coherence and analogy benchmarks, reduced direct stereotypes, and promising qualitative improvements in indirect bias.
- Evaluation objective: Debiasing algorithms were evaluated for reducing both direct and indirect gender biases while preserving desirable properties of the original embedding.The evaluation explicitly targeted both bias reduction and preservation of embedding utility.
- Direct Bias: 19% of the top 150 analogies in the initial w2vNEWS embedding were judged by a majority of ten workers to show gender stereotypes.The same analogy-generation task was applied to hard- and soft-debiased embeddings for comparison.
- Embedding quality: The debiasing transformation did not negatively impact performance on standard benchmarks measuring semantic coherence and analogy-solving ability.Table 1 compares the original and debiased w2vNEWS embeddings using RG, WS, and MSR-analogy, with higher scores better.
- Indirect bias: Strict debiasing produced promising qualitative improvements in indirect gender bias, illustrated by the softball–football example.Because no ground truth exists for indirect effects, the passage describes the improvement qualitatively rather than quantitatively.
9 Discussion
The discussion frames gender bias as a largely one-dimensional property of word embeddings that can be quantified and reduced by neutralizing gender associations. It also emphasizes residual biases, broader societal and application risks, and open extensions to other stereotypes and languages.
- Geometric account of bias: A single embedding direction largely captures gender and enables quantification of gender associations and indirect inequality for gender-neutral words.Projecting a gender-neutral word onto this direction indicates its degree of female or male bias.
- Debiasing and residual bias: Debiasing changes gender-neutral word embeddings by removing gender associations, such as moving nurse to be equally male and female along direction g.Gender-specific words can also exhibit biases beyond the main gender direction, including pair-specific associations such as babysit being closer to grandmother than grandfather.
- Societal implications: Because machine learning increasingly relies on word embeddings, reducing their bias may help avoid amplifying gender stereotypes and discrimination in computer systems.The discussion presents debiasing as a small possible contribution to reducing gender bias in society, while stressing that systems should not inadvertently amplify it.
- Application guidance: Despite potentially useful gender statistics in specific applications, the authors recommend erring toward neutrality and using debiased embeddings as much as possible.They identify the risk of algorithms amplifying gender stereotypes and discrimination as outweighing the case for routinely retaining the original biased embeddings.
- Limitations and extensions: The approach should be extended beyond gender because embeddings can reflect other stereotypes, including strong racial stereotypes, and its applicability to other languages remains an open question.The discussion specifically highlights languages with grammatical gender, where most noun definitions carry a gender marker.
A Generating analogies
The section compares objectives for generating analogy pairs, including the parallelogram approach, cosAdd, and 3CosMul. It then motivates an angle-based method and examines how varying δ changes the generated analogies.
- Analogy objectives: The parallelogram approach scores analogies by matching vector differences and is equivalent to maximizing cos(⃗y,⃗x+⃗b−⃗a), known as cosAdd.This equivalence assumes all vectors have unit length; the approach works well in some cases but has identified weaknesses.
- Analogy objectives: 3CosMul is designed to find a single y, but its objective is not directly applicable to generating both x and y because it is asymmetric.An additional ϵ ensures that the denominator is positive.
- Angle-based generation: The desired angle-based property is for a − b and x − y to have similar directions even when their magnitudes differ.However, maximizing cos(a − b, x − y) generally produces an extreme y; with a = he and b = she, her maximizes the expression for most x.
- Angle-based generation: As δ increases, generated analogies range from very similar x and y to loosely related pairs whose relationship is more creative.The parameter therefore controls the relationship tightness between the generated words.
- Comparison of generated analogies: The section compares the first 100 gender-neutral analogies from two approaches and displays the first 10 that are unique to each list.Figure 9 highlights differences between analogies generated by the proposed approach and the earlier minimizing approach.
B Learning the linear transform
The soft debiasing method formulates learning the transformation as a semidefinite program that balances preserving pairwise inner products against reducing gender-neutral words’ gender-subspace bias. An SVD-based reformulation reduces the computational burden to solving a 300 × 300 matrix problem, yielding the debiasing transformation.
- The soft debiasing algorithm solves an optimization problem equivalent to a semidefinite programming problem.
- The objective preserves pairwise inner products while making gender-neutral words’ biases in the gender subspace small, with λ balancing these terms.
- Direct SDP solution is impractical because W is about 300 × 400,000, making W^T XW and W^T W 400,000 × 400,000.
- Singular value decomposition W = UΣV^T enables reformulation in which ΣU^T(X − I)UΣ is a 300 × 300 matrix solved efficiently.
- The resulting solution T is the debiasing transformation of the word embedding.
C Details of gender specific words base set … G Analogies Generated by Word Embeddings
The appendices define gender-specific and stereotypical word sets through WordNet filtering and worker questionnaires, then collect and rate gender-stereotypical analogies while distinguishing them from appropriate analogies.
- C Details of gender specific words base set: The base gender-specific set was derived by checking WordNet definitions for gender terms across all 26,377 filtered embedding words.The method considered singular and plural forms of terms such as female, male, woman, man, girl, boy, sister, and brother.
- C Details of gender specific words base set: A strict policy classified doubtful cases as gender neutral, leaving 218 gender-specific words after filtering.Clothing terms such as vest were treated as gender neutral because people of every gender may wear them.
- E Questionnaire for generating gender stereotypical analogies: The analogy-generation questionnaire defines an analogy as two word pairs sharing the same relationship.It gives apple:fruit::asparagus:vegetable as an example.
- E Questionnaire for generating gender stereotypical analogies: Workers were asked to enter at least 10 analogies reflecting gender stereotypes to improve the analogy-generating system.The questionnaire illustrated stereotypes such as tall : man :: short : woman and doctor : man :: nurse : woman.
- F Questionnaire for rating stereotypical analogies: A separate rating task asked workers to identify which analogies were stereotypes, appropriate analogies, or nonsensical.Examples classified for evaluation included tall : man :: short : woman, doctor : man :: nurse : woman, and King: man :: Queen : woman.
- F Questionnaire for rating stereotypical analogies: The rating materials included appropriate gender analogies involving kinship, names, pronouns, occupations, and anatomical terms.Examples included brother : man :: sister : woman, John : man :: Mary : woman, His : man :: Hers : woman, and penis : man :: vagina : woman.
H Debiasing the full w2vNEWS embedding.
Hard debiasing was applied to the full w2vNEWS embedding, which contains 26,377 lower-case words in the cleaned version discussed in the main text. Standard evaluations indicate that debiasing preserves the embedding’s utility, including coherence and analogy-solving abilities.
- H Debiasing the full w2vNEWS embedding.: Hard debiasing was applied to the full w2vNEWS dataset in addition to the cleaned 26,377-word version.The cleaned version consisted of 26,377 lower-case words.
- H Debiasing the full w2vNEWS embedding.: Standard metrics show that debiasing does not degrade w2vNEWS performance on coherence and analogy-solving abilities.The reported metrics are RG, WS, and MSR-analogy, with higher scores indicating better performance.