Source-linked AI summary
Understanding the Origins of Bias in Word Embeddings
Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, Richard Zemel
TL;DR
Existing work shows that word embeddings can reproduce stereotypical biases, but how those biases arise from particular training documents remains poorly understood. The paper uses influence functions to approximate the bias change caused by corpus perturbations, finding highly accurate predictions across Wikipedia and New York Times experiments. The method also exposes sensitivity to rare words as a limitation of the WEAT metric.
Problem
The origins of word-embedding bias in specific training documents are poorly understood, while direct document-by-document retraining is infeasible.
Method
Influence functions approximate how removing a small corpus subset changes the learned embedding and its measured bias.
Results
The method very accurately approximates the true bias changes from manually removing training documents and retraining embeddings across Wikipedia and New York Times experiments.
Takeaways & Limitations
The approach can trace bias to documents, support selective corpus pruning, and analyze how bias is distributed or evolves across training data.
Takeaways & Limitations
Rare-word representations are more sensitive to corpus perturbations, leaving WEAT vulnerable to manipulation through selected rare words.
Abstract
from arXiv · showhide
The power of machine learning systems not only promises great technical progress, but risks societal harm. As a recent example, researchers have shown that popular word embedding algorithms exhibit stereotypical biases, such as gender bias. The widespread use of these algorithms in machine learning systems, from automated translation services to curriculum vitae scanners, can amplify stereotypes in important contexts. Although methods have been developed to measure these biases and alter word embeddings to mitigate their biased representations, there is a lack of understanding in how word embedding bias depends on the training data. In this work, we develop a technique for understanding the origins of bias in word embeddings. Given a word embedding trained on a corpus, our method identifies how perturbing the corpus will affect the bias of the resulting embedding. This can be used to trace the origins of word embedding bias back to the original training documents. Using our method, one can investigate trends in the bias of the underlying corpus and identify subsets of documents whose removal would most reduce bias. We demonstrate our techniques on both a New York Times and Wikipedia corpus and find that our influence function-based approximations are very accurate.
1. Introduction
Word embeddings can acquire and amplify stereotypical biases from training text, yet the documents responsible for those biases are difficult to identify efficiently. The paper introduces an influence-function method to trace bias to corpus documents and validates its accuracy on Wikipedia and New York Times data.
- Bias in machine-learning systems can reflect and magnify human biases in data used for training and deployment.
- Word embeddings trained on text can associate male terms with science and female terms with art, risking stereotype perpetuation in translation and hiring systems.
- Attributing embedding bias to individual documents is desirable but naïve leave-one-document-out retraining is computationally prohibitive.
- The proposed method approximates how removing a small corpus subset changes embedding bias by separating data effects on embeddings from embedding effects on bias.
- Experiments on Wikipedia and New York Times corpora found extremely accurate predictions of how corpus perturbations affect embedding bias and improved identification of bias-inducing documents.
2. Related Work
Prior work established that word embeddings preserve both useful linguistic structure and problematic social associations, and developed tests and debiasing algorithms. This paper builds on that literature by using influence functions to investigate where such biases originate in training data.
- Word embeddings encode statistical, syntactic, and semantic relationships from training corpora, including both useful analogies and problematic relationships.
- WEAT showed that embeddings trained on large public corpora replicate human-measured gender associations, such as female terms being closer to family and arts than career and math terms.
- Debiasing algorithms were developed to remove problematic relationships while preserving unproblematic ones.
- Stereotypical and racial biases have also been documented in visual models, multilabel object-classification datasets, and other machine-learning applications.
- Influence functions approximate how perturbing training examples changes model parameters, enabling explanations that trace learned states back to individual examples.
3. Background
The paper’s background describes GloVe as optimizing word vectors from a sparse co-occurrence matrix and introduces influence functions for approximating parameter changes after data perturbations. It also defines WEAT as a cosine-similarity-based bias measure over target and attribute word sets.
- The GloVe word embedding algorithm: GloVe first extracts a weighted word-context co-occurrence matrix, then optimizes embedding and bias parameters to minimize its loss.
- The GloVe word embedding algorithm: The learned word representation has dimension D, while context vectors and separate bias terms are also optimized in GloVe.
- Influence Functions: Influence functions approximate how optimal model parameters change when a small subset of training points is perturbed.
- Influence Functions: The multiple-perturbation formulation assumes the perturbed subset is much smaller than the full training set, |δ| ≪ n.
- The Word Embedding Association Test: WEAT compares two target-word sets and two attribute-word sets using cosine similarities, then summarizes differential associations with an effect size.
4. Methodology
The method formalizes document-level bias contributions and efficiently approximates how corpus perturbations change word embeddings and their bias. It exploits GloVe’s structure to avoid retraining for every document while identifying bias-inducing corpus components.
- Formalizing the Problem: The authors define differential bias as the change in a bias metric when a corpus part is removed from training.The perturbed embedding is trained on the corpus with part p excluded.
- Efficient Algorithm: The algorithm subtracts each document’s co-occurrences and computes changes only for WEAT words appearing in that document.This combines the differential-bias approximation with the sparsity of the WEAT metric.
- Formalizing the Problem: The bias gradient links changes in the co-occurrence matrix to changes in embedding bias and indicates which co-occurrences most affect bias.For metrics such as WEAT, the gradient can be sparse because only a small vocabulary subset affects the metric.
- Computing the Differential Bias for GloVe: Influence functions approximate each document’s perturbed embedding and differential bias without directly removing the document and retraining the model.The naive approach is computationally infeasible when differential bias must be computed for every document.
- Computing the Differential Bias for GloVe: A simplifying assumption treats GloVe’s context vectors and bias parameters as constant during the approximation, while the original embedding still trains all parameters dynamically.Only the derivation of the influence-function approximation fixes u, b, and c.
- Computing the Differential Bias for GloVe: The resulting Hessian is block diagonal, allowing word-vector perturbations to be solved independently and reducing computation to affected word vectors.For document-level perturbations, only WEAT word vectors whose co-occurrences change need to be computed.
5. Experimentation
Experiments evaluate approximation accuracy, baseline comparisons, transfer across embedding methods and metrics, and the semantic and frequency-related properties of influential documents. The method closely tracks retraining-based ground truth, identifies stronger bias-influencing documents than PPMI, transfers to word2vec and other metrics, and reveals sensitivity to rare words.
- Accuracy: The method’s predictions correlated extremely strongly with retraining-based ground truth in every configuration (r2 ≥0.985).Ground truth used several retraining runs with different random seeds.
- Accuracy: Targeted perturbation sets generally changed bias significantly, unlike random sets, which rarely differed from baseline.Only 2 of 36 random sets differed significantly, compared with 38 of 40 targeted sets.
- Comparison to a PPMI Baseline: PPMI-based document selection was weaker: removing its 10 most bias-increasing Wiki documents reduced WEAT1 by 4%, versus 40% for the method’s selections.PPMI changes can be computed rapidly, enabling corpus-wide scans, but the selected documents had smaller effects.
- Impact on Word2Vec and Other Bias Metrics: Removing the 10k documents identified for GloVe changed bias in PPMI and word2vec; word2vec’s WEAT effect size fell from 1.35 to 0.11 without significant TOP-1 analogy change.The response in PPMI and word2vec was weaker than in GloVe, but remained clear.
- Qualitative Analysis: The same perturbations produced strong changes in gender-axis projections, while influential documents often had interpretable semantic content or affected bias through secondary co-occurrences.Some influential documents contained synonyms rather than explicit WEAT terms, indicating that direct WEAT-word co-occurrences would miss them.
- Qualitative Analysis: Rare words were more sensitive to corpus perturbations, with log frequency correlating with relative-position effects at r2 = 0.828.This sensitivity allowed removing 0.07% of NYT articles to reverse the WEAT effect size.
6. Conclusion
The paper formalizes tracing bias origins in word embeddings and validates an influence-function methodology for estimating how training-document removal changes bias.
- The authors define differential bias as the change in embedding bias caused by removing a document or small corpus subset.
- They develop an efficient influence-function approximation for differential bias in GloVe embeddings.
- The method closely approximates the true bias change measured after manually removing documents and retraining embeddings.
- Experiments on Simple Wikipedia and New York Times corpora cover two WEAT bias metrics.
- The methodology could measure how text-corpus bias evolves over time and could generalize to other perturbation-sensitive learning settings.
A. Computing the Bias Gradient for GloVe
The bias gradient uses the chain rule to connect corpus co-occurrence perturbations to embedding-bias changes, with influence functions approximating the embedding response in GloVe.
- The bias gradient is a V × V matrix describing corpus perturbation directions that produce maximal bias change.
- The chain rule decomposes bias sensitivity into the bias gradient with respect to word vectors and the word vectors’ sensitivity to corpus perturbations.
- For WEAT, the calculation simplifies because the bias metric depends only on a small subset U of vocabulary words.
- Influence functions approximate how GloVe word vectors change under an almost arbitrary co-occurrence perturbation Y, including document removal.
- The required higher-order Jacobian is sparse because each perturbed word-vector loss depends only on its corresponding co-occurrence row.
- The bias gradient is defined only for non-zero co-occurrences, but this does not restrict analyses of corpus removals.
B. Experimental Setup
The experiments use WEAT tests over word groups representing gender–science/arts and instrument–weapon associations, with the setups summarized in Table 3.
- The experimental setups summarize the corpora and embedding hyperparameters used in the study.
- The complete word lists for both WEAT tests are provided below the experimental setup summary.
- The first WEAT uses science and arts as target groups and male and female terms as attribute groups.
- The second WEAT uses instruments and weapons as target groups and pleasant and unpleasant terms as attribute groups.
C. Detailed Experimental Methodology
The evaluation estimates document-level bias effects, builds targeted and random removal sets, approximates set-level effects, and compares predictions with retrained-embedding ground truth.
- The methodology begins by training 10 baseline embeddings with identical parameters but different random seeds.
- The method approximates each document’s differential bias for every WEAT test using the main-text equations and multiple baseline embeddings.
- Perturbation sets include increase, random, and decrease groups, with targeted sets selected from documents predicted to have extreme differential bias.
- Set-level differential bias is approximated directly because GloVe loss is not linear in the co-occurrence matrix.
- Ground truth comes from removing each target set and training five new embeddings with the same hyperparameters and varied random seeds.
D. Additional experimental results
Additional experiments compare approximated and ground-truth bias effects across Wiki and NYT embeddings, using WEAT1 and WEAT2. They also examine agreement with PPMI and compare the method with a PPMI baseline.
- Approximation accuracy: The experiments evaluate approximated versus ground-truth WEAT bias effects after removing non-random perturbation sets in Wiki and NYT.Figure 7 plots means with one-standard-deviation error bars, least-squares fits, and baseline means; Figure 8 compares differential bias of removal.
- Baseline comparison: Table 5 compares documents selected by the PPMI baseline with documents selected by the proposed method using mean WEAT1 in 10 retrained Wiki GloVe embeddings.The supplied table caption specifies the Wiki setup and the 10-embedding mean.
- PPMI comparison: The comparable word2vec experiment reports r2 = 0.803 for the same randomly generated word-set comparison.The reported values are r2 = 0.725 for the NYT GloVe comparison and r2 = 0.803 for word2vec.
E. Influential Documents - NYT WEAT 1
The section identifies influential NYT documents for WEAT1 and presents the influence-function extension used to analyze multiple document perturbations. The listed documents include articles associated with both decreasing and increasing bias.
- Influential NYT documents: The 50 most WEAT1 bias-influencing NYT documents were identified, with publication dates ranging from January 1, 1987 to June 19, 2007.The supplied list contains article titles and notes that most can be found through the New York Times search service.
- Multiple perturbations: The influence-function derivation is extended from one perturbed training point to multiple perturbed points by replacing the single-point loss difference with a sum over perturbed indices.The extension relies on gradient linearity and assumes |δ| ≪ n.