Gender-preserving Debiasing for Pre-trained Word Embeddings

1 Introduction

• Word representations have been reported to encode racist, offensive, and discriminatory language, including human-like associations involving gender, profession, and ethnicity.Prior examples include gender bias amplified by embeddings trained on a 300GB news corpus and abusive language learned by Tay from Twitter.Source: §1 · p. 1
• Because pre-trained embeddings are used in downstream NLP tasks that interact with or affect people, the introduction argues they should be debiased before deployment.It cites legal, business, and ethical obligations to make NLP systems unbiased.Source: §1 · p. 1
• Debiasing must remove discriminatory information without discarding information useful for downstream tasks, such as gender associations used in recommendations.The introduction contrasts stereotypically gendered professions with associations such as beard–male and bikini–female.Source: §1 · pp. 1–2
• The method separately preserves feminine and masculine information, protects neutrality, and removes unfair gender bias from stereotypical words.It uses an encoder in a denoising autoencoder and trains the decoder to reconstruct original embeddings from debiased ones, balancing the four criteria end-to-end.Source: §1 · p. 2
• The proposed method outperforms hard-debiasing and GN-GloVe on SemBias stereotypical analogies while preserving semantic information and removing gender biases.It can also further debias embeddings already processed by GN-GloVe and is presented as a drop-in replacement for NLP applications.Source: §1 · p. 2

2 Related Work

• Hard- and soft-debiasing project embeddings away from a gender direction, but exclude words classified as gender-definitional from subsequent debiasing.Thus, a stereotypical word mistakenly classified as gender-definitional may remain biased.Source: §2 · p. 2
• GN-GloVe confines gender-related information to a sub-vector, but learns from scratch and does not debias its predefined feminine, masculine, and neutral word lists.It therefore cannot be applied directly to pre-trained embeddings.Source: §2 · pp. 2–3
• Adversarial methods jointly train a task predictor, a protected-attribute predictor, and an encoder, but chance-level development accuracy does not guarantee invariant representations.Elazar and Goldberg (2018) found that a post-hoc classifier could still predict protected attributes with substantially high accuracy.Source: §2 · p. 3
• Reported gender bias in coreference and machine translation includes biased coreference predictions and male or female translations when source-language gender is ambiguous.The examples indicate that gender bias has been studied across distinct NLP tasks.Source: §2 · p. 3

3.1 Formulation

• The method projects pretrained embeddings into a debiased lower-dimensional space without requiring access to their training algorithm, corpora, or lexicons.This decoupling supports embeddings produced with different algorithms and resources.Source: §3.1 · p. 3
• The method divides words into four mutually exclusive categories and preserves feminine and masculine information, preserves neutrality, and removes stereotypical gender bias.The categories are feminine, masculine, gender-neutral, and stereotypical.Source: §3.1 · pp. 3–4
• Gender-neutral and stereotypical words are pushed orthogonal to a gender direction computed from feminine–masculine pairs, with squared inner-product loss and epoch-wise re-estimation.The gender direction is held fixed during each epoch and re-estimated between epochs.Source: §3.1 · p. 4
• A reconstruction loss preserves embedding semantics, and the total objective is a weighted sum of the defined losses.The nonnegative loss weights sum to 1 and are selected by cross-validation in the experiments.Source: §3.1 · p. 4

3.2 Implementation and Training

• Feminine and masculine regressors use one-hidden-layer feed-forward networks, while the encoder and decoder use two-hidden-layer feed-forward networks.The regressors use sigmoid activation; the autoencoder uses hyperbolic tangent throughout.Source: §3.2 · p. 4
• Training uses SGD for the full objective, optional classifier and autoencoder pretraining, seed-word reuse, and one-sided undersampling for category imbalance.Classifier pretraining uses Adam; autoencoder pretraining uses 5,000 randomly selected embeddings, and dropout is used during training.Source: §3.2 · pp. 4–5

4.1 Training and Development Data

• The experiments use 223 feminine and 223 masculine words, 1,031 annotator-verified neutral words, and 166 stereotypically gender-associated professions.GloVe embeddings are trained on the January 2017 English Wikipedia dump; 20 words from each category form the development set, and reconstruction loss is an important requirement during debiasing.Source: §4.1 · p. 5

4.2 Baselines and Comparisons

• The comparisons include original GloVe, Hard-GloVe, GN-GloVe, autoencoded versions, and the proposed method applied to GloVe or GN-GloVe.AE (GloVe) serves as a non-debiasing autoencoding baseline; GP (GN-GloVe) tests further debiasing of embeddings already processed by another method.Source: §4.2 · p. 5

4.3 Evaluating Debiasing Performance

• GP (GN-GloVe) reports the best Definition and Stereotype accuracies in SemBias and SemBias-subset, significantly outperforming GloVe and Hard-GloVe on both datasets.Applying the method to GN-GloVe also removes residual bias; applying it to GloVe reduces bias while preserving useful gender-related information.Source: §4.3 · p. 6

4.4.1 Analogy Detection

• On Google, MSR, and SemEval analogy tasks, the proposed method causes no significant performance degradation.Analogy completion uses CosAdd, and the reported measure is the percentage of correctly solved questions.Source: §4.4.1 · p. 7, Table 2 · p. 7

4.4.2 Semantic Similarity Measurement

• Semantic-similarity evaluation measures Spearman correlation with human ratings, but existing benchmarks contain gender-stereotyped examples that can reward biased embeddings.The authors balance benchmarks by adding opposite-gender word pairs and reusing the original pair's rating as a proxy rather than re-annotating.Source: §4.4.2 · p. 7, Table 3 · p. 7
• GP (GloVe) and GP (GN-GloVe) perform best on balanced similarity benchmarks, while GP (GloVe) remains comparable to GloVe on original and balanced datasets.The results support preservation of embedding information while removing stereotypical gender bias.Source: §4.4.2 · pp. 8–9, Table 4 · p. 8

4.5 Visualising the Effect of Debiasing

• Figure 1 compares cosine similarity to the gender direction: GP (GloVe) reduces stereotypical-word bias while nearly preserving gender-oriented and neutral words, unlike Hard-GloVe and GN-GloVe.Hard-GloVe also shifts gender-oriented words toward the masculine direction.Source: Figure 1 · p. 8, §4.5 · p. 9

• The proposed method removes gender-specific biases from pre-trained word embeddings while preserving useful semantic information.Source: §5 · p. 9
• Experiments on multiple benchmark datasets show accurate debiasing and better performance than previously proposed debiasing methods.Source: §5 · p. 9
• Future work will extend the method to ethnic, age, or religious biases.Source: §5 · p. 9