PaperlayerView exact revision on arXiv

Source-linked AI summary

On Measuring and Mitigating Biased Inferences of Word Embeddings

Sunipa Dev, Tao Li, Jeff Phillips, Vivek Srikumar

arXiv:1908.09369v3cs.CLcs.LG

TL;DR

Word embeddings encode real-world biases, while existing intrinsic tests mismatch downstream use and do not suit contextualized representations. The paper probes bias through invalid NLI inferences and evaluates projection-based mitigation, finding effectiveness for static GloVe and for gender in selected ELMo and BERT components.

  • Problem

    Existing intrinsic bias tests measure vector distances rather than downstream inference behavior and are designed for word types, limiting their applicability to contextualized embeddings.

  • Method

    The paper constructs neutral sentence pairs and measures deviations from neutral NLI probabilities as evidence of bias, then applies projection-based debiasing to embedding subspaces.

  • Results

    The probe finds gender, religion, and nationality biases in GloVe, ELMo, and BERT; projection works for GloVe and attenuates gender bias in ELMo and BERT when applied to non-contextual components.

  • Takeaways & Limitations

    NLI-based probing provides a systematic way to measure bias in static and contextualized word representations, while mitigation effectiveness depends on the embedding component and bias type.

  • Takeaways & Limitations

    Residual bias may originate from either word embeddings or stereotypes and annotation artifacts in SNLI data, and removing the latter remains an open question.

Abstract

from arXiv · show

Word embeddings carry stereotypical connotations from the text they are trained on, which can lead to invalid inferences in downstream models that rely on them. We use this observation to design a mechanism for measuring stereotypes using the task of natural language inference. We demonstrate a reduction in invalid inferences via bias mitigation strategies on static word embeddings (GloVe). Further, we show that for gender bias, these techniques extend to contextualized embeddings when applied selectively only to the static components of contextualized embeddings (ELMo, BERT).

Introduction

Word embeddings encode real-world stereotypes, but conventional intrinsic bias tests do not match downstream use and often do not apply to contextualized representations. The paper therefore uses NLI-based invalid inferences to measure bias and tests projection-based mitigation across static and contextualized embeddings.

  • Motivation: Intrinsic vector-distance tests mismatch how embeddings function as downstream features and do not directly apply to contextualized token representations.ELMo and BERT produce context-dependent embeddings rather than one vector per word type.
  • NLI-based probing: A GloVe-based NLI model assigned 0.842 entailment probability to a neutral pair differing between “rude” and “Uzbekistani.”The paper treats such errors as potentially reflecting model error or representational bias and develops probes over millions of targeted pairs.
  • Motivation: An NLI model made invalid predictions when sentence pairs differed only by gendered words.For the driver/man pair, entailment received probability 0.497; for the driver/woman pair, contradiction received probability 0.654, although both pairs were neutral.
  • NLI-based probing: The paper defines aggregate NLI-based measures over many predictions and finds substantial gender, demonym, and religion bias in GloVe, ELMo, and BERT.It presents the work as among the first demonstrations of national or religious bias in word embeddings.
  • Bias mitigation: Projection-based debiasing reduces gender effects on occupations and similarly removes subspaces associated with religions and demonyms in static GloVe embeddings.The approach extends projection-based mitigation beyond intrinsic measures to NLI-based probes.
  • Bias mitigation: For ELMo and BERT, removing gender information from non-contextual components reduces NLI-measured gender bias, whereas removing a direction from full representations fails.The approach is ineffective or inapplicable for religion and nationality.

Measuring Bias with Inference

The paper measures embedding bias through systematically invalid NLI inferences on sentence pairs that should be neutral. It applies these probes across gender, nationality, and religion-related tests using GloVe, ELMo, and BERT-based models.

  • Probe construction: NLI treats entailment, contradiction, and neutrality as labels for evaluating whether embeddings support invalid inferences.The probes use sentence pairs whose bias-free relationship should be neutral, such as occupation–gender pairs.
  • Probe construction: Over 2,000,000 gender-occupation sentence pairs are generated from 164 occupations, 27 verbs, 184 objects, and three gendered word pairs.Only the subject changes between premise and hypothesis, while gendered occupation terms are excluded from the occupation list.
  • Bias measures: Bias is quantified by aggregate neutrality measures: average neutral probability, fraction labeled neutral, and fractions exceeding neutral-probability thresholds of 0.5 and 0.7.In the ideal bias-free case, all three measures equal 1.
  • Gender bias: Gender-occupation neutrality scores are far below the desired value of 1 across GloVe, ELMo, and BERT, demonstrating bias in static and contextualized embeddings.The paper also reports template fillers producing the largest non-neutral probabilities for GloVe.
  • Religion bias: Religion tests contain 1,133,730 pairs, show about 25% non-neutral predictions, and perform notably worse with the ELMo-based model.The tests compare religion-related terms with polarity words using the same aggregate measures.

Attenuating Bias in Static Embeddings

The paper applies projection-based debiasing to GloVe by removing subspaces associated with gender, nationality, and religion. Neutrality improves across these static-embedding probes, while standard SNLI accuracy changes little or slightly improves after gender projection.

  • Method: Projection removes a bias subspace from all word representations, avoiding residual information associated with hard debiasing.The method identifies dominant directions for gender, demonyms, or religions and projects them out post hoc.
  • Bias Subspaces: The gender subspace uses the he-she vector, while nationality and religion use principal components from eight demonym and adherent training items.The nationality subspace is one-dimensional; the religion subspace uses two principal components.
  • Gender: 25% to 160% increases in GloVe neutrality scores followed gender projection, versus -6% to +3.5% for random vectors.The comparison uses eight random projected vectors as a baseline.
  • Religion: Removing the adherent subspace similarly improved all religion neutrality measures to approximately the same level as nationality.The reported improvements are summarized in Table 9.

Embeddings

The paper tests projection-based debiasing in contextual ELMo and BERT representations. Applying projection only to static components reduces gender bias, but nationality and religion remain ineffective or technically difficult to debias.

  • Representation Structure: ELMo combines a static character-based first layer with two contextual layers, whereas BERT uses contextual layers over static subword input embeddings.ELMo has three 1024-dimensional layers; base BERT has twelve 768-dimensional layers.
  • ELMo Full Representation: Projecting a learned gender subspace from all ELMo layers does not significantly improve NLI neutrality over random-direction projection.The authors conclude that full-representation projection is ineffective for NLI-measured bias.
  • ELMo Gender: Projecting only ELMo layer 1, a context-independent word lookup, significantly attenuates gender bias before contextual layers generate the remaining representation.This selectively debiases the static component rather than the full contextual representation.
  • ELMo Gender: The probability of a neutral prediction exceeds 0.5 after layer-1 projection, while examples with neutral probability above 0.7 rise from 0.063 to 0.364.The latter is reported as a nearly 500% increase; random-direction changes stay within 3%.
  • ELMo Nationality & Religion: Layer-1 projection decreases neutrality for nationality and religion, so removing their information from ELMo’s first layer does not attenuate bias in the full model.The authors hypothesize that distortion or contextual layers reintroducing bias may explain this result.
  • BERT: BERT gender debiasing projects the he-she direction only from context-independent subword embeddings, because nationality and religion subspaces are harder to identify over subwords.The authors leave BERT nationality and religion debiasing for future work.
  • BERT: BERT projection at test time is ineffective, whereas applying the projection during training and testing reduces NLI-measured gender bias.Random-direction projections have negligible effects in both comparisons.

Discussions, Related works & Next Steps

The discussion compares debiasing across embedding types, relates NLI-based bias measurement to model error and coreference probes, and identifies residual bias as an open challenge. Projection is consistently effective for GloVe, while contextualized models retain stronger predictive performance but are harder to debias.

  • Debiasing across embeddings: Projection-based attenuation is consistently successful on GloVe, whereas ELMo debiasing is not universally successful.The paper reports that GloVe neutrality scores are higher on almost all tasks after debiasing, while ELMo's mechanisms are less reliable.
  • Debiasing across embeddings: ELMo retains slightly higher SNLI scores than GloVe before and after debiasing, suggesting better accuracy but greater debiasing difficulty.The reported effect of debiasing on ELMo's original prediction goal is fairly minor.
  • Debiasing across embeddings: BERT achieves gender dev and test scores of 90.70 and 90.23 while also attaining the highest neutrality scores.Nationality and religion debiasing were not evaluated for BERT because associated projection subspaces were unavailable.
  • Further resolutions: For Iraqi nationality templates involving rude, average entailment falls from 99.3 to 62.9 after projecting out the demonym subspace.This illustrates an intermediate-resolution bias measure rather than only aggregate template-level error.
  • Limitations and next steps: Residual non-neutrality may reflect either embeddings or stereotypes and artifacts in SNLI data, whose removal remains an open question.The authors conjecture that suitable loss functions might address bias originating in the NLI data.
  • Model error: The authors argue the observed non-neutrality exceeds ordinary model error because neutral predictions are far below roughly 40–50% despite test-set scores near 90%.Random-direction projection produces fairly insignificant effects, supporting the interpretation that the measured effects are real.
  • Related work and extensions: NLI-based probes complement coreference probes because incorrect coreference decisions also constitute invalid inferences, although coreference is itself difficult.The paper presents NLI as a potentially extensible mechanism for other socially consequential biases.

Conclusion

The conclusion frames biased representations as a source of biased inferences and applies NLI to measure them. It reports broad bias across static and contextualized embeddings, with projection-based attenuation effective for GloVe and for gender in contextualized models through selective first-layer debiasing.

  • Conclusion: The paper constructs an NLI-based probe from the observation that biased representations produce biased inferences.The probe is designed to measure biases in word representations systematically.
  • Conclusion: Experiments find that GloVe, ELMo, and BERT encode gender, religion, and nationality biases.The conclusion summarizes the paper's cross-embedding, cross-bias finding.
  • Conclusion: Projection-based attenuation works for static GloVe embeddings and, for the characterized gender direction, for ELMo and BERT after debiasing only their first non-contextual layer.The contextualized-embedding approach preserves entailment accuracy for the reported gender result.

Word Lists

The word lists provide template-generation and evaluation vocabularies spanning gender, polarity, objects, people, occupations, rulers, religions, nationalities, and related country names. Training and test lists are separated for subspace learning and template assessment, and the experiments use deterministic seeds with only small observed run-to-run variation.

  • List construction: Train and Test word lists do not intersect: Train populates templates, while Test learns embedding subspaces and full lists assess subspace stability.When no subscript is shown, the list denotes the union of both sets.
  • Demonym and religion lists: Religion lists contain separate training and test vocabularies, including terms such as atheist, Catholic, Hindu, Muslim, Sikh, Sunni, and Taoist.The passages identify the lists but do not report their empirical effects here.
  • Demonym and religion lists: Nationality lists pair demonyms such as American, Iraqi, Uzbekistani, and Zambian with corresponding country names such as America, Iraq, Uzbekistan, and Zambia.These vocabularies support nationality-template construction and evaluation.
  • Gender and polarity lists: The lists include gender terms, pronouns, names, kinship terms, and male- or female-associated lexical forms.Examples include man, woman, himself, herself, John, Mary, father, mother, his, and her.
  • Gender and polarity lists: Polarity vocabularies contain evaluative adjectives such as awful, dishonest, intelligent, nice, rude, smart, stupid, and weak.These words populate polarized adjective targets in the template-based probes.
  • Template vocabularies: Template objects draw from object nouns, rulers, and person hyponyms, while other lists provide relations, actions, and occupations.The inventories include everyday objects, social relations, verbs, professions, and political roles.
  • Reproducibility: Random-seed variation changes testing accuracies by approximately −0.3 to 0.3 points on a 100-point scale in preliminary experiments.The code is described as deterministic apart from randomness buried in learning libraries.
Loading 1908.09369v3…