PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Bias Metrics Do Not Correlate with Application Bias

Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sanchez, Mugdha Pandya, Adam Lopez

arXiv:2012.15859v5cs.CL

TL;DR

NLP bias research lacks evidence that intrinsic embedding metrics predict bias in downstream applications. This paper compares the metrics across varied tasks, languages, embeddings, and interventions, finding no reliable relationship in most settings and recommending direct application-level evaluation.

  • Problem

    Bias research lacks consistent metrics that meaningfully detect bias in applications, while embedding debiasing relies on an unconfirmed link between intrinsic and downstream bias.

  • Method

    The study compares intrinsic and extrinsic bias across varied embedding, task, language, and bias-modification conditions.

  • Results

    Changes in WEAT do not reliably correlate with changes in application bias across tasks, languages, or most experimental conditions.

  • Takeaways & Limitations

    Researchers should not use WEAT to measure debiasing progress and should instead evaluate bias in downstream applications with suitable datasets and challenge sets.

  • Takeaways & Limitations

    All downstream tasks are discriminative classification tasks, so the findings do not establish whether generative models show the same relationship.

Abstract

from arXiv · show

Natural Language Processing (NLP) systems learn harmful societal biases that cause them to amplify inequality as they are deployed in more and more situations. To guide efforts at debiasing these systems, the NLP community relies on a variety of metrics that quantify bias in models. Some of these metrics are intrinsic, measuring bias in word embedding spaces, and some are extrinsic, measuring bias in downstream tasks that the word embeddings enable. Do these intrinsic and extrinsic metrics correlate with each other? We compare intrinsic and extrinsic metrics across hundreds of trained models covering different tasks and experimental conditions. Our results show no reliable correlation between these metrics that holds in all scenarios across tasks and languages. We urge researchers working on debiasing to focus on extrinsic measures of bias, and to make using these measures more feasible via creation of new challenge sets and annotated test data. To aid this effort, we release code, a new intrinsic metric, and an annotated test set focused on gender bias in hate speech.

1 Introduction

NLP bias research commonly uses intrinsic embedding metrics despite limited evidence that they reflect bias in downstream applications. This study tests that assumption across languages, embedding algorithms, and tasks, finding mostly absent or negative correlations.

  • Motivation: Bias research lacks consistent metrics that meaningfully detect bias in applications.Existing work often favors metrics that are easy to measure rather than metrics tied to application harms.
  • Research gap: Intrinsic embedding metrics are widely used, but their relationship to downstream application bias remains unconfirmed.The paper identifies this relationship as a central assumption in embedding debiasing research.
  • Why it matters: Untested metric assumptions risk misleading fairness claims, misdirected research effort, and false confidence in progress.These risks arise because researchers may not understand the limitations of their evaluation metrics.
  • Study scope: The study compares intrinsic and extrinsic bias across English and Spanish, word2vec and fastText, and coreference and hate-speech detection.This design varies languages, embedding algorithms, and downstream tasks.
  • Main finding: Most conditions show no correlation or negative correlation, despite moderately high correlations in a handful of conditions.The authors therefore recommend testing bias directly in specific applications rather than relying on intrinsic metrics.
  • Contributions: The paper releases new Spanish WEAT metrics and a gender-annotated English hate-speech test set.Both resources were created during the research.

2 Bias Metrics

The paper distinguishes intrinsic metrics, which measure geometric associations in word embeddings, from extrinsic metrics, which measure performance disparities in applications. WEAT is the main intrinsic metric studied, while Predictive Parity and Equality of Opportunity quantify downstream disparities.

  • Intrinsic bias metrics: Intrinsic metrics measure geometric relationships among curated wordlists representing concepts such as gender, career, math, and family.Each concept is represented by words in the embedding space.
  • Intrinsic bias metrics: WEAT measures the difference in mean cosine similarity between two target concepts and two attribute concepts.The difference represents imbalance in associations and is normalized into an effect size for the experiments.
  • Intrinsic bias metrics: WEAT originated as an embedding analogue of human Implicit Association Tests and was later repurposed as a predictor of embedding bias.It has also been extended across languages, sentences, and contextual language models.
  • Intrinsic bias metrics: WEAT is sensitive to corpus word frequency and to the selected target and attribute wordlists.The paper notes these known issues when motivating careful metric selection.
  • Extrinsic bias metrics: Extrinsic metrics quantify application bias through performance disparities between groups.Examples include unequal error rates in speech recognition and unequal false-negative rates in hiring classification.
  • Extrinsic bias metrics: Predictive Parity measures differences in precision, whereas Equality of Opportunity measures differences in recall between groups.The paper examines both because harmful errors vary by application.
  • Extrinsic bias metrics: Hate-speech detection can produce harmful false positives that censor content or false negatives that permit abuse against targeted populations.The paper therefore measures performance gaps in both precision and recall.

3 Methodology

The methodology varies embedding, task, language, and bias-modification conditions while tracking intrinsic and downstream bias together. Correlations are computed across embeddings whose bias is deliberately decreased or increased.

  • Correlation design: Each experiment measures a specific WEAT metric against a specific extrinsic metric for a downstream task using the embedding.The study evaluates both the original and modified embeddings.
  • Correlation design: Pearson correlations and scatterplots quantify the relationship between intrinsic and extrinsic metrics.The analysis is performed after reaching a predefined stopping condition.
  • Experimental variation: The experiment grid varies two embedding algorithms, two tasks, and two languages rather than relying on one experiment.The paper uses this variation to seek more robust conclusions across experimental conditions.
  • Correlation design: Bias is adjusted upward or downward, then WEAT and downstream bias are remeasured across repeated intervention steps.The resulting paired measurements provide multiple data points per experiment.
  • Bias modification: The study uses preprocessing and postprocessing methods capable of both debiasing and overbiasing embeddings.Preprocessing changes training data, while postprocessing changes an already trained embedding space.
  • Bias modification: Preprocessing balances training data, while Attract-Repel modifies relationships among words in an existing embedding space.The selected methods are intended to represent practical settings for retrained and pretrained embeddings.
  • Models and tasks: The embeddings include fastText and Skip-gram word2vec, with fastText additionally representing words through character n-grams.The experiments use simpler contextless embeddings rather than contextual embeddings.
  • Models and tasks: Tasks include English coreference resolution and English and Spanish hate-speech detection using standard task architectures.Spanish is used only for hate-speech detection because grammatical gender would make the coreference challenge evaluation trivial.

4 Experiments

The experiments use task-matched data, bias-specific wordlists, and modified WEAT resources to evaluate gender and migrant bias across English and Spanish settings. Spanish WEAT is substantially redesigned to address translation and grammatical-gender problems.

  • Datasets: Embeddings are trained on Wikipedia for coreference and on English or Spanish tweets for hate-speech detection.The training data are matched to the downstream task.
  • Datasets: English coreference uses OntoNotes for training and Winobias for evaluating gender bias in coreference resolution.Winobias is a challenge set designed to measure gender bias.
  • Datasets: English hate-speech detection uses abusive tweets and a hand-labelled 10,000-tweet test set.The labels identify targeted male, targeted female, and neutral content.
  • Wordlists: WEAT and bias-modification wordlists are matched by bias type, including gender and migrant bias.The modification lists are expanded from WEAT terms using nearest neighbors and then used for balancing and Attract-Repel.
  • Wordlists: English gender experiments use WEAT tests 6, 7, and 8, with a small substitution of general gender words for proper names.The substitution reflects the vocabulary used in the coreference task.
  • New Spanish WEAT: Spanish XWEAT is substantially modified because literal translations can be culturally inappropriate or uncommon.Examples include names with unintended class connotations and literal weapon terminology absent from the vocabulary.
  • New Spanish WEAT: Spanish grammatical gender can confound associations because many math, science, and art nouns share feminine grammatical gender.The paper identifies this as a problem for translated attribute wordlists.
  • New Spanish WEAT: The study introduces a Spanish WEAT test for migrant bias using migrant-associated versus non-migrant-associated names and pleasant versus unpleasant terms.The names draw on prior work connecting perceived class status with migrant status.

5 Results

Across tasks and experimental settings, intrinsic and extrinsic bias metrics do not show a reliable positive relationship. Coreference shows the clearest but embedding-dependent association, while hate-speech results are mostly absent, negative, or restricted to narrow conditions.

  • Overall results: None of the experiments shows a correlation trend that holds across all experimental details.Figure 3 plots intrinsic bias on the y-axis against extrinsic bias on the x-axis for individual runs, varying tasks, embeddings, metrics, and bias modification methods.
  • Coreference: Coreference has the clearest relationship, with significant moderate positive correlations for precision and recall using word2vec.FastText shows no significant correlation under any conditions, weakening the overall pattern.
  • English hate speech: English hate-speech detection shows no aggregate relationship, with correlations emerging only after separating preprocessing from postprocessing.Recall is moderately positive for postprocessing but moderately negative for preprocessing, while precision shows no correlation.
  • English hate speech: English hate-speech baselines differ by embedding type for precision, while recall shows six fewer points for male-targeted hate speech across both algorithms.FastText has 10 additional precision points for male-targeted versus female-targeted hate speech, whereas word2vec has 4 fewer points.
  • Spanish hate speech: Spanish hate-speech gender and migrant bias show no positive correlations in any experimental conditions.Migrant bias has a positive recall correlation only with fastText preprocessing, while precision trends in word2vec are negative.
  • Spanish hate speech: Spanish results form central clusters because bias measures were harder to spread evenly and fewer data points were available.The passage attributes this difficulty partly to limited WEAT tests, richer grammatical agreement, and less suitable language-model wordlists.

6 Discussion

WEAT changes do not reliably track downstream application bias, even when bias modification directly targets embeddings. This leaves researchers choosing between task-specific validation and direct application testing, while motivating better annotated downstream measures.

  • 6 Discussion: WEAT changes do not reliably correlate with changes in downstream application bias across tasks, languages, or most within-task comparisons.The study used relatively simple fullword embeddings and isolated one bias type at a time; greater complexity may make relationships less clear.
  • 6 Discussion: Researchers must either identify configurations predictive for a specific task, language, and model or test application bias directly.Direct testing may be no more onerous than exhaustively searching for predictive intrinsic configurations.
  • 6 Discussion: More datasets need subgroup demographic and identity annotations, and more challenge sets are needed to measure application bias.These downstream measures are required by either practical strategy for investigating or mitigating bias.
  • 6 Discussion: The efficacy of debiasing efforts can be assessed more fully only when suitable downstream measures are readily available.
  • 6 Discussion: All downstream tasks are discriminative classification tasks, limiting coverage of representational harms that generative models could capture.Classification primarily covers allocational harms such as performance disparity, whereas generation may expose misleading or harmful portrayals.

7 Conclusion

Across multiple tasks and languages, WEAT showed positive correlations with extrinsic bias metrics only in very restricted settings. The authors retain WEAT for descriptive analysis but advise against using it to measure debiasing progress.

  • 7 Conclusion: Positive correlations between WEAT and Equality of Opportunity or Predictive Parity occur only in very restricted settings.Many cases show either negative correlation or no correlation.
  • 7 Conclusion: WEAT remains useful for descriptive computational social science and examining bias in human texts, but not for measuring debiasing progress.

A Bias Metric Definitions & Formulas

This appendix defines the performance-gap metrics used for application bias and documents the wordlists and modifications used in English and Spanish WEAT tests. It also records comparability concerns arising from different gender-term choices.

  • A Bias Metric Definitions & Formulas: Performance Gap metrics quantify differences in model performance across demographic data splits, primarily for classification tasks.
  • A Bias Metric Definitions & Formulas: Equality of Opportunity requires equal true-positive rates across demographic groups, expressed using A, Y, and predicted label ˆY.
  • A Bias Metric Definitions & Formulas: Predictive Parity requires equal false-positive rates across groups; together with Equality of Opportunity, it defines Equalized Odds.
  • A Bias Metric Definitions & Formulas: The experiments operationalize these criteria as recall gaps and precision gaps, with positive privileged-minus-underprivileged differences indicating unfairness.
  • B.1 English WEAT lists: English WEAT tests all target gender bias, using gender, career, family, math, art, science, or related wordlists.WEAT 6 was adapted from proper names to general gender terms, while WEAT 7 and 8 retain their listed gender and attribute categories.
  • B.2 Changes to English List: The English WEAT 6 modification replaces proper-name gender terms with the gender terms used in WEAT 7 and 8.
  • B.2 Changes to English List: Using different gender-term types creates a comparability confound, especially because names are uncommon in Twitter and Winobias uses professions instead.
  • B.3 Spanish WEAT lists: Spanish WEAT lists balance grammatical gender and define gender, science, art, European-Spanish, migrant-Spanish, pleasant, and unpleasant word groups.

C Training Data and Preprocessing

The appendix points readers to the cited papers for training data used in the final embedding models. It does not provide those final-model training datasets here.

  • C Training Data and Preprocessing: Final-model training data are documented in the relevant papers cited in Section 4.1.

C.1 Wikipedia

The Wikipedia dataset contains 439,935,872 words after preprocessing and vocabulary filtering.

  • C.1 Wikipedia: 439,935,872 words remain after tokenization and replacing words occurring fewer than 10 times with <unk>.The data comes from the latest Wikipedia article dump.

C.2 Twitter

The study uses 2019 Twitter data from English and Spanish, with retweets removed and text normalized before training.

  • C.2 Twitter: 2019 Twitter data was downloaded from the Internet Archive.
  • C.2 Twitter: 3,641,306 English tweets and 10,683,846 Spanish tweets were lowercased, tokenized, normalized, and vocabulary-filtered.Hashtags and mentions were replaced with <HASH> and <MENTION>, and rare words were replaced with <unk>.

D Further Results Graphs

The further-results graphs examine how bias modification method, experimental conditions, and Winobias challenge type relate to intrinsic and extrinsic bias measures across tasks.

  • Bias modification breakouts: English hate-speech recall is positively correlated with WEAT only after post-processing, while precision gaps show greater spread when embeddings are modified.For Spanish, modifying bias appears more difficult with preprocessing than postprocessing.
  • Bias modification breakouts: Figure 4 separates each task’s gender-bias results by pre-processing versus post-processing for precision and recall.The breakout is intended to show how the modification method affects the observed relationships.
  • Full scatterplots: Figure 5 plots WEAT against group performance gaps for four English and Spanish coreference and hate-speech task conditions.Black points represent original embeddings before modification; the plots expose variation across experimental conditions.
  • Full scatterplots: None of the Figure 5 task plots shows a correlation that remains consistent across embedding type, bias-modification method, and WEAT test.
  • Winobias challenge types: Figure 6 breaks English coreference results down by Winobias challenge type, with Type 1 relying on semantic cues and Type 2 adding syntactic cues.The two challenge types differ in difficulty because Type 1 lacks syntactic cues.
Loading 2012.15859v5…