Source-linked AI summary
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Svetlana Kiritchenko, Saif M. Mohammad
TL;DR
Machine-learning systems can perpetuate inappropriate human biases, while prior evaluations covered few systems and lacked a benchmark dataset. The paper introduces the 8,640-sentence Equity Evaluation Corpus and uses it to assess 219 sentiment-analysis systems. More than 75% show higher intensity scores for one gender or race, with race biases more prevalent and task-dependent; the corpus supports measurement but is not a complete solution.
Problem
Prior bias evaluations largely focused on one or two systems, and no benchmark dataset existed for examining inappropriate bias in natural language systems.
Method
The study constructs the 8,640-sentence Equity Evaluation Corpus and compares predictions from 219 shared-task systems on sentence pairs differing only in race- or gender-related terms.
Results
More than 75% of systems assign higher intensity to one gender or race; race biases are more prevalent, vary by affect dimension, and average score differences are below 0.03 but reach 0.34 for some systems.
Takeaways & Limitations
The EEC provides a freely available way for developers and users to measure race and gender bias in sentiment-analysis systems.
Takeaways & Limitations
The analysis cannot determine which datasets, language resources, or learning methods cause the observed biases, and mitigation approaches are outside its scope.
Abstract
from arXiv · showhide
Automatic machine learning systems can inadvertently accentuate and perpetuate inappropriate human biases. Past work on examining inappropriate biases has largely focused on just individual systems. Further, there is no benchmark dataset for examining inappropriate biases in systems. Here for the first time, we present the Equity Evaluation Corpus (EEC), which consists of 8,640 English sentences carefully chosen to tease out biases towards certain races and genders. We use the dataset to examine 219 automatic sentiment analysis systems that took part in a recent shared task, SemEval-2018 Task 1 'Affect in Tweets'. We find that several of the systems show statistically significant bias; that is, they consistently provide slightly higher sentiment intensity predictions for one race or one gender. We make the EEC freely available.
1 Introduction
The paper addresses inappropriate bias in sentiment and emotion systems by introducing a benchmark corpus and evaluating many shared-task systems. It finds systematic race and gender differences in predicted intensity and emphasizes that bias measurement remains necessary despite the lack of simple remedies.
- Past bias studies largely examined only one or two systems, and no benchmark dataset existed for evaluating inappropriate bias in natural language systems.
- The Equity Evaluation Corpus contains 8,640 English sentences designed to expose biases associated with race and gender.
- The study evaluates 219 sentiment analysis systems from SemEval-2018 Task 1 using sentence pairs differing only in a race- or gender-related word.
- The majority of examined systems show statistically significant bias, consistently assigning slightly higher sentiment intensity to one race or gender.
- Bias may vary with the affect dimension being predicted, so a system’s race or gender bias is not necessarily consistent across tasks.
- The EEC is intended as one fairness-measurement approach rather than a complete solution to inappropriate machine-learning bias.
2 Related Work
Related work shows that human biases can enter predictive systems through historical data and language resources. In NLP, prior sentiment-bias research focused mainly on word embeddings, while this paper distinguishes its scope from accuracy disparities and mitigation methods.
- Predictive models trained on historical data may inherit biases involving gender, ethnicity, race, or religion.
- NLP research has primarily examined bias in word embeddings derived from large collections of human-written text.
- Prior sentiment-analysis bias work examined simple systems using pre-computed word embeddings rather than systematically quantifying bias across many systems.
- This paper does not study accuracy differences across demographic groups or approaches for mitigating inappropriate biases.
3 The Equity Evaluation Corpus
The Equity Evaluation Corpus is an 8,640-sentence benchmark designed to test gender and race bias in natural language systems. It uses controlled templates and person and emotion-word substitutions to create comparable sentence pairs.
- Template design: The corpus uses eleven short, grammatically simple templates, with seven including emotion words and four omitting them.The authors developed the templates through discussion and consensus building.
- Person substitutions: Each template includes a <person> variable instantiated with gender-associated noun phrases and African American or European American names.The study selected ten names of each kind from prior work and also used paired terms such as “my daughter” and “my son.”
- Emotion substitutions: The <emotion word> variable covers anger, fear, joy, and sadness through five words per emotion selected at varying intensities.The words were drawn from four corresponding Roget’s Thesaurus categories.
- Corpus purpose: 8,640 English sentences form the Equity Evaluation Corpus, designed to determine whether systems assign consistently different intensity scores to sentences involving particular races or genders.The corpus can also be used to examine bias in NLP systems beyond the emotion-intensity task that motivated some design choices.
- Corpus construction: 8,640 sentences were generated from the combinations of person and emotion-word values across the eleven templates, then manually checked for grammatical well-formedness.The construction permits sentence pairs differing in only one gender- or race-associated word.
4 Measuring Race and Gender Bias in Automatic Sentiment Analysis Systems
The study evaluates bias by comparing system predictions on controlled EEC sentence pairs across outputs from a shared task. It tests 219 submissions using paired score comparisons and multiple-comparison correction.
- Evaluation setup: 219 automatic-system submissions from SemEval-2018 Task 1 were evaluated on the EEC, alongside the task’s five affect-prediction subtasks.The subtasks covered emotion intensity regression, emotion intensity ordinal classification, valence regression, valence ordinal classification, and emotion classification.
- Evaluation setup: Participants trained systems on tweet data with gold intensity scores and ran the same final systems on both the regular tweet test set and the unlabeled EEC test set.The EEC test set’s nature was disclosed only after the competition.
- Bias measurement: Gender bias was measured by comparing predicted scores for corresponding female and male noun-phrase sentences, including averages over female and male name subsets.The comparisons preserve the template while changing the gender-associated person expression.
- Bias measurement: The analysis compared 1,584 gender score pairs formed from eleven pairings across 20 emotion words in seven templates and four templates without emotion words.The calculation was 11 × (20 × 7 + 4) = 1,584.
- Statistical testing: For each system, paired two-sample t-tests assessed mean differences, with significance set at 0.05/438 after Bonferroni correction.The correction reflects 438 assessments across 219 submissions evaluated for both gender and race bias.
5 Results
Across sentiment and emotion tasks, many systems showed statistically significant gender or race-associated score differences, with bias patterns varying by affect dimension. Gender-associated words could also affect neutral sentences, while the analyses leave the causes of these biases unresolved.
- Gender Bias Results: 75% to 86% of submissions consistently marked sentences of one gender higher than another across the emotion and valence tasks.Only about 12 of 46 emotion-task submissions and 5 of 36 valence submissions showed no statistically significant gender difference.
- Gender Bias Results: 21–25 systems favored female-associated sentences on anger, joy, or valence, whereas 23 favored male-associated sentences on fear.Sadness showed a near balance: 18 systems favored female-associated sentences and 16 favored male-associated sentences.
- Gender Bias Results: Gender score differences were usually ≤0.03, but the maximal difference reached 0.34 and system-level ∆–spreads reached 0.57.These differences resulted from changing just one word, and larger effects may occur when complex sentences contain several gender-associated words.
- Gender Bias Results: Neutral-sentence analyses showed similar trends, while gender-associated words could have a bigger impact on predictions for neutral sentences.Emotion-matched subsets produced results similar to the full set.
- Race Bias Results: Only about 5–11 submissions for emotion tasks and 3 for valence showed no statistically significant race-associated score difference.These correspond to about 11%–24% of emotion submissions and about 8% of valence submissions, lower than the gender results.
- Race Bias Results: Most systems favored African American names on anger, fear, and sadness, but favored European American names on joy and valence.Race-pair ∆–spreads ranged from 0 to 0.15; medium- to low-performing systems showed greater sensitivity on valence.
6 Discussion
The analysis finds bias in systems using varied algorithms and language resources, while the unigram baseline indicates that training data itself can contain demographic associations.
- Bias appeared in systems trained with both traditional and deep neural algorithms and with varied language resources.The analysis cannot isolate bias to one component because datasets, resources, and learning methods may each contribute.
- The SVM-Unigrams baseline showed small gender and race bias despite using no language resources beyond its training data.Its Δ-spreads were 0.09 to 0.2 for gender pairs and less than 0.002 for race pairs.
- For gender pairs, male-associated sentences received higher scores for anger, fear, and sadness, but the tendency reversed for valence.
- For race pairs, the baseline favored European American names across emotion-intensity tasks but favored African American names for valence.
- The results indicate that distant-supervision training data can be a source of bias, while learning methods and language resources may accentuate, reverse, or mask it.
7 Conclusions and Future Work
The paper introduces the EEC and applies it to 219 shared-task systems to measure demographic bias. Bias was widespread, generally small on average but sometimes much larger, and varied across race, gender, and affect dimension.
- The EEC contains 8,640 sentences designed to expose gender and race biases in NLP systems, and it was used to analyze 219 shared-task systems.
- Race bias was more prevalent than gender bias, and the direction of bias differed across affect dimensions.
- Average score differences across genders and races were below 0.03, but some systems reached differences as high as 0.34.The paper notes that the downstream impact of consistent bias, even below 3% on average, requires further investigation.
- Future work will extend the EEC to additional demographic, occupational, geographic, and educational variables and examine prediction quality across demographic groups.
- The EEC and its methodology are not comprehensive, motivating broader examination using multiple approaches and datasets.
Appendix
The appendix visualizes demographic score differences across systems and emotion-intensity tasks, ordering systems by their official shared-task performance rank.
- Figures 3 and 4 plot score differences for each system on gender and race sentence pairs across four emotion-intensity regression tasks.
- Each plotted point represents the predicted-score difference for one sentence pair.
- The systems are ordered from first to last by their official performance rank on the relevant tweets test set.
- Figure 3 uses analogous gender groups: significant F↑–M↓, significant F↓–M↑, and nonsignificant F=M.