Source-linked AI summary
Evaluating Gender Bias in Machine Translation
Gabriel Stanovsky, Noah A. Smith, Luke Zettlemoyer
TL;DR
Gender-biased machine translation can assign stereotypical genders despite contextual evidence, but broad evidence across languages and systems was limited. The paper builds WinoMT from two coreference datasets and evaluates translations using alignment and morphological analysis. All tested systems showed significant gender bias across eight target languages, and the authors release the data and code.
Problem
The paper addresses limited large-scale multilingual evidence on whether machine translation systems preserve contextual gender rather than relying on stereotypes.
Method
The authors concatenate Winogender and WinoBias into WinoMT and evaluate translations for eight target languages using alignment and morphological analysis.
Results
All four tested commercial systems and two academic models were significantly prone to gender-biased translations across all eight tested languages.
Takeaways & Limitations
The findings provide large-scale multilingual quantitative evidence that tested MT systems translate according to gender stereotypes rather than meaningful context.
Takeaways & Limitations
WinoMT uses synthetic English examples and serves only as a proxy for gender bias, so natural in-the-wild data and larger annotated corpora remain needed.
Abstract
from arXiv · showhide
We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT). Our approach uses two recent coreference resolution datasets composed of English sentences which cast participants into non-stereotypical gender roles (e.g., "The doctor asked the nurse to help her in the operation"). We devise an automatic gender bias evaluation method for eight target languages with grammatical gender, based on morphological analysis (e.g., the use of female inflection for the word "doctor"). Our analyses show that four popular industrial MT systems and two recent state-of-the-art academic MT models are significantly prone to gender-biased translation errors for all tested target languages. Our data and code are made publicly available.
1 Introduction
The paper introduces a multilingual evaluation of gender bias in machine translation, motivated by evidence that systems can assign stereotypical genders despite contextual cues. It uses coreference-based challenge data and morphological analysis to detect these errors across target languages.
- Learned models can encode social stereotypes from training data even when those correlations are irrelevant to the task.
- The study conducts the first large-scale multilingual evaluation of gender bias in machine translation, extending earlier small-scale observations of stereotypical translations.
- In the Spanish example, translating the female doctor as “el doctor” changes the source meaning by assigning the doctor a stereotypically male gender.
- Winogender and WinoBias contain English scenarios with role-based entities and pronouns whose coreference requires contextual resolution.
- Faithful translation into grammatically gendered languages may require implicit gender identification, recoverable through morphological markers.
- The proposed challenge-set evaluation covers eight target languages and uses alignment plus morphological analysis without requiring additional gold translations.
2 Challenge Set for Gender Bias in MT
The authors construct WinoMT by combining two coreference test sets and evaluate translations through alignment and language-specific morphological gender extraction. The procedure produces entity-gender predictions that can be compared with the original English annotations, while introducing possible noise from alignment and analysis errors.
- WinoMT concatenates Winogender and WinoBias into a balanced challenge set of 3,888 instances.The corpus is balanced across male and female genders and stereotypical and non-stereotypical role assignments.
- The evaluation translates every WinoMT sentence into a target language and aligns the resulting bilingual corpus with the English source.
- English entities are mapped to their target-language translations, whose genders are extracted using language-specific morphological-analysis tools and heuristics.
- The extracted translated genders are evaluated against gold entity-gender annotations from the original English datasets.
- Evaluation noise can arise from incorrect alignments or erroneous morphological analysis, with human evaluation assessing the frequency of these errors.
3 Evaluation
The evaluation tests six commercial and academic MT systems across eight grammatical-gender languages using WinoMT and three gender-preservation metrics. All tested systems show gender bias, with weaker performance on non-stereotypical roles and male–female performance differences.
- Experimental Setup: Six widely used MT systems were evaluated: four commercial services and two recent state-of-the-art academic models.The commercial systems were Google Translate, Microsoft Translator, Amazon Translate, and SYSTRAN; the academic models were from Ott et al. and Edunov et al.
- Experimental Setup: The evaluation covered eight grammatical-gender languages spanning Romance, Slavic, Semitic, and Germanic families.The tested languages were Spanish, French, Italian, Russian, Ukrainian, Hebrew, Arabic, and German.
- Results: All tested MT systems were gender biased according to three metrics measuring gender accuracy and male–female and stereotypical–non-stereotypical performance differences.Accuracy measures whether the translation preserves the source entity’s gender; ∆G and ∆S are F1-score differences, with larger values indicating stronger biases.
- Results: Most systems performed poorly on overall gender Accuracy, with the best model for each language often near random-guess performance; German was an exception for several systems.Three of four commercial systems achieved their best performance on German.
- Results: All systems except Microsoft Translator on German performed significantly better on male roles than female roles.The authors suggest this pattern may stem from male roles being more frequent in training data.
- Results: Performance was consistently better for pro-stereotypical than anti-stereotypical role assignments across tested systems and languages.Google Translate and the other systems deteriorated on non-stereotypical assignments, while adding gendered adjectives improved performance in some languages and significantly reduced bias in Spanish, Russian, and Ukrainian.
- Human Validation: Human validation found that the automatic method agreed with sentence-level annotations above 85% for every language and system, averaging 87%.Human inter-annotator agreement averaged 90%, with some noise from incoherent translations.
- Human Validation: The evaluation can be ambiguous when language-specific morphology does not uniquely express gender.Examples include Hebrew words requiring diacritics, gender-neutral contracted determiners in French and Italian, and professions lacking female or male inflections.
4 Discussion
The work uses challenge sets to evaluate linguistic phenomena beyond traditional BLEU, while acknowledging that its synthetic, medium-sized benchmark has important scope limitations. Future work should broaden the data and improve real-world applicability.
- Related work: Challenge sets evaluate machine translation on specially created test sets targeting linguistic phenomena beyond traditional BLEU.
- Limitations and future work: WinoMT's synthetic English examples enable controlled experiments but may introduce artificial biases into the data and evaluation.
- Limitations and future work: WinoMT provides only a proxy estimate of gender bias because its medium-sized test set could be easy to overfit.
- Limitations and future work: The benchmark could be strengthened with natural instances from many source languages annotated with ground-truth entity gender.
- Limitations and future work: The demonstrated debiasing scheme relies on an oracle coreference system and is therefore not applicable in a real-world scenario.
5 Conclusions
The paper provides large-scale multilingual evidence that commercial and academic machine translation systems can translate according to gender stereotypes rather than meaningful context. The evidence covers eight diverse target languages, and the data and code are publicly available.
- Eight diverse target languages were evaluated for gender bias in machine translation.
- Four popular commercial systems and two state-of-the-art academic models were significantly prone to gender-biased translation.
- The authors made their data and code publicly available.