PaperlayerView exact revision on arXiv

Source-linked AI summary

Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem

Danielle Saunders, Bill Byrne

arXiv:2004.04498v3cs.CL

TL;DR

Gender bias in NMT training data can reduce translation quality, particularly for gender-inflected target languages, while fully debiasing datasets is difficult. The paper treats debiasing as domain adaptation using small trusted gender-balanced data, with EWC and lattice rescoring to limit forgetting. Across three language pairs, the approach improves gender debiasing efficiently, and lattice rescoring raises WinoMT accuracy by up to 30% without reducing general-test BLEU.

  • Problem

    Gender bias in NMT training data harms translation quality, especially for gender-inflected target languages, while effectively debiasing full datasets is difficult.

  • Method

    The paper treats gender debiasing as domain adaptation with small trusted gender-balanced data, using EWC during adaptation and lattice rescoring during inference.

  • Results

    Lattice rescoring improves WinoMT accuracy by up to 30% with no decrease in BLEU on the general test set, while experiments cover three language pairs.

  • Takeaways & Limitations

    Gender bias can be reduced without degrading overall translation quality, including when debiasing the output of black-box online commercial MT systems.

  • Takeaways & Limitations

    A robust method for gender-swapping multiple entities in inflected-language parallel sentences is beyond the paper's scope.

Abstract

from arXiv · show

Training data for NLP tasks often exhibits gender bias in that fewer sentences refer to women than to men. In Neural Machine Translation (NMT) gender bias has been shown to reduce translation quality, particularly when the target language has grammatical gender. The recent WinoMT challenge set allows us to measure this effect directly (Stanovsky et al, 2019). Ideally we would reduce system bias by simply debiasing all data prior to training, but achieving this effectively is itself a challenge. Rather than attempt to create a `balanced' dataset, we use transfer learning on a small set of trusted, gender-balanced examples. This approach gives strong and consistent improvements in gender debiasing with much less computational cost than training from scratch. A known pitfall of transfer learning on new domains is `catastrophic forgetting', which we address both in adaptation and in inference. During adaptation we show that Elastic Weight Consolidation allows a performance trade-off between general translation quality and bias reduction. During inference we propose a lattice-rescoring scheme which outperforms all systems evaluated in Stanovsky et al (2019) on WinoMT with no degradation of general test set BLEU, and we show this scheme can be applied to remove gender bias in the output of `black box` online commercial MT systems. We demonstrate our approach translating from English into three languages with varied linguistic properties and data availability.

1 Introduction

Gender bias in training data harms NMT, especially for gender-inflected target languages, while existing debiasing approaches can require difficult dataset construction or inefficient retraining. This paper instead treats debiasing as domain adaptation using small trusted datasets, regularization, and inference-time lattice rescoring.

  • Motivation: Gender bias is especially consequential in NMT for gender-inflected languages, where imbalanced training data produces identifiable translation errors.Translations are more reliable for men and stereotypical gender roles, including male doctors versus male nurses.
  • Prior approaches: Existing approaches use gender-balanced data or debiased embeddings, but training from scratch and balancing large parallel datasets remain challenging.These approaches may be effective, yet they can be inefficient or difficult to construct at scale.
  • Approach: The paper treats gender debiasing as domain adaptation and studies trusted adaptation data, parameter adaptation, and inference with debiased models.The approach uses a small handcrafted profession-based dataset, alongside counterfactual and artificially balanced alternatives.
  • Approach: Naive adaptation improves the debiased domain at the cost of general translation quality through catastrophic forgetting.The paper addresses this trade-off with Elastic Weight Consolidation and two-step lattice rescoring.
  • Results: Continued training on the handcrafted set produces stronger and more consistent debiasing improvements with orders of magnitude less training time, although general BLEU decreases.Experiments cover three language pairs and evaluate both general-domain BLEU and WinoMT.
  • Results: Lattice rescoring improves WinoMT accuracy by up to 30% with no decrease in BLEU on the general test set.The rescoring procedure uses gender-inflected search spaces generated from baseline hypotheses and rescored by the adapted model.
  • Implications: Post-training debiasing avoids identifying and removing every bias before training and can be applied to black-box online commercial MT systems.The paper presents this as an efficient alternative to debiasing the full NMT dataset while preserving privacy because original data access is unnecessary.

2 Gender bias in machine translation

Gender bias in NMT produces gendered translation errors, especially for coreference sentences and stereotypical roles, while WinoMT provides cross-language evaluation metrics. The paper frames debiasing as adaptation to a small gender-balanced domain, while recognizing limitations in counterfactual data generation and catastrophic forgetting.

  • Gender bias in NMT: Gender bias causes NMT systems to mistranslate gendered entities and pronouns, particularly in gender-inflected target languages.Examples include masculine defaults for female doctors and downstream pronoun errors in German translation.
  • WinoMT challenge set and metrics: WinoMT evaluates correctly gendered primary entities using accuracy, gender-bias difference ∆G, stereotype-bias difference ∆S, and male:female prediction ratio M:F.Its 3888 coreference sentences are derived from Winogender and WinoBias and support evaluation across eight target languages with grammatical gender.
  • WinoMT challenge set and metrics: Low-accuracy systems can obtain deceptively small ∆S values, so stereotype bias must be interpreted alongside ∆G and the M:F prediction ratio.A system predicting male forms for most sentences may achieve approximately 50% accuracy on both stereotype classes.
  • Adaptation data: The paper treats gender debiasing as a small-domain adaptation problem and hypothesizes that a well-formed balanced dataset can outperform attempts to debias the full training corpus.The handcrafted set contains 194 professions and 388 gender-balanced sentences, but lacks coreference examples and grammatical-gender variety.
  • Counterfactual datasets: Counterfactual augmentation is difficult for inflected languages because all co-referent gendered entities and corresponding parallel text must be updated consistently.The paper therefore presents a rough approach rather than a robust general solution for multiple entities in inflected-language sentences.
  • Maintaining general translation performance: Fine-tuning on a distinct, tiny adaptation domain risks catastrophic forgetting of general translation performance.The paper identifies this trade-off as especially acute for the small handcrafted adaptation sets.

3.1 Languages and data

The experiments cover English-to-German, English-to-Spanish, and English-to-Hebrew translation, spanning Germanic, Romance, and Semitic languages with varied data conditions. The datasets contain similar proportions of gendered sentences, but their gender balance can reflect topical phrases rather than balanced professional entities.

  • Language pairs: Experiments use English-to-German, English-to-Spanish, and English-to-Hebrew translation, covering Germanic, Romance, and Semitic language groups.Training data quantity and quality vary across the language pairs.
  • Training data: The English-to-German corpus contains 17.6M sentence pairs from WMT19 news data, while English-to-Spanish uses 10M UN Corpus pairs.German is validated on newstest17 and tested on newstest18; Spanish is validated on newstest12 and tested on newstest13.
  • Training data: The English-to-Hebrew corpus contains 185K TED-talk sentence pairs, making it both specialized and much smaller than the other training sets.It is validated on IWSLT 2012 and tested on IWSLT 2014.
  • Data definition: A gendered sentence contains at least one English gendered stopword, and Table 1 reports male-to-female ratios among gendered training sentences.The datasets use filtering based on parallel sentence lengths and length ratios.

3.2 Training and inference

All models use Transformer-base settings and are trained to validation BLEU convergence, while decoding uses SGNMT3 with beam size 4.

  • Training: All models use the Transformer base configuration from Tensor2Tensor and train baselines to validation-set BLEU convergence on one GPU.Gradient updates are delayed by a factor of 4 to simulate four GPUs.
  • Inference: Normal and lattice-constrained decoding use SGNMT3 with beam size 4.

3.3 Lattice rescoring with debiased models

Lattice rescoring constructs gender-inflected target-side search spaces from vocabulary forms and applies debiased models to rescore them.

  • Lattice construction: The rescoring transducer generates gender-inflected forms for words in the target vocabulary to create alternative search paths.German uses the full target training vocabulary; Spanish and Hebrew use OpenSubtitles word lists.
  • Lattice construction: Morphological tools generate alternate-gender forms for German and Spanish, while Hebrew uses a small set of gendered suffixes.The procedures approximately lemmatize vocabulary words before generating alternatives.
  • Lattice construction: Although the lattice may contain non-words, the authors expect debiasing models to assign them low likelihood.

3.4 Results

Results show that handcrafted-data adaptation and lattice rescoring produce stronger and more consistent WinoMT improvements than counterfactual fine-tuning. Lattice rescoring preserves general BLEU while improving adapted or commercial-system outputs, though effectiveness varies by language and baseline quality.

  • Baselines: Baseline systems achieve WinoMT accuracy, ∆G, and ∆S comparable to four commercial systems, outperforming at least one system on every metric and language pair.
  • Counterfactual adaptation: Fine-tuning on original data slightly improves WinoMT accuracy and ∆G across language pairs but worsens ∆S, while FTrans original sharply increases ∆G.The authors suspect bias amplification from forward-translation in FTrans original; over-predicting male entities can lower ∆S artificially.
  • Counterfactual adaptation: Counterfactual fine-tuning improves results inconsistently across language pairs, with swapped and balanced sets not clearly better than equivalent non-counterfactual sets.The balanced set produces a similar effect to FTrans swapped, with a smaller test-BLEU difference from baseline.
  • Handcrafted adaptation: Handcrafted-set adaptation raises WinoMT accuracy by up to 19 points and improves ∆G by at least a factor of 4, consistently across WinoMT metrics and language pairs.The no-overlap model also improves WinoMT without simply memorising challenge-set vocabulary.
  • Regularisation: EWC limits BLEU loss to about 0.5 BLEU per language pair while trading general translation performance against WinoMT accuracy.Further adaptation toward convergence yields additional WinoMT gains at the expense of BLEU.
  • Lattice rescoring: Lattice rescoring preserves or slightly improves general BLEU and delivers WinoMT accuracy gains of up to 30% relative to baseline.For converged models, BLEU degradation is limited to 0.2 BLEU, with gains of 36%, 38%, and 24% for English-German, English-Spanish, and English-Hebrew.
  • Commercial systems: Rescoring substantially improves WinoMT accuracy for commercial systems without requiring access to their models or training data.A 25.5% baseline English-German accuracy range becomes 3.6% after rescoring.

4 Conclusions

The paper treats gender bias in NMT as a domain adaptation problem and uses tiny, handcrafted gender-balanced datasets for three language pairs. EWC and lattice rescoring limit catastrophic forgetting, enabling bias reduction while preserving general translation performance.

  • The study frames NMT gender debiasing as domain adaptation using tiny, handcrafted gender-balanced datasets across three language pairs.
  • Naive adaptation can cause catastrophic forgetting, but EWC and lattice rescoring limit this trade-off between debiasing and general translation quality.
  • Lattice rescoring enables substantially more debiasing with potentially no degradation in general translation performance and without access to the original model.
  • The authors suggest small-domain adaptation is more effective and efficient than counterfactual data augmentation, while explicitly not claiming to eliminate NMT bias.

A WinoMT male:female prediction ratio

The section introduces the WinoMT male:female prediction ratio as an intuitive metric and situates expanded tables alongside earlier results. Table 6 reports general BLEU and WinoMT scores after unregularised fine-tuning on four gender-based datasets.

  • The M:F prediction ratio on WinoMT is presented as an intuitive metric for assessing gender prediction behavior.
  • Table 6 reports general test set BLEU and WinoMT scores after unregularised fine-tuning on four gender-based adaptation datasets.
  • Table 7 compares handcrafted and counterfactual adaptation data, training procedures, and lattice rescoring results, with its first two lines duplicated from Table 6.
Loading 2004.04498v3…