Source-linked AI summary
Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques
Joel Escudé Font, Marta R. Costa-jussà
TL;DR
Neural MT can inherit gender stereotypes from training corpora, motivating methods that improve fairness without sacrificing translation quality. The paper integrates two debiasing techniques into GloVe embeddings used by a Transformer, evaluates them on English-Spanish translation and an occupations test, and reports slightly better BLEU with debiased models alongside changed gender-pronoun behavior.
Problem
Neural MT can inherit social and gender biases from human-generated corpora, while evidence on gender bias in MT remains limited.
Method
The paper integrates Hard-Debiased GloVe and GN-GloVe embeddings into Transformer translation systems and evaluates them with standard and occupation-based tests.
Results
Debiased models achieved slightly better BLEU than the baseline with pre-trained embeddings, while occupation tests found male-pronoun identification was consistent and female-pronoun rates varied by model.
Takeaways & Limitations
The study provides progress toward gender-debiased translation systems using word embedding techniques and analyzes whether they equalize bias in MT.
Takeaways & Limitations
The evaluation focuses on news articles and professions, leaving other domains, language pairs, and social biases for further study.
Abstract
from arXiv · showhide
Neural machine translation has significantly pushed forward the quality of the field. However, there are remaining big issues with the output translations and one of them is fairness. Neural models are trained on large text corpora which contain biases and stereotypes. As a consequence, models inherit these social biases. Recent methods have shown results in reducing gender bias in other natural language processing tools such as word embeddings. We take advantage of the fact that word embeddings are used in neural machine translation to propose a method to equalize gender biases in neural machine translation using these representations. Specifically, we propose, experiment and analyze the integration of two debiasing techniques over GloVe embeddings in the Transformer translation architecture. We evaluate our proposed system on the WMT English-Spanish benchmark task, showing gains up to one BLEU point. As for the gender bias evaluation, we generate a test set of occupations and we show that our proposed system learns to equalize existing biases from the baseline system.
1 Introduction
Neural machine translation improves translation through deep learning, but models trained on human-generated corpora can inherit and amplify social and gender biases. This study evaluates gender bias in MT and proposes debiased word embeddings as a mitigation approach.
- Neural machine translation uses deep learning to improve translation quality across natural languages.
- Models trained on human-generated corpora can learn social stereotypes and gender bias that affect downstream applications.
- A translation may encode gender bias when it ignores contextual gender and instead follows occupational stereotypes.
- The study evaluates gender bias in MT and examines the impact of debiasing in translation systems.
- The proposed contribution combines a framework for detecting and evaluating MT gender bias with debiased word embeddings.
2 Background
The paper uses Transformer translation with word embeddings and examines two approaches for reducing gender information in embeddings. Hard debiasing removes gender direction from neutral words, while GN-GloVe restricts gender information during embedding learning.
- Transformer: The Transformer is a self-attention architecture with encoder and decoder components for producing translated sentences.
- Transformer: Input words are represented as vectors, while positional embeddings preserve sequence information during self-attention processing.
- Word embeddings: Word embeddings represent words as vectors that capture semantic similarity from contextual usage.
- Word embeddings: GloVe learns word vectors from statistical word-occurrence information in an unsupervised, count-based process.
- Equalizing biases in word embeddings: Hard-debiased embeddings identify a gender direction, neutralize gender-neutral words along it, and equalize selected word sets.
- Equalizing biases in word embeddings: GN-GloVe learns embeddings while restricting attributes such as gender to specific dimensions using seed gender words and neutral words.
3 Related work
Prior MT research has documented occupational gender stereotypes and explored speaker information, but studies of bias in MT remain limited. This paper proposes an explicit gender-debiased NMT approach with a coreference- and stereotype-based evaluation.
- Studies of bias in machine translation remain limited compared with broader research on bias in NLP applications.
- A prior machine-translation study found strong male stereotypes for occupations related to science, engineering, and mathematics.
- Adding speaker information to a multilingual politics-domain dataset improved translation quality in previous work.
- This paper differs by explicitly proposing gender-debiased NMT and evaluating it through coreference and occupational stereotypes.
4 Methodology
The methodology adapts Transformer encoder and decoder embedding layers to use pre-trained GloVe-based representations. It compares learned, standard GloVe, Hard-Debiased GloVe, and GN-GloVe embeddings across encoder-only, decoder-only, and dual-side configurations.
- The study adapts the Transformer’s encoder and decoder embedding layers to use pre-trained word embeddings.
- The comparison includes embeddings learned during model training and pre-trained GloVe, Hard-Debiased GloVe, and GN-GloVe variants.
- Pre-trained embeddings are tested separately in the encoder, separately in the decoder, and jointly in both components.
5 Experimental framework
The study evaluates English-Spanish translation with Transformer models using standard or pre-trained word embeddings, including debiased GloVe variants. It combines standard WMT data with an occupations-focused test set designed to assess gender-sensitive translation.
- The experiments use English-Spanish data from WMT, with 16,554,790 training sentences, 3,003 validation sentences, and 3,000 test sentences.
- The custom Occupations test uses sentences pairing gendered pronouns or names with occupations to evaluate translation quality and gender handling.The ambiguous English word “friend” is evaluated through its Spanish alternatives “amiga” and “amigo”.
- Pre-trained embeddings are tested in the encoder, decoder, or both, using GloVe, Hard-Debiased GloVe, and GN-GloVe.
- The baseline is a Transformer without pre-trained word embeddings, implemented with OpenNMT using the toolkit’s baseline parameter settings.
- Hard debiasing defines a gender direction and neutralizes or equalizes gender information in word vectors using gendered word sets and equalization pairs.
- Translation quality is evaluated with BLEU, while training uses separate groups of four GPUs and gradient accumulation equivalent to eight GPUs.
6 Results
The models maintain or improve standard translation quality while reducing gender bias on the occupations evaluation. Debiased embeddings are especially associated with more accurate gendered translations in several occupation contexts.
- 6.1 Translation: Pre-trained embeddings can improve translation, and GN-GloVe preserves or increases this improvement when used in both the encoder and decoder.The authors report that these models do not decrease BLEU quality on the standard machine-translation task.
- 6.2 Gender Bias: The Occupations test evaluates whether context enables correct translation of the neutral English word “friend” as “amigo” or “amiga”.
- 6.2 Gender Bias: “Him” is predicted at almost 100% accuracy for all models, whereas accuracy decreases for “her” and is even lower for “Mary” than for “John”.
- 6.2 Gender Bias: GloVe and Hard-Debiased GloVe produce the most neutral systems, improving accuracy by 30 percentage points over the baseline and over 10 percentage points over non-debiased pre-trained embeddings.Gender debiasing is assessed by increasing “amiga” predictions in female-pronoun contexts while preserving translation quality.
- 6.2 Gender Bias: GN-GloVe in both encoder and decoder yields higher accuracy for gender prediction in Spanish for several technical and other occupations.Reported examples include criminal investigator, heating mechanic, refrigeration mechanic, and mine shuttle car operator.
7 Conclusions and further work
The study applies debiased word embeddings in Transformer-based neural MT and finds modest translation-quality gains alongside reduced gender bias. Its evaluation focuses on English–Spanish news and occupation contexts, leaving broader domains, language pairs, and social biases for future work.
- Conclusions: The study integrates standard, post-process debiased, and gender-neutral GloVe embeddings into Transformer encoder and decoder configurations.Embeddings were evaluated on both sides of the model or on only the encoder or decoder side.
- Results: BLEU performance increases with pre-trained embeddings and is slightly better for the debiased models.Models were evaluated on the WMT newstest2013 standard task.
- Results: The Occupations test shows improved gender-prediction accuracy for some settings using debiased and gender-neutral embeddings while slightly improving baseline translation quality.The evaluation uses English–Spanish sentences where context determines whether “friend” should be translated as “amiga” or “amigo.”
- Contribution: The work is presented as an early proposal for gender-debiased translation systems based on word-embedding techniques.The authors frame the study as a pioneer effort in this direction.
- Further work: The study is limited to news articles and professions, while other domains, language pairs, and social biases remain open evaluation areas.The authors specifically mention industrial, medical, and legal domains, as well as race, religion, and age.