Source-linked AI summary
Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation
Eva Vanmassenhove, Dimitar Shterionov, Andy Way
TL;DR
The paper asks whether MT loses lexical diversity and favors frequent forms, potentially worsening gender bias. It compares MT systems and finds general lexical loss alongside frequency shifts that leave many words absent.
Problem
Whether MT generalization suppresses lexical diversity and amplifies frequent forms, potentially exacerbating gender bias, lacked empirical evidence.
Source: §1 · pp. 1–2Method
Results
MT causes general lexical-diversity loss versus human-generated text, while systems increase frequencies of more frequent words and decrease those of less frequent words, leaving many words lost.
Source: §6 · p. 9Takeaways & Limitations
The authors aim to investigate solutions to translation-model overgeneralization, while noting that randomness may conflict with strong learning and generalization.
Source: §6 · p. 9The RNN and Transformer models were not optimized to state-of-the-art settings; they used no BPE and unrestricted vocabularies.
Source: §5 · p. 8
Abstract
from arXiv · showhide
This work presents an empirical approach to quantifying the loss of lexical richness in Machine Translation (MT) systems compared to Human Translation (HT). Our experiments show how current MT systems indeed fail to render the lexical diversity of human generated or translated text. The inability of MT systems to generate diverse outputs and its tendency to exacerbate already frequent patterns while ignoring less frequent ones, might be the underlying cause for, among others, the currently heavily debated issues related to gender biased output. Can we indeed, aside from biased data, talk about an algorithm that exacerbates seen biases?
1 Introduction
The introduction frames lexical loss as an established feature of translation and asks whether MT systems also suppress lexical diversity and amplify frequent forms. It proposes examining these patterns across MT architectures and their possible connection to issues such as gender bias.
1 Introduction
- Prior research describes lexical impoverishment in translation; empirical work found lexical loss in human translations and greater loss in SMT than HT.Berman (2000) discusses quantitative impoverishment, and Kruger (2012) empirically examined lexical loss; the cited MT study reports greater loss for SMT than HT.Source: §1 · p. 1
- MT may exacerbate frequent patterns while suppressing alternatives, potentially contributing to gender bias, morphological errors, and difficulties with agglutinative languages.The authors note that statistical systems may treat lexical alternatives and morphological variants similarly when context does not resolve ambiguity.Source: §1 · p. 1
- Although diverse output has been studied in language generation, the authors hypothesize that MT overgeneralization may also exacerbate gender bias.They note that empirical evidence for algorithmic bias in MT had not yet been provided.Source: §1 · pp. 1–2
- The study compares lexical diversity across MT systems and analyzes word frequencies to assess whether architectures favor translations frequent in training data.These are the work’s two stated objectives: compare NMT with SMT and HT, and quantify preference for frequent training-data translations.Source: §1 · p. 2
2 Related Work
Related work connects lexical loss to translation simplification, cohesion, associative texture, and evaluation of MT lexical diversity. The cited studies report differences between human translation and MT, and between SMT and NMT.
2 Related Work
- Kruger (2012) found human-translated English texts more simplified in language use than comparable nontranslated originals.Berman (2000) had described quantitative impoverishment as an inherent, though partly mitigable, tendency of translation.Source: §2 · p. 2
- Klebanov and Flor (2013) found that SMT suffers more lexical loss than HT when comparing associative texture across originals, backtranslations, references, and system translations.They examined the destruction of networks of signification, which the authors describe as potentially unavoidable in translation generally.Source: §2 · p. 2
- Bentivogli et al. (2016) used type-token ratio to assess vocabulary size and subject-matter variety, finding NMT better able to cope with lexical diversity than SMT.Wong and Kit (2012) related under-use of lexical devices in SMT to cohesion.Source: §2 · p. 2
3 Hypothesis
The authors hypothesize that data-driven MT favors frequent words and neglects rarer ones. They expect this pattern to affect lexical diversity and the use or disappearance of morphological variants.
3 Hypothesis
- The hypothesis is that data-driven MT generalization favors frequent words and disregards less frequent ones, producing measurable effects on word choice.The authors expect the most visible effects in word frequencies and the disappearance of scarce words.Source: §3 · p. 3
- This frequency skew may alter morphological variants, including male-form overuse, preference for third over first person, and difficulty handling morphologically richer target languages.The authors present these as possible effects of the hypothesized behavior, not as results established in this section.Source: §3 · p. 3
- The authors further hypothesize that NMT is more susceptible to overgeneralization because it jointly handles translation and language modeling, making optimization harder than in SMT.The paper presents this as an additional hypothesis.Source: §3 · p. 3
4 Empirical evaluation
The evaluation compares three MT architectures on EN-FR and EN-ES Europarl data, including systems trained on back-translated data. Reverse-system English translations show a particularly large vocabulary-size drop for neural systems.
4 Empirical evaluation
- The experiments used attentional RNN, Transformer, and Moses systems for EN-FR and EN-ES on Europarl data, assessing seen and unseen inputs.The authors used about 2 million sentence pairs per language pair and made test sets large to maximize language variability.Source: §4 · p. 3
- Neural systems use no subword units so that subword splitting would not add variability to the word-frequency analysis.Their dictionaries include all words in the training data.Source: §4 · p. 4
- Reverse-direction systems translated the same training-set language data into English, and those outputs became source data for EN*→FR/ES BACK systems.This setup was intended to examine algorithm effects in a favorable scenario where the input had already been seen.Source: §4 · p. 4
- Neural REV systems' English translations showed a huge drop in vocabulary size, while SMT REV systems showed a significant but less profound decrease; original-data source and target vocabularies were comparable.Table 3 reports vocabularies for REV-generated English used as BACK input and French/Spanish BACK output.Source: §4 · p. 4, Table 3 · p. 4
- BLEU and TER are reported for 12 analyzed systems, and all Table 4 results are statistically significant at p < 0.05.The authors analyze FF and BACK outputs; REV systems are used only to generate back-translated data.Source: §4 · pp. 4–5, Table 4 · p. 5
5 Analysis
The analysis compares lexical richness and word-frequency patterns across human and machine translations, using separate seen- and unseen-data evaluations. All MT systems fall short of human lexical richness, while their outputs favor frequent words and lose less frequent ones.
Lexical richness
- All MT systems fall short of human lexical richness; SMT retains more than neural systems on Yule’s I and TTR, but Transformers lead on MTLD, and RNN systems are consistently worst.The results cover both language pairs and both seen and unseen data.Source: Table 6 · p. 6, Table 7 · p. 6, §5 · p. 7
Word frequencies and bias
- Word frequencies are normalized by sentence length and total frequency, with zero-frequency words counted separately; frequent words are defined using average normalized HT frequency.The analysis counts normalized word classes and accumulated frequency differences against HT.Source: §5 · pp. 6–7
- All systems lose less-frequent words, while accumulated frequency differences favor frequent words even when some systems produce fewer higher-frequency words than HT.The analysis compares frequency increases, decreases, and zero-frequency words across training and test translations.Source: Table 8 · p. 7, Table 9 · p. 7, Table 10 · p. 7
Automatic evaluation
- Transformer systems perform best on automatic quality evaluation, and MTLD is the only lexical-richness metric that corroborates BLEU and TER.The authors identify incorporating lexical richness, possibly through MTLD features, as a future research direction.Source: §5 · pp. 7–8
Seen and unseen data
- Seen and unseen data show similar lexical-richness and frequency trends, but unseen test translations have a slight MTLD decrease and lose more words altogether.The MTLD decrease is reported for the unseen test set.Source: §5 · p. 8
Semi-manual evaluation
- For English “picture,” MT systems use Spanish “imagen” more often than the original data, while less-frequent translation variants decline or disappear.The same pattern—frequent translations gaining relative frequency at less-frequent options’ expense—appears for “happen” and “also.”Source: §5 · pp. 8–9, Figure 5 · p. 9
6 Conclusions and Future Work
The study examines lexical richness across 12 MT systems and finds that MT outputs lose lexical diversity relative to human-generated text. It concludes that systems overgeneralize, while noting differences among paradigms and the challenge of addressing overgeneralization without weakening learning.
6 Conclusions and Future Work
- The analysis measures lexical richness across 12 SMT, RNN, and Transformer systems for EN–FR and EN–ES using original and back-translated data.Source: §6 · p. 9
- MT causes a general loss of lexical diversity and richness compared with human-generated text, with many words becoming rarer or disappearing.The authors interpret this pattern as evidence of overgeneralization and a form of algorithmic bias.Source: §6 · p. 9
- SMT retains the most lexical richness on TTR, Yule’s I, and MTLD, although Transformer models perform best on evaluation metrics.RNN systems are among the worst for lexical diversity; the authors caution that not using BPE may disadvantage neural models relative to SMT.Source: §6 · p. 9
- Future work will investigate ways to address translation-model overgeneralization, but adding randomness while preserving strong learning and generalization may be complex and contradictory.Source: §6 · p. 9