Source-linked AI summary
Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation
Eva Vanmassenhove, Dimitar Shterionov, Matthew Gwilliam
TL;DR
The paper asks whether algorithmic bias in MT, beyond familiar societal biases, can impoverish language by favoring frequent forms. It measures lexical and morphological richness across PB-SMT and NMT outputs and finds reduced richness across all investigated systems, language pairs, and directions.
Problem
The paper investigates whether algorithmic bias can reduce linguistic richness in MT beyond the societal biases commonly studied.
Method
The study compares training data with outputs from PB-SMT, LSTM NMT, and Transformer NMT using lexical, morphological, and synonym-frequency measures.
Results
All 9 metrics indicate that original training data is more lexically and morphologically diverse than translations across all language pairs and directions.
Takeaways & Limitations
Machine translationese is measurably less linguistically rich than the training data across the investigated MT architectures and language settings.
Takeaways & Limitations
Diversity may be undesirable for some tasks and domains that prioritize consistent in-domain translations, and the morphological analysis excludes single-wordform lemmas.
Abstract
from arXiv · showhide
Recent studies in the field of Machine Translation (MT) and Natural Language Processing (NLP) have shown that existing models amplify biases observed in the training data. The amplification of biases in language technology has mainly been examined with respect to specific phenomena, such as gender bias. In this work, we go beyond the study of gender in MT and investigate how bias amplification might affect language in a broader sense. We hypothesize that the 'algorithmic bias', i.e. an exacerbation of frequently observed patterns in combination with a loss of less frequent ones, not only exacerbates societal biases present in current datasets but could also lead to an artificially impoverished language: 'machine translationese'. We assess the linguistic richness (on a lexical and morphological level) of translations created by different data-driven MT paradigms - phrase-based statistical (PB-SMT) and neural MT (NMT). Our experiments show that there is a loss of lexical and morphological richness in the translations produced by all investigated MT paradigms for two language pairs (EN<=>FR and EN<=>ES).
1 Introduction
The paper examines whether algorithmic bias in MT can reduce lexical and morphological richness, creating an artificially impoverished form of language called machine translationese. It introduces metrics and compares MT outputs with their training data across major data-driven paradigms.
- 1 Introduction: Because MT systems increasingly produce widely used translations, the paper argues that output richness deserves attention alongside accuracy and fluency.The authors note that maintaining diversity has not traditionally been a priority.
- 1 Introduction: MT systems may prefer frequent words and morphological variants, reducing lexical and grammatical variety in their outputs.The authors connect this hypothesized loss to algorithmic bias and possible longer-term language change.
- 1 Introduction: The authors acknowledge that diversity can be harmful for some tasks and domains that prioritize consistent in-domain translations.This qualification limits the claim that greater diversity is always desirable.
- 1 Introduction: The study asks whether MT outputs differ measurably from their training data in lexical and morphological diversity.It presents this as the first study of both dimensions in machine translationese.
- 1 Introduction: The analysis goes beyond standard lexical-diversity metrics by adapting language-acquisition measures and adding entropy, Simpson diversity, and synonym-frequency measures.The metrics target lexical sophistication, morphological richness, and synonym use.
2 Related Work
Related work shows that translationese and algorithmic bias can be detected through systematic linguistic differences, while prior MT studies generally report reduced lexical diversity. This paper shifts comparison toward the difference between an MT system’s training data and its own output, including morphological diversity.
- 2 Related Work: Research on algorithmic bias has focused heavily on societal categories such as gender and race, but bias can also mean systematic inaccuracy in any direction.The paper positions linguistic richness as a broader dimension of algorithmic bias.
- 2 Related Work: Prior MT research found PB-SMT and NMT translations consistently less lexically diverse than original training data.Earlier work used TTR, MTLD, and Yule’s measures, while one study also examined post-editese across multiple language pairs.
- 2 Related Work: Earlier studies primarily compared human or post-edited translations with machine translations using lexical-diversity and translationese features.The cited work also examined lexical loss in PB-SMT and automatic identification of machine versus human translations.
- 2 Related Work: The paper’s novelty is to compare MT output with the data used to train the system, rather than primarily comparing machine output with human translation.This design targets effects attributable to the algorithm on language itself.
3 Machine Translation Systems
The study trains PB-SMT, LSTM, and Transformer systems on Europarl data for English–French and English–Spanish in both directions. It evaluates translation quality and reports statistically significant system comparisons, with Transformer ranked highest on the quality metrics.
- 3 Machine Translation Systems: The evaluated paradigms are PB-SMT, LSTM-based NMT, and Transformer NMT.PB-SMT uses Moses, while the neural systems use OpenNMT-py; NMT vocabularies use BPE subwords.
- 3 Machine Translation Systems: The experiments cover English–French and English–Spanish translations in both directions using Europarl data.The bidirectional setup compares translations from English with translations into English.
- 3 Machine Translation Systems: Translation quality is evaluated with BLEU and TER, while unknown PB-SMT words are mapped to one UNK token to avoid artificially inflated linguistic-diversity scores.The UNK treatment affects the linguistic analysis but not BLEU or TER.
- 3 Machine Translation Systems: All reported system comparisons are statistically significant under bootstrap resampling with a 95% confidence interval.The analysis uses 1000 iterations and samples of 100 sentences.
- 3 Machine Translation Systems: Transformer systems outperform the other systems on all metrics, followed by PB-SMT and then LSTM.The passage states that this ranking holds across all language pairs.
4 Experiments and Results
The experiments compare lexical and morphological diversity between original training data and machine-generated translations using multiple complementary metrics. Across language pairs and MT paradigms, translations consistently show reduced lexical and morphological richness, although retention varies by system and language.
- Lexical Diversity: Original training data is consistently more lexically diverse than machine translationese across the evaluated language pairs and MT systems.This pattern appears in LFP measures and in TTR, Yule’s I, and MTLD.
- Lexical Diversity: LFP compares three frequency bands, with original data using fewer 0–1000-word tokens and more words beyond the first 2000 than MT outputs.The adapted profile derives bands from the original training data rather than predefined L2 word lists.
- Lexical Diversity: French and Spanish show higher lexical richness than English, reflecting their richer morphology in token-based frequency profiles and diversity scores.This language contrast appears in both the LFP analysis and the broader lexical metrics.
- Lexical Diversity: Synonym-frequency metrics rank systems PB-SMT > TRANS > LSTM, while reference translations remain most diverse across both language pairs.Reference data has the lowest primary translation frequency and cosine distance from uniform, and the highest SynTTR.
- Grammatical Diversity: Shannon entropy and Simpson’s index show consistently higher morphological diversity in original data than in MT outputs.Among MT systems, TRANS retains morphological diversity best; LSTM exceeds PB-SMT for French and Spanish, whereas PB-SMT is better for English.
5 Conclusions
The study finds that MT outputs are less lexically and morphologically rich than their training data across language pairs and directions. It develops a nine-metric evaluation suite and observes that Transformer models generally preserve more richness than other tested architectures.
- The study analyzes linguistic richness through lexical diversity and sophistication, morphological variety, and synonymy using 9 established, adapted, and newly proposed metrics.
- The developed metric suite is presented as unprecedented in MT quality research and as a basis for future MT evaluation.
- All 9 metrics show greater lexical and morphological diversity in original training data than in MT translations across language pairs and directions.
- Transformer models generally outperform PB-SMT and LSTM in lexical and morphological richness across most metrics.