Source-linked AI summary
The Curious Case of Hallucinations in Neural Machine Translation
Vikas Raunak, Arul Menezes, Marcin Junczys-Dowmunt
TL;DR
The paper asks why NMT produces severe hallucinations, including source-perturbation failures and natural outputs detached from their inputs. It combines memorization analysis with controlled corpus noise and studies Backtranslation and Knowledge Distillation, finding that memorized samples are more vulnerable, specific noise patterns generate distinct natural hallucinations, and these patterns can be amplified by data-generation processes.
Problem
NMT hallucinations are severe adequacy failures, yet their generation lacks a coherent empirically validated explanation across perturbations, natural cases, and data-generation pipelines.
Method
The paper extends a memorization estimator to sequence-to-sequence models, introduces corpus-level noise, and analyzes hallucinations produced by Backtranslation and Knowledge Distillation.
Results
Memorized samples hallucinate more under perturbation; specific corpus noise patterns generate known natural hallucinations; and Backtranslation and Knowledge Distillation amplify hallucination patterns.
Takeaways & Limitations
Hallucinations can be studied through memorization and corpus-noise mechanisms, with amplification affecting generated training data and potentially downstream systems.
Takeaways & Limitations
Most experiments use the IWSLT 2014 corpus because the analysis is compute-intensive, while downstream effects of amplified hallucinations remain for future work.
Abstract
from arXiv · showhide
In this work, we study hallucinations in Neural Machine Translation (NMT), which lie at an extreme end on the spectrum of NMT pathologies. Firstly, we connect the phenomenon of hallucinations under source perturbation to the Long-Tail theory of Feldman (2020), and present an empirically validated hypothesis that explains hallucinations under source perturbation. Secondly, we consider hallucinations under corpus-level noise (without any source perturbation) and demonstrate that two prominent types of natural hallucinations (detached and oscillatory outputs) could be generated and explained through specific corpus-level noise patterns. Finally, we elucidate the phenomenon of hallucination amplification in popular data-generation processes such as Backtranslation and sequence-level Knowledge Distillation.
1 Introduction
NMT has surpassed statistical approaches in many settings but still exhibits severe pathologies. Hallucinations are especially egregious because fluent or moderately fluent outputs can be decoupled from the source.
- Hallucinations are extreme NMT adequacy failures in which generated outputs are decoupled from the source.
1. NMT models tend to generate hallucinated
The paper frames NMT hallucinations as an insufficiently explained family of failures and proposes a systematic account spanning perturbations, corpus noise, and data-generation pipelines. Its contributions link perturbation hallucinations to memorization, show that specific noise patterns generate natural hallucinations, and demonstrate amplification during Backtranslation and Knowledge Distillation.
- The paper studies hallucinations through generalization, memorization, and optimization in sequence-to-sequence models.
- Memorized training samples hallucinate more under source perturbation, explained through an extension of the long-tail theory.
- Specific corpus-level noise patterns interact with sequence-to-sequence training dynamics to generate prominent hallucination patterns.
- Hallucination amplification occurs in outputs generated through Backtranslation and Knowledge Distillation.
2 Related Work
Prior work documents several hallucination phenomena and detection approaches but lacks a coherent empirically validated explanatory theory. This paper connects hallucinations to generalization and memorization while also addressing natural hallucinations and corpus-level detection.
- Existing work reports hallucinations under source perturbations and greater hallucination propensity for out-of-domain inputs.
- The literature lacks a coherent, empirically validated theory explaining hallucination phenomena.
- Manual detection of natural hallucinations impedes rapid experiments, motivating an approximate corpus-level detection algorithm.
- Long-tail theory links memorization of rare subpopulations with improved generalization, providing the conceptual lens for this study.
3 Categorizing Hallucinations in NMT
The paper separates hallucinations under perturbations from natural hallucinations, then divides natural hallucinations into detached and oscillatory types. This categorization enables quantitative analysis distinct from other NMT pathologies.
- Hallucinations under Perturbations: Hallucinations under Perturbations occur when translations for perturbed and unperturbed inputs differ drastically.
- Natural Hallucinations: Natural Hallucinations are severely inadequate translations produced for unperturbed source inputs, whether fluent or not.
- Natural Hallucinations: Detached Hallucinations are fluent but completely inadequate translations.
- Natural Hallucinations: Oscillatory Hallucinations are inadequate translations containing repeating ngrams.
- Scope of categorization: The categorization excludes shorter outputs and source copying, allowing hallucinations to be quantitatively disentangled from other NMT pathologies.
4 Origins of Hallucinations
The section tests two explanations for NMT hallucinations: memorized samples are especially vulnerable under source perturbation, while corpus-level noise patterns generate distinct natural hallucination behaviors.
- Overview: The study validates two hypotheses linking hallucinations to sample memorization under perturbation and specific corpus-level noise patterns.The experiments construct perturbations and four noise patterns, then analyze the resulting translations.
- Hallucinations under Perturbations: MVE estimates a sample’s memorization value from the prediction-metric change between models trained with and without that sample.The sequence-to-sequence adaptation uses overlap metrics such as chrF or BLEU.
- Hallucinations under Perturbations: The Memorized set produces far more unique hallucinations under perturbation than an equally sized Random set across chrF, BLEU, and accuracy-based memorization estimates.The difference remains consistent as memorization estimates are refined through additional sample exclusions, although coarser metrics reduce the gap.
- Hallucinations under Perturbations: Unique and total hallucination counts both increase as the minimum memorization-value threshold rises.This establishes a strong positive correlation between hallucination frequency and memorization values in the Figure 3 comparisons.
- Hallucinations under Perturbations: Memorized samples also show distinct decoder cross-attention distributions, including more fixed, lower-entropy attention than Random samples.The attention comparison further links memorization with hallucination under perturbation.
- Natural Hallucinations: Corpus-level noise experiments train a no-noise baseline and four models with specific added invalid source-target pair patterns.The noise corpus uses an invalid reference set and varies source-target sampling strategies; BPE is learned on each noise-added corpus.
3. On the IRS set, the UR model produces a num-
Specific corpus-level noise patterns produce distinct natural hallucination behaviors: UR promotes copied training references, while RU promotes oscillatory outputs.
- These noise–hallucination links provide a diagnostic tool for tracing hallucinated outputs to corpus-level noise and removing it from training data.
- UR noise produces natural hallucinations that directly copy training references, even when IRS source sequences are absent from training.
- RU noise produces prominent oscillatory hallucinations, reflected by considerably fewer unique bigrams than other noise types.
- Among the four noise patterns, RU leads to the most oscillatory hallucinations in translations of IRS source sequences.
5 Hallucination Amplification
The paper evaluates whether hallucinations caused by corpus-level noise are amplified when noisy models generate data for Backtranslation and Knowledge Distillation. UR models cause severe amplification, while the estimator misses many oscillatory hallucinations.
- The reference-free corpus-level estimator uses multilingual similarity scoring and repeat analysis to measure natural hallucinations at scale.
- UR models cause severe hallucination amplification in both Backtranslation and Knowledge Distillation.
- For Knowledge Distillation, every noisy model increases natural hallucinations relative to the initial parallel corpus.
- For Backtranslation, UU and UR models generate many repeated outputs, whereas RR models cause the fewest hallucinations across both procedures.
- The study leaves analysis of hallucination effects in downstream systems trained on amplified Knowledge Distillation data for future work.
6 Discussion
The discussion examines memorized long-tail samples and outlines possible prevention strategies targeting data augmentation, learning robustness, and corpus-level filtering.
- 6.1 Memorized Samples: The most memorized training samples represent long-tail data and appear syntactically different from a random training subset.
- 6.1 Memorized Samples: The paper leaves quantitative analysis of differences between memorized and random samples, and their link to out-of-domain samples, for future work.
- 6.2 Preventing Hallucinations: Data augmentation could target long-tail sample characteristics to bring memorized samples out of the long tail, although transition dynamics require further study.
- 6.2 Preventing Hallucinations: Robust early-learning methods designed to prevent memorization may help prevent perturbation-based hallucinations.
- 6.2 Preventing Hallucinations: Loss truncation, Tilted-ERM, and related robust-learning methods are proposed as ways to reduce the impact of noisy references and outliers.
- 6.2 Preventing Hallucinations: Filtering invalid source-target pairs or bitext indeterminacy could reduce natural hallucinations.
7 Conclusion
The paper concludes that memorization, corpus-level noise, and downstream data-generation processes are connected to distinct hallucination phenomena in NMT. It also notes scope and analysis boundaries while motivating further work.
- Memorized training samples are far more likely than non-memorized samples to hallucinate under source perturbation.
- Specific corpus-level noise patterns generate specific well-known hallucination patterns.
- Backtranslation and sequence-level Knowledge Distillation amplify these hallucination patterns in generated outputs.
- Most experiments use IWSLT 2014 because the analysis involves compute-intensive algorithms, although the authors argue the insights should apply to larger-scale models.
- The authors identify learning-centric fixes for memorization and corpus-level noise as future research directions.
A Appendices
Table 10 presents random training-set samples with low Memorization Values, measured using the MV Estimator with chrF for De-En. Their syntactic differences from Table 9 require further quantitative analysis.
- The low-MV sentences differ in syntactic properties from the samples in Table 9.
- Quantitative analysis is still required to establish the extent of these syntactic differences.
- Table 10 lists random training-set samples with Memorization Values between 0.0 and 0.6, below the top 100.The values were measured using the Memorization Value Estimator (Algorithm 1) with chrF as the metric for De-En.