Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

• Zero-shot NMT translates between unseen language pairs without direct parallel corpora by exploiting multilingual training.Source: §1 · p. 1
• Naive zero-shot NMT is sensitive to training conditions and usually underperforms pivot-based translation.Source: §1 · p. 1
• The paper quantitatively attributes zero-shot degeneracy to capturing spurious correlations between source and decoded language identities.Source: §1 · p. 1
• Decoder pre-training and back-translation substantially improve zero-shot NMT across Europarl, IWSLT, and MultiUN experiments.The approaches outperform vanilla zero-shot NMT and exceed pivot-based translation by 2–3 BLEU points on some language pairs.Source: §1 · p. 1

• NMT factorizes the probability of a target sentence into conditional token probabilities given the source and preceding target tokens.The decoder uses beginning- and end-of-sentence markers.Source: §2 · pp. 1–2
• Multilingual NMT trains one model across available language pairs while providing the target language ID to control the output language.The target ID is implemented as the decoder's initial token, and the model shares a representation space across languages.Source: §2 · p. 2
• Pivot-based NMT handles an unseen direction by translating from the source language into an intermediate pivot and then into the target language.For example, Spanish-to-French can be performed through English when direct Spanish–French data is unavailable.Source: §2 · p. 2
• Pivoting has at least twice the latency of comparable direct translation and makes the second model process errors produced by the first.Source: §2 · p. 2
• Zero-shot NMT directly translates unseen pairs when both languages appeared in multilingual training, without direct supervision for that direction.Its feasibility has been associated with multilingual encoders learning a shared representation space.Source: §2 · p. 2

3.1 Zero-shot NMT is Sensitive to Training Conditions

• Zero-shot translation quality is highly sensitive to hyperparameters, while parallel-direction performance remains comparatively stable across tested variants.The layerwise-attn variant completely fails on zero-shot pairs.Source: §3.1 · p. 3

3 Degeneracy Issue of Zero-shot NMT

• Zero-shot NMT remains well below pivot-based translation on some pairs, including Fr-De, even under the stable default setting.Source: §3.1 · p. 3

3.2 Performance Degeneracy is Due to Capturing Spurious Correlation

• The output can be viewed as combining a target-language factor z and language-invariant semantics s, with zero-shot translation requiring them to be controlled by the target ID and source sentence separately.Source: §3.2 · p. 3
• Maximum-likelihood training can capture a spurious dependency between source language i and output language z, reducing the decoder's reliance on the requested target language j.When this dependency dominates, the model may fail to produce the desired target language.Source: §3.2 · pp. 3–4
• Zero-shot performance varies inversely with estimated mutual information I(i; z), supporting spurious correlation as the source of degeneracy.Source: §3.2 · p. 4
• Training conditions that make machine translation easier, such as larger batches, can increase I(i; z), making zero-shot performance unstable and causing wrong-language outputs.Source: §3.2 · p. 4

4 Approaches

• Decoder pre-training and back-translation are proposed to address zero-shot NMT degeneracy caused by spurious correlations.These are presented as approaches for the zero-shot NMT setting.Source: §4 · p. 4, §4.2 · p. 5

4.1 Language Model Pre-training

• Language-model pre-training strengthens the decoder’s language-model prior so it better follows the requested target language and can ignore source-language correlations.It uses a few pre-training steps before continuing with machine-translation training on the same parallel data.Source: §4.1 · p. 4
• Language-model pre-training is efficient and requires no additional data, but its objective does not necessarily align with the NMT objective.The method replaces source representations with zero vectors during pre-training, equivalent to ignoring encoder–decoder attention in a Transformer.Source: §4.1 · p. 4

4.2 Back-Translation

• Back-translation generates synthetic parallel data for every zero-shot direction and trains on merged real and synthetic datasets, forcing broader multilingual translation coverage.Synthetic corpora can be initialized through a pivot-based model (BTTP) or a zero-shot model (BTZS).Source: §4.2 · p. 5
• Back-translation explicitly avoids spurious correlations but is more computationally expensive, requiring synthetic corpora for up to O(K2) language-pair directions and depending on starting-model quality.BTZS may benefit from a zero-shot model augmented with language-model pre-training.Source: §4.2 · p. 5

5.1 Experimental Settings

• The experiments cover Europarl, IWSLT, and MultiUN, including ordinary English-pivot settings, no-pivot settings, and an IWSLT chain-of-pivots setting.Europarl-b includes zero-shot pairs with neither direct nor pivot parallel sentences, where pivot-based methods are not applicable.Source: §5.1 · p. 5, A Chain of Pivots · p. 7

5.2 Model Selection for Zero-shot NMT

• The proposed model-selection score combines likelihood on available parallel validation data with a language-model score to detect zero-shot degeneracy.The language-model term decreases when spurious source–decoded-language correlations are captured, helping select a model for unseen directions.Source: §5.2 · p. 6

5.3 Results and Analysis

• 4–22 BLEU points: BTTP and BTZS significantly improve zero-shot translation over the naïve approach across the evaluated language settings.LM pre-training alone closes the gap with strong pivot-based baselines on the evaluated datasets, and ZS+LM can outperform pivoting for lexically similar pairs.Source: §5.3 · p. 6, Table 2 · p. 7
• BTTP and BTZS are competitive with the fully supervised upper bound, with BTZS often slightly better than BTTP across languages.Source: §5.3 · p. 6
• Without pivot languages, LM and BTZS still substantially improve the vanilla model and achieve competitive performance with fully supervised training.Source: §5.3 · pp. 6–7

A Chain of Pivots

• As pivot-chain length increases, pivot-based performance degrades sharply, whereas zero-shot translation avoids this degradation and achieves large gains over pivoting.Direct zero-shot translation avoids the computational and error-accumulation costs of traversing the chain.Source: A Chain of Pivots · p. 7
• LM pre-training and BTZS remain robust across default and large-batch conditions, while vanilla zero-shot training is unstable and fails after few large-batch iterations.On Europarl French→German, the proposed learning curves can reach competitive or better performance than the pivot baseline.Source: Figure 4 · p. 6, A Chain of Pivots · pp. 7–8
• In the MultiUN Russian→Chinese case study, vanilla zero-shot output eventually shifts into English, whereas LM-pretrained output is initially fluent.The vanilla model learns translation faster at first, but later produces English tokens and fully switches languages.Source: A Chain of Pivots · p. 8, Figure 5 · p. 8

Zero-shot Neural Machine Translation

• Prior zero-shot NMT studies proposed contextual parameter generation, language-agnostic encoder representations, and dual learning to improve unseen-language-pair translation.Source: §6 · p. 8
• Unlike prior zero-shot NMT work, this paper quantitatively investigates the underlying cause of degeneracy.Source: §6 · p. 8

Zero Resource Translation

• Zero-resource translation covers translation between languages without parallel resources, with pivot-based, multilingual, and unsupervised NMT as possible solutions.Source: §6 · p. 8

Unsupervised Translation

• Unsupervised translation typically studies zero-resource settings with many monolingual corpora, unlike this work's focus.Source: §6 · p. 8

Neural Machine Translation Pre-training

• NMT pre-training commonly transfers parameters from high-resource to low-resource languages, whereas the paper's language-model pre-training follows a different motivation.Source: §6 · p. 9

7 Conclusion

• Experiments on Europarl, IWSLT, and MultiUN show that the proposed methods significantly improve vanilla zero-shot NMT and consistently outperform pivot-based methods.Source: §7 · p. 9
• The analysis successfully closes the performance gap between zero-shot translation and pivot-based zero-resource translation.Source: §7 · p. 9

A.1 Trade-off between decoding speed and translation quality

• Table 3 reports decoding speed in ms/sentence and average BLEU scores for Europarl zero-shot pairs.Source: Table 3 · p. 10
• Vanilla zero-shot NMT is faster but performs worse than pivot-based methods.Source: §A.1 · p. 10
• The proposed approaches improve zero-shot NMT, outperform pivot-based translation, and use shorter decoding time, making the speed-quality trade-off irrelevant.Source: §A.1 · p. 11

A.2 Effect of Using Multi-way Data

• On the non-multi-way Europarl-c dataset, language-model pre-training consistently improves vanilla zero-shot NMT despite the performance drop caused by removing multi-way data.Source: §A.2 · p. 11, Table 5 · p. 11
• The authors conjecture that the performance drop without multi-way sentences is mainly due to reduced dataset size.Source: §A.2 · p. 11
• The proposed methods beat Chen et al. (2017) by large margins on the modified Europarl setting.Source: §A.2 · p. 11
• Europarl denotes the multi-way dataset, whereas Europarl-c denotes the non-multi-way dataset.Source: Table 4 · p. 11