PaperlayerView exact revision on arXiv

Source-linked AI summary

The Missing Ingredient in Zero-Shot Neural Machine Translation

Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, Wolfgang Macherey

arXiv:1903.07091v1cs.CLcs.AIcs.LG

TL;DR

Zero-shot multilingual NMT remains weaker than pivoting because parameter sharing does not ensure language-invariant representations. The paper adds auxiliary encoder alignment losses, treating English as a source domain, and achieves pivot-level WMT14 zero-shot performance without meaningful supervised regression while scaling to more languages.

  • Problem

    Zero-shot translation between language pairs unseen during training consistently lags pivoting, reflecting inadequate language-invariant representations despite parameter sharing.

    Source: §1 · p. 1, §3.4 · p. 5

  • Method

    The method applies auxiliary encoder losses that explicitly align representations across languages, using English as a source domain and non-English languages as a target domain.

    Source: §4 · p. 5, §4.2 · p. 6

  • Results

    On WMT14 English-German-French, zero-shot performance reaches parity with pivoting without meaningful regression on supervised directions, and the approach scales to IWSLT17.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Explicitly incentivizing cross-lingual transfer provides a simple alternative to pivoting and quadratic data-synthesis pipelines for multilingual zero-shot translation.

    Source: §1 · p. 2, §5.4 · p. 8, §6 · p. 8

    Adversarial alignment is sensitive to initialization and hyperparameters and can suffer unstable failure modes, whereas the paired cosine loss uses one alignment-weight hyperparameter.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Multilingual Neural Machine Translation (NMT) models are capable of translating between multiple source and target languages. Despite various approaches to train such models, they have difficulty with zero-shot translation: translating between language pairs that were not together seen during training. In this paper we first diagnose why state-of-the-art multilingual NMT models that rely purely on parameter sharing, fail to generalize to unseen language pairs. We then propose auxiliary losses on the NMT encoder that impose representational invariance across languages. Our simple approach vastly improves zero-shot translation quality without regressing on supervised directions. For the first time, on WMT14 English-FrenchGerman, we achieve zero-shot performance that is on par with pivoting. We also demonstrate the easy scalability of our approach to multiple languages on the IWSLT 2017 shared task.

1 Introduction

Zero-shot multilingual NMT lags pivoting because parameter sharing alone does not guarantee language-invariant representations. The paper addresses this by explicitly aligning encoder representations, reaching pivot-level zero-shot quality without meaningful supervised regression.

  • Zero-shot directions trail pivoting by 2-10 BLEU points despite strong supervised translation quality.Source: §1 · p. 1
  • Pivoting avoids the quality gap but doubles latency and compounds errors, while existing one-step data-augmentation methods require multiple phases and scale quadratically with languages.Source: §1 · p. 1
  • Parameter sharing alone does not reliably produce language-invariant features, causing multilingual models to fail on zero-shot directions.Source: §1 · p. 1
  • Auxiliary encoder losses explicitly incentivize source-language-invariant representations by treating English as a source domain and other languages as a target domain.Source: §1 · pp. 1–2
  • On WMT14 English-German-French, the method reaches zero-shot performance on par with pivoting without meaningful supervised regression and scales to more languages on IWSLT17.Source: §1 · p. 2

2 Related Work

Related work improves zero-shot translation through architecture changes, language-invariance objectives, and synthetic supervision, but important quality or scalability gaps remain. These studies motivate directly aligning multilingual representations without quadratic data synthesis.

  • Early multilingual models demonstrated zero-shot transfer, but zero-shot quality remained below pivoting, including a 6 BLEU-point Portuguese-to-Spanish deficit.Source: §2 · p. 2
  • Language-bias remedies improved zero-shot quality but still remained 4-5 BLEU points behind pivoting.Source: §2 · p. 2
  • Architectural approaches using selective sharing, contextual parameter generation, or interlingua layers still lagged pivoting by 2-10 BLEU points, suggesting parameter sharing alone is insufficient.Source: §2 · p. 2
  • Language-invariance research aligns representations through parallel-data objectives or feature-distribution discrepancy, including adversarial encoder losses for cross-lingual transfer.Source: §2 · pp. 2–3
  • Zero-resource methods can perform strongly through pseudo-parallel data, but their teacher-training, synthesis, and multilingual-training pipeline scales quadratically with language count.Source: §2 · p. 3

3 An Error Analysis of Zero-Shot Translation

The error analysis evaluates multilingual NMT on WMT14 and finds that zero-shot failure reflects both wrong-language decoding and inadequate cross-lingual generalization. Standard parameter sharing and capacity constraints do not directly enforce invariant representations.

3. An Error Analysis of Zero-Shot Translation

  • The study compares bilingual and multilingual Transformer models on WMT14 English-German-French data, using 158M merged sentence pairs with shared 32k BPE vocabulary.Source: §3 · p. 3, §3.1.1 · p. 3, §3.1.2 · p. 3

3.2 Baseline Result

  • Multilingual models perform well on supervised directions, with an expected 1-2 BLEU-point regression versus one-to-one models from sharing capacity across tasks.Source: §3.2 · pp. 3–4
  • Zero-shot translation is far below pivoting and therefore unusable, despite the multilingual model's ability to translate directions absent from training.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Target Language is Entangled with Source Language

  • A significant fraction of zero-shot outputs use the wrong language or copy the source because training never teaches the model to attribute target language correctly to the target-language token.Source: §3.3 · p. 4, Table 2 · p. 4

3.4 Problems with Cross-lingual Generalization

  • After restricting evaluation to outputs in the correct language, zero-shot translation still trails pivoting by 0.5 BLEU for French-to-German and 2.7 BLEU for German-to-French.Source: §3.4 · p. 4, Table 3 · p. 4
  • Increasing dropout from 0.1 to 0.3 improves zero-shot performance but overregularizes supervised directions, exposing the trade-off of capacity-based remedies.Source: §3.4 · p. 4
  • End-to-end translation training can partition capacity by language instead of learning invariant encoder features, so direct incentives for transferable representations are needed.Source: §3.4 · p. 5, Figure 1 · p. 5

4 Aligning Latent Representations

The paper frames zero-shot translation as domain adaptation and adds encoder losses that make representations invariant across source languages. It uses distribution-level and instance-level alignment so English representations act as an implicit latent-space pivot.

4 Aligning Latent Representations

  • Aligning encoder representations to English treats multilingual NMT as a two-domain adaptation problem, with English as the source domain and all non-English languages as the target domain.The regularizer encourages non-English sentence representations to resemble their English counterparts, making the encoder source-language agnostic.Source: §4 · p. 5
  • The total training objective combines the cross-entropy translation loss with an alignment regularizer weighted by λ.The regularizer is applied to encoder representations because the encoder maps source text into the latent space consumed by the decoder.Source: §4 · p. 5

4.1 Aligning Distributions

  • Adversarial alignment minimizes discrepancy between English and non-English encoder-representation distributions using a binary discriminator independent of the number of languages.The discriminator operates on temporally max-pooled encoder outputs; the authors also distinguish this approach from language-specific adversarial alignment.Source: §4.1 · pp. 5–6

4.2 Aligning Known Translation Pairs

  • Instance-level alignment uses parallel translations to make semantically equivalent sentences have similar encoder representations rather than only matching domain distributions.Because word alignments are unavailable, the method uses pooled representations under a bag-of-words assumption and minimizes cosine distance; max pooling works well empirically.Source: §4.2 · p. 6

5 Experiments

Across WMT14 and IWSLT17, encoder alignment losses improve zero-shot translation, match pivoting, preserve supervised quality, and scale to more languages. Analyses indicate that explicit representation alignment produces more language-agnostic and functionally equivalent representations, while cosine alignment is simpler than adversarial adaptation and zero-resource synthesis.

5.1 Zero-Shot Now Matches Pivoting

  • Zero-shot translation matches pivoting on WMT14 en-de-fr and does so without meaningful regression on supervised directions.The aligned models close the gap to pivoting in both zero-shot directions; Table 4 reports the comparison.Source: §5.1 · p. 7, Table 4 · p. 7

5.2 Quantifying the Improvement to Language Invariance

  • Alignment losses make encoder representations more language-agnostic because baseline alignment arrests without an external incentive.The analysis compares mean cosine distances between context vectors for 100 parallel en-de-fr sentences during training.Source: §5.2 · p. 7

5.3 Cosine vs Adversarial

  • Cosine alignment achieves nearly indistinguishable quality from adversarial adaptation while using a simpler and more robust objective with only λ as a hyperparameter.Adversarial training is sensitive to initialization and hyperparameters and can suffer unstable behavior, whereas cosine alignment directly matches translation pairs.Source: §5.3 · pp. 7–8

5.4 Zero-Shot with Adaptation vs Zero-Resource

  • Zero-resource synthesis reaches BLEU 29.04 for de →fr and 21.66 for fr →de, but its sequential, quadratic process is unsuitable for many-language multilingual settings.The synthesized-data systems are one-to-one models, giving them a single-task focus advantage; the proposed approach avoids extra preprocessing, training phases, and data generation.Source: §5.4 · p. 8

5.5 IWSLT17: Scaling to more languages

  • On IWSLT17, cosine alignment matches bridging quality across five languages, although the baseline already performs well on zero-shot directions.The dataset uses English-only training directions and evaluates multilingual translation with zero-shot results marked in Table 5.Source: §5.5 · p. 8, Table 5 · p. 8

6 Conclusion

The paper diagnoses why naïvely trained multilingual NMT models fail to generalize to zero-shot directions and introduces auxiliary losses that enforce source-language-invariant representations. Across WMT English-French-German and IWSLT 2017, the approach improves zero-shot performance while maintaining supervised-direction quality.

6 Conclusion

  • Auxiliary losses force source-language-invariant representations that improve zero-shot generalization while maintaining supervised-direction performance.The approach is demonstrated on the WMT English-French-German and IWSLT 2017 benchmarks.Source: §6 · p. 8

Related papers

Loading 1903.07091v1…