Source-linked AI summary
Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
Thanh-Le Ha, Jan Niehues, Alexander Waibel
TL;DR
Multilingual NMT lacks a simple unified way to handle many languages without extra architectural complexity and parameters. This paper introduces a universal encoder-decoder approach using language-specific coding and target forcing, achieving improvements in under-resourced translation and promising zero-resourced results.
Problem
Existing multilingual NMT approaches require multiple encoders or decoders, architecture modifications, and more free parameters, complicating standard training.
Method
The paper uses language-specific coding and target forcing to share one encoder and attention-enabled decoder across languages in a unified multilingual NMT framework.
Results
The approach improves under-resourced translation by up to 2.6 BLEU points and achieves interesting, promising results without direct parallel data between source and target languages.
Takeaways & Limitations
The framework supports many-to-many multilingual attention-based NMT without special network treatment and keeps the number of learned free parameters comparable to a single NMT system.
Takeaways & Limitations
The authors identify imbalanced data and a primitive target-forcing mechanism as issues for future work.
Abstract
from arXiv · showhide
In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework under a unified approach. We are then able to employ attention-based NMT for many-to-many multilingual translation tasks. Our approach does not require any special treatment on the network architecture and it allows us to learn minimal number of free parameters in a standard way of training. Our approach has shown its effectiveness in an under-resourced translation scenario with considerable improvements up to 2.6 BLEU points. In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages.
1 Introduction
The paper proposes a unified multilingual NMT framework that uses one encoder and decoder across languages without architecture-specific attention changes. It evaluates this approach in under-resourced and zero-resourced translation scenarios.
- 1 Introduction: Attention-based NMT is presented as an end-to-end encoder-decoder system that represents source sentences and predicts target words.The paper describes the encoder, recurrent decoder, and attention-based state of the art as background for the multilingual framework.
- 1 Introduction: Multilingual NMT can exploit information across languages, but prior systems require multiple encoders or decoders and architecture modifications for attention.These designs introduce more free parameters and make standard training more difficult than original NMT.
- 1 Introduction: The proposed framework integrates any language on either encoder-decoder side using one encoder and one decoder for all involved languages.It extends original NMT to multilingual settings without requiring network modification for attention.
- 1 Introduction: The framework is applied to under-resourced and zero-resourced translation, including settings without direct parallel data between source and target languages.The paper’s organization describes evaluation in these two scenarios after introducing the approach.
- 1 Introduction: Multilinguality improves individual translations, and the paper uses analyses of these results to motivate a fully multilingual NMT framework.This is reported as the central result of the introductory evaluation summary.
2 Neural Machine Translation: Background
Attention-based NMT encodes a source sentence into annotation vectors and uses a recurrent decoder with context vectors to generate target words. Attention computes a differentiable soft alignment over source annotations.
- Encoder: A bidirectional recurrent encoder reads each source word and forms annotation vectors by concatenating forward and backward representations.The encoder represents the source sentence through recurrent states from both directions.
- Decoder: The decoder recursively combines its previous hidden state, previous target-word representation, and a time-specific context vector to calculate the current state.The current decoder state supports generation of the translated target sentence.
- Attention: Attention selects which encoder annotation vectors contribute to predicting the next target word.The context vector is computed from encoder annotations after an attention mechanism evaluates their relevance to the previous decoder state.
- Attention: Attention normalizes relevance scores into weights and computes the context vector as a weighted sum of annotation vectors.The resulting soft alignment is differentiable and can be trained jointly with the full architecture by backpropagation.
- Challenges: NMT addresses rare and out-of-vocabulary words as a major challenge, with prior methods including source-word copying for rare target words.The cited copying heuristic helps with OOV words and named entities but cannot translate unseen words.
3 Universal Encoder and Decoder for Multilingual Neural Machine Translation
The paper proposes a unified multilingual NMT framework using one shared encoder and decoder across languages, avoiding architecture changes for attention. It combines language-specific coding with target forcing to support many-to-many translation.
- Motivation and related approaches: Previous multilingual NMT systems commonly used multiple encoders or decoders and modified attention or other architecture components to handle multiple languages.
- Core approach: The approach codes words from different languages as distinct vocabulary items and forces translation toward a representation of the desired target language.
- Core approach: One encoder and one decoder support many-to-many translation by exploiting semantic correlations across source and target languages.
- Core approach: Language-specific coding aims to learn cross-lingual word representations while distinguishing words by their language-specific surface forms.
- Core approach: Target-language information supplied during training and testing helps guide the decoder toward the desired language and limit translated candidates.
- Properties and scope: The framework uses two preprocessing steps to enable multilingual attention-based NMT with a shared encoder and decoder, conventional attention, and standard training.
4 Evaluation
The evaluation tests the unified multilingual NMT approach in under-resourced and zero-resourced translation settings. Results show gains in one under-resourced setup, while large monolingual-data training requires adaptation and zero-resourced systems underperform the pivot baseline.
- Evaluation settings: The experiments evaluate the approach on an under-resourced English→German task and a zero-resourced German→French task without direct parallel data.The study uses TED and larger multilingual or monolingual corpora, with NMT systems trained using BPE preprocessing and fixed vocabulary settings.
- Under-resourced translation: 2.6 BLEU points improvement is achieved by mix-source over the baseline on tst2013, while mix-source also gains 2.1 BLEU points on tst2014.Mix-multi-source gains 2.2 BLEU points on tst2013 and 1.6 BLEU points on tst2014, but performs below mix-source.
- Large monolingual data: +1.52 BLEU on tst2013 and +1.06 BLEU on tst2014 follow continued training on genuine parallel data after mixing nearly four million sentences dominated by monolingual data.Without adaptation, the mixed-data system performs worse than the baseline trained only on the small parallel corpus.
- Zero-resourced translation: Bridge and universal systems are evaluated against direct and pivot systems on two German→French test sets in the zero-resourced scenario.The direct system is included only as a comparison because it would not exist in a real zero-resourced setting; the pivot system translates through English.
- Zero-resourced translation: Both bridge and universal systems perform worse than the pivot system in the zero-resourced task.The paper attributes this result partly to primitive target forcing, which may fail to keep generation in the desired language after an initial language error.
5 Conclusion and Future Work
The paper presents an initial unified multilingual NMT framework that supports attention-based multilingual translation without extensive architecture redesign. It reports improvements in under-resourced translation and promising results for translation without direct source-target parallel data, while identifying data balance and target-language forcing as future challenges.
- Conclusion: The framework treats words in different languages distinctly and forces translation toward the desired target language.It supports attention-based multilingual translation without redesigning the network architecture.
- Conclusion: The network learns no more free parameters than a single NMT system.The authors present this parameter economy as part of the framework’s unified design.
- Conclusion: The approach improves individual translations in the under-resourced translation scenario.The paper describes these improvements as considerable but does not quantify them in the conclusion passage.
- Conclusion: The approach also produces promising results without direct parallel data between source and target languages.This concerns the paper’s zero-resourced translation scenario.
- Future Work: More balanced data, improved target-language forcing, and deeper strategy analyses remain future-work priorities.The authors identify these issues as areas requiring continued work.