Source-linked AI summary
Multi-Source Neural Translation
Barret Zoph, Kevin Knight
TL;DR
Multi-source translation asks whether multiple language inputs can reduce ambiguity beyond single-source systems. The paper trains a neural model directly on trilingual data, using separate source encoders and state-combination methods. It reports up to +4.8 Bleu over a strong single-source attention-based system, with smaller gains for more similar source languages.
Problem
Multi-source translation seeks to exploit multiple input strings because a second language can reduce ambiguity, while earlier systems relied on separately trained bilingual models and hypothesis combination.
Method
The paper trains P(e|f, g) directly on trilingual data and combines hidden and cell states from separate source encoders within a neural encoder-decoder model.
Results
+4.8 Bleu is obtained over the strongest single-source, attention-based system for French and German sources, while English-French sources yield gains up to +1.1.
Takeaways & Limitations
Improvements are best when the two source languages are more distant from each other, consistent with stronger cross-language disambiguation.
Takeaways & Limitations
Concatenating cell states and applying a linear transformation caused training to diverge because of large cell values.
Abstract
from arXiv · showhide
We build a multi-source machine translation model and train it to maximize the probability of a target English string given French and German sources. Using the neural encoder-decoder framework, we explore several combination methods and report up to +4.8 Bleu increases on top of a very strong attention-based neural translation model.
1 Introduction
Multi-source MT uses multiple languages to reduce ambiguity, extending prior hypothesis-combination approaches with a neural model trained directly on trilingual data. The paper reports positive Bleu gains, especially when the source languages are more distant.
- A second source string can reduce translation ambiguity through triangulation, such as distinguishing “bank” using German context.
- Earlier multi-source systems separately translate sources and select or merge candidate hypotheses using scores, word-level combinations, or expanded corpora.
- The paper trains P(e|f, g) directly on trilingual data and simultaneously decodes a source pair into a target string.
- +4.8 Bleu is achieved over the strongest single-source baselines, with improvements best when the source languages are more distant.
2 Multi-Source Neural MT
The model uses recurrent encoder-decoder networks with separate encoders for each source language and combines their states before decoding. Two combination methods are explored: a Basic transformation-and-sum method and a Child-Sum LSTM variant.
- Neural encoder-decoder framework: The encoder converts each source sentence into a dense vector, while the decoder converts that representation into the target sentence.
- Neural encoder-decoder framework: The four-layer LSTM encoder-decoder is trained by maximum likelihood with a softmax layer and back-propagation through time.
- Multi-source architecture: Each source language has its own encoder, and a combiner merges the encoders’ hidden and cell states into one state for the decoder.
- Basic Combination Method: The Basic method concatenates hidden states, applies Wc, passes the result through tanh, and sums the two encoder cell states.
- Basic Combination Method: The Basic method learns Wc and other weights from example string triples drawn from a trilingual training corpus.
- Basic Combination Method: Concatenating cell states followed by a linear transformation caused training to diverge because of large cell values.
- Child-Sum Method: The Child-Sum method uses an LSTM variant that computes standard gates and cell updates, while assigning separate forget gates to the encoder cell states.
I J K
The model combines information from separate source encoders and extends attention to both encoders when generating the target. Its attention mechanism predicts a source position, computes local alignment scores, and forms context vectors from weighted encoder states.
- Local attention: A local attention position is predicted before examining a source window centered on that position.The window has size 2D + 1, with D = 10.
- Context construction: Each context vector is formed as a weighted sum of source hidden states using the corresponding alignment weights.The resulting context is used to produce the final hidden state sent to the softmax layer.
- Multi-source attention: The multi-source attention model creates a context vector from each source encoder and maintains two attention variables and two alignment sets.The model uses distinct parameters for each encoder.
3 Experiments
Experiments use English, French, and German data from WMT 2014 with specified development and test subsets. For English targets, Basic combination improves BLEU over strong single-source baselines, while gains are smaller for English–French sources.
- Data and evaluation: The training data come from a subset of WMT 2014, with 3000 development sentences and a 1503-line trilingual test subset.The experiments use English, French, and German data.
- Results: +4.8 Bleu is obtained by Basic combination over the strongest single-source attention-based system for English targets from French and German.Basic combination also improves Bleu by +2.2 over the non-attention baseline.
- Results: +4.4 and +1.4 are the Child-Sum improvements over the strongest attention-based and non-attention baselines, respectively.These results use French and German sources to predict English.
- Results: Two copies of the same French input yield no BLEU improvement.This comparison tests whether the gain comes simply from duplicating a source.
- Results: Up to +1.1 BLEU gains are observed with English and French sources in the German-target experiment.The reported gains are smaller than those obtained with French and German sources for English targets.
4 Conclusion
The paper presents a multi-source neural machine translation system with novel encoder-vector combination and multi-attention methods. It reports up to +4.8 Bleu over a strong attention-based single-source baseline.
- Conclusion: +4.8 Bleu gains are reported over a very strong attention-based single-source baseline.The result is achieved with a novel encoder-vector combination method and a novel multi-attention system.