Source-linked AI summary
Does Neural Machine Translation Benefit from Larger Context?
Sebastien Jean, Stanislas Lauly, Orhan Firat, Kyunghyun Cho
TL;DR
The paper asks whether neural machine translation benefits from modeling text surrounding the current source sentence. It adds context-aware encoding and attention, finding better translation and pronoun-prediction performance on smaller corpora, while the benefit largely disappears with larger training data.
Problem
The impact of larger context on average translation quality is unknown, although it may capture discourse properties such as coherence and anaphora.
Method
The paper extends attention-based neural machine translation with encoders and attention mechanisms for surrounding source sentences, evaluating translation and cross-lingual pronoun prediction.
Results
Larger-context models generally outperform vanilla models on BLEU, RIBES, and macro-average recall for smaller training corpora, but the improvement vanishes as training data grows.
Takeaways & Limitations
Larger context provides a moderate benefit when training data are limited, while large corpora may let models capture subtle word relations from the source sentence alone.
Abstract
from arXiv · showhide
We propose a neural machine translation architecture that models the surrounding text in addition to the source sentence. These models lead to better performance, both in terms of general translation quality and pronoun prediction, when trained on small corpora, although this improvement largely disappears when trained with a larger corpus. We also discover that attention-based neural machine translation is well suited for pronoun prediction and compares favorably with other approaches that were specifically designed for this task.
1 Introduction
The paper investigates whether adding surrounding source sentences improves neural machine translation, especially translation quality and discourse-sensitive pronoun prediction. It introduces a larger-context attention-based model and evaluates it against a source-sentence-only baseline.
- 1 Introduction: Neural machine translation can integrate information beyond source and target tokens when that information is projected into a vector space.This flexibility has supported multilingual, multimodal, and syntax-aware translation extensions.
- 1 Introduction: The proposed model incorporates preceding and/or following source sentences into attention-based neural machine translation.It models p(Y | X, X−n, ..., X−1, X1, ..., Xn) rather than only p(Y | X).
- 1 Introduction: The study evaluates larger-context models using BLEU, RIBES, and cross-lingual pronoun prediction, alongside WMT’16 shared-task systems.
Neural Machine Translation
The baseline system encodes the source sentence, attends to its annotation vectors, and decodes target symbols recurrently. The larger-context extension adds a context encoder and attention mechanism whose context vector influences decoder transitions.
- 2.1 Attention-based Neural Machine Translation: The baseline attention-based system uses an encoder, decoder, and attention model to convert source annotations into target-symbol predictions.The encoder produces annotation vectors, while attention forms a time-dependent source vector for the decoder.
- 2.1 Attention-based Neural Machine Translation: Attention scores use the previous target symbol, previous decoder state, and an annotation vector to select source information relevant to each decoding step.The resulting source vector is a weighted sum of annotation vectors.
- 2.1 Attention-based Neural Machine Translation: The complete encoder-decoder-attention network is differentiable and jointly trained by maximizing corpus log-likelihood with stochastic gradient descent and backpropagation through time.
- 2.2 Larger-Context Neural Machine Translation: The larger-context model adds a bidirectional encoder and attention mechanism for a preceding source sentence, then incorporates its context vector into decoding.The method can be extended to multiple preceding or following sentences using additional encoder-attention sets or concatenation.
3 Evaluating Larger-Context Neural Machine Translation
The evaluation compares vanilla and larger-context neural machine translation on average translation quality and cross-lingual pronoun prediction. It examines whether surrounding context improves these outcomes and uses pronoun prediction as a targeted context-sensitive task.
- Larger context is motivated by its potential to capture style, genre, topical patterns, discourse coherence, and anaphora beyond a single source sentence.
- Cross-lingual pronoun prediction provides an established task for evaluating whether translation models incorporate larger-context information.
- The study compares vanilla and larger-context models using BLEU for translation quality and macro-averaged recall for pronoun prediction.
- Unlike specialized pronoun-prediction systems, these models are trained to maximize average translation quality and then used for pronoun prediction.
4 Experimental Settings
The experiments use English–French and English–German data for translation and cross-lingual pronoun prediction, with additional IWSLT English–German evaluation. They compare naive and larger-context attention-based models across training-corpus sizes.
- The experiments use En-Fr and En-De corpora, with macro-average recall as the main pronoun-prediction metric and millions of training sentence pairs.
- Pronoun prediction replaces target pronouns with REPLACE and selects the pronoun combination maximizing log-probability, exhaustively testing combinations when needed.
- Translation receives only the source sentence and uses beam search to approximate the maximum-a-posteriori translation.
- The study additionally evaluates average translation quality on IWSLT’15 En-De, using IWSLT’12 for development and IWSLT’14 for testing.
- Larger-context models generally outperform vanilla models on BLEU, RIBES, and macro-average recall, but the improvement vanishes as training data grows.
5 Conclusion
The paper extends attention-based neural machine translation with surrounding-sentence context and evaluates its effect on translation quality and pronoun prediction. Benefits are moderate with few training pairs and are not similarly observed with larger corpora.
- The proposed architecture incorporates context from surrounding sentences into attention-based neural machine translation.
- Benefits from larger context are moderate with few training sentence pairs and do not persist at the same level with larger training corpora.
- The authors suspect that large corpora let models capture subtle word relations from the source sentence alone.
- A more focused evaluation metric may be needed to properly assess discourse-level information in translation.