Source-linked AI summary
Context-Aware Neural Machine Translation Learns Anaphora Resolution
Elena Voita, Pavel Serdyukov, Rico Sennrich, Ivan Titov
TL;DR
Sentence-isolated machine translation ignores discourse context that can help resolve ambiguity and improve coherence. The paper introduces a controllable, interpretable context-aware NMT model, finding that it captures pronoun translation and implicitly models anaphora, while improving overall BLEU over both context-agnostic and simple concatenation baselines.
Problem
Machine translation systems typically process sentences in isolation, while it remains unclear which discourse phenomena context-aware NMT systems handle and how they model them.
Method
The paper modifies a Transformer to encode source and context sentences independently, using an attention layer and gating function to control contextual information flow.
Results
The model improves overall BLEU over context-agnostic and simple context-aware baselines, with especially prominent gains for ambiguous-pronoun sentences, and its attention aligns with anaphora evidence.
Takeaways & Limitations
Context-aware NMT can implicitly capture coreference phenomena while improving translation quality, particularly for pronoun translation requiring grammatical gender agreement.
Takeaways & Limitations
The analysis focuses on pronoun translation, while other discourse phenomena remain potential subjects for future investigation; the Transformer setting also limits the effectiveness of earlier recurrent-encoder techniques.
Abstract
from arXiv · showhide
Standard machine translation systems process sentences in isolation and hence ignore extra-sentential information, even though extended context can both prevent mistakes in ambiguous cases and improve translation coherence. We introduce a context-aware neural machine translation model designed in such way that the flow of information from the extended context to the translation model can be controlled and analyzed. We experiment with an English-Russian subtitles dataset, and observe that much of what is captured by our model deals with improving pronoun translation. We measure correspondences between induced attention distributions and coreference relations and observe that the model implicitly captures anaphora. It is consistent with gains for sentences where pronouns need to be gendered in translation. Beside improvements in anaphoric cases, the model also improves in overall BLEU, both over its context-agnostic version (+0.7) and over simple concatenation of the context and source sentences (+0.6).
1 Introduction
The paper addresses discourse information that sentence-isolated translation systems ignore by introducing an interpretable context-aware NMT model. It analyzes the model’s contextual information flow and finds pronoun translation and implicit anaphora capture to be central.
- Machine translation systems typically translate sentences in isolation, despite extended context’s potential benefits for ambiguous cases and translation coherence.
- The paper identifies unclear modeling of discourse phenomena in context-aware NMT as a gap relevant to designing future architectures and linguistic features.
- The proposed model controls and interprets information flow between extended context and the translation model.
- Pronoun translation is the key contextual phenomenon captured by the model, particularly when translating from English without grammatical gender into Russian with grammatical gender.
- Comparisons with automatic and human coreference annotations indicate that the model implicitly captures anaphora without specialized task features.
2 Neural Machine Translation
This section introduces the encoder-decoder and attention-based foundations of NMT, then describes the Transformer’s self-attentive architecture and practical training advantages.
- NMT encoders map source symbol sequences to distributed representations, while decoders generate target symbols one element at a time.
- Attention-based NMT: Attention mechanisms selectively focus on source positions, with attention weights determining how information from those positions is combined.
- Transformer: The Transformer removes recurrence and uses stacked self-attention and fully connected layers in both encoder and decoder.
- Transformer: The Transformer is more parallelizable and faster to train than recurrent encoder-decoder models, motivating its use as the experimental baseline.
- Transformer: Each Transformer encoder layer contains multi-head attention and a feed-forward network, while decoder layers add attention over the corresponding encoder output.
- Transformer: The decoder masks attention to prevent access to future target positions and illegal leftward information flow.
3 Context-aware model architecture
The discourse-aware model preserves the Transformer decoder while adding a controlled context pathway to the encoder. Source and context representations interact through attention and a gated combination.
- The model leaves the Transformer decoder intact and incorporates contextual information on the encoder side.
- The first N −1 source-encoder layers retain the original Transformer structure, while the last layer incorporates contextual information.
- The final source-encoder layer combines source self-attention with multi-head attention over the context encoder output.
- A gated sum combines the source-attention and context-attention outputs to form the contextualized source representation.
- The context encoder replicates the Transformer encoder and shares the first N −1 layers’ parameters with the source encoder.
- A special <bos> token is added only to context sentences so shared layers can distinguish context from source inputs.
4 Experiments
The experiments use cleaned English-Russian OpenSubtitles2018 data, with randomly selected training, development, and test instances and byte-pair encoded vocabularies.
- The experiments use English and Russian from OpenSubtitles2018, selecting 2 million training instances and two 10000-instance development and test subsets from unseen movies.
- The data are cleaned and encoded with byte-pair encoding using source and target vocabularies of about 32000 tokens.
5 Results and analysis
The context-aware model improves overall translation and concentrates useful context attention on linguistically ambiguous cases, especially pronouns requiring gender or number resolution in Russian. Attention patterns also align with discourse structure and coreference, although attention usage does not translate into uniformly larger BLEU gains for shorter sentences.
- 5.1 Overall performance: 0.7 BLEU improvement over the discourse-agnostic model was achieved by the best model using a context encoder for the previous sentence.The next sentence did not appear beneficial, so subsequent experiments focused on the previous sentence.
- 5.1 Overall performance: The context-aware model appears more accurate than the concatenation baseline and offers a more analyzable interface.The comparison suggests potential effectiveness beyond simply concatenating context and source sentences.
- 5.2 Context attention: Contextual attention is high for ambiguous Russian translations of “it”, “yours”, “ones”, “you” and “I”.For example, English “it” can map to masculine, feminine, neuter, or plural Russian forms depending on its antecedent.
- 5.2 Context attention: Context attention is disproportionately high for short sentences and positively correlated with context length, while sentence-initial source words attend more to context.The analysis used average attention mass to context words and examined source-token positions.
- 5.2 Context attention: There is no straightforward dependence of BLEU improvement on source length, despite higher contextual attention for shorter sentences.Context therefore does not appear disproportionately more useful for short sentences.
- 5.3.1 Ambiguous pronouns and translation quality: 4-5 BLEU improvement occurs when “it” is translated into a feminine or plural pronoun by the reference.The improvement is smaller for masculine translations because masculine gender is more frequent and favored by the context-agnostic baseline.
6 Related work
Prior work used external systems or engineered features for discourse phenomena, while this work studies whether context-aware NMT can learn coreference implicitly through attention.
- Earlier statistical translation systems relied on external coreference resolution, whose low performance limited this approach.
- This model learns coreference phenomena without feature engineering, extending prior attention-based analyses of contextual history.
7 Conclusions
The paper concludes that its Transformer-based context-aware NMT system improves translation quality and especially helps with ambiguous pronouns while inducing anaphora relations.
- The Transformer-based context-aware NMT system outperforms both context-agnostic and simple context-aware baselines on an English-Russian parallel corpus.
- Improvements are especially prominent for sentences containing ambiguous pronouns.
- The model induces anaphora relations, while future work could examine other discourse phenomena and specialized attention features.
A.1 Data preprocessing
The study uses English-Russian OpenSubtitles2018 data filtered for alignment quality and temporally nearby context, with byte-pair encoded vocabularies and length-based batching.
- The dataset is the publicly available OpenSubtitles2018 corpus for English and Russian.
- Sentence pairs require at least 0.9 overlap, and context is the previous sentence when timestamps differ by no more than 7 seconds.
- Sentences use byte-pair encoding with source and target vocabularies of about 32000 tokens, while batches contain approximately 5000 source tokens.
A.2 Model parameters
The model follows the Transformer base configuration, using six encoder and decoder layers, eight attention heads, and specified representation and feed-forward dimensions.
- The Transformer configuration uses N = 6 layers in both the encoder and decoder.
- It employs h = 8 parallel attention layers, or heads.
- The input and output dimensionality is dmodel = 512, and the feed-forward inner-layer dimensionality is dff = 2048.
A.3 Optimizer
The optimizer uses Adam with specified hyperparameters and varies the learning rate during training using a schedule with 4000 warmup steps.
- Adam uses β1 = 0.9, β2 = 0.98, and ε = 109.
- The learning rate varies over the course of training according to a formula.
- Training uses 4000 warmup steps.