Source-linked AI summary
Neural Machine Translation with Extended Context
Jörg Tiedemann, Yves Scherrer
TL;DR
The paper asks whether neural machine translation can use discourse context beyond individual translation units without explicit guidance. It tests simple extended-source and bilingual-context models on German-English subtitles, finding robust translation quality and attention patterns linked to cross-sentential coherence in selected cases. The evidence remains preliminary and largely anecdotal.
Problem
The paper investigates whether neural machine translation can discover cross-sentential dependencies without explicit annotation or guidance.
Method
The study extends context during training and translation without changing the model, using either previous source-language units or larger bilingual translation units.
Results
The models handle larger units without harming translation performance and learn to distinguish information from different segments or discourse history.
Takeaways & Limitations
The experiments provide selected examples of referential expressions handled across sentence boundaries and cross-sentential attention patterns associated with improved textual coherence.
Takeaways & Limitations
The evidence is still rather anecdotal, requiring more systematic experiments, detailed analyses, and evaluations.
Abstract
from arXiv · showhide
We investigate the use of extended context in attention-based neural machine translation. We base our experiments on translated movie subtitles and discuss the effect of increasing the segments beyond single translation units. We study the use of extended source language context as well as bilingual context extensions. The models learn to distinguish between information from different segments and are surprisingly robust with respect to translation quality. In this pilot study, we observe interesting cross-sentential attention patterns that improve textual coherence in translation at least in some selected cases.
1 Introduction
Prior discourse-aware machine translation efforts addressed coherence and related phenomena but did not significantly improve translation quality. This motivates testing whether neural machine translation can learn cross-sentential dependencies through extended context without explicit guidance.
- Prior work: Earlier studies targeted textual coherence, document-wide consistency, referential elements, and discourse connectives in machine translation.These efforts included pronominal anaphora, terminology consistency, and discourse-level connective generation.
- Motivation: Overall, existing ideas did not lead to significant improvements in translation quality.
- Motivation: The paper presents a pilot study of simple context extensions within standard attention-based encoder-decoder models.Its purpose is to test whether NMT can discover cross-sentential dependencies without explicit annotation or guidance.
2 Attention-Based NMT
Attention-based encoder-decoder NMT represents source sentences with recurrent networks and uses attention to combine information across encoding positions. This mechanism supports studying whether models can exploit context beyond sentence boundaries.
- Model architecture: Attention-based encoder-decoder models encode a source sentence into a distributed representation and decode it into the target language with another recurrent network.
- Attention mechanism: Attention weights specify how information from different source positions is combined during decoding.
- Attention mechanism: The framework can handle arbitrarily long sequences without limiting the capacity of the internal representation.
- Attention mechanism: Its attention distributions can expose intuitively plausible connections between source and target language.
3 Data Sets
The experiments use German-English movie-subtitle data because subtitles provide abundant training material and short translation units suited to adding context beyond sentence boundaries. The corpus is drawn from OpenSubtitles2016 and contains 13.9 million translation units.
- Corpus choice: The experiments translate German movie subtitles into English, with the language choice motivated partly by easier qualitative inspection.German-English translation also includes discourse phenomena such as referential pronouns requiring grammatical agreement.
- Corpus choice: Movie subtitles provide large quantities of training data and unusually short translation units for extending context beyond sentence boundaries.Short segments are also important because neural MT struggles with long sequences.
- Corpus construction: Aligned translation units average about 8 tokens per language and may contain one or more sentences or sentence fragments.
- Corpus construction: 13.9 million translation units comprise the final OpenSubtitles2016 corpus after reserving 400 movies for development and testing.
4 Extended Context Models
The paper extends context during training and translation without changing the NMT architecture, using either previous source-language units or larger bilingual translation units. The models are designed to learn which contextual information influences the focus sentence and to distinguish dependencies across segments.
- 4 Extended Context Models: The approach extends context in training and translation while leaving the model architecture unchanged.The training procedures are expected to discover what information is needed for translation.
- Extended source: Extended source context adds the previous source-language unit to the focus sentence and marks contextual tokens with a special prefix.A variant instead inserts sentence-break tokens without distinguishing contextual from sentence-internal words.
- Learning contextual influence: The central task is learning how specific context word sequences influence translation of the focus sentence.Ambiguous German pronouns such as “sie” provide a test of whether discourse information is used for cross-sentential decisions.
- Extended translation units: Extended translation units concatenate the previous source and target units, requiring the decoder to generate larger bilingual segments.Segment boundaries are marked with BREAK tokens, and the design may capture target-language dependencies such as grammatical agreement.
- Implementation: Subword-based models use BPE, with separate vocabulary settings for extended-source models and the 2+2 bilingual model.
5 Experiments and Results
Extended-context models preserve translation quality while learning selective cross-segment attention. The analyses reveal coherence-related patterns, especially for pronouns and dialogue context, but also expose limits from narrow context windows and ambiguous cases.
- All models achieve similar translation quality, with slightly higher BLEU for 2+1 with sentence breaks and slightly higher chrF3 for sentence-break variants.The differences are small, but larger units do not harm performance.
- 5.1 2+1: Extended Source Language Context: The 2+1 model learns to ignore most contextual information while still using relevant history to produce adequate translations.Attention must distinguish useful context from substantial extra source information.
- 5.1 2+1: Extended Source Language Context: Pronouns rise in the external-attention rankings when attention peaks are measured, suggesting more focused attention to these entities.This supports targeted analysis of cross-lingually ambiguous pronouns, although the authors leave systematic investigation for future work.
- 5.2 2+2: Larger Translation Units: The 2+2 model merges information across segments, directs beginning-of-sentence items toward previous sentences, and directs final items toward following sentences.It also uses segment-initial tokens and punctuation when generating segment-break symbols.
6 Conclusions and Future Work
The study presents two simple extended-context neural MT models and finds that neural MT can use wider context while distinguishing information from different segments. German-English subtitle experiments provide encouraging examples, but the evidence remains largely anecdotal and motivates more systematic evaluation.
- Two models extend neural MT context by adding source-language history or concatenating subsequent training segments.
- Neural MT distinguishes information from different segments or discourse history when translating with wider context.
- German-English subtitle experiments include examples where referential expressions across sentence boundaries are handled properly.
- Evidence so far is rather anecdotal, so future work will conduct more systematic experiments with detailed analyses and evaluations.