Improving the Transformer Translation Model with Document-Level Context

• Transformer translates only at the sentence level, ignoring document context needed for coreference, lexical cohesion, and lexical disambiguation.Source: §1 · p. 1
• The proposed extension represents document-level context with multi-head self-attention and incorporates it into the encoder and decoder through multi-head attention.Source: §1 · p. 1
• The two-step training strategy first trains sentence-level parameters on larger sentence-level corpora, then estimates document-level parameters using limited document-level data.Source: §1 · pp. 1–2
• Multi-head attention improves the model’s capability to capture document-level context by reducing the path length between long-range dependencies.Source: §1 · pp. 1–2
• The newly introduced multi-head-attention modules impose no significant slowdown in training or decoding.Source: §1 · p. 2
• 1.96 and 0.89 BLEU points improve over Transformer on Chinese-English and French-English translation, respectively, when exploiting document-level context.Source: §1 · p. 2

2.1 Problem Statement

• The model omits right source-side context and target-side document context, using only left source-side context for simplicity and to avoid translation error propagation.The authors state that errors in one sentence can propagate to subsequent target sentences when target-side context is used.Source: §2.1 · p. 3

2.2 Document-level Context Representation

• The model represents left source-side document context with a self-attentive encoder built from stacked multi-head self-attention and feed-forward layers.Multi-head self-attention uses the contextual word-embedding matrix as query, key, and value, enabling long-range dependency modeling and relative location weighting.Source: §2.2 · p. 3

2.3 Document-level Context Integration

• Document-level context is incorporated into both the original Transformer encoder and decoder through multi-head attention.The encoder combines source-sentence representations with context, while the decoder integrates context alongside target-side and source-sentence representations.Source: §2.3 · p. 4, §2.3.1 · p. 4, §2.3.2 · p. 4

2.3.3 Context Gating

• Context gating replaces residual connections after context attention because the source sentence usually matters more for generating target words.The gating sub-layer is designed to control the contribution of document-level context rather than allowing the residual connection to increase its influence unchecked.Source: §2.3.3 · p. 5

2.4 Training

• The approach is designed to preserve sentence-level superiority when only small-scale document-level parallel corpora are available.The paper identifies limited document-level data as a setting in which document-level NMT can otherwise underperform because low-frequency events are poorly estimated.Source: §2.4 · p. 5
• A two-step training strategy first learns original sentence-level parameters on sentence-level parallel data, then estimates newly introduced document-level parameters on document-level data while freezing the original parameters.This strategy addresses the scarcity of large-scale document-level parallel corpora and separates parameters into sentence-level and document-level subsets.Source: §2.4 · p. 5

Chinese-English Setup

• Chinese-English training uses 2M sentence pairs, including a document-level subset of 41K documents and 940K sentence pairs.Each training document contains 22.9 sentences on average.Source: §3.1 · p. 5

French-English Setup

• French-English training uses 1,824 documents containing 220K sentence pairs, with IWSLT 2010 development and test sets.The development set contains 8 documents and 887 sentence pairs, while the test set contains 11 documents and 1,664 sentence pairs.Source: §3.1 · p. 5

Evaluation and Configuration

• Both translation tasks are evaluated using case-insensitive BLEU scores calculated with the multi-bleu.perl script.The original and extended Transformer models use the same reported architectural and optimization settings.Source: §3.1 · pp. 5–6
• The experiments use 32K BPE merges, hidden size 512, filter size 2,048, eight attention heads, and six layers in each Transformer stack.Training uses Adam and four Tesla P40 GPUs; decoding uses beam size 4 on one Tesla P40 GPU.Source: §3.1 · pp. 5–6

Context Length

• Two preceding source sentences achieve the best translation performance on the development set.The experiment defines context length as the number of preceding sentences.Source: §3.2 · p. 6
• Using more preceding sentences brings no improvement and increases computational cost.The authors therefore use two preceding sentences in subsequent experiments.Source: §3.2 · p. 6
• The result agrees with the reported finding that long-distance context has limited influence on translation.This motivates restricting the document-level context window.Source: §3.2 · p. 6
• When no preceding sentence exists, the model uses a single begin-of-sentence token.This provides the context input for documents or sentences without available preceding context.Source: §3.3 · p. 6

Compared Methods

• The comparison includes hierarchical-RNN and cache-based document-context methods for RNNsearch, alongside Transformer and its adapted cache baseline.The table reports case-insensitive BLEU scores for Chinese-English translation.Source: §3.4 · p. 6, Table 3 · p. 7

Chinese-English Comparison

• Using the same data, the approach significantly improves over the original Transformer on Chinese-English translation, with a 1.96 BLEU-point gain on the concatenated test set.The improvement is statistically significant at p < 0.01, and the “All” set excludes MT06.Source: §3.4 · p. 6, Table 3 · p. 7
• The approach also significantly outperforms the cache-based method adapted for Transformer, which uses the same two-step training strategy.The adapted cache baseline is trained on the same 2M sentence pairs.Source: §3.4 · p. 6

French-English Comparison

• On French-English translation, the model outperforms Transformer by 0.89 BLEU points.The comparison uses case-insensitive BLEU on the French-English task.Source: §3.4 · p. 6, Table 4 · p. 7

Evaluation Protocol

• Three human evaluators compared outputs from the original Transformer and the proposed model on 20 randomly sampled documents containing 198 sentences.The documents were sampled from the test sets.Source: §3.5 · p. 7

Human Judgments

• On average, 19% of Transformer translations were judged better, 51% equal, and 31% worse than the proposed model.The evaluators’ judgments were generally consistent.Source: §3.5 · p. 7
• The subjective evaluation supports a benefit from exploiting document-level context for translation quality.This conclusion follows the authors’ interpretation of the human comparison results.Source: §3.5 · p. 7

Efficiency

• The original Transformer converged in about 6.7 hours, whereas the proposed model required about 7.8 hours in its second training step.Source: §3.6 · p. 7
• Training speed decreased from 41K to 31K words/second, while decoding speed decreased from 872 to 364 words/second.The speed measurements are reported in words/second.Source: §3.6 · p. 7, Table 6 · p. 7
• Training speed was reduced by 25%, while the decoding gap was larger because target words are generated autoregressively in Transformer.Source: §3.6 · p. 7

Effect of Two-Step Training

• Two-step training combines sentence-level and document-level parallel corpora to improve performance over using limited document-level data alone.Using only document-level data achieved 36.52 BLEU, while the two-step strategy achieved 40.22 and 47.93 BLEU in the reported settings.Source: §3.7 · pp. 7–8

Encoder Integration

• Integrating document-level context into the encoder improved the reported BLEU score from 45.97 to 47.51.Source: §3.8 · p. 8

Combined Integration

• Decoder integration was also beneficial, and combining encoder and decoder integration produced further improvements.Source: §3.8 · p. 8

Context Gating

• Replacing residual connections with context gating produced an overall improvement of 0.38 BLEU point.Source: §3.9 · p. 8

Illustrative Example

• Document context helped infer the meanings of “yundong” and “rezhong” from related words elsewhere in the document.The example uses “saiche” to infer “yundong” and the antonym “yanjuan” to interpret “rezhong.”Source: §3.10 · p. 8

Prior Document-Level NMT

• Document-level NMT research has focused primarily on integrating context into the RNNsearch model.Source: §4 · p. 8

Approach Categories

• Existing approaches either compute representations of full document context or use a cache to memorize relevant information.Source: §4 · p. 9

This Work

• The proposed approach belongs to the full-context category, using multi-head attention to represent and integrate document-level context.Source: §4 · p. 9

Comparison with Transformer Work

• Although Voita et al. also extended Transformer for document-level context, their work focused on anaphora resolution, whereas this model improves overall translation quality.Source: §4 · p. 9

• The method exploits document-level context within the Transformer translation model.Source: §5 · p. 9
• Experiments on Chinese-English and French-English translation tasks show significant improvement over Transformer.Source: §5 · p. 9
• Further validation on additional language pairs is planned.Source: §5 · p. 9