Source-linked AI summary
Improving the Transformer Translation Model with Document-Level Context
Jiacheng Zhang, Huanbo Luan, Maosong Sun, FeiFei Zhai, Jingfang Xu, Min Zhang, Yang Liu
TL;DR
The paper addresses Transformer’s difficulty using document-level context for discourse-related translation phenomena. It adds a context encoder integrated with the encoder and decoder, trains it through a two-step strategy using sentence- and document-level corpora, and reports significant improvements over Transformer on Chinese-English and French-English tasks.
Problem
Transformer operates at the sentence level and lacks document-level context for phenomena such as coreference, lexical cohesion, and lexical disambiguation.
Method
The model uses multi-head self-attention to encode document context, integrates it into the encoder and decoder, and trains sentence-level then document-level parameters in two steps.
Results
1.96 and 0.89 BLEU points improve over Transformer on Chinese-English and French-English translation, respectively, and the approach also outperforms a Transformer-adapted cache-based method.
Takeaways & Limitations
The method improves Transformer translation on the reported Chinese-English and French-English tasks while exploiting document-level context.
Takeaways & Limitations
Large-scale document-level parallel corpora are usually unavailable, and small-data training can make document-level NMT underperform sentence-level NMT because of poor low-frequency-event estimates.
Abstract
from arXiv · showhide
Although the Transformer translation model (Vaswani et al., 2017) has achieved state-of-the-art performance in a variety of translation tasks, how to use document-level context to deal with discourse phenomena problematic for Transformer still remains a challenge. In this work, we extend the Transformer model with a new context encoder to represent document-level context, which is then incorporated into the original encoder and decoder. As large-scale document-level parallel corpora are usually not available, we introduce a two-step training method to take full advantage of abundant sentence-level parallel corpora and limited document-level parallel corpora. Experiments on the NIST Chinese-English datasets and the IWSLT French-English datasets show that our approach improves over Transformer significantly.
1 Introduction
The paper extends Transformer to use document-level context, addressing context-dependent translation phenomena while adapting training to limited document-level data. Experiments report significant gains over Transformer, with added context-capture capability and little computational slowdown.
- Transformer translates only at the sentence level, ignoring document context needed for coreference, lexical cohesion, and lexical disambiguation.
- The proposed extension represents document-level context with multi-head self-attention and incorporates it into the encoder and decoder through multi-head attention.
- The two-step training strategy first trains sentence-level parameters on larger sentence-level corpora, then estimates document-level parameters using limited document-level data.
- Multi-head attention improves the model’s capability to capture document-level context by reducing the path length between long-range dependencies.
- The newly introduced multi-head-attention modules impose no significant slowdown in training or decoding.
- 1.96 and 0.89 BLEU points improve over Transformer on Chinese-English and French-English translation, respectively, when exploiting document-level context.
2 Approach
The approach extends Transformer with a self-attentive encoder for left source-side document context, integrates that representation into the encoder and decoder, and uses context gating to limit its influence. A two-step training strategy leverages larger sentence-level corpora when document-level data are limited.
- 2.1 Problem Statement: The model omits right source-side context and target-side document context, using only left source-side context for simplicity and to avoid translation error propagation.The authors state that errors in one sentence can propagate to subsequent target sentences when target-side context is used.
- 2.2 Document-level Context Representation: The model represents left source-side document context with a self-attentive encoder built from stacked multi-head self-attention and feed-forward layers.Multi-head self-attention uses the contextual word-embedding matrix as query, key, and value, enabling long-range dependency modeling and relative location weighting.
- 2.3 Document-level Context Integration: Document-level context is incorporated into both the original Transformer encoder and decoder through multi-head attention.The encoder combines source-sentence representations with context, while the decoder integrates context alongside target-side and source-sentence representations.
- 2.3.3 Context Gating: Context gating replaces residual connections after context attention because the source sentence usually matters more for generating target words.The gating sub-layer is designed to control the contribution of document-level context rather than allowing the residual connection to increase its influence unchecked.
- 2.4 Training: The approach is designed to preserve sentence-level superiority when only small-scale document-level parallel corpora are available.The paper identifies limited document-level data as a setting in which document-level NMT can otherwise underperform because low-frequency events are poorly estimated.
- 2.4 Training: A two-step training strategy first learns original sentence-level parameters on sentence-level parallel data, then estimates newly introduced document-level parameters on document-level data while freezing the original parameters.This strategy addresses the scarcity of large-scale document-level parallel corpora and separates parameters into sentence-level and document-level subsets.
3 Experiments
The experiments evaluate the approach on Chinese-English and French-English translation using sentence- and document-level parallel data, with case-insensitive BLEU as the metric.
- Chinese-English Setup: Chinese-English training uses 2M sentence pairs, including a document-level subset of 41K documents and 940K sentence pairs.Each training document contains 22.9 sentences on average.
- French-English Setup: French-English training uses 1,824 documents containing 220K sentence pairs, with IWSLT 2010 development and test sets.The development set contains 8 documents and 887 sentence pairs, while the test set contains 11 documents and 1,664 sentence pairs.
- Evaluation and Configuration: Both translation tasks are evaluated using case-insensitive BLEU scores calculated with the multi-bleu.perl script.The original and extended Transformer models use the same reported architectural and optimization settings.
- Evaluation and Configuration: The experiments use 32K BPE merges, hidden size 512, filter size 2,048, eight attention heads, and six layers in each Transformer stack.Training uses Adam and four Tesla P40 GPUs; decoding uses beam size 4 on one Tesla P40 GPU.
3.2 Effect of Context Length
The study finds that two preceding source sentences provide the best context length, while additional context raises computational cost without improving development-set translation quality.
- Context Length: Two preceding source sentences achieve the best translation performance on the development set.The experiment defines context length as the number of preceding sentences.
- Context Length: Using more preceding sentences brings no improvement and increases computational cost.The authors therefore use two preceding sentences in subsequent experiments.
- Context Length: The result agrees with the reported finding that long-distance context has limited influence on translation.This motivates restricting the document-level context window.
- Context Length: When no preceding sentence exists, the model uses a single begin-of-sentence token.This provides the context input for documents or sentences without available preceding context.
3.4 Comparison with Previous Work
On Chinese-English and French-English translation, the proposed document-context Transformer outperforms the original Transformer and a Transformer-adapted cache method under comparable training conditions.
- Compared Methods: The comparison includes hierarchical-RNN and cache-based document-context methods for RNNsearch, alongside Transformer and its adapted cache baseline.The table reports case-insensitive BLEU scores for Chinese-English translation.
- Chinese-English Comparison: Using the same data, the approach significantly improves over the original Transformer on Chinese-English translation, with a 1.96 BLEU-point gain on the concatenated test set.The improvement is statistically significant at p < 0.01, and the “All” set excludes MT06.
- Chinese-English Comparison: The approach also significantly outperforms the cache-based method adapted for Transformer, which uses the same two-step training strategy.The adapted cache baseline is trained on the same 2M sentence pairs.
- French-English Comparison: On French-English translation, the model outperforms Transformer by 0.89 BLEU points.The comparison uses case-insensitive BLEU on the French-English task.
3.5 Subjective Evaluation
A subjective evaluation by three human evaluators compared the original Transformer with the proposed model on randomly sampled documents and generally found the proposed model preferable.
- Evaluation Protocol: Three human evaluators compared outputs from the original Transformer and the proposed model on 20 randomly sampled documents containing 198 sentences.The documents were sampled from the test sets.
- Human Judgments: On average, 19% of Transformer translations were judged better, 51% equal, and 31% worse than the proposed model.The evaluators’ judgments were generally consistent.
- Human Judgments: The subjective evaluation supports a benefit from exploiting document-level context for translation quality.This conclusion follows the authors’ interpretation of the human comparison results.
3.6 Evaluation of Efficiency
The evaluation examines efficiency and the contribution of two-step training. The approach retains broad improvements while incurring additional training and decoding cost.
- Efficiency: The original Transformer converged in about 6.7 hours, whereas the proposed model required about 7.8 hours in its second training step.
- Efficiency: Training speed decreased from 41K to 31K words/second, while decoding speed decreased from 872 to 364 words/second.The speed measurements are reported in words/second.
- Efficiency: Training speed was reduced by 25%, while the decoding gap was larger because target words are generated autoregressively in Transformer.
- Effect of Two-Step Training: Two-step training combines sentence-level and document-level parallel corpora to improve performance over using limited document-level data alone.Using only document-level data achieved 36.52 BLEU, while the two-step strategy achieved 40.22 and 47.93 BLEU in the reported settings.
3.8 Effect of Context Integration
Integrating document-level context into either Transformer component improves translation, and integrating it into both produces further improvements.
- Encoder Integration: Integrating document-level context into the encoder improved the reported BLEU score from 45.97 to 47.51.
- Combined Integration: Decoder integration was also beneficial, and combining encoder and decoder integration produced further improvements.
3.9 Effect of Context Gating
The context-gating evaluation reports an overall BLEU improvement when residual connections are replaced with context gating. An example illustrates how document context supports ambiguity resolution and lexical cohesion.
- Context Gating: Replacing residual connections with context gating produced an overall improvement of 0.38 BLEU point.
- Illustrative Example: Document context helped infer the meanings of “yundong” and “rezhong” from related words elsewhere in the document.The example uses “saiche” to infer “yundong” and the antonym “yanjuan” to interpret “rezhong.”
4 Related Work
Related work on document-level NMT has mainly focused on RNNsearch, using either full-document representations or cache-based information retrieval. This work applies the full-context strategy to Transformer and targets overall translation quality.
- Prior Document-Level NMT: Document-level NMT research has focused primarily on integrating context into the RNNsearch model.
- Approach Categories: Existing approaches either compute representations of full document context or use a cache to memorize relevant information.
- This Work: The proposed approach belongs to the full-context category, using multi-head attention to represent and integrate document-level context.
- Comparison with Transformer Work: Although Voita et al. also extended Transformer for document-level context, their work focused on anaphora resolution, whereas this model improves overall translation quality.
5 Conclusion
The method incorporates document-level context into Transformer translation and significantly improves performance on Chinese-English and French-English tasks.
- The method exploits document-level context within the Transformer translation model.
- Experiments on Chinese-English and French-English translation tasks show significant improvement over Transformer.
- Further validation on additional language pairs is planned.