Source-linked AI summary
Exploiting Cross-Sentence Context for Neural Machine Translation
Longyue Wang, Zhaopeng Tu, Andy Way, Qun Liu
TL;DR
The paper addresses ambiguity and document-level inconsistency that arise when NMT translates sentences independently. It summarizes previous source sentences hierarchically and integrates that history through state initialization and auxiliary decoding context. The combined approach significantly outperforms a strong attention-based baseline, by up to +2.1 BLEU points.
Problem
Sentence-level NMT has difficulty resolving ambiguity and maintaining consistent translations of repeated terms across documents.
Method
A hierarchical RNN summarizes previous source sentences, then supplies the history through encoder-decoder warm-start initialization or auxiliary context for decoder updates.
Results
+2.1 BLEU points over NEMATUS is achieved by combining the best initialization and auxiliary context variants.
Takeaways & Limitations
Initialization and auxiliary-context strategies are complementary, while a context gate helps distinguish the global-context needs of generated target words.
Takeaways & Limitations
The models use source-side history and leave full-length documents with additional linguistic features, such as discourse relations, for future work.
Abstract
from arXiv · showhide
In translation, considering the document as a whole can help to resolve ambiguities and inconsistencies. In this paper, we propose a cross-sentence context-aware approach and investigate the influence of historical contextual information on the performance of neural machine translation (NMT). First, this history is summarized in a hierarchical way. We then integrate the historical representation into NMT in two strategies: 1) a warm-start of encoder and decoder states, and 2) an auxiliary context source for updating decoder states. Experimental results on a large Chinese-English translation task show that our approach significantly improves upon a strong attention-based NMT system by up to +2.1 BLEU points.
1 Introduction
Sentence-level NMT struggles with ambiguity and consistency because it usually translates each sentence independently. The paper therefore models previous source sentences as cross-sentence context and integrates that history into NMT through several strategies.
- Motivation: Sentence-only NMT struggles with ambiguous words and inconsistent translations of repeated terms across documents.These problems are difficult to alleviate using limited intra-sentence context.
- Motivation: Cross-sentence context has helped sequential tasks but received relatively little attention in NMT.The proposed model considers previous source sentences in the same document.
- Approach: A hierarchy of RNNs summarizes previous source sentences by adding a document-level RNN above the sentence-level encoder.The resulting representation captures historical context for translating the current sentence.
- Integration strategies: Initialization warm-starts the encoder, decoder, or both with the history representation.This strategy uses global context to initialize NMT states before current-sentence decoding.
- Integration strategies: Auxiliary Context combines static cross-sentence history with dynamic intra-sentence attention context during decoding.Gating Auxiliary Context additionally controls how much global context contributes to generating each target word.
- Results: The proposed mechanisms each improve translation performance, and combining them achieves further improvement.The experiments report significant gains over a strong attention-based NMT system.
2 Approach
The approach summarizes previous source sentences hierarchically, then integrates the resulting global context into NMT through initialization, auxiliary-context, and gated auxiliary-context strategies.
- Global context: The model represents the K previous source sentences as cross-sentence context C before integrating it into NMT.C consists of the previous sentences in the same document preceding the sentence being translated.
- Hierarchical context summarization: A sentence-level RNN reads each previous sentence sequentially, using its last hidden state as the sentence representation S_k.The last state stores order-sensitive information about all words in the sentence.
- Hierarchical context summarization: A document-level RNN processes the sequence of sentence representations and uses its last hidden state D as the global-context summary.This hierarchical design first summarizes each sentence, then summarizes the processed sentence sequence.
- Context integration: Initialization uses D to initialize the NMT encoder, decoder, or both instead of relying only on standard initial states.For the decoder, D is added to the transformed encoder final state when computing the initial hidden state.
- Context integration: Auxiliary Context adds D alongside the intra-sentence context c_i to jointly update each decoder state.The combined representation [c_i, D] supplies both intra- and cross-sentence information to the decoder.
- Context integration: Gating Auxiliary Context applies a context gate that dynamically controls how much global-context information flows at each decoding step.The gate produces an element-wise weight for D using the decoding environment and a sigmoid layer.
3 Experiments
Experiments on Chinese–English translation evaluate cross-sentence context methods against the attention-based NEMATUS baseline. Initialization and auxiliary-context strategies each improve BLEU, their combination reaches +2.1 BLEU, and error analysis shows both corrections and new errors.
- 3.1 Setup: The experiments use 1M Chinese–English sentence pairs from document-boundary LDC corpora, with MT05 for development and MT06/MT08 for testing.Case-insensitive BLEU and sign-tests assess translation quality and significance.
- 3.2 Results: NEMATUS significantly outperforms Moses by 2.3 BLEU points on average, establishing a strong NMT baseline.The comparison uses a commonly used phrase-based SMT system.
- 3.2 Results: Initenc and Initdec improve translation performance by around +1.0 and +1.3 BLEU points individually, while combining them yields further improvement.These strategies warm-start encoder and decoder states with cross-sentence context.
- 3.2 Results: Gating auxiliary context improves performance by around +1.0 BLEU point over its non-gating counterpart.The gate learns when to use global context for generating target words.
- 3.2 Results: +2.1 BLEU points over NEMATUS is achieved by combining the best initialization and auxiliary-context variants.The reported result is the best performance among the evaluated approaches.
- 3.3 Analysis: The method corrects 76% of identified ambiguity errors and 75% of inconsistency errors, but introduces relative 21% new errors.The analysis samples 15 documents containing about 60 sentences.
4 Related Work
The paper distinguishes its cross-sentence context approach from earlier hierarchical recurrent and auxiliary-context methods. Its differences include attention compatibility, separate control of context types, and a context gate.
- 4 Related Work: Compared with Sordoni et al.’s HRED, the approach supports attention-based encoder–decoders and uses cross-sentence representations beyond decoder initialization.The earlier model summarizes current and previous sentences into a single representation.
- 4 Related Work: Unlike Serban et al., the approach uses separate parameters for cross-sentence and intra-sentence contexts.Serban et al. use one parameter matrix for a representation encoding both contexts.
- 4 Related Work: The approach introduces a context gate because not every target word requires equivalent cross-sentence context.This design controls the effects of global context during generation.
- 4 Related Work: Other work uses additional encoders and attention to select previous-source information or incorporate global image features, but additional attention increases computational cost.Those approaches therefore incorporate limited information.
- 4 Related Work: Multi-source and multi-target NMT integrate additional encoders or decoders for one-to-many or many-to-one translation tasks.The cited experiments report promising results for these multilingual settings.
5 Conclusion and Future Work
The paper presents two complementary ways to integrate cross-sentence context into NMT and reports significant gains over a strong attention-based baseline. It also identifies richer document-level information and full-length documents as future directions.
- Contributions: The model integrates global context by warm-starting encoder and decoder states or by using context as an auxiliary decoder source.The auxiliary-source strategy includes a context gate for updating decoder states.
- Results: The presented model significantly outperforms a strong attention-based NMT baseline in quantitative and qualitative evaluations.
- Future Work: The authors plan to study full-length documents with additional linguistic features, including discourse relations.They also report that the models benefit from larger contexts.