Source-linked AI summary
Document Context Neural Machine Translation with Memory Networks
Sameen Maruf, Gholamreza Haffari
TL;DR
Document-level NMT must account for dependencies beyond independently translated sentences. This paper adds source and target memory networks to sentence-based NMT and uses structured prediction with iterative decoding, achieving statistically significant improvements over prior work across three language pairs.
Problem
Sentence-based NMT neglects long-range document dependencies, including discourse phenomena requiring context beyond a few previous sentences.
Method
The model represents source and target document context with external memory networks, trains document-conditioned NMT, and decodes through block coordinate descent.
Results
Statistically significant BLEU and METEOR improvements were obtained across French-English, German-English, and Estonian-English translation.
Takeaways & Limitations
Using both source and target document context is beneficial for improving machine translation performance.
Takeaways & Limitations
At test time, multiple decoding passes were not helpful, so reported results use two-pass decoding with one update from base-model translations.
Abstract
from arXiv · showhide
We present a document-level neural machine translation model which takes both source and target document context into account using memory networks. We model the problem as a structured prediction problem with interdependencies among the observed and hidden variables, i.e., the source sentences and their unobserved target translations in the document. The resulting structured prediction problem is tackled with a neural translation model equipped with two memory components, one each for the source and target side, to capture the documental interdependencies. We train the model end-to-end, and propose an iterative decoding algorithm based on block coordinate descent. Experimental results of English translations from French, German, and Estonian documents show that our model is effective in exploiting both source and target document context, and statistically significantly outperforms the previous work in terms of BLEU and METEOR.
1 Introduction
Sentence-based NMT often neglects document-wide dependencies needed for phenomena such as anaphora and lexical consistency. This paper addresses that gap with a model using both source and target document context, outperforming previous work on three language pairs.
- Most neural MT models translate sentences independently, neglecting discourse phenomena that depend on dependencies beyond a few previous sentences.
- Prior document-level MT attempts were restrictive or failed to achieve significant improvements, while recent neural approaches mainly used local source context.
- The proposed model combines sentence-based NMT with memory networks containing separate source and target document memories.
- Experiments on French-English, German-English, and Estonian-English show statistically significant BLEU and METEOR improvements over previous work.
2 Background
The paper reviews attentional sentence-level NMT and memory networks as the foundations for document-level translation. NMT encodes source sentences, decodes target words with attention, and memory networks retrieve information from external vector memories.
- Neural Machine Translation (NMT): The sentence-based NMT foundation uses an encoder to read the source sentence and an attentional decoder to generate its translation.
- Neural Machine Translation (NMT): The encoder is bidirectional, representing each source word by concatenated forward and backward RNN hidden states.
- Neural Machine Translation (NMT): The decoder generates each target word from prior target words and a dynamic context vector that attends to relevant source parts.
- Memory Networks (MemNets): Memory networks use external collections of vector cells to perform inference over long-range dependencies.
- Memory Networks (MemNets): Reading a memory uses a query to compute relevance-weighted probabilities over memory cells and produce an output vector.
3 Document NMT as Structured Prediction
The paper formulates document translation as structured prediction over interdependent source sentences and target translations. It trains a document-conditioned neural model and decodes with iterative block coordinate descent.
- Document translation is modeled as predicting all target sentences while accounting for interdependencies imposed by the document.
- Two factors capture dependencies between each translation, its source sentence, and other source sentences, and between each translation and other translations.
- The document translation probability is proportional to the product of these source- and target-side factors, whose neural parameters are denoted by θ.
- Maximum-likelihood training is difficult because the partition function spans many translation variables with an unbounded target-language domain.
- The model therefore maximizes pseudo-likelihood over bilingual training documents.
- Because exact decoding is hard, block coordinate descent initializes sentence translations with base NMT and repeatedly updates one sentence while fixing the others.
4 Context Dependent NMT with MemNets
The model augments sentence-level NMT with source and target external memories, attending to document context while generating each sentence. It constructs these memories from document representations and decoder states, with multiple integration architectures explored.
- Document Context: The decoder conditions each generated word on prior target words, the current source sentence, and other document sentences with their current translations.This extends vanilla left-to-right attentional NMT with cross-sentence source and target information.
- Document Context: Document context is represented as source and target external memories, which the decoder consults while generating each target sentence.The memories contain cells for all document sentences except the current one, and attention retrieves relevant context.
- Memory Attention: Source and target queries retrieve relevant context using the current source representation and a target representation combined with source information.Combining decoder and encoder representations makes the target query robust to noise in the current translation and helps limit error propagation.
- Memory Integration: The model incorporates memory contexts either into the next decoder hidden state through Memory-to-Context or into the output layer through Memory-to-Output.Experiments vary source-only, target-only, and combined memories together with the two integration architectures.
- Memory Construction: The source memory uses hierarchical sentence- and document-level bidirectional RNNs, while the target memory stores the last decoder states of current document translations.The source memory is built once per minibatch; target translations are iteratively updated using coordinate descent.
5 Experiments and Analysis
Experiments across French–English, German–English, and Estonian–English evaluate document-context NMT variants against sentence-level and local-context baselines. The best models improve translation quality, with dual memories especially effective, while performance also depends on architecture, target-memory training, and decoding choices.
- Experimental setup: Experiments use French–English, German–English, and Estonian–English corpora, evaluating BLEU and METEOR with bootstrap significance testing at p < 0.05.The German–English evaluation includes news-test2011 and news-test2016.
- Main results: +1.15/+1.13 BLEU/METEOR improvements are observed across the three language pairs for the best Memory-to-Context model versus S-NMT.The both-memory Memory-to-Context variant is the most effective overall across all three language pairs.
- Analysis: Using gold translations to construct target memory decreases BLEU by −0.16 for target-only and −0.25 for both-memory variants.Training with generated translations exposes the model to noisy target-memory inputs encountered during testing.
- Main results: For French→English, the dual-memory model is best on METEOR; for Estonian→English, it is best overall, while German→English target-memory variants are comparable.Memory-to-Context variants generally outperform Memory-to-Output variants, attributed to the latter’s larger parameter count and limited data.
- Main results: +0.72 and +1.44 METEOR gains occur for source-memory and dual-memory Memory-to-Context models when the base NMT uses a larger German–English corpus.The target-memory Memory-to-Output model improves by +1.09 METEOR relative to its baseline.
6 Related Work
Earlier document-level MT approaches used restrictive statistical mechanisms or limited local source context. Neural approaches extended sentence-based translation with previous-sentence attention or summaries of three previous source sentences.
- Document-level Statistical MT: Statistical document-level MT attempts were restrictive or failed to produce significant improvements.Approaches included word-dependency links, caches of previous translations, and two-pass refinement.
- Document-level Statistical MT: Docent modifies Moses translations through stochastic local search and hill-climbing, but subsequent enhancements failed to improve automatic evaluation.
- Larger Context Neural MT: Jean et al. condition the decoder on the previous sentence through attention over its words.
- Larger Context Neural MT: Wang et al. summarize three previous source sentences with a two-level hierarchical RNN before passing the summary to the decoder.
7 Conclusion
The paper proposes document-level neural MT that captures global source and target context through external memories. Across three language pairs, it reports statistically significant translation-quality improvements and identifies discourse-level phenomena as future work.
- The model captures global source and target document context with external memories that augment sentence-based NMT.
- The authors report statistically significant translation-quality improvements on French-English, German-English, and Estonian-English.
- Future work will investigate models incorporating specific discourse-level phenomena.