Source-linked AI summary
Selective Attention for Context-aware Neural Machine Translation
Sameen Maruf, André F. T. Martins, Gholamreza Haffari
TL;DR
Sentence-level NMT and many context-aware systems do not adequately use long-range document context. The paper proposes sparse, top-down selective attention with sentence- and word-level variants integrated into Transformer encoders or decoders. Across English-German document MT settings, the approach surpasses context-agnostic and recent context-aware baselines in most cases.
Problem
Sentence-level NMT overlooks useful long-range dependencies, while most context-aware systems use only a few previous sentences and neglect the rest of the document.
Method
The paper uses sparse hierarchical attention to select relevant document sentences before attending to key words, alongside flat sentence- or word-level attention variants.
Results
The selective attention approach surpasses context-agnostic and two recent context-aware baselines across three English-German datasets and online and offline settings.
Takeaways & Limitations
Sentence- and word-level sparsity identifies key context and efficiently compresses document memory for context-aware NMT.
Takeaways & Limitations
The sparsemax implementation is CPU-based and operates only on column vectors, and the authors expect a GPU matrix implementation to improve speed.
Abstract
from arXiv · showhide
Despite the progress made in sentence-level NMT, current systems still fall short at achieving fluent, good quality translation for a full document. Recent works in context-aware NMT consider only a few previous sentences as context and may not scale to entire documents. To this end, we propose a novel and scalable top-down approach to hierarchical attention for context-aware NMT which uses sparse attention to selectively focus on relevant sentences in the document context and then attends to key words in those sentences. We also propose single-level attention approaches based on sentence or word-level information in the context. The document-level context representation, produced from these attention modules, is integrated into the encoder or decoder of the Transformer model depending on whether we use monolingual or bilingual context. Our experiments and evaluation on English-German datasets in different document MT settings show that our selective attention approach not only significantly outperforms context-agnostic baselines but also surpasses context-aware baselines in most cases.
1 Introduction
Sentence-level NMT still overlooks useful long-range document dependencies, while many context-aware systems use only a few preceding sentences. The paper introduces scalable selective and hierarchical attention to focus on relevant document context.
- Sentence-level NMT ignores long-range dependencies that may help translate discourse phenomena.
- Most context-aware NMT systems consider only a few previous sentences and neglect the rest of the document.
- The proposed selective attention first identifies relevant document sentences, then attends to key words within them using sparse attention.
- Hierarchical Attention is computed dynamically for each query word, while Flat Attention uses sentence- or word-level context information.
- The context representation is integrated into the Transformer encoder or decoder for monolingual or bilingual context, respectively.
- +1.34, +2.06 and +1.18 BLEU over the Transformer are reported for TED Talks, News-Commentary and Europarl, respectively.
2 Background
Document-level NMT conditions each sentence translation on the rest of the source and target document, while retaining left-to-right word generation. Training typically combines sentence-level pre-training with whole-model optimization, and decoding uses iterative updates.
- Transformer encoders use multi-head self-attention and feed-forward sub-layers, while decoders add attention over encoder outputs and mask future positions.
- Document translation models estimate each target sentence conditioned on its source sentence and the other document sentences.
- The document context D−j contains all other source and target sentences besides the jth sentence.
- Word-level factorization generates each target sentence one word at a time from previously generated words and document context.
- Training usually pre-trains a sentence-level NMT model before jointly optimizing document-level and sentence-level parameters.
- Two-pass iterative decoding initializes sentences with a sentence-based model, then updates each translation while fixing the others.
3 Proposed Approach
The proposed document-level NMT model uses a context layer with hierarchical or flat attention, integrating document context into the Transformer encoder or decoder. Hierarchical attention selectively identifies relevant sentences and then key words, while sparse attention supports efficient compression of document context.
- 3 Proposed Approach: The model augments a sentence-level Transformer with a document-level context layer designed to be memory-efficient, scalable, and attentive to the entire document.The context layer contains multi-head context attention and a feed-forward sub-layer, with layer normalization after each sub-layer.
- 3.1.1 Hierarchical Attention: Hierarchical attention first matches query words against sentence-level keys, then performs word-level matching within document sentences.Its five inputs include sentence- and word-level queries and keys, while values are word-based.
- 3.1.1 Hierarchical Attention: Sparsemax assigns zero attention to irrelevant sentences, enabling selective context use instead of softmax’s long-tail distribution.Masks exclude the current sentence in offline training and the current and future sentences in online training.
- 3.1.1 Hierarchical Attention: The hierarchical word weights are re-scaled by sentence-level attention before multiplying by word-level values to produce the context representation.This re-weighting lets the model prune the hierarchy and focus on selected words within selected sentences.
- 3.1.2 Flat Attention: Flat attention provides single-level sentence- or word-level alternatives, using standard scaled dot-product attention with dynamic query-word-specific context attention.The paper notes that sparse flat attention is left for future work.
- Context Integration: Monolingual context is integrated alongside the encoder, whereas bilingual context is integrated alongside the decoder through separate context representations.Encoder-side keys and values are derived from source words or sentence representations; decoder-side keys match source context and values match target context.
4 Experiments
Experiments evaluate English→German document-level NMT across three domains, two context settings, and multiple baselines. Selective-attention models generally improve translation quality, with gains varying by integration strategy and dataset.
- Experimental Setup: Experiments cover TED talks, News-Commentary, and Europarl English→German datasets with different genre, style, and formality characteristics.TED treats each talk as a document; Europarl retains documents longer than five sentences.
- Experimental Setup: The study compares offline settings using past and future context with online settings using only past context.Experiments are divided into offline and online document MT, with Transformer and other context-aware or context-agnostic baselines.
- Offline Document MT: In offline experiments, Transformer outperforms RNNSearch by at least +2.5 BLEU and +2.1 Meteor in all cases.All reported model results are significantly better than both context-agnostic baselines.
- Offline Document MT: Hierarchical Attention is strongest for News and Europarl encoder integration, while Flat Attention leads on TED; Hierarchical Attention usually wins decoder integration.News and Europarl encoder gains reach +1.98 and +1 BLEU, while TED Flat Attention improves BLEU by +1.27.
- Online Document MT: Online models significantly outperform the context-agnostic baseline and usually surpass context-aware baselines, with Europarl gaining +1.15 BLEU and +1.13 Meteor over Miculicich et al.The authors conclude that efficiently using available context is crucial for context-aware NMT.
- Offline vs. Online Document MT: Offline and online best scores are close, with online results slightly better, while future-context benefits depend on the data.The datasets consist of talks, speeches, or commentaries produced in an online manner.
- Contrastive Pronoun Evaluation: Hierarchical Attention is most useful when pronoun antecedents occur more than three previous sentences away, and offline models perform better beyond distance two.The comparison uses a contrastive test set organized by antecedent distance.
5 Related Work
Document-level MT work spans conventional and neural approaches, with neural systems differing mainly in how much past or future context they use. Sparse attention has also been applied to limit attention and improve efficiency in related NLP tasks.
- Document-level MT is broadly divided into conventional MT and neural MT.
- Conventional Document-level MT: Conventional approaches use cache-based memories or target discourse phenomena such as anaphora, lexical cohesion, and coreference.
- Document-level Neural MT: Neural document-level MT is categorized as online, using previous context, or offline, using both past and future contexts.
- Document-level Neural MT: Most neural approaches use limited context, while only one cited offline work uses full source- and target-side document context with memory networks.
- Sparse Attention: Sparse attention has been used for NMT coverage and has also shown promising results in textual entailment and summarization.
6 Conclusion
The paper proposes sparsemax-based hierarchical attention for scalable context-aware NMT and evaluates it across online and offline English→German document translation. The approach surpasses context-agnostic and two recent context-aware baselines, while sparsity identifies relevant sentences and words for efficient memory compression.
- Sparse attention makes the proposed hierarchical context-aware NMT approach scalable and efficient.
- Experiments on three English→German datasets show that the approach surpasses context-agnostic and two recent context-aware baselines in online and offline settings.
- Sentence-level sparsity identifies key document-context sentences, while word-level sparsity focuses on key words and efficiently compresses memory.