Source-linked AI summary
Learning to Remember Translation History with a Continuous Cache
Zhaopeng Tu, Yang Liu, Shuming Shi, Tong Zhang
TL;DR
Existing NMT models translate sentences in isolation, missing document-level information and producing inconsistency or ambiguity. The paper augments NMT with a lightweight continuous cache that stores bilingual hidden representations and updates word probabilities using retrieved translation history. Experiments show consistent improvements over a strong baseline across domains and styles, while the authors identify long-range context modeling as an area for future work.
Problem
Existing NMT models process documents sentence by sentence, overlooking cross-sentence information needed to address translation inconsistency and ambiguity.
Method
The proposed CACHE augments NMT with an external key-value memory storing attention vectors and decoder states from previous translations, retrieving history online during decoding.
Results
The proposed approach significantly and consistently outperforms a strong NMT baseline across multiple domains with different topics and styles.
Takeaways & Limitations
The cache remembers exact, fuzzy, word-level, and phrase-level translation patterns while using internal representations to incorporate target-side context.
Takeaways & Limitations
The current caching mechanism does not model long-range context well, motivating improved architectures and discourse-aware cache values.
Abstract
from arXiv · showhide
Existing neural machine translation (NMT) models generally translate sentences in isolation, missing the opportunity to take advantage of document-level information. In this work, we propose to augment NMT models with a very light-weight cache-like memory network, which stores recent hidden representations as translation history. The probability distribution over generated words is updated online depending on the translation history retrieved from the memory, endowing NMT models with the capability to dynamically adapt over time. Experiments on multiple domains with different topics and styles show the effectiveness of the proposed approach with negligible impact on the computational cost.
1 Introduction
Existing NMT models usually translate sentence pairs independently, overlooking cross-sentence information and causing inconsistency and ambiguity. The paper proposes a lightweight continuous cache that uses bilingual translation history and improves translation across domains and styles.
- Existing NMT models treat documents as independent sentence pairs, ignoring cross-sentence information that contributes to translation inconsistency and ambiguity.
- Prior cross-sentence methods summarize or attend to previous source sentences but fail to use target-side information and can propagate translation errors.
- The proposed CACHE stores recent hidden representations as bilingual translation history, enabling large-scale context while reducing error propagation from output words.
- The cache is designed as a lightweight alternative that can exploit long translation histories with limited computational burden.
- The proposed approach significantly and consistently improves translation performance over a strong NMT baseline across domains with different topics and styles.
- The cache remembers translation patterns from exact lexical matches to fuzzy matches and from word-level to phrase-level patterns.
2 Neural Machine Translation
Standard NMT computes word probabilities from the previous target word, decoder state, and source representation while translating each sentence independently. This sentence-level process leaves ambiguity and consistency problems that translation history can alleviate.
- NMT models translation word by word, computing each target-word probability from the previous target word, decoder state, and source representation.
- The decoder state is recurrently computed, while the source representation dynamically summarizes source words through attention.
- Current NMT models translate each document sentence alone, requiring substantial capacity to disambiguate source and target words from sentence-level context.
- The illustrated translations show inconsistent rendering of “觉得” and ambiguity in translating “机遇” without document-level history.
- The example contains ambiguity for “机遇” (“opportunity”) and tense inconsistency for “觉得” (“feel”).
3 Approach
The approach augments NMT with a continuous key-value cache that retrieves target-side translation history using current source-side context, combines it with the decoder state, and stores generated sentence histories for future translations.
- Architecture: A cache-like memory stores key-value pairs whose keys represent source-side attention contexts and whose values represent target-side decoder contexts from translation history.The cache is designed to match source-side context while retrieving target-side information associated with previously generated words.
- Architecture: At each decoding step, the current attention context queries the cache, which returns a target-side vector that is combined with the current decoder state to generate the next word.The architecture performs key matching, value reading, and representation combining online during decoding.
- Reading from Cache: Cache reading uses attention-context keys to retrieve similar records, sums stored values weighted by matching probabilities, and returns the retrieved vector.The matching operation uses simple dot products and avoids learning additional memory-matching parameters.
- Reading from Cache: A dynamic element-wise interpolation combines the decoder state and retrieved cache vector, allowing different dimensions to retain different amounts of translation-history information.The interpolation weights are computed at each decoding step from the decoder state, attention context, and retrieved vector.
- Writing to Cache: The cache is updated after each translation sentence: new words use an empty or least-recently-used slot, while repeated words update stored key and value vectors by averaging them with new contexts.The averaging operation gives more recent histories greater influence for repeated words.
- Training and Inference: During inference, beam search remains standard, and only the 1-best output representations are written to the cache to improve consistency and avoid the sparsity issues of k-best histories.Preliminary experiments reported no improvement from writing k-best outputs or all beam hypotheses instead of the 1-best output.
4 Experiment
Experiments across diverse Chinese–English domains show that CACHE improves translation quality and consistency while adding little computational overhead. Its behavior depends on cache size and domain, and continuous representations capture lexical, fuzzy, word-level, and phrase-level translation patterns.
- 4.2 Effect of Cache Size: Small caches, such as size=25, generally achieve similar performance to larger caches, such as size=500, on tuning sets.Larger capacity improves history length but makes related records harder to match; the current mechanism also models long-range context imperfectly.
- 4.2 Effect of Cache Size: The cache attends more to recent history, with larger caches placing even more attention on the most recent records.Attention to older records varies by domain: Subtitle and TED use farther-back records more often than News.
- 4.3 Main Results: CACHE significantly outperforms the BASE system across News, Subtitle, and TED domains with different textual styles.Compared with source-only cross-sentence approaches, it remains robust across all domains, including Subtitle.
- 4.3 Main Results: The cache adds 4M parameters versus 84.2M in the existing model, while decoding speed does not decrease.The model trains cache-related parameters only, although it cannot use mini-batch length sorting because it depends on preceding hidden states.
- 4.4 Deep Fusion vs. Shallow Fusion: Deep fusion with continuous vectors outperforms shallow word-level fusion, which does not improve over the baseline.Continuous vectors reduce noise from lexical caching and can store information beyond a single word.
- 4.5 Translation Patterns Stored in the Cache: CACHE improves tense consistency and stores both fuzzy-match indicators and phrase-level translation patterns.Decoder states provide phrasal information, while translation history can guide choices such as “felt” for a repeated term.
5 Related Work
The paper situates its approach among memory-augmented networks, cache models, and methods for exploiting cross-sentence context in NLP and machine translation.
- Cache in NLP: The paper adapts cache and memory ideas to NMT by storing bilingual information rather than monolingual information and by removing hand-tuned cache-matching parameters.Its approach builds on continuous caches for language modeling while changing the stored information and matching procedure for translation.
- Exploitation of Cross-Sentence Context: Cross-sentence context has been used in statistical machine translation to reduce inconsistency and ambiguity, including through discrete caches of bilingual phrases.The paper connects its continuous cache to earlier statistical approaches that stored translation information from previous sentences.
- Exploitation of Cross-Sentence Context: Related NMT methods encode or summarize previous source sentences, but the proposed approach instead learns from bilingual translation history.Earlier methods use hierarchical recurrent networks or additional encoders and attention over previous source-side context.
- Memory-Augmented Neural Networks: Prior cache models and memory networks motivate storing contextual information externally to improve language modeling and document-oriented NLP tasks.The cited work spans statistical cache language models, Memory Networks, Key-Value Memory Networks, and continuous caches.
6 Conclusion
The paper concludes that a cache-like memory can incorporate translation history into NMT, while future work should improve the modeling of long-range context and discourse information.
- Conclusion: The cache stores bilingual hidden representations from prior decoding steps and updates word probabilities online using retrieved translation history.Keys are attention vectors, values are decoder states, and dot-product matching enables efficient access.
- Conclusion: Future work should investigate architectures for long-range contexts and use discourse relations instead of directly using decoder states as cache values.These directions are proposed to clarify how long-range context should be represented and utilized.