Memory-augmented Neural Machine Translation

1 Introduction

• NMT’s shared parameters let frequent pairs dominate, so infrequent pairs may be treated as noise and rare entities can produce meaning drift.The paper gives an infrequent chromosome as an example of a keyword that NMT may fail to translate correctly.Source: §1 · pp. 1–2
• SMT uses discrete word and phrase statistics with little parameter sharing, so frequent pairs have less impact on infrequent pairs.SMT typically memorizes observed patterns in a phrase table.Source: §1 · p. 2
• The proposed combination keeps the neural architecture primary and uses symbolic knowledge as complementary support.The paper contrasts this design with approaches that use neural models to improve SMT.Source: §1 · p. 2
• M-NMT pairs a neural model for frequent patterns with memory of source-target word pairs for infrequent words and pairs.The memory knowledge is used to improve the neural model.Source: §1 · p. 2

2 Attention-based NMT

• The attention-based RNN uses a bidirectional GRU encoder to represent source words as hidden states and an RNN decoder to produce the target sentence.This model serves as the baseline and as M-NMT’s neural component.Source: §2 · p. 2
• At each decoding step, attention measures relevance between the decoder state and source-word representations to derive the content the decoder focuses on.The attention factor for source word x_j at step i is denoted α_ij.Source: §2 · p. 2
• The decoder updates its hidden state recurrently and generates the next target word according to a posterior distribution.The posterior uses a softmax projection, with an intermediate variable computed by a neural net with a single maxout hidden layer.Source: §2 · pp. 2–3
• The model was implemented in TensorFlow using a training recipe that largely followed Bahdanau et al. (2015).Source: §2 · p. 3

3.1 The Architecture

• The attention-based RNN handles frequent words and pairs, while memory supplies knowledge for infrequent ones; their outputs combine into a consolidated translation.Source: §3.1 · p. 3

3.2 Memory Elements

• Each global memory element maps a source word to one or more likely target translations; a sentence-specific local memory uses context annotations and merges entries with the same target.The global memory is static; repeated source-word occurrences retain distinct annotations, and merged source representations use translation probabilities.Source: §3.2 · pp. 3–4

3.3 Memory Attention

• Memory attention is combined with the neural posterior, and the memory is trained separately while the neural model remains unchanged.The memory posterior is normalized over target words in local memory, so treating it as a posterior over all words is an approximation; joint training is left for future work because of GPU demands and overfitting risk.Source: §3.3 · pp. 4–5

3.4 Memory for SMT Integration

• For SMT integration, the authors derive a word-translation dictionary from bidirectional GIZA++ alignments refined by intersection, using conditional probabilities to select and merge memory entries.Phrase pairs longer than one word are left for future work.Source: §3.4 · p. 5

3.5 Memory for OOV Treatment

• For true OOVs absent from bilingual and monolingual training data, a manually specified dictionary builds run-time memory, with similar-word vectors standing in for OOVs and predictions redirected to them.Several substitute candidates are pre-defined for run-time selection; the method can handle an OOV target as well as an in-vocabulary target.Source: §3.5 · p. 5

Memory in NMT

• Unlike Wang et al. (2016), whose memory provides temporary variables for RNN decoding, M-NMT memory stores translation knowledge like a dictionary.Source: §4 · p. 5

Combining SMT and NMT

• Earlier SMT-NMT combinations mostly used an SMT framework; Cohn et al. (2016) improved attention with alignment knowledge, while M-NMT promotes target words directly from mappings.Source: §4 · p. 5

OOV Treatment

• For OOVs, prior methods enlarge vocabularies, use subwords, or replace unknowns after decoding; M-NMT prepares dictionary knowledge before decoding and can offer multiple targets.The authors also relate their similar-word approach to Li et al. (2016).Source: §4 · p. 6

5.1 Data

• The experiments use Chinese-English translation tasks: IWSLT has 44K tourism and travel sentences, while NIST uses 1M sentence pairs.The memory is constructed from bidirectional GIZA++ alignments; its global size is 80K for IWSLT and 500K for NIST, with at most two target candidates retained per source word.Source: §5.1 · p. 6

5.2 Systems

• The M-NMT model retains the NMT baseline and learns a separate memory-attention function; memory attention can use target history and source-target memory factors.During training, target words that are UNK or absent from memory are skipped from back-propagation to avoid bias from numerous UNK symbols; the trained system can handle OOV words without retraining.Source: §5.2 · pp. 6–7

5.3 SMT-NMT Integration Experiments

• 9.0 and 2.7 BLEU points: the best M-NMT system outperforms the NMT baseline on the two datasets, respectively.It also outperforms the SMT baseline on IWSLT05.Source: Table 3 · p. 7, §5.3 · pp. 7–8
• On IWSLT05, SMT outperforms NMT, while on the larger NIST dataset, NMT outperforms SMT; lexical augmentation improves NMT on both.The authors report that the lexical approach’s improvement is less significant than in Arthur et al. (2016), likely because their implementation does not retrain the neural model.Source: §5.3 · p. 7

5.4 OOV Treatment Experiments

• The M-NMT system with OOV words in memory is much more effective at OOV translation than the basic NMT, M-NMT, and place-holder systems.The experiment used 312 NIST test sentences containing OOV words, split into T-INV and T-OOV cases.Source: §5.4 · p. 8, Table 4 · p. 8

5.5 Frequency Analysis

• M-NMT improves recall more for infrequent words, measured across four bins based on each sentence’s lowest word frequency.The analysis uses the NIST dataset and counts a generated word as a hit when it appears in any reference.Source: Figure 2 · p. 8, §5.5 · p. 9

• 9.0 and 2.7 BLEU score improvements were reported on two Chinese-English translation datasets, respectively.Source: §6 · p. 9
• The memory mechanism assists conventional NMT with translating words not well learned by the neural model.Source: §6 · p. 9
• 28% and 40% OOV recall were reported for OOV inputs whose target words are INV and OOV, respectively.Source: §6 · p. 9
• Future work will investigate better model-memory integration, including joint training.Source: §6 · p. 9