Source-linked AI summary
End-To-End Memory Networks
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus
TL;DR
The paper addresses how to perform multiple computational steps over memory while learning with limited supervision. It introduces an end-to-end recurrent attention model with multiple memory hops per output, and reports competitive question answering and slightly better language-modeling performance than tuned recurrent baselines.
Problem
Existing memory models required strong layerwise supervision, limiting their use in tasks such as language modeling and realistically supervised question answering.
Method
The paper introduces a recurrent neural network that repeatedly reads a continuous external memory through soft attention, with tied multi-hop layers updating an internal state.
Results
The model approaches strongly supervised Memory Networks on question answering and slightly outperforms tuned RNNs and LSTMs of comparable complexity on language modeling, with more hops improving performance.
Takeaways & Limitations
Multiple computational hops are important across question answering and language modeling, while end-to-end training enables use with less supervision.
Takeaways & Limitations
The model does not match strongly supervised memory networks, fails on several 1k question-answering tasks, and smooth lookups may not scale to larger memories.
Abstract
from arXiv · showhide
We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (Weston et al., 2015) but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings. It can also be seen as an extension of RNNsearch to the case where multiple computational steps (hops) are performed per output symbol. The flexibility of the model allows us to apply it to tasks as diverse as (synthetic) question answering and to language modeling. For the former our approach is competitive with Memory Networks, but with less supervision. For the latter, on the Penn TreeBank and Text8 datasets our approach demonstrates comparable performance to RNNs and LSTMs. In both cases we show that the key concept of multiple computational hops yields improved results.
1 Introduction
The paper targets multi-step computation and long-term dependencies by introducing an end-to-end recurrent architecture that repeatedly reads an external memory before producing each output.
- The model performs multiple computational hops over a possibly large external memory before outputting each symbol.
- Unlike the Memory Network implementation in, it is trained end-to-end from input-output pairs without supervision at every layer.This broadens applicability to tasks such as language modeling and realistically supervised question answering.
- The architecture can also be viewed as RNNsearch [2] extended with multiple computational steps per output symbol.
2 Approach
The approach embeds inputs and queries into a continuous memory, uses soft attention to read it, and stacks recurrent memory hops whose outputs update the internal state before prediction.
- 2.1 Single Layer: The model stores discrete inputs as continuous memory vectors, embeds the query, and uses softmax-weighted matching to read the memory.The resulting probability vector weights output memory representations, producing a response vector through a smooth differentiable operation.
- 2.1 Single Layer: The response vector and query state are combined, transformed by W, and passed through a softmax to predict the answer.All embedding matrices and W are jointly learned with cross-entropy and stochastic gradient descent.
- 2.2 Multiple Layers: Multiple layers stack memory hops by feeding each layer the sum of its input and output states before producing the final prediction.The three-layer architecture replaces hard layerwise selection with continuous softmax weighting.
- 2.2 Multiple Layers: Weight tying can share embeddings across adjacent layers or across all layers, reducing parameters and enabling an RNN-like recurrent interpretation.With layer-wise tying, intermediate steps read memory and update the hidden state before an external output is emitted.
3 Related Work
The paper connects its continuous external-memory model to RNNs, stacks, Neural Turing Machines, and attention-based sequence models, while distinguishing its global memory and soft recurrent reads.
- With layer-wise tying, the model can be viewed as an RNN whose fixed intermediate steps perform memory input/output operations before prediction.Unlike a standard RNN, it explicitly conditions on stored memory outputs during the hops and keeps them soft rather than sampling them.
- Compared with latent-state RNNs and LSTMs [7] [15], the model uses a global memory with shared read and write functions.
- Compared with the Neural Turing Machine [8], it supports explicit content-based access but is simpler because memory is written sequentially and does not require sharpening.The paper notes that temporal features can provide a form of address-based access.
- The model is related to Bahdanau et al. [2] because both use attention to select useful representations for generating the next output.
- For language modeling, it builds on cache, neural-network, RNN, and LSTM approaches used to capture sequence structure.
4 Synthetic Question and Answering Experiments
The experiments evaluate MemN2N on synthetic question-answering tasks where the model must identify relevant statements without supporting-fact supervision. Across 20 tasks, position encoding, training choices, random-memory regularization, joint training, and multiple hops improve performance, bringing the best models reasonably close to supervised Memory Networks.
- Model choices: Position encoding improves over bag-of-words on tasks where word order matters, especially tasks 4, 5, 15, and 18.Position encoding makes sentence order affect the memory representation, unlike bag-of-words.
- Model choices: Linear-start training can avoid poor solutions: on task 16, position encoding alone reaches 53.6% error, whereas adding linear start reduces error to 1.6%.Linear start initially removes softmaxes before training with them, and random empty memories provide a small, consistent boost, especially with 1k examples.
- Results: All MemN2N variants beat weakly supervised baselines, while the best models approach supervised MemNN performance but remain inferior.For 1k training examples, MemNN has 6.7% error versus 12.6% for a strong MemN2N configuration; for 10k, the corresponding errors are 3.2% and 4.2%.
- Results: More computational hops improve performance, and joint training across tasks also helps.The experiments compare one to three memory hops and independently trained versus jointly trained models.
5 Language Modeling Experiments
MemN2N applies recurrent memory hops to word-level language modeling on Penn Treebank and Text8. It achieves lower perplexity than tuned recurrent baselines, while attention patterns suggest alternating local and broad memory access.
- 5 Language Modeling Experiments: The model stores previous words as separate memory cells and predicts the next vocabulary word through a softmax trained with cross-entropy.The question input is fixed to a constant vector, and gradients are backpropagated through multiple memory layers.
- 5 Language Modeling Experiments: Unlike a traditional RNN, recurrence occurs across memory hops, with weight tying restricting parameters and supporting deeper models.The architecture’s recurrent sequence is over memory hops rather than text positions.
- 5.2 Results: 111 vs 115 perplexity on Penn Treebank and 147 vs 154 on Text8, outperforming the reported RNN/SCRN and LSTM baselines.The baselines were tuned for optimal perplexity, while MemN2N used approximately 1.5× more parameters than comparable RNNs and fewer than LSTMs.
- 5.2 Results: Increasing the number of memory hops improves MemN2N performance on the Penn Treebank and Text8 test sets.The experiments vary both memory hops and memory size to assess their contribution.
- 5.2 Results: Across six hops, some attention concentrates on recent words while other hops attend broadly, and these two patterns tend to alternate.The broad and local behaviors are consistent with combining a smoothed n-gram model and a cache.
6 Conclusions and Future Work
The paper concludes that end-to-end recurrent attention over explicit memory works across question answering and language modeling. Multiple memory hops improve performance, but the model remains weaker than strongly supervised memory networks and may not scale to larger memories.
- 6 Conclusions and Future Work: The model is successfully trained by backpropagation across question answering and language modeling using explicit memory and recurrent attention.Compared with Memory Networks, it requires no supervision for supporting facts and applies to a wider range of settings.
- 6 Conclusions and Future Work: Multiple memory hops improve performance on both question answering and language modeling tasks.
- 6 Conclusions and Future Work: The model does not exactly match strongly supervised memory networks, both approaches fail on several 1k QA tasks, and smooth lookups may not scale to larger memories.The authors propose exploring multiscale attention or hashing for larger-memory settings.
Appendix A Results on 10k QA dataset
The appendix reports 10k-example bAbI results across 20 QA tasks using several representation, training, tying, and nonlinearity variants. Its table defines the abbreviations and conditions needed to interpret the reported test errors.
- Appendix A Results on 10k QA dataset: Table 3 reports test error rates (%) for models trained with 10k examples on all 20 bAbI QA tasks.
- Appendix A Results on 10k QA dataset: The table distinguishes bag-of-words and position-encoding representations, plus linear-start training and random time-index noise.BoW, PE, LS, and RN are the corresponding abbreviations.
- Appendix A Results on 10k QA dataset: The table identifies layer-wise RNN-style weight tying, joint training across tasks, and larger nonlinear models with d = 100 and ReLU after each hop.The nonlinear variant was inspired by [17] and was crucial for tasks 17 and 19.
Appendix B Visualization of attention weights in QA problems
Figure 4 visualizes three-hop attention for a PE+LS+RN model trained separately on each bAbI task with 10k examples. It compares learned attention with the sentences required to answer each question and marks mistakes.
- Appendix B Visualization of attention weights in QA problems: The figure shows attention weights across three memory hops for PE+LS+RN models trained separately on each task.Each model uses 10k training examples.
- Appendix B Visualization of attention weights in QA problems: The support column identifies necessary sentences, while hop columns show where the model assigns greater weight using values and blue color.
- Appendix B Visualization of attention weights in QA problems: The model learns to focus on correct support sentences on most tasks, and mistakes are highlighted in red.