Source-linked AI summary
Frustratingly Short Attention Spans in Neural Language Modeling
Michał Daniluk, Tim Rocktäschel, Johannes Welbl, Sebastian Riedel
TL;DR
Neural language models with attention aim to use differentiable memory for mid- and long-range dependencies, but conventional mechanisms reuse one representation for prediction and memory. The paper separates key, value, and prediction representations, yet finds that a simple recent-history model performs comparably to more complex attentive models.
Problem
Attention-augmented neural language models reuse one output vector for next-word prediction and for the key and value of differentiable memory, limiting how these functions are separated.
Method
The paper introduces key-value and key-value-predict attention mechanisms, alongside an N-gram RNN that concatenates recent output representations for prediction.
Results
The key-value-predict model outperforms existing memory-augmented models on two corpora, while the simpler N-gram RNN is on par with more sophisticated attentive models.
Takeaways & Limitations
The findings indicate that modeling short attention spans properly can match more elaborate memory-augmented neural language models.
Abstract
from arXiv · showhide
Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning mid- and long-range dependencies. However, conventional attention mechanisms used in memory-augmented neural language models produce a single output vector per time step. This vector is used both for predicting the next token as well as for the key and value of a differentiable memory of a token history. In this paper, we propose a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution. This model outperforms existing memory-augmented neural language models on two corpora. Yet, we found that our method mainly utilizes a memory of the five most recent output representations. This led to the unexpected main finding that a much simpler model based only on the concatenation of recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models.
1 INTRODUCTION
Neural language models need to represent dependencies across different time scales, but attention-based models overload one output representation with prediction and memory roles. The paper separates these roles with key-value-predict attention, then finds that a simpler short-history model performs comparably.
- 1 INTRODUCTION: Neural language models must represent context-specific dependencies across multiple time scales, while conventional attention mechanisms can directly capture long-range connections.Classical N-gram models capture short-distance dependencies explicitly but suffer from data sparsity; neural models maintain dense sequence representations.
- 1 INTRODUCTION: Key-value-predict attention separates representations for next-word prediction, memory keys, and memory values, and outperforms existing memory-augmented language models on two corpora.The model is evaluated on the Children’s Book Test and a corpus of 7,500 Wikipedia articles.
- 1 INTRODUCTION: The proposed model mainly attends to the previous five memories, indicating that increasing the attention history does not yield substantial benefits.The paper therefore questions whether these models can effectively exploit long-range dependencies.
- 1 INTRODUCTION: A simpler model that concatenates recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models.This finding motivates modeling short attention spans directly rather than relying on large differentiable memories.
2 METHODS
The paper develops attention-based neural language models that separate representations for memory keys, values, and next-word prediction, then introduces a simpler model that directly concatenates recent outputs.
- 2.1 ATTENTION FOR NEURAL LANGUAGE MODELING: Standard attention stores a sliding window of previous LSTM outputs and computes a weighted context representation for predicting the next word.Attention weights compare the current output with previous outputs, whose weighted sum forms the context vector.
- 2.2 KEY-VALUE ATTENTION: Key-value attention compares the current key with previous keys to compute attention weights, then retrieves a weighted context from their associated values.This separates the representation used for addressing memory from the representation used as memory content.
- 2.3 KEY-VALUE-PREDICT ATTENTION: Key-value-predict attention separates each output into key, value, and predict representations for attention retrieval and next-word prediction.The output vector is split into three equal parts: key, value, and predict.
- 2.4 N-GRAM RECURRENT NEURAL NETWORK: Initial experiments found that memory-augmented models usually utilize only the previous five output representations, motivating a simpler alternative.The simpler model directly uses previous output representations to calculate next-word probabilities rather than applying attention.
3 RELATED WORK
The related work situates the paper among external-memory architectures, recurrent models using preceding states, and methods that separate memory keys from content.
- External memory: Memory Networks explicitly separate memory storage from neural computation, while recurrent memory models add memory access to sequence modeling.The paper contrasts these approaches with models that produce memory representations directly and use fewer parameters.
- Attention and recurrent models: Attention over recurrent outputs has improved performance across translation, entailment, summarization, image captioning, and speech recognition.These systems attend over previous state outputs on top of an RNN encoder.
- Recurrent alternatives: Higher-order recurrent networks condition computation on multiple preceding states, whereas this paper uses preceding output vectors for prediction without incorporating them into the hidden state.The distinction concerns where earlier representations enter the model.
- Key-value separation: The paper extends functional separation of look-up keys and memory content from Memory Networks, Neural Programmer-Interpreters, Dynamic Neural Turing Machines, and Fast Associative Memory to language modeling.This separation principle is presented as prior work motivating the proposed attention mechanism.
4 EXPERIMENTS
The study evaluates language models on a categorized Wikipedia corpus and the Children’s Book Test, using controlled optimization and truncated backpropagation settings.
- Datasets: Experiments use two corpora: a 22.5M-word Wikipedia training set and the Children’s Book Test for wider-context language modeling.The Wikipedia corpus contains 7,500 articles across five categories, with 1.2M words each for development and test.
- Datasets: The Wikipedia dataset targets entity-referential context, while the Children’s Book Test evaluates exploitation of wider linguistic context.Wikipedia articles come from People, Cities, Countries, Universities, and Novels.
- Training setup: Training uses ADAM with learning rate 0.001, batch size 64, gradient clipping at 5, and 20-step backpropagation through time.The LSTM forget-gate bias is initialized to 1, while other parameters are initialized uniformly from −0.1 to 0.1.
5 RESULTS
The proposed attention models improve Wikipedia perplexity, but longer histories do not significantly help, and a simple 4-gram RNN performs comparably to complex memory-augmented models.
- History length: Longer attention histories did not significantly improve perplexity for any attentive model, although the proposed models attended farther back more often than the Recurrent-memory Model.The results are summarized in Figure 2a and the attention distributions in Figure 3.
- Attention: 3.2 points lower perplexity than a vanilla LSTM was observed for attention on Wikipedia, with notable CBT differences only for verbs and prepositions.The experiments compared memory-augmented models across Wikipedia and CBT, with results shown in Figure 2.
- Key-Value: 7.0 points lower perplexity than a baseline LSTM and 1.9 points lower than RM(+tM-g) were obtained by Key-Value attention.CBT improvements were small.
- Key-Value-Predict: 9.4 points lower perplexity than a baseline LSTM, 4.3 points lower than RM(+tM-g), and 2.4 points lower than Key-Value attention were achieved by Key-Value-Predict.It also increased CBT accuracy by 1.0 percentage points for verbs and 1.7 for prepositions.
- N-gram RNN: 4-gram RNN perplexities were on par with or better than more complicated neural language models with attention.The model uses output vectors from previous time steps and performed best with the previous three output vectors.
6 CONCLUSION
Separating key, value, and prediction representations improves attentive language modeling on Wikipedia and CBT, but the models mainly use recent history. A simpler model using three previous output representations performs comparably, underscoring the difficulty of learning long-range dependencies.
- The key-value-predict attention model outperformed simpler attention mechanisms on Wikipedia and the Children Book Test (CBT, Hill et al., 2016).
- All attentive models mainly used the most recent history and failed to exploit long-range dependencies.
- A simpler N-gram RNN using concatenated output representations from the previous three time steps was on par with more sophisticated memory-augmented models.
- Training neural language models that use long-range dependencies remains notoriously hard and requires further investigation.