Source-linked AI summary
Improving Neural Language Models with a Continuous Cache
Edouard Grave, Armand Joulin, Nicolas Usunier
TL;DR
Neural language models have limited ability to adapt to recent history, while existing memory-augmented approaches can be computationally expensive. The paper introduces a Neural Cache Model that stores recent hidden activations and retrieves words using dot products, achieving significant gains across language-modeling tasks and LAMBADA at negligible cost.
Problem
Neural language models lack capacity to adapt to recent history, and existing external-memory models can be computationally expensive.
Method
The Neural Cache Model stores recent hidden activations as memory and uses dot-product retrieval to define cache-based word probabilities without learned memory lookup.
Results
Significant performance gains are reported on language-modeling tasks and the challenging LAMBADA dataset after adding the neural cache component.
Takeaways & Limitations
The neural cache can be added to a pre-trained language model at negligible cost while supporting larger cache sizes than memory-augmented networks.
Abstract
from arXiv · showhide
We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.
1 INTRODUCTION
Neural language models perform well but do not adapt readily to recent history, motivating external-memory approaches. The paper proposes a lightweight continuous cache that adapts predictions using recent context and scales to larger memories and datasets.
- Neural language models lack capacity to adapt to recent history, limiting their application in dynamic environments.
- Existing memory-augmented networks can store new information for adaptation but are computationally expensive.Their learned memory read/write mechanisms may restrict usable memory size and training-data scale.
- The proposed Neural Cache Model is a continuous cache that stores recent hidden activations and uses them for prediction.It connects neural external memory with cache models that store representations of recent history.
- The model requires no training, applies to pre-trained neural language models, and scales to thousands of memory cells.
2 LANGUAGE MODELING
Language modeling estimates word probabilities from preceding context using count-based or neural sequence models. Recurrent networks encode history recursively, while cache models exploit recent words to capture long-range document dependencies and adapt efficiently.
- Language modeling assigns probabilities to word sequences by factorizing them into conditional probabilities over preceding history.
- Recurrent networks: Recurrent language models encode the history in a vector computed recursively from the current input and previous hidden state.The recurrent function depends on the chosen architecture, including Elman networks, LSTMs, and GRUs.
- Recurrent networks: In the Elman network, the hidden state applies a nonlinearity to the current word embedding and recurrent transformation of the previous state.The matrices L and R represent word embeddings and recurrent connections, respectively.
- Recurrent networks: Recurrent neural network language models learn parameters by minimizing training-data negative log-likelihood with stochastic gradient methods and truncated backpropagation through time.
- Cache model: Cache models fit a simple language model on recent words and interpolate it with a static model to capture long-range document dependencies.They can adapt efficiently to new domains and predict out-of-vocabulary words after observing them once.
3 NEURAL CACHE MODEL
The Neural Cache Model augments a neural language model with recent hidden activations, retrieving likely next words through dot-product similarity. It requires no additional training and can use large caches, improving validation perplexity over the base model.
- Neural cache mechanism: The cache stores pairs of hidden representations and their corresponding next words, using the current hidden state to define a distribution over cached words.Previous hidden states are used as memory keys, without transforming the stored representations during writing or reading.
- Neural cache mechanism: When θ = 0, the history distribution is uniform and the model becomes equivalent to a unigram cache model.θ controls the flatness of the cache distribution.
- Neural cache mechanism: The cache probability can be interpreted as retrieving the next word from memory using the current hidden activation as a query.The lookup uses simple dot products rather than a learned memory lookup operator.
- Combining model probabilities: The final word probability combines the vocabulary model and cache model through linear interpolation or global normalization.λ controls linear interpolation, while α controls the cache weight under global normalization.
- Validation results: 74.6 validation perplexity is achieved by the best linear interpolation on Penn Tree Bank, compared with 86.9 for the base model.The cache size is 500; global normalization reaches 74.9.
- Training procedure: The method is applied only at test time, with θ and λ or α selected on validation data, avoiding backpropagation over large contexts.This permits cache sizes larger than one thousand and use with already trained neural models.
4 RELATED WORK
The paper connects neural caching to earlier cache, adaptive, and memory-augmented language models. Its approach differs by using hidden activations directly, avoiding learned transformations and scaling to very large caches.
- Cache models: Traditional cache models store recent words and use them to capture long-range document dependencies in n-gram language models.Earlier work also extended cache models with smoothing and document-specific marginal constraints.
- Adaptive language models: Adaptive language models have dynamically adjusted model parameters using interpolation schemes, latent semantic analysis, topic features, and related contextual representations.These approaches span n-gram, maximum entropy, and recurrent-network models.
- Memory-augmented networks: Memory-augmented neural networks store recent representations and access them with attention mechanisms, improving perplexity in language modeling.Related memory mechanisms have also been applied to question answering and sequence reordering.
- Empirical comparison: 72.1 validation perplexity is achieved by the best linear interpolation on wikitext2, compared with 104.2 for the base model.The best global normalization reaches 73.5 with a cache size of 2000.
- Empirical comparison: Test results across the wikitext datasets are directly comparable because the datasets share the same validation and test sets.The results are summarized in Table 2.
- Distinction from related work: Unlike the related recurrent-network cache approach, this method avoids learning a transformation between current and past representations and scales to very large caches.The comparison identifies effortless scaling as the principal advantage of the proposed approach.
5 EXPERIMENTS
Experiments evaluate neural cache models across small, medium-scale, and LAMBADA language-modeling datasets, comparing them with static LSTM and unigram-cache baselines. The method benefits from larger caches and yields substantial perplexity improvements, including 30% on WikiText-2 and 16% on WikiText-103.
- Experimental setup: Experiments train static LSTM language models, then interpolate neural-cache predictions with the static model and compare against unigram caches.The neural cache uses hidden representations from the trained LSTM; cache hyperparameters are selected on validation data.
- Medium-scale results: 16% improvement over the LSTM baseline is obtained on WikiText-103, smaller than the 30% improvement reported on WikiText-2.The authors attribute the reduced gain with more training data to a pattern observed previously and emphasize evaluation on larger datasets.
- Medium-scale results: 99.9 test perplexity is achieved by the neural cache model versus 121.8 for the authors’ LSTM-1024 baseline.These values are reported among the medium-scale experimental results.
- Reported metrics: Table 3 reports perplexity on text8 and LAMBADA and identifies WB5 as a Witten–Bell-smoothed 5-gram language model.The table caption defines the baseline abbreviation but does not provide the table’s numerical entries.
- LAMBADA results: On LAMBADA, adding a neural cache strongly improves the LSTM baseline, while the best interpolation parameter differs between development and control sets.The development set contains the target word in more than 83% of passages, compared with 14% for the control set.
6 CONCLUSION
The conclusion presents the neural cache as a lightweight external memory that dynamically updates neural language-model predictions using longer-term context. Its design avoids learned memory lookup, enabling large cache sizes and easy application to pretrained models, with significant gains across evaluated tasks.
- Conclusion: The neural cache dynamically updates word probabilities from longer-term context and can be added to a pretrained language model at negligible cost.The conclusion reports significant gains on language-modeling tasks and LAMBADA after adding this external memory component.
- Conclusion: Avoiding a learned memory lookup lets the neural cache use larger cache sizes than memory-augmented networks and remain as easy to apply as count-based caches.The conclusion links this design choice to the method’s practicality.