Source-linked AI summary
Using Fast Weights to Attend to the Recent Past
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu
TL;DR
The paper addresses the limitation of RNNs that primarily use neural activities and slow weights by introducing fast weights for temporary memories of recent sequence history. Fast associative memory combines these memories with ordinary recurrent computation, yielding attention to recent hidden states; across retrieval, visual-attention, and partially observable-control tasks, the reported results favor fast-weight models, with layer normalization improving robustness.
Problem
RNNs largely use neural activities and slow weights, despite a proposed need for memory with greater capacity than activities and faster dynamics than standard weights.
Method
The paper introduces fast associative memory that stores recent hidden states in rapidly changing weights and uses them to influence recurrent hidden-state updates.
Results
Fast-weight models improve performance across associative retrieval, visual-attention classification, and partially observable Catch tasks, while layer normalization makes the mechanism more robust to hyperparameters.
Takeaways & Limitations
Fast weights provide a neurally plausible form of attention to the recent past without storing copies of neural activity patterns.
Abstract
from arXiv · showhide
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These "fast weights" can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proved very helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.
1 Introduction
Ordinary RNNs primarily use hidden activities for short-term sequence history and slow weights for long-term regularities. The paper motivates a third memory system with greater capacity than activities and faster dynamics than standard weights.
- Memory in ordinary RNNs: Hidden activities store the current sequence history and are updated at every time step, giving short-term memory with capacity O(H).H is the number of hidden units.
- Memory in ordinary RNNs: Slow weight matrices store how inputs and hidden states determine future hidden and output vectors, with capacity O(H2) + O(IH) + O(HO).I and O are the numbers of input and output units.
- Long-range sequence learning: LSTMs improve long-range sequence learning by computing hidden-state increments and allowing hidden activities to control gates that scale weight effects.These multiplicative interactions dynamically adjust effective weights.
- Motivation for fast weights: The proposed third memory would store sequence-specific information with higher capacity than neural activities and faster dynamics than standard slow weights.Earlier work suggested fast weights for recursion, but the cited proposals did not implement this method.
2 Evidence from physiology that temporary memory may not be stored as neural activities
Temporary cognitive processes operate on an intermediate timescale that falls between neural activation dynamics and long-term synaptic plasticity. Physiological short-term plasticity mechanisms provide a synapse-specific alternative for storing temporary information.
- Intermediate timescales: Working memory, attention, and priming operate from 100ms to minutes, between activation dynamics of about 10ms and long-term plasticity over minutes to hours.This timing motivates mechanisms distinct from both transient neural activity and long-term synaptic change.
- Synaptic mechanisms: Short-term facilitation, short-term depression, and spike-time-dependent plasticity operate on intermediate timescales through synapse-specific mechanisms.Facilitation involves leftover axonal Ca2+, while depression involves presynaptic neurotransmitter depletion.
- Synaptic mechanisms: Because these plasticity mechanisms are synapse-specific, they are more accurately modeled by memory with O(H2) capacity than by standard recurrent activity with O(H).The supplied passage introduces this capacity comparison but is truncated before completing the contrast.
3 Fast Associative Memory
Fast associative memory stores recent hidden states in rapidly changing synaptic variables and uses them to influence the next hidden state. Its operation is equivalent to attention over recent states weighted by similarity and temporal decay, with layer normalization improving robustness.
- Fast associative memory: Fast associative memory reconstructs temporary memories from synaptic information rather than storing copies of neural activity patterns.The paper connects this associative-memory principle to recent hidden-state storage.
- Fast associative memory: Each hidden-state transition combines the new input through C, the previous hidden state through W, and recent hidden-state history through fast weights A.The fast memory is updated continuously, without separate decisions about where or when to read and write.
- Attention to the recent past: With exponential decay, fast weights attend to recent hidden vectors according to their scalar products with the current vector and a decay factor.The equivalent stored-vector formulation avoids explicitly computing the full fast matrix and supports efficient simulation.
- Fast-memory update: The fast memory matrix decays existing values by λ and adds ηh(t)h(t)^T, storing an outer product of the current hidden state.This update is the paper’s fast associative-memory learning rule.
- Hidden-state computation: The next hidden state begins with f(Wh(t) + Cx(t)) and is then refined through an S-step inner loop involving A(t)hs(t + 1).The recurrent transition therefore separates a preliminary state from iterative settling.
- Layer normalized fast weights: Layer normalization is applied during each inner-loop iteration because hidden-vector norms can make scalar products vanish or explode.The paper reports that this improves robustness to learning-rate and decay-hyperparameter choices.
4 Experimental results
Across associative retrieval, visual attention, facial expression recognition, and partially observable Catch, fast-weight models were evaluated against recurrent baselines and showed benefits that depended on memory capacity and task setting.
- Associative retrieval: The associative retrieval task presents sequential key-value pairs and later queries one key, requiring temporary storage beyond a standard RNN’s hidden activities.The training set contained 100,000 examples, with 10,000 validation and 20,000 test examples.
- Associative retrieval: When recurrent-unit counts were small, fast associative memory significantly outperformed same-sized LSTMs in retrieval accuracy and converged faster.The comparison used a ReLU RNN augmented with fast associative memory and LSTM variants with the same architecture.
- Integrating glimpses in visual attention models: The multi-level visual attention model receives a fixed sequence of recursively subdivided image glimpses, using fast associative memory to retain information across processing levels.For MNIST, the sequence contained 24 patches of 7 × 7 pixels; the simplified setup supplied glimpse and storage control signals.
- Integrating glimpses in visual attention models: On MNIST, fast weights significantly improved classification when hidden-unit capacity was limited and consistently outperformed LSTM as memory capacity increased.The comparison used a ReLU RNN with multi-level fast associative memory and an LSTM receiving the same glimpse sequence.
- Facial expression recognition: On near-frontal Multi-PIE faces, the multi-level fast-weights model outperformed LSTM and IRNN, but the ConvNet performed better under the rigid predetermined glimpse policy.The authors attribute the ConvNet advantage to weight sharing, architectural engineering, and simultaneous access to information at each processing level.
- Agents with memory: In partially observable Catch, fast-weights agents learned faster than ReLU RNN and LSTM agents, with a larger improvement on the larger memory-demanding game variant.The tested variants were N = 16, M = 3 and N = 24, M = 5.
5 Conclusion
The paper improves RNN performance across varied tasks by using fast weights to attract each new hidden state toward recent states according to scalar products with the current state.
- Fast weights let each new hidden state be attracted toward recent hidden states in proportion to their scalar products with the current state.Layer normalization makes this recent-past attention work much better.
A.1 Associative retrieval
The associative-retrieval experiments compare fast-weights memory with several RNN architectures using specified initialization and hyperparameter settings.
- The experiments compare fast weights with IRNN, standard LSTM, and associative LSTM architectures.All models use a single hidden-layer recurrent network with a 100-dimensional embedding input.
- Hyperparameter settings are provided for the models used in the experiments.The supplied settings include slow-weight initialization, learning rates, decay, activation, and associative-LSTM memory configuration.
- Fast weights use learning rate η=0.5, decay rate λ=0.9, and updates at every time step.The recurrent slow weights are initialized to an identity matrix scaled by 0.05, with ReLU activation.
- The IRNN uses recurrent slow weights initialized to an identity matrix scaled by 0.5 with ReLU nonlinearity.
- The associative LSTM uses four copies of memory cells and three read-write heads for storage and retrieval.
A.2 Integrating glimpses in visual attention models: MNIST and Facial expression recognition
The visual-attention experiments use comparable parameter initialization and hyperparameter settings across both tasks.
- Both tasks use similar parameter initialization and hyperparameter settings comparable to the associative-retrieval task.
A.3 Agents with memory
The agents use recurrent policies with 128-unit hidden and recurrent layers, softmax action outputs, and a linear value estimate, with random-search hyperparameter selection.
- Each agent passes inputs through 128 ReLU units and then a recurrent core with 128 recurrent cells.
- At every step, the policy outputs a softmax over valid actions and a single linear estimate of the value function.
- Random search selects hyperparameters including learning rate, Hebbian steps, and fast-weight learning rate and decay.
B Implementing the fast weights “inner loop” in biological neural networks
The paper considers two ways to sustain hidden activity during fast-weight inner-loop settling: stored boundary conditions or adding the identity matrix to the fast weights.
- Method 1 stores hidden-unit inputs after an outer-loop transition using W, providing sustained boundary conditions during inner-loop settling.
- Method 2 adds the identity matrix to the fast-weight matrix so inner-loop settling tends to sustain the hidden activity vector.
- For ReLUs, the two settling methods are equivalent under the condition stated by the paper.