Source-linked AI summary
Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
Colin Raffel, Daniel P. W. Ellis
TL;DR
Very long sequences challenge recurrent models because training can suffer from vanishing and exploding gradients. The paper introduces feed-forward attention and shows it handles addition and multiplication sequences with highly variable lengths, achieving 99.9% and 99.4% accuracy.
Problem
Recurrent networks often struggle to capture dependencies across very long sequences because backpropagation through time is hindered by vanishing and exploding gradients.
Method
The paper applies simplified attention to feed-forward networks by adaptively averaging state representations across sequence positions into a fixed-length context vector.
Results
A single feed-forward attention model achieved 99.9% accuracy on addition and 99.4% on multiplication for sequences uniformly varying from 50 to 10000 steps.
Takeaways & Limitations
The experiments show that attention can let feed-forward models integrate information over very long, widely variable-length sequences and refer to specific sequence points.
Takeaways & Limitations
The model fails on tasks where temporal order matters because averaging over time discards order information.
Abstract
from arXiv · showhide
We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.
1 MODELS FOR SEQUENTIAL DATA
The section introduces attention as a way to integrate information across time and proposes a simplified feed-forward variant that embeds variable-length sequences while enabling parallel computation. It motivates investigating this model as an alternative to recurrent temporal integration.
- Attention mechanisms: Attention provides more direct dependencies between model states at different sequence positions, helping models capture long-term temporal relationships.Attention computes context vectors as weighted means of hidden states, with weights determined by learned importance values.
- Simplified feed-forward attention: The simplified mechanism produces a fixed-length sequence embedding by taking an adaptive weighted average of feed-forward hidden states h_t = f(x_t).Its learned weighting function depends only on each hidden state h_t, unlike the recurrent attention formulation.
- Simplified feed-forward attention: Feed-forward attention can integrate information over time despite feed-forward state computation, allowing models to handle variable-length sequences.The adaptive weighted average collapses the temporal dimension into a single vector c.
- Motivation: Because feed-forward computation can be completely parallelized, the proposed model may offer substantial efficiency gains over recurrent processing.The section investigates the capabilities of this feed-forward attention model in Section 2.
- Alternative integration: Without attention, feed-forward models can process sequential data directly when sequence length T is fixed, but varying T requires temporal integration.The section identifies unweighted averaging as a straightforward alternative and notes its use for collapsing audio and text sequences.
2 TOY LONG-TERM MEMORY PROBLEMS
The section evaluates an attention-based feed-forward model on synthetic addition and multiplication long-term memory problems. It describes the model and experiments spanning long and highly variable sequence lengths, while noting that temporal-order tasks remain a limitation.
- Tasks and evaluation: The experiments focus on the synthetic “addition” and “multiplication” problems proposed by Hochreiter & Schmidhuber (1997), with accuracy defined as the proportion of sequences meeting the task criterion.The supplied passage truncates the remainder of the accuracy definition.
- Model: The model computes each time-step state with a leaky rectifier and then uses an attention-based context vector to produce its output.The leaky rectifier improved early convergence and was used in all models.
- Sequence-length experiments: Models were traditionally tested on sequence lengths uniformly distributed over [T0, 1.1T0], so increasing T0 requires handling longer-term dependencies.Across prior RNN results, the largest attained T0 ranged from a few hundred to a few thousand, according to the supplied passage.
- Sequence-length experiments: T0 = 10000 required 254 seconds for the proposed model versus 917 seconds for a comparable single-layer vanilla RNN on the same GPU.The proposed model had 10,602 parameters versus 10,501 for unweighted averaging, an increase of less than 1%.
- Limitation: Because averaging over time discards order information, the model fails on tasks where temporal order matters, including distinguishing the X, Y and Y, X cases in the twosymbol temporal order task.It can nevertheless distinguish the X, X and Y, Y cases perfectly, according to the supplied passage.