Source-linked AI summary
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
Trieu H. Trinh, Andrew M. Dai, Minh-Thang Luong, Quoc V. Le
TL;DR
Long-term dependencies are difficult for RNNs because BPTT suffers from unstable gradients and sequence-length-dependent memory costs. The paper adds auxiliary losses that reconstruct past or predict future subsequences around random anchors, enabling effective training with shorter supervised BPTT. Across image and document benchmarks, the approach achieves competitive performance with fewer resources, including on sequences up to 16 000 elements.
Problem
RNNs struggle with long-term dependencies because BPTT gradients can vanish or explode and storing intermediate states scales with sequence length.
Method
The method adds an unsupervised auxiliary loss that reconstructs or predicts a random sequence segment around an anchor point alongside the supervised objective.
Results
Across pixel-by-pixel image and character-level document benchmarks, the models achieved competitive performance over recurrent and non-recurrent baselines, using fewer resources for long sequences.
Takeaways & Limitations
Auxiliary losses improve LSTM optimization and generalization while making short supervised BPTT effective for very long sequences.
Takeaways & Limitations
The current work focuses on recurrent networks and leaves incorporating the technique into scalable Transformer variants for future exploration.
Abstract
from arXiv · showhide
Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences. This paper proposes a simple method that improves the ability to capture long term dependencies in RNNs by adding an unsupervised auxiliary loss to the original objective. This auxiliary loss forces RNNs to either reconstruct previous events or predict next events in a sequence, making truncated backpropagation feasible for long sequences and also improving full BPTT. We evaluate our method on a variety of settings, including pixel-by-pixel image classification with sequence lengths up to 16\,000, and a real document classification benchmark. Our results highlight good performance and resource efficiency of this approach over competitive baselines, including other recurrent models and a comparable sized Transformer. Further analyses reveal beneficial effects of the auxiliary loss on optimization and regularization, as well as extreme cases where there is little to no backpropagation.
1. Introduction
Long-term dependencies are difficult for RNNs because BPTT gradients can vanish or explode and storing intermediate states scales with sequence length. The paper adds an unsupervised auxiliary loss so truncated BPTT can remain effective on long sequences.
- BPTT gradients can vanish or explode, while storing intermediate hidden states requires memory proportional to sequence length.
- RNNs use O(l) training storage for fixed BPTT length l and O(1) inference storage, unlike architectures requiring storage tied to the full input.
- The method adds an unsupervised auxiliary loss that reconstructs or predicts a random sequence segment around an anchor point.
- The auxiliary loss improves LSTM optimization and generalization, while reducing the need for lengthy BPTT during training.
- Sequences of up to 16 000 elements were processed with faster, lower-memory auxiliary-loss training, whereas full-backpropagation LSTM training became very difficult.
2. Related works
Prior work addresses long-term dependencies through specialized recurrent architectures, optimization methods, convolutions, attention, and input skipping. This paper differs from recurrent pretraining work by studying auxiliary losses for longer sequences rather than short-sequence generalization.
- Proposed approaches include specialized recurrent structures, LSTMs, GRUs, multiplicative units, optimizers, identity or highway connections, unitary weights, convolutions, attention, and input skipping.
- Earlier recurrent pretraining studies used sequence autoencoders or language models mainly to improve generalization on short sequences.
- Other work combines auxiliary losses with truncated BPTT for online learning or reconstructs recurrent states using a backward-running network.
3. Methodology
The method samples random anchor positions and attaches auxiliary reconstruction or prediction losses to subsequences around them. It combines unsupervised pretraining with semi-supervised joint optimization while controlling auxiliary-segment sampling and loss computation.
- Random anchor positions receive unsupervised auxiliary losses for subsequences before or after the anchor.
- Reconstruction auxiliary loss: Reconstruction inserts the first token of a past subsequence into a decoder, which predicts the remaining tokens.
- Reconstruction auxiliary loss: Anchor points act as temporary memory for past events, allowing the classifier to use only a few supervised BPTT steps for final weight tuning.
- Prediction auxiliary loss: Prediction uses an auxiliary decoder to sequentially predict tokens in a subsequence beginning at the anchor point.
- r-LSTM and p-LSTM use unsupervised auxiliary pretraining followed by joint minimization of supervised and auxiliary losses; reconstruction uses Scheduled Sampling.
- Sampling frequency and subsequence length: Each sampled segment loss sums token losses, using cross-entropy for character sequences and L2 distance for continuous inputs.
- Sampling frequency and subsequence length: The method usually sets all segment lengths equal and samples one segment per example, reducing hyperparameter-tuning cost for long sequences.
4. Experiments
The experiments evaluate auxiliary-loss RNNs across image and language classification tasks spanning sequence lengths from 784 to 16,384. Benchmarks include MNIST variants, CIFAR10, StanfordDogs, and DBpedia character-level classification.
- The evaluation covers sequence lengths from 784 to 16384 across several datasets.The benchmark suite includes image sequences and a real language classification task.
- Pixel-by-pixel MNIST, permuted MNIST, and CIFAR10 test long-term dependency learning with sequential image pixels.Permuting pixels removes local structure and creates more complex dependencies.
- StanfordDogs provides eight image resolutions, producing controlled sequence lengths from 40×40 to 128×128 before flattening.The dataset contains images categorized into 120 dog breeds.
- DBpedia character-level classification supplies a large real-language benchmark with 560K training examples.The task follows the normalization procedure suggested in prior work.
4.1. Model Setup
The models use a main LSTM for supervised classification and auxiliary LSTMs for reconstruction or prediction, trained through unsupervised pretraining followed by joint optimization. Training keeps backpropagation bounded while comparing recurrent models with a similarly sized Transformer.
- A single-layer 128-cell LSTM reads sequences, and its final state feeds a two-layer 256-unit classifier.Dropconnect with probability 0.5 is applied to the classifier’s second layer.
- Auxiliary losses use a two-layer LSTM initialized from the main classifier’s current state.The auxiliary LSTM reconstruction model is trained with scheduled sampling.
- Gradients are truncated to 300 time steps for both supervised and auxiliary losses, while one auxiliary segment of length 600 is sampled per example.This keeps backpropagation cost constant as input length grows.
- The Transformer baseline uses an off-the-shelf configuration with approximately 0.5M weights and average-pooled outputs.Its outputs are passed to a two-layer feedforward classifier.
4.2. Main results
Auxiliary losses improve truncated and full-BPTT LSTMs across increasingly difficult image-sequence tasks, while preserving accuracy and reducing resource use on very long StanfordDogs sequences.
- MNIST, pMNIST, and CIFAR10: 11.3% accuracy leaves the truncated MNIST LSTM nearly untrainable, whereas auxiliary-loss r-LSTM and p-LSTM reach parity with fully trained recurrent models.The limitation arises because truncated gradients reach largely uninformative pixels near the sequence end.
- MNIST, pMNIST, and CIFAR10: On pMNIST, r-LSTM and p-LSTM outperform fully trained LSTM and uRNN baselines while using less than half the classification-loss gradients.The task tests more complex long-range dependencies than MNIST.
- MNIST, pMNIST, and CIFAR10: On CIFAR10, r-LSTM leads p-LSTM, while fully trained LSTM accuracy is more than 7% lower in absolute terms.The comparison reports a larger gap on this harder benchmark.
- MNIST, pMNIST, and CIFAR10: With full classification backpropagation, the proposed r-LSTM achieves the best accuracy across datasets against other recurrent models.The strongest margin over fully trained LSTM occurs on pMNIST and CIFAR10.
- StanfordDogs: At 9,000 elements, fully backpropagated LSTM produces no better than random predictions, and after 12,000 elements its memory limit is exceeded.By contrast, r-LSTM shows virtually no accuracy loss from 12,000 to 16,000 elements.
- StanfordDogs: Training time rises from 4 seconds at length 1,600 to 26 seconds at 12,000 for LSTM, versus under 3 seconds initially and about 8 seconds at over 16,000 for the proposed methods.These measurements use a single mini-batch of 128 examples.
4.3. Comparing with Transformer
The Transformer is competitive on shorter image sequences but degrades faster as sequences lengthen. Auxiliary-loss recurrent models remain feasible and outperform it beyond sufficiently long sequence lengths.
- MNIST, pMNIST, and CIFAR10: Transformer outperforms the best proposed model on MNIST and pMNIST, but its CIFAR10 performance drops below most recurrent models.The table reports test accuracy on all three datasets.
- StanfordDogs: On StanfordDogs, Transformer begins at nearly twice the accuracy of r-LSTM or p-LSTM but degrades much faster as sequences lengthen.The comparison uses the same transferred hyper-parameter settings.
- StanfordDogs: After the 3,000 mark, Transformer performs worse than the proposed methods and is only slightly better than random prediction around 9,200.Training becomes infeasible on the same resource after this point.
- The auxiliary-loss technique is described as orthogonal to most sequence-processing models, but its integration with scalable Transformer variants is left for future work.The experiments focus on improving recurrent networks.
4.4. Classifying DBpedia documents at character level
On character-level DBpedia classification, auxiliary losses make truncated LSTM training substantially more effective and competitive with strong recurrent baselines.
- Sequences average 300 characters, with 99% of training examples shorter than 600 characters.
- More than 10% absolute accuracy separates auxiliary-loss models with truncated BPTT from the LSTM baseline.
- Auxiliary-loss methods outperform truncated LM-LSTM and SA-LSTM baselines on DBpedia.
- With unrestricted model size and gradient truncation, p-LSTM performs on par with strong character-level baselines.
- r-LSTM with n = 20 and l = 15 significantly outperforms SA-LSTM’s full auto-encoder and ranks behind only Very-deep CNN with 29 layers.
5. Analysis
The analyses show that auxiliary losses improve performance under severe supervised-gradient limits, with effects tied to sampling, optimization, regularization, and joint training.
- 5.1. Shrinking supervised BPTT length: With only 50 supervised BPTT steps, r-LSTM and p-LSTM approximate fully trained LSTM accuracy; with one step, they reach 46.1% and 47.0%.
- 5.2. Multiple reconstructions with fixed BPTT cost: Accuracy peaks at 20 samples per sequence and declines when sampling is either too infrequent or too frequent.
- 5.2. Multiple reconstructions with fixed BPTT cost: At sampling frequency 20, single-step supervised BPTT gains 2.0%, while completely unsupervised main-LSTM training gains 4.0%.
- 5.3. Generalizing sampling settings: r-LSTM 20×30 closes the Transformer gap on shorter StanfordDogs sequences and outperforms recurrent models and Transformer by a large margin from 3000 onward.
- 5.4. Ablation Study: r-LSTM and p-LSTM gain early optimization advantages, while later semi-supervised loss minimization produces a regularization effect.
- 5.4. Ablation Study: Jointly training supervised and unsupervised losses is the most important ablated factor, with a loss exceeding 3.6%.
- 5.4. Ablation Study: Randomly sampling distant reconstruction segments provides almost a 2% accuracy gain, and extending segments beyond the anchor also improves results.
6. Conclusion
The paper concludes that auxiliary losses improve long-term dependency learning in RNNs by combining regularization with resource-efficient training. Across image and document benchmarks, the models remain competitive while using fewer resources on long sequences.
- Auxiliary losses regularize models that generalize to sequences up to length 16 000.
- Competitive performance can be obtained by backpropagating through only a small number of time steps, including cases with little or no backpropagation.
- Across pixel-by-pixel image and character-level document benchmarks, the models are competitive with strong recurrent and non-recurrent baselines.
- For long sequences, the models achieve superior results despite using much fewer resources.
- The authors anticipate applications in online learning systems and systems processing unusually long sequences.