Source-linked AI summary
Advances in Optimizing Recurrent Networks
Yoshua Bengio, Nicolas Boulanger-Lewandowski, Razvan Pascanu
TL;DR
Learning long-term dependencies is difficult because recurrent computation repeatedly transforms information across time, producing vanishing or exploding gradients and challenging optimization. The paper reviews and combines training improvements, including gradient clipping, leaky integration, advanced momentum, richer output models, and sparse gradients, and evaluates them on text and music. Across the reported experiments, the enhanced methods generally improve training and test error and can make SGD competitive with Hessian-Free optimization.
Problem
Learning long-term dependencies is difficult because recurrent gradients can vanish or explode across time, making optimization challenging.
Method
The paper reviews and combines gradient clipping, leaky integration, advanced momentum, richer output probability models, and sparse gradients for training RNNs.
Results
The techniques generally improve training and test error on text and music data and can make SGD competitive with Hessian-Free optimization.
Takeaways & Limitations
Enhanced SGD can compete with or improve on a second-order method that was found particularly effective for RNN training.
Takeaways & Limitations
The leading-eigenvalue condition is necessary but not sufficient, and the equivalent parameter formulation assumes zero initial and converged velocity.
Abstract
from arXiv · showhide
After a more than decade-long period of relatively little research activity in the area of recurrent neural networks, several new developments will be reviewed here that have allowed substantial progress both in understanding and in technical solutions towards more efficient training of recurrent networks. These advances have been motivated by and related to the optimization issues surrounding deep learning. Although recurrent networks are extremely powerful in what they can in principle represent in terms of modelling sequences,their training is plagued by two aspects of the same issue regarding the learning of long-term dependencies. Experiments reported here evaluate the use of clipping gradients, spanning longer time ranges with leaky integration, advanced momentum techniques, using more powerful output probability models, and encouraging sparser gradients to help symmetry breaking and credit assignment. The experiments are performed on text and music data and show off the combined effects of these techniques in generally improving both training and test error.
1. INTRODUCTION
RNNs can represent nonlinear sequence relationships, but learning long-term dependencies makes their optimization difficult. The paper reviews and combines techniques that generally improve training and generalization, sometimes making SGD competitive with second-order methods.
- RNN sequence modeling: RNNs iteratively summarize past observations through nonlinear state transitions governed by tunable parameters.Their state is computed from the current input and previous state, while a loss is defined against observed outputs.
- Optimization difficulty: As dependency spans exceed tens or hundreds of steps, few random initializations reach a sufficiently good solution under local optimization.The proportion of trials entering a good basin of attraction rapidly decreases as temporal dependency length grows.
- Paper focus: The paper reviews and combines several techniques intended to improve RNN training and optimization.The techniques are presented as a response to difficulties surrounding long-term dependency learning.
- Reported outcome: The combined techniques generally improve both training and generalization performance and can let online SGD compete with batch or large-minibatch second-order methods.The comparison includes Hessian-Free optimization, which had been found to help RNN training.
2. LEARNING LONG-TERM DEPENDENCIES AND THE OPTIMIZATION DIFFICULTY WITH DEEP LEARNING
Deep and recurrent networks face optimization difficulties when repeated nonlinear transformations must preserve information across long temporal spans. Vanishing or exploding Jacobian products and poorly conditioned objectives make long-term credit assignment difficult.
- Deep learning optimization: Deep learning optimization is difficult because deeper networks compose nonlinearities across many representation levels, producing highly nonlinear objectives.The text also describes many apparent local minima and qualitatively different regions reached by different initializations.
- RNNs as deep networks: RNNs become deep networks with shared weights when unfolded through time, linking their training difficulties to deep-learning optimization problems.Hessian-Free optimization has extended the temporal dependency span that RNNs can learn, suggesting ill-conditioning contributes to the difficulty.
- Gradient propagation: Long-range gradients decompose into products of Jacobians whose magnitudes tend to vanish or explode as the temporal gap increases.These products weight effects across different time spans in the total gradient.
- Vanishing gradients: Vanishing products make long-term effects exponentially smaller than short-term effects, hindering their capture.The problem arises as the temporal separation between the linked events increases.
- Exploding gradients: Exploding gradients can produce destructive finite gradient-descent updates in steep regions of the training criterion.The gradient may be locally correct while finite steps move parameters toward poor configurations.
- Conditions and caveats: A leading eigenvalue above one is necessary but not sufficient for the desired long-term-dependency regime, which also requires gradients not to explode.The paper explicitly qualifies the eigenvalue condition as necessary rather than sufficient.
3. ADVANCES IN TRAINING RECURRENT NETWORKS
The paper reviews training advances that address exploding and vanishing gradients, underfitting, and inefficient credit assignment in recurrent networks. It also derives a simplified Nesterov formulation and relates its update behavior to improved RNN performance.
- Gradient clipping: Gradient clipping limits destructive parameter updates when exploding gradients create steep regions in the objective.The method clips gradients above a threshold while preserving a descent direction.
- Longer temporal paths: Leaky integration assigns different units slow or fast dynamics, allowing signals and gradients to propagate farther through time.The update uses α_i values sampled from (0.02, 0.2); standard RNNs correspond to α_i = 0.
- Output modeling: NADE output models capture high-order dependencies among multivariate outputs in the music experiments.The paper uses NADE as a more powerful output probability model for music data.
- Sparse gradients: Rectifier units with L1 penalties encourage sparser hidden-unit gradients, which may help break symmetries and assign credit less uniformly.The proposed motivation is that sparse gradients can make hidden units specialize more easily.
- Nesterov momentum: Nesterov accelerated gradient evaluates the gradient at a look-ahead parameter, differing from regular momentum in its evaluation point and update coefficients.The new formulation commits to peeked-ahead parameters and backtracks before updating.
- Nesterov momentum: The reformulation stores past velocities with higher momentum while using them more conservatively, a mechanism suspected to support empirical performance.Under zero initial and converged velocity, the transformed parameters are equivalent to the original parameters.
4. EXPERIMENTS
The experiments combine gradient clipping, leaky units, rectifier units with L1 penalties, and Nesterov momentum across symbolic music and text prediction tasks. These enhancements generally improve training and test performance over regular SGD.
- Experimental design: The study compares vanilla SGD with combinations of clipping, leaky integration, rectifier units with L1 penalties, and Nesterov momentum.The labels C, L, R, and M identify these enhancements in the experiments.
- Symbolic music: The music evaluation uses four polyphonic datasets spanning piano, folk, orchestral, and Bach chorale sequences.The symbolic data represent pitch and timing in high-dimensional piano-roll matrices with complex temporal dependencies.
- Experimental design: Hyperparameters are selected by random search using validation log-likelihood, with final performance reported on the test set.The search covers hidden units, regularization, nonlinearities, momentum, learning rate, and leaky-unit settings.
- Music results: The music results show that the enhancements improve on regular SGD in almost all cases and make SGD competitive with HF for sigmoid recognition layers.Table 1 reports log-likelihood and expected frame-level accuracy for the evaluated RNNs.
- Text experiments: The text experiments evaluate word and character prediction on the Penn Treebank corpus using different sequence lengths and model configurations.Word prediction uses 40-step gradient sequences, while character prediction uses 150-step sequences and 500 hidden units.
- Text results: Substantial improvements appear in both training and test perplexity, suggesting that the techniques make recurrent-network optimization easier.Table 2 reports entropy in bits per character and perplexity for next-character and next-word prediction.
5. CONCLUSIONS
Experiments attribute part of RNN training difficulty to rough error surfaces that SGD handles poorly. Incremental improvements to SGD generally improve training and test error, sometimes matching or surpassing Hessian-Free optimization.
- Rough error surfaces contribute to RNN training difficulty and are not easily handled by SGD.
- Incremental improvements to SGD generally improve both training and test error in the experiments.
- Enhanced SGD competes with or improves on Hessian-Free optimization, a second-order method effective for RNNs.