Source-linked AI summary
Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness
Antônio H. Ribeiro, Koen Tiels, Luis A. Aguirre, Thomas B. Schön
TL;DR
The paper argues that gradient-centered accounts need refinement to distinguish attractor learning and cost-function geometry. It analyzes RNN attractors and reframes exploding gradients through cost-function smoothness; its discussion shows that eigenvalue constraints below one can produce vanishing gradients and, when training and inference modes match, prevent attractors.
Problem
Gradient-centered accounts do not fully distinguish vanishing gradients from attractor learning or characterize the cost-function geometry associated with exploding gradients.
Source: §1 · p. 1Method
The paper analyzes RNN attractors and reformulates exploding-gradient analysis in terms of cost-function smoothness, including higher-order derivatives.
Source: §1 · p. 1Results
Constraining RNN eigenvalues below one yields exponentially decreasing transients and vanishing gradients, and when training and inference modes match it also prevents attractors.
Source: §6 · p. 8Takeaways & Limitations
Learning attractors with matched training and inference requires exploring non-contractive parameter regions, where objective smoothness may be poor for traditional optimizers.
Source: §6 · p. 8The deterministic sine-wave task is limited in representing real-world sequence problems, which often have stochastic elements.
Source: §5.1 · p. 5
Abstract
from arXiv · showhide
The exploding and vanishing gradient problem has been the major conceptual principle behind most architecture and training improvements in recurrent neural networks (RNNs) during the last decade. In this paper, we argue that this principle, while powerful, might need some refinement to explain recent developments. We refine the concept of exploding gradients by reformulating the problem in terms of the cost function smoothness, which gives insight into higher-order derivatives and the existence of regions with many close local minima. We also clarify the distinction between vanishing gradients and the need for the RNN to learn attractors to fully use its expressive power. Through the lens of these refinements, we shed new light on recent developments in the RNN field, namely stable RNN and unitary (or orthogonal) RNNs.
1 Introduction
The paper argues that gradient-centered explanations of RNN training need refinement: attractor analysis clarifies long-term information storage, while cost-function smoothness extends exploding-gradient analysis to higher-order derivatives and complex optimization regions.
1 Introduction
- Unitary and orthogonal RNNs fix hidden-to-hidden eigenvalues at one to address exploding and vanishing gradients [Mhammedi et al., 2017; Vorontsov et al., 2017; Lezcano-Casado and Martínez-Rubio, 2019; Helfrich et al., 2018; Arjovsky et al., 2016; Jing et al., 2017; Maduranga et al., 2019; Wisdom et al., 2016; Lezcano-Casado, 2019].Stable RNNs instead constrain eigenvalues below one; projecting eigenvalues below one has been reported not to affect performance on some tasks [Miller and Hardt, 2019].Source: §1 · p. 1
- The paper distinguishes vanishing gradients from learning attractors, which are important for RNNs to store long-term information.Its numerical examples examine stable and orthogonal RNNs.Source: §1 · p. 1
- Cost-function smoothness extends exploding-gradient analysis to higher-order derivatives and regions with many close local minima that are difficult to navigate.The introduction contrasts these intricate regions with the conventional picture of high-curvature walls.Source: §1 · pp. 1–2
2 Recurrent neural networks
The section presents RNNs as state-transition and output systems, then describes training and inference inputs and parameter estimation through a cost function.
2 Recurrent neural networks
- An RNN maps a hidden state and input to the next state and a predicted output, a representation broad enough to include vanilla RNNs, LSTMs, GRUs, and stacked layers.The state is x_t, the input is z_t, and the parameters are θ.Source: §2 · p. 2
- Teacher forcing uses observed previous outputs as training inputs, while inference may instead feed the network's own predictions back into the model.The training and inference modes need not use the same transition and output functions.Source: §2 · p. 2
- Training estimates parameters by minimizing a cost function, commonly using squared error for regression or cross-entropy for classification.For multiple independent sequences, the objective may be a weighted average of their costs.Source: §2 · p. 2
3 Information in a recurrent network
The section connects RNN information retention to entropy and attractors, showing how contractive dynamics relate to vanishing gradients and how learning long-term memory can involve bifurcations.
3 Information in a recurrent network
- Vanishing gradients are commonly used to explain why RNNs struggle to learn long-term dependencies, but dynamic attractors also matter for storing information.The section treats these as related but distinct aspects of RNN behavior.Source: §3 · p. 2
3.1 Entropy of the internal states
- The entropy bound permits decay when L_f < 1, possible retention when L_f = 1 and the bound is tight, and growth when L_f > 1; information retention and vanishing gradients are distinct.L_f < 1 is sufficient for gradients to vanish, making escape from those parameter regions harder during training.Source: §3.1 · p. 3
3.2 Contractive vs non-contractive systems
- A contractive system has a unique fixed point in its contractive region, and its trajectories converge to that point; richer dynamics such as limit cycles and chaotic attractors are non-contractive.The section defines contractiveness for a given input sequence.Source: §3.2 · p. 3
3.3 Long-term memory
- An attractor can store long-term memory because a state in it remains there; nearby states in its basin converge back to it after sufficiently small disturbances.The basin condition describes robustness of this memory.Source: §3.3 · p. 3
- For tasks requiring long-term memory, an RNN may undergo bifurcations during training and learn attractors; stable and orthogonal RNNs use different mechanisms.Attractor diagrams show steady-state values as a function of a bifurcation parameter.Source: §3.3 · p. 3
4 Smoothness of the cost function
The paper connects RNN dynamics to cost-function smoothness, extending exploding-gradient analysis to higher-order derivatives and intricate regions with many local minima. An LSTM example and theoretical bounds relate these effects to the state-transition Lipschitz constant and simulation length.
4 Smoothness of the cost function
- Lower smoothness constants imply a less intricate cost function, allow optimization algorithms to take larger steps while still converging, and bound performance differences between close local minima.The smoothness measures are the Lipschitz constant of the cost and the Lipschitz constant of its gradient, also called β-smoothness.Source: §4 · p. 4
4.1 Example: chaotic LSTM
- For 0.9 ≲ s ≲ 1.45, the system exhibits rich nonlinear and chaotic steady-state behavior, corresponding to a region where the cost function is intricate and full of local minima.The RNN weights are scaled as θ(s) = sθtrue, where θtrue denotes the true data-generating weights.Source: §4.1 · p. 4
4.2 Theoretical results
- The cost and gradient Lipschitz constants connect internal RNN dynamics to exploding gradients and higher-order curvature, extending the analysis beyond first-order gradients.For Lf > 1, these quantities may grow exponentially or polynomially with maximum simulation length, producing intricate cost functions.Source: §4.2 · pp. 4–5
- The cost-function Lipschitz bound is O(N) when Lf = 1 and O(1) when Lf < 1; the gradient has a separate bound when the Jacobians are Lipschitz.The theorem assumes Lipschitz transition and output functions on a compact, convex set with an invariant set; it covers specified squared-error and cross-entropy losses.Source: §4.2 · p. 4
5 Learning attractors during training
The paper compares LSTM, stable LSTM, and orthogonal RNN behavior across sine-wave generation, sequence classification, and word-level language modeling. It analyzes training through attractors as well as task performance.
5.1 Sine-wave generation
- The stable LSTM performs poorly on sine-wave generation because its contractive dynamics retain only a single stable attractor, while oscillations require sustained periodic behavior.The LSTM develops a periodic solution through a bifurcation; the orthogonal RNN also learns the periodic attractor despite its eigenvalue constraint.Source: §5.1 · p. 5
5.2 Sequence classification based on few relevant symbols
- For sequence length 200, the orthogonal RNN solves the task with smoother convergence than the unsteady LSTM, while the stable LSTM fails at lengths of 200 or more.The standard LSTM's convergence depends strongly on initialization; the stable LSTM eventually solves the task only at a slow linear rate for shorter sequences.Source: §5.2 · p. 6
- The LSTM develops multiple fixed points and periodic behavior during sequence classification, whereas the orthogonal RNN shifts a persistent cloud of fixed points that the output layer learns to weigh.The LSTM's periodic behavior under distractor input is spurious because distractors do not determine the output.Source: §5.2 · p. 7
5.3 Word-level language model
- With constant input in the language-model analysis, the LSTM exhibits varied dynamics, while the orthogonal RNN and stable LSTM converge to a single fixed point.The LSTM's training curve is smooth in this task, and its chaotic behavior under constant input may have limited influence on output.Source: §5.3 · p. 8
6 Discussion
The discussion distinguishes vanishing gradients from attractor-based memory and connects both to training and inference behavior. It argues that cost-function smoothness and attractor analysis can clarify RNN methods.
6 Discussion
- Eigenvalues below one produce exponentially decreasing transients and vanishing gradients; with identical training and inference modes, they also prevent nontrivial attractors.The paper argues that conflating these effects limits analysis of RNN training mechanisms.Source: §6 · p. 8
- When training and inference modes match, learning nontrivial attractors requires exploring non-contractive parameter regions, where objective smoothness may be poor for traditional optimization.Attractors are needed to store information non-volatilely.Source: §6 · p. 8
- The paper's artificial symbol-classification benchmark, like adding, multiplication, and copy-memory tasks, assesses attractor learning and exploration of non-contractive regions; its results reinforce oRNN effectiveness on this class of problems.Source: §6 · p. 8
- Teacher forcing can make training and inference dynamics differ, allowing attractors during inference even when training remains in contractive regions.The discussion suggests this distinction may help explain stable-model performance on some tasks.Source: §6 · p. 8
7 Related work
The related-work discussion positions the paper's contribution as analyzing attractors during training, extending prior attractor analyses conducted after training.
7 Related work
- Prior RNN attractor analyses focus on post-training behavior, whereas this paper uses attractors to analyze the training procedure.Source: §7 · p. 8
- The paper extends earlier long-term-memory analyses by considering chaotic and periodic attractors and the possibility of different training and inference modes.It contrasts this with work that frames memory around single fixed points and a trade-off with robust information retention under noise.Source: §7 · p. 8
- In the examples, the authors do not observe the connection between bifurcations and performance jumps conjectured by Pascanu et al. (2013).Source: §7 · pp. 8–9
Supplementary material - Beyond exploding and vanishing gradients:
The supplement derives parameter sensitivities of recurrent outputs by applying the chain rule to the recurrent state and output equations.
A Computing the derivatives
- The derivative appendix recursively computes hidden-state and predicted-output derivatives with respect to parameters using the state and output Jacobians.The state derivative follows D_(t+1) = A_tD_t + B_t, with D_0 = 0; the output derivative is J_t = C_tD_t + F_t.Source: A Computing the derivatives · p. 11
B Proofs
The appendix establishes an entropy lower-bound result under regularity assumptions, then supplies preliminary Lipschitz properties for the losses used in the proofs.
B.1 Entropy lower bound
- The entropy argument assumes a one-to-one continuously differentiable state map and Lipschitz dynamics on a compact, convex domain.The proof changes variables through the state transition and then applies Hadamard’s inequality.Source: §B.1 · p. 11
B.2 Preliminary results
- The preliminary results establish Lipschitz bounds for sums and products of functions, with product bounds also requiring bounded factors.These lemmas provide tools for controlling composed quantities in the later proofs.Source: §B.2 · p. 11
- For squared-error and binary cross-entropy losses, the appendix states bounds on loss differences and specifies the form of the first loss derivative.The binary cross-entropy case assumes target values in {0, 1}.Source: §B.2 · p. 11
B.3 Proof of Theorem 1 (a)
The proof compares trajectories generated from different parameters and initial states, using Lipschitz bounds to control their outputs and derive the stated asymptotic result.
B.3 Proof of Theorem 1 (a)
- The proof begins with two trajectories generated from distinct parameter vectors and initial conditions, then recursively applies Lipschitz bounds on the dynamics and outputs.The comparison is made over the same inputs.Source: §B.3 · p. 12
- Compactness and an invariant-set premise bound the outputs, with the proof stating that their bound scales as O(S(t)).A simple way to satisfy the premise is to choose f so that f(0, u_t; 0) = 0.Source: §B.3 · p. 12
- The proof combines the trajectory and loss bounds, then analyzes their asymptotic behavior in N to obtain Eq. (6).The supplied passages do not state the result’s formula.Source: §B.3 · p. 12
B.4 Proof of Theorem 1 (b)
The appendix bounds changes in the proof quantities for nearby parameter and initial-state choices; it also illustrates an intricate LSTM loss landscape with neighboring local minima.
B.4 Proof of Theorem 1 (b)
- The proof bounds changes in J_t and J_tΨ(ŷ_t) using Lipschitz assumptions on the dynamics, outputs, and Jacobian.The resulting bounds scale with the difference between the parameter and initial-state vectors, and their asymptotic analysis yields Eq. (7).Source: §B.4 · p. 12
C Chaotic LSTM example
- The LSTM example evaluates mean-square loss on data generated by the same model while varying parameters along a grid interpolated and extrapolated from θtrue and θrandom.The parameterization also includes zero parameter values.Source: C Chaotic LSTM example · p. 13
- An intricate region around θtrue contains many neighboring local minima, making it hard for optimization algorithms to navigate.The example uses the same initial conditions for the data-generating model and the loss evaluations.Source: C Chaotic LSTM example · p. 13
D Additional Experiments
The additional experiments visualize loss landscapes, initialization-dependent accuracy, and training trajectories for LSTM, stable LSTM, and orthogonal RNN models.
Additional experiments
- The chaotic-LSTM figure plots mean-square cost over parameter vectors formed from θtrue and θrandom.The parameter vectors are θ(s1, s2) = s1θtrue + s2θrandom.Source: Figure 1 · p. 14
- Different random initializations produce either abrupt convergence to good training and validation accuracy or slow convergence to high training accuracy with validation no better than guessing.In the latter case, training accuracy exceeds 90%.Source: Figure 2 · p. 14
- Bifurcation diagrams track output or internal-state projections across epochs for LSTM, oRNN, and stable LSTM language and symbol-classification experiments.The language-model diagrams distinguish constant inputs from inputs based on the previous highest-probability prediction; some projections follow a 1500-sample burnout period.Source: Figure 3 · p. 15, Figure 4 · p. 16, Figure 5 · p. 17