PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Recurrent Models

John Miller, Moritz Hardt

arXiv:1805.10369v4cs.LGstat.ML

TL;DR

Stability has had little impact on recurrent-network practice, raising questions about whether sequence modeling is inherently unstable. The paper investigates stable recurrent models theoretically and empirically, finding that they can be approximated by feed-forward networks and often perform as well as unstable counterparts.

  • Problem

    Stability has had little impact on recurrent-network practice, leaving unresolved whether sequence modeling in practice is inherently unstable.

    Source: §1 · p. 1

  • Method

    The paper investigates stable recurrent models theoretically and empirically, including their feed-forward approximations for inference and training by gradient descent.

    Source: §1 · p. 1

  • Results

    Across varied sequence tasks, recurrent models can often be made stable without loss in performance, and nominally unstable models often operate stably on the data distribution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Stable models cannot have long-term memory, so similar results from stable and unstable models imply either that the task does not require it or that the unstable model lacks it.

    Source: §1 · p. 2

    The present proof techniques rely on truncated recurrent models, and their feed-forward approximations use k times as many weights; more parsimonious approximations and training-time approximation guarantees for natural feed-forward models remain open.

    Source: §5 · p. 12

Abstract

from arXiv · show

Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks. Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime. Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.

1 Introduction

Stability has had little impact on recurrent-network practice, despite its importance for dynamical systems. The paper investigates stable recurrent models theoretically and empirically, finding that stable models can often match unstable counterparts and are approximable by feed-forward networks for inference and training.

1 Introduction

  • A recurrent sequence that cannot be approximated by a feed-forward model must come from an unstable model, whose gradients must explode and can impede learning by gradient descent.The paper notes that exploding gradients are generally an impediment to learnability via gradient descent.Source: §1 · p. 2
  • Across varied sequence tasks, recurrent models can often be made stable without performance loss, and nominally unstable models often operate stably on the data distribution.Stable models cannot have long-term memory, so similar results mean either the task does not require it or the unstable model does not have it.Source: §1 · p. 2
  • Similar performance between stable and unstable models suggests either that a task does not require long-term memory or that the unstable model does not have it.These findings help explain why feed-forward replacements have succeeded in applications including translation [6] [26], speech synthesis [25], and language modeling [5].Source: §1 · p. 2

1.1 Contributions

  • The paper gives a generic stability definition, sufficient conditions for common models including LSTMs, and an efficient projection operator for enforcing stability during training.Its experiments cover sequence benchmarks and examine whether stable models incur a performance cost.Source: §1.1 · p. 2
  • Under the stability assumption, feed-forward networks approximate recurrent networks for both inference and training by gradient descent.The training result relies on non-trivial stability properties of gradient descent.Source: §1 · p. 1, §1.1 · p. 2

2 Stable Recurrent Models

The paper treats recurrent models as nonlinear dynamical systems and defines stability through non-exploding gradients, then gives conditions for common architectures. It also describes how to impose those conditions during training, while noting that the LSTM conditions are restrictive.

2.1 Defining Stable Recurrent Models

  • A recurrent model updates its hidden state through a differentiable transition map, and stability is motivated as the regime where training-objective gradients cannot explode.The formal definition is contraction in the hidden state, with a contraction factor below one.Source: §2.1 · pp. 2–3
  • An unstable system can make gradient descent fail to converge to a stationary point while the parameter-gradient norm grows without bound.The paper presents this as an example of the difficulty of analyzing gradient descent for unstable models.Source: §2.1 · p. 3

2.2 Examples of Stable Recurrent Models

  • Stability conditions can be enforced by projecting parameters after each gradient-descent iteration, though the conditions given for LSTMs are restrictive and the original LSTM’s stability remains unresolved.The authors report that models satisfying the conditions still perform well on several tasks.Source: §2.2 · pp. 3–4
  • For linear systems and tanh recurrent networks, stability follows when the recurrent weight matrix has spectral norm below one; training can enforce this by projection onto the spectral-norm ball.More generally, a Lipschitz nonlinearity with constant Lρ requires ∥W∥ < 1/Lρ.Source: §2.2 · p. 3
  • For LSTMs, the paper gives sufficient norm bounds that make an iterated transition contractive when the iteration count is O(log(d)).The conditions use induced ℓ∞ matrix norms and bounded weights and inputs; the authors describe them as restrictive and leave other stable regimes open.Source: §2.2 · p. 4

3 Stable Recurrent Models Have Feed-forward Approximations

The section establishes that stable recurrent models can be approximated by finite-context feed-forward models for inference and gradient-descent training. The approximation relies on truncation and holds under stated stability, smoothness, and Lipschitz conditions.

3.1 Truncated recurrent models

  • The feed-forward approximation is formed by truncating recurrence to a finite context: a sliding input window produces an output at each step.The truncated model can be implemented as a depth-k autoregressive feed-forward network with significant weight sharing.Source: §3.1 · p. 5

3.2 Approximation during inference

  • Under contraction and Lipschitz conditions, truncating distant inputs keeps hidden states close enough for Lipschitz prediction functions to yield nearly identical outputs.The paper characterizes stable models as lacking long-term memory: distant inputs do not change system states.Source: §3.2 · p. 5

3.3 Approximation during training via gradient descent

  • Tracking weight divergence shows that a context of approximately O(log(N/ε)) keeps recurrent and truncated models’ weights within ε after N gradient-descent steps.The proof argues that gradient descent is itself stable, preventing small differences between gradients from accumulating into divergent weights.Source: §3.3 · pp. 5–6
  • With sufficiently large context, projected gradient descent with step size α_t = 1/t makes recurrent and truncated models’ predictions differ by at most ε after N steps.The theorem assumes bounded inputs and smooth, Lipschitz loss, recurrence, and prediction functions; the required context grows as Ω(log(γNβ/ε)). The theorem’s O(1/t) decay is faster than many learning rates used in practice.Source: §3.3 · p. 6

4 Experiments

Experiments compare stable and unstable recurrent models across four sequence benchmarks. Stable models generally retain performance, while further analyses examine data-dependent stability, short-sequence benefits of instability, and truncation.

4.1 Tasks

  • Benchmarks span word- and character-level language modeling, polyphonic music, and slot-filling, measured by perplexity, bits per character, negative log-likelihood, and F1, respectively.The tasks use WikiText-2, Penn Treebank, JSB Chorales, and the ATIS benchmark.Source: §4.1 · p. 7

4.2 Comparing Stable and Unstable Models

  • Unconstrained RNNs and LSTMs were retrained with projected gradient descent to enforce stability, without retuning hyperparameters.The RNN constraint bounds the recurrent matrix norm; LSTM weight rows are normalized to satisfy sufficient stability conditions.Source: §4.2 · p. 7
  • Across tasks, stable and unstable RNNs perform similarly; LSTMs match on music and slot-filling but show a small language-modeling gap.The comparisons use held-out test metrics.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 What is the “price of stability” in sequence modeling?

  • Under a weaker data-dependent test, character-level LSTMs are iteratively stable for about 80 steps and word-level LSTMs for about 100.The test uses data inputs and hidden states reachable via gradient descent; it indicates that nominally unstable language models operate in the stable regime on the data distribution.Source: §4.3 · p. 9
  • For short music sequences, an unstable model with λ≈3.5 achieves better test likelihood; at longer lengths, λ≈1.5 and the gain disappears.The reported comparison varies the truncation parameter k.Source: §4.3 · p. 9

4.4 Unstable Models Operate in the Stable Regime

  • Unstable language-model RNNs and LSTMs also have vanishing gradients, with LSTM gradients decaying more slowly than RNN gradients.The gradient norm with respect to earlier inputs is averaged over the training set as the distance to the loss increases.Source: §4.4 · p. 10, Figure 3 · p. 10
  • On language and music tasks, increasing truncation length yields diminishing returns; larger k leaves LSTM performance unchanged and slightly degrades unstable RNN performance.These diminishing returns imply truncation, and thus feed-forward approximation, is possible even for the unstable models.Source: §4.4 · p. 11, Figure 4 · p. 11
  • For unstable word-level language models, truncated-model weights diverge slowly from the k=65 proxy after an initial increase, with slower growth at larger k.Approximation accuracy also has diminishing returns as k increases.Source: §4.4 · p. 11

5 Are recurrent models truly necessary?

The experiments suggest recurrent models trained in practice operate in the stable regime, and the theoretical results show stable recurrent models can be approximated by feed-forward networks. The paper conjectures that recurrent networks trained in practice are therefore approximable by feed-forward networks, but identifies unresolved costs and proof requirements.

5 Are recurrent models truly necessary?

  • The paper conjectures that recurrent networks trained in practice are approximable by feed-forward networks, based on its experimental and theoretical results.The experiments indicate practical recurrent models operate in the stable regime, and the theory establishes feed-forward approximation for stable recurrent models.Source: §5 · p. 11
  • For unstable word-level language models, the distance between truncated-model weights and a k=65 full-model proxy grows slowly after an initial increase, with slower growth at larger k.The authors assume k=65 well-captures the full recurrent model.Source: Figure 5 · p. 12
  • Feed-forward replacement is not established: truncation may provide useful inductive bias, and its implementation increases the number of weights by a factor of k.The authors call for more parsimonious feed-forward approximations and proofs that natural architectures can approximate stable recurrent models during training.Source: §5 · p. 12

6 Related Work

Prior work studies learning stable dynamical systems, stability conditions, and recurrent alternatives; this work extends stable-system approximation to nonlinear models and training, and examines stable recurrent models empirically.

6 Related Work

  • Earlier work showed gradient descent can learn classes of stable linear and nonlinear systems, while this paper examines theoretical and empirical consequences of that stability assumption.Related approaches include moment-based learning for some stable nonlinear RNN classes.Source: §6 · p. 12
  • For linear systems, [24] connect stability with truncation; this work extends the approximation result to nonlinear systems and analyzes truncation’s effect on gradient-descent training.The training-time analysis builds on [8] but uses stability analysis for a different purpose.Source: §6 · p. 12
  • Prior work gives stability conditions for one-layer RNNs; this work adds conditions for LSTMs and studies how stability enables feed-forward approximation.The prior conditions use the norm of W and the nonlinearity’s Lipschitz constant.Source: §6 · pp. 12–13
  • Unitary and soft-isometry approaches constrain recurrent weights to address gradient issues; this work argues much practical sequence learning, including with nominally unstable models, occurs in the stable regime.The cited work reports strict unit-norm constraints reduce performance on several tasks, while soft-isometry relaxes that constraint.Source: §6 · p. 13
  • Prior empirical studies examined non-chaotic recurrent models and feed-forward sequence models; this work finds recurrent models can often be replaced by stable recurrent models equivalent to feed-forward networks.The cited evaluation found feed-forward models outperforming recurrent counterparts in diverse settings.Source: §6 · p. 13

A Proofs from Section 2

A scalar linear-system example shows that gradient descent need not converge on an unstable system: outside the stable range, its iterates can diverge.

A.1 Gradient descent on unstable systems need not converge

  • The example uses an identity-mapping target with unit inputs, squared loss, b = 1, and learning the parameter a = 0.Stability in this scalar system corresponds to |a| < 1.Source: §A.1 · p. 16
  • If the initial parameter a0 lies outside [−1, 1], gradient descent with step sizes αi = 1/i diverges, so its iterates are not stationary points.For large sequence length, the gradient grows approximately exponentially when |a| > 1.Source: §A.1 · p. 16

A.2 Proofs from section 2.2

The appendix establishes contraction-based stability conditions for recurrent neural networks and LSTMs, with boundedness assumptions used in the LSTM argument.

A.2.1 Recurrent neural networks

  • For recurrent neural networks, the proof assumes ∥W∥ ≤ λ < 1 and uses tanh’s Lipschitz and smoothness properties to establish parameter regularity.It shows the model and its parameter derivative are Lipschitz under the stated bounds.Source: §A.2.1 · p. 17

A.2.2 LSTMs

  • For LSTMs, bounded weights and inputs imply bounded cell states, supplying the condition needed for the state-transition stability argument.The cell-state bound is ∥ct∥∞ ≤ 1/(1−∥f∥∞).Source: §A.2.2 · p. 18
  • The LSTM proof establishes contraction in the ℓ∞ norm and then considers contraction in the ℓ2 norm for reachable hidden states.The argument uses bounded cell states and the Lipschitz properties of sigmoid and tanh.Source: §A.2.2 · pp. 18–19

B Proofs from section 3

The proof assumes a zero initial state and stable transition maps over a compact, convex parameter domain, then bounds differences between hidden states by unrolling the dynamics.

B Proofs from section 3

  • The proof assumes h0 = 0, φw(0, 0) = 0, and a compact, convex parameter domain where each transition map is stable.The zero-at-origin condition can be achieved by reparameterization without changing expressivity.Source: B Proofs from section 3 · p. 19
  • Applying stability and Lipschitz assumptions and summing a geometric series yields a bound used in the truncation argument.The proof then unrolls k updates to bound hidden-state differences and solves for k.Source: B Proofs from section 3 · p. 19
  • The truncation depth is obtained by solving for k after unrolling the hidden-state iterates and applying the preceding bound.The argument connects recurrent-state differences to a finite-step approximation.Source: B Proofs from section 3 · p. 19

B.1 Proofs from section 3.3

The appendix bounds the gradient difference caused by truncating a stable recurrent model and establishes smoothness properties used in the gradient-descent analysis. These results support the proof of the main gradient-descent result.

B.1 Proofs from section 3.3

  • The proof assumes four smoothness conditions on φ_w for reachable states and weights, with corresponding smoothness scalars.Source: §B.1 · p. 20

B.1.1 Gradient difference due to truncation is negligible

  • For sufficiently large truncation length k, the gradient difference is O(kλ^k), independent of sequence length, because long-term gradient components vanish and short-term components remain to be bounded.The argument separates the long-term components of the recurrent-model gradient from the short-term difference between full and truncated gradients.Source: §B.1.1 · p. 20

B.1.2 Stable recurrent models are smooth

  • Under the stated contraction and smoothness conditions, the proof bounds the difference between hidden-state trajectories in terms of the weight difference and controls the resulting gradient variation.The proof states that the relevant sum is O(1/(1 − λ)^3).Source: §B.1.2 · pp. 22, 24

B.1.3 Gradient descent analysis

  • The gradient-descent analysis uses a recurrence for parameter divergence, applies the truncation and smoothness lemmas, and bounds the inner sum via an integral.Source: §B.1.3 · pp. 24–25

B.1.4 Proof of theorem 1

The proof of Theorem 1 bounds the relevant terms using the loss’s Lipschitz property, the triangle inequality, and Lemma 5. It then chooses k so that both terms are below ε/2.

B.1.4 Proof of theorem 1

  • Theorem 1’s proof uses the L_f-Lipschitz property of f and the triangle inequality to bound the quantity under consideration.Source: §B.1.4 · p. 25
  • The proof bounds its first term using Lemma 5.Source: §B.1.4 · p. 25
  • Choosing k so that both terms are below ε/2 yields the stated result.Source: §B.1.4 · p. 25

C Experiments

On random Gaussian instances, the observed parameter error scales comparably with Proposition 4’s bound, while larger step sizes undermine that bound. The experiments also describe how stability is enforced or estimated and restrict evaluation to single-layer models trained with SGD.

C Experiments

  • Observed parameter error scales comparably with Proposition 4’s bound; larger α/t or constant-α step sizes make the bound fail, suggesting the O(1/t) rate may be necessary.Source: C Experiments · pp. 25–26, Figure 6 · p. 26
  • The experiments use single-layer RNNs or LSTMs and vanilla SGD, because the stated sufficient stability conditions are single-layer and the theory applies to SGD rather than adaptive methods.Source: C Experiments · p. 26
  • Stability is enforced by projecting RNN weights onto the spectral-norm ball and constraining LSTM parameters; the specified LSTM constraints yield a stable iterated model.The RNN projection uses an SVD after each gradient update; the LSTM implementation normalizes weight-matrix rows.Source: C Experiments · pp. 26, 28
  • For LSTMs, whose stability parameter cannot be computed analytically here, the authors estimate it by maximizing S(h, h′, x) with gradient ascent from 20 random initializations.They use training-set inputs, 1,000 steps per initialization, and take the maximum observed value as the estimate.Source: C Experiments · p. 28
Loading 1805.10369v4…