PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz

arXiv:1806.05394v2stat.MLcs.LG

TL;DR

Gating’s role in recurrent-network trainability is not fully understood, so the paper analyzes signal propagation after random initialization with mean field and random matrix theory. Using vanilla RNNs and a simpler minimalRNN, it finds that gating creates a broader trainable region and that critical initialization improves training dynamics.

  • Problem

    The paper addresses why gated recurrent networks train substantially better than vanilla RNNs and seeks to distinguish trainability benefits from increased expressivity.

    Source: §1 · p. 1

  • Method

    The authors combine mean field theory and random matrix theory to analyze signal propagation, gradient propagation, and dynamical isometry in vanilla RNNs and minimalRNNs.

    Source: §1 · p. 2

  • Results

    Gated recurrent networks have a broader, more robust trainable region than vanilla RNNs, while closed-form critical initialization achieves dynamical isometry and improves training.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The theory predicts recurrent-network trainability and supports gating and critical initialization as mechanisms for more reliable training.

    Source: §1 · p. 2, §6 · p. 8

    The recurrent analysis approximates weight sharing; ignoring tied weights changes quantitative agreement with experiments but not the qualitative picture.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the exact mechanism that enables such remarkable performance is not well understood. We develop a theory for signal propagation in recurrent networks after random initialization using a combination of mean field theory and random matrix theory. To simplify our discussion, we introduce a new RNN cell with a simple gating mechanism that we call the minimalRNN and compare it with vanilla RNNs. Our theory allows us to define a maximum timescale over which RNNs can remember an input. We show that this theory predicts trainability for both recurrent architectures. We show that gated recurrent networks feature a much broader, more robust, trainable region than vanilla RNNs, which corroborates recent experimental findings. Finally, we develop a closed-form critical initialization scheme that achieves dynamical isometry in both vanilla RNNs and minimalRNNs. We show that this results in significantly improvement in training dynamics. Finally, we demonstrate that the minimalRNN achieves comparable performance to its more complex counterparts, such as LSTMs or GRUs, on a language modeling task.

1. Introduction

RNNs are widely used, but vanilla models are difficult to train and gated variants outperform them despite an incomplete explanation of gating’s benefits. This paper extends signal-propagation theory to recurrent networks and finds that gating broadens trainability while critical initialization improves training.

Motivation

  • Gated RNN variants such as LSTMs and GRUs perform significantly better than vanilla RNNs, but the mechanism behind gating’s success remains insufficiently explained.Prior evidence suggests gating benefits trainability more than capacity or expressivity.Source: §1 · p. 1

Dynamical isometry

  • Dynamical isometry requires finite outputs, input sensitivity, and gradients that neither explode nor vanish, and well-conditioned Jacobians can accelerate training by orders of magnitude.These conditions motivate the paper’s recurrent analysis and initialization strategy.Source: §1 · pp. 1–2

Approach

  • The paper combines mean field theory and random matrix theory to study signal and gradient propagation in vanilla RNNs and the simpler gated minimalRNN.The framework is intended to clarify gating’s role in recurrent trainability and potentially extend to more complex gated architectures.Source: §1 · p. 2

Contributions

  • The theory identifies a maximum signal-propagation timescale, relates forward signal propagation to backward gradients, and predicts the maximum trainable sequence length.Theory and practice show remarkable alignment across recurrent architectures.Source: §1 · p. 2

Main findings

  • Vanilla RNNs achieve dynamical isometry only along a one-dimensional boundary with orthogonal weights, whereas minimalRNNs have a robust multidimensional region of suitable initializations.The contrast supports gating as a source of more robust signal propagation and trainability.Source: §1 · p. 2

2. Related Work

Related work connects this paper to initialization strategies, gradient stabilization, and mean field analyses of neural networks. The paper distinguishes dynamical isometry from orthogonality and extends mean field techniques to recurrent networks with gating.

Initialization and orthogonality

  • Identity and orthogonal initialization aim to improve trainability, while other approaches constrain recurrent transition matrices throughout learning.Strict orthogonality can hinder training speed and generalization performance.Source: §2 · p. 2
  • Orthogonal weight matrices do not guarantee dynamical isometry because nonlinearities affect Jacobian conditioning, and orthogonality has little impact on gated-RNN Jacobians.The paper therefore targets dynamical isometry rather than orthogonality alone.Source: §2 · p. 2

Gradient stabilization

  • Edge-of-chaos initialization, normalization, orthogonal initialization, random-walk initialization, composition kernels, and residual architectures share a goal of stabilizing gradients and improving training dynamics.These approaches provide context for the paper’s critical-initialization analysis.Source: §2 · p. 2

Mean field approaches

  • The paper follows recent mean field analyses of trainability and expressivity in fully connected and residual networks, extending their techniques to recurrent networks with gating.Earlier recurrent analyses used alternatives including Gershgorin circle arguments and time invariance.Source: §2 · p. 2

3. Theory and Critical Initialization

The paper develops mean-field and random-matrix analyses of vanilla RNNs and minimalRNNs, relating forward signal propagation to gradient behavior and dynamical isometry. It shows that gating yields more robust long-term propagation and trainable initializations than vanilla recurrence.

3. Theory and Critical Initialization

  • The theory combines mean-field analysis of forward signals with random-matrix analysis of end-to-end Jacobians to study trainability in recurrent networks.The framework contrasts vanilla RNNs with the simpler gated minimalRNN.Source: §3 · p. 2

3.1. Vanilla RNN

  • Ignoring weight tying changes quantitative predictions but preserves the qualitative signal-propagation behavior observed in simulations.The approximation is evaluated by comparing tied and untied recurrent weights.Source: §3.1 · pp. 3–4
  • Vanilla RNN memory is fundamentally limited by input noise because Σ12 < 1 destroys the ordered-to-chaos critical point.The maximum memory timescale is τ = −1/log χc∗ when the fixed point is stable.Source: §3.1 · pp. 3–4

3.2.1. MEAN-FIELD THEORY

  • The minimalRNN uses an update gate to interpolate between the previous hidden state and a transformed input, retaining an essential LSTM-like gating mechanism in a simpler cell.Its simplified update is intended to isolate the role of gating.Source: §3.2.1 · p. 4
  • Gating permits arbitrarily long-term signal propagation as μb → ∞, independent of input similarity Σ12, although the critical point at μb = 0 appears only when Σ12 = 1.Theory and Monte Carlo simulations agree quantitatively in several comparisons, while the untied approximation remains qualitatively accurate overall.Source: §3.2.1 · p. 5

3.2.2. DYNAMICAL ISOMETRY

  • The same χ1 = 1 boundary separates ordered from chaotic forward dynamics and vanishing from exploding gradients, while positive μb makes minimalRNN dynamical isometry robust to variances and input similarity.Unlike vanilla RNNs, minimalRNNs can reduce the linear growth coefficient of JJT by adjusting the bias mean, producing a higher-dimensional trainable region.Source: §3.2.2 · pp. 5–6

4. Experiments

The experiments test whether theoretical signal-propagation timescales predict trainability and whether critical initialization improves recurrent learning. Across these settings, gating broadens the trainable region, while critical initialization substantially accelerates convergence.

4.1. Trainability

  • Untied-weight experiments show strong quantitative agreement between the theoretical timescale τ and the maximum trainable sequence depth T, while tied weights preserve the qualitative relationship.Source: §4.1 · p. 7
  • The minimalRNN remains trainable across a much wider hyperparameter range than the vanilla RNN, despite being trained for an order of magnitude less time.Source: §4.1 · p. 7

4.2. Critical initialization

  • The experiments use MNIST-derived sequences with T = 196 and T = 784 to vary task difficulty, comparing critical and off-critical orthogonal or Gaussian initialization.Source: §4.2 · p. 7
  • Critical initialization produces about a two-orders-of-magnitude training-speed advantage over off-critical initialization for vanilla RNNs.Vanilla RNNs reach 90% test accuracy after 750 optimization steps under critical initialization versus more than 16,000 updates off criticality.Source: §4.2 · p. 8
  • The same critical-versus-off-critical convergence trend appears in minimalRNNs, and the longer T = 784 task causes both architectures initialized off criticality to fail.Source: §4.2 · p. 8
  • The well-conditioned minimalRNN trains three times faster than the vanilla RNN, while weight-distribution choice matters for vanilla RNNs but not for minimalRNNs.Source: §4.2 · p. 8

5. Language modeling

The language-modeling experiments compare minimalRNN with more complex recurrent architectures on Penn Tree-Bank. Despite its simpler design, minimalRNN performs comparably to GRUs, although LSTMs achieve lower perplexity.

5. Language modeling

  • The comparison uses Penn Tree-Bank validation and test perplexities for single-layer RNNs with approximately 5M- and 20M-parameter configurations.Source: §5 · p. 8
  • MinimalRNN performs comparably to GRUs on Penn Tree-Bank despite being significantly simpler, while a perplexity gap remains between minimalRNN and LSTM.The authors hypothesize that removing an independent input gate contributes to the gap.Source: §5 · p. 8, Table 1 · p. 8
  • Single-layer RNNs perform on par with their multi-layer counterparts in the reported language-modeling comparisons.Source: §5 · p. 8

6. Discussion

The discussion concludes that the theory predicts trainability and that gating enlarges the trainable region for recurrent networks. The authors identify extensions to more complex cells and multilayer RNNs as future work.

6. Discussion

  • The theory rigorously predicts trainability for random vanilla RNNs and simple gated RNNs, with gating enabling a significantly larger trainable region.Source: §6 · p. 8
  • The stated scope for future extension includes more complicated RNN cells and multilayer recurrent networks.Source: §6 · p. 8

A. MinimalRNN Architecture

The minimalRNN maps inputs into hidden space and uses an update gate to control recurrence. Mean-field analysis then derives covariance recurrences and interprets the gate bias as controlling the balance between retaining state and incorporating inputs.

A. MinimalRNN Architecture

  • The minimalRNN first transforms each input into hidden space, then computes a gated recurrent update using the transformed input, pre-activation, update gate, and hidden state.Source: B. Diagonal Recurrence Relation · p. 11

B. Diagonal Recurrence Relation

C. Existence of a Q∗Fixed Point

With time-independent input norm, minimalRNN admits a hidden-state norm fixed point Q*; the subsequent analysis examines dynamics near that point.

C. Existence of a Q∗Fixed Point

  • A fixed point Q* exists whenever the input norm is constant over time, because a continuous function changes sign between Q*→0 and Q*→∞.The argument applies the intermediate value theorem to the transcendental fixed-point equation.Source: C. Existence of a Q∗Fixed Point · p. 12

D. Q∗Dynamics

  • The analysis then perturbs the hidden-state norm as Q_t = Q* + ϵ_t to study local dynamics around the fixed point.The expansion is performed around Q* with ϵ ≪ 1.Source: D. Q∗Dynamics · p. 12

E. Off-Diagonal Recurrence Relation

The off-diagonal recurrence is linearized around its fixed point, yielding a multiplicative update controlled by χ_c*.

E. Off-Diagonal Recurrence Relation

F. Additional Hyperparameter Ranges

The models are tuned over specified ranges of learning rate, maximum epochs, decay, and dropout.

F. Additional Hyperparameter Ranges

  • Learning rate, maximum epochs, decay, and dropout are varied over predefined grids for all models.The ranges are learning rate {0.1, 0.2, 0.3, 0.5, 1, 2}, max-epoch {4, 7, 11}, decay {0.5, 0.65, 0.8}, and dropout {0.0, 0.2, 0.3, 0.5}.Source: F. Additional Hyperparameter Ranges · p. 13
Loading 1806.05394v2…