PaperlayerView exact revision on arXiv

Source-linked AI summary

AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

Bo Chang, Minmin Chen, Eldad Haber, Ed H. Chi

arXiv:1902.09689v1stat.MLcs.LG

TL;DR

RNNs struggle to learn long-term dependencies because gradients can explode or vanish. The paper uses ODE discretization to design AntisymmetricRNN, which shows predictable dynamics and competitive performance with simpler construction, while its stability and numerical-solver conditions impose important boundaries.

  • Problem

    RNNs face an open challenge in modeling long-term dependencies because back-propagated error signals can grow or decay exponentially.

  • Method

    The paper connects RNNs to ordinary differential equations and designs recurrent architectures by discretizing ODEs, yielding AntisymmetricRNN from stability criteria.

  • Results

    AntisymmetricRNN exhibits well-behaved dynamics, outperforms regular LSTMs on long-term-memory tasks, and matches them on short-term tasks with fewer parameters.

  • Takeaways & Limitations

    The dynamical-systems viewpoint provides a way to exploit ODE theory for understanding and improving RNN trainability.

  • Takeaways & Limitations

    Stability alone does not ensure long-term memory, and the antisymmetric ODE is incompatible with forward Euler stability because its eigenvalues are imaginary.

Abstract

from arXiv · show

Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.

1 INTRODUCTION

RNNs are widely used for sequential data, but long-term dependency learning remains difficult because back-propagated error signals can explode or vanish. The paper addresses this through an ODE-based perspective and the AntisymmetricRNN architecture.

  • Motivation: Long-term dependencies remain difficult for RNNs because back-propagated error signals can grow or decay exponentially.This produces the exploding- or vanishing-gradient problem during backpropagation through time.
  • Existing approaches: Gated RNNs such as LSTMs and GRUs partially address vanishing gradients by allowing information to flow across earlier time steps.
  • Existing approaches: Orthogonal-weight approaches can add significant computational overhead, hinder representation power, and still fail to prevent exploding or vanishing gradients in nonlinear networks.
  • Proposed perspective: The paper connects RNNs with ordinary differential equations and designs recurrent architectures by discretizing ODEs.
  • Proposed architecture: AntisymmetricRNN uses stable ODE solutions to capture long-term dependencies with a simpler construction than existing methods for improving RNN trainability.
  • Empirical results: AntisymmetricRNN exhibits well-behaved dynamics, outperforms regular LSTMs on long-term-memory tasks, and matches them on short-term tasks with fewer parameters.

2 RELATED WORK

Prior work improves recurrent trainability through gating, architectural constraints, or dynamical-systems analyses. This paper belongs to the latter direction while extending ODE-based reasoning to recurrent architectures.

  • Trainability of RNNs: Gated architectures, especially LSTMs and GRUs, are prominent approaches for mitigating exploding and vanishing gradients.
  • Trainability of RNNs: Another line of work constrains recurrent weight matrices so the input-output Jacobian has unitary singular values.
  • Dynamical systems view of recurrent networks: Dynamical-systems analyses have examined chaotic behavior in LSTMs and GRUs and proposed simplified non-chaotic recurrent networks with comparable language-modeling performance.
  • Dynamical systems view of residual networks: Related work on residual networks interprets ResNets as ODEs and develops stable or reversible architectures that motivate the proposed recurrent networks.

3 ANTISYMMETRICRNNS

The paper designs AntisymmetricRNN by discretizing stable ordinary differential equations, linking dynamical-system stability to recurrent-network trainability and long-term dependency preservation.

  • ODE discretization: Forward Euler discretization turns an ODE into a recurrent update with hidden state, model parameters, input, bias, and step size.The resulting skip connection resembles residual and Fourier RNN architectures proposed for gradient stabilization.
  • Stability and trainability: ODE stability motivates recurrent architectures whose dynamics can preserve long-term dependencies without exploding or vanishing gradients.Stability controls sensitivity to initial conditions, while strongly negative Jacobian eigenvalue real parts create lossy systems that forget past inputs.
  • Stability and trainability: The critical criterion Re(λi(J(t))) ≈0 keeps the sensitivity magnitude approximately constant over time, avoiding exploding or vanishing gradients.This criterion is intended to combine stability with preservation of long-term dependencies.
  • AntisymmetricRNN design: Antisymmetric matrices have purely imaginary eigenvalues, making them suitable building blocks for a stable recurrent architecture.The construction uses an antisymmetric transition component, whose Jacobian eigenvalues have zero real parts under bounded activation derivatives.
  • AntisymmetricRNN design: The model can be parameterized efficiently because the antisymmetric transition component has n(n −1)/2 degrees of freedom.A strictly upper triangular parameterization provides this reduction relative to an unstructured recurrent matrix of the same hidden-state size.
  • Forward Euler stability and diffusion: Naive forward Euler is unstable for the purely imaginary eigenvalues of the antisymmetric ODE because |1 + ϵλi(Jt)| exceeds 1.Adding diffusion by subtracting γI shifts eigenvalues slightly into the negative-real-part region and improves numerical stability.

4 SIMULATION

The simulation compares vanilla RNNs and RNNs with feedback under different weight matrices. AntisymmetricRNN feedback produces predictable trajectories, while diffusion prevents outward spiraling and maintains distance from the origin.

  • Simulation setup: The simulation uses three initial states over T = 50 steps to plot hidden-state trajectories for vanilla and feedback RNNs.The plots compare first-row vanilla RNNs with second-row feedback RNNs across transition matrices.
  • Vanilla RNNs: Vanilla RNN states converge differently under random, identity, and orthogonal matrices, revealing the influence of weight structure and activation contraction.Random weights produce unpredictable behavior, while identity and orthogonal weights lead states toward the origin.
  • RNNs with feedback: For feedback RNNs, trajectory behavior is predictable from the eigenvalues of the weight matrix W.The simulation overlays the underlying ODE vector field and uses step size ϵ = 0.1.
  • RNNs with feedback: Antisymmetric weights with eigenvalues 2i and −2i produce circular vector fields, but discretization causes trajectories to become outward spirals.Each tangential step slightly increases distance from the origin, creating numerical instability.
  • RNNs with feedback: Adding diffusion γ = 0.15 tilts the vector field toward the origin and makes trajectories maintain a constant distance from it.The resulting eigenvalues are −0.15 + 2i and −0.15 − 2i.
  • Simulation conclusion: AntisymmetricRNN achieves predictable dynamics without maintaining orthogonal or unitary matrices, which can still suffer vanishing gradients from activation contraction.The simulation identifies this as a simpler alternative to the orthogonal-matrix approach.

5 EXPERIMENTS

Experiments evaluate AntisymmetricRNNs on image-sequence classification tasks designed to test long-term and short-term dependencies. The models achieve strong performance with fewer parameters, while Jacobian analyses and ablations examine their stability and antisymmetric parametrization.

  • Image classification experiments: AntisymmetricRNNs outperform regular LSTMs on pixel-by-pixel MNIST and permuted MNIST using only 1/7 of the parameters.The comparison also reports worse results for strictly orthogonal weights and large-margin improvements over softened orthogonality methods without orthogonality-enforcement overhead.
  • Image classification experiments: On pixel-by-pixel CIFAR-10, AntisymmetricRNN matches LSTM performance, while its gated variant is slightly better using about half as many parameters.The task is mostly dominated by short-term dependencies, and LSTM reaches about 48.3% accuracy using only the last eight image rows.
  • Long-range dependency experiment: On noise padded CIFAR-10, only the first 32 of 1000 time steps contain image information, requiring models to retain salient inputs across 968 noise steps.LSTM fails to train, whereas the proposed methods perform reasonably well with fewer parameters.
  • Dynamics and gradients: Across total sequence lengths T ∈{100, 200, 400, 800}, LSTM Jacobian eigenvalues approach zero, while AntisymmetricRNN eigenvalues remain centered around 1 across many diffusion constants.Eigenvalues near 1 indicate non-exploding and non-vanishing gradients; very large diffusion constants can also produce vanishing gradients.
  • Ablation study: Removing antisymmetric parametrization worsens performance on both pixel-by-pixel and noise padded CIFAR-10.The authors characterize antisymmetry as sufficient but not necessary for stability, leaving other stabilizing conditions possible.

6 CONCLUSION

The paper reframes RNN trainability through dynamical systems and constructs AntisymmetricRNNs by discretizing stable ODEs. The resulting models combine theoretical properties with competitive performance, while motivating further study of stable ODEs and numerical methods.

  • Conclusion: The paper connects recurrent networks with ordinary differential equations and designs recurrent architectures by discretizing ODEs.This perspective is intended to exploit dynamical-systems theory for understanding and improving RNN trainability.
  • Conclusion: AntisymmetricRNN is an ODE discretization satisfying the paper’s critical stability criterion and achieves competitive performance against strong recurrent baselines.The conclusion describes this performance across a comprehensive set of benchmark tasks.
  • Future directions: Future work includes investigating other stable ODEs and numerical methods that could yield novel, well-conditioned recurrent architectures.The proposed connection is also expected to motivate research in both recurrent networks and dynamical systems.

A AN OVERVIEW OF STABILITY THEORY

The stability overview explains how perturbations evolve in scalar and matrix ODEs. System stability is governed by eigenvalues, whose real parts determine whether perturbations grow, decay, or persist.

  • Scalar test equation: For the scalar ODE y′(t) = λy(t), the solution is y(t) = e^λt y(0), with λ allowed to be complex.The complex formulation prepares the analysis for eigenvalues of system matrices.
  • Scalar test equation: If Re(λ) > 0, perturbations grow exponentially; if Re(λ) < 0, they decay exponentially; and if Re(λ) = 0, they are preserved.These cases distinguish instability, stability with decay, and perturbation preservation.
  • Matrix ODEs: For the matrix ODE y′(t) = Ay(t), diagonalization decouples the system into scalar equations whose eigenvalues determine stability.The solution is y(t) = e^At y(0), and the transformed components satisfy w′_i(t) = λ_i w_i(t).

B PROOF OF A PROPOSITION

The proposition establishes that multiplying an antisymmetric matrix by an invertible diagonal matrix yields a matrix with purely imaginary eigenvalues. The proof uses the eigenvalue equation and the fact that the relevant quadratic form is imaginary.

  • Proposition: For antisymmetric W and invertible diagonal D, every eigenvalue of DW is imaginary.Equivalently, Re(λ_i(DW)) = 0 for all i.
  • Proof: The proof starts with an eigenvalue–eigenvector pair of DW and shows that v∗Wv is imaginary.This forces the corresponding eigenvalue to be imaginary, and the argument applies to all eigenvalues.

C EXPERIMENTAL DETAILS

The experiments use varied initialization, optimization, and training settings across recurrent architectures and sequence tasks.

  • Initialization: Input-to-hidden matrices use N(0, 1/m), while hidden-to-hidden matrices use N(0, σ²_w/n).Here, m is the input dimension and n is the number of hidden units.
  • Initialization: The recurrent weight scale σ_w is selected from {0, 1, 2, 4, 8, 16}, with zero biases except an LSTM forget-gate bias initialized to 1.
  • AntisymmetricRNN settings: AntisymmetricRNN experiments vary step size ϵ ∈ {0.01, 0.1, 1} and diffusion γ ∈ {0.001, 0.01, 0.1, 1.0}.
  • Optimization: Models use SGD with momentum or Adagrad, batch size 128, and learning rates selected from 0.1 to 1.The tested learning-rate set is {0.1, 0.2, 0.3, 0.4, 0.5, 0.75, 1}.
  • Training schedule: MNIST and pixel-by-pixel CIFAR-10 models train for 50,000 iterations, while noise-padded CIFAR-10 models train for 10,000 iterations.

D ADDITIONAL VISUALIZATIONS

Additional visualizations compare recurrent dynamics across random seeds and input conditions, showing random vanilla-RNN behavior and more predictable AntisymmetricRNN behavior.

  • Cross-seed comparison: The visualizations illustrate random vanilla-RNN behavior and predictable AntisymmetricRNN dynamics across different random seeds.
  • Vanilla RNN: Figure 3 visualizes vanilla-RNN dynamics with standard Gaussian random weights across seeds 1 to 16.
  • Feedback RNN: Figure 4 visualizes feedback-RNN dynamics with standard Gaussian random weights, seeds 1 to 16, and diffusion strength γ = 0.1.
  • Input perturbation: Figure 5 shows AntisymmetricRNN dynamics under independent standard Gaussian input, which become noisier than Figure 1 while retaining the same trend.
Loading 1902.09689v1…