PaperlayerView exact revision on arXiv

Source-linked AI summary

Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies

T. Konstantin Rusch, Siddhartha Mishra

arXiv:2010.00951v2cs.LGcs.NEstat.ML

TL;DR

Long-sequence RNN training faces exploding and vanishing gradients. The paper introduces a recurrent architecture based on discretized coupled-oscillator ODEs, proves bounds on its states and gradients, and reports competitive performance across sequential learning tasks.

  • Problem

    RNN training over long sequences is difficult because products of hidden-state Jacobians can make gradients vanish or explode exponentially with recurrent interactions.

    Source: §1 · p. 1

  • Method

    The paper proposes a recurrent network based on time-discretizing a second-order ODE that models coupled, controlled nonlinear oscillators.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Under verifiable hypotheses, the authors establish bounds on hidden states and their gradients, and report competitive performance on a variety of sequential learning tasks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results support coRNN as a potential stable and expressive architecture for processing complex sequential data.

    Source: §1 · p. 2

    The authors caution that coRNN may not suit every sequential task; in the chaotic regime of Lorenz-96 time-series prediction, LSTMs outperform it.

    Source: §5 · p. 9

Abstract

from arXiv · show

Circuits of biological neurons, such as in the functional parts of the brain can be modeled as networks of coupled oscillators. Inspired by the ability of these systems to express a rich set of outputs while keeping (gradients of) state variables bounded, we propose a novel architecture for recurrent neural networks. Our proposed RNN is based on a time-discretization of a system of second-order ordinary differential equations, modeling networks of controlled nonlinear oscillators. We prove precise bounds on the gradients of the hidden states, leading to the mitigation of the exploding and vanishing gradient problem for this RNN. Experiments show that the proposed RNN is comparable in performance to the state of the art on a variety of benchmarks, demonstrating the potential of this architecture to provide stable and accurate RNNs for processing complex sequential data.

1 INTRODUCTION

Training RNNs over long sequences is difficult because gradients can vanish or explode. The paper proposes an oscillator-based RNN intended to control state and gradient dynamics while retaining expressivity.

1 INTRODUCTION

  • BPTT multiplies hidden-state Jacobians across long sequences, allowing gradients to grow or decay exponentially with recurrent interactions.This exploding and vanishing gradient problem makes long time-scale training notoriously hard.Source: §1 · p. 1
  • Additive gating in LSTMs and GRUs can address vanishing gradients, but gradients may still explode and very long dependencies remain challenging.Orthogonal or unitary recurrent matrices alleviate the gradient problem by constraining Jacobian spectra, but may reduce expressivity on realistic tasks.Source: §1 · p. 1
  • The proposed RNN discretizes second-order ODEs for coupled nonlinear oscillators, with precise state and gradient bounds under verifiable hypotheses and competitive benchmark performance.The authors present this as a possible way to mitigate exploding and vanishing gradients while retaining sufficient expressivity for complex inputs.Source: §1 · pp. 1–2

2 THE PROPOSED RNN

The proposed coRNN models coupled forced and damped nonlinear oscillators with a second-order ODE, then uses an IMEX discretization to define recurrent hidden-state updates.

2 THE PROPOSED RNN

  • The architecture starts from a second-order ODE for a network of controlled, forced, damped nonlinear oscillators, with the hidden state and input entering the oscillator dynamics.The continuous-time formulation uses tanh as its activation and includes parameters for oscillation frequency and damping.Source: §2 · p. 2
  • The RNN discretizes the corresponding first-order system in variables y and z using an IMEX scheme, with the timestep defining recurrent states.The damping term may be treated implicitly or explicitly, while both versions retain the implicit treatment of the first equation.Source: §2 · p. 2
  • The authors call the architecture coRNN; its diagonal parameters set oscillator frequency and damping, while off-diagonal entries modulate neuron interactions.Source: §2 · p. 2
  • For a single neuron, input signals drive superpositions of oscillatory waveforms, and interactions can produce rich, non-oscillatory hidden states.The authors argue that these dynamics may support expressive outputs for complicated sequential inputs.Source: §2 · pp. 2–3
  • Unlike constrained-weight ODE-inspired RNNs, coRNN relies on its underlying ODE dynamics and IMEX discretization for gradient stability, without gating mechanisms.The authors note that strong weight constraints may reduce expressivity and lead to inadequate performance on complex tasks.Source: §2 · p. 3

3 RIGOROUS ANALYSIS OF THE PROPOSED RNN

The analysis establishes bounds on coRNN hidden states and gradients under stated conditions. It also derives a long-term gradient lower bound tied to the chosen discretization.

3 RIGOROUS ANALYSIS OF THE PROPOSED RNN

  • Analogous results and proofs are provided for the explicit damping variant and for general values of epsilon and gamma.Source: §3 · p. 3
  • The hidden states satisfy an energy bound, and the resulting estimate on input sensitivity rules out chaotic hidden-state behavior.The result is stated for the RNN hidden states generated under the specified setup.Source: §3 · p. 4
  • Under hypotheses on the timestep and weight matrices, the loss gradient with respect to network parameters is bounded above, mitigating exploding gradients.The proof identifies a bound involving recurrent matrices with close to unit norm as the crux of the gradient-control argument.Source: §3 · pp. 4–5
  • With the stated hidden-state growth assumption, the long-term gradient bound is independent of lag k, so distant states can still contribute at later steps.The authors say this mitigates vanishing gradients; the gradient can nevertheless be small.Source: §3 · p. 5
  • The lower bound for long-term gradients follows from this specific discretization and has no continuous analogue, making the discretization choice crucial to mitigating vanishing gradients.Source: §3 · p. 5

4 EXPERIMENTS

Experiments test coRNN on long-dependency and classification tasks, reporting strong benchmark comparisons and robustness across training conditions.

Experimental protocol

  • Experiments use n̄=n−1, selected for marginally better performance, and validation-based tuning without dropout, gradient clipping, or batch normalization.Source: §4 · p. 5

Adding problem

  • On the adding problem, coRNN converges faster with lower test MSE at T=500 and remains effective at T=2000 and 5000, where expRNN falters.At T=2000, only coRNN and expRNN beat the baseline MSE of 0.167, but expRNN does not reach 0.01; at T=5000, it inconsistently beats baseline.Source: §4 · p. 6, Figure 1 · p. 6

Sequential (permuted) MNIST

  • On sMNIST and psMNIST, coRNN reportedly outperforms all published single-layer recurrent architectures; on the fixed 128-unit psMNIST permutation, it outperforms other tested architectures.The psMNIST comparison uses the same fixed random permutation across models.Source: §4 · pp. 6–7, Table 1 · p. 6, Figure 2 · p. 7

Human activity recognition

  • On HAR-2, coRNN outperforms the compared methods, and a 1k-parameter coRNN also beats all baselines.The authors conclude that coRNN can be used on resource-constrained IoT microcontrollers.Source: §4 · p. 7, Table 3 · p. 8

IMDB sentiment analysis

  • On IMDB, coRNN compares favorably with gated baselines while requiring significantly fewer parameters.The comparison uses models with 128 hidden units.Source: §4 · p. 8, Table 4 · p. 8

Further experimental results

  • Across LTD tasks, assumptions (8) hold during training; noise-padded CIFAR-10 accuracy is robust across tested (ϵ, γ), and 10-retraining results have low standard deviations.The mean accuracy over retrainings is comparable to the best reported value.Source: Figure 3 · p. 7, Figure 4 · p. 7, §4 · p. 8

5 DISCUSSION

The paper proposes coRNN, an oscillator-based recurrent architecture with bounded hidden states and precisely bounded gradients under verifiable hypotheses. Its experiments support performance on varied tasks, while the discussion identifies chaotic prediction and the rigor of its expressivity account as boundaries.

5 DISCUSSION

  • Under verifiable hypotheses, coRNN has bounded hidden states and precise hidden-state gradient bounds, and experiments show it can perform complex sequential tasks.The authors present this as a way to mitigate the exploding and vanishing gradient problem.Source: §5 · p. 9
  • The paper focuses on long-term-dependency tasks and reports coRNN performance comparable to the best published results in the literature.Oscillatory-input applications such as EEG, EMG, and seismic data, as well as language modeling, are left to future work.Source: §5 · p. 9
  • coRNN may be ineffective for chaotic time-series prediction; in the Lorenz-96 chaotic regime, LSTMs outperform it.The authors link this boundary to the underlying ODE and discretization not allowing super-linear-in-time separation of nearby trajectories.Source: §5 · p. 9
  • The paper provides heuristics and numerical evidence for coRNN's expressivity, but not a rigorous universality result.A rigorous investigation of universality is proposed as future work.Source: §5 · p. 9

Supplementary Material for: Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable

The supplied heading identifies the supplement as relating to an architecture for learning long time dependencies.

Supplementary Material for: Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable

A CHAOTIC TIME-SERIES PREDICTION.

This section evaluates coRNN and LSTM on Lorenz-96 prediction, contrasting non-chaotic and chaotic settings. Their performance is similar in the non-chaotic setting, but LSTM outperforms coRNN when the dynamics are chaotic.

A CHAOTIC TIME-SERIES PREDICTION

  • Because coRNN's underlying ODE and discretization do not permit super-linear-in-time separation of nearby trajectories, the authors do not expect it to predict chaotic time series effectively.Source: A CHAOTIC TIME-SERIES PREDICTION · p. 13
  • Lorenz-96's external force F controls the level of chaos, and the figure presents example x1–x2 trajectories for different forces.The text identifies F = 0.9 as non-chaotic and F = 8 as highly chaotic.Source: A CHAOTIC TIME-SERIES PREDICTION · p. 13, Figure 5 · p. 13
  • The task predicts a trajectory's 25th next state using 128 trajectories of length 2,000 for each training, validation, and test set.Trajectories are sampled at 0.01 time intervals, with initial values chosen uniformly around the system's equilibrium point.Source: A CHAOTIC TIME-SERIES PREDICTION · p. 13
  • In the chaotic Lorenz-96 case, LSTM clearly outperforms coRNN; in the non-chaotic case, they perform similarly.The comparison reports test NRMSE, defined as RMSE divided by the target trajectory's root mean square.Source: A CHAOTIC TIME-SERIES PREDICTION · p. 13, Table 6 · p. 13

B TRAINING DETAILS

The training details describe random-search hyperparameter selection, initialization, and task-specific training schedules. The supplied table captions identify hyperparameter search ranges and the best-performing configurations, but do not give their values.

B TRAINING DETAILS

  • The best coRNN was selected by validation-set performance after random-search hyperparameter optimization.Table 7 covers search settings and Table 8 the rounded hyperparameters of the best-performing architecture; their values are not given in the supplied passages.Source: B TRAINING DETAILS · p. 14, Table 7 · p. 14, Table 8 · p. 14
  • The weights and biases were initialized uniformly based on each affine transformation's input dimension; making Δt, γ, and ϵ trainable produced no major performance difference.Source: B TRAINING DETAILS · p. 14
  • Training used 100 epochs for sMNIST, psMNIST, noise-padded CIFAR-10, and IMDB, and 250 epochs for HAR-2.The image tasks received 20 additional epochs with the learning rate reduced by a factor of 10.Source: B TRAINING DETAILS · p. 14

C HEURISTICS OF NETWORK FUNCTION

The section illustrates how forcing, damping, control, and coupling shape oscillator-network outputs. Controlled oscillators and neuron coupling generate complex waveforms, while the Duffing-type example is heavily damped and sensitive to parameter changes.

C HEURISTICS OF NETWORK FUNCTION

  • Controlled oscillators produce complex outputs: coRNN modulates the controlled forced-and-damped oscillator’s wave superpositions, while the Duffing-type solution is heavily damped and can blow up after slight parameter changes.The authors conclude that bounded nonlinearity seems essential in this context.Source: C HEURISTICS OF NETWORK FUNCTION · p. 15
  • Adding damping to the forced oscillator reduces wave amplitude without changing frequency, while control substantially changes waveform shape and phase.The comparison uses an oscillatory signal and a combination of step functions; forcing alone makes only a minor change over the simple harmonic oscillator for the oscillatory input.Source: C HEURISTICS OF NETWORK FUNCTION · pp. 14–15
  • Coupling accentuates waveform superpositions; with ordered connections, the first neuron turns an oscillatory input into an approximate step-like output in the second neuron.Fully connecting the neurons further accentuates that step-function approximation, while slightly changing the first neuron’s output.Source: C HEURISTICS OF NETWORK FUNCTION · p. 15
  • The authors describe oscillator-network outputs as superpositions whose frequencies, phases, and amplitudes are modulated by learned weights, allowing a rich set of output states.They argue that networks with many neurons can have rich global dynamics.Source: C HEURISTICS OF NETWORK FUNCTION · p. 15

D BOUNDS ON THE DYNAMICS OF THE ORDINARY DIFFERENTIAL EQUATION (1)

The analysis bounds oscillator dynamics and their sensitivity to inputs, then extends the reasoning to hidden-state gradients. Its supplementary proofs derive energy and gradient estimates for the discretized system.

D BOUNDS ON THE DYNAMICS OF THE ORDINARY DIFFERENTIAL EQUATION (1)

  • The ODE energy estimates bound the oscillator dynamics, with the uniformly bounded activation σ = tanh playing a crucial role.The analysis also shows that trajectories for different inputs separate at most linearly in time, ruling out chaotic behavior for this ODE structure.Source: D BOUNDS ON THE DYNAMICS OF THE… · pp. 15–17

D.1 GRADIENT DYNAMICS FOR THE ODE SYSTEM (2)

  • Hidden-state gradients remain bounded over time when their initial values are bounded and the weights are controlled by the damping parameter.The parameter-gradient dynamics form an oscillator system with additional coupling and forcing terms proportional to hidden states or inputs.Source: §D.1 · pp. 17–18

E.1 PROOF OF PROPOSITION 3.1

  • The supplementary proof of Proposition 3.1 obtains an energy bound by iterating an inequality for the discretized recurrence.Source: §E.1 · p. 18

E.2 SENSITIVITY TO INPUTS

  • For the discretized RNN, changes in the input produce hidden-state differences that grow at most linearly in time, so the proposed RNN is stable with respect to input changes.The authors attribute this property largely to the globally bounded activation function.Source: §E.2 · p. 18

E.3 PROOF OF PROPOSITION 3.2

  • The Proposition 3.2 proof bounds entries of an activation-derivative matrix by 1 and uses estimates to obtain the desired loss-gradient bound.Source: §E.3 · pp. 19–20

E.4 ON THE ASSUMPTION (8) AND TRAINING

This section asks whether the stability condition used in the analysis can hold as training changes the weights. Its estimates and experiments indicate that it can remain valid over many training steps.

E.4 ON THE ASSUMPTION (8) AND TRAINING

  • The section asks whether a timestep can keep the stability condition satisfied at every stochastic-gradient-descent step as the weights are learned.Source: §E.4 · p. 20
  • The worst-case estimate says the condition holds for many steps when r ≤ 1, but the authors describe that estimate as a large underestimate on the number of steps.The argument notes δ < 1 and, in general, a learning rate ζ << 1.Source: §E.4 · p. 21
  • In the experiments, the authors report taking many training steps while gradients remain within a range.Source: §E.4 · p. 21

E.5 PROOF OF PROPOSITION 3.3

The proofs extend stability estimates to another coRNN discretization and general control parameters. They derive hidden-state and gradient bounds under stated conditions, while noting an assumption used to estimate leading-term constants.

E.5 PROOF OF PROPOSITION 3.3

  • The Jacobian-product proof uses an inductive representation and expands the product into leading and higher-order terms before deriving the estimate.The derivation neglects terms of order O(∆t^4) in one expansion.Source: §E.5 · pp. 21–22
  • The leading-term constants rely on assuming energy is equidistributed across hidden-state entries, because the energy bound is too coarse to resolve individual entry magnitudes.The authors note that experiment-specific terms can be used to evaluate a more precise bound.Source: §E.5 · p. 23
  • The alternative discretization has the same hidden-state bound as the other version, but its sufficient timestep condition is more stringent; the authors report similar practical stability ranges.The stated timestep conditions are sufficient, not necessary.Source: §E.5 · p. 24
  • Under the stated conditions, the alternative version’s hidden-state gradients cannot explode, and its long-term gradient estimate rules out vanishing gradients.Source: §E.5 · pp. 24–25
Loading 2010.00951v2…