PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming the vanishing gradient problem in plain recurrent networks

Yuhuang Hu, Adrian Huber, Jithendar Anumula, Shih-Chii Liu

arXiv:1801.06105v3cs.NE

TL;DR

Plain recurrent networks face the vanishing-gradient problem; this paper analyzes gating mechanisms and proposes the gate-free Recurrent Identity Network (RIN). RINs are competitive across benchmarks and converge faster, with small models reporting 12%–67% higher accuracy on Sequential and Permuted MNIST.

  • Problem

    Plain recurrent networks face the vanishing-gradient problem, motivating approaches that mitigate it.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes a plain RNN formulation with a non-trainable identity matrix as surrogate memory to preserve representations across recurrent steps without gates.

    Source: §1 · p. 1, §2.3 · p. 4

  • Results

    RINs are competitive across benchmarks and converge faster; small RINs produce 12%–67% higher accuracy in Sequential and Permuted MNIST.

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    RINs model long-range dependencies without gates.

    Source: §4 · p. 8

    The empirical identity-mapping analysis shows evidence across tested RNNs but does not explain how that analysis relates to network performance.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%--67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.

1 INTRODUCTION

The introduction analyzes how gates may encourage identity mappings, then proposes a gate-free plain RNN formulation intended to address vanishing gradients while retaining long-range dependencies.

1 INTRODUCTION

  • Prior mitigation approaches include second-order optimization, training schedules, and special weight initialization; gated networks instead control information flow through layers or sequences.Examples include LSTM, GRU, Highway Networks, and ResNets.Source: §1 · p. 1
  • Analysis of dual-gate designs in LSTMs and GRUs suggests that gates encourage identity mappings that can benefit deep-network training.The paper also investigates whether iterative estimation applies to recurrent gated networks.Source: §1 · p. 1
  • The proposed Recurrent Identity Network (RIN) encourages identity mappings without gates and uses ReLU activations and non-trainable parameters.The introduction presents the formulation as a way for plain recurrent networks to address vanishing gradients while modeling long-range dependencies.Source: §1 · pp. 1–2
  • RINs show competitive performance across the reported benchmarks and faster early training convergence than IRNNs and LSTMs.The paper also reports that plain RNNs with ReLUs do not suffer the usual training instability in its experiments.Source: §1 · p. 2

2 METHODS

The methods interpret recurrent state updates as iterative estimation, analyze how gated recurrences preserve representations, and use that analysis to formulate a plain RNN with an identity component.

2.1 ITERATIVE ESTIMATION VIEW IN RNNS

  • In the adding problem, LSTM average estimation errors are close to zero, consistent with iterative estimation and identity mapping across recurrent updates.The measure compares hidden activations across steps; the reported result suggests activations remain similar on average.Source: §2.1 · p. 2, Figure 1 · p. 2

2.2 ANALYSIS OF GNNS

  • The gate analysis shows that state can be preserved without a carry gate, indicating that plain recurrent networks can learn identity mappings.The analysis represents each step using a hidden transformation, transform gate, and carry gate, with a plain RNN as a special case when Tt = 1 and Ct = 0.Source: §2.2 · p. 3

2.3 RECURRENT IDENTITY NETWORKS

  • RIN adds a non-trainable identity matrix to the hidden-to-hidden weights as surrogate memory, encouraging preservation of past representations without gates.The paper notes that this is equivalent to initializing a plain RNN's hidden-to-hidden weight matrix with an additional identity matrix.Source: §2.3 · p. 4
  • Ororbia II et al. (2017) independently discovered the same RIN formulation using the Differential State Framework, which also motivates implicit hidden-state stability.Source: §2.3 · p. 4

3 RESULTS

Across the adding problem, MNIST classification, and bAbI question answering, RINs are compared with IRNNs and LSTMs on tasks involving long-range dependencies. Results vary by task and model size, with faster convergence reported for RINs in several comparisons.

3.1 THE ADDING PROBLEM

  • RINs and IRNNs reached the same MSE level on adding sequences of 200, 300, and 400 steps, while LSTMs performed worst and failed to converge at 400 steps.The task predicts the sum of two randomly selected numbers in each sequence.Source: Figure 2 · p. 4, §3.1 · pp. 4–5

3.2 SEQUENTIAL AND PERMUTED MNIST

  • RIN-DTs preserve the error signal across 1,568 computation steps, twice the 784-step sequence length used for MNIST.Each recurrent step in RIN-DT performs two hidden transitions, doubling the inherent depth.Source: §3.2 · p. 5
  • Across recurrent steps, all tested networks show average activation levels consistent with identity mapping, but variance grows in the final 200 steps for IRNNs and RINs.The authors caution that this empirical analysis does not explain the relationship between identity mapping and network performance.Source: §3.2 · pp. 5–6
  • On small Sequential and Permuted MNIST networks, RINs outperform IRNNs in accuracy; on larger networks, their accuracy is similar, while RINs converge faster early in training.LSTMs perform worst on both tasks in convergence speed and final accuracy.Source: §3.2 · p. 5, Table 1 · p. 6, Figure 3 · p. 6

3.3 BABI QUESTION ANSWERING TASKS

  • On 20 bAbI question-answering tasks, RINs reach mean performance similar to the state-of-the-art results reported by Jing et al. (2017).The table reports test accuracy for the 20 tasks.Source: §3.3 · p. 7, Table 3 · p. 8

4 DISCUSSION

The discussion interprets recurrent gated networks through iterative representation refinement and proposes RIN as a gate-free way to maintain representations across recurrent steps. RINs are competitive across the reported tasks, with faster convergence, while the authors identify unresolved questions about their stability and convergence.

4 DISCUSSION

  • In gated networks, each recurrent step’s contribution depends jointly on the representation formed so far and later carry-gate openness.The authors note that carry gates can select useful representation patterns, as in Phased LSTM’s temporal-frequency selection.Source: §4 · p. 8
  • RIN adds non-trainable weights for a surrogate memory that maintains learned representations across two recurrent steps without gates.The discussion presents it as a plain recurrent variant that can model long-range dependencies.Source: §4 · p. 8
  • RINs are competitive with IRNNs and LSTMs; small RINs achieve 12%–67% higher accuracy on Sequential and Permuted MNIST.RINs also converge faster early, work without advanced weight initialization, and are relatively insensitive to learning rate, batch size, and optimizer selection.Source: §4 · p. 8
  • The authors do not claim RINs generally outperform LSTMs, which may reach comparable performance with finely tuned hyperparameters.This qualifies the reported competitiveness rather than establishing universal superiority.Source: §4 · p. 8
  • Although repeated ReLU in RNNs may risk gradient explosion, the authors report that RINs are less sensitive to training conditions, and experiments support faster convergence with ReLU.They contrast RIN with IRNN, which they found more sensitive and unstable than RINs and LSTMs.Source: §4 · pp. 8–9
  • RIN complements DiracNet’s plain feedforward-network findings by showing an RNN can be trained without gates or special activation functions.DiracNet instead uses NCReLU to approximate identity mappings under a stated weight-expectation condition.Source: §4 · p. 9
  • The authors leave the effectiveness of the non-trainable identity matrix, RIN’s faster convergence, and its stability with repeated ReLU for future investigation.They also identify RIN’s lower sensitivity to training parameters as an open question.Source: §4 · p. 9

A ALGEBRA OF EQS. 8–9

The appendix sets out the dual-gate recurrent formulation and auxiliary boundary states, then derives the final layer output and its residual-based expansion. The notation treats carry-gate openness as a way to select useful representation contributions.

A ALGEBRA OF EQS. 8–9

  • The dual-gate formulation combines a hidden transformation with transform and carry gates; a plain recurrent network is the case Tt = 1 and Ct = 0.The gates’ values indicate openness at each neuron, and the gate terms are defined through recurrent layers.Source: A ALGEBRA OF EQS. 8–9 · p. 12
  • Auxiliary states h−1 and hT+1, with specified gate values, place the initial and final activations within the recurrent formulation.The conventional initial hidden activation h0 is zero.Source: A ALGEBRA OF EQS. 8–9 · p. 12
  • Starting from recursive Eq. 4, the appendix writes final-layer output hT and defines Q as element-wise multiplication; under Eq. 3 and assuming Eq. 5 fulfills Eq. 1, it introduces a zero-mean residual ϵt.It then rewrites Eq. 16, substitutes Eq. 18 into Eq. 15, and rearranges Eqn. 20.Source: A ALGEBRA OF EQS. 8–9 · p. 12

B DETAILS IN THE ADDING PROBLEM EXPERIMENTS

The adding-problem appendix reports training-loss behavior across sequence lengths and highlights greater IRNN instability at the longest setting. Figure 5 plots MSE on a log scale through 300 training epochs.

B DETAILS IN THE ADDING PROBLEM EXPERIMENTS

C DETAILS IN SEQUENTIAL AND PERMUTED MNIST EXPERIMENTS

The MNIST appendix presents training and testing curves for Sequential and Permuted MNIST, alongside figures assessing the Iterative Estimation hypothesis across network types. The reported training curves show faster convergence for RIN and RIN-DT than for IRNN and LSTM.

C DETAILS IN SEQUENTIAL AND PERMUTED MNIST EXPERIMENTS

D RINS WITH DEEP TRANSITIONS

RIN-DTs add multiple nonlinear transitions within each recurrent step rather than stacking recurrent layers, increasing network depth.

D RINS WITH DEEP TRANSITIONS

Related papers

Loading 1801.06105v3…