PaperlayerView exact revision on arXiv

Source-linked AI summary

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, Yanbo Gao

arXiv:1803.04831v3cs.CVcs.LG

TL;DR

RNNs remain difficult to train over long sequences and deep stacks because of gradient instability, activation-related decay, and entangled neurons. The paper proposes IndRNN, which uses independent within-layer neurons and regulated scalar recurrent weights. IndRNN processes over 5000 steps, supports 21-layer networks, and outperforms traditional RNNs and LSTMs across multiple tasks.

  • Problem

    RNNs face gradient vanishing and exploding problems, difficulty learning long-term patterns, gradient decay in LSTM and GRU layers, and hard-to-interpret entangled neurons.

  • Method

    IndRNN uses Hadamard-product recurrent inputs so neurons are independent within layers, with gradients regulated through scalar recurrent weights and layers stackable with residual connections.

  • Results

    IndRNN outperformed traditional RNN and LSTM models across adding, sequential MNIST, language modelling, and action recognition tasks, while processing over 5000 time steps.

  • Takeaways & Limitations

    IndRNN supports robust long-sequence processing, ReLU-based training, and deep recurrent networks, including a demonstrated 21-layer language-modelling network.

  • Takeaways & Limitations

    The theoretical relationship between IndRNN and traditional nonlinear RNNs remains unestablished, and GPU memory limited one experiment to an 11-layer residual IndRNN.

Abstract

from arXiv · show

Recurrent neural networks (RNNs) have been widely used for processing sequential data. However, RNNs are commonly difficult to train due to the well-known gradient vanishing and exploding problems and hard to learn long-term patterns. Long short-term memory (LSTM) and gated recurrent unit (GRU) were developed to address these problems, but the use of hyperbolic tangent and the sigmoid action functions results in gradient decay over layers. Consequently, construction of an efficiently trainable deep network is challenging. In addition, all the neurons in an RNN layer are entangled together and their behaviour is hard to interpret. To address these problems, a new type of RNN, referred to as independently recurrent neural network (IndRNN), is proposed in this paper, where neurons in the same layer are independent of each other and they are connected across layers. We have shown that an IndRNN can be easily regulated to prevent the gradient exploding and vanishing problems while allowing the network to learn long-term dependencies. Moreover, an IndRNN can work with non-saturated activation functions such as relu (rectified linear unit) and be still trained robustly. Multiple IndRNNs can be stacked to construct a network that is deeper than the existing RNNs. Experimental results have shown that the proposed IndRNN is able to process very long sequences (over 5000 time steps), can be used to construct very deep networks (21 layers used in the experiment) and still be trained robustly. Better performances have been achieved on various tasks by using IndRNNs compared with the traditional RNN and LSTM. The code is available at https://github.com/Sunnydreamrain/IndRNN_Theano_Lasagne.

1. Introduction

IndRNN separates recurrent processing across neurons, enabling regulated gradients, long-sequence learning, robust ReLU training, interpretable neurons, and deeper stacked networks. Experiments report performance gains over traditional RNNs and LSTMs across multiple tasks.

  • Motivation: RNN training suffers from vanishing and exploding gradients, while LSTM and GRU activation functions cause gradient decay over layers that complicates deep-network training.Existing RNN neurons are also entangled, making individual neuron roles difficult to interpret.
  • IndRNN design: IndRNN processes recurrent inputs with a Hadamard product, making neurons within a layer independent while connecting information across layers.Each neuron receives the current input and its own previous hidden state.
  • IndRNN advantages: IndRNN backpropagation can be regulated to address gradient vanishing and exploding problems.The recurrent contribution to the gradient depends on a scalar recurrent weight rather than repeated recurrent matrix products.
  • IndRNN advantages: Over 5000 time steps, IndRNN processed sequences that LSTM could process for fewer than 1000 steps.This result demonstrates the reported long-term sequence-processing capability.
  • IndRNN advantages: IndRNN works robustly with the non-saturated ReLU activation function.This supports using ReLU-based recurrent units without the saturation behavior associated with sigmoid and hyperbolic tangent functions.
  • Deep and interpretable networks: Multiple IndRNN layers can be efficiently stacked, especially with residual connections, and a 21-layer IndRNN was demonstrated for language modelling.Neuron independence also makes IndRNN behavior easier to interpret.
  • Experimental results: IndRNN performed much better than traditional RNN and LSTM models on adding, sequential MNIST, language modelling, and action recognition tasks.The reported comparison spans four task families.

2. Related Work

Related work addresses recurrent gradient problems through gates, constrained recurrent weights, initialization, and training techniques. IndRNN instead supports trainable recurrent connections without gate parameters and can be stacked deeply with residual connections.

  • Gated RNNs: LSTM and GRU use gates to regulate information flow and maintain constant error flow, but gating prevents parallel computation and increases network complexity.Their sigmoid and hyperbolic tangent activations also contribute to gradient vanishing over layers.
  • Alternative recurrent designs: Unitary-evolution RNNs bound backpropagated-gradient norms through empirically defined unitary recurrent weights.The passage contrasts this approach with IndRNN's trainable recurrent connections without gate parameters.
  • Initialization and training methods: Initialization and gradient-regulation methods ease exploding gradients, but recurrent eigenvalues may still exceed 1 during training, so the problem is not completely avoided.The cited methods include IRNN initialization, positive-definite recurrent matrices, path-normalized optimization, and hidden-state norm penalties.
  • Deep architectures: Most existing RNN architectures have only several layers, whereas residual CNNs can exceed 100 layers, largely because recurrent gradient problems make deep RNN training difficult.LSTM gates and modulation functions use sigmoid or hyperbolic tangent activations that suffer gradient vanishing over layers.
  • Deep architectures: Multiple IndRNN layers can be efficiently combined with residual connections, enabling construction of a deep RNN with ReLU-like stacking behavior.This design is presented as a route to deeper recurrent networks.

3. Independently Recurrent Neural Network

IndRNN separates recurrent processing across neurons, enabling direct regulation of temporal gradients and independent gradient propagation. Its scalar recurrent weights support long- and short-term memory while facilitating deeper architectures.

  • Independently Recurrent Neural Network: IndRNN processes each neuron’s recurrent input independently through a Hadamard product, while stacking layers provides connections between neurons across layers.Each neuron receives the current input and only its own previous hidden state.
  • Backpropagation Through Time for An IndRNN: IndRNN backpropagation is independently calculated for each neuron because neurons do not interact within a layer.This removes the recurrent matrix products that couple gradient propagation across neurons.
  • Backpropagation Through Time for An IndRNN: The IndRNN gradient depends directly on each scalar recurrent weight rather than a recurrent matrix product, making training more robust to small parameter changes.Regulating the exponential recurrent-weight term controls gradient vanishing and exploding over time.
  • Backpropagation Through Time for An IndRNN: To preserve long-term memory across T − t steps, the recurrent weight magnitude is constrained by a range determined by the minimum effective gradient.A further constraint limits the range to avoid gradient explosion, while activation derivatives are bounded for common activations such as relu and tanh.
  • Backpropagation Through Time for An IndRNN: With relu, recurrent weights can include zero, allowing some neurons to use only current input while others retain memory from the past.This supports different memory lengths across neurons.
  • Network Architectures: The basic IndRNN architecture can be stacked into deep networks, and residual IndRNN architectures provide an additional stacking design.The supplied figure caption identifies both basic and residual IndRNN architectures.

4. Multiple-layer IndRNN

Multiple IndRNN layers keep neurons independent within each layer while using cross-layer connections to explore cross-channel information. Under linear activation, stacked IndRNNs can represent traditional RNNs with diagonalizable recurrent weights and support deeper, flexible architectures.

  • IndRNN neurons are independent within each layer, while multiple layers explore cross-channel information over time.
  • Under linear activation, a traditional RNN with diagonalizable recurrent weight U is a special case of a two-layer IndRNN.The second layer has zero recurrent weights and an invertible input weight.
  • A two-layer IndRNN with N neurons has M × N + N × N + 2 × N parameters, similar in order to a traditional RNN.
  • The input transformation Wxt + b is independent across timesteps, allowing parallel implementation and extension to convolutional IndRNNs.The convolutional extension replaces the fully connected operation Wxt with W ∗ xt.
  • IndRNN layers can be stacked, including with residual structures, to deepen input processing and construct deep networks.The architecture supports simple stacking, deeper input-processing layers, and residual arrangements.

5. Experiments

Experiments evaluate IndRNNs on long-sequence memory, neuron behavior, image classification, language modeling, and action recognition. Across these tasks, IndRNNs support long sequences, deeper architectures, and improved performance over traditional RNNs and LSTMs.

  • 5.1. Adding Problem: At sequence length 1000, IRNN and LSTM failed to minimize error, whereas IndRNN converged quickly to a small error.For length 5000, IndRNN still modeled the adding problem well after the learning rate decreased.
  • 5.1.1 Analysis of Neurons’ Behaviour: IndRNN neuron activations were interpretable: first-layer neurons selected relevant values, while second-layer neurons separated long-term and short-term state processing.Neuron independence permits analysis without accounting for effects from other neurons.
  • 5.1.1 Analysis of Neurons’ Behaviour: 5000 time steps remained tractable with one IndRNN neuron in the second layer, after retaining the trained first-layer weights and fine-tuning only the output layer.The experiment supports the conjecture that one second-layer neuron can model the adding problem effectively.
  • 5.2. Sequential MNIST Classification: A six-layer IndRNN with 128 neurons per layer was evaluated on sequential and permuted MNIST classification, with batch normalization inserted after each layer.The supplied passages identify the evaluation setup and the error-rate reporting table, but not its numerical results.
  • 5.3. Penn Treebank: IndRNNs achieved better performance than traditional RNNs and LSTMs on language modeling, while deeper residual IndRNNs and longer temporal dependencies further improved performance.A six-layer model and a 21-layer residual model were evaluated on character-level PTB; GPU memory required 11 layers at time step 150.
  • 5.4. Skeleton based Action Recognition: For NTU RGB+D action recognition, a 4-layer IndRNN reached 78.58% on CS and 83.75% on CV, while a 6-layer IndRNN reached 81.80% and 87.97%.These results exceeded the reported two-layer RNN and LSTM baselines and state-of-the-art methods cited by the paper.

6. Conclusion

IndRNNs use independent neurons within each layer and regulated recurrent weights to address gradient problems, support longer sequences, and enable robust deep stacking. Experiments across multiple fundamental tasks verified advantages over existing RNN models.

  • IndRNN neurons are independent within each layer, while recurrent inputs use a Hadamard product with the previous hidden state.This independence supports interpretation of individual neuron roles and distinguishes IndRNNs from traditional entangled recurrent connections.
  • IndRNN gradient backpropagation can be regulated to address vanishing and exploding gradients during sequence processing.
  • IndRNNs can process much longer sequences than existing RNN models, while stacked IndRNNs with residual connections form deep networks that train robustly.
  • Experiments on multiple fundamental tasks verified IndRNN advantages over existing RNN models, including LSTM and GRU.
Loading 1803.04831v3…