PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Advances in Recurrent Neural Networks

Hojjat Salehinejad, Sharan Sankar, Joseph Barfett, Errol Colak, Shahrokh Valaee

arXiv:1801.01078v3cs.NE

TL;DR

RNNs can learn sequential dependencies, but gradient-based training struggles with long-range dependencies. This survey reviews RNN fundamentals and advances, reports developments including gating and memory-network results, and identifies open challenges and applications.

  • Problem

    Training RNNs is difficult because gradient-based methods struggle to learn long-range temporal dependencies.

  • Method

    The paper systematically reviews RNN fundamentals, recent advances, training challenges, architectures, and applications.

  • Results

    On the Facebook bAbI dataset, DMN passes 18 tasks with accuracy above 95%, while MemNN passes 16 tasks with lower accuracy.

  • Takeaways & Limitations

    Gating mechanisms have enabled RNNs to learn long-term sequential dependencies, while unitary RNNs offer a potentially simpler alternative for this capability.

Abstract

from arXiv · show

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies. In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field. The fundamentals and recent advances are explained and the research challenges are introduced.

I. INTRODUCTION

RNNs extend neural networks with recurrent connections that model sequential data through nonlinear hidden states serving as memory. The paper introduces their architecture, activations, applications, and training challenges.

  • I. INTRODUCTION: RNNs use recurrent connections to model sequential data for sequence recognition and prediction.Their high-dimensional hidden states have nonlinear dynamics and function as memory.
  • I. INTRODUCTION: RNN training with gradient descent faces vanishing and exploding gradients when modeling long-term dependencies.These issues arise in training recurrent dynamics through time and are central challenges discussed by the paper.
  • I. INTRODUCTION: The survey reviews RNN fundamentals, training methods, architectures, regularization, applications, and research challenges.It focuses on discrete-time RNNs and recent advances in the field.
  • Model Architecture: A simple RNN contains input, recurrent hidden, and output layers, with hidden units connected across time.The hidden state summarizes necessary information from the past and supports output prediction at each timestep.
  • Activation Function: Sigmoid, tanh, and ReLU are common activation functions whose suitability depends on the task and data.Sigmoid and tanh can saturate and contribute to vanishing gradients, whereas ReLU can accelerate stochastic-gradient convergence but may leave neurons inactive.

C. Loss Function

RNN training minimizes timestep-wise output-target losses, but gradient-based learning becomes difficult on long sequences and can be computationally constrained. The section presents BPTT and related optimization considerations.

  • C. Loss Function: The loss function compares each output y_t with its target z_t and sums losses across timesteps.The appropriate loss depends on the task, including Euclidean distance for real-valued forecasting and cross-entropy for classification.
  • Gradient-based Learning Methods: Gradient descent updates model weights using derivatives of the error with respect to the weight matrices.Batch GD computes the gradient over the whole dataset for one update per optimization iteration.
  • Gradient-based Learning Methods: Batch gradient descent is computationally expensive for very large datasets and unsuitable for online training.Its update uses the full training set rather than arriving inputs individually.
  • Gradient-based Learning Methods: BPTT extends gradient descent through time by unfolding the RNN and propagating error signals backward across timesteps.The procedure is difficult because recurrent parameters interact with unstable hidden-state dynamics.
  • Gradient-based Learning Methods: Gradient-based algorithms struggle to capture dependencies as their duration increases because gradients can vanish or explode during temporal back-propagation.The paper identifies exponential gradient decay as the vanishing-gradient problem and exponential blow-up as the exploding-gradient problem.

1) Back-propagation through time (BPTT):

BPTT transports loss information through recurrent states, but repeated temporal Jacobian factors can suppress long-term contributions. The survey links this decay to conditions on recurrent weights.

  • 1) Back-propagation through time (BPTT):: When recurrent-matrix eigenvalues are below one, gradients can converge rapidly to zero, normally after 5∼10 back-propagation steps.This is attributed to repeatedly back-propagating through the recurrent matrix.
  • 1) Back-propagation through time (BPTT):: For long sequences, small recurrent weights can shrink gradients, while larger weights can make products grow without bound.The paper characterizes these effects through repeated matrix products and the spectral radius of the recurrent weight matrix.
  • 1) Back-propagation through time (BPTT):: As t−k increases, long-term dependencies move toward zero, producing the vanishing-gradient problem.The cited analysis describes this behavior through the temporal propagation of gradients.
  • 1) Back-propagation through time (BPTT):: The sufficient condition stated for vanishing gradients is that the largest singular value λ1 of WHH satisfies λ1 < 1.Under this condition, long-term gradient contributions decay as temporal distance increases.

3) Exploding Gradient Problem:

Exploding gradients arise when recurrent dynamics amplify back-propagated errors, destabilizing training. The section discusses clipping, stochastic optimization, momentum, and mini-batch methods as training approaches.

  • 3) Exploding Gradient Problem:: Gradient norm clipping constrains large gradients to mitigate exploding-gradient problems during RNN training.Mikolov and Pascanu described clipping methods using a threshold, which may be selected heuristically.
  • Stochastic Gradient Descent: Momentum accelerates optimization by accumulating a velocity vector along directions of persistent objective reduction.With curvature condition number R, the cited result says momentum can require R times fewer iterations than steepest descent for equal accuracy.
  • Stochastic Gradient Descent: Nesterov accelerated gradient changes the velocity update rule and can converge more efficiently than GD in particular settings such as deterministic convex optimization.Its performance depends on the momentum coefficient and suitable fine-tuning.
  • Mini-Batch Gradient Descent: Mini-batch gradient descent computes updates from multiple training samples and supports parallel processing on GPUs.Typical mini-batches contain 50 ≤ b ≤ 256 samples, though the size varies by application.
  • Mini-Batch Gradient Descent: Mini-batch GD can outperform GD and SGD because it supports parallelization and vectorized implementation.The cited discussion contrasts its performance with the faster individual updates of SGD.

6) Adam Stochastic Optimization:

The section surveys gradient-based, Kalman-based, second-order, and Hessian-free approaches for training RNNs, emphasizing their optimization mechanisms and trade-offs. These methods address difficult recurrent optimization problems but vary in computational cost, assumptions, and applicability.

  • Adam stochastic optimization: Adam estimates first- and second-order gradient moments to optimize a stochastic objective function.It uses initialized moment vectors, bias correction, and parameter updates with recommended decay rates β1 = 0.9 and β2 = 0.999.
  • Adam stochastic optimization: Adam is relatively simple to implement and suitable for very large datasets.The update equations use ϵ = 10−8.
  • Kalman-based learning: Extended Kalman filtering can reach training steady state faster than back-propagation for non-stationary processes and train online with limited data.Its RNN formulation assumes the optimum weight setting is stationary.
  • Kalman-based learning: DEKF reduces computational complexity and storage by applying extended Kalman filtering independently to each neuron while retaining only previous-step results.It ignores interdependencies between mutually exclusive weight groups and recursively solves a least-squares fitting problem.
  • Second-order optimization: Second-order methods can improve optimization near saddle points or local extrema but require difficult Hessian and inverse-Hessian computations in RNNs.Their advantages rely on a good quadratic approximation and their learning-rate burden differs from gradient descent.
  • Hessian-free optimization: Hessian-free optimization addresses Hessian computation challenges and can reduce gradient-based training difficulties in RNNs.It uses local curvature information and is compatible with weight sharing across time.
  • Hessian-free optimization: A novel damping mechanism enabled Hessian-free optimization to train RNNs on pathological synthetic datasets described as impossible to learn with gradient descent.The same discussion connects the optimizer with multiplicative RNNs for next-character prediction.

F. Global Optimization

The section surveys global optimization and deep architectural extensions for RNNs. These approaches target difficult nonlinear optimization and richer representations, while introducing computational, gradient-flow, and training-time constraints.

  • Global optimization: Evolutionary algorithms search over populations for local or global solutions across diverse optimization landscapes, but may converge slowly or become stuck.Reported drawbacks for RNN optimization include local minima or maxima and network stagnancy.
  • Global optimization: Genetic algorithms are the most common global optimization method for RNN training, alongside AEA and CMA-ES approaches.CMA-ES has been used to tune RNN parameters for language models.
  • Global optimization: Global optimization research for RNNs remains scattered and computationally complex, although small-population multi-agent methods may outperform SGD.The potential performance comparison is presented as a possibility rather than a general finding.
  • Deep RNN architectures: Deep transition and deep output RNNs insert MLP structures into transition or output stages to form richer nonlinear representations.The plus operator combines input and hidden state into a summary, while the predictor maps a summary to an output.
  • Deep RNN architectures: Higher-level input representations can simplify temporal-relationship modeling and have produced better results than original inputs in speech recognition and word embedding.A shortcut between input and hidden layers is used to enhance long-term dependencies.
  • Deep RNN architectures: Stacked recurrent layers encourage operation across multiple input timescales, but shallow transitions limit representational capacity unless higher layers feed back to lower layers.Deep hidden-to-output functions can also produce more compact historical summaries.
  • Deep RNN architectures: Adding nonlinear transition layers can worsen vanishing or exploding gradients and substantially increase training time because the layers must be unrolled and cannot be parallelized.These effects can impair long-term dependency learning.

B. Bidirectional RNN

Bidirectional RNNs extend recurrent modeling by using both past and future sequence context, while multidimensional variants generalize recurrence across data dimensions. These benefits come with sequence-boundary, memory, gradient, and evaluation limitations.

  • Bidirectional RNN: BRNNs use both past and future information instead of relying only on inputs preceding the current time.This avoids choosing a handpicked output delay to incorporate future context.
  • Bidirectional RNN: BRNN forward states iterate from t = 1 to T, backward states iterate from t = T to 1, and outputs combine the directional processing.The two directions require more complicated state and output updates than simple RNNs.
  • Bidirectional RNN: BRNNs require the start and end of the input sequence to be known in advance.This constraint matters for applications such as phoneme labeling in spoken sentences.
  • Multi-dimensional RNN: Multidimensional RNNs replace a one-dimensional update chain with a D-dimensional grid and use recurrent connections for each dimension.They are described as a special case of directed acyclic graph RNNs.
  • Multi-dimensional RNN: MDRNNs offer robustness to warping and flexible context use, including sequential patterns in image and video analysis.Their recurrent structure can leverage dependencies that other architectures may ignore.
  • Multi-dimensional RNN: MDRNN memory usage can become very large as recurrent connections increase, while vanishing gradients can prevent learning long-term information across dimensions.The passage also reports no comparative examinations between MDRNNs and RCNNs.

1) Standard LSTM:

Standard LSTM uses gated memory cells to control information flow and preserve signals over multiple timesteps. The architecture supports longer sequence correlations but has higher hidden-layer complexity and may not capture structures beyond sequences.

  • Standard LSTM: An LSTM cell contains input, forget, and output gates plus a cell activation component.These components receive activation signals from different sources and control cell activation through designed multipliers.
  • Standard LSTM: LSTM gates prevent the rest of the network from modifying memory-cell contents for multiple timesteps, allowing signals and errors to persist longer than in ordinary RNNs.This mechanism supports learning longer sequence correlations.
  • Standard LSTM: The input, forget, cell, and output gates are computed from weighted signals from inputs, hidden states, and, for specified gates, cell activation.The resulting hidden state is computed after the gate and cell updates.
  • Standard LSTM: S-LSTM extends memory blocks to hierarchical tree structures so nodes can incorporate information from multiple descendants.The architecture is designed to address gradient vanishing and longer-term dependencies for inputs more complex than a sequence.

2) S-LSTM:

S-LSTM is presented among recurrent architectures that extend LSTM-style processing for sequential and multidimensional data, with competitive results but limited comparison evidence.

  • S-LSTM achieves competitive results against recursive and LSTM models.
  • S-LSTM performance is not compared with other state-of-the-art LSTM models.
  • Stacked LSTMs increase network capacity by arranging LSTM hidden layers in space.
  • Stacked LSTM structures can use a stack pointer to select state and memory from a previous timestep.
  • Deep bidirectional LSTMs stack LSTM hidden layers and theoretically use information from the full input sequence.
  • Multidimensional and spatial LSTMs extend memory across input dimensions, including pixel neighborhoods in image modeling.

7) Differential Recurrent Neural Networks:

Differential recurrent neural networks use changes in hidden-state information to emphasize informative dynamics, while the survey also situates them among alternative gated and recurrent architectures.

  • 7) Differential Recurrent Neural Networks:: dRNNs detect important spatio-temporal sequences by monitoring changes in hidden states between successive frames.A large derivative of state indicates sudden action-state change and informative dynamics.
  • 7) Differential Recurrent Neural Networks:: The derivative of state controls dRNN gating so large information changes update the memory cell, while small changes limit input influence.
  • 7) Differential Recurrent Neural Networks:: First- and second-order dRNNs perform better in training than simple LSTMs but add computational complexity.
  • F. Gated Recurrent Unit: GRUs adaptively capture dependencies at different time scales without separate memory cells.
  • F. Gated Recurrent Unit: Comparisons found that GRUs and LSTMs each performed better than the other only on certain tasks.The cited study therefore does not identify one model as universally better.

G. Memory Networks

Memory-oriented recurrent models augment conventional RNN state with explicit, contextual, or constrained memory mechanisms to retain information across sequences.

  • G. Memory Networks: Memory neural networks add readable and writable memory components to address the small memory size of conventional RNNs.
  • G. Memory Networks: Recurrent memory networks compare an LSTM hidden state with recent inputs through attention and extract knowledge from retained information over time.
  • G. Memory Networks: Dynamic memory networks retrieve experiential facts from inputs conditioned on a question and reason over them for a final representation.
  • G. Memory Networks: On Facebook bAbI, DMN passed 18 tasks with accuracy above 95%, while MemNN passed 16 tasks with lower accuracy.
  • H. Structurally Constrained Recurrent Neural Network: SCRN adds an identity recurrent matrix and context features to represent longer-term dependencies alongside rapidly changing hidden states.
  • H. Structurally Constrained Recurrent Neural Network: Adaptive context features learn different time delays, while fixed context weights force hidden units to capture one time scale.

I. Unitary Recurrent Neural Networks

The survey reviews unitary and related recurrent approaches for stabilizing long-range learning, alongside subsampling and regularization strategies for computational and generalization challenges.

  • I. Unitary Recurrent Neural Networks: Unitary matrices replace general recurrent matrices because their eigenvalues have absolute value exactly one.
  • I. Unitary Recurrent Neural Networks: Unitary RNNs preserve vector-flow and gradient norms across timesteps, preventing vanishing or exploding gradients from arising.
  • I. Unitary Recurrent Neural Networks: Unitary RNNs retain vanilla-RNN internal structure without additional memory requirements or noticeable increases in computational cost.
  • I. Unitary Recurrent Neural Networks: Gated orthogonal recurrent units combine an orthogonal hidden-state matrix with GRU-style forgetting mechanisms and an augmented ReLU for complex inputs.
  • K. Hierarchical Subsampling Recurrent Neural Networks: HSRNNs shorten long sequences through fixed-window subsampling, reducing computational costs but decreasing robustness to sequential distortions.
  • K. Hierarchical Subsampling Recurrent Neural Networks: HSRNN performance requires task-dependent window-size tuning because reducing sequence size makes the model less robust to sequential distortions.
  • Regularization adds penalties to control network capacity and prevent parameter configurations from overfitting.

B. Dropout

Dropout regularizes RNNs by randomly omitting connections or preserving selected hidden activations across time. RNN-specific masks and activation stabilization methods aim to improve temporal information flow, robustness, and long-sequence generalization.

  • Dropout: Dropout randomly omits a fraction of connections between network layers during training, while testing retains all units with possibly weighted activations.A binary mask and element-wise product implement the withdrawal operation.
  • Dropout: RNNDrop generates one dropout mask per training sequence and adjusts it over time, keeping network connections constant through the sequence.Other RNN dropout implementations instead drop the previous hidden state.
  • Activation stabilization: The norm-stabilizer adds a loss penalty that stabilizes hidden-vector norms when the RNN generalizes over long-term sequences.The hidden activations at consecutive time steps are used, and β controls the regularization strength.
  • Activation stabilization: Experiments in language modeling and phoneme recognition report state-of-the-art performance for activation stabilization, although inputs and nonlinearities can affect transition-matrix stability.Orthogonal matrices are another approach for stabilizing hidden-to-hidden transitions.
  • Zoneout: Zoneout preserves some units' previous-timestep activations, injecting stochasticity that improves hidden-state robustness and helps avoid overfitting.Zoneout has slightly better performance than dropout and can be combined with dropout and other regularization methods.

C. Image

RNNs are applied to image and video tasks by modeling contextual or temporal dependencies alongside learned visual features. The survey covers scene labeling, image generation and recognition, medical imaging potential, and CNN–RNN video systems.

  • Image: RNNs have shown promising results in image labeling, image modeling, and handwriting recognition, although CNNs dominate many vision applications.Their value lies in modeling dependencies that extend beyond individual spatial features.
  • Image: A DAG-RNN scene-labeling system combines convolutional feature mapping, recurrent contextual modeling, and deconvolutional upsampling to process image dependencies.The reported implementation outperformed other state-of-the-art models on SiftFlow, CamVid, and Barcelona.
  • Image: DRAW combines spatial attention with sequential variational auto-encoding to construct complex images iteratively rather than modeling all pixels with one latent distribution.The model generates image sections independently over successive steps.
  • Image: RNNs support handwriting recognition by modeling contextual dependencies in letter sequences, including online recognition with BLSTM models.Online recognition records pen-tip positions at intervals and maps them to word sequences.
  • Video: Video-description systems combine CNN visual features with LSTM sequence decoding, progressing from supervised intermediate representations toward end-to-end training.Later work further uses 3-dimensional convolution for feature extraction.
  • Image: RNNs have potential for 3-dimensional medical images, where temporal dependencies between scans may support cancer detection and segmentation.Examples include head MRI, lung CT, and abdominal MRI modalities.
Loading 1801.01078v3…