How to Construct Deep Recurrent Neural Networks

Razvan Pascanu; Caglar Gulcehre; Kyunghyun Cho; Yoshua Bengio

Source-linked AI summary

How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio

arXiv:1312.6026v5Dec 20, 2013cs.NEcs.LGstat.ML

TL;DR

The paper addresses how to define and construct depth in recurrent neural networks, where depth is ambiguous because temporal unfolding already creates nonlinear composition. It analyzes shallow RNN components, proposes deeper transition and output designs, and evaluates them on music prediction and language modeling. The proposed deeper variants outperform conventional shallow RNNs across the reported tasks, while deeper models introduce training difficulty that shortcut connections and pretraining are intended to mitigate.

Problem
Depth is ambiguous in RNNs, and the paper investigates how recurrent architectures can benefit from deeper component functions.
Method
The paper analyzes RNN components and proposes deep hidden-to-hidden transitions, deep hidden-to-output functions, and combined variants, including shortcut-connected designs.
Results
The proposed deeper variants outperformed conventional shallow RNNs across polyphonic music prediction and language modeling, with DOT(S)-RNN outperforming conventional and stacked RNNs on language modeling.
Takeaways & Limitations
The experiments support the paper’s claim that recurrent networks benefit from deeper architectures, similarly to feedforward neural networks.
Takeaways & Limitations
Deep RNNs are difficult to train, and the paper expects this difficulty may increase as model size and depth grow.

Abstract

from arXiv · show

In this paper, we explore different ways to extend a recurrent neural network (RNN) to a \textit{deep} RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

1 Introduction

The paper argues that depth is ambiguous in RNNs because temporal unfolding already creates nonlinear composition, while individual architectural components can remain shallow. It therefore studies alternative deep designs beyond stacked recurrent layers and evaluates them on sequence-modeling tasks.

Depth in an RNN is ambiguous because temporal unfolding already expresses the model as multiple nonlinear layers.
Stacking recurrent hidden states can let levels operate at different timescales, but each level’s hidden-state transition remains shallow.
The paper examines which RNN components are shallow and proposes deeper designs for each such part.
The resulting variants are empirically evaluated on polyphonic music prediction and language modeling.

2 Recurrent Neural Networks

An RNN is presented as a discrete-time dynamical system with inputs, outputs, and hidden states. Its transition and output functions are parameterized and trained by minimizing a divergence-based sequence cost.

An RNN simulates a discrete-time dynamical system with input x_t, output y_t, and hidden state h_t.
The state transition and output functions are parameterized by θ_h and θ_o, respectively.
The training objective sums a divergence between target outputs and predicted outputs across sequences and timesteps.
The conventional RNN uses transition, input, and output matrices with element-wise nonlinear functions.
Parameters can be estimated using stochastic gradient descent with gradients computed by backpropagation through time.

3.1 Why Deep Recurrent Neural Networks?

The paper motivates deep RNNs by extending the deep-learning hypothesis that hierarchical models can represent some functions more efficiently than shallow models. It argues that this rationale should also apply to recurrent networks.

Deep learning hypothesizes that hierarchical models can be exponentially more efficient than shallow models for representing some functions.
The paper uses theoretical and empirical evidence for deep feedforward models to motivate applying the same argument to RNNs.

3.2 Depth of a Recurrent Neural Network

RNN depth is ambiguous because temporal unrolling can make any RNN appear deep, while individual input-to-hidden, hidden-to-hidden, and hidden-to-output mappings may remain shallow. The paper distinguishes deep transition and deep output designs from stacked RNNs, which address different aspects and introduce distinct trade-offs.

Depth ambiguity: Unfolding an RNN through time makes computational paths cross multiple nonlinear layers, but this does not define architectural depth as clearly as in feedforward networks.The ambiguity arises from conflating temporal depth with depth within a single time step.
Three depth locations: At each time step, the input-to-hidden, hidden-to-hidden, and hidden-to-output functions are shallow because they lack intermediate nonlinear hidden layers.The paper treats these three mappings as separate opportunities for increasing depth.
Input and output depth: Deep input-to-hidden mappings can extract higher-level representations, while deep hidden-to-output mappings can disentangle hidden-state variation and support more compact summaries.The paper restricts its output-layer alternative to feedforward intermediate layers rather than conditional generative output models.
Deep transition: A deep hidden-to-hidden transition uses an MLP to model highly nonlinear updates that combine the previous hidden state with new input.The proposed DT-RNN is intended to represent complex state-summary updates beyond affine transformation followed by element-wise nonlinearity.
Trade-off: Deep transitions may make long-term dependencies harder to train because backpropagated gradients traverse more nonlinear steps; shortcut connections provide shorter paths.The shortcut variant is called DT(S)-RNN.
Combined variants: Combining deep hidden-to-output and deep hidden-to-hidden functions produces a DOT-RNN, with a shortcut variant called DOT(S)-RNN.These models extend the conventional RNN along multiple depth dimensions.
Stacked versus transition depth: Stacked RNNs encourage different recurrent levels to operate at different timescales, whereas DT-RNNs deepen within-level transitions; the two features are orthogonal.The paper notes that stacking DT-RNN levels could combine both properties, but does not explore that model.

3.3 Formal descriptions of deep RNNs

The paper formalizes deep RNN variants by replacing selected transition or output functions with multilayer perceptrons, and contrasts these designs with recursively computed stacked recurrent levels. The resulting formulations include DT-RNN, DO-RNN, DOT-RNN, and stacked RNN variants with optional shortcut connections.

Formal descriptions of deep RNNs: The formal section describes implementations of DT-RNN, DO-RNN, and stacked RNN architectures.These models correspond to different ways of deepening recurrent computations.
Deep Transition RNN: DT-RNN replaces the unrestricted state-transition function fh with a multilayer perceptron containing L intermediate nonlinear layers.The construction uses element-wise nonlinearities φl and weight matrices Wl for successive layers.
Deep Transition RNN: The DT-RNN transition is illustrated with one intermediate layer and is intended to learn a non-trivial, highly nonlinear mapping between consecutive hidden states.The intermediate network is inserted into the recurrent state update.
Deep Output RNN: DO-RNN models the output function fo with a multilayer perceptron containing L intermediate layers.Its layers use element-wise nonlinearities φl and output-side weight matrices Vl.
Deep Output RNN: Using both multilayered transition and output functions yields the DOT-RNN, illustrated with one intermediate layer in each function.The combined architecture deepens both recurrent transition and output computation.
Stacked RNN: A stacked RNN recursively computes hidden states from the bottom recurrent level upward, using the input xt at the first level.Each higher level receives the preceding level’s hidden state as its input.
Stacked RNN: After the top-level state is computed, the output may use that state alone or all hidden states, and shortcut connections may also connect levels to the input.These alternatives broaden how stacked hidden representations contribute to the output and recurrent computation.

4 Another Perspective: Neural Operators

The operator-based framework represents RNN computation through predefined neural operators, providing a way to construct deep-transition and deep-output RNNs. In this paper, the framework uses plus and predict operators and notes a possible but unexplored regularization direction.

The operator-based framework builds deep-transition and/or deep-output RNNs from a set of predefined neural operators.Each operator is implemented by a multilayer perceptron.
The plus operator receives an input and hidden-state vector and returns their hidden-state summary, while the predict operator produces the most likely output symbol from a summary.The hidden-state and summary vectors may be constrained to have identical dimensionality.
At each time step, an RNN applies the plus operator to update its hidden state and then the predict operator to compute its output.This gives the recurrence h_t = x_t ⊕ h_{t−1} followed by y_t = ▷h_t.
Using multilayer perceptrons for the operators yields the proposed deep transition, deep output RNN, called DOT-RNN.The MLP parameterization allows the operators to represent nonlinear operations rather than assuming linearity.
The framework could support regularizing the plus operator to be commutative, but the paper does not investigate this approach.

5 Experiments

The experiments compare four RNN variants on polyphonic music prediction and character- and word-level language modeling, using validation-selected model sizes and task-specific training procedures. Across these evaluations, deep variants generally outperform the conventional shallow RNN, with the strongest model depending on the task.

Experimental setup: Four RNN variants—conventional RNN, DT(S)-RNN, DOT(S)-RNN, and stacked RNN—are compared on next-symbol prediction tasks.The tasks include polyphonic music prediction, character-level language modeling, and word-level language modeling.
Experimental setup: Model sizes are selected from limited candidate sets to minimize validation error for each task.Polyphonic music models use sizes from {100, 200, 400, 600, 800}; language-modeling sizes use task-specific candidate sets.
Training: Training uses stochastic gradient descent with gradient clipping, and training stops when validation cost stops decreasing.For music prediction, subsequences are truncated but hidden states are preserved across subsequences from the same song.
Polyphonic music prediction: On polyphonic music prediction, at least one proposed deep RNN outperforms the conventional shallow RNN on every dataset, although the best deep variant depends on the data.The reported test metric is negative log-probability on test sequences.
Polyphonic music prediction: DOT(S)-RNN with Lp units, maxout units, and dropout performs significantly better than the other models on every music dataset.The authors connect this result to applying non-saturating activations and dropout in recurrent models.
Language modeling: On language modeling, deep RNNs significantly outperform the conventional shallow RNN, with DOT(S)-RNN outperforming the other evaluated models.The reported metrics are bit-per-character for character-level modeling and perplexity for word-level modeling.

6 Discussion

The proposed deep RNN designs improved on conventional recurrent models across the evaluated tasks, but their relative suitability depended on the dataset and training remained difficult for some architectures.

The proposed deep RNNs modeled shallow recurrent relationships with deeper networks and used shortcut connections to ease potentially difficult learning.The targeted relationships included consecutive hidden states and hidden states to outputs.
On language modeling, DOT(S)-RNN outperformed both conventional and stacked RNNs and achieved the state-of-the-art word-level result.
On polyphonic music prediction, different deeper variants achieved the best performance on different datasets, while conventional shallow RNNs did not outperform the deeper variants.
A quick experiment using non-saturating activations and dropout significantly improved deep-RNN performance on polyphonic music prediction tasks.The authors leave systematic investigation of these feedforward techniques for recurrent networks to future research.
Training DOT(S)-RNN and stacked RNN was not trivial, and the authors expect learning to become more problematic as model size and depth increase.Shortcut connections and pretraining with conventional RNNs or DT(S)-RNNs were used to address this issue.