Source-linked AI summary
Recurrent Highway Networks
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, Jürgen Schmidhuber
TL;DR
Deep recurrent transitions are difficult to train because gradient propagation becomes unstable, motivating better ways to increase per-step nonlinear depth. The paper analyzes RNN Jacobians with Geršgorin’s circle theorem and introduces Recurrent Highway Networks, whose experiments show strong language-modeling performance. RHNs improve Penn Treebank perplexity as transition depth rises and achieve 1.27 bits per character on text8 and enwik8 while outperforming previous models with fewer parameters.
Problem
Deep recurrent transition functions are difficult to train because gradients can vanish or explode during backpropagation.
Method
The paper uses Geršgorin-circle analysis of RNNs to motivate Recurrent Highway Networks with Highway layers inside recurrent transitions.
Results
RHNs outperform previous results on text8 and enwik8, achieving 1.27 bits per character with significantly fewer total parameters.
Takeaways & Limitations
RHNs provide a more versatile mechanism for dynamically remembering, forgetting, and transforming information while increasing recurrent-transition depth.
Takeaways & Limitations
Coupling the carry gate to the transform gate reduces model size and stabilizes training but imposes a modeling bias that may be suboptimal for certain tasks.
Abstract
from arXiv · showhide
Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with 'deep' transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Gersgorin's circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks, which extend the LSTM architecture to allow step-to-step transition depths larger than one. Several language modeling experiments demonstrate that the proposed architecture results in powerful and efficient models. On the Penn Treebank corpus, solely increasing the transition depth from 1 to 10 improves word-level perplexity from 90.6 to 65.4 using the same number of parameters. On the larger Wikipedia datasets for character prediction (text8 and enwik8), RHNs outperform all previous results and achieve an entropy of 1.27 bits per character.
1. Introduction
RNNs benefit from depth but typically use only a single nonlinear transformation per recurrent step, while deeper transitions are difficult to optimize. The paper addresses this challenge by introducing Recurrent Highway Networks, which place Highway layers inside recurrent transitions.
- RNN state updates typically use a single trainable linear transformation followed by a non-linearity, limiting their use of depth.
- Deep networks and standard RNNs can suffer vanishing or exploding gradients because gradient magnitudes may shrink or grow exponentially during backpropagation.
- Highway Layers enabled training of feedforward networks with hundreds of stacked layers and improved performance in speech recognition and language modeling.
- Recurrent Highway Networks use Highway layers inside recurrent transitions to create long credit-assignment paths in both time and per-step computation.
- The paper combines a mathematical analysis of RNNs with RHNs to build more powerful and trainable sequential models that outperform existing architectures on benchmarks.
2. Related Work on Deep Recurrent Transitions
Prior approaches increased recurrent depth through stacking, micro-time-step recurrence, or added nonlinear transition layers, but deeper credit-assignment paths made optimization difficult. The paper motivates RHNs as an alternative way to obtain recurrence depth and modeling power.
- Stacked RNNs require credit assignment across both space and time, creating practical training difficulties.
- Micro-time-step methods adapt recurrence depth to the task but require the RNN to learn how to allocate parameters between memory and nonlinear processing.
- Deep Transition RNNs add multiple nonlinear layers to recurrent transitions, but their extremely long credit-assignment paths exacerbate gradient-propagation problems.
- A stacked RNN of depth d has maximum path length d + T − 1 across states T steps apart, whereas recurrence depth d gives a maximum path length of d × T.
- Recurrence depth can provide greater modeling power and efficiency than stacking, while also making architectures harder to train.
3. Revisiting Gradient Flow in Recurrent Networks
The paper analyzes recurrent gradient flow through Jacobians and Geršgorin circles to connect recurrent-matrix structure with eigenvalue behavior. This analysis clarifies conditions for vanishing or exploding gradients and motivates more flexible control of recurrent dynamics.
- Gradient propagation: A standard RNN applies y[t] = f(Wx[t] + Ry[t−1] + b), and loss derivatives are expanded through time using the chain rule.
- Gradient propagation: The error transported from time step t2 to t1 is obtained by chaining Jacobian derivatives across all intervening time steps.
- Gradient conditions: Vanishing gradients arise when the activation-derivative bound multiplied by the largest singular value of the recurrent matrix is below 1.
- Geršgorin analysis: Geršgorin circles bound eigenvalue locations using diagonal entries as centers and sums of absolute off-diagonal entries as radii.
- Gradient conditions: If the Jacobian spectral radius ρ exceeds 1, exploding gradients emerge because its norm is at least ρ.
- Geršgorin analysis: Largest-singular-value and spectral-radius analyses give boundary conditions but do not describe the overall eigenvalue distribution.
- Initialization: Small recurrent-weight initialization tends to place eigenvalues below unit magnitude, whereas larger variance expands the circles and can produce eigenvalues associated with exploding gradients.
- Gating: Unlike LSTM variants, other RNNs lack a direct mechanism for rapidly regulating Jacobian eigenvalues across time steps.
4. Recurrent Highway Networks (RHN)
Recurrent Highway Networks place one or more Highway layers inside the recurrent state transition, extending LSTM-like gating to recurrence depth greater than one. Their gated Jacobian analysis explains how carry and transform gates regulate temporal dynamics and supports deeper, trainable transitions.
- RHN construction: Highway layers use H, T, and C transformations, with transform and carry gates regulating nonlinear transformation versus carrying the input.T and C typically use sigmoid nonlinearities; H produces the transformed signal.
- RHN construction: RHN layers insert one or multiple Highway layers into the recurrent state transition, with the number of layers defining recurrence depth L.The first layer directly transforms x[t], while subsequent layers process preceding layer outputs; the final layer produces y[t].
- RHN construction: An RHN layer with L = 1 is essentially a basic variant of an LSTM layer, while L > 1 supports more complex state transitions.The architecture retains multiplicative gates controlling information flow through self-connected additive cells.
- Jacobian analysis: Gershgorin-circle analysis shows that RHN gates flexibly adjust the temporal Jacobian spectrum between copying information and applying the nonlinear transform.Fully open carry gates and closed transform gates set all eigenvalues to 1; the opposite limit yields the eigenvalues of H′.
- Jacobian analysis: The analysis motivates stacking multiple H, T, and C transformations with Highway layers to increase expressive power while retaining simplicity and ease of training.The authors distinguish RHNs from approaches that merely stack recurrent layers or use multiple micro time steps.
5. Experiments
Experiments evaluate whether Recurrent Highway Networks make deep recurrent transitions easier to optimize and improve language modeling at fixed parameter budgets. Across JSB Chorales, Penn Treebank, text8, and enwik8, deeper RHNs remain trainable and achieve strong reported results.
- 5. Experiments: The experiments couple the carry gate to the transform gate, reducing model size and preventing unbounded state-value blow-up while introducing modeling bias.For optimization and Wikipedia experiments, transform gates are biased toward being closed at the start of training.
- 5.1. Optimization: RHNs are evaluated as architectures intended to enable optimization of recurrent networks with deep transitions.The optimization study compares RHNs with DT-RNNs and DT(S)-RNNs on JSB Chorales next-step prediction.
- 5.1. Optimization: As recurrence depth increases, DT-RNN and DT(S)-RNN optimization becomes considerably harder, whereas RHN optimization is not adversely affected.The comparison uses equal parameter counts across architectures at each depth, while total parameters increase with depth.
- 5.2.1. Penn Treebank: 67.9/65.4 validation/test perplexity is achieved on Penn Treebank with a 32 M-parameter RHN at recurrence depth 10.The same hyperparameters are used across depths, and test perplexity improves as recurrence depth increases from 1 to 10.
- 5.2.1. Penn Treebank: Increasing recurrence depth from 1 to 10 reduces network width from 1275 to 830 units while keeping the parameter count fixed.The result demonstrates benefits from allocating parameters to depth even as the RNN state size decreases.
- 5.2.2. Wikipedia: On text8, a 1500-unit RHN with recurrence depth 10 achieves 1.19/1.27 validation/test BPC and outperforms previous best models with fewer parameters.The same model and hyperparameters achieve 1.26/1.27 validation/test BPC on enwik8, differing only in embedding size.
6. Analysis
The analysis examines how RHN layers use adaptive computation and contribute to performance. It also describes how transform-gate activity varies across layers, sequences, and time steps.
- Gate analysis: All six recurrent-transition layers are utilized after training, despite transform gates being biased toward carry behavior at initialization.The first layer is typically highly active, while later layers vary their activity over time.
- Gate analysis: RHN Highway layers perform adaptive computation by dynamically adjusting the effective amount of transformation for each sequence and time step.The maximum depth is bounded by the RHN recurrence depth.
- Gate analysis: Transform-gate activity differs across four example sequences and across Highway layers, with high values indicating greater input transformation.Low activity corresponds to carrying inputs forward to the next layer.
- Lesioning experiment: Layer contributions can be measured by biasing one Highway layer at a time toward carry behavior and recording the resulting training-loss change.The loss change estimates each layer’s contribution to network performance.
7. Conclusion
The paper develops a Geršgorin Circle Theorem analysis of recurrent networks and introduces Recurrent Highway Networks. Experiments support their optimization advantages and improved sequence-modeling performance.
- Conclusion: The paper analyzes recurrent-network behavior using the Geršgorin Circle Theorem.The analysis provides insights into how gates variably influence learning in a simplified LSTM version.
- Conclusion: Recurrent Highway Networks increase depth in the recurrent transition while retaining the ease of training associated with LSTMs.The model is designed to exploit deeper recurrent transitions.
- Conclusion: Experiments confirmed the proposed theoretical optimization advantages and improved performance on established sequence-modeling tasks.
8. Supplementary Material
The supplementary material documents experimental configurations and lesioning analysis for RHNs across language-modeling and sequence-modeling experiments.
- Experimental setup: The supplementary material provides precise experimental settings and reproduction code for Penn Treebank, enwik8, and text8 experiments.
- Experimental setup: Penn Treebank experiments compare RHNs with Deep Transition RNNs and skip-connected variants across multiple architectures and depths.The study uses 60 random hyperparameter settings for each architecture and depth.
- Experimental setup: Penn Treebank recurrence-depth experiments train one-layer RHNs from depth 1 to 10 under a fixed 32 M-parameter budget.The same hyperparameters are used across depths, with hidden-state sizes adjusted accordingly.
- Lesioning analysis: The lesioning experiment biases individual Highway layers toward carry behavior and measures the resulting log likelihood or training-error change.Figure 6 uses the lesioned-layer index on the x-axis and prediction log likelihood on the y-axis.