Capacity and Trainability in Recurrent Neural Networks

Jasmine Collins; Jascha Sohl-Dickstein; David Sussillo

Source-linked AI summary

Capacity and Trainability in Recurrent Neural Networks

Jasmine Collins, Jascha Sohl-Dickstein, David Sussillo

arXiv:1611.09913v3Nov 29, 2016stat.MLcs.AIcs.LGcs.NE

TL;DR

RNN expressiveness may be limited by task information stored in parameters or input history stored in units, and it is unclear whether gated models are more powerful or merely easier to train. The paper experimentally compares common and novel architectures across tasks and depths with careful training, finding similar capacity bounds but substantial trainability differences. Task capacity is approximately 5 bits per parameter, while input-history capacity is about one real number per hidden unit.

Problem
It was unclear whether gated RNNs outperform vanilla RNNs because they are computationally more powerful or because they are easier to train, amid limited evidence on RNN task capacity.
Method
The paper experimentally compares six RNN variants across tasks, depths, and parameter sizes, separating task-parameter capacity, input-history memory, and trainability with extensive hyperparameter optimization.
Results
Approximately 5 bits per parameter of task information and nearly one random input per hidden unit are stored across architectures, while vanilla RNNs are harder to train despite slightly higher capacity.
Takeaways & Limitations
For several tasks, parameter-limited task capacity—not per-unit input memory—determines performance, making trainability a primary practical distinction among architectures.
Takeaways & Limitations
The authors cannot claim that any model was perfectly trained, because hyperparameter optimization may encounter local minima and stochastic variation.

Abstract

from arXiv · show

Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.

1 INTRODUCTION

RNNs are widely used but difficult to design and train, leaving open whether gated architectures are more powerful or simply easier to optimize. This paper separates task-parameter capacity, input-history memory, computational primitives, and trainability across architectures and depths.

Motivation: Gated architectures such as LSTMs and GRUs are perceived as easier to train and lower-error than vanilla RNNs, but their practical computational advantage remains unresolved.The paper frames this as a distinction between greater computational power and improved trainability.
Experimental approach: The experiments disentangle task-parameter efficiency and input-history memory while applying extremely thorough hyperparameter optimization.The study also includes trainability experiments on difficult tasks.
Capacity framework: The paper introduces task memory and input-history memory as distinct capacity bottlenecks, with computational primitives representing another possible source of architectural differences.Different primitives may make some computations resource-heavy even with the same number of units and time steps.
Experimental approach: The study compares vanilla RNNs, IRNNs, GRUs, LSTMs, and two novel architectures—UGRNN and +RNN—across multiple stacking depths and mostly fixed parameter budgets.The UGRNN uses a coupled recurrent/update gate, while +RNN couples recurrent and depth gates.
Related work: RNN task-capacity theory is limited despite universal approximation results, while prior input-memory work generally studies short-term traces rather than storing a single input vector.The paper contrasts its empirical capacity measures with VC-dimension bounds and prior memory studies.
Novel architectures: The +RNN was developed to address trainability problems observed for LSTM and GRU in deeper architectures by combining coupled recurrent and depth gating.It is not intended for depth-one networks or layers whose input and output dimensions differ.

2 CAPACITY EXPERIMENTS

The experiments separate task information stored in parameters from input history stored in units, finding broadly similar capacity across RNN architectures but a stronger per-parameter bottleneck in tested tasks.

Experimental setup: The capacity experiment trains RNNs on random binary input-label mappings and maximizes mutual information between predicted and true labels.The number of mappings is treated as a hyperparameter, and the resulting mutual information is normalized by parameter count.
Per-parameter capacity: RNN task capacity scales roughly linearly with parameter count and averages approximately 5 bits per parameter across architectures and depths.Across architectures and depths, estimates range from 3 to 6 bits per parameter; larger networks are considered more reliable because of small-size effects.
Architecture comparisons: Gatedness slightly reduces capacity, with the observed ordering RNN > UGRNN > GRU > LSTM > +RNN after excluding IRNN.The comparison holds across most depths and parameter counts, while the architectures differ in their number of gates.
Activation effects: ReLU activation reduces capacity to roughly 3.5 bits per parameter in both RNN and UGRNN results when only ReLU-trained models are considered.This pattern provides evidence that the ReLU activation function is problematic for the capacity task, separate from IRNN’s identity initialization.
Per-unit input memory: Every studied architecture reconstructs an input of dimension n_in when hidden units per layer n_h satisfy n_h >= n_in, with comparable reconstruction-error curves.The curves correspond to reconstructing an n_h-dimensional subspace of the n_in-dimensional input.
Interpretation: The experiments identify per-parameter task capacity, rather than per-unit input memory, as the apparent performance bottleneck and make unit-based comparisons misleading.The paper therefore argues that architectures should be normalized by parameter count, which also matches the stated computational-cost scaling.

3 ADDITIONAL TASKS WHERE ARCHITECTURES ACHIEVE VERY SIMILAR LOSS

On language modeling and random function fitting, architectures perform extremely similarly when compared at the same parameter count. In these tasks, parameters rather than units provide the performance bottleneck.

Tasks: Two additional tasks—text8 character prediction and Random Continuous Functions—were selected to be trainable enough for evaluation loss to reveal capacity differences.The tasks were designed so that none of the tested model sizes could learn them perfectly, allowing performance to vary with model size.
Results: Across both tasks, parameter count rather than unit count limits performance, and architectures achieve extremely close losses at the same number of parameters.Matching one model’s loss to another requires adjusting parameter count by only a small factor.
Figure 3: Figure 3 compares text8 bits per character and RCF square error across model sizes, depths, and architecture types.Panels a-c and e-g use parameter count, while panel d uses hidden units for one-layer text8 models.

4 TASKS THAT ARE VERY HARD TO LEARN

On difficult parentheses and arithmetic tasks, gated architectures were substantially easier to train than vanilla RNNs, especially as depth increased. Random hyperparameter trials likewise favored GRU at depth 1 and +RNN at depth 8.

Across both difficult tasks and depths, vanilla RNN and IRNN performed worst and took longest to learn.Both eventually solved the tasks at depth 1.
As stacking depth increased, gated architectures became the only models able to solve the tasks.
GRU was most trainable at depth 1, whereas +RNN performed best on both trainability metrics for both tasks at depth 8.
Figure 4 tracks median and minimum evaluation error across hyperparameter-search iterations for parentheses and arithmetic tasks at depths 1, 2, 4, and 8.Vanilla-RNN loss was still decreasing after more than 2400 evaluations.
Across 1000 random hyperparameter trials, GRU had the smallest error and median error at depth 1, while +RNN had both at depth 8.The trials used 200k-parameter models on the parentheses task.

5 DISCUSSION

The experiments indicate that RNN architectures have similar capacity when carefully trained, while trainability and parameter capacity matter more for practical performance. The authors frame architecture choice as a trade-off between training resources, inference resources, task difficulty, and depth.

RNN variants store 3-6 bits per parameter about their task and nearly one random input per hidden unit.
Input-history memory does not appear to be a practical bottleneck for tasks not explicitly designed to measure it.Language-modeling error curves overlapped across architectures at equal parameter counts.
About 5 bits per parameter is the average task-capacity result across architectures.The authors note agreement with an estimate of 4.7 bits per biological synapse.
Per-parameter capacity rises for a few time steps and then saturates, suggesting diminishing returns from recurrence.
Performance is nearly constant across architectures when parameter count is fixed.The authors suggest this may motivate architectures with small compute-per-parameter ratios.
The authors cannot claim that any model was perfectly trained because hyperparameter optimization may suffer from local minima and stochasticity.Repeated runs of the best hyperparameters showed no serious deviations.
The authors recommend vanilla RNNs with abundant training resources, gated models with limited training resources, GRU for shallow depth, and +RNN for deeper depth.They caution that UGRNN and +RNN require further experiments.

A RNN HPS SET BY THE HP TUNER

The study uses extensive hyperparameter optimization to compare RNN architectures under controlled evaluation settings. The tuner selects architectural and learning hyperparameters, while parameter budgets and several implementation choices are specified across experiments.

Hyperparameters were optimized with batched Gaussian Process Bandits using expected improvement and a Matérn 5/2 kernel.
Evaluation performance was reported on the evaluation dataset, with dataset relationships varying by task.
Each RNN variant had 10-27 tunable hyperparameter dimensions spanning architecture, optimization, and regularization.
The tuner selected nonlinearities, matrix initializations, initialization scales, and bias distributions.
Learning-related hyperparameters included training steps, optimizer choice, learning-rate initialization, and learning-rate decay.Candidate optimizers were SGD, momentum SGD, RMSProp, and ADAM.
For the perceptron capacity task, dataset size was tuned between 0.1x and 10x the number of model parameters.
The tuner could control whether an input vector was presented only initially or at every time step.

PERCEPTRON CAPACITY

The capacity experiments measure how much task information RNN parameters can encode and how much input information hidden units can reconstruct. They use random-label classification for task capacity and delayed reconstruction for memory capacity.

The perceptron capacity objective trains RNNs on random inputs and labels while the tuner maximizes mutual information between outputs and true labels.
The inner optimization trains with cross-entropy, while the outer hyperparameter optimization selects sample count to maximize mutual information.
The memory-capacity task asks the network to reconstruct a 64-dimensional random input after 12 time steps using squared error.
A separate delayed-loss task uses 10^6 random unit-norm Gaussian inputs of dimension 50 and evaluates weighted squared error after 50 time steps.Weights follow a power law with characteristic time constant τ = 5000.
The text8 task predicts the next character in 10^8 Wikipedia characters using one-hot inputs and outputs with cross-entropy loss.Random pointers replace partial unrolling during language-model training.

PARENTHESES COUNTING TASK

The parentheses counting task tests whether an RNN can independently count opened parentheses across multiple parenthesis types while ignoring closing parentheses and noise characters.

Task definition: The task independently counts opened parentheses without counting their corresponding closing parentheses.It includes 10 parenthesis type pairs, such as <> and [].
Input representation: Each parenthesis type is represented with a 30D hot-one encoding, producing 300 input dimensions across parentheses and noise symbols.The 30D encoding combines 20D parenthesis symbols with 10D noise characters.
Input representation: The task includes 10 noise characters, ‘a’ through ‘j’, alongside the parenthesis symbols.

ARITHMETIC TASK

The arithmetic task presents an encoded addition problem and requires the network to output the correctly formatted encoded answer, with computation delayed after the equals sign.

Task definition: The network receives a hot-one encoded character sequence representing an addition problem and outputs the hot-one encoded answer.An example input is ‘-343243+93851= ’ and the answer is ‘-249392’.
Output requirements: The output must include the correct number of left-padded spaces before the first non-space answer character.
Timing condition: The task varies computation steps from 1 to 6 between reading ‘=’ and producing the first non-space output character.
Data generation: The two input numbers are sampled uniformly and independently from [−1e7, 1e7].

C HP ROBUSTNESS

The robustness analysis tests whether selected hyperparameters remain reliable across repeated optimization runs and whether randomly sampled hyperparameters produce statistically distinct architecture losses.

Best-hyperparameter robustness: 100 reruns using the best hyperparameters for 100k-parameter, one-layer parentheses models measured infeasible trials and loss variability.The reruns examined minimum, mean, maximum, and standard deviation of loss.
Best-hyperparameter robustness: The best hyperparameters produced a loss distribution very close to the original result across repeated runs.
Random-hyperparameter comparisons: Welch’s t-tests found statistically significant evaluation-loss distribution differences for 8-layer GRU–UGRNN, IRNN–RNN, and LSTM–RNN pairs under randomly sampled hyperparameters.The table reports significance using p > 0.05 as stated in the passage.