PaperlayerView exact revision on arXiv

Source-linked AI summary

LSTM: A Search Space Odyssey

Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber

arXiv:1503.04069v2cs.NEcs.LG

TL;DR

The utility of LSTM components across diverse learning problems lacked systematic evidence. This study compares eight variants across three benchmark tasks using optimized hyperparameters and finds that none significantly improves vanilla LSTM, while the forget gate and output activation are most critical.

  • Problem

    A systematic study of the utility of LSTM components across learning problems differing from those used to test them was missing.

  • Method

    The study compares vanilla LSTM with eight single-change variants across three benchmark tasks, optimizing hyperparameters by random search and analyzing their importance with fANOVA.

  • Results

    None of the eight modifications significantly improves performance, while the forget gate and output activation function are the most critical LSTM components.

  • Takeaways & Limitations

    Vanilla LSTM performs reasonably well, and hyperparameters can be treated as approximately independent for practical tuning.

Abstract

from arXiv · show

Several variants of the Long Short-Term Memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful fANOVA framework. In total, we summarize the results of 5400 experimental runs ($\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

I. INTRODUCTION · II. VANILLA LSTM · A. Forward Pass

The paper motivates LSTMs as scalable models for sequential data, evaluates vanilla LSTM against isolated architectural variants, and defines vanilla LSTM’s gated memory and forward-pass computations.

  • I. INTRODUCTION: LSTMs address sequential-learning problems involving long time dependencies and have been applied to handwriting, language, speech, audio, video, and other data.Their central mechanism combines a memory cell that maintains state with nonlinear gates regulating information flow.
  • I. INTRODUCTION: Modern LSTM studies incorporate many architectural improvements, but these changes were initially tested on problems differing from current applications in scale and nature.This motivates systematic evaluation of commonly used LSTM components.
  • I. INTRODUCTION: The study compares vanilla LSTM with eight variants on acoustic modeling, handwriting recognition, and polyphonic music modeling, with each variant differing by a single change.Isolating one change per variant allows its effect on architecture performance to be assessed.
  • II. VANILLA LSTM: Vanilla LSTM is the literature’s most commonly used setup and serves as the reference architecture for comparing all variants.It incorporates changes introduced by Gers et al. and Gers and Schmidhuber into the original LSTM and uses full gradient training.
  • II. VANILLA LSTM: The vanilla LSTM block contains input, forget, and output gates, a block input, one Constant Error Carousel cell, an output activation function, and peephole connections.Its output is recurrently connected to the block input and all gates.
  • A. Forward Pass: The forward pass computes block input and gate preactivations from the current input, the previous output, biases, and peephole-connected cell states.The input and forget gates use c_t−1, while the output gate uses c_t.
  • A. Forward Pass: Gate activations use the logistic sigmoid, while the block input and output activations usually use tanh; vector products are elementwise.The notation uses σ for gates and g and h for block input and output nonlinearities.

B. Backpropagation Through Time

This section describes backpropagation through time in an LSTM by calculating deltas inside the block, optionally propagating input deltas downward, and computing weight gradients. The upstream delta corresponds formally to the loss derivative with respect to the output, excluding recurrent dependencies.

  • Deltas inside the LSTM block are calculated during backpropagation through time.
  • The upstream delta vector is formally associated with ∂E/∂y_t, excluding recurrent dependencies.Here, Δ_t is passed down from the layer above.
  • Input deltas are computed only when a lower layer requires them for training.
  • Weight gradients are calculated for each LSTM component using the outer product of two vectors.The component ⋆ can be any of {z̄, ī, f̄, ō}.

III. HISTORY OF LSTM … IV. EVALUATION SETUP

The paper traces LSTM’s evolution from an initial gated architecture through forget gates, peephole connections, and full BPTT, then evaluates eight one-aspect variants against vanilla LSTM across three domains. Hyperparameters are tuned separately for each variant to support fair comparison of achievable performance.

  • III. HISTORY OF LSTM: The initial LSTM included cells, input and output gates, but omitted forget gates and peephole connections in some experiments.Some experiments also omitted the output gate, unit biases, or input activation function, using mixtures of RTRL and BPTT.
  • A. Forget Gate: The forget gate enabled LSTM to reset its own state and learn continual tasks such as embedded Reber grammar.
  • B. Peephole Connections: Peephole connections let cells control gates, making precise timing easier to learn, while the output activation function was omitted as nonessential evidence was lacking.
  • C. Full Gradient: Full BPTT was introduced for the vanilla LSTM architecture and evaluated on TIMIT, while also enabling finite-difference gradient checks for more reliable implementations.
  • D. Other Variants: Other variants changed training methods, projections, gate slopes, or recurrent architecture, including Extended Kalman Filtering, evolution-based training, linear projections, and GRUs.The GRU omitted peephole connections and output activations, coupled input and forget gates, and produced mixed comparison results against vanilla LSTM.
  • IV. EVALUATION SETUP: The evaluation compares vanilla LSTM with eight variants, each adding, removing, or modifying exactly one baseline aspect across three datasets from different domains.The study prioritizes empirical comparison rather than state-of-the-art results, using a simple setup intended to keep comparisons fair.
  • IV. EVALUATION SETUP: Hyperparameters such as learning rate and input noise were tuned individually for each variant because variants may require different settings to achieve their best performance.Random search was used because the hyperparameter space was too large to traverse completely.

A. Datasets · TIMIT:

The study uses separate training, validation, and test sets, with validation supporting early stopping and hyperparameter optimization. It evaluates speech recognition on TIMIT, alongside handwriting and polyphonic music datasets described in the supplied passages.

  • A. Datasets: Each dataset is divided into training, validation, and test sets, with validation used for early stopping and hyperparameter optimization.The test set is reserved for final evaluation.
  • TIMIT:: TIMIT is used as a manageable acoustic-modeling benchmark for speech recognition, with frame-wise classification of each audio frame into one of 61 phones.The task uses 39 normalized inputs derived from MFCCs, energy, and their first and second derivatives.
  • TIMIT:: TIMIT performance is measured by classification error percentage on splits containing 3696 training, 400 testing, and 192 validation sequences.The sequences have 304 frames on average.
  • TIMIT:: The study restricts TIMIT to the established core test set and removes SA samples from training because they bias the phone distribution.The validation set uses some discarded samples from the full test set.
  • TIMIT:: IAM-OnDB represents English handwriting as pen-movement time series mapped to characters, with 775 training, 192 and 216 validation, and 544 test boards.The experiments use one line per sequence and combine the two validation sets.
  • TIMIT:: Handwriting sequences are subsampled to half length and represented by four-dimensional vectors encoding pen displacement, elapsed time, and pen lifts.No additional preprocessing was used, and training employs CTC with 82 outputs; performance is measured by Character Error Rate.
  • TIMIT:: JSB Chorales contains 382 four-part Bach chorales, represented as preprocessed piano-rolls sampled every quarter note for next-step prediction.The networks are trained by minimizing negative log-likelihood.

B. Network Architectures & Training · C. LSTM Variants

The study used task-specific LSTM architectures, losses, and training procedures, then defined eight variants of the vanilla LSTM by removing, altering, or expanding gating and activation components. The variants include gate omissions, identity activations, coupled input-forget gating, and full gate recurrence.

  • B. Network Architectures & Training: JSB Chorales used a single LSTM hidden layer with a sigmoid output layer.
  • B. Network Architectures & Training: TIMIT and IAM Online used bidirectional LSTMs with two hidden layers connected to one softmax output layer.The two hidden layers process inputs forwards and backwards in time.
  • B. Network Architectures & Training: Cross-Entropy Error was used for TIMIT and JSB Chorales, while IAM Online used Connectionist Temporal Classification loss.The IAM Online loss was the CTC loss by Graves et al.
  • B. Network Architectures & Training: Training used full BPTT for LSTMs and stopped after 150 epochs or after more than fifteen validation epochs without improvement.The learning rate was rescaled by a factor of (1 − momentum).
  • C. LSTM Variants: The vanilla LSTM used logistic sigmoid activation for σ and hyperbolic tangent activation for both g and h.The paper refers to this architecture as Vanilla (V).
  • C. LSTM Variants: NIG, NFG, and NOG remove the input, forget, and output gates, respectively, by setting the corresponding gate to 1.These variants are defined as it = 1, ft = 1, and ot = 1.
  • C. LSTM Variants: NIAF and NOAF replace the input or output activation function with the identity function.They set g(x) = x and h(x) = x, respectively.
  • C. LSTM Variants: CIFG couples input and forget gating with ft = 1 − it, whereas FGR adds recurrent connections between all gates and nine additional recurrent weight matrices.CIFG therefore uses one gate for both input and cell recurrent self-connection instead of independently learned forget-gate weights.

D. Hyperparameter Search

The study used parallelizable random searches to evaluate LSTM hyperparameters across nine variants and three datasets. It also tested dataset-specific TIMIT options and allowed parameter counts to vary across variants to avoid biased comparisons.

  • Search design: Random search was chosen because it is easy to implement, trivial to parallelize, and uniformly covers the search space for hyperparameter-importance analysis.This approach supported the study’s follow-up fANOVA analysis of hyperparameter importance.
  • Search design: 5400 trials came from 27 random searches, each containing 200 trials for one variant–dataset combination.The searches covered nine variants across three datasets.
  • TIMIT-specific hyperparameters: For TIMIT, momentum type and gradient clipping were additionally tested; momentum choice had no measurable effect, while clipping gradients to [−1, 1] hurt overall performance.Nesterov-style momentum was therefore selected for subsequent experiments, and gradient clipping was not retained.
  • Comparison assumptions: Parameter counts were not fixed across variants because variants may use their parameters differently, making fixed-count comparisons potentially biased.This differed from an earlier small-scale study.

V. RESULTS & DISCUSSION · A. Comparison of the Variants

Across 5400 experiments, the study reports strong task-specific best results and finds that most LSTM variants perform similarly to vanilla LSTM, with forget and output mechanisms especially important. Simplifying some components can preserve performance, whereas removing key gates or activations often harms recognition tasks.

  • V. RESULTS & DISCUSSION: 5400 experiments averaged 24.3 h each, totaling just below 15 years of single-CPU computation time.Each run used one of 128 AMD Opteron CPUs at 2.5 GHz.
  • V. RESULTS & DISCUSSION: 29.6% classification error was the best TIMIT result with CIFG, versus 26.9% for the best reported result.The paper also reports -8.38 log-likelihood for NIAF on JSB Chorales and 9.26% CER for NP on IAM Online, compared with 11.5% previously published.
  • A. Comparison of the Variants: The comparison used Welch’s t-test at p = 0.05, with Bonferroni correction for eight variant tests against vanilla LSTM.Variants were marked when their mean test-set performance differed significantly from vanilla LSTM.
  • A. Comparison of the Variants: The top 10% of trials were analyzed to reduce dependence on selected search ranges and represent reasonable hyperparameter-tuning efforts.The full distributions contain 200 test-set performances for each dataset–variant combination.
  • A. Comparison of the Variants: Removing the output activation function or forget gate significantly hurt performance on all three datasets.The authors identify forgetting old information and squashing the cell state as critical, except for the CEC.
  • A. Comparison of the Variants: CIFG and NP did not significantly change mean performance, while slightly improving the best result for music modeling and handwriting recognition, respectively.Both variants simplify LSTMs and reduce computational complexity.
  • A. Comparison of the Variants: NIG, NOG, and NIAF significantly reduced speech and handwriting performance but had no significant effect on music modeling.FGR did not significantly change TIMIT or IAM Online performance, worsened JSB Chorales results, and greatly increased parameters.

B. Impact of Hyperparameters

The study uses fANOVA to estimate hyperparameter importance from randomly sampled configurations rather than a full grid search. Regression trees, aggregated in a 100-tree random forest, predict averaged performance across hyperparameter slices and support variance decomposition.

  • Method: fANOVA efficiently estimates a hyperparameter’s marginal error by averaging over all other dimensions using regression trees.This avoids the full hyperparameter grid search traditionally required for marginalization.
  • Method: Random sampling of the hyperparameter space enables fANOVA analysis without exhaustively evaluating a full grid.The framework predicts marginal performance from sampled configurations.
  • Method: 100 regression trees are averaged to predict performance for hyperparameter-space slices, improving generalization and enabling uncertainty estimates.Slice performance is obtained by summing predictions over the relevant subset of dimensions.
  • Method: The resulting marginal predictions can decompose performance variance into additive components.This decomposition uses the obtained marginals from the regression-forest predictions.

Learning rate:

Learning rate is the most important hyperparameter, with dataset-dependent optima lying within broad basins of similarly good performance. Efficient tuning therefore favors coarse searches from high values, while allowing transfer from small to large networks because interactions with hidden-layer size are comparatively weak.

  • Learning rate:: Learning rate is the most important hyperparameter for achieving good performance.Figure 4 averages predicted test-set performance over all other hyperparameters and regression-forest trees.
  • Learning rate:: Up to two orders of magnitude can form a dataset-dependent basin of learning rates with little performance variation.The optimal value differs across datasets, with a sweet spot near the basin’s high end.
  • Learning rate:: A coarse search can start at 1.0 and divide the learning rate by ten until performance stops increasing.The high end of the good-performance basin combines good performance with small training time.
  • Learning rate:: Learning-rate variance is much larger than its interaction with hidden-layer size, supporting tuning on a small network before training a large one.The interaction contributes only part of the higher-order variance component.
  • Learning rate:: The investigated learning-rate range excluded higher values that would ideally have been included for IAM Online and JSB Chorales.This limitation affects interpretation of the apparent high-end sweet spot for those datasets.

Hidden Layer Size: … Analysis of Variance:

Hidden layer size improves LSTM performance with diminishing returns but increases wall-time training cost. Input noise and momentum generally provide little benefit, while learning rate dominates test-performance variance across hyperparameters.

  • Hidden Layer Size:: Larger hidden layers improve performance, but with diminishing returns.The hidden layer size is an important hyperparameter affecting LSTM performance.
  • Hidden Layer Size:: Increasing network size also increases wall-time training, reflecting both slower epochs and convergence effects.Wall-time includes increased computation per epoch and changes in convergence speed.
  • Input Noise:: Input noise almost always hurts performance and slightly increases training time.Additive Gaussian input noise is the only traditional regularizer discussed here.
  • Input Noise:: For TIMIT, input noise is the sole exception, producing a small error dip for noise values from 0.2 to 0.5.The observed exception applies specifically to the [0.2, 0.5] range.
  • Momentum:: Momentum affects neither performance nor training time significantly.For none of the datasets did momentum account for more than 1% of test-set performance variance.
  • Momentum:: For TIMIT, the learning rate × momentum interaction accounts for 2.5% of total variance.The passage notes this interaction without identifying a significant effect on performance or training time.
  • Analysis of Variance:: Learning rate is the most important hyperparameter, accounting for more than two thirds of test-set performance variance in every case.The hidden layer size ranks second, input noise third, and momentum contributes less than 1% of the variance.
  • Analysis of Variance:: Higher-order interactions are important for TIMIT but much less important elsewhere.The supplied passage truncates the comparison after stating that these interactions are much less important in other cases.

Interaction of Hyperparameters: · VI. CONCLUSION

The interaction analysis finds that hyperparameter interactions explain 5%–20% of test-performance variance but reveal little actionable structure. Overall, vanilla LSTM remains competitive, with the forget gate, output activation, and learning rate identified as especially important.

  • Interaction of Hyperparameters:: 5%–20% of test-set performance variance is explained jointly by hyperparameter interactions.The study visualizes pairwise interactions to help assess whether hyperparameter search can be accelerated.
  • Interaction of Hyperparameters:: ANOVA-based residual heat maps remove single-hyperparameter effects, so independent pairs should show zero residual interaction.The resulting plots distinguish total marginal performance from variation attributable specifically to interactions.
  • Interaction of Hyperparameters:: For TIMIT, hidden size and learning rate show performance patterns consistent with their separate marginal effects.Performance first decreases and then increases along learning rate, while it decreases slightly as hidden size grows.
  • Interaction of Hyperparameters:: For IAM Online and JSB, high learning rates (⪆10^-4) favor lower input noise (⪅.5), whereas lower learning rates reverse this trend.The reversal is not practically useful because performance is generally poor at low learning rates.
  • VI. CONCLUSION: None of the eight investigated LSTM modifications significantly improves performance over the vanilla LSTM.Coupled input-forget gates and removed peephole connections simplify the model without significantly reducing performance, while also reducing parameters and computational cost.
  • VI. CONCLUSION: The forget gate and output activation function are the most critical LSTM components, and removing either significantly impairs performance.The paper hypothesizes that output activation prevents the unbounded cell state from destabilizing learning.
  • VI. CONCLUSION: The learning rate is the most crucial hyperparameter, followed by network size; momentum is unimportant, while input noise helps TIMIT but harms other datasets.These conclusions concern the study’s online gradient-descent setting.
  • VI. CONCLUSION: Hyperparameters can be treated as approximately independent because interactions show no apparent structure and remain small, enabling learning-rate tuning with a small network first.The study uses experimental results to support practical intuitions and provide guidance on architecture selection and hyperparameter tuning.
Loading 1503.04069v2…