PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularizing and Optimizing LSTM Language Models

Stephen Merity, Nitish Shirish Keskar, Richard Socher

arXiv:1708.02182v1cs.CLcs.LGcs.NE

TL;DR

The paper addresses how to regularize and optimize LSTM language models, given that naïvely applying common regularization methods to recurrent networks has been unsuccessful. It proposes recurrent regularization, adaptive averaged SGD, and related strategies, achieving state-of-the-art perplexities and further gains with a neural cache.

  • Problem

    Naïvely applying successful feed-forward regularization methods to recurrent neural networks has not been highly successful, motivating effective regularization for LSTM language models.

  • Method

    The paper combines weight-dropped LSTMs, non-monotonic-trigger averaged SGD, variable-length BPTT, embedding dropout, activation regularization, and temporal activation regularization.

  • Results

    The proposed strategies achieve state-of-the-art perplexities on Penn Treebank and WikiText-2, with further improvement when combined with a neural cache.

  • Takeaways & Limitations

    The demonstrated regularization and optimization strategies are applicable to language modeling and are anticipated to generalize across other sequence learning tasks.

  • Takeaways & Limitations

    The proposed strategies may prevent use of black-box RNN implementations that can be many times faster through low-level hardware-specific optimizations.

Abstract

from arXiv · show

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

1. Introduction

The introduction motivates recurrent-specific regularization and optimization for LSTM language models, because naïve feed-forward techniques can disrupt recurrent dependencies and regularized models can be harder to train. The paper proposes implementation-compatible regularization strategies and a non-monotonic ASGD variant.

  • Regularization: Naïve dropout on RNN hidden states can disrupt long-term dependency retention, motivating recurrent-specific regularization.Keeping the same dropout mask across time steps is one proposed response.
  • Related approaches: Existing recurrent regularization approaches target hidden states, recurrent matrices, or activations, but activation-based methods can add parameters and training sensitivity.The surveyed approaches include matrix restrictions, element-wise interactions, batch normalization, recurrent batch normalization, and layer normalization.
  • Contributions: The paper proposes a weight-dropped LSTM that applies DropConnect to hidden-to-hidden recurrent weights without modifying existing LSTM implementations.It also investigates randomized-length BPTT, embedding dropout, activation regularization, and temporal activation regularization.
  • Practical compatibility: Implementation-compatible regularization can work with black-box libraries such as NVIDIA cuDNN, which may be many times faster than naïve LSTM implementations.Compatibility follows from requiring no LSTM implementation modifications.
  • Optimization: NT-ASGD determines ASGD’s averaging threshold through a non-monotonic criterion instead of user tuning and achieves better training outcomes than SGD.The introduction frames optimizer choice as especially important because dropout and other regularizers can impede training.

2. Weight-dropped LSTM

The weight-dropped LSTM regularizes recurrent connections by applying DropConnect to hidden-to-hidden weights while preserving the standard LSTM formulation and compatibility with optimized implementations.

  • LSTM formulation: The standard LSTM formulation uses weight matrices, timestep inputs, hidden states, memory-cell states, and element-wise multiplication.The supplied formulation passage explicitly defines x_t, h_t, c_t, and ⊙ alongside the gate and recurrent matrices.
  • Motivation: Prior recurrent regularization methods commonly modify hidden states or memory-state updates, which can prevent use of optimized black-box RNN implementations.The limitation is tied to modifications of the standard LSTM before the weight-dropped formulation.
  • Implementation: Applying DropConnect once to recurrent weight matrices before forward and backward passes preserves the RNN formulation and keeps training-speed impact minimal.This permits standard implementations, including NVIDIA’s cuDNN LSTM.
  • Weight-dropped LSTM: DropConnect is applied to the hidden-to-hidden matrices [U_i, U_f, U_o, U_c] to prevent overfitting on recurrent LSTM connections.The same approach is described as applicable to recurrent weight matrices in other RNN cells.
  • Mask behavior: The same dropped recurrent weights remain dropped across all timesteps, making the method similar to variational dropout but on recurrent weights.DropConnect can also be applied to non-recurrent weights, although the paper focuses on recurrent overfitting.

3. Optimization

The optimization section contrasts SGD and ASGD, then introduces NT-ASGD, which triggers averaging conservatively from validation behavior and avoids tuning the averaging threshold. The method uses a constant learning rate and fixed practical settings for its added logging and non-monotone intervals.

  • SGD: SGD trains deep networks by taking stochastic-gradient steps in a non-convex optimization problem.The stochastic gradient may be computed on a minibatch, and the learning rate can vary by iteration.
  • Language-model optimization: In neural language modeling, plain SGD without momentum has traditionally outperformed momentum SGD, Adam, Adagrad, and RMSProp by a statistically significant margin.This motivates improving SGD-based training rather than replacing it with another commonly used optimizer.
  • ASGD: ASGD averages iterates after a user-specified trigger instead of returning the final iterate, but its trigger and learning-rate schedule can be difficult to tune.Triggering too early harms efficacy, whereas triggering too late may require many additional iterations.
  • NT-ASGD: NT-ASGD removes the need to tune the averaging trigger by using a non-monotonic validation criterion that waits for multiple cycles without improvement.The algorithm checks validation perplexity at logging intervals and sets the trigger when it exceeds the recent minimum after the non-monotone interval.
  • Practical settings: NT-ASGD uses a constant learning rate and two added hyperparameters, with logging interval L set to an epoch and non-monotone interval n = 5 in the reported experiments.The authors report that these settings worked across various models and data sets and produced better training outcomes than SGD.

4. Extended regularization techniques

The paper extends LSTM regularization with variable-length BPTT, dropout variants, activation penalties, weight tying, and dimensionality reduction. These techniques target inefficient data usage, recurrent overfitting, and excessive parameterization while preserving practical training compatibility.

  • Variable length backpropagation sequences: Randomized BPTT sequence lengths ensure that, given enough epochs, all data elements experience a full backpropagation window.The method samples a base length, jitters it from a normal distribution, and rescales the learning rate to avoid favoring short sequences.
  • Dropout: Variational dropout reuses masks across an LSTM pass, while each minibatch example receives a distinct mask.DropConnect is reserved for hidden-to-hidden recurrent weights; variational dropout is used for other LSTM inputs and outputs.
  • Embedding dropout: Embedding dropout removes entire word embeddings during a forward and backward pass, causing every occurrence of a selected word to disappear.The remaining embeddings are scaled after dropout.
  • Weight tying and dimensionality: Weight tying shares embedding and softmax weights, reducing parameters and improving the standard LSTM language model.The architecture also reduces word-vector dimensionality by modifying the first and last LSTM layers.
  • Activation Regularization (AR) and Temporal Activation Regularization (TAR): Activation regularization penalizes large hidden activations, while temporal activation regularization penalizes large changes between successive hidden states.Both penalties are applied only to the final recurrent layer and use scaling coefficients.

5. Experiment Details

Experiments evaluate the methods on preprocessed Penn Treebank and WikiText-2 using a common three-layer, 1150-unit LSTM configuration. Training uses NT-ASGD, extensive dropout and regularization, gradient clipping, and dataset-specific batch and dropout settings.

  • Datasets: Experiments use preprocessed Penn Treebank and WikiText-2 language-modeling datasets.Penn Treebank has a 10,000-word vocabulary, while WikiText-2 is approximately twice as large and has over 30,000 words.
  • Model configuration: All experiments use a three-layer LSTM with 1150 hidden units and 400-dimensional embeddings.Embedding weights are initialized uniformly in [−0.1, 0.1], while other weights use a hidden-size-dependent interval.
  • Optimization: NT-ASGD trains models for 750 epochs with batch sizes of 80 for WikiText-2 and 40 for Penn Treebank.A subsequent ASGD run with T = 0 provides a fine-tuning step terminated by the same nonmonotonic criterion.
  • Regularization and training settings: Training uses gradient clipping at norm 0.25, initial learning rate 30, randomized BPTT, and dropout values of (0.4, 0.3, 0.4, 0.1).Weight-dropped recurrent matrices use dropout 0.5; WikiText-2 input dropout increases to 0.65.

6. Experimental Analysis

The AWD-LSTM models improve state-of-the-art single-model perplexity on both evaluated datasets. The results also show that a vanilla LSTM can remain highly competitive with more complex architectures when regularization and optimization are effective.

  • Perplexity results: AWD-LSTM improves state-of-the-art single-model perplexity on both Penn Treebank and WikiText-2.The vanilla LSTM beats the prior state of the art by approximately 1 perplexity unit on Penn Treebank and 0.1 units on WikiText-2.
  • Comparison with complex architectures: The model uses a vanilla LSTM rather than recurrent highway networks or task-tailored cells generated through reinforcement learning.Those competing approaches add architectural complexity beyond the standard LSTM.
  • Related findings: A separately studied LSTM language model also finds the underlying LSTM highly effective with well-tuned hyperparameters.The two approaches differ in architecture, optimizer, and hyperparameter-search strategy, and the paper identifies them as potentially complementary.

7. Pointer models

The paper augments its pretrained language model with a neural cache that combines cached hidden-state predictions with language-model probabilities. This simple pointer mechanism further lowers perplexity, particularly for repeated or rare content, but is less useful for frequent function-like words.

  • Neural cache: A neural cache stores previous hidden states and convexly combines its predictions with those of the pretrained language model.The cache has memory-window, mixture-coefficient, and additional hyperparameters, but no trained components are added.
  • Perplexity impact: The cache lowers perplexity by as much as 6 points on Penn Treebank and 11 points on WikiText-2.The gains are smaller than those reported for an LSTM without weight tying, but remain substantial.
  • Word-level effects: The largest cumulative cache gain concerns <unk> across over 11540 instances, while punctuation and stop words do not benefit.The cache also helps some relatively rare words, including Meridian, Churchill, Blythe, and Sonic.
  • Word-level analysis: Table 3 ranks words by cumulative summed log-perplexity differences between the LSTM-only and cached models on WikiText-2 validation data.The right column contains the twenty greatest improvements, while the left contains the twenty most deteriorated words.

8. Model Ablation Analysis

Ablations show that both NT-ASGD-related optimization choices and the proposed regularization strategies materially support the best LSTM performance. Removing recurrent regularization causes the largest degradation, while several other removals also worsen perplexity.

  • Optimization: Optimization ablations show that replacing averaged SGD with reduced-learning-rate SGD or a monotonic trigger significantly degrades performance.Removing fine-tuning also hurts performance, while repeating fine-tuning more than once provides little benefit.
  • Regularization: The full set of proposed regularization strategies was pivotal in ensuring state-of-the-art performance.The ablation table reports validation and test perplexities for variants on Penn Treebank and WikiText-2.
  • Regularization: Up to 11 perplexity points are added when hidden-to-hidden LSTM regularization from the weight-dropped LSTM is removed.This was the most extreme perplexity increase among the regularization ablations.
  • Architecture and training choices: Approximately one perplexity unit is lost with static sequence lengths, which the authors associate with inefficient data usage.The experiment tested the authors’ hypothesis that fixed lengths would use data less efficiently.
  • Architecture and training choices: Matching embedding and hidden-state sizes increases parameters to 43M on PTB and 70M on WT2, while degrading performance by almost 8 perplexity points.The authors attribute this degradation to overfitting in the word embeddings and note the computational overhead of larger embeddings.
  • Regularization: Removing embedding dropout, AR/TAR, or weight decay increases perplexity by 2–6 points, consistent with insufficient regularization.The authors report this increase for each of these removal variants.

9. Conclusion

The paper develops regularization and optimization strategies for neural language models, including weight-dropped LSTM and nonmonotonic averaged SGD. These methods achieve new state-of-the-art perplexities on PTB and WikiText-2, with a neural cache improving performance further.

  • Contributions: The weight-dropped LSTM applies DropConnect to hidden-to-hidden weight matrices to prevent overfitting across recurrent connections.The strategy is presented as a recurrent regularization method.
  • Contributions: Averaged SGD with a nonmonotonic trigger outperforms SGD by a significant margin for training language models.The trigger determines when iterate averaging is applied.
  • Results: Variable BPTT length and other regularization strategies produce new state-of-the-art perplexities on Penn Treebank and WikiText-2.The models also outperform custom-built RNN cells and regularization strategies that preclude optimized libraries such as NVIDIA cuDNN LSTM.
  • Results: A neural cache further improves the proposed model’s performance, attaining an even lower state-of-the-art perplexity.The cache is evaluated in conjunction with the proposed model.
Loading 1708.02182v1…