PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Evaluation of Neural Sequence Models

Ben Krause, Emmanuel Kahembwe, Iain Murray, Steve Renals

arXiv:1709.07432v2cs.NEcs.CL

TL;DR

Neural sequence models often miss patterns that recur in a test sequence. This paper develops dynamic evaluation, adapting model parameters to recent history with gradient descent, and reports state-of-the-art word- and character-level results across four datasets.

  • Problem

    Neural sequence models often cannot exploit new patterns that repeatedly occur in test sequences.

  • Method

    Dynamic evaluation continuously adapts training-time model parameters to recent sequence segments using gradient descent.

  • Results

    Dynamic evaluation gives large test-time improvements across character- and word-level language modeling and achieves state-of-the-art results on both datasets in each setting.

  • Takeaways & Limitations

    Dynamic evaluation is an effective method for exploiting pattern re-occurrence in sequences.

  • Takeaways & Limitations

    Neural caching and dynamic evaluation apply to autoregressive sequence prediction and generation, but not directly to more general supervised learning tasks.

Abstract

from arXiv · show

We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.

1 INTRODUCTION

Neural sequence models predict elements autoregressively across applications including language, speech, and forecasting. Dynamic evaluation addresses their difficulty exploiting repeated patterns in test sequences by adapting to recent history.

  • 1 INTRODUCTION: Neural networks model diverse sequence tasks autoregressively by predicting each element conditioned on preceding elements.Applications include speech recognition, machine translation, dialogue generation, speech synthesis, forecasting, and music generation.
  • 1 INTRODUCTION: Repeated patterns are common in sequences, but RNNs often fail to exploit patterns that recur during testing.Examples include repeated words, handwriting styles, and speech voices.
  • 1 INTRODUCTION: Dynamic evaluation adapts models to recent sequences with gradient descent and improves past approaches, enabling state-of-the-art results across language-modeling benchmarks.The methodology also reduces adaptation parameters and analyzes varying time-scales and distribution shifts.

2 MOTIVATION

Sequence distributions can vary locally across examples and over time, while static models use fixed parameters. Recent history can therefore inform adaptation toward the current sequence distribution.

  • 2 MOTIVATION: Autoregressive sequence models factorize sequence probability into conditional predictions based on preceding elements.This factorization underlies fixed-context and recurrent approaches.
  • 2 MOTIVATION: Long sequence histories contain recurring patterns that fixed-parameter models often struggle to capture.This motivates adapting beyond static modeling approaches.
  • 2 MOTIVATION: A sequence’s recent history contains useful information about its specific generating distribution, supporting adaptation of global parameters θg to local parameters θl.The local distribution may vary across sequences or continuously within one sequence.
  • 2 MOTIVATION: During evaluation, a model that infers the current local distribution from recent history has an advantage over a model optimized only for the global distribution.Training seeks a fixed model for Pg(x), whereas evaluation can exploit recent-sequence information about Pl(x).

3 DYNAMIC EVALUATION

Dynamic evaluation continuously adapts training-time model parameters during testing to better model a sequence’s local distribution. It segments long sequences, updates after each segment using gradient descent, and can also adapt during generation.

  • 3 DYNAMIC EVALUATION: Dynamic evaluation divides a test sequence into shorter segments and continuously adapts global parameters θg toward local parameters θl.The objective is a better model of the local sequence distribution Pl(x).
  • 3 DYNAMIC EVALUATION: Each segment produces a cross-entropy gradient through truncated back-propagation, which updates the model before processing the next segment.The same procedure is repeated sequentially across segments.
  • 3 DYNAMIC EVALUATION: Dynamic evaluation uses one forward pass and one gradient computation through the data, with computation linear in sequence length.Gradients are backpropagated only to the beginning of each segment, and each update approximates the current local distribution.
  • 3 DYNAMIC EVALUATION: During generation, the model generates each segment with fixed weights and then updates using that segment’s loss.This can produce generated sequences with more consistent regularities.

4 BACKGROUND

Prior adaptive language models include neural caching, while dynamic evaluation adapts model parameters with gradient descent. Both operate autoregressively at test time, but neural caching cannot alter recurrent hidden-state dynamics.

  • 4 BACKGROUND: Neural caching stores hidden-state and next-input pairs and interpolates cache probabilities with base-network probabilities.This increases weight for outputs associated with similar past hidden states.
  • 4 BACKGROUND: Dynamic evaluation differs from neural caching by changing model parameters with gradient descent rather than using a non-parametric nearest-neighbor-like adaptation.Both methods can be added to a base model for test-time adaptation.
  • 4 BACKGROUND: Both neural caching and dynamic evaluation apply to autoregressive sequence prediction and generation, but not directly to general supervised learning tasks.Their adaptation depends on observing sequence elements after prediction.
  • 4 BACKGROUND: Neural caching cannot adjust recurrent hidden-state dynamics, limiting its ability to capture jointly occurring information between successive elements.The limitation is especially relevant when individual elements have little independent meaning, such as character-level language modeling.
  • 4 BACKGROUND: Traditional dynamic evaluation used SGD updates at every time step, while this work explores broader methodology and prior applications had not examined it deeply.The paper’s later modification reduces update frequency by backpropagating over longer segments.

5 UPDATE RULE METHODOLOGY FOR DYNAMIC EVALUATION

The paper modifies dynamic evaluation with less frequent updates, a global decay prior, and RMSprop-derived scaling to improve gradient information, efficiency, and adaptation behavior.

  • Less frequent updates backpropagate over more timesteps, providing more accurate gradients and improving computational efficiency; segments use length 5 for word-level and 20 for character-level tasks.
  • A global decay prior exponentially reduces the influence of previous adaptation updates, biasing parameters toward the training-time values as the local distribution changes.
  • RMSprop scales each weight’s learning rate using a moving average of squared gradients, with mean squared gradients collected from training data rather than recent test data.
  • The final RMS global-prior update normalizes mean squared gradients, scales parameter-wise decay accordingly, and clips RMSnorm at 1/λ so no decay rate exceeds 1.

6 SPARSE DYNAMIC EVALUATION

Sparse dynamic evaluation reduces the memory cost of adapting separate parameters for multiple sequences by updating a smaller adaptation matrix instead of the network’s original parameters.

  • Mini-batched dynamic evaluation requires separate parameter sets for each sequence, creating high memory cost during parallel test-time processing.
  • The sparse variant initializes an adaptation matrix M to zero and applies it to RNN hidden states at every timestep, replacing the original hidden state throughout the network.
  • Applying dynamic evaluation only to M reduces adaptation parameters and makes mini-batching less memory intensive; restricting M to H hidden units yields H×H parameters.

7 EXPERIMENTS

The experiments evaluate dynamic evaluation on word- and character-level language modeling tasks, comparing it with static models and neural caching. Across PTB, WikiText-2, Hutter Prize, and text8, dynamic evaluation improves benchmark performance, while sparse adaptation and timescale analyses examine efficiency and behavior under distribution shifts.

  • 7.1 WORD-LEVEL LANGUAGE MODELLING: Dynamic evaluation improves both base models on PTB, surpasses the neural cache, and improves the state of the art.Experiments use a standard LSTM and an AWD-LSTM reimplementation; PTB perplexities are reported in Table 1.
  • 7.1 WORD-LEVEL LANGUAGE MODELLING: Dynamic evaluation improves state-of-the-art WikiText-2 perplexity and outperforms neural caching on both base models.WikiText-2 preserves dependencies across non-shuffled articles, which adaptive methods can exploit.
  • 7.2 CHARACTER-LEVEL LANGUAGE MODELLING: Dynamic evaluation achieves state-of-the-art results on both Hutter Prize and text8, with sparse evaluation improving Hutter Prize using 0.5% of regular adaptation parameters.Results are reported in Tables 3 and 4; sparse evaluation adapts 500 hidden units on Hutter Prize.
  • 7.3 TIME-SCALES OF DYNAMIC EVALUATION: Dynamic evaluation gains a noticeable advantage after a few hundred characters, growing on Spanish but peaking after roughly 2–3k characters on Hutter.The Spanish advantage is substantially larger than the Hutter advantage, based on Figure 2 analyses of average losses over processed characters.
  • 7.3 TIME-SCALES OF DYNAMIC EVALUATION: After 10k Spanish characters, static sampling switches toward English-like Hutter text, whereas dynamic sampling retains Spanish-like features throughout 300-character samples.Dynamic evaluation remains active during dynamic-model sampling.

8 CONCLUSION

The proposed dynamic evaluation methodology improves sequence modelling across character- and word-level language modelling, with applications to longer-context speech recognition and machine translation.

  • Dynamic evaluation gives large test-time improvements across character- and word-level language modelling.
  • The methodology can support speech recognition and machine translation over longer contexts, including broadcast speech and paragraph-level translation.
  • Overall, dynamic evaluation is shown to be effective for exploiting pattern re-occurrence in sequences.

P. J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78:

The supplied passages list prior work on neural sequence modelling and recurrent networks.

  • The cited related work includes backpropagation through time and recurrent neural network methods.
  • The references include multiplicative integration, recurrent neural network regularization, and recurrent highway networks.
  • The related work also includes neural architecture search with reinforcement learning.

A.1 DYNAMIC SAMPLES CONDITIONED ON SPANISH

The dynamic model generates 300-character samples after conditioning on 10,000 Spanish characters.

  • 300-character samples are generated from the dynamic model trained on the Hutter Prize dataset.
  • The model is conditioned on 10k Spanish characters before generating text.
  • The conditioning context ends with a Spanish sentence fragment supplied to the reader.

A.2 STATIC SAMPLES CONDITIONED ON SPANISH

The static model generates 300-character samples after conditioning on 10,000 Spanish characters.

  • 300-character samples are generated from the static model trained on the Hutter Prize dataset.
  • The static model is conditioned on 10k Spanish characters before generating text.
  • The conditioning context ends with a Spanish sentence fragment supplied to the reader.
Loading 1709.07432v2…