PaperlayerView exact revision on arXiv

Source-linked AI summary

Pointer Sentinel Mixture Models

Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher

arXiv:1609.07843v1cs.CLcs.AI

TL;DR

Neural language models struggle to retain and reproduce rare words from earlier context. The paper introduces a pointer-sentinel mixture that combines copying recent words with softmax generation, achieving state-of-the-art Penn Treebank perplexity while using few additional parameters and computation.

  • Problem

    Neural language models struggle to reproduce rare or unseen words from earlier context when the relevant context exceeds hidden-state capacity.

  • Method

    The pointer-sentinel mixture combines a softmax vocabulary distribution with a pointer distribution over recent context, using a sentinel-controlled gate.

  • Results

    The model achieves state-of-the-art Penn Treebank language-modeling results with few additional parameters and little additional prediction-time computation.

  • Takeaways & Limitations

    The paper introduces WikiText-2 and WikiText-103 as benchmarks for studying rare-word prediction and long-range dependencies in language modeling.

  • Takeaways & Limitations

    The pointer component evaluates only a tunable window of the most recent L words rather than the entire preceding dataset.

Abstract

from arXiv · show

Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.

1. Introduction

The introduction identifies rare, unseen-word prediction as a weakness of standard neural language models, even when context makes the answer clear. It presents a pointer-sentinel mixture that combines copying from context with softmax vocabulary prediction and improves Penn Treebank perplexity.

  • Motivation: Standard neural sequence models must store a context-specific rare word in their limited hidden state before decoding it later.This becomes difficult when the relevant context is separated from the prediction by many time steps.
  • Motivation: Even with attention, softmax classifiers often struggle to predict rare or previously unknown words.Attention can increase effective hidden-state capacity and provide gradient paths not tied to timesteps, but does not resolve this vocabulary limitation.
  • Prior approaches: Pointer networks can produce unseen input tokens and improve rare-word and long-term-dependency performance, but cannot select words absent from the input.This motivates combining pointer-based copying with a standard vocabulary classifier.
  • Contribution: The proposed mixture combines a softmax classifier with a pointer component, using a sentinel to let the pointer decide when to use the softmax vocabulary.The architecture is designed for effective and efficient language modeling and improves state-of-the-art Penn Treebank perplexity.

2. The Pointer Sentinel for Language Modeling

The pointer sentinel mixture model combines a standard RNN softmax vocabulary with a pointer distribution over recent context, using a sentinel gate to decide when to copy or back off. The design supports repeated words, unseen outputs, and long pointer windows while adding relatively little computational and parameter overhead.

  • Pointer component: Pointer sum attention aggregates probability across every input position containing the predicted word.This lets multiple occurrences of the same word share probability mass while preserving occurrence-specific context.
  • Pointer component: The pointer matches only the L most recent words, with L tuned on held-out data or estimated from word recurrence patterns.A finite window avoids evaluating attention over the entire dataset.
  • Mixture model: The model combines an RNN softmax component with a pointer component over positional vocabulary through a learned gating function.The gate selects between the two base distributions, with 0 indicating pointer use and 1 indicating softmax-RNN use.
  • Pointer sentinel: The sentinel adds a gate position to the pointer attention distribution, assigning its probability mass to the standard RNN softmax vocabulary.The gate is the final element of the attention distribution over the pointer window and sentinel state.
  • Pointer sentinel: Integrating the gate into pointer computation makes it depend on both the RNN hidden state and pointer-window states, enabling competition between copying and softmax back-off.This competition was crucial for obtaining the best model and avoids requiring the fixed-size RNN state to store every word identity and position in a long window.
  • Efficiency: The pointer sentinel-LSTM mixture model causes a relatively minor increase in parameters and computation time compared with standard LSTM models targeting similar performance.Parameter sharing between components further minimizes the total number of model parameters.

3. Related Work

Prior language-modeling work spans n-gram mixtures, neural sequence models, attention and memory mechanisms, and pointer-based architectures. The paper distinguishes its mixture model from switching-based pointer softmax approaches by combining pointer and vocabulary probabilities for ambiguous contexts.

  • Neural sequence models: Neural recurrent language models achieve state-of-the-art results, with research also exploring dropout regularization and deeper recurrent architectures.These methods target overfitting in complex LSTM models and improved handling of increased recurrence depth.
  • Attention and memory: Soft attention and memory mechanisms retrieve relevant previous hidden states but must encode and then decode a relevant word, even when it is unchanged.This creates an unnecessary representation cycle when the output word is identical to the input word associated with the retrieved state.
  • Pointer mechanisms: Pointer networks select elements from the input as output and have supported geometric problem solving, code generation, summarization, and question answering.They can only output elements present in the input; the cited example excludes February because it does not appear there.
  • Pointer mechanisms: Pointer softmax combines an RNN vocabulary softmax with a pointer location softmax, improving handling of rare and unknown words and enabling output of out-of-vocabulary words from the input.Unlike the present work, it uses a switching network rather than a mixture model.
  • Pointer mechanisms: The pointer softmax approach selects the word or location with the highest scaled attention score, whereas this model mixes pointer and RNN probabilities when both can produce the same word.Combining probabilities supports better predictions when the context is ambiguous.
  • Pointer mechanisms: The latent predictor network conditions output generation on an arbitrary number of base models with potentially different granularities, including character-level softmax and word-copying pointers.In code generation, it can generate characters with a standard softmax or copy entire words from referenced text fields.

4. WikiText - A Benchmark for Language Modeling

The section motivates WikiText as a more realistic, publicly available language-modeling benchmark than processed PTB and describes its construction, scale, and variants. WikiText preserves linguistic features and article structure that support longer-term dependency modeling.

  • PTB limitations: 929k training words, 73k validation words, and 82k test words comprise the pre-processed Penn Treebank dataset.PTB lower-cases words, replaces numbers, and limits the vocabulary to the most frequent 10k words, with remaining tokens replaced by ⟨unk⟩.
  • PTB limitations: 10k words and stripped punctuation make processed PTB unrealistic for language use involving large vocabularies and rare words.The dataset is also entirely lower case, and its Zipfian frequency curve ends abruptly at the vocabulary limit.
  • WikiText construction: 23,805 Good articles and 4,790 Featured articles were selected from Wikipedia to construct the WikiText corpus.The selected articles were editor-reviewed for writing quality, factual accuracy, broad coverage, neutrality, and stability, then extracted using the Wikipedia API.
  • WikiText scale: 103 million words make WikiText a hundred times larger than PTB while retaining articles suitable for learning longer-term dependencies.The corpus was created to avoid restrictive licensing and randomized sentence ordering found in other larger-scale datasets.
  • WikiText variants: WikiText-2 and WikiText-103 both preserve punctuation, original casing, larger vocabularies, and numbers, with WikiText-2 twice the size of PTB.WikiText-103 contains all extracted articles; both datasets use the same validation and test articles, differing only in vocabulary.

5. Experiments

Experiments use extended truncated-BPTT training and compare pointer sentinel-LSTM models with LSTM variants on Penn Treebank and WikiText-2. The pointer sentinel-LSTM achieves the lowest reported Penn Treebank perplexity and improves similarly over variational LSTM baselines on WikiText-2.

  • Training setup: The pointer component uses k1 = 1 and k2 = L, backpropagating L timesteps for each prediction while advancing one timestep at a time.Only the loss for the final predicted word is used for backpropagation through each window.
  • Training setup: Training uses 100 timesteps to match window length L, batch size 32, learning-rate halving after worsening validation perplexity, and early stopping by three unimproved epochs or 64 epochs.Gradients are rescaled when their global norm exceeds 1.
  • Models: The experiments compare dropout-regularized LSTM and pointer sentinel-LSTM models, using zoneout and variational-inference-based dropout variants.Zoneout is applied to recurrent LSTM connections, while variational dropout locks a layer’s mask across timesteps.
  • Penn Treebank results: The pointer sentinel-LSTM achieves the lowest perplexity on Penn Treebank, and its medium model outperforms the large LSTM models.It is followed by Recurrent Highway Networks, while the best large variational LSTM uses computationally intensive Monte Carlo dropout averaging.
  • Penn Treebank results: The pointer sentinel-LSTM’s variational-LSTM ablation performs worse than Gal (2015)’s variational LSTM without Monte Carlo dropout averaging.The ablation uses the same L = 100 timestep BPTT length as the pointer sentinel-LSTM.
  • WikiText-2 results: On WikiText-2, the pointer sentinel-LSTM makes a similar gain over the variational LSTM models, with no existing baselines available for the newly introduced dataset.The provided comparisons are against a zoneout variational LSTM and Gal (2015)’s medium variational LSTM.

6. Analysis

The pointer sentinel-LSTM benefits most on rare words but also improves predictions for frequent tokens. Qualitative analyses show pointer use for names, repeated measurements, and words far back in the context window.

  • Frequency effects: Pointer sentinel-LSTM improvements over LSTM grow as words become rarer, including on Penn Treebank despite its 10k-word vocabulary.The analysis attributes this pattern partly to reproducing rare words and enabling sharper word selection.
  • Frequency effects: The pointer component remains helpful for relatively frequent words, potentially selecting them directly or improving the RNN through pointer supervision.Pointer supervision can allow gradients to flow directly to other occurrences within the context window.
  • Qualitative analysis: Pointer use is heavy for rare names, including Seidman (23 times in training), Iverson (7 times in training), and Rosenthal (3 times in training).The analysis also reports heavy use for company names such as Honeywell (8 times in training) and Integrated (41 times in training).
  • Qualitative analysis: The pointer also handles frequent measurement and number-scale tokens through recent repetition, and can mix with softmax for verbs such as said.Examples include phrases like increased from N tons to N tons and tokens such as tons, kilograms, thousands, millions, and billions.
  • Long-range context: Pointer attention can reach position 97 in the 100-word window, beyond the 35-step truncation horizon, motivating integration of the gating function into the pointer component.This integration avoids requiring the RNN hidden state to maintain intensive bookkeeping about words near the pointer tail.

7. Conclusion

The paper introduces the pointer sentinel mixture model and WikiText datasets, reporting state-of-the-art Penn Treebank language modeling with few additional parameters and little added prediction complexity. WikiText-2 and WikiText-103 are proposed to support research on rare words and long-range dependencies.

  • Conclusion: The pointer sentinel mixture model achieves state-of-the-art Penn Treebank language modeling with few additional parameters and little additional prediction-time computational complexity.The model is the paper’s central contribution.
  • Conclusion: WikiText-2 and WikiText-103 are introduced as potential datasets for studying long-range dependencies beyond Penn Treebank.The conclusion motivates moving to a new language modeling dataset.
  • Conclusion: The WikiText datasets are intended to improve handling of rare words and the use of long-term dependencies in language modeling.The authors position the datasets as a platform for future progress.

Supplementary material

The supplementary analysis visualizes how the pointer sentinel mixture balances RNN and pointer components across predictions, including recent-context copying and long-range named-entity retrieval. It also shows that WikiText-103 provides reasonable coverage of the vocabulary’s long tail.

  • Pointer usage: The gate interpolates between RNN and pointer components, with 1 denoting exclusive RNN use and 0 denoting exclusive pointer use.The qualitative analysis orders examples from primarily RNN-based to primarily pointer-based predictions.
  • Pointer usage: For retailer and cancer-mortality predictions, the pointer suggests contextually fitting historical words while the gate remains primarily RNN-weighted.Suggested words include retailers, investments, chains, institutions, mortality, and rates.
  • Pointer usage: The pointer can receive substantial weight even for frequent words, selecting said nearly equally with the RNN and relying more heavily for a recent occurrence of billion.The billion example uses highly similar recent context despite the word’s frequency in Penn Treebank.
  • Long-range retrieval: The model retrieves named-entity surnames from 45 and 65 timesteps back, exceeding the 35-step truncation used by most language models.It correctly identifies iverson and rosenthal from longer context, with the pointer almost exclusively used for rosenthal.
  • WikiText-103: Over 100 million tokens in WikiText-103 provide reasonable coverage of the vocabulary’s long tail.The figure shows a Zipfian plot over the training partition.
Loading 1609.07843v1…