PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Limits of Language Modeling

Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu

arXiv:1602.02410v2cs.CL

TL;DR

Large-scale language modeling must handle huge corpora and vocabularies while modeling long-term structure. This paper studies recurrent and character-based approaches on the One Billion Word Benchmark, achieving 30.0 perplexity for a single model and 23.7 with an ensemble while reducing single-model parameters by 20×.

  • Problem

    Large-scale language modeling requires effective methods for huge vocabularies and corpora while preserving long-term language structure.

  • Method

    The paper explores recurrent language models, importance sampling, and character-CNN parameterizations for input and Softmax embeddings.

  • Results

    30.0 perplexity improves the best single-model result from 51.3 with 20× fewer parameters, while an ensemble reaches 23.7 perplexity.

  • Takeaways & Limitations

    The results show that large, regularized LSTM language models with projected states and approximate Softmax training can outperform competing N-grams without interpolation.

Abstract

from arXiv · show

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

1. Introduction

Language modeling is central to NLP and language understanding, but large-scale settings expose challenges in vocabulary size, corpus scale, and long-term structure. This work studies recurrent models and character-based techniques on large benchmarks, reporting substantial perplexity improvements and releasing models and training recipes.

  • Motivation: Language models support language understanding and downstream tasks including speech recognition, machine translation, and text summarization.They can encode grammatical structure and information from training corpora.
  • Motivation: Large-scale benchmarks matter because methods that work on small datasets can fail to improve on larger datasets.The authors emphasize that overfitting is a defining characteristic of PTB but not the main limitation in current language modeling.
  • Setting: The study focuses on the One Billion Word Benchmark, with 800k vocabulary and 1B training words, making it substantially larger and more challenging than PTB.The benchmark is presented as a clear large-scale setting for advancing language modeling.
  • Approach: The work explores and extends large-scale language-modeling research using recurrent models, character CNNs, and related architectures.Figure 1 contrasts a standard LSTM language model with character-CNN embeddings and a character-prediction LSTM.
  • Results: 51.3 to 30.0 perplexity improves the best single-model result while reducing parameters by a factor of 20.An ensemble further reaches 23.7 perplexity.
  • Impact: The authors release models and training recipes to help motivate further research in large-scale language modeling.The paper positions open resources alongside clear benchmarks as a way to support progress.

2. Related Work

The paper situates large-scale neural language modeling within work on recurrent models, character-based embeddings, and efficient vocabulary normalization. It emphasizes the One Billion Word Benchmark, large projected recurrent states, and the computational challenge of million-word Softmax layers.

  • Language Models: Recurrent language models retain long-term dependencies through LSTM gating, complementing successful count-based N-gram models.The paper frames RNNs and N-grams as having different, complementary strengths.
  • Large-Scale Modeling: The One Billion Word Benchmark is treated as very large for neural language models, making scale and data availability central to evaluation.The authors contrast this regime with smaller benchmarks such as PTB and relate progress to large-scale datasets in other fields.
  • Large-Scale Modeling: Projected recurrent states preserve large memory capacity while keeping the matrices involved relatively small, reducing computation despite many parameters.This design follows a projection strategy in which a large recurrent state space is mapped to a smaller representation.
  • Convolutional Embedding Models: Character-level models construct word embeddings from character sequences, extending their use beyond the small benchmarks where they had previously shown benefits.Prior approaches used bidirectional LSTMs or CNNs with max-pooling and highway layers for character-derived embeddings.
  • Softmax Over Large Vocabularies: Million-word vocabularies make full Softmax training prohibitively slow because every context vector must be compared with every word embedding.The paper reviews importance sampling, Noise Contrastive Estimation, self-normalizing methods, and hierarchical Softmax as solutions.

3. Language Modeling Improvements

The paper improves language modeling by approximating large-vocabulary normalization and replacing independently stored word embeddings with character-based parameterizations. It also evaluates character-level prediction as a vocabulary-independent alternative, while identifying capacity and efficiency trade-offs.

  • Language Modeling Improvements: RNN language models apply the chain rule, encoding previous-word context with an LSTM and predicting the next word with a Softmax.The model represents the joint sequence probability as a product of conditional word probabilities.
  • Relationship between Noise Contrastive Estimation and Importance Sampling: Importance sampling approximates the large Softmax partition function through a multiclass classification loss over one data word and sampled noise words.The paper contrasts this with NCE, which uses binary classification and a logistic loss.
  • CNN Softmax: CNN Softmax generates each word embedding from its characters, producing a compact parameterization that supports arbitrary words and permits precomputation for inference.The CNN Softmax avoids a full |V| × |h| embedding matrix and does not increase inference complexity relative to regular Softmax after precomputation.
  • CNN Softmax: A per-word low-dimensional correction factor restores capacity for distinguishing similarly spelled words, and a 128-dimensional correction halves the gap to regular Softmax.The correction is projected back to the LSTM hidden-state dimensionality through a bottleneck linear layer.
  • Character-Level LSTM: A character-level LSTM replacement scales independently of vocabulary size for training and inference, but performs worse than regular and CNN Softmax models.The word-level LSTM is trained to convergence and frozen before its Softmax is replaced by character-level prediction.

4. Experiments

Experiments use the One Billion Word Benchmark, standard perplexity evaluation, distributed recurrent training, and multiple architectural variations. The study examines efficient large-vocabulary training and reports single-model and ensemble results in benchmark tables.

  • Data and Evaluation: The benchmark contains about 0.8B words and a 793471-word vocabulary, with shuffled duplicate-free sentences and approximately 0.3% OOV tokens.OOV words are represented by a special UNK token.
  • Data and Evaluation: Perplexity is computed from average per-word log-probability on held-out data, including the end-of-sentence symbol.The paper follows the standard aggregation procedure over all words.
  • Model Training: LSTMs with projection layers trained using 20-step truncated BPTT performed well across the evaluated architectural variations.The experiments varied embedding, state, projection, and layer dimensions, with dropout and forget-gate bias initialization applied consistently.
  • Large-Vocabulary Training: 8192 noise samples per step reduce the approximate Softmax computation by about 100-fold compared with evaluating all 793471 vocabulary entries.The samples are shared across the 2560 target words in each unrolled batch.

5. Results and Analysis

Experiments show that larger LSTMs, dropout, vocabulary choices, and character CNN embeddings materially affect large-scale language-model performance. The best single model reaches 30.0 perplexity while using substantially fewer parameters.

  • 30.0 perplexity improves the best single-model result from 51.3 while considerably reducing model size.The authors report this improvement in their overall results summary.
  • 5.1. Size Matters: Larger LSTM layers improve results, with the largest fitted model using a 2-layer LSTM with 8192+1024-dimensional recurrent states.Increasing embedding and projection sizes also helps but substantially increases parameter count.
  • Dropout improves results and mitigates holdout degradation caused by overfitting, although training perplexity can remain up to 6 points below test perplexity.The authors used 10% dropout for smaller LSTMs and 25% for larger ones.
  • A 100,000-word vocabulary reduces the train–test gap, suggesting that excessive capacity is allocated to rare words.Character CNN embeddings reduce this issue because their representations are shared across words.

5.3. Importance Sampling is Data Efficient

The experiments compare importance sampling with noise-contrastive estimation and examine character-based representations. Importance sampling improves both training speed and overall performance, while character embeddings remain compact and competitive.

  • 5.3. Importance Sampling is Data Efficient: Importance sampling significantly improves training speed and overall model performance compared with noise-contrastive estimation.The comparison uses an LSTM-2048-512 evaluated after several epochs.
  • 5.4. Word Embeddings vs Character CNN: Character CNN embeddings produce nearest neighbors for out-of-vocabulary words, and these neighbors are usually reasonable.The examples are drawn from the character CNN embedding space.
  • 5.4. Word Embeddings vs Character CNN: Character-level embeddings are feasible without degrading performance, and the best single model uses a Character CNN embedding.The result supports replacing conventional word embeddings with character-derived representations.
  • 5.4. Word Embeddings vs Character CNN: Character CNN embeddings reduce input-layer parameters by a factor of 11, from 820M to 72M weights, with slightly slower training.Precomputing embeddings removes the inference-time speed penalty.

5.5. Smaller Models with CNN Softmax

The study evaluates CNN Softmax and character-level alternatives for reducing the parameter cost of large language models. CNN Softmax enables smaller models, with a size–perplexity trade-off, while the character-level LSTM performs worse.

  • 820M of 1.04B total parameters reside in the linear layer before the Softmax, leaving the model large even with character-level embeddings.This identifies the Softmax-side output layer as the dominant parameter bottleneck.
  • 39.8 perplexity is achieved with 293M weights by replacing the Softmax with a CNN Softmax sub-network after freezing the word-LSTM.The result requires no fine-tuning after the replacement.
  • 35.8 perplexity is achieved by adding 100M weights through a 128-dimensional bottleneck embedding, demonstrating a size–perplexity trade-off.The added correction word-embedding term narrows the gap between regular and CNN Softmax.
  • 49.0 test perplexity is reached by a single-layer 1024-unit character-level LSTM Softmax, substantially below the best model.Expensive marginalization improves this result to 47.9 perplexity.
  • Figure 2 compares best-LSTM and KN-5 log-probability differences across 25 equal-sized word-frequency buckets.Higher differences favor the LSTM.

5.6. Training Speed

Training large language models required substantial GPU resources and time, although smaller models surpassed prior state of the art quickly. The best results emerged only after extended training.

  • 32 Tesla K40 GPUs were used to train the models, with the best results achieved after 3 weeks.The best model required about 5 days to reach 35 perplexity and 10 days to reach 32.5.
  • Below 45 perplexity is reached in under 10 hours for the smaller 2048-unit, 512-projection LSTM.After 2 hours, this model beats the previous state of the art on the dataset.

5.7. Ensembles

Averaging several strong models reaches 23.7 test perplexity, while adding an N-gram model provides only a modest further gain despite its weak standalone performance.

  • 23.7 test perplexity was achieved by averaging several of the best models, improving by more than 40% over previous work.The result is reported for an ensemble of the authors’ models.
  • 1.2 perplexity points were gained by including the best N-gram model in the ensemble, despite its standalone perplexity of 67.6.
  • The results suggest that N-grams provide limited benefit and that a carefully trained LSTM language model is the most competitive model.

5.8. LSTMs are best on the tail words

The best LSTM outperforms KN-5 across all rare-word buckets, while generated sentences are generally high quality but still contain occasional mistakes.

  • The best LSTM at 30.0 perplexity significantly outperforms KN-5 on rare words and performs better across all buckets.
  • Generated samples are high quality overall, although they still contain occasional mistakes.The samples were otherwise presented as raw rather than hand-picked, after discarding short and politically incorrect sentences.
  • The ensemble generated sentences at about 26 perplexity.
  • The reported ensemble samples include coherent news-like, biographical, and sports-related sentences, alongside factual and grammatical errors.

6. Discussion and Conclusions

The paper shows that large-scale RNN language models can outperform carefully tuned N-grams. Its main result comes from a regularized projected LSTM trained with an approximate Softmax, while sharing code and models is intended to encourage further research.

  • RNN language models trained on large amounts of data outperform competing models, including carefully tuned N-grams.
  • 51.3 to 30.0 perplexity reduction is attributed to several studied components, including a large regularized LSTM with projection layers.The model uses an approximation to the true Softmax with importance sampling.
  • N-gram interpolation is unnecessary in this setup, and its additional gains are reported as marginal.
  • Releasing the findings, code, and models is intended to inspire research on large-scale language modeling.
Loading 1602.02410v2…