Source-linked AI summary
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, Tony Robinson
TL;DR
The paper addresses the need for a large, accessible benchmark to measure progress in statistical language modeling and compare techniques at scale. It introduces a roughly one-billion-word corpus with baseline evaluations, reporting a 35% perplexity reduction from combined techniques and identifying scaling as a practical constraint.
Problem
Language-model performance depends heavily on training-data scale, data match, and modeling technique, but scaling new methods to large datasets creates a substantial evaluation barrier.
Method
The paper constructs a web-based benchmark from processed English corpora, removes duplicates, builds a vocabulary, randomizes sentences, partitions the data, and provides baseline model comparisons.
Results
35% perplexity reduction was achieved by combined techniques relative to the baseline, with the best results from a recurrent neural network language model.
Takeaways & Limitations
The benchmark supports fair, reproducible comparison of language-modeling techniques and lowers the barrier to evaluating competitive approaches.
Takeaways & Limitations
The approximately one-billion-word scale may restrict immediate competitiveness for new techniques, although smaller subsets can support computationally expensive comparisons.
Abstract
from arXiv · showhide
We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned Kneser-Ney 5-gram model achieves perplexity 67.6; a combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models.
1 Introduction
The paper introduces a one-billion-word benchmark intended to make statistical language-modeling progress easier to evaluate and compare. It also provides strong neural-network baselines for comparison with competitive techniques.
- Language models support applications including speech recognition, machine translation, spelling correction, soft keyboards, and natural language processing.
- Training-data amount, data–test-data match, and modeling technique are major determinants of language-model performance.
- One billion training words balance relevance to abundant-data settings with the ease of evaluating new modeling approaches.
- The benchmark supplies strong neural-network language-model baselines so researchers can compare competitive techniques with current state-of-the-art results.
2 Description of the Benchmark Data
The benchmark is built from normalized, tokenized, deduplicated web text and organized into randomized partitions for training, held-out evaluation, and testing. Rebuild scripts and baseline log probabilities support reproducible comparisons, while randomized sentence order limits long cross-sentence-context experiments.
- Duplicate removal reduced approximately 2.9 billion words to 829250940 words, and the vocabulary contains 793471 words including sentence markers.
- The data was randomized and split into 100 disjoint partitions, with 1% selected as held-out data and 2% of that held-out data used for testing.
- The test set contains 159658 words excluding the never-predicted sentence-beginning marker, and its out-of-vocabulary rate is 0.28%.
- The benchmark provides scripts to rebuild the data and log-probability values for each word across ten held-out sets for baseline n-gram models.
- Randomized sentence order makes the benchmark unsuitable for models that capture long-context dependencies across sentence boundaries.
3 Baseline Language Models
The benchmark uses Katz and interpolated Kneser–Ney 5-gram language models as prevalent baselines and examines the effect of entropy pruning. Kneser–Ney smoothing degrades more rapidly under pruning than Katz smoothing.
- Katz and interpolated Kneser–Ney 5-gram language models were selected as prevalent baseline models.
- Entropy pruning was applied to both baseline models to illustrate its negative effect in practical, commonly pruned systems.
- Kneser–Ney smoothing degrades much more rapidly than Katz under entropy pruning, requiring a discerning choice for a given application.
4 Advanced Language Modeling Techniques
The paper compares several advanced language-modeling approaches, including distributed backoff, maximum-entropy, hierarchical-softmax, and recurrent-neural-network models. These approaches emphasize scalable training or efficient probability estimation on the benchmark.
- Models evaluated: The comparison covers Stupid Backoff, Binary MaxEnt, hierarchical softmax, and recurrent neural network language models.
- Stupid Backoff: Stupid Backoff omits discounting and uses one backoff weight, so its scores are normalized over the vocabulary only for reported perplexity.
- Maximum Entropy Models: Binary MaxEnt uses independent binary predictors with heavily down-sampled negative examples to avoid expensive probability normalization during training.
- Maximum Entropy Models: Hierarchical softmax estimates probabilities through a binary Huffman tree, reducing evaluation work for frequent words.
- Recurrent Neural Network Model: RNN training was accelerated 20–50-fold, enabling typical training times of 1–10 days on billions of words and vocabularies exceeding one million words.
- Model Combination: The model-combination table reports interpolation weights tuned to minimize held-out-data perplexity.
- Recurrent Neural Network Model: RNN experiments varied neuron counts and used ten data iterations, while optimization targeted test performance rather than model size or training speed.
5 Results
The benchmark compares language models using perplexity-focused hyper-parameter selection while also reporting training time. Linear interpolation achieved the best perplexity, reducing it by about 35% versus the unpruned modified Kneser-Ney 5-gram baseline, equivalent to about a 10% cross-entropy reduction.
- Training times were reported alongside perplexity, but are not necessarily comparable because implementations differ.
- 5.2 Model Combination: Linear interpolation of probabilities from all models produced the best perplexity results.
- 35% lower perplexity was achieved than the unpruned modified Kneser-Ney 5-gram baseline, corresponding to about 10% lower cross-entropy.
- 5.2 Model Combination: Only some models received significant combination weights, which were tuned on held-out data.
- 5.2 Model Combination: The SBO model received relatively high combination weight despite poor perplexity, while the KN baseline received relatively little weight.
6 Conclusion
The paper introduces a reproducible, web-based benchmark for measuring progress in statistical language modeling and compares modeling approaches trained on a large dataset. Its results include a 35% perplexity reduction, while its scope remains constrained by the computational demands of approximately one billion words and incomplete technique coverage.
- 6 Conclusion: The benchmark uses freely available web resources to support fair comparison of statistical language modeling techniques.
- 6 Conclusion: Approximately one billion words can restrict immediate competitiveness for computationally expensive techniques, although smaller dataset partitions remain available for comparison.
- 6 Conclusion: The paper compares novel modeling approaches trained on a large dataset and reports the largest recurrent neural network language model known to the authors.
- 6 Conclusion: 35% perplexity reduction was reported as large enough to support hope for significant improvements in various applications.
- 6 Conclusion: The benchmark does not yet include some well-known techniques, and future work proposes adding application-specific ASR or SMT evaluation resources.