Source-linked AI summary
On the State of the Art of Evaluation in Neural Language Models
Gábor Melis, Chris Dyer, Phil Blunsom
TL;DR
Neural language-model comparisons can be distorted by inadequately controlled hyperparameter variation. The paper uses black-box hyperparameter optimisation to compare recurrent architectures and finds that properly controlled LSTMs outperform newer models, while careful control has substantial computational costs.
Problem
Hyperparameter dependence is an inadequately controlled source of variation in empirical neural language-model comparisons, risking empirically unsound claims.
Method
The paper uses black-box hyperparameter optimisation to compare LSTM, Recurrent Highway Network, and NAS model families under controlled architecture, budget, learning, and regularisation settings.
Results
Properly controlled hyperparameters make LSTMs outperform the more recent models, contrary to published claims.
Takeaways & Limitations
Reliable comparisons of competing deep architectures require more methodological care and attention to hyperparameter sensitivity.
Takeaways & Limitations
The model comparisons retain uncertainty from the Gaussian Process, and the authors estimate that a perplexity gap of 1.0 is statistically robust on these datasets.
Abstract
from arXiv · showhide
Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing code bases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.
1 INTRODUCTION
The paper argues that hyperparameter variation can undermine empirical comparisons of neural language models. Using black-box optimisation to control these effects, it finds that properly regularised LSTMs outperform newer architectures, while careful controls remain computationally costly.
- Motivation: Hyperparameter values affecting learning, regularisation, and capacity are an inadequately controlled source of experimental variation.The paper links this variation to a risk of empirically unsound claims.
- Method: Black-box hyperparameter optimisation compares LSTM, Recurrent Highway Network, and NAS model families under adjustable architectures, parameter budgets, and regularisation settings.The search controls embedding and recurrent cell sizes alongside learning and regularisation hyperparameters.
- Result: Properly controlled hyperparameters lead LSTMs to outperform the more recent models, contrary to published claims.The result is presented as evidence that replication failures can arise from poorly controlled hyperparameter variation.
- Implications: Careful experimental controls are possible but require considerable computational cost.The paper therefore highlights a trade-off between rigorous evaluation and practical resource demands.
- Implications: Models with many hyperparameters or narrow successful tuning ranges should have that sensitivity identified in standard publication practice.The authors also call for methodology that balances careful experimentation against the risks of false claims.
2 MODELS
The study compares LSTM, Recurrent Highway Network, and NAS recurrent architectures using shared model components and dropout variants, while preserving their key recurrent-state differences.
- Architectures: The comparison focuses on LSTM, Recurrent Highway Network, and NAS architectures.LSTM is the established baseline; RHN and NAS were selected for reported state-of-the-art or impressive performance.
- Model components: LSTM and NAS models use embeddings, stacked recurrent cells, additive skip connections, and an optional down-projection.The down-projection reduces output embedding parameters, while shared input and output embeddings are the default.
- Regularisation: Feedforward dropout is applied to inputs, inter-layer connections, and outputs, with masks independently sampled per time step.Dropout on recurrent states instead uses the same mask across all time steps in a sequence.
- Architecture differences: RHNs pass recurrent state from the topmost layer to the lowest layer at the next time step, unlike LSTMs, whose layers have separate recurrent states.Intra-layer dropout therefore does not apply to RHNs because only the recurrent state passes between layers.
- Regularisation: All three model types use variational or recurrent dropout for recurrent states unless explicitly noted otherwise.The paper excludes techniques viewed as orthogonal to comparing the relative merits of recurrent cells.
3 EXPERIMENTAL SETUP
Models are evaluated on three language-modelling datasets spanning word-level and character-level prediction: Penn Treebank, Wikitext-2, and Enwik8.
- Datasets: The experiments use Penn Treebank, Wikitext-2, and Enwik8 from the Hutter Prize dataset.Penn Treebank is the smallest; Wikitext-2 is about twice its size with a larger vocabulary and lighter preprocessing.
- Datasets: For Enwik8, the first 90 million characters are used for training and the remaining 10 million are evenly split between validation and test.This establishes the character-level dataset partition used in the experiments.
4 TRAINING DETAILS
Training uses truncated backpropagation and Adam with dataset-specific batch, checkpoint, and optimisation settings for word-level and character-level models.
- Word-level training: Word-level models use batch size 64, 35-step truncated backpropagation, and carry final states between successive batches.Models start from a zero state at training and test time, with a 0.01 training probability of receiving a constant zero initial state.
- Optimisation: Adam uses β1 = 0, β2 = 0.999, and ϵ = 10−9 for optimisation.Setting β1 to zero removes the exponential moving average for gradient means and makes Adam close to RMSProp without momentum.
- Scheduling: For Penn Treebank and Wikitext-2, validation checkpoints occur after every 100 and 200 optimisation steps, respectively.The learning rate is multiplied by 0.1 after 30 consecutive checkpoints without validation improvement.
- Character-level training: Enwik8 uses 50-step truncated backpropagation, batch size 128, Adam settings β2 = 0.99 and ϵ = 10−5, checkpoints every 400 optimisation steps, and unshared embeddings.These settings distinguish character-level training from the word-level setup.
5 EVALUATION
The evaluation compares recurrent language-model architectures under black-box hyperparameter tuning, parameter-budget controls, and specified evaluation procedures.
- 5.1 HYPERPARAMETER TUNING: Evaluation loads the tuner-selected checkpoint with the best validation perplexity and tests it using batch size 1.Using the training batch size worsens word-level results by 0.3 PPL, while Enwik8 is practically unaffected.
- 5.1 HYPERPARAMETER TUNING: Google Vizier optimises a restricted set of learning and regularisation hyperparameters, requiring thousands of evaluations for convergence.The tuned variables include learning rate, embedding ratio, input, state, output and intra-layer dropout, and weight decay.
- 5.1 HYPERPARAMETER TUNING: Table 1 reports validation and test perplexities across model depths and parameter budgets, with shared input and output embeddings except for Zaremba’s results.VD denotes Variational Dropout, and the dagger marks parallel work.
- 5.1 HYPERPARAMETER TUNING: Models are compared by total trainable parameters, with down-projection controlling the tradeoff between embedding and recurrent-cell capacity.Hidden and embedding sizes are determined by the parameter budget, depth, and input embedding ratio.
- 5.1 HYPERPARAMETER TUNING: For Enwik8, embeddings are not shared and down-projection is omitted because the vocabulary contains only 205 symbols.The small vocabulary means relatively few parameters are allocated to embeddings.
6 RESULTS
Results compare LSTMs, RHNs, and NAS across Penn Treebank, Wikitext-2, and Enwik8, showing strong shallow-LSTM performance but a result slightly below state of the art on Enwik8.
- 6.1 PENN TREEBANK: At 24M parameters, Penn Treebank models reach 58.3 test perplexity at depth 4, while a shallow 10M LSTM retains a comfortable margin over RHN.The 10M RHN also improves on the 24M RHN reported in the original publication, and NAS performs nearly as well as the tuned models.
- 6.2 WIKITEXT-2: 65.9 is the reported best Wikitext-2 result, comparing favourably with the Neural Cache despite using a base model without that innovation.The paper also reports results for both Penn Treebank-tuned and Wikitext-2-tuned hyperparameters.
- 6.2 WIKITEXT-2: Table 2 reports validation and test perplexities on Wikitext-2 using shared input and output embeddings.The table marks parallel work with a dagger.
- 6.2 WIKITEXT-2: Shallow LSTMs perform especially well on Wikitext-2, while deeper models degrade gradually and RHNs lag significantly.NAS falls short of LSTM performance, possibly because its architecture overfit Penn Treebank, though deeper variants would be needed to assess that explanation.
- 6.3 ENWIK8: Enwik8 results are slightly below state of the art, with optimisation limited to 14 epochs—about one tenth of the training used by the compared RHN.The models match the smaller RHN, while NAS lags the other models by a surprising margin.
7 ANALYSIS
The analysis examines model features, tuning robustness, and sources of evaluation variance across recurrent architectures and datasets. It finds that careful hyperparameter optimization and regularization materially affect comparisons, while residual uncertainty remains bounded but computationally costly to estimate.
- Analysis: On two of three datasets, careful model specification and hyperparameter optimization substantially improved previous results, with smaller gains for RHNs than LSTMs.The authors note that their setup might favor LSTMs, although they consider differential prior tuning a more likely explanation.
- Analysis: The three recurrent cell architectures were closely matched across all three datasets, with only minuscule differences on Enwik8.The results support the view that apparent architectural differences may arise from trainability and regularization, although the comparison cannot prove that claim.
- The effect of individual features: 2–5 perplexity points: down-projection improved Penn Treebank results for selected depths and parameter budgets; on Wikitext-2, gaps reached about 10–18 points.The larger Wikitext-2 gains are attributed to its larger vocabulary size.
- The effect of individual features: About 6 perplexity points: untying input and output embeddings worsened results across the evaluated models.This finding was consistent with earlier reported results.
- The effect of individual features: Without variational dropout, RHNs suffer substantially and deep LSTMs also lose perplexity; recurrent dropout showed no consistent advantage over variational dropout.The deep LSTM loss is attributed to insufficient regularization from intra-layer dropout alone.
8 CONCLUSION
As deep architectures become harder to distinguish, reliable evaluation requires more methodological care. The paper demonstrates that careful comparisons are possible, but computational cost remains a central constraint.
- Smaller effect sizes among competing deep architectures make poorly controlled hyperparameter noise more consequential.
- The paper demonstrates that models can be meaningfully compared by estimating noise from multiple sources with extensive computation.
- Reliable baselines are the paper’s main practical contribution, while a broadly applicable low-cost methodological solution remains open.
- Evaluation reliability and computational expenditure form an apparent tradeoff that future methodology must address.