Source-linked AI summary
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging
Nils Reimers, Iryna Gurevych
TL;DR
The paper examines how random seeds affect evaluation of nondeterministic LSTM sequence-tagging systems. Across five tasks, it advocates comparing score distributions over multiple executions and identifies configurations with superior, more stable performance.
Problem
Single-run performance scores can produce misleading comparisons for nondeterministic sequence-tagging systems because random seeds substantially affect results.
Method
The study evaluates LSTM networks across five sequence-tagging tasks and compares score distributions from multiple executions and network configurations.
Results
Lample et al.'s implementation produced a superior score distribution generalizing better to unseen data than Ma and Hovy's, contrary to conclusions based on published single scores.
Takeaways & Limitations
Comparing score distributions reduces the risk of rejecting promising approaches or falsely accepting weaker ones, while some configurations may require less tuning on new tasks or domains.
Takeaways & Limitations
The study evaluates provided pre-trained embeddings rather than the underlying embedding-generation algorithms, so it cannot conclude that one generation approach is superior.
Abstract
from arXiv · showhide
In this paper we show that reporting a single performance score is insufficient to compare non-deterministic approaches. We demonstrate for common sequence tagging tasks that the seed value for the random number generator can result in statistically significant (p < 10^-4) differences for state-of-the-art systems. For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre. Instead of publishing and reporting single performance scores, we propose to compare score distributions based on multiple executions. Based on the evaluation of 50.000 LSTM-networks for five sequence tagging tasks, we present network architectures that produce both superior performance as well as are more stable with respect to the remaining hyperparameters.
1 Introduction
Neural-network results depend on random seeds, so single scores can misrepresent performance and hinder fair comparison. The paper studies this problem across five sequence-tagging tasks and advocates score distributions from multiple executions.
- Motivation: Random initialization, data shuffling, and dropout make neural-network training non-deterministic, with different seeds reaching different local minima.These minima can have different performance on unseen data.
- Empirical finding: 89.99% to 91.00% F1-score was observed for Ma and Hovy’s NER system across random seeds, with the difference statistically significant at p < 10^-4.The selected seed could make the approach appear state-of-the-art or mediocre.
- Evaluation risks: Single-run comparisons can falsely attribute improvements to superior approaches when they instead reflect favorable random-number sequences.Promising approaches may also be rejected after an unfavorable run.
- Scope and contributions: The study evaluates LSTM architectures on POS-tagging, Chunking, Named Entity Recognition, Entity Recognition, and Event Detection.It examines superior performance, result stability, and the importance of tuning parameters.
- Proposed evaluation: The paper proposes comparing score distributions rather than single performance scores because no seed is justified over another.This comparison can also provide insights into algorithm functioning.
- Scope and contributions: The paper’s contributions include analyzing two recent NER systems and examining LSTM design choices across the five tasks.The NER comparison specifically addresses how single performance scores can mislead.
2 Background
Prior work frames reproducibility as difficult, especially for non-deterministic neural networks whose random seeds affect convergence. Background discussion links local-minimum geometry to generalization on unseen data.
- Reproducibility: Reproducing exact results is difficult because preprocessing, data splits, component versions, feature implementations, and tie handling can affect performance.The difficulty is greater for neural networks because randomness also influences outcomes.
- Non-deterministic optimization: Neural-network training can converge to different local minima because the error function is highly non-convex and the random-number sequence affects optimization.Different minima need not generalize equally well to unseen data.
- Local-minimum geometry: The figure plots error-function values on the Y-axis against weight-space on the X-axis to sketch flat and sharp minima.It is a conceptual illustration rather than an empirical result.
- Local-minimum geometry: A flat minimum keeps the error function approximately constant over a large connected weight-space region, whereas a sharp minimum rises rapidly nearby.The paper describes flat minima as generally having better generalization than sharp minima.
- Optimization choices: Weight initialization and small-batch training are described as methods that can help avoid bad, such as sharp, minima.The paper nevertheless states that non-deterministic behavior must be considered during evaluation.
3 Impact of Randomness in the Evaluation of Neural Networks
The section shows that changing only the random seed can substantially alter neural sequence-tagging results, weakening conclusions drawn from single runs. Score distributions and repeated executions expose differences in performance, variance, and model comparisons.
- 86 Ma and Hovy runs and 41 Lample et al. runs produced statistically different score distributions, with Lample et al. showing higher quartiles and lower F1-score variance.The distribution difference had p < 0.01, while the standard-deviation difference had p < 0.05.
- Selecting the highest development score did not reliably identify the best test run: 94.44% development yielded 90.31% test, whereas 94.28% yielded 91.00%.The development–test Spearman correlation was ρ = 0.229, and the difference was statistically significant at p < 0.002.
- 0.38% was the median seed-related difference and 1.08% the 95th percentile for CoNLL 2003 NER, while ACE 2005 ranged from 82.5% to 74.3% F1.The ACE 2005 result was described as a rare extreme case.
- Comparing single scores can produce false conclusions, including rejecting promising approaches or selecting weaker ones because of favorable or unfavorable random-number sequences.The section motivates comparing score distributions rather than fixing or reporting one arbitrary seed.
4 Experimental Setup
The experiments benchmark BiLSTM sequence-tagging architectures across five NLP tasks while varying network, representation, optimization, regularization, and training parameters. Evaluation uses accuracy for POS tagging and F1-score for the other tasks.
- Five benchmark tasks were used: POS tagging, Chunking, Named Entity Recognition, Entity Recognition, and Event Detection.The setups draw on established benchmarks including CoNLL 2000, CoNLL 2003, ACE 2005, and TempEval3 Task B.
- POS tagging was evaluated with accuracy, whereas Chunking, NER, Entity Recognition, and Event Detection were evaluated with F1-score.
- BiLSTM networks were used for sequence tagging, with an implementation optimized to reduce per-epoch time by a factor of 6 compared with Ma and Hovy (2016).
- The study varied character representations, pre-trained word embeddings, optimizers, classifiers, tagging schemes, dropout strategies, gradient handling, layer counts, recurrent units, and mini-batch sizes.Character representations included CNNs and LSTM-networks; classifiers included Softmax and CRF.
- Dropout comparisons included no dropout, naive dropout, and variational dropout with rates {0.05, 0.1, 0.25, 0.5}.Variational dropout uses the same mask across time steps within a sentence and applies dropout to recurrent units.
5 Robust Model Evaluation
The section evaluates robustness by sampling many network configurations and comparing architectural choices across tasks. CRF-based setups generally outperform Softmax and show lower sensitivity to remaining configuration choices for NER.
- Stable performance across a wide parameter range is desirable because narrow good-parameter regions make adaptation to new tasks, domains, or languages costly.
- 975 configurations were sampled, with each trained using both Softmax and CRF classifiers, producing 1950 trained networks.
- For NER, CRF outperformed Softmax in 210 of 232 configurations (90.5%), with a median F1 difference of −0.66%.The negative ΔF1 indicates that Softmax averaged 0.66 percentage points below CRF.
- The CRF classifier had σ = 0.0060 versus σ = 0.0082 for Softmax, a statistically significant standard-deviation difference at p < 10^-3.Standard deviation was used to detect sensitivity to remaining hyperparameters and the random-number generator.
6 Results
Across five sequence-tagging tasks, the experiments compare LSTM design choices for performance, stability, and sensitivity to configuration. The results identify choices associated with stronger or more stable outcomes, including CRF classifiers, Nadam, suitable embeddings, variational dropout, and gradient normalization.
- Classifier: A CRF classifier achieved better performance on 4 of 5 tasks and reduced NER standard deviation by 27%.The reduction was statistically significant at p < 10^-3.
- Classifier: For Event Detection, the CRF classifier failed because tag dependencies were nearly absent.Events often contain only a single annotated event-trigger token per sentence.
- Optimizer: RMSProp, Adam, and Nadam produced more stable results than SGD, Adagrad, and Adadelta.For NER, their standard deviations were 0.0096, 0.0091, and 0.0092, versus 0.0139 for Adagrad and 0.1328 for SGD.
- Optimizer: Nadam produced the highest performance in 453 of 882 configurations (51.4%) and averaged 0.82 percentage points better NER performance than Adagrad.It also converged after 9 epochs on NER, compared with 42 epochs for SGD.
- Word Embeddings: Word embeddings substantially affected performance, with a 4.97% median POS difference between Komninos and GloVe2 embeddings.Komninos embeddings performed best for POS, Entities, and Events, while Levy and Goldberg embeddings slightly led for Chunking.
- Character Representation: Character representations significantly improved only POS, Chunking, and Events; differences were not significant for NER or Entity Recognition.The CNN and LSTM character-representation approaches were statistically insignificant from each other across all tasks.
- Gradient Clipping and Normalization: Gradient normalization improved performance, with median gains ranging from 0.29 percentage points F1-score for Chunking to 0.82 percentage points for POS.Not normalizing was best in only 5.6% of configurations, while a threshold of 1 was best in 30.5%.
- Dropout: Variational dropout was best in 83.5% of configurations and increased median performance over no dropout by 0.31 to 1.98 percentage points.Applying dropout to both output and recurrent dimensions was best in 62.6% of cases.
7 Conclusion
The paper shows that random seeds can materially alter neural sequence-tagging results, making single-run comparisons unreliable. By comparing score distributions across executions and testing LSTM configurations, it identifies more robust performance patterns and configuration choices.
- Random-number sequences significantly affected test performance, so single-run comparisons could produce wrong conclusions.
- For two NER systems, re-execution showed that Lample et al.’s implementation had a superior score distribution generalizing better to unseen data.This contrasted with the improvement inferred from the systems’ published single scores.
- Comparing score distributions reduced the risk of rejecting promising approaches or falsely accepting weaker approaches.The paper demonstrated this across ten LSTM design choices and hyperparameters for five tasks.
- Standard deviations indicated greater dependence for SGD, Adagrad, and Adadelta than for RMSProp, Adam, or Nadam.Variational dropout also reduced dependence on hyperparameters and random seed values.
- Testing many configurations identified choices that consistently produced superior performance and depended less on the remaining network configuration.The authors report that such choices may require less tuning on new tasks or domains.