PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks

Nils Reimers, Iryna Gurevych

arXiv:1707.06799v2cs.CL

TL;DR

Hyperparameter optimization for LSTM sequence taggers lacks clear evidence about which choices matter most. The paper evaluates more than 50,000 configurations across five tasks and finds that embeddings, classifier choice, optimizer, and dropout have high impact, while some architectural dimensions matter less. It also identifies task similarity and hyperparameter sensitivity as important conditions for multi-task learning.

  • Problem

    Published evidence is limited on which LSTM hyperparameters and architectural extensions most affect performance, making optimization difficult and resource-intensive.

  • Method

    The paper evaluates more than 50,000 BiLSTM configurations, architectural variants, and hyperparameter choices across five linguistic sequence-tagging tasks.

  • Results

    Pretrained embeddings, the final classifier, optimizer, dropout, and gradient normalization have high impact, while recurrent-unit count has relatively small impact; multi-task gains occur mainly for similar tasks.

  • Takeaways & Limitations

    Komninos and Manandhar embeddings, Nadam, gradient normalization, variational dropout on output and recurrent units, CRF classification, and usually two recurrent layers are supported choices.

  • Takeaways & Limitations

    The character-representation comparison uses the original papers’ hyperparameters, and the study leaves multi-task learning versus pipeline approaches for future work.

Abstract

from arXiv · show

Selecting optimal parameters for a neural network architecture can often make the difference between mediocre and state-of-the-art performance. However, little is published which parameters and design choices should be evaluated or selected making the correct hyperparameter optimization often a "black art that requires expert experiences" (Snoek et al., 2012). In this paper, we evaluate the importance of different network design choices and hyperparameters for five common linguistic sequence tagging tasks (POS, Chunking, NER, Entity Recognition, and Event Detection). We evaluated over 50.000 different setups and found, that some parameters, like the pre-trained word embeddings or the last layer of the network, have a large impact on the performance, while other parameters, for example the number of LSTM layers or the number of recurrent units, are of minor importance. We give a recommendation on a configuration that performs well among different tasks.

1 Introduction

The paper addresses limited evidence about which LSTM hyperparameters and architectural extensions matter most for linguistic sequence tagging. It evaluates more than 50,000 configurations across five tasks and identifies consistently useful choices alongside lower-impact parameters.

  • Hyperparameter selection can separate mediocre from state-of-the-art LSTM sequence-tagging performance.Relevant choices include recurrent units, network depth, dropout, and pretrained word embeddings.
  • Published evidence remains limited on which hyperparameters and architectural extensions strongly affect performance.The paper frames this uncertainty as a risk for inefficient optimization and missed improvements.
  • More than 50,000 BiLSTM configurations were evaluated across POS, Chunking, NER, Entity Recognition, and Event Detection.The study also compares BiLSTM-CRF, BiLSTM-CNN-CRF, and BiLSTM-LSTM-CRF architectures.
  • Pretrained embeddings, the classifier, optimizer, gradient normalization, tagging scheme, dropout, and recurrent depth showed practically important choices.The reported results favor Komninos and Manandhar embeddings, Nadam, gradient normalization, BIO tagging, variational dropout, and usually two recurrent layers.
  • Multi-task learning improved performance only under certain circumstances, especially when jointly learned tasks were linguistically similar.The experiments also found benefits from combining shared and task-dependent LSTM layers.

2 Related Work

Related work contrasts exhaustive, randomized, Bayesian, and manual hyperparameter search. These approaches differ in computational cost, adaptability to prior results, and sensitivity to poorly chosen parameter ranges.

  • Grid search exhaustively tests a manually defined subset of hyperparameter combinations, creating rapidly increasing computational demands.Testing 11 hyperparameters with small option sets would require more than 6 million combinations per task.
  • Randomized search samples parameter settings and can be more efficient than grid search while remaining easy to stop, resume, or modify.Its effectiveness depends on choosing suitable parameter options and ranges.
  • Randomized search does not adapt to previous outcomes and can waste trials when broad ranges include poorly chosen values.The paper gives a learning-rate example where over 90% of trials fail when good values occupy only 0.01–0.1 within a 0–1 range.
  • Bayesian optimization models the relationship between hyperparameters and objective values to select subsequent configurations using training history.The approach assumes a smooth but noisy mapping between hyperparameters and the objective function.
  • Manual tuning can remain effective, while coarse grid or randomized searches can help identify high-performance regions.The inventor iteratively narrows the search toward promising architectures and hyperparameter settings.
  • Prior work recognizes that architectural additions and hyperparameters can substantially change performance but reports little about their relative importance.The paper situates its analysis alongside methods for estimating hyperparameter importance.

3 LSTM-Networks for Sequence Tagging

The sequence-tagging model concatenates word, capitalization, and character representations, processes them with a BiLSTM, and predicts tags with either softmax or CRF decoding. Character representations are derived using CNN or BiLSTM encoders.

  • Each word is represented by a pretrained embedding, a capitalization feature, and an optional fixed-size character representation.The capitalization feature encodes categories such as numeric, lowercase, uppercase, and initial uppercase.
  • Character encodings use either a CNN with 30 filters of length 3 and max-over-time pooling or a BiLSTM with 25 recurrent units in each direction.The CNN produces a 30-dimensional representation, while the BiLSTM produces a 50-dimensional representation.
  • The concatenated representations enter a bidirectional LSTM whose two directional outputs feed the final classifier.One LSTM processes the sentence forward and the other in reverse.
  • The softmax classifier predicts each token independently, whereas the CRF classifier maximizes tag probability for the complete sentence.The CRF is therefore suited to tasks with strong dependencies between token tags.
  • Figure 1 depicts the BiLSTM-with-CRF architecture and its alternative CNN- or BiLSTM-based character representation.
  • Figure 2 contrasts the CNN and BiLSTM character-representation pathways.

4 Benchmark Tasks

The study evaluates five sequence-tagging benchmarks: POS, Chunking, NER, Entity Recognition, and Event Detection. The tasks differ in annotation targets, datasets, splits, and evaluation settings.

  • The benchmark includes POS, Chunking, NER, Entity Recognition, and Event Detection, with Table 1 summarizing their datasets.For POS, only the first 500 training sentences are used; the other listed tasks include BIO tags in their tag counts.
  • Part-of-Speech tagging: POS tagging labels each token with its syntactic role and is evaluated on a reduced-training-set setup to increase task difficulty.Using the first 500 WSJ training sentences yields approximately 94–95% accuracy, below the estimated 97% upper bound.
  • Part-of-Speech tagging: POS performance above 97% accuracy may be difficult to interpret meaningfully because it approaches estimated annotation and task error rates.
  • Chunking: Chunking assigns tags to syntactic constituents such as noun or verb phrases and requires exact span matching for correct evaluation.The benchmark uses CoNLL 2000 and reports F1, with state-of-the-art performance around 95% F1.
  • NER: NER identifies person, location, organization, and miscellaneous entities in the CoNLL 2003 setup using F1 over all named entities.The reported state-of-the-art performance is 91.21% F1.
  • Entities: The ACE 2005 Entities task labels all words referring to entities across seven categories rather than only named entities.
  • Events: Event Detection identifies the smallest text extent expressing situations that happen or occur, usually a single word.In the example, “sent” and “spent” express events.
  • Evaluation: Chunking, NER, and Entity Recognition performance is measured with F1, while the event benchmark reports a TempEval3 reference score of 81.05% F1.

5 Evaluation Methology

The evaluation targets robust design choices across five sequence-tagging tasks by comparing many randomly sampled configurations, classifiers, and statistical outcomes. It also measures variability from random seeds and configuration choices.

  • Evaluation goal: The study evaluates BiLSTM design choices across five NLP tasks, seeking options that perform robustly across hyperparameters and random seeds.The tasks are part-of-speech tagging, chunking, NER, entity recognition, and event recognition.
  • Evaluation procedure: Each sampled network configuration is evaluated with both Softmax and CRF classifiers to compare alternative final layers under matched settings.This paired evaluation supports comparison across randomly sampled configurations rather than one selected setup.
  • Evaluation procedure: For chunking, several hundred configurations are summarized with violin plots showing the probability density, median, and quartiles of test performance.The plot compares Softmax and CRF classifier outcomes as the final BiLSTM layer.
  • Classifier comparison: For Chunking, CRF outperformed Softmax on 219 of 230 configurations, or 95.2%, in the reported comparison.The configurations were sampled randomly and each was evaluated with both classifiers.
  • Classifier comparison: For Chunking, the median Softmax-minus-CRF F1 difference was ΔF1 = −0.38%, meaning Softmax scored 0.38 percentage points lower on average.The difference is computed across 230 paired configurations.
  • Statistical analysis: The analysis uses two-sided binomial tests for option comparisons and Brown-Forsythe tests to compare standard deviations, with p < 0.01 thresholds.Standard deviation measures dependence on remaining configuration choices and random seed values.

6 Experimental Setup

The experimental setup implements BiLSTM sequence taggers and varies embeddings, character representations, optimization, regularization, architecture, batching, tagging, and classifier choices. Training and preprocessing procedures are specified alongside these evaluated hyperparameters.

  • Implementation: The experiments implement BiLSTM networks in Keras version 1.2.2 across the evaluated sequence-tagging configurations.Source code and a spreadsheet containing results from 50,000 runs are reported as available on GitHub.
  • Input representations: The setup evaluates multiple publicly available pre-trained word embeddings, including GoogleNews, Levy, GloVe, and Komninos–Manandhar embeddings.The embeddings differ in training corpora and generation procedures.
  • Input representations: Character-level representations are learned during training to encode word-internal information such as prefixes and suffixes.The evaluated approaches use CNNs or LSTMs for character encoding.
  • Optimization and gradients: The evaluated optimization choices include SGD-sensitive learning-rate settings and strategies for handling recurrent-network gradients.Gradient clipping limits individual components, whereas normalization rescales gradients based on their norm.
  • Output and labeling: The classifier comparison contrasts token-independent Softmax predictions with a linear-chain CRF that models complete tag-sequence probabilities.The CRF is especially suited to tagging schemes with dependencies between valid consecutive tags.
  • Regularization: The experiments vary dropout, using no dropout, naive dropout, or variational dropout as alternatives for addressing overfitting.Naive dropout uses a randomly selected mask for each LSTM output and changes across time steps.
  • Network architecture: The architecture varies from one to three stacked BiLSTM layers and selects recurrent-unit counts from {25, 50, 75, 100, 125}.For multiple layers, increasing layer sizes are forbidden; additional experiments use 60 ≤ u ≤ 300 units.
  • Training configuration: Mini-batch sizes of 1, 8, 16, 32, and 64 sentences are evaluated, while Theano and TensorFlow serve as alternative backends.The two backends can differ because of implementation and numerical-instability effects, although the paper reports approximately equivalent test performance.

7 Evaluation Results

Across five sequence-tagging tasks, the evaluation compares word embeddings, character representations, and other architecture choices using randomly sampled network configurations. Pre-trained word embeddings had a large impact, whereas character representations often added little and CNN- and LSTM-based character encoders performed on-par.

  • Word Embeddings: Komninos and Manandhar embeddings performed best on most tasks, with a 4.97 percentage-point median POS accuracy advantage over GloVe embeddings.
  • Word Embeddings: The Komninos and Manandhar embeddings were not significantly different from alternatives on Chunking and NER.Levy and Goldberg embeddings were often best for Chunking, while GloVe 3 was often best for NER, but both differences were statistically insignificant.
  • Character Representation: Character-based representations significantly affected only POS, Chunking, and Events, not NER or Entities.
  • Character Representation: CNN- and LSTM-based character representations performed on-par, while character representations often failed to improve network performance.The CNN approach was preferred for higher computational efficiency.

7.3 Optimizers

The optimizer experiments compare six optimization methods and learning-rate adaptations across the benchmark tasks. Nadam, Adam, and RMSProp were more stable than the alternatives, with Nadam generally producing the best performance and fastest convergence.

  • Optimizer Comparison: Adam, Nadam, and RMSProp showed lower performance variance than SGD, Adagrad, and Adadelta.The authors conclude these optimizers are less dependent on the remaining network configuration or random seed.
  • Optimizer Comparison: Nadam achieved the highest score in 48.7% of tested configurations, although its advantage over Adam and RMSProp was often not statistically significant.
  • Convergence: Nadam converged fastest, whereas Adadelta, Adagrad, and SGD required the most epochs to reach a minimum.
  • Learning-Rate Adaptation: Increasing Adam’s learning rate to 0.01 and then 0.005 before returning to 0.001 improved performance and convergence for POS and Chunking.
  • Gradient Handling: Gradient clipping produced no statistically significant improvement, while gradient normalization improved performance by 0.45 to 0.82 percentage points on average.A threshold τ = 1 usually performed best, but the threshold had minor importance.

7.5 Tagging Schemes

The tagging-scheme and classifier experiments show that structured label dependencies matter for choosing the final classifier. BIO and IOBES generally outperformed IOB, while CRF usually outperformed Softmax except when label dependencies were negligible.

  • Tagging Scheme: IOB performed poorly across all tasks, whereas BIO and IOBES generally performed on-par.For Entities, BIO performed much better than IOBES; the authors recommend BIO because it creates less overhead.
  • Classifier: CRF improved performance over Softmax by 0.19 to 0.85 percentage points on the evaluated tasks, except for event detection.
  • Classifier: Softmax produced invalid BIO tags for around 1.5%–2.0% of named entities with one BiLSTM layer, compared with 0.3%–0.8% using three layers.
  • Classifier: CRF provided its largest improvement for shallow BiLSTM networks, although a significant difference remained at depth three.With CRF, the difference among one, two, and three BiLSTM layers was smaller than with Softmax.
  • Classifier: Softmax slightly outperformed CRF on TempEval 3 because the training labels contained essentially no tag dependencies.The I- tag appeared only three times in the corpus and never in training.

7.7 Dropout

The experiments find that variational dropout is the most reliable dropout strategy, while network depth is more consequential than recurrent-unit count only within a limited range. Two BiLSTM layers and roughly 100 recurrent units per LSTM network are robust choices for the tested tasks.

  • Dropout Placement: Applying variational dropout to both output and recurrent units achieved the best results across the benchmark tasks.
  • Dropout Scheme: Variational dropout outperformed no dropout and naive dropout on all benchmark tasks.It also yielded the lowest standard deviation, indicating greater robustness to configuration and random-seed variation.
  • Network Depth: Two BiLSTM layers were a robust rule of thumb, producing the best results except for the reduced POS tagging task.
  • Recurrent Units: The optimal recurrent-unit count was around 100 per LSTM network, but changing it by 25 units altered performance by only roughly 0.01% to 0.06%.
  • Recurrent Units: The recurrent-unit count had only a minor effect when it was not far too small or far too large.The study models performance with a degree-2 polynomial relating units to test performance, reflecting underfitting at low values and overfitting at high values.

7.10 Mini-Batch Size

Mini-batch size effects depend on training-set size, while backend choice produces no clear runtime winner in this setup.

  • For small training sets, mini-batch sizes from 1 to 16 are good choices, whereas larger training sets favor sizes from 8 to 32.The conclusion identifies 8 as robust for small training sets and 32 as robust for larger ones.
  • 2.83 percentage points: using mini-batch size 1 instead of 32 reduced median performance on ACE 2005 Entities.
  • Theano and TensorFlow showed an insignificant performance difference, but their runtime trade-offs prevented a clear winner.TensorFlow started training faster, while Theano benefited from caching when architectures were re-executed; TensorFlow epochs took longer on CPU.

8 Multi-Task Learning

Multi-task learning helps mainly for linguistically similar tasks, but it is more variable and hyperparameter-sensitive than single-task learning. Task-specific recurrent layers alongside shared layers and supervision at different levels can improve results, while pipeline comparison remains untested.

  • 8 Multi-Task Learning: Sharing embedding and BiLSTM layers while using task-specific output layers is the paper’s basic multi-task architecture.Training minimizes the loss averaged across tasks, with one main task and an optional auxiliary task.
  • 8 Multi-Task Learning: Multi-task learning benefits some linguistically similar task combinations, while single-task learning is better in other cases.Chunking clearly outperformed with POS as an auxiliary task, whereas NER performed best in the single-task setup and Entities did not improve it.
  • 8 Multi-Task Learning: POS with Chunking as an auxiliary task had a statistically significant improvement, but its median performance difference was -0.08%.The multi-task setup also showed greater variance, including both poorly performing and clearly outperforming configurations.
  • 8 Multi-Task Learning: Multi-task learning has higher performance variance and requires more careful hyperparameter testing than single-task learning.The paper reports that finding a stable local minimum with low generalization error is more challenging in the multi-task setup.
  • 8.3 Supervising tasks at different levels: Supervising tasks at different network levels usually improves performance, and task-specific LSTM layers appear beneficial alongside shared layers.The reported pattern extends beyond supervising only low-level tasks at lower layers.
  • 8.3 Supervising tasks at different levels: The study leaves multi-task learning versus a pipeline approach for future work.The paper notes that pipeline features from auxiliary-task outputs could outperform multi-task learning when only the main task improves.

9 Conclusion

The conclusion identifies several high-impact BiLSTM choices and emphasizes that multi-task learning is more variable and difficult to optimize. It also cautions that one-dimensional hyperparameter evaluations may miss interactions between settings.

  • Random initialization changed ACE 2005 Entities test performance by an average of 0.72 percentage points across otherwise identical runs.The authors evaluated 50,000 configurations to reduce the influence of random noise when comparing parameters.
  • Word embeddings, optimizer, final classifier, and dropout mechanism had high impact, whereas character representation, tagging scheme, LSTM depth, and recurrent-unit count had minor impact.Komninos and Manandhar embeddings, Nadam, gradient normalization threshold 1, variational dropout, and CRF were the reported preferred choices.
  • Multi-task learning improved performance consistently for some similar task combinations but generally showed higher variance and greater sensitivity to hyperparameter selection.
  • Task-specific LSTM layers alongside shared layers appear beneficial, including when tasks are supervised at different network levels.
  • Evaluating one hyperparameter dimension at a time can miss interactions, as the best recurrent-unit count changed with the number of recurrent layers.Thus, individually best settings need not form the global optimum.
Loading 1707.06799v2…