Source-linked AI summary
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Xuezhe Ma, Eduard Hovy
TL;DR
Sequence labeling systems traditionally rely on costly hand-crafted features and task-specific resources. This paper introduces an end-to-end CNN-BLSTM-CRF architecture that combines character- and word-level representations, achieving state-of-the-art results on POS tagging and NER. The model also avoids task-specific feature engineering and data pre-processing beyond pre-trained word embeddings.
Problem
Traditional high-performance sequence labeling models rely on costly hand-crafted features and task-specific resources, limiting adaptation to new tasks or domains.
Method
The model encodes character-level information with CNNs, combines it with word embeddings, models context using BLSTM, and jointly decodes labels with a CRF.
Results
The model achieves state-of-the-art performance on POS tagging and NER, with 97.55% POS accuracy and 91.21% NER F1.
Takeaways & Limitations
The end-to-end architecture requires no task-specific resources, feature engineering, or data pre-processing beyond pre-trained word embeddings.
Takeaways & Limitations
The authors identify applying the model to other domains, such as social media, as future work.
Abstract
from arXiv · showhide
State-of-the-art sequence labeling systems traditionally require large amounts of task-specific knowledge in the form of hand-crafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word- and character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data pre-processing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks --- Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both the two data --- 97.55\% accuracy for POS tagging and 91.21\% F1 for NER.
1 Introduction
Sequence labeling models traditionally depend on costly hand-crafted features and task-specific resources. The paper proposes an end-to-end neural architecture that automatically combines word- and character-level information without such engineering.
- Motivation: Traditional high-performance sequence labeling models rely heavily on hand-crafted features and task-specific resources.Examples include spelling features for POS tagging and orthographic features or gazetteers for NER.
- Motivation: Task-specific knowledge is costly to develop, making sequence labeling models difficult to adapt to new tasks or domains.
- Prior approaches: Neural sequence labeling systems using word embeddings have achieved competitive performance, but often augment rather than replace hand-crafted features.Their performance drops rapidly when those features are unavailable, according to the cited discussion.
- Proposed approach: The proposed model encodes characters with CNNs, combines character- and word-level representations, models context with BLSTM, and jointly decodes labels with CRF.It requires no task-specific resources, feature engineering, or data pre-processing beyond pre-trained word embeddings.
2 Neural Network Architecture
The architecture extracts character information with CNNs, combines it with word embeddings, models bidirectional context with BLSTM, and uses CRF for joint label decoding. Dropout is applied within the network, and sequence CRF training and decoding use likelihood optimization and efficient Viterbi search.
- 2.1 CNN for Character-level Representation: CNNs encode character-level information from character embeddings into a representation for each word.The CNN is intended to capture morphological information such as prefixes and suffixes; dropout is applied before the CNN input.
- 2.2 LSTM Unit: An LSTM uses multiplicative gates to control information forgotten and passed to the next time step.The formulation defines sigmoid gating, element-wise products, input vectors, hidden states, weights, and biases, without peephole connections.
- 2.2 BLSTM: BLSTM processes sequences in both directions, concatenating hidden states to capture past and future context.This addresses the limitation that a unidirectional LSTM hidden state receives information only from the past.
- 2.3 CRF: The CRF models correlations between neighboring labels and jointly decodes the best label sequence for an input sentence.For a sequence CRF with successive-label interactions, training and decoding can be solved efficiently with the Viterbi algorithm.
- 2.4 BLSTM-CNNs-CRF: The full network concatenates each word’s CNN character representation with its word embedding, feeds the result to BLSTM, and sends BLSTM outputs to CRF.Dropout is applied to both the input and output vectors of BLSTM, and experiments report that dropout significantly improves performance.
3 Network Training
Training uses pretrained word embeddings, SGD with momentum, gradient clipping, dropout, and validation-based early stopping. Hyperparameters are tuned on development sets, with mostly shared settings across POS tagging and NER.
- Model training requires about 12 hours for POS tagging and 8 hours for NER on a GeForce GTX TITAN X GPU.
- The model uses Stanford GloVe 100-dimensional embeddings trained on 6 billion words from Wikipedia and web text.
- Senna, Google Word2Vec, and randomly initialized embeddings are also evaluated to test embedding choices and pretraining effectiveness.
- Training uses minibatch SGD with batch size 10, momentum 0.9, task-specific initial learning rates, decay rate ρ = 0.05, and gradient clipping of 5.0.
- Dropout is applied for regularization, while early stopping selects parameters based on validation-set performance.
- Hyperparameters are tuned by random search on development sets, with nearly identical settings for POS tagging and NER except for initial learning rate.
4 Experiments
The experiments evaluate the neural sequence-labeling model on English POS tagging and NER using standard benchmark datasets and no dataset preprocessing. The tasks differ in label inventories and corpus settings.
- The evaluation covers POS tagging and named entity recognition, two sequence labeling tasks.
- The corpora statistics are reported in Table 2, with SENT and TOKEN denoting sentence and token counts.
- POS Tagging: POS tagging uses the Wall Street Journal portion of Penn Treebank with 45 POS tags and standard sections 0–18, 19–21, and 22–24 for training, development, and testing.
- NER: NER uses English CoNLL 2003 data containing PERSON, LOCATION, ORGANIZATION, and MISC entity types.
- No preprocessing is performed on either dataset, preserving the system’s end-to-end setup.
4.2 Main Results
Ablation experiments compare recurrent and character-aware architectures, showing that BLSTM improves over BRNN and that adding CRF joint decoding significantly improves BLSTM-CNN performance on both tasks.
- The ablation compares BRNN, BLSTM, BLSTM-CNNs, and the full model using GloVe embeddings and shared hyperparameters.
- BLSTM achieves better performance than BRNN on all evaluation metrics for both POS tagging and NER.
- Table 3 reports performance on development and test sets for both tasks alongside three baseline systems.
- Adding a CRF layer for joint decoding significantly improves BLSTM-CNN models on all metrics for both POS tagging and NER.
4.3 Comparison with Previous Work
The model reaches state-of-the-art results on POS tagging and NER while avoiding the hand-crafted features, external resources, and preprocessing used by several comparison systems. Some reported comparisons are limited by different dataset splits or training settings.
- POS Tagging: 0.23% accuracy improvement over CharWNN demonstrates a gain for the proposed POS model over a character-aware neural comparison.
- POS Tagging: 0.05% accuracy improvement over Søgaard (2011) yields state-of-the-art POS tagging accuracy against traditional statistical models.
- POS Tagging: The POS result of Huang et al. (2015) is not directly comparable because that study used a different training, development, and test split.
- NER: The model improves on neural and feature-engineered NER systems without requiring carefully designed features or data preprocessing.
- NER: The 90.77% F1 result from Chiu and Nichols (2015) is incomparable because their final model was trained on combined training and development data.
- NER: 0.01% F1 improvement over the previous best reported 91.20 F1 score yields state-of-the-art NER performance.
4.4 Word Embeddings
Pretrained word embeddings substantially improve performance over random initialization, with their importance greater for NER than POS tagging. Among the evaluated embeddings, GloVe performs best on both tasks, while Word2Vec is especially weaker on NER.
- Pretrained word embeddings significantly outperform randomly initialized embeddings.
- The reported precision, recall, and F1 figures contain an unresolved numerical inconsistency.
- NER relies more heavily on pretrained embeddings than POS tagging.
- GloVe 100-dimensional embeddings achieve the best results on both tasks, exceeding Senna by about 0.1% POS accuracy and 0.9% NER F1.
- Word2Vec performs similarly to Senna on POS tagging but falls far behind both GloVe and Senna on NER.
4.5 Effect of Dropout
Adding dropout improves performance on both POS tagging and NER under otherwise unchanged hyper-parameters, supporting its role in reducing overfitting.
- Dropout produces an essential improvement on both POS tagging and NER.
- The comparison keeps all other hyper-parameters unchanged.
- The authors interpret the improvement as evidence that dropout reduces overfitting.
4.6 OOV Error Analysis
The error analysis partitions words by their presence in the training and embedding vocabularies and compares the proposed model with an LSTM-CNN baseline. The largest gains occur for words absent from both vocabularies.
- The analysis partitions words into IV, OOTV, OOEV, and OOBV subsets based on training- and embedding-vocabulary membership.
- The proposed model is compared with a baseline LSTM-CNN model across the word subsets.
- The largest improvements appear on OOBV subsets in both corpora.
- The authors attribute the OOBV advantage to adding CRF for joint decoding.
5 Related Work
The paper situates its model among neural sequence-labeling systems combining recurrent, convolutional, and CRF components. Its distinguishing combination uses character CNNs, word-level BLSTM representations, and no hand-crafted features or preprocessing.
- Related neural architectures include BLSTM-CRF, LSTM-CNNs, and other recurrent-convolutional sequence-labeling models.
- Huang et al. combine BLSTM and CRF but omit character CNNs and use hand-crafted features.
- Chiu and Nichols use BLSTM and CNNs to model word- and character-level information.
- CharWNN uses CNNs to model character-level information and was inspired by a feed-forward network.
6 Conclusion
The proposed architecture achieves state-of-the-art performance on two linguistic sequence labeling tasks without task-specific resources, feature engineering, or data pre-processing. Future work includes multi-task learning and applying the model to other domains.
- The model achieves state-of-the-art performance on two linguistic sequence labeling tasks.
- The architecture requires no task-specific resources, feature engineering, or data pre-processing.
- Future work includes multi-task learning and applying the model to domains such as social media.