Source-linked AI summary
Neural Architectures for Named Entity Recognition
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer
TL;DR
Named entity recognition is difficult to adapt because limited supervised data makes generalization challenging and language-specific features and resources costly to develop. This paper introduces two neural architectures using character-based and unsupervised word representations, achieving state-of-the-art or near-state-of-the-art NER performance across four languages without hand-engineered features or gazetteers.
Problem
Limited supervised data makes NER generalization difficult, while language-specific features and resources are costly to develop for new languages and domains.
Method
The paper presents LSTM-CRF and transition-based Stack-LSTM architectures that combine output-label dependencies with character-based and unsupervised word representations.
Results
The models achieve state-of-the-art or near-state-of-the-art NER performance in English, Dutch, German, and Spanish without hand-engineered features or gazetteers.
Takeaways & Limitations
Neural sequence-labeling architectures can provide leading NER results in standard evaluations while avoiding language-specific resources such as gazetteers.
Takeaways & Limitations
The greedy Stack-LSTM applies locally optimal actions, so beam search or exploration-based training might improve it.
Abstract
from arXiv · showhide
State-of-the-art named entity recognition systems rely heavily on hand-crafted features and domain-specific knowledge in order to learn effectively from the small, supervised training corpora that are available. In this paper, we introduce two new neural architectures---one based on bidirectional LSTMs and conditional random fields, and the other that constructs and labels segments using a transition-based approach inspired by shift-reduce parsers. Our models rely on two sources of information about words: character-based word representations learned from the supervised corpus and unsupervised word representations learned from unannotated corpora. Our models obtain state-of-the-art performance in NER in four languages without resorting to any language-specific knowledge or resources such as gazetteers.
1 Introduction
The paper addresses NER’s difficulty with scarce supervision and unconstrained name vocabularies by introducing two neural architectures that avoid language-specific resources and hand-engineered features. Experiments across four languages show strong performance, with LSTM-CRF outperforming the transition-based model.
- Motivation: NER is difficult because supervised data are scarce while few constraints limit generalization over possible names.These conditions have motivated orthographic features and language-specific resources such as gazetteers.
- Motivation: Unsupervised representations from unannotated corpora improve generalization but traditionally augment rather than replace hand-engineered features and specialized resources.Prior systems used unsupervised features alongside capitalization, character-class, and other language-specific information.
- Approach: The paper presents neural NER architectures using only limited supervised data and unlabeled corpora, with joint reasoning over token-level tagging decisions.The two models are a bidirectional LSTM with a sequential CRF layer and a transition-based model inspired by shift-reduce parsing.
- Experiments: Experiments in English, Dutch, German, and Spanish evaluate the proposed architectures without language-specific resources or features.The introduction frames the systems as relying on supervised and unlabeled corpora rather than gazetteers or hand-engineered knowledge.
- Results: The LSTM-CRF reaches state-of-the-art NER performance in Dutch, German, and Spanish and near-state-of-the-art performance in English, while the transition-based model surpasses prior results in several languages but performs worse than LSTM-CRF.These results are reported without hand-engineered features or gazetteers.
2 LSTM-CRF Model
The model combines bidirectional LSTM representations of words in context with a CRF that jointly models neighboring tags. It uses IOBES tagging for named entity recognition and decodes valid label sequences with dynamic programming.
- LSTM representations: A bidirectional LSTM concatenates left- and right-context representations to encode each word in context.The forward and backward LSTMs are distinct networks, and their concatenated representations support tagging applications.
- CRF tagging: The CRF replaces independent tagging decisions because NER labels impose hard constraints across neighboring outputs.It combines bidirectional-LSTM tag scores with bigram transition compatibility scores.
- CRF tagging: The CRF scores tag sequences using per-token scores and a transition matrix with start and end tags.The transition matrix A is square with size k+2, where k is the number of distinct tags.
- CRF tagging: Training maximizes the log-probability of the correct tag sequence, while decoding selects the maximum-scoring sequence.Because the model includes only bigram interactions between outputs, both operations can be computed with dynamic programming.
- Model architecture: The final architecture projects contextual word representations to tag scores, applies a CRF, and includes an additional hidden layer that marginally improved results.All reported results for this model incorporate the extra hidden layer.
- Tagging scheme: The model uses the IOBES tagging scheme for named entity recognition.NER assigns an entity label to every word, and entities may span several tokens.
3 Transition-Based Chunking Model
The transition-based chunking model incrementally constructs and labels multi-token chunks using a Stack-LSTM architecture inspired by transition-based parsing. It represents parser state with stacks and a buffer, predicts actions, and directly labels completed chunks.
- Model overview: The model constructs and labels multi-token names directly, composing tokens such as Mark Watney into a single representation.It is presented as an alternative to the LSTM-CRF architecture.
- Stack-LSTM representation: A Stack-LSTM incrementally represents a stack whose objects can be added with push and removed with pop operations.The stack pointer enables embeddings of stack contents for predicting subsequent actions.
- Transition system: The transition system uses output and scratch stacks plus a buffer of unprocessed words, with transitions including SHIFT and labeled REDUCE operations.The inventory is inspired by transition-based parsers, particularly Nivre’s arc-standard parser.
- Action prediction: At each step, the model predicts an action distribution from Stack-LSTM embeddings of the stack, buffer, output, and action history.These embeddings are concatenated into a fixed-dimensional representation of the algorithm state.
- Chunk composition: REDUCE(y) creates a completed chunk by moving a token sequence to the output and encoding it with a bidirectional LSTM plus a learned embedding of label type y.The operation directly predicts labeled chunks, making the model agnostic to the tagging scheme.
4 Input Word Embeddings
The models represent words using character-level and pretrained word embeddings, combining spelling-sensitive information with corpus-derived representations. Dropout is used to encourage reliance on both sources and is crucial for generalization.
- Character-level representations: Character-level features are learned during training rather than hand-engineered as word prefixes and suffixes, producing task- and domain-specific representations.The paper highlights their usefulness for morphologically rich languages and out-of-vocabulary problems.
- Character-level representations: The models construct word representations from character embeddings processed by forward and backward LSTMs, whose outputs are concatenated with a lookup-table embedding.Character embeddings are learned from the supervised corpus, while the lookup table provides word-level representations.
- Pretrained word representations: Pretrained skip-n-gram word embeddings initialize the lookup table, are sensitive to word order, and are fine-tuned during training.The authors observe significant improvements over randomly initialized word embeddings.
- Dropout: Dropout is applied to the final embedding layer because character-level embeddings alone did not improve overall performance alongside pretrained word representations.The authors report a significant improvement after applying dropout and describe it as crucial for good generalization.
5 Experiments
The experiments evaluate the models on multilingual named entity recognition and compare their performance across architectures, resources, and configurations. The LSTM-CRF achieves strong results across languages, while ablations identify pretrained word embeddings as the largest contributor to performance.
- Model configuration: The Stack-LSTM uses greedy locally optimal actions until each sentence is processed, and the authors suggest beam search as a possible source of further improvement.Its configuration includes two 100-dimensional layers per stack, 16-dimensional action embeddings, and a 20-dimensional output embedding.
- Multilingual evaluation: The models are evaluated on CoNLL-2002 and CoNLL-2003 NER datasets in English, Spanish, German, and Dutch, covering locations, persons, organizations, and miscellaneous entities.The comparisons include systems with and without external labeled resources such as gazetteers and knowledge bases, while the proposed models use none.
- Multilingual evaluation: On German, Dutch, and Spanish NER, the LSTM-CRF significantly outperforms previous methods, except for Dutch where Gillick et al. (2015) benefits from other NER datasets.The Stack-LSTM also achieves state-of-the-art or near-state-of-the-art results on these languages.
- Model analysis: The Stack-LSTM depends more on character-based representations, whereas the LSTM-CRF can rely less on orthographic information because its bidirectional LSTMs capture more context.The Stack-LSTM processes words sequentially and relies on word representations when chunking words.
- Configuration ablations: +7.31 in F1 is the improvement from pretrained word embeddings in the LSTM-CRF ablation, compared with +1.79 from the CRF layer, +1.17 from dropout, and about +0.74 from character-level word embeddings.The experiments varied the CRF, character-level representations, pretrained word embeddings, and dropout; analogous Stack-LSTM configurations were also tested.
6 Related Work
Prior NER systems combined fixed-depth decision trees, diverse classifiers, or neural architectures, while other work explored language-independent, semi-supervised, and letter-based representations. The paper positions its models among these approaches, including CNN-CRF, LSTM-CRF, bootstrapping, Bayesian, and character-based systems.
- Earlier NER systems: Earlier CoNLL systems achieved strong results by combining fixed-depth decision trees or diverse classifiers, while Qi et al. (2009) added unsupervised neural learning on massive unlabeled data.Carreras et al. performed strongly on Dutch and Spanish, Florian et al. achieved the best German score, and Qi et al. later improved on this.
- Neural architectures: Collobert et al. (2011) used a CNN over word embeddings with a CRF layer, while Huang et al. (2015) and Zhou and Xu (2015) developed similar LSTM-CRF-style models.The paper characterizes Collobert et al.’s architecture as its first model without character-level embeddings and with a CNN replacing the bidirectional LSTM; Huang et al. used hand-crafted spelling features.
- Language-independent NER: Earlier language-independent NER research included semi-supervised bootstrapping, Bayesian nonparametric entity databases, and supervised models aggregating contextual information.Cucerzan is introduced as language-independent work, while the supplied passages describe contributions by Yarowsky, Eisenstein et al., and Ratinov and Roth.
- Letter-based representations: Recent NER models increasingly used letter-based representations, including character-based encoder features and CNNs for learning character-level features.Gillick et al. framed sequence labeling as sequence-to-sequence learning, while Chiu and Nichols used CNNs similarly to Santos and Guimarães.
7 Conclusion
The paper presents two neural sequence-labeling architectures that achieve the best reported NER results in standard evaluations, including against models using external resources. Their success depends on modeling output-label dependencies and combining pretrained word representations with character-based representations.
- The proposed architectures achieve the best NER results reported in standard evaluation settings, even versus models using gazetteers.
- Both models capture output-label dependencies through either a CRF or a transition-based algorithm that constructs and labels input chunks.
- The models combine pre-trained word representations with character-based representations encoding morphological and orthographic information.