PaperlayerView exact revision on arXiv

Source-linked AI summary

Named Entity Recognition with Bidirectional LSTM-CNNs

Jason P. C. Chiu, Eric Nichols

arXiv:1511.08308v5cs.CLcs.LGcs.NE

TL;DR

NER systems have traditionally relied heavily on hand-crafted features and lexicons, while earlier neural models had limited context and weak character-level modeling. This paper combines bidirectional LSTMs with character-level CNNs and introduces partial lexicon matching, achieving competitive or state-of-the-art results across CoNLL-2003 and OntoNotes 5.0. The authors also identify lexicon construction and embedding data as areas requiring further research.

  • Problem

    Earlier NER approaches relied on extensive feature engineering, while feed-forward neural models had limited context and could not explicitly exploit character-level features.

  • Method

    The paper combines bidirectional LSTMs and character-level CNNs with a lexicon encoding scheme and matching algorithm that supports partial matches.

  • Results

    The model is competitive on CoNLL-2003 without external knowledge beyond word embeddings and establishes a new state of the art on OntoNotes and CoNLL-2003 with lexicons.

  • Takeaways & Limitations

    The results suggest that neural networks can learn relevant NER features with little feature engineering and that combining complementary lexicons improves CoNLL-2003 performance.

  • Takeaways & Limitations

    Lexicon matches are noisy and ambiguous on OntoNotes, with poor coverage for several entity categories because the lexicons do not align well with its tagset.

Abstract

from arXiv · show

Named entity recognition is a challenging task that has traditionally required large amounts of knowledge in the form of feature engineering and lexicons to achieve high performance. In this paper, we present a novel neural network architecture that automatically detects word- and character-level features using a hybrid bidirectional LSTM and CNN architecture, eliminating the need for most feature engineering. We also propose a novel method of encoding partial lexicon matches in neural networks and compare it to existing approaches. Extensive evaluation shows that, given only tokenized text and publicly available word embeddings, our system is competitive on the CoNLL-2003 dataset and surpasses the previously reported state of the art performance on the OntoNotes 5.0 dataset by 2.13 F1 points. By using two lexicons constructed from publicly-available sources, we establish new state of the art performance with an F1 score of 91.62 on CoNLL-2003 and 86.28 on OntoNotes, surpassing systems that employ heavy feature engineering, proprietary lexicons, and rich entity linking information.

1 Introduction

The paper addresses limitations of feature-engineered NER and feed-forward neural models by combining bidirectional LSTMs with character-level CNNs and introducing partial lexicon matching. It evaluates this architecture on established English NER datasets.

  • Motivation: Feed-forward NER models restrict context to a fixed window and word embeddings cannot explicitly capture character features such as prefixes and suffixes.These limitations are especially relevant for rare words with poorly trained word embeddings.
  • Motivation: Bidirectional LSTMs model long-distance context on both sides of a word for sequential labeling tasks such as NER.The paper contrasts this effectively unlimited context with feed-forward models’ limited windows.
  • Contributions: The proposed contribution combines bidirectional LSTMs and CNNs to learn word- and character-level features for NER.The architecture was evaluated on well-established English-language datasets.
  • Contributions: The paper introduces a lexicon encoding scheme and matching algorithm that uses partial matches, then compares it with a simpler existing approach.The method is motivated by the importance of lexicons to NER performance.

2 Model

The model replaces a feed-forward network with a stacked bidirectional LSTM and uses a character-level CNN to construct word features before predicting tag scores.

  • Architecture: Lookup tables transform discrete word and character features into vectors that are concatenated and fed into a bidirectional LSTM network.The character-level vectors are produced by a CNN, while the BLSTM replaces the feed-forward architecture.
  • Architecture: The stacked recurrent network processes extracted word features in forward and backward directions to model sequence context.Each direction can contain multiple LSTM layers connected in sequence.
  • Output: The forward and backward outputs are separately decoded into tag log-probabilities and then added to produce the final output.A linear layer and log-softmax layer perform decoding for each tag category.
  • Output: The output-layer architecture was selected from minor variants based on preliminary performance.The paper does not describe this selection as a central architectural contribution.

2.2 Extracting Character Features Using a Convolutional Neural Network

The character CNN converts per-character representations into a fixed-size feature vector for each word, which can then be used by the sequence model.

  • Character feature extraction: The CNN applies convolution and max pooling to per-character feature vectors, producing a new feature vector for each word.Inputs include character embeddings and optionally character-type features.
  • Character feature extraction: Character embeddings and optional character-type vectors are concatenated before entering the CNN.Words are padded on both sides with special characters according to the CNN window size.
  • Character feature extraction: The CNN’s tuned hyperparameters are its window size and output vector size.

2.3 Core Features

The model uses publicly available word embeddings and character representations as core inputs, with preprocessing and embedding choices specified for the experiments.

  • Word embeddings: The best model uses publicly available 50-dimensional word embeddings trained on Wikipedia and the Reuters RCV-1 corpus.These embeddings were released by Collobert et al. (2011b).
  • Word embeddings: The experiments compare Stanford GloVe embeddings trained on 6 billion words with Google word2vec embeddings trained on 100 billion words.
  • Word embeddings: The authors also train embeddings on Wikipedia and Reuters RCV1 to test whether in-domain training data improves performance.
  • Preprocessing: All words are lower-cased before lookup-table conversion to their corresponding embeddings.Pre-trained embeddings can be modified during training.
  • Character representations: Character embeddings are randomly initialized at 25 dimensions, using PADDING and UNKNOWN tokens for CNN padding and unseen characters.The same random character embeddings are used across experiments.

2.4 Additional Word-level Features

The word-level feature section describes lexicon construction, partial-match encoding, and comparisons between lexicon configurations for NER.

  • Lexicon construction: DBpedia entities were grouped into CoNLL-2003 categories, while separate OntoNotes lexicons were not constructed because category correspondences were often unavailable.
  • Partial matching: Lexicon matching tests n-grams against prefixes or suffixes of entries, requiring matches to cover at least half an entry.Partial matches shorter than two tokens are discarded for categories other than Person.
  • Partial-match encoding: BIOES annotations encode each token’s position within a matched lexicon entry, distinguishing boundaries in partial matches.B is excluded from suffix-only matches, and E from prefix-only matches.
  • Comparison: The proposed partial-match method outperforms Collobert et al.’s approach, which treats exact and partial matches equally and uses YES/NO markers.
  • Lexicon combination: The SENNA and DBpedia lexicons complement each other and improve CoNLL-2003 performance when used together.The best configuration uses SENNA exact matching and DBpedia partial matching with BIOES annotation.

2.5 Additional Character-level Features

The character-level feature section represents each character by a four-way type vector for use in the model.

  • Character representation: A lookup table outputs a 4-dimensional vector encoding whether each character is uppercase, lowercase, punctuation, or other.

2.6 Training and Inference

The model is trained with a sentence-level log-likelihood and uses learned tag transitions with dynamic programming and Viterbi decoding for sequence prediction.

  • Training setup: Training and inference operate per sentence, with zero initial LSTM states and randomly initialized lookup tables except for described character and word embeddings.
  • Sequence scoring: The model learns a tag-transition matrix whose entries score successive tag transitions and sentence-start tags.
  • Sequence scoring: A tag sequence receives the sum of neural-network scores and transition scores across the sentence.
  • Objective function: The sentence-level log-likelihood normalizes the true sequence score over all possible tag sequences with a softmax.
  • Inference: Dynamic programming computes the objective and gradients efficiently, while Viterbi decoding selects the highest-scoring tag sequence at inference.
  • Tagging scheme: Output tags use BIOES annotation, which identifies each token’s position within an entity and has been reported to outperform BIO.

3 Evaluation

Evaluation covers CoNLL-2003 and OntoNotes 5.0, with preprocessing and hyper-parameter searches tailored to the datasets. Results examine optimization, training stability, overfitting, and feature effects.

  • Datasets and protocol: Evaluation uses the CoNLL-2003 and OntoNotes 5.0 named entity recognition datasets, with results averaged over 10 successful trials.
  • Feature effects: Character-level CNNs can replace some hand-crafted character features, while weak lexicons may make additional character features beneficial.
  • Preprocessing: Preprocessing replaces digit sequences with “0,” groups sentences by length for mini-batches, and splits OntoNotes tokens around digits.
  • Datasets and protocol: CoNLL-2003 contains Reuters newswire labeled with location, organization, person, and miscellaneous entities, while OntoNotes is larger and more varied.
  • Hyper-parameter optimization: Particle swarm produced better hyper-parameters than random search on CoNLL-2003, but not on OntoNotes.
  • Training behavior: CoNLL-2003 models continued improving without observed overtraining, whereas OntoNotes performance declined after about 18 epochs because of overfitting.
  • Training stability: BLSTM-CNN training failed in 5–10% of CoNLL-2003 trials and 1.5% of OntoNotes trials; lower learning rates, gradient clipping, or AdaDelta reduced failures.AdaDelta eliminated failures but increased training cost without improving performance.

4 Results and Discussion

The model achieves strong NER performance through bidirectional LSTM-CNN architecture, dropout, trained embeddings, and lexicon features. Results also show that lexicon quality and dataset-domain alignment substantially affect gains.

  • Overall results: The best models surpass previous reported F1 scores on both CoNLL-2003 and OntoNotes, while using only word embeddings for the latter comparison.The authors report competitiveness on CoNLL-2003 and a new state of the art on OntoNotes without external knowledge beyond word embeddings.
  • Overall results: LSTM models are essential for NER on the larger-domain OntoNotes dataset, where the feed-forward baseline is clearly inadequate.The reimplemented FFNN performs reasonably on CoNLL-2003 but poorly on OntoNotes.
  • Character-level features: BLSTM-CNN models significantly outperform BLSTM models with the same features on CoNLL-2003, but the difference is smaller and nonsignificant on OntoNotes with capitalization features.The comparison uses a Wilcoxon rank sum test with p < 0.05 for the four corresponding model comparisons.
  • Word embeddings: Trained word embeddings produce a large, significant improvement over random embeddings regardless of the additional features used.The reported Wilcoxon rank sum test gives p < 0.001.
  • Word embeddings: Public GloVe and Google embeddings trail Collobert’s embeddings by about one point on CoNLL-2003, while 300-dimensional vectors show no significant improvement over 50-dimensional vectors.On OntoNotes, GloVe performs close to Collobert embeddings and Google embeddings are again about one point behind.
  • Lexicon features: Combining SENNA and DBpedia lexicons significantly improves CoNLL-2003 performance because the lexicons are complementary in cleanliness, domain tailoring, and coverage.SENNA is relatively clean and newswire-oriented, whereas DBpedia is noisier but broader in coverage.
  • Lexicon features: DBpedia partial matching with BIOES encoding improves its performance, but noisy and ambiguous category matches limit its benefit on OntoNotes.OntoNotes contains spurious matches and low coverage for several entity categories, motivating improved DBpedia-to-OntoNotes tag mappings.
  • Per-genre performance: The model performs best on clean broadcast news and newswire genres and worst on telephone conversations and Web text.It improves over previous work on all genres except telephone conversation, where the small training set likely hinders learning.

5 Related Research

Earlier NER systems relied heavily on engineered features and varied across statistical, neural, and multitask paradigms. This work builds on neural embeddings, recurrent models, and character-level CNNs while reducing feature engineering.

  • 5.1 Named Entity Recognition: Recent high-performing NER approaches commonly use CRF, SVM, or perceptron models whose performance depends heavily on feature engineering.Examples include non-local features, gazetteers, clustered representations, private query logs, and phrase vectors.
  • 5.1 Named Entity Recognition: Public-data systems reached an F1 score of 91.02 on CoNLL-2003 without external knowledge, establishing the prior state of the art for that setting.Suzuki et al. used large-scale unlabeled data for feature reduction.
  • 5.1 Named Entity Recognition: Multitask systems combining NER with coreference resolution and entity linking improved OntoNotes results, although one such system was not evaluated on CoNLL-2003.The missing CoNLL-2003 evaluation was attributed to a lack of coreference annotations.
  • 5.2 Neural Networks: Early neural NER systems were limited by insufficient computational power, scalable learning algorithms, and high-quality word embeddings.These constraints preceded later progress from neural word embeddings and improved training methods.
  • 5.2 Neural Networks: SENNA used a deep feed-forward neural network and word embeddings across several NLP tasks, providing the feature encoding and objectives adopted by this work.The paper explicitly builds on SENNA’s word embeddings, feature encoding method, and objective functions.
  • 5.2 Neural Networks: Character-level CNNs had improved Spanish and Portuguese NER, and this work incorporates them into a stronger bidirectional LSTM architecture for English NER.Related systems also used BLSTMs or character CNNs for other sequential labeling tasks, but often with heavier feature engineering or different recurrent units.

6 Conclusion

The model achieves state-of-the-art NER with little feature engineering, improving over previous best results on two major datasets. Partial lexicon matching and better lexicon and embedding construction remain promising research directions.

  • The bidirectional LSTM–character CNN model achieves state-of-the-art named entity recognition with little feature engineering.Its robust training uses dropout.
  • Preliminary evaluation suggests that partial lexicon matching could further improve performance through more flexible use of existing lexicons.
  • More effective lexicon and word-embedding construction and application require further research.The authors also propose extending the model to extended-tagset NER and entity linking.
Loading 1511.08308v5…