Source-linked AI summary
Lexicon Infused Phrase Embeddings for Named Entity Resolution
Alexandre Passos, Vineet Kumar, Andrew McCallum
TL;DR
NER systems need external information such as lexicons and word representations, while existing representations have scalability or performance limitations. The paper injects lexicon supervision into efficiently trained Skip-Gram phrase embeddings and applies them to NER, achieving a public-data state-of-the-art CoNLL 2003 system and outperforming Brown clusters on Ontonotes.
Problem
NER systems need external information such as domain-specific lexicons and word representations because syntax alone is insufficient.
Method
The paper extends Skip-Gram training so curated lexicons encourage similar embeddings for phrases sharing lexicon membership.
Results
The method provides a public-data state-of-the-art CoNLL 2003 NER system and embeddings that outperform Brown clusters on Ontonotes NER.
Takeaways & Limitations
Lexicon-infused embeddings enable high-quality phrase embeddings and a public state-of-the-art NER system across the reported tasks.
Takeaways & Limitations
Brown-cluster training has complexity O(kV 2 + N)x and is infeasible for large corpora with millions of word types.
Abstract
from arXiv · showhide
Most state-of-the-art approaches for named-entity recognition (NER) use semi supervised information in the form of word clusters and lexicons. Recently neural network-based language models have been explored, as they as a byproduct generate highly informative vector representations for words, known as word embeddings. In this paper we present two contributions: a new form of learning word embeddings that can leverage information from relevant lexicons to improve the representations, and the first system to use neural word embeddings to achieve state-of-the-art results on named-entity recognition in both CoNLL and Ontonotes NER. Our system achieves an F1 score of 90.90 on the test set for CoNLL 2003---significantly better than any previous system trained on public data, and matching a system employing massive private industrial query-log data.
1 Introduction
NER systems require external information beyond syntax, chiefly lexicons and word representations. The paper combines scalable neural embeddings with lexicon supervision to improve NER performance.
- Motivation: NER systems require external information beyond syntax, chiefly domain-specific lexicons and word representations.Representations capture syntactic and semantic behavior and help generalize to unseen word types.
- Motivation: Brown clusters are widely used but their quadratic training complexity makes large corpora and millions of word types infeasible.The computational cost scales with corpus vocabulary or, in some implementations, the number of base clusters.
- Motivation: Neural language-model representations historically underperformed Brown clusters, despite greater scalability, and large models could require days or weeks to train.Their nonlinear structure also limits how much information a log-linear CRF can exploit.
- Motivation: CBOW and Skip-Gram provide efficiently trainable alternatives, with Skip-Gram models scalable to more than a billion tokens on one machine in under half a day.These models can also learn phrase embeddings for finer-grained meaning.
- Contributions: The paper injects curated lexicon supervision into Skip-Gram embeddings and evaluates them across similarity, CoNLL 2003 NER, and Ontonotes NER.The contributions include a public-data state-of-the-art CoNLL system and embeddings outperforming Brown clusters on Ontonotes.
2 Background and Related Work
This section introduces language-model embeddings, contrasts Brown clusters with neural and log-linear alternatives, and outlines their use in NER with CRFs. It also describes the skip-gram tree formulation, phrase handling, datasets, evaluation, and baseline architecture.
- 2.1 Language models and word embeddings: Brown clustering provides hierarchical word representations but has computational complexity O(kV^2 + N)x, making it infeasible for vocabularies with millions of word types.Here, k is the number of base clusters, V the vocabulary size, and N the number of tokens.
- 2.1 Language models and word embeddings: Neural language models map word types to low-dimensional vectors, but their application to NLP tasks has generally been less successful than Brown clusters and can require slow training.Log-linear models such as CBOW and Skip-Gram were designed to produce useful embeddings efficiently rather than highly accurate language models.
- 2.2 The Skip-gram Model: The skip-gram model represents each n-gram probability as a product of conditional probabilities for context words given the central word.Hierarchical binary classifiers compute these conditional probabilities along paths from the root to context-word leaves, and parameters are learned with stochastic gradient descent.
- 2.2 The Skip-gram Model: Phrase embeddings extend the representation from individual tokens to multi-word expressions, while phrase-building considers candidate trigrams formed from high-PMI bigram pairs sharing a token.The binary Huffman tree represents tokens that may be multi-word.
- 2.3 Named Entity Recognition: NER systems require external knowledge beyond syntax; the paper evaluates on CoNLL 2003 and Ontonotes using precision, recall, and F1, alongside a stacked linear-chain CRF baseline.CoNLL has approximately 320k tokens and four entity types, whereas Ontonotes has 1.6M tokens and eighteen entity types.
- 2.3.1 Baseline System: Phrase embeddings are incorporated into the chain-CRF system as token features, with their vector scale tuned on development data; neighboring phrase embeddings did not improve performance.Connecting neighboring token embeddings was mildly beneficial for token embeddings.
3 Lexicon-infused Skip-gram Models
The lexicon-infused Skip-gram model adds lexicon-membership prediction to contextual word prediction, encouraging related phrases to receive similar embeddings while reducing the impact of noisy lexicons.
- Model extension: The added supervision tends to bring together vectors for words sharing lexicon memberships.The approach is intended to make embeddings more relevant to named-entity recognition.
- Noise handling: Lexicon training can reduce the effect of noisy entries by combining lexicon membership with contextual distributions.The paper illustrates this with “He,” whose context differs from typical named-person entities despite appearing in a person-name lexicon.
- Training example: Figure 3 illustrates that “New York” predicts “state” and, after lexicon infusion, also predicts classes such as US-State and Wiki-location.
- Model extension: Lexicon-infused Skip-gram predicts both neighboring context words and the central word’s membership in each lexicon.Each lexicon uses its own binary classifier rather than being inserted into the word tree.
- Training: Only 1% of possible negative lexicon examples are randomly selected to address the severe imbalance between negative and positive examples.The paper describes this as aggressive negative subsampling for each lexicon class.
- Training: The model contains V + K binary classifiers, with K additional lexicon classes alongside the V word classifiers.The experiments used one million vocabulary words and twenty-two lexicons, so no class tree was needed.
4 Experiments
Experiments evaluate phrase embeddings on semantic-syntactic similarity and named-entity recognition, finding gains over Skip-gram on similarity and strong NER performance across CoNLL and Ontonotes.
- 4.1 Syntactic and Semantic Similarity: Lexicon-infused embeddings outperform Skip-gram on the semantic-syntactic task, with Lex-0.01 performing best among the tested models.The models use 50 dimensions, and the authors report no perceptible computation-cost difference from standard Skip-gram.
- 4.2 CoNLL 2003 NER: CoNLL hyperparameters are tuned on the development set before retraining the selected configuration on training plus development data for final test evaluation.
- 4.2 CoNLL 2003 NER: 90.90 F1 is achieved by Lex-0.01 on the CoNLL 2003 test set, matching the highest previously reported result and exceeding the previous best trained on public data.Traditional unsupervised Skip-gram performs worse than Brown clusters in this comparison.
- 4.3 Ontonotes 5.0 NER: On Ontonotes, both Skip-gram and lexicon-infused embeddings outperform Brown-cluster features, while Skip-gram is marginally better in this setting.The authors describe these as the first published performance numbers on the Ontonotes NER task.
5 Conclusions
The paper concludes that external supervision can improve phrase embeddings and that these embeddings can support strong NER results when integrated with a log-linear CRF system.
- Conclusions: The method is evaluated on syntactic-semantic similarity, CoNLL 2003 NER, and Ontonotes 5.0 NER.
- Conclusions: Phrase embeddings can be integrated into an existing log-linear CRF system for named-entity recognition.
- Conclusions: The authors report that high-quality phrase embeddings can be learned and fine-tuned with external supervision from billions of tokens within one day of computation time.
- Future work: Future work proposes applying embeddings to dependency parsing and coreference resolution and using NER error gradients to improve embeddings.