Source-linked AI summary
Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji
TL;DR
Named Entity Disambiguation must resolve ambiguous mentions to knowledge-base entities while modeling textual context and global coherence. This paper jointly embeds words and entities using skip-gram extensions based on KB links and anchor contexts, then combines the resulting contexts with standard NED features. The method outperforms state-of-the-art NED methods on the CoNLL and TAC 2010 datasets.
Problem
NED must resolve ambiguous entity mentions to knowledge-base referents, while effective textual context and global coherence modeling remain central challenges.
Method
The method jointly embeds words and entities by extending skip-gram with KB graph and anchor context models, then combines embedding-based contexts with standard NED features.
Results
The method outperformed all state-of-the-art methods on both the CoNLL and TAC 2010 datasets.
Takeaways & Limitations
The proposed embedding enables effective modeling of both textual and global contexts for NED.
Takeaways & Limitations
The method depends on KB anchors and may learn unsatisfactory representations for tail entities; the authors identify reducing this dependence as future work.
Abstract
from arXiv · showhide
Named Entity Disambiguation (NED) refers to the task of resolving multiple named entity mentions in a document to their correct references in a knowledge base (KB) (e.g., Wikipedia). In this paper, we propose a novel embedding method specifically designed for NED. The proposed method jointly maps words and entities into the same continuous vector space. We extend the skip-gram model by using two models. The KB graph model learns the relatedness of entities using the link structure of the KB, whereas the anchor context model aims to align vectors such that similar words and entities occur close to one another in the vector space by leveraging KB anchors and their context words. By combining contexts based on the proposed embedding with standard NED features, we achieved state-of-the-art accuracy of 93.1% on the standard CoNLL dataset and 85.2% on the TAC 2010 dataset.
1 Introduction
Named Entity Disambiguation resolves ambiguous entity mentions to knowledge-base referents, using contextual and global coherence signals. The paper proposes a joint word–entity embedding for these signals and evaluates it on two standard datasets.
- NED resolves ambiguous entity mentions to their correct referent entities in a knowledge base such as Wikipedia.
- Ambiguous mentions such as “Washington” may refer to multiple entities, motivating contextual and global-coherence disambiguation.
- Word embeddings represent semantically similar words as nearby vectors learned from large, unstructured text corpora.
- The proposed embedding jointly maps words and entities into one vector space, allowing cosine similarity to measure word–entity context and entity relatedness.
- The model extends skip-gram with conventional word prediction, a KB graph model, and an anchor context model, jointly learning word and entity embeddings.
- The resulting NED method combines embedding-based textual context similarity and entity coherence with standard features using supervised machine learning.
- The method was evaluated on the CoNLL and TAC 2010 datasets and reportedly outperformed state-of-the-art methods on both.
2 Joint Embedding of Words and Entities
The joint embedding extends skip-gram with models for words, linked entities, and anchor contexts, then combines their objectives to produce shared word–entity representations. Negative sampling makes training computationally feasible while Wikipedia supplies the training data.
- Joint embedding: The method extends skip-gram by jointly learning word and entity vectors in the same continuous d-dimensional space.
- Skip-gram model: The conventional skip-gram model predicts neighboring words from a target word in text corpora.
- KB graph model: The KB graph model places entities with similar incoming links near one another by predicting an entity’s incoming-link set.
- Anchor context model: Adding only the KB graph model leaves word and entity vectors potentially separated, so the anchor context model aligns them through shared context words.
- Anchor context model: The anchor context model predicts context words surrounding entities identified by KB anchors; each anchor pairs a referent entity with its surrounding words.
- Joint training: The three objectives are linearly combined as L = Lw + Le + La, and the resulting matrix V embeds words and entities.
- Optimization: Negative sampling replaces expensive full-vocabulary or full-entity softmax calculations with binary classification against noise samples.
- Optimization: The models are trained jointly on Wikipedia using stochastic gradient descent and a multiprocess implementation.
3 Named Entity Disambiguation Using Embedding
The method disambiguates mentions by modeling textual context and entity coherence in a shared word–entity embedding, then ranking candidates with supervised learning.
- Mention Disambiguation: NED selects the most relevant referent entity from candidates generated for each mention in a document.
- Modeling Context: The proposed embedding supports both word–entity textual-context similarity and entity–entity coherence through cosine similarity.Context words and context entities are represented as averaged vectors before similarity is computed.
- Modeling Coherence: Coherence is estimated in two stages, first from unambiguous context entities and then from predicted entity assignments.A mention is treated as unambiguous when P(e|m) exceeds 0.95.
- Feature Combination: The ranking model combines embedding-based context features with prior, candidate-count, and string-similarity features.The supervised ranker is Gradient Boosted Regression Trees, using contextual similarities and their candidate ranks.
4 Experiments
The experiments assess the proposed embedding through NED evaluation and a separate entity-relatedness evaluation.
- Experimental Setup: The experimental setup includes NED experiments and a separate assessment of pairwise entity relatedness.
4.1 Training for the Proposed Embedding
The embedding is trained on a processed December 2014 Wikipedia dump containing millions of pages, billions of tokens, and anchors.
- Training Data: Training uses the December 2014 Wikipedia dump after removing navigation, maintenance, and discussion pages.The remaining corpus contains 4.9 million pages and approximately 2 billion tokens.
- Training Configuration: The embedding has 500 dimensions, uses a context window of 10 and 30 negative samples, and is trained for 10 passes over Wikipedia.Training lasted approximately five days on a 40-core Amazon EC2 server.
4.2 Entity Relatedness
Entity relatedness is evaluated by ranking candidate entities with cosine similarity and comparing the proposed method against WLM using NDCG and MAP.
- Dataset: The relatedness test set contains 3,314 entities, each paired with 91 candidate entities and gold-standard relatedness labels.
- Evaluation: NDCG and MAP are computed from cosine-similarity rankings, with WLM used as the baseline.The candidate entities are ordered by cosine similarity to the target entity.
- Results: The proposed method clearly outperformed WLM, indicating accurate capture of pairwise entity relatedness.
4.3 Named Entity Disambiguation
The NED evaluation tests the proposed method on CoNLL and TAC 2010, comparing it with state-of-the-art systems and examining feature contributions and errors. Results show strong performance, while candidate generation, dataset mention density, popularity features, and KB-anchor coverage affect outcomes.
- Experimental setup: The evaluation uses CoNLL and TAC 2010 datasets, with candidate generation based on PPRforNED and YAGO resources.CoNLL includes training, development, and test documents; TAC 2010 includes training and test documents, using mentions with valid KB entries.
- Results: The choice of candidate generation method considerably affected performance on the CoNLL dataset.Table 2 reports the proposed method’s experimental results.
- Comparison with state-of-the-art methods: The proposed method outperformed all state-of-the-art methods on both datasets.Table 3 reports accuracy scores for the proposed and comparison methods.
- Feature study: Adding string similarity features slightly improved performance, while embedding-based textual context features produced significant improvement.Base features alone achieved results comparable with some state-of-the-art methods on both datasets.
- Feature study: Coherence improved performance on CoNLL but not TAC 2010 because TAC 2010 has approximately one entity mention per document.CoNLL contains approximately 20 entity mentions per document, which provides more opportunity to model coherence.
- Error analysis: Approximately 48.6% of errors were caused by metonymy mentions, especially when an incorrect entity was popular and exactly matched the mention surface.The model uses KB popularity statistics and string similarity between entity titles and mention surfaces.
- Error analysis: Nearly 9.6% of errors involved referent entities with fewer than 10 inbound KB anchors, and 4.5% involved entities with no inbound KB anchor.The method depends on KB anchors to learn entity representations, limiting representations for tail entities.
5 Related Work
Prior NED work emphasized textual context or document-level entity coherence, while earlier embedding approaches represented entities separately or omitted KB link structure. The proposed approach jointly learns word and entity representations to model both textual context and coherence.
- NED approaches: Early NED methods modeled local textual-context similarity, whereas later state-of-the-art methods emphasized coherence among entities in the same document.The latter approaches are also called collective or global approaches.
- Entity representations: Earlier representation-learning approaches used random walks, entity descriptions, deep neural networks, or hierarchical KB information for NED.These methods constructed entity or document representations for disambiguation or coherence modeling.
- Proposed embedding: The proposed approach jointly learns word and entity vectors, enabling semantic similarity computations for both textual context and coherence.This distinguishes it from methods that do not jointly learn both representation types.
- Comparison with related embeddings: A related word-and-entity embedding for entity typing used annotated FACC1 text but did not incorporate KB link-graph data.The cited work addressed entity typing rather than NED.
- Knowledge-graph embedding: Knowledge-graph embedding methods primarily learn entity representations for link prediction, while joint word-and-entity modeling has also been reported to improve performance.These approaches target predicting new facts from existing KB facts.
6 Conclusions
The proposed embedding jointly maps words and entities into one continuous vector space, enabling the NED method to model textual and global contexts. The authors identify richer knowledge-graph relations and applications beyond NED as future directions.
- The embedding jointly maps words and entities into the same continuous vector space.
- The resulting context models allow the NED method to outperform state-of-the-art NED methods.
- Future work will leverage relevant knowledge, such as relations in a knowledge graph such as Freebase.
- The authors also plan to investigate applications of the embedding beyond NED.