Source-linked AI summary
Deep Joint Entity Disambiguation with Local Neural Attention
Octavian-Eugen Ganea, Thomas Hofmann
TL;DR
Entity disambiguation must resolve ambiguous mentions while existing competitive systems often depend on engineered features. The paper combines entity embeddings, local neural attention, and differentiable joint inference, achieving competitive or state-of-the-art performance across datasets. Errors remain when gold entities are absent from candidate sets, have very low priors, or context is misleading.
Problem
Entity disambiguation is challenging because surface-form mentions can ambiguously refer to multiple knowledge-base entities, while competitive systems largely rely on manually designed features.
Method
The model combines shared word–entity embeddings, neural attention over local context, mention–entity priors, and differentiable joint inference for document-level disambiguation.
Results
The approach obtains state-of-the-art accuracy on AIDA, remains competitive on other datasets, and outperforms established methods on entity similarity using less information.
Takeaways & Limitations
The experiments support a fully differentiable neural approach that learns effective entity-disambiguation features without expert-designed features, apart from the mention–entity prior.
Takeaways & Limitations
Errors occur when gold entities are missing from candidate sets, have very low mention priors, or are favored by misleading context.
Abstract
from arXiv · showhide
We propose a novel deep learning model for joint document-level entity disambiguation, which leverages learned neural representations. Key components are entity embeddings, a neural attention mechanism over local context windows, and a differentiable joint inference stage for disambiguation. Our approach thereby combines benefits of deep learning with more traditional approaches such as graphical models and probabilistic mention-entity maps. Extensive experiments show that we are able to obtain competitive or state-of-the-art accuracy at moderate computational costs.
1 Introduction
Entity disambiguation resolves ambiguous mentions to entities in a knowledge base using local and document-level contextual information. The paper targets neural learning of features and their combinations instead of relying mainly on manually designed features.
- Entity disambiguation resolves references to knowledge-base entities despite many-to-many ambiguity between surface forms and entities.
- Existing approaches use local context windows, document-level entity coherence, or combinations of both.
- The paper uses deep learning to learn basic features and their combinations from scratch, addressing the reliance of competitive systems on manually designed features.
2 Contributions and Related Work
The paper contributes common word–entity embeddings, neural attention for local disambiguation, and differentiable message passing for joint document-level inference.
- Entity Embeddings: Entity and word embeddings provide a common representation for assessing local and global evidence without entity–entity co-occurrence statistics.
- Context Attention: The local model uses attention to select informative context words, then combines context-based scores with mention–entity priors.
- Related Work: Unlike related local approaches using CNNs or auto-encoders, the paper learns entity scores from shared word–entity representations and attention.
- Collective Disambiguation: The proposed global model resolves document mentions jointly with a CRF whose parametrized potentials are learned through rolled-out loopy belief propagation.
- Collective Disambiguation: Differentiable message passing enables backpropagation through truncated inference, distinguishing the approach from prior methods that use message passing only for inference.
3 Learning Entity Embeddings
The paper learns entity embeddings from word–entity statistics so entities can share a semantic vector space with words while reducing dependence on engineered features and co-occurrence statistics.
- Entity vectors are trained for entity disambiguation and potentially other tasks, compressing entity semantics and reducing manually designed features or co-occurrence statistics.
- Embeddings are trained independently for each entity, avoiding sparse entity-cooccurrence statistics and enabling domain-specific training with possible speed and memory savings.
- Words and entities are embedded in the same low-dimensional vector space to exploit geometric similarity between them.
- Word–entity counts come from canonical knowledge-base pages and fixed windows surrounding annotated entity mentions.
- A max-margin objective makes an entity closer by dot product to sampled positive words than to sampled negative words.
- The entity vector is optimized on the unit sphere using projected stochastic gradient descent.
- Embedding quality is assessed on entity similarity and entity-disambiguation tasks, while document embeddings are left for future comparison.
4 Local Model with Neural Attention
The local model scores candidate entities from a mention’s context and prior using differentiable neural attention. It prunes noisy context words, combines contextual and prior signals, and supports gradient-based training.
- Local Model: For each candidate entity, the model computes a score from a K-word local context and mention–entity prior through differentiable functions.
- Context Attention: Neural attention assigns higher support to context words related to candidate entities and retains only the top R ≤ K words before softmax weighting.
- Context Attention: The attention mechanism produces a β-weighted context-based entity–mention score using trainable parameters.
- Architecture: Figure 1 maps context word vectors, candidate priors, and entity embeddings to entity scores, with all components trainable by backpropagation.
- Local Score Combination: The final local model combines the context score Ψ(e, c) with the context-independent log prior log ˆp(e|m).
- Prediction: Prediction independently selects the entity with the highest Ψ(e, mi, ci) score for each mention and context.
- Learning the Local Model: Training uses a max-margin loss that requires the correct entity score to exceed every other candidate by margin γ.
5 Document-Level Deep Model
The document-level model jointly disambiguates mentions with a fully connected pairwise CRF whose inference is implemented as an unrolled, differentiable LBP network. Learned neural scoring combines local evidence, document-level entity interactions, and mention–entity priors.
- CRF Model: The model represents a document as mentions with candidate entities and defines a joint CRF score over one candidate assignment per mention.The assignment space grows exponentially with the number of mentions, motivating approximate inference.
- CRF Model: Unary factors are local context scores, while pairwise factors use bilinear forms of entity embeddings to model document coherence.A normalization balances unary and pairwise terms across documents with different numbers of mentions.
- Differentiable Inference: The architecture encodes T synchronous max-product LBP message-passing iterations as T neural-network layers, enabling end-to-end differentiation and training.Messages are damped, initialized to zero, and backpropagated through the truncated inference procedure.
- Global Scores: After T iterations, marginal beliefs are combined nonlinearly with mention–entity priors to produce final marginal scores for prediction.The same learned combination function is used to integrate document-level beliefs and priors.
- Global Scores: The learned scoring function reduces prior influence when document evidence is strong but lets the prior dominate when that evidence is weak.Directly integrating the prior into unary factors performed worse because strong incorrect priors could resist global correction.
- Global Scores: The neural scoring function improves AIDA-B accuracy by 1.7% over a weighted-average scheme.This comparison concerns the learned nonlinear combination of belief and mention prior.
6 Candidate Selection
Candidate selection uses mention–entity priors and local context to construct small candidate sets, reducing the cost of quadratic LBP while retaining candidates from complementary ranking signals.
- Prior-Based Selection: Mention–entity priors serve both as a model feature and as the basis for selecting entity candidates.The prior averages probabilities from Wikipedia and Web-corpus hyperlink-count indexes and adds YAGO candidates with uniform prior.
- Candidate Pruning: The system first selects the top 30 candidates by mention–entity prior, then retains only 7 candidates for efficiency.The reduction is motivated by LBP’s quadratic complexity in the candidate-set size.
- Candidate Pruning: The final 7 candidates combine the top 4 prior-ranked entities with the top 3 ranked by local context–entity similarity.This heuristic preserves candidates supported by either prior probability or local contextual evidence.
- Evaluation Scope: Training and evaluation exclude mentions without candidate entities, so precision and recall can differ in this setup.The dataset statistics define gold recall as the share of mentions whose candidate set contains the ground-truth entity.
- Coreference Heuristic: The system merges candidate sets for generic person mentions with those of more specific mentions containing them as continuous subsequences.It also classifies a mention as a person when its most probable prior candidate is a person.
7 Experiments
Experiments evaluate the approach on entity similarity and multiple entity-disambiguation datasets, with analyses of efficiency, difficult cases, attention, and errors. The system achieves competitive or state-of-the-art performance while using learned representations and limited iterative inference.
- Datasets: The evaluation covers AIDA-CoNLL, MSNBC, AQUAINT, ACE2004, WNED-WIKI, and WNED-CWEB, with the latter two automatically extracted and less reliable.AIDA-CoNLL is manually annotated; the other listed datasets provide broader evaluation settings.
- Entity Similarity Results: The method outperforms the Wikipedia link measure and Yamada et al. on entity similarity using only word–entity statistics.Huang et al. report the best dataset result using embeddings trained from substantially more information sources.
- ED Baselines & Results: The system obtains state-of-the-art accuracy on AIDA, the largest and hardest manually created ED dataset, and remains competitive on other datasets.Evaluation uses in-KB accuracy and micro F1, averaged per mention, with five runs and validation-selected models.
- ED Baselines & Results: The method performs well on AIDA-B cases where gold entities have low Wikipedia frequency or low mention prior.Table 6 restricts analysis to cases where the gold entity appears in the candidate set.
- Hyperparameter Studies: Hard attention reduces noise from uninformative context words, while a small number of LBP iterations is sufficient for good accuracy and faster training and testing.The authors attribute the efficiency benefit to using truncated message passing during both training and testing.
- Error Analysis: Attention usually selects words relevant to at least one candidate in correctly solved hard cases, while errors arise from annotation issues, missing candidates, weak priors, or misleading context.Examples include resolving “Italians” to Italy national football team and “Australia” to Australia national cricket team in unsuitable contexts.
8 Conclusion
The paper concludes with a differentiable deep architecture for entity disambiguation that combines learned representations, local attention, adaptive scoring, and global message passing. Experiments show competitiveness across corpora, while future work targets nil detection, coreference resolution, and mention detection.
- Conclusion: The architecture combines entity embeddings, contextual attention, adaptive local score combination, and unrolled differentiable message passing for global inference.The system is fully differentiable, though word and entity embeddings are pretrained.
- Conclusion: Unlike many methods, the system avoids hand-engineered features and extensive entity co-occurrence or relatedness corpora.The conclusion presents this as a central design distinction of the approach.
- Conclusion: Extensive experiments show competitive performance across a wide range of corpora.
- Conclusion: Future extensions include nil detection, coreference resolution, and mention detection.