Source-linked AI summary
Learning Dense Representations for Entity Retrieval
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, Diego Garcia-Olano
TL;DR
Entity resolution traditionally relies on alias tables and reranking, which have difficulty using context and covering varied references. The paper trains a dual encoder to retrieve entities directly in a shared vector space, using hard-negative mining and approximate nearest-neighbor search. It reports stronger performance than discrete retrieval baselines, competitive TACKBP-2010 accuracy, and retrieval across 5.7 million entities in around 3ms per mention.
Problem
Entity resolution commonly uses a two-stage alias-table and reranking pipeline, but alias tables cannot systematically use context or capture all entity references.
Method
A dual encoder maps mentions and entities into a shared vector space, with unsupervised hard-negative mining and nearest-neighbor retrieval.
Results
The model significantly outperforms alias-table and BM25 baselines, is competitive on TACKBP-2010, and retrieves across 5.7 million Wikipedia entities in around 3ms per mention.
Takeaways & Limitations
Fully learned single-stage retrieval is reported as a viable substitute for more cumbersome two-stage retrieval and reranking systems.
Takeaways & Limitations
Evaluation comparisons cannot control candidate sets, and the larger candidate set used here may place the model at a disadvantage.
Abstract
from arXiv · showhide
We show that it is feasible to perform entity linking by training a dual encoder (two-tower) model that encodes mentions and entities in the same dense vector space, where candidate entities are retrieved by approximate nearest neighbor search. Unlike prior work, this setup does not rely on an alias table followed by a re-ranker, and is thus the first fully learned entity retrieval model. We show that our dual encoder, trained using only anchor-text links in Wikipedia, outperforms discrete alias table and BM25 baselines, and is competitive with the best comparable results on the standard TACKBP-2010 dataset. In addition, it can retrieve candidates extremely fast, and generalizes well to a new dataset derived from Wikinews. On the modeling side, we demonstrate the dramatic value of an unsupervised negative mining algorithm for this task.
1 Introduction
Entity resolution systems traditionally use alias tables to nominate candidates before reranking, but alias tables struggle with contextual ambiguity and varied references. This work replaces that pipeline with a learned dual encoder that retrieves entities directly, achieving strong accuracy and efficient retrieval.
- Knowledge bases are large, so entity resolution typically uses a two-stage candidate-generation and reranking pipeline.
- The paper introduces a dual encoder that represents mentions and entities in the same vector space for direct nearest-neighbor retrieval.Its modular hierarchy of sub-encoders captures different aspects of mentions and entities.
- Alias tables miss contextual cues and cannot capture all ways of referring to entities, limiting recall.Context can distinguish ambiguous mentions such as Costa, while colloquial references such as “Refoundation Communists” challenge fixed alias lists.
- A fully unsupervised hard-negative mining strategy produces major retrieval gains compared with using only random negatives.
- Approximate nearest-neighbor search retrieves high-quality candidate entities efficiently.
- The model significantly outperforms alias-table and BM25 baselines and is competitive with the best reported accuracy on TACKBP-2010.
- The authors describe the system as the first accurate, robust, and highly efficient fully learned substitute for two-stage retrieval and reranking, retrieving across 5.7 million entities in around 3ms per mention.
2 Related work
Prior entity-resolution systems typically retrieve candidates with alias tables and then rerank them, while this dual encoder trades cross-encoder comparisons for efficient nearest-neighbor retrieval.
- 2 Related work: Most entity-resolution systems use a two-stage pipeline: alias-table candidate nomination followed by neural or other reranking.The alias table maps possible names to entities, while recent neural work mainly improves the reranking stage.
- 2 Related work: The dual encoder cannot jointly attend to mentions and entities or directly compare their features.This constraint prevents direct mention-span/entity-title comparisons but permits nearest-neighbor retrieval.
- 2 Related work: Nearest-neighbor retrieval therefore provides efficiency at the cost of restricting allowable network structures.The model represents the two sides separately rather than using cross-encoder interactions.
3 Data
The paper evaluates disambiguation over English Wikipedia entities using Wikipedia hyperlinks for training, TACKBP-2010 for comparison, and a new Wikinews set for validation.
- 3 Data: The study focuses on entity disambiguation, ignores NIL mentions, and treats every English Wikipedia entry as a candidate.End-to-end mention detection and NIL prediction are outside the chosen task scope.
- 3 Data: Wikipedia provides 5.7M entities and 112.7M linked mentions from the 2018-10-22 English dump, split 99.9% for training and the remainder for model selection.Internal hyperlinks supply anchor-text training labels.
- 3 Data: Wikipedia version changes can substantially affect results because the candidate set grows and diverges from older evaluation knowledge bases.The paper notes that TACKBP-2010 was published when Wikipedia had around 3M entities, nearly half the later candidate count.
- 3 Data: The Wikinews evaluation contains 2263 annotated mention/entity pairs from 1801 English news documents published in 2018.Its annotations align with the contemporaneous Wikipedia training dump.
4 Entity retrieval model
The model encodes mention contexts and entities independently into a shared dense space, trains with random and mined hard negatives, and retrieves candidates by cosine similarity.
- 4 Entity retrieval model: The system retrieves entities from dense, fixed-length mention and entity representations using nearest-neighbor search.Candidate entities are ranked by cosine similarity between the two encodings.
- 4.1 Dual Encoder model: A two-tower architecture uses separate mention and entity networks whose equal-dimensional outputs interact only through cosine similarity.No direct cross-side interaction enables efficient retrieval but constrains the network design.
- 4.1 Dual Encoder model: Compound encoders combine mention context with its span, and entity paragraph/categories with its title.The architecture adds modular sub-structure to both towers.
- 4.1 Dual Encoder model: The mention encoder uses local context and sentence features, while the entity encoder uses title, first paragraph, and categories without entity IDs.Excluding IDs supports encoding candidate entities unseen during training.
- 4.1 Dual Encoder model: The model shares text embeddings across features and combines averaged unigram and bigram representations before a feed-forward layer.Category names are handled as sparse inputs.
- 4.2 Training: Training uses Wikipedia alone rather than explicit in-domain tuning, with in-batch random negatives and softmax loss over the batch similarity matrix.Training uses batch size 100, SGD with momentum 0.9, and learning rate 0.01.
- 4.3 Hard negative mining: Random negatives are insufficient because the model can often distinguish them using mention text and entity titles without exploiting context.Hard negatives are introduced to force more challenging contextual discrimination.
- 4.3 Hard negative mining: Hard-negative mining retrieves each mention’s ten nearest entities and labels those ranked above the correct entity as negatives.The procedure begins after training an initial model with random negatives.
5 Experiments
Experiments compare DEER with alias-table and BM25 retrieval baselines, evaluate recall on TACKBP-2010 and Wikinews, and examine efficiency, hard-negative mining, qualitative behavior, and related work. DEER achieves strong retrieval and resolution performance while approximate search and iterative hard-negative mining provide practical and modeling benefits.
- Evaluation setup: DEER uses alias-table, BM25, and related dual-encoder systems as retrieval or resolution baselines, with Recall@1 and Recall@100 as the main metrics.Recall@1 measures whether the top result is correct; Recall@100 measures whether the true entity appears among the top 100 retrieved items.
- Resolution performance: DEER achieves Recall@100 of 96.3% on TACKBP-2010, compared with 91.2% reported for Sun et al.’s alias-table-based method.Sun et al. score entries from an alias table, whereas DEER performs direct retrieval and uses batch negatives plus hard-negative mining.
- Approximate search: Approximate search reduces retrieval time by an order of magnitude with a small R@100 loss, or by two orders of magnitude while losing less than 3 points, relative to brute force.The benchmark uses quantization-based search on a single machine and evaluates the trade-off on Wikinews.
- Impact of hard negative mining: The first hard-negative-mining iteration produces a large Recall@1 improvement, while later iterations add gains that eventually flatten out.Each round mines high-scoring negatives from the current model and appends them to the classification task, producing a stable convergent process.
- Qualitative analysis: DEER uses context to retrieve entities missed or poorly ranked by alias tables and differentiates identical mention spans according to surrounding context.Qualitative examples include Costa, Refoundation Communists, European EADS, and distinct senses of Phoenix.
6 Conclusion
DEER is presented as a highly effective single-stage entity retrieval approach that performs at least as well as comparable two-stage systems and supports fast retrieval. The dual-encoder design also suggests extensions to incremental reference resolution and broader multilingual training.
- DEER performs at least as well as the best comparable two-stage systems without domain-specific tuning.
- Fast retrieval and approximate search make DEER likely to scale to much larger knowledge bases.
- Context encodings can retrieve entities relevant to an evolving document, supporting incremental reference resolution and collective entity resolution.
- Cross-lingual training offers an opportunity to use more examples across more languages beyond English Wikipedia.