Select, Don't Train: The Benefits of Modular Entity Disambiguation with LLM-Based Selection

Fina Polat; Daniel Daza; Pengyu Zhang; Klim Zaporojets; Paul Groth

Source-linked AI summary

Select, Don't Train: The Benefits of Modular Entity Disambiguation with LLM-Based Selection

Fina Polat, Daniel Daza, Pengyu Zhang, Klim Zaporojets, Paul Groth

arXiv:2608.27470v1Aug 17, 2026cs.CLcs.AIcs.DB

TL;DR

Entity disambiguation combines candidate retrieval with contextual selection, but prior systems often couple these stages and rely on costly trained retrievers. This paper compares retrieval strategies under a shared LLM selector and finds that training-free retrieval can achieve strong performance while enabling abstention-aware analysis of retrieval failures.

Problem
Prior entity-disambiguation systems often couple retrieval and selection, although the stages require different capabilities and trained retrievers are costly to maintain as knowledge bases evolve.
Method
The paper systematically compares sparse, dense, and Web KB candidate retrieval with open- and closed-source LLM selectors, including abstention-aware evaluation.
Results
Training-free modular pipelines with strong LLM selectors remain competitive, while abstention-aware evaluation provides additional insight into retrieval failure and selector behavior.
Takeaways & Limitations
Retrieval and contextual selection can be studied and optimized as partially independent components rather than relying exclusively on tightly coupled trained retriever architectures.
Takeaways & Limitations
The experiments focus on English ZELDA benchmarks, and NIL entities absent from the knowledge base remain outside the study’s scope.

Abstract

from arXiv · show

Entity Disambiguation (ED) is a key task for constructing and using knowledge graphs. State-of-the-art neural approaches commonly model ED as a single task, although it consists of two distinct subproblems: retrieving candidate entities and selecting the correct one given context. Dual-encoder models optimize for both within a shared embedding space, forcing representations to balance high-recall retrieval with fine-grained selection, and they require trained retrievers, which are costly to maintain as knowledge graphs change. While recent work has begun to combine retrievers with LLM-based selectors, the interplay between the two stages has not been studied systematically. In this paper, we present a systematic comparison of retrieval strategies for candidate generation under a shared LLM-based selection stage, combining sparse retrieval (BM25), Web KB search, and a state-of-the-art trained dense retriever with several open- and closed-source LLMs. We show that, once selection is delegated to a capable LLM, training the retriever provides only modest additional value: a fully training-free BM25 retriever paired with an LLM selector reaches a new state of the art on the ZELDA benchmark, raising inKB micro-F1 from 82.3 to 86.3 (+4); pairing the same LLM with a trained dense retriever reaches 88.5. Decoupling retrieval from selection also exposes a limitation of current ED systems: when the correct entity is missing from retrieved candidates, they are forced to predict an incorrect entity. In contrast, our framework allows for abstention when retrieval failure is detected. In an evaluation setting that rewards correct abstentions, the training-free BM25 + LLM pipeline reaches 90.7 F1.

1 Introduction

The paper argues that entity disambiguation should separate candidate retrieval from contextual selection, then systematically studies retrieval strategies under a shared LLM selector. It finds that training-free retrieval remains competitive and that abstention-aware evaluation reveals retrieval failures.

ED maps ambiguous textual mentions to knowledge-base entries and supports knowledge-graph construction, information extraction, and question answering.
Retrieval seeks broad coverage across entity collections, whereas selection requires fine-grained contextual discrimination among similar candidates.
Dense retrievers rely on trained encoders and precomputed indexes, making them comparatively costly to maintain as knowledge bases evolve.
The paper systematically compares sparse, dense, and Web KB retrieval under LLM-based selection across several entity-disambiguation datasets.
Abstention-aware evaluation lets the model reject a candidate set when retrieval appears to have failed instead of being forced to predict an entity.
The evaluation covers the ZELDA suite, and results indicate that training-free BM25 can remain competitive while avoiding dense-retrieval retraining costs.
The contributions include comparing retrieval strategies, evaluating training-free and low-resource LLM selection, and analyzing abstention-aware entity disambiguation.

2 Related Work

Related work contrasts end-to-end ED systems with modular retrieval-and-selection pipelines, while positioning this paper around retrieval-selection modularity and None-of-the-Candidates evaluation. RAISED is presented as a two-stage framework that supports multiple retrieval backends and LLM-based selection.

Entity Disambiguation and Entity Linking: Entity disambiguation is the final stage of entity linking, mapping ambiguous mentions to entities in a knowledge base.
End-to-End and Modular ED: End-to-end systems jointly optimize candidate retrieval and disambiguation, but typically depend on trained retrievers and fixed indexes.
End-to-End and Modular ED: Modular systems use LLMs for contextual selection or reranking over externally retrieved candidates, often without task-specific fine-tuning.
LLMs for Contextual Augmentation: A complementary line of work uses LLMs to enrich mention context or generate entity descriptions, whereas this paper focuses on retrieval-selection modularity.
RAISED Framework: RAISED separates retrieval from LLM selection and supports abstention when no retrieved candidate is appropriate.
Abstention, NIL, and NoC: NIL prediction concerns entities absent from the knowledge base, while None-of-the-Candidates prediction concerns entities that exist but were not retrieved.

3 Methodology

RAISED decomposes entity disambiguation into high-recall candidate retrieval and context-aware LLM selection, with abstention available when candidates are unsuitable. It compares Web-based, sparse, and trained dense retrievers under a shared selection stage, alongside an abstention-aware evaluation metric.

Problem statement: ED maps each ambiguous mention to a knowledge-base entity or an abstention symbol when no candidate is appropriate.The formulation represents a document, its mention spans, and a finite entity set, then maps each mention to an entity or ⊥.
RAISED framework: RAISED separates high-recall candidate generation from fine-grained, context-aware selection, allowing the selector to abstain.The retriever produces a candidate set, while the selector returns a candidate entity or the literal abstention output when none applies.
Candidate generation: Retrieval is evaluated only for recall, requiring the correct entity to appear in the top-k candidate set whenever it exists in the knowledge base.RAISED imposes no architectural commitment on retrieval and treats it as a shortlisting stage.
Retrievers for candidate generation: The study compares Wikipedia API search, BM25 over the ZELDA dictionary, and the trained dense VERBALIZED retriever under a shared selector.Wikipedia API search is training-free and uses top-16 pages with shortened abstracts; BM25 ranks titles and descriptions using the mention and surrounding context; VERBALIZED ranks entities by embedding similarity.
LLM selection: The LLM selector receives the passage, mention, and retrieved candidates in a multiple-choice prompt and outputs a candidate ID or “None of the candidates.”Zero-shot experiments use four open- and closed-source models with an identical prompt and temperature 0.01.
Evaluation: Abstention-aware micro-F1 counts correct entity predictions and correct abstentions as positive outcomes, separating retrieval failures from selection errors.The metric rewards true abstentions when the gold entity is absent from the retrieved set and penalizes wrong predictions and false abstentions.

4 Results

Modular retrieval-selection pipelines generally outperform the end-to-end dual-encoder baseline, with strong LLM selectors making training-free BM25 competitive. Results also show that retrieval quality, candidate-set quality, selection ability, and abstention behavior contribute differently across datasets.

Retrieval–Selection Decoupling: 88.5 average micro-F1 is achieved by GPT-5.4-mini with VERBALIZED retrieval, versus 82.3 for VERBALIZED end-to-end.
Retrieval–Selection Decoupling: 86.3 micro-F1 is achieved by BM25 combined with GPT-5.4-mini without requiring a trained retriever.
Retrieval–Selection Decoupling: Improvements are relatively consistent across evaluated selectors, suggesting that gains are not tied to a single LLM.
Retrieval Strategy Comparison: VERBALIZED has the strongest average retrieval results, but BM25 remains comparatively close despite requiring no retraining or dense indexing.
Retriever Recall Analysis: At larger candidate sizes, BM25 and VERBALIZED recall differences narrow, consistent with strong LLM selectors compensating for moderate retrieval differences once the gold entity is included.
Abstention-Aware Evaluation: 90.7 abstention-aware F1 is reached by BM25 with GPT-5.4-mini, compared with 86.3 under standard inKB evaluation.

5 Conclusion

The paper finds that modular entity disambiguation can achieve competitive performance with training-free retrieval and LLM-based selection. Retrieval and contextual selection can be optimized as partially independent components, while abstention-aware evaluation reveals retrieval failures.

Training-free modular ED with LLM selection achieves competitive performance against fully trained transformer-based approaches.
BM25 remains effective with strong LLM selectors despite lower retrieval recall than dense retrievers.
Larger zero-shot models outperform fine-tuned 8B–12B models across most configurations.The gap is smallest on abstention F1 with BM25, where Qwen3-8B matches larger zero-shot selectors.
Retrieval and contextual selection can be studied and optimized as partially independent components.

6 Limitations and Future Work

The evaluation is limited to English ZELDA benchmarks, small fine-tuning sets, and the NoC abstention setting. Future work should examine broader languages and domains, more supervision, and NIL entities absent from the KB.

The experiments focus on English benchmarks within the ZELDA suite, limiting coverage of multilingual and domain-specific knowledge bases.Extending the analysis to multilingual and domain-specific KBs is identified as future work.
The low-resource experiments use relatively small fine-tuning sets, so larger-scale supervision may improve smaller selectors.
NIL entities absent from the knowledge base remain outside the study’s scope.The abstention analysis focuses on the NoC setting.

7 GenAI Usage

The authors used ChatGPT and Claude for manuscript language editing, rewriting, and stylistic improvements, while retaining responsibility for the research and conclusions.

ChatGPT and Claude were used for language editing, rewriting, and stylistic improvements during manuscript preparation.
The authors state that the research ideas, experimental design, analyses, and scientific conclusions are their own.

A Selector Prompt: Instantiated Example

The selector receives a mention, context, and candidate descriptions, then chooses the best matching entity ID or abstains when no candidate fits. The example illustrates BM25 retrieval for CHINA in a soccer-news context with k = 16 candidates.

The CHINA example uses a soccer-news passage and BM25-retrieved candidates including China, History of China, Taiwan, and other similarly named entities.
The selector prompt asks the model to identify the knowledge-base entry matching a marked mention in context.
The prompt supplies marked text, the mention, and Wikipedia candidates containing IDs, titles, and descriptions.
The selector should use surrounding context and prefer candidates whose descriptions match the mention’s role, domain, or attributes.
The selector receives k = 16 candidates and outputs a chosen ID or the literal string None of the candidates for abstention.The abstention string is mapped to ⊥ during evaluation.