PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking

Parinthapat Pengpun, Simran Khanuja, Graham Neubig

arXiv:2609.10745v1cs.CL

TL;DR

Multimodal entity-linking systems struggle with rare entities, while popularity-based rarity measures miss structurally underdocumented or culturally specific cases. This paper characterizes rarity structurally and introduces a training-free VLM framework that iteratively retrieves and reasons over Wikipedia, improving MERLIN accuracy by 6.9% overall and up to 23.3% on rare-entity slices.

  • Problem

    Entity-linking models degrade on rare and culturally niche entities, while popularity-based rarity proxies may not reflect cultural specificity or distinguish different failure modes.

  • Method

    The paper characterizes rarity with Wikidata structural metrics and combines a reasoning-capable VLM with iterative Wikipedia retrieval and reasoning.

  • Results

    +6.9% overall and up to +23.3% on the hardest rare-entity slices are achieved over the state of the art on MERLIN, with reasoning and retrieval combined performing best.

  • Takeaways & Limitations

    Different rarity definitions expose distinct entity-linking failure modes, and combining retrieval with reasoning provides the strongest reported system.

  • Takeaways & Limitations

    Retrieval accounts for 72% of pipeline errors, and transliteration differences can prevent recall when source-language mentions diverge from English Wikipedia titles.

Abstract

from arXiv · show

Multimodal entity linking grounds entity mentions in text and images to knowledge-base entries. These systems degrade on rare entities, but prior work measures rarity primarily through popularity-based metrics such as pageviews. We broaden this view using knowledge-graph structural metrics that capture how well an entity is documented and connected. These metrics identify many rare entities that popularity metrics miss. Across the resulting rare-entity slices, state-of-the-art accuracy drops by 15.4-39.9%, showing that different rarity definitions expose different failure modes. To address these failures, we introduce a simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically. Controlled experiments show that reasoning and retrieval are complementary. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval without reasoning improves rare-entity accuracy but can hurt overall accuracy. Their combination performs best. On MERLIN, a multilingual multimodal entity linking benchmark over five languages (Hindi, Indonesian, Japanese, Tamil, Vietnamese), our best system improves over the state of the art by 6.9% overall and by up to 23.3% on rare-entity slices. We release MERLIN-Rare, rare-entity test slices for targeted evaluation, with our framework.

1 Introduction

The paper broadens rare-entity evaluation beyond popularity by using knowledge-graph structure, then combines iterative Wikipedia retrieval with reasoning-capable vision-language models to address the resulting failures. On MERLIN, this framework improves state-of-the-art accuracy by 6.9% overall and up to 23.3% on rare-entity slices.

  • 1 Introduction: +6.9% average accuracy over SOTA is achieved across MERLIN’s five languages, rising to +23.3% on bottom-5% rare-entity slices.Figure 2 reports 63.9% accuracy for the proposed system versus 47.6% for Pangea on bottom-5% rare entities by language editions.
  • 1 Introduction: The study defines rarity multidimensionally using Wikidata structural metrics, distinguishing cultural specificity and structural sparsity from mere unpopularity.It releases MERLIN-RARE rare-entity test slices for targeted evaluation on MERLIN, which spans Hindi, Indonesian, Japanese, Tamil, and Vietnamese.
  • 1 Introduction: Accuracy drops 15.4–39.9% across bottom-5% rarity slices, while structural metrics expose failures on largely different entities than popularity metrics.Structural metrics show drops up to 37.0%, comparable to 37.7% for pageviews, but identify different rare entities.
  • 1 Introduction: Reasoning and retrieval are complementary: reasoning alone does not significantly improve rare-entity accuracy, retrieval helps rare entities but can hurt full-set accuracy, and their combination performs best.The framework uses iterative retrieval over Wikipedia with reasoning-capable vision-language models; reasoning helps models use retrieved evidence more effectively.
  • 1 Introduction: Retrieval hurts non-reasoning models on common entities but helps rare ones, while retrieval failure accounts for 72% of residual errors and reasoning models make fewer, more targeted calls.These analyses motivate combining retrieval with contextual reasoning rather than relying on retrieval alone.

2 Related Work

Prior work advances generative, multimodal, multilingual, and retrieval-augmented entity linking, but typically treats rarity as popularity and has not applied interleaved reasoning-retrieval to entity linking. This work addresses these gaps with multidimensional rarity characterization and retrieval-augmented reasoning.

  • Entity Linking with Large Language Models: Entity linking has shifted from candidate-set classification toward direct generation, beginning with GENRE (Cao et al., 2021a) and extending through context enrichment, adaptive routing, and retrieval-reasoning methods.
  • Multimodal and Multilingual Entity Linking: MERLIN marks an early benchmark at the intersection of multimodal and multilingual entity linking, alongside progress using visual disambiguation and autoregressive or end-to-end multilingual methods.
  • Entity Rarity and Cultural Representation: Entity-linking systems degrade on rare entities, but prior work typically equates rarity with low popularity despite evidence of cultural, linguistic, and Western-centric representation biases.
  • Entity Rarity and Cultural Representation: The paper distinguishes structural sparsity from popularity through a multidimensional rarity characterization.
  • Retrieval-Augmented Reasoning: Reasoning-retrieval approaches interleave thinking with search, but this paradigm had not been applied to entity linking, which the paper addresses.

3 Task Definition

The paper studies multilingual, multimodal entity linking: mapping a marked textual mention to the corresponding knowledge-base entity using text and an accompanying image. Following MERLIN, it predicts the entity’s English Wikipedia title and evaluates exact-match accuracy against gold annotations.

  • 3 Task Definition: Entity linking maps textual entity mentions to entries in a knowledge base.
  • 3 Task Definition: The task takes a source-language text passage T, an accompanying image I, and a marked mention m ∈ T as input.
  • 3 Task Definition: The system predicts the referenced entity’s English Wikipedia title and is evaluated with exact-match accuracy against gold annotations on MERLIN (Ramamoorthy et al., 2025).

4 Entity Rarity Analysis

Entity rarity is multidimensional: popularity, documentation, structural connectivity, and cross-lingual coverage identify substantially different rare-entity sets. On these slices, Cultural Pangea’s accuracy drops sharply, including on structurally sparse entities that popularity-only evaluation can miss.

  • Rarity metrics: The analysis combines Wikipedia documentation metrics with Wikidata structural and cross-lingual metrics to expose distinct rarity dimensions.Metrics include pageviews, backlinks, article size, revisions, editors, categories, links, references, images, language editions, statements, qualifiers, and entity age.
  • Definitions: Entities are rare when a metric falls in the bottom 5% of the test-set distribution, while “rare” serves as an umbrella for metric-specific tails.The tails include unpopular, under-documented, and structurally sparse entities; findings remain stable at 1%, 5%, and 10% thresholds.
  • Rarity metrics: 37% average overlap: bottom-5% rare-entity sets defined by different metrics identify substantially different entities, with some pairs overlapping by only 10%.An entity can therefore be popular yet structurally rare, or have thin Wikipedia coverage but dense knowledge-graph connectivity.
  • Cultural representation: Structural sparsity in cross-lingual knowledge-base signals is treated as culturally meaningful because coverage differences can amplify multilingual model bias.The paper relates this motivation to language-specific Wikipedia coverage, English-centric model outputs, and thin descriptions of digitally underrepresented cultures.
  • Rare-entity evaluation: 15.4%-39.9%: Cultural Pangea’s accuracy drops across bottom-5% rarity slices relative to its 81.1% full-set MERLIN accuracy.Degradation reaches 37.7% on the pageview slice and 37.0% on the Wikidata statement-count slice.

5 Methodology

The methodology combines matched reasoning and instruct VLMs with iterative Wikipedia retrieval, enabling controlled comparisons of reasoning, retrieval, and model scale. Its best configuration achieves a 6.9% average improvement over Cultural Pangea on the full MERLIN test set.

  • Model Selection: The study uses matched Qwen3-VL Thinking and Instruct variants at 2B, 4B, and 8B scales to isolate reasoning, retrieval, and model-size effects.The matched architecture supports controlled comparisons across reasoning and non-reasoning variants.
  • Implementation: The pipeline iteratively analyzes image and text context, issues retrieval queries, incorporates Wikipedia snippets, and repeats until confident before extracting a final title.A second pass re-prompts the model with its complete reasoning to extract the final answer.
  • Retrieval System: Retrieval uses English Wikipedia with either BM25 lexical search or multilingual embedding search over title-description pairs indexed in FAISS.The embedding strategy addresses BM25’s cross-lingual limitation for non-Latin entity mentions and returns top-k nearest pairs as snippets.
  • Implementation: Every retrieval-augmented configuration is forced to make an initial search, after which the model autonomously chooses whether and how to continue for up to 20 iterations.This prevents nominal RAG settings from behaving like No RAG, particularly for 2B models that often skipped tool use without the control.
  • Results: 6.9%: the 8B-Think+Embed system outperforms Cultural Pangea on average across the full MERLIN test set.Reasoning models consistently outperform instruct models, embedding retrieval outperforms BM25, and BM25 helps reasoning models but hurts instruct models.

6 Experimental Setup

The experiments vary Qwen3-VL’s reasoning mode, parameter scale, and retrieval strategy in a factorial design, comparing 18 configurations with published and retrieval-aware MERLIN baselines.

  • Model variants: Qwen3-VL is evaluated in Thinking and Instruct variants at 2B, 4B, and 8B parameter sizes.Thinking produces extended reasoning traces, while Instruct is standard instruction-tuned; both share the same base architecture.
  • Retrieval methods: The factorial design crosses each model variant and size with No RAG, BM25 retrieval, and multilingual embedding retrieval, yielding 18 configurations.These conditions isolate the contributions of model scale, reasoning, and retrieval to overall and rare-entity accuracy.
  • Baselines: Published MERLIN baselines are GEMEL (58.7%), mGENRE (72.9%), and Cultural Pangea (81.1%), the current state of the art.GEMEL is generative multimodal entity linking; mGENRE uses constrained beam search over Wikipedia titles; Cultural Pangea is the strongest published baseline listed.
  • Baselines: CulturalPangea-RAG prepends the top-5 embedding-retrieved title–description pairs because CulturalPangea does not support tool calling.This provides an additional retrieval-aware baseline using the paper’s embedding retrieval.

7 Results

On MERLIN, 8B-Think+Embed improves substantially over the state of the art, with larger gains on rare-entity slices. The results show that retrieval and reasoning are complementary, while retrieval failures remain the dominant error bottleneck, especially for rare and non-Latin mentions.

  • Full MERLIN Results: 87.9% average accuracy: 8B-Think+Embed outperforms Cultural Pangea by +6.9% on full MERLIN, including +10.0% on Hindi and Indonesian.The advantage remains +4.9% under redirect-aware scoring.
  • RQ1: How Does the Advantage Scale with Entity Rarity?: +5.5% to +23.3% gains across all 15 rare-entity slices exceed the +6.9% full-dataset gain on 14 slices.The largest gains are for qualifiers (+23.3%), statements (+22.1%), and Wikidata outgoing links (+21.7%).
  • RQ2: What Drives the Gains: Reasoning, Retrieval, or Their Combination?: +18.8% RAG improvement on language-edition rare entities versus +3.8% on the full dataset shows retrieval becomes essential when parametric knowledge fails.Reasoning alone has no significant rare-entity effect (p > 0.5 across all slices), whereas reasoning enables more effective use of retrieved evidence.
  • RQ2: What Drives the Gains: Reasoning, Retrieval, or Their Combination?: −4.9% on the full dataset versus +8.1% to +12.6% on structural rare slices captures BM25’s instruct-model reversal under retrieval.Instruct models issue 3.2–3.7 searches without deliberation, flooding context; thinking models make 1.0–2.2 deliberate searches and 56–58% of transitions refine prior queries, versus 35–40% for instruct models.
  • RQ3: Can Smaller Reasoning Models Match Larger Ones?: +5 to +7% on rare entities: 4B-Think+Embed beats 8B-Instr despite being virtually identical overall (+0.3%).At 2B, no-RAG baselines outperform retrieval configurations because 2B-Think makes only one forced search on 97% of examples, indicating effective tool use requires sufficient model capability.
  • Error Analysis: 72% of errors occur because search never surfaces the correct entity, while 23.5% involve reasoning that finds but rejects it.Rare entities have a 31.1% error rate versus 8.4% for head entities, with more completely-wrong, concept-granularity, and name-format errors; retrieval failures are also a larger share for non-Latin inputs.

8 Conclusion

The conclusion shows that structural sparsity reveals greater cultural-long-tail difficulty than pageview-based rarity measures. A reasoning-capable VLM combining iterative Wikipedia retrieval and reasoning achieves the strongest results, while retrieval failures remain the main limitation.

  • 8 Conclusion: Unpopularity is not equivalent to rarity: sparse Wikidata structure and few Wikipedia language editions cause larger performance drops than pageview-based analyses suggest.This indicates that prior work underestimated the difficulty of the cultural long tail.
  • 8 Conclusion: +6.9% over the state of the art on MERLIN and up to +23.3% on the hardest rare-entity slices are achieved by the proposed framework.The framework uses a reasoning-capable VLM that iteratively searches and reasons over Wikipedia.
  • 8 Conclusion: Combining reasoning and retrieval produces the strongest overall system, whereas reasoning without retrieval has no significant effect on rare entities.Retrieval alone can help rare entities but may not match the combined approach.
  • 8 Conclusion: +18.8% retrieval gain on structurally sparse entities versus +3.8% on the full dataset represents a 5.0× increase for the 8B reasoning model.A 4B reasoning model with retrieval matches an 8B instruct model overall and outperforms it by +5 to +7% on rare entities.
  • 8 Conclusion: Retrieval failure accounts for 72% of residual errors, and long-tail disambiguation remains an open problem even with retrieval.The conclusion identifies cross-lingual retrieval as a useful direction for future work.

Limitations

The study’s evidence is limited by model-family coverage, English-Wikipedia coverage, and retrieval reliability, while exact-match evaluation penalizes valid non-canonical titles. Retrieval is the dominant pipeline bottleneck, and generalization beyond the evaluated setting remains uncertain.

  • Model-family evidence: The controlled factorial experiments primarily use Qwen3-VL (Team, 2025), while the GLM check supports rare-entity retrieval benefits but does not establish that the full reasoning-by-retrieval interaction generalizes across model families.GLM’s non-thinking mode could not sustain the retrieval loop, although adding retrieval in thinking mode improved all 15 rare-entity slices.
  • Scope and generalization: The evaluation covers five MERLIN languages and English Wikipedia, limiting applicability to entities without English coverage and leaving performance in languages with sparser Wikipedia representation unknown.African languages are cited as an example of settings that could test the limits of retrieval-augmented approaches for the cultural long tail.
  • Retrieval limitations: Retrieval accounts for 72% of pipeline errors, and substantial transliteration differences can prevent either retrieval method from recalling the correct English Wikipedia title.Embedding retrieval partially mitigates cross-lingual mismatch for non-Latin scripts; improving cross-lingual retrieval for LLM tool use is identified as the most impactful future direction.
  • Evaluation protocol: MERLIN’s standard exact-match protocol treats valid but non-canonical title strings, such as redirects or common abbreviations, the same as genuinely incorrect predictions.The protocol is retained for consistency with prior MERLIN work and comparability across systems.

Ethical Considerations

The work relies on publicly available resources and involves no personal data or human subjects, but inherits Wikipedia’s coverage biases. These biases can reduce reliability for marginalized communities, and the system cannot correct them.

  • Ethical Considerations: The study uses publicly available MERLIN, Wikipedia, and Wikidata data and involves no personal data or human subjects.
  • Ethical Considerations: The system inherits Wikipedia’s underrepresentation of non-Western entities and perspectives, which its rarity analysis can surface but cannot correct.
  • Ethical Considerations: Entity linking may perform less reliably on entities from marginalized communities systematically underrepresented in Wikipedia.

A Appendix … A.17 Licenses for Artifacts Used and Released

Across its appendix analyses, the paper shows that rarity metrics identify substantially different entities, retrieval-enhanced reasoning improves robustness across rare-entity definitions and model settings, and the released artifacts follow specified licensing terms. The analyses also characterize search behavior, computational tradeoffs, error sources, and cross-model limitations.

  • A.6 Robustness Analysis; A.7 Threshold Robustness: The final system degrades less than Pangea on 14 of 15 rare-entity slices, with robustness gaps up to 16.4%, while its advantage persists across rarity thresholds from 1% to 10% (+8.0% to +41.1%).A positive robustness gap means the proposed system’s accuracy drops less; entity age is the sole exception.
  • A.1 Rarity Metric Independence: The 15 rarity metrics identify largely different entities: mean bottom-5% Jaccard overlap is 37%, with 48% within Wikipedia metrics, 38% within Wikidata metrics, and 30% between families.Qualitative examples include entities that are structurally rare in Wikidata despite substantial Wikipedia documentation and entities showing the reverse pattern.
  • A.8 Accuracy by Rarity Decile: The system’s accuracy declines roughly half as fast as the baseline toward rare entities: the decile-1-versus-10 gap is approximately 26% versus 45–47%, yielding +6.9% overall despite a 2–3% deficit on common entities.The baseline declines steeply and near-monotonically across Wikidata-structural rarity deciles, with Spearman ρ ≥0.92; the curves cross as entities become rarer.
  • A.11 Reasoning Compensates for Size; A.13 Search Behavior Analysis; A.15 Search Query Analysis: Reasoning with retrieval compensates for model size: 4B-Think+Embed is nearly tied with 8B-Instruct on the full set (83.7% vs. 83.5%) but leads by +5.9%, +6.9%, and +7.1% on rare language editions, statements, and Wikidata outgoing links.Thinking models stop earlier, whereas Instruct queries increasingly repeat or degrade; query length itself is similar across configurations, and refinement and variation dominate transitions.
  • A.10 Pipeline Decomposition; A.13 Search Behavior Analysis: Embedding retrieval finds the target on the first search 25–40% of the time versus 6–15% for BM25, while retrieval failures still dominate and 8B-Think+Embed has the lowest failure rate at 72.1%.Search 2 is most productive, later searches show diminishing returns, and reasoning models use longer deliberation between searches despite making fewer searches than Instruct models.
  • A.14 Computational Cost Analysis: The accuracy–cost frontier ranges from 4B-Instruct at 76.2% and 2.8k tokens to 8B-Think+Embed at 87.9% and 11.1k tokens; the best configuration costs 3.9× more for +11.7% accuracy.The no-RAG 8B-Think baseline reaches 84.2% at 5.0k tokens, or 96% of the best system’s accuracy at 45% of its token cost.
  • A.16 Second Model Family: GLM-4.6V-Flash: Adding embedding retrieval improves GLM-4.6V-Flash thinking mode on all 15 rare-entity slices, with significant gains on 11, but GLM retrieval loses 2.7% on the full set and its non-thinking mode cannot sustain the retrieval loop.Without retrieval, GLM thinking and Qwen3-VL-8B-Thinking are near parity at 84.5 and 84.2; Qwen produces roughly 4.5× more completion tokens, potentially explaining the different full-set effects.
Loading 2609.10745v1…