PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking

Fang Wang, Tianwei Yan, Zonghao Yang, Minghao Hu, Jun Zhang, Zhunchen Luo, Xiaoying Bai

arXiv:2508.15876v1cs.CLcs.AIcs.MA

TL;DR

Incomplete context, coarse cross-modal fusion, and challenges coordinating language and vision models complicate multimodal entity linking. DeepMEL uses specialized agents to align modalities and refine candidate selection, and reports improved accuracy over existing methods across five public benchmarks.

  • Problem

    MEL must resolve ambiguous textual and visual mentions despite incomplete context, coarse cross-modal fusion, and limitations in coordinating LLMs with LVMs.

    Source: §1 · p. 1

  • Method

    DeepMEL coordinates four specialized agents that combine LLM-generated context summaries, LVM-extracted image representations, adaptive candidate retrieval, and cloze-style entity selection.

    Source: §1 · pp. 2–3

  • Results

    Accuracy improves by 1% to 57% over existing methods across five public benchmark datasets, with ablations supporting each module's effectiveness.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DeepMEL presents a multi-agent approach to MEL that combines language and visual capabilities through specialized roles and dynamic coordination.

    Source: §1 · p. 2

    The setup assumes relevant information for target mentions has already been obtained through a named entity recognizer.

    Source: §3 · p. 4

Abstract

from arXiv · show

Multimodal Entity Linking (MEL) aims to associate textual and visual mentions with entities in a multimodal knowledge graph. Despite its importance, current methods face challenges such as incomplete contextual information, coarse cross-modal fusion, and the difficulty of jointly large language models (LLMs) and large visual models (LVMs). To address these issues, we propose DeepMEL, a novel framework based on multi-agent collaborative reasoning, which achieves efficient alignment and disambiguation of textual and visual modalities through a role-specialized division strategy. DeepMEL integrates four specialized agents, namely Modal-Fuser, Candidate-Adapter, Entity-Clozer and Role-Orchestrator, to complete end-to-end cross-modal linking through specialized roles and dynamic coordination. DeepMEL adopts a dual-modal alignment path, and combines the fine-grained text semantics generated by the LLM with the structured image representation extracted by the LVM, significantly narrowing the modal gap. We design an adaptive iteration strategy, combines tool-based retrieval and semantic reasoning capabilities to dynamically optimize the candidate set and balance recall and precision. DeepMEL also unifies MEL tasks into a structured cloze prompt to reduce parsing complexity and enhance semantic comprehension. Extensive experiments on five public benchmark datasets demonstrate that DeepMEL achieves state-of-the-art performance, improving ACC by 1%-57%. Ablation studies verify the effectiveness of all modules.

1. Introduction

DeepMEL addresses incomplete context, coarse text–image fusion, and limited LLM–LVM collaboration with a role-specialized multi-agent framework for multimodal entity linking. Its agents align modalities, adapt candidates, and select entities, and experiments report accuracy gains on five public benchmarks.

1. Introduction

  • MEL combines text and visual information to resolve mentions that may be ambiguous or contextually incomplete; the paper identifies fragmented context, coarse fusion, and LLM–LVM collaboration as challenges.Its Apple example contrasts Apple Inc. with the fruit, illustrating how both modalities can aid disambiguation.Source: §1 · pp. 1–2
  • DeepMEL introduces a role-specialized multi-agent framework for MEL, with Modal-Fuser, Candidate-Adapter, Entity-Clozer, and Role-Orchestrator coordinating cross-modal linking.The authors describe it as the first multi-agent solution applied to MEL.Source: §1 · p. 2
  • The modality-conversion strategy aligns fine-grained LLM-generated context descriptions with structured image representations extracted through LVM visual question answering.It uses a Context Summary Prompt for text and a Visual Q&A Prompt for images.Source: §1 · p. 2
  • Adaptive Iteration combines tool-based retrieval with LLM semantic reasoning to dynamically optimize the candidate set.The Candidate-Adapter progressively filters candidates to balance recall and precision.Source: §1 · pp. 2–3
  • A unified structured cloze prompt recasts MEL as a cloze-style task, simplifying input parsing and improving LLM semantic understanding.The Entity-Clozer uses this prompt to select an entity for the target mention.Source: §1 · pp. 2–3
  • Accuracy improves by 1% to 57% over existing methods across five public benchmark datasets, and ablations support the effectiveness of each module.The authors connect these results to their multi-agent design.Source: §1 · p. 3

2. Related Work

Related work spans neural textual entity linking, multimodal methods and datasets, and research on LLM- and LVM-based agents. The review highlights benchmark saturation and data-diversity constraints alongside complementary strengths and limitations of language and vision models.

2.1. Textual Entity Linking

  • Textual entity linking commonly encodes mention and entity contexts, scores vector-space similarity, and may use an FFN with softmax; saturation on traditional benchmarks has motivated harder settings.The review also notes that text-based methods remain limited for multimodal tasks.Source: §2.1 · p. 3

2.2. Multimodal Entity Linking

  • MEL research has introduced diverse datasets and multimodal models, but dataset scale, entity-type coverage, and task-system completeness remain constraints.Reviewed models include approaches for joint feature extraction, knowledge-level representation, dynamic alignment, and cross-modal interaction.Source: §2.2 · pp. 3–4

2.3. Agents-based Tasks

  • LLMs support text comprehension, planning, summarization, and reasoning, while LVMs offer fine-grained visual understanding but can struggle with complex semantic reasoning.The review also notes that LLM visual perception remains constrained by text-only training.Source: §2.3 · p. 4

3. Preliminary

MEL retrieves candidate entities from a multimodal knowledge graph for a document mention, then links the mention to the equivalent entity. Mentions and entities are represented with textual and visual information.

3. Preliminary

  • A mention combines its words, summary, and corresponding image, while each knowledge-graph entity includes a name, images, description, and attributes.Source: §3 · p. 4
  • MEL retrieves a candidate subset from the knowledge graph’s entity set, then links the ground-truth entity from that subset.Source: §3 · p. 4
  • A mention and candidate entity are equivalent when they refer to the same real-world object.Source: §3 · p. 5

4. Method

DeepMEL uses role-specialized multi-agent collaboration and modality conversion to address limitations of fusion-before-matching. It also iteratively optimizes candidates and frames disambiguation as a cloze task.

4.1. Overview

  • DeepMEL replaces fusion-before-matching with role-specialized collaboration that aligns visual information to text for multimodal entity linking.The overview identifies coarse pre-matching fusion as limiting fine-grained alignment between mentions and candidates.Source: §4.1 · p. 5
  • Its modality-conversion strategy aligns visual and contextual information, while adaptive iteration dynamically enhances candidate-set recall and precision.Source: §4.1 · p. 5
  • DeepMEL reformulates entity disambiguation as a cloze-style prompt to improve agent comprehension and reasoning.Source: §4.1 · p. 5

4.2. Role-Orchestrator: Multi-Agent Task Coordination

The Role-Orchestrator coordinates a hierarchical, role-specialized agent system. Dedicated agents handle fusion, candidate generation, and disambiguation, while validation-based updates manage agent versions.

4.2. Role-Orchestrator: Multi-Agent Task Coordination

  • The Role-Orchestrator uses hierarchical task allocation and coordination to optimize multimodal entity linking.Source: §4.2 · p. 5

4.2.1. Specialized Task Decoupling

  • Modal-Fuser aligns multimodal information, Candidate-Adapter retrieves and ranks entities, and Entity-Clozer uses context-aware reasoning and scoring to make the final choice.Source: §4.2.1 · p. 5

4.2.2. Model Update Strategy

  • Each agent tracks validation-set performance over a sliding window; a drop below a predefined threshold triggers a version update, subject to interval and candidate-validation rules.Source: §4.2.2 · p. 5

4.3. Modal-Fuser: Cross-Modal Alignment & Fusion

Modal-Fuser aligns visual and textual information through context distillation and image-description generation. Its prompts guide LLMs and LVMs to produce structured, task-relevant representations.

4.3. Modal-Fuser: Cross-Modal Alignment & Fusion

  • The Dual-Path Representation Learning Strategy aligns visual-textual semantics through structured modality conversion.Source: §4.3 · p. 5

4.3.1. Contextual Summary Distill

  • The Context Summary Prompt distills mention context into a concise sentence; it combines task-oriented CoT, 32 Wikidata-crawled examples, and standardized input/output formatting.The paper characterizes the prompt as supporting long-text comprehension, information selection, and structured output generation.Source: §4.3.1 · p. 7, §4.3.2 · p. 7

4.3.2. Visual Concept Generation

  • The Multi-Granularity Visual Question Answering prompt guides LVMs to identify image entities, adjust description detail from identity to fine-grained attributes, and generate structured questions for descriptions.Source: §4.3.2 · p. 7

4.4. Candidate-Adapter: Dynamic Candidate Generation

Candidate-Adapter narrows retrieval to a manageable set, then uses iterative feedback to refine candidates when the correct entity is missing. This combines tool-based search with semantic judgment and prompt revision.

4.4. Candidate-Adapter: Dynamic Candidate Generation

  • Candidate generation narrows the search space so subsequent disambiguation sorts and selects among candidates rather than searching the full knowledge graph.This is intended to reduce computational costs.Source: §4.4 · p. 7

4.4.1. Tool-based Candidate Search

  • Wikidata search retrieves the top 200 candidates, then BLEU and BERTScore similarity to the mention understanding filter them to the top five.Source: §4.4.1 · p. 7

4.4.2. Candidates Adaptive Iteration

  • A Judgment Agent checks whether the candidates contain the correct entity using mention semantics and contextual alignment; otherwise, it sends failure signals and candidates back to Modal-Fuser.Modal-Fuser revises the prompt using a context summary, semantic-conflict analysis, and examples of irrelevant candidates, then candidate generation runs again; the loop stops after five iterations and returns failure if the entity is still absent.Source: §4.4.2 · p. 8
  • The feedback–prompt revision–retrieval loop progressively refines the candidate set, with a default limit of five iterations.If the correct entity is not found after five iterations, the process terminates with a matching-failed signal.Source: §4.4.2 · p. 8

4.5. Entity-Clozer: Cloze-Style Entity Linking

Entity-Clozer reframes entity linking as an LLM cloze-style reasoning task. Its structured prompt supplies mention semantics and candidate options for the model to select from.

4.5. Entity-Clozer: Cloze-Style Entity Linking

  • The method reformulates string-matching entity linking as a cloze-style reasoning task to use LLM semantic understanding and reasoning.Source: §4.5 · p. 8

4.5.1. Cloze-style Prompt

  • The prompt contains a semantic summary, target mention, candidate options, and an instruction to select the best option, enabling LLM reasoning over candidates.Source: §4.5.1 · p. 8

5. Experimental setting

The evaluation uses five public MEL datasets and compares DeepMEL with textual-only and textual-visual baselines. Experiments use Wikidata, fixed prompt and retrieval settings, and accuracy at rank one.

5.1. Dataset

  • Experiments evaluate DeepMEL on five MEL datasets—WikiMEL, Richpedia, WikiDiverse, WikiPerson, and M3EL—with M3EL comprising 79,625 instances across nine multimodal tasks and five distinct topics.Source: §5 · p. 8, §5.1 · pp. 8–9
  • Evaluation uses the test sets for WikiMEL and Richpedia, WikiDiverse, WikiPerson, and M3EL, following the stated prior-work split where applicable.The test sets include 1,570 WikiDiverse image-caption pairs, 6,142 WikiPerson images, and 7,962 M3EL instances.Source: §5.1 · p. 9

5.2. Baselines

  • Baselines are grouped into textual-only and textual-visual methods, including neural entity-linking models and multimodal foundation models.Examples include ARNN, BERT, and BLINK among textual-only approaches, and CLIP, GHMFC, LLaVA, and GPT-4 among textual-visual approaches.Source: §5.2 · p. 9

5.3. Implementation Details

  • In this framework, Wikidata is the source knowledge graph; MEL effectiveness in all experiments is evaluated with Accuracy (precision@1), with temperature 0.75 and maximum input length 512 tokens.The semantic-summary prompt uses 32 shots, the cloze-style prompt 2 shots, candidate adaptive iteration is capped at 5 iterations, and search results are set to 200.Source: §5.3 · p. 11

6. Results

DeepMEL is compared with textual-only and textual-visual baselines and evaluated through agent ablations, model substitutions, and hyperparameter tests. The reported findings favor DeepMEL over baselines and identify differences across agent choices and settings.

6.1. Comparison with Previous Textual-Only EL Models.

  • DeepMEL outperforms textual-only entity-linking models across five datasets; SumMC and LLMaEL report accuracy improvements of 10% to 15%.The paper attributes their performance to LLM-based compression of document content into mention-related sentences, while noting that semantic disambiguation remains unresolved.Source: §6.1 · p. 11

6.2. Comparison with Previous Textual-Visual MEL Models.

  • DWE+ is the strongest listed baseline, ranking behind DeepMEL in the comparison of textual-visual MEL methods across datasets.Source: §6.2 · p. 11, Table 2 · p. 12

6.3. Comparison with Multimodal Large Language Models.

  • DeepSeek performs best among the evaluated multimodal LLMs, with average accuracy 5%–15% higher than the other models.The paper suggests input text length limits may contribute to the difference.Source: §6.3 · p. 13

6.4. Comparison the Different Agent of DeepMEL.

  • Ablations show accuracy drops when cloze-style prompting, Candidate-Adapter, Visual Generation, Modal-Fuser, or Role-Orchestrator is disabled.The cloze-style module is especially effective on WikiDiverse and M3EL; Modal-Fuser contributes contextual information and structured image output for subsequent candidate construction.Source: §6.4 · p. 13, Table 3 · p. 13

6.5. Comparison of Media Agent in DeepMEL Using Different LLMs and LVMs.

  • With the remaining architecture fixed, DeepSeek is the best textual agent, while GPT-4 leads on most visual-agent datasets.On M3EL, GPT-4 scores 0.88 as visual agent, 0.01 above MiniGPT, 0.19 above LLaVA, and 0.26 above Qwen2.5-VL.Source: Table 4 · p. 13, §6.5 · p. 14

6.6. Comparison of Different Key Hyperparameter Settings.

  • BERTScore filtering yields 15% higher average accuracy than BLEU, and temperature 0.75 gives the best overall performance across datasets.Source: Table 5 · p. 14, §6.6 · p. 15
  • Increasing initial candidates from 50 to 200 improves performance by 21% on Wikidata-MEL and 25% on M3EL, while increasing to 300 reduces it by 8% on WikiPerson and 4% on M3EL.The authors associate excessive candidates with noise, redundancy, and longer inputs.Source: Table 5 · p. 14, §6.6 · p. 15

7. Conclusion

DeepMEL frames MEL as a multi-agent approach that combines LLM summarization, LVM visual question answering, adaptive candidate filtering, and cloze-style entity selection. Experiments on five public datasets demonstrate effectiveness compared with several state-of-the-art baselines.

7. Conclusion

  • Experiments on five public datasets demonstrate DeepMEL’s effectiveness compared with several state-of-the-art baselines.Source: §7 · p. 15
  • DeepMEL uses LLM summarization and LVM visual question answering to enhance mention representations by fusing textual and visual information.Source: §7 · p. 15
  • Tool-based search and adaptive strategies dynamically filter the candidate set before LLM reasoning selects an entity through a cloze-style task.Source: §7 · p. 15
  • The paper presents DeepMEL as a new multi-agent paradigm for handling multimodal entity linking.Source: §7 · p. 15

CRediT authorship contribution statement

The contribution statement assigns conceptualization, methodology, software, investigation, writing, supervision, and funding acquisition across the listed authors.

CRediT authorship contribution statement

Loading 2508.15876v1…