PaperlayerView exact revision on arXiv

Source-linked AI summary

MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains

Xuying Ning, Dongqi Fu, Tianxin Wei, Mengting Ai, Jiaru Zou, Ting-Wei Li, Hanghang Tong, Yada Zhu, Hendrik Hamann, Jingrui He

arXiv:2603.00873v1cs.AI

TL;DR

MC-Search addresses the limited evaluation of adaptive, long-chain multimodal retrieval and planning by introducing a verified benchmark with process-level assessment and training signals. Across six MLLMs, it diagnoses retrieval and modality-planning weaknesses, while Search-Align improves open-source models’ planning and retrieval fidelity.

  • Problem

    Existing MM-RAG benchmarks focus mainly on simple QA and short retrieval chains, leaving long, adaptive, multimodal reasoning underexplored.

  • Method

    MC-Search provides 3,333 HAVE-verified examples averaging 3.7 hops, with step-wise annotations, five reasoning structures, process metrics, and Search-Align process supervision.

  • Results

    Benchmarking six MLLMs reveals over-retrieval, under-retrieval, and modality-specific planning errors, while Search-Align improves open-source models’ planning and retrieval fidelity.

  • Takeaways & Limitations

    MC-Search supports fine-grained diagnosis and alignment of adaptive, modality-aware retrieval planning in agentic MM-RAG systems.

Abstract

from arXiv · show

With the increasing demand for step-wise, cross-modal, and knowledge-grounded reasoning, multimodal large language models (MLLMs) are evolving beyond the traditional fixed retrieve-then-generate paradigm toward more sophisticated agentic multimodal retrieval-augmented generation (MM-RAG). Existing benchmarks, however, mainly focus on simplified QA with short retrieval chains, leaving adaptive planning and multimodal reasoning underexplored. We present MC-Search, the first benchmark for agentic MM-RAG with long, step-wise annotated reasoning chains spanning five representative reasoning structures. Each example specifies sub-questions, retrieval modalities, supporting facts, and intermediate answers, with fidelity ensured by HAVE (Hop-wise Attribution and Verification of Evidence), resulting in 3,333 high-quality examples averaging 3.7 hops. Beyond answer accuracy, MC-Search introduces new process-level metrics for reasoning quality, stepwise retrieval and planning accuracy. By developing a unified agentic MM-RAG pipeline, we benchmark six leading MLLMs and reveal systematic issues such as over- and under-retrieval and modality-misaligned planning. Finally, we introduce Search-Align, a process-supervised fine-tuning framework leveraging verified reasoning chains, showing that our data not only enables faithful evaluation but also improves planning and retrieval fidelity in open-source MLLMs.

1 INTRODUCTION

MC-Search addresses the limits of shallow, fixed MM-RAG benchmarks by introducing long, step-wise multimodal reasoning evaluation. Its benchmark, process metrics, unified pipeline, and Search-Align training framework target adaptive retrieval and planning fidelity.

  • 1 INTRODUCTION: Existing MM-RAG benchmarks mainly use simple question answering and short 1–2-hop retrieval, omitting adaptive multimodal reasoning trajectories.
  • 1 INTRODUCTION: MC-Search provides long, step-wise annotated multimodal reasoning chains across five structures, verified by HAVE for hop necessity and non-redundancy.The benchmark contains 3,333 examples averaging 3.7 hops.
  • 1 INTRODUCTION: MC-Search introduces process-level metrics for reasoning quality, per-step retrieval fidelity, and execution drift beyond answer accuracy.The metrics are LLM-as-a-Judge, Structure-Aware per Step Hit Rate, and Rollout Deviation.
  • 1 INTRODUCTION: A unified agentic MM-RAG pipeline benchmarks six MLLMs and exposes over-retrieval, under-retrieval, and modality-specific planning errors.
  • 1 INTRODUCTION: Search-Align uses HAVE-verified reasoning chains for process-supervised fine-tuning that improves open-source models’ multimodal planning and retrieval fidelity.

2 MC-SEARCH BENCHMARK

MC-SEARCH is a benchmark for long, structured multimodal search reasoning, with verified chains, diverse topologies, and process-level evaluation beyond final-answer accuracy.

  • Benchmark representation: MC-SEARCH represents multimodal reasoning as graphs of sub-questions, retrieval modalities, evidence, and intermediate answers that are aggregated into a final answer.The retrieval function maps each sub-question and modality to evidence, while the reasoning procedure combines intermediate answers.
  • Reasoning structures: Five topologies cover serial and parallel workflows: Image-Initiated, Text-Initiated, Parallel Image-Text, Multi-Images, and Text-Only chains.These structures encode modality dependencies such as visual comparison followed by textual support or concurrent image-text retrieval.
  • Dataset construction: HAVE filters hallucinated, unnecessary, and redundant hops by testing evidence utility and navigational dependence, producing 3,333 high-quality samples after multi-stage verification.Verification also checks that no unused text or image can independently resolve a sub-question, supporting unique reasoning graphs.
  • Dataset construction: The dataset receives an overall quality score of 4.87 across factual correctness, step necessity, clarity, and multimodal alignment.Low-scoring samples are refined through targeted answer editing.
  • Evaluation protocol: MC-SEARCH evaluates both answer correctness and reasoning fidelity using F1-based measures, step-level evidence matching, and rollout deviation.Hit per Step measures recovered golden evidence, while Rollout Deviation captures under- or over-retrieval through differences in reasoning-graph length.

3 AGENTIC MM-RAG PIPELINE AND PROCESS-LEVEL ALIGNMENT

The unified agentic MM-RAG pipeline iteratively plans sub-goals, retrieves modality-appropriate evidence, and synthesizes answers. SEARCH-ALIGN converts verified trajectories into process-level supervision for open-source MLLMs.

  • Agentic MM-RAG pipeline: The pipeline loops through sub-query generation, evidence acquisition, and iterative reasoning until accumulated evidence supports a final response.Each iteration selects a retrieval action, integrates evidence, generates a sub-answer, and decides whether to continue or terminate.
  • Agentic MM-RAG pipeline: The agent chooses among text search with text queries, image search with text queries, and image search with input images.These actions provide modality-aware access to the local multimodal knowledge base.
  • Process-level alignment: SEARCH-ALIGN uses HAVE-verified trajectories containing sub-questions, retrieval actions, evidence, and intermediate answers as process-level training supervision.The framework differs from conventional SFT by supervising the reasoning process rather than only final answers.
  • Process-level alignment: Conversation-style fine-tuning trains open-source MLLMs to plan, select retrieval modalities, and integrate evidence across multiple reasoning steps.The resulting training signal targets long-horizon multimodal reasoning capabilities.

4 EXPERIMENTS

The experiments evaluate six MLLMs under a unified agentic MM-RAG pipeline and analyze performance across reasoning structures, chain lengths, retrieval deviations, modality coverage, and error types. SEARCH-ALIGN substantially improves open-source models, while long chains, retrieval imbalance, and missing visual cues expose persistent weaknesses.

  • RQ1: Backbone Performance and Search-Align Gains: +13.7 F1 and +16.0 HPS on average, with a 3.1-point RD reduction, are achieved by Qwen2.5-VL-7B after SEARCH-ALIGN, while InternVL3.5-8B gains +2.8 F1 and +12.0 HPS.After alignment, Qwen2.5-VL-7B nearly matches Gemini-2.5-Pro on text-centric chains, and InternVL improves across all topologies.
  • RQ2: Performance vs. Chain Length: F1 drops consistently with chain length, with all models degrading sharply at 4–5 hops and Qwen2.5-VL-7B remaining weakest across lengths.Gemini-2.5-Pro is the most robust at longer hops, whereas GPT-4o-Mini and Claude-3.7-Sonnet drop sharply beyond three hops.
  • RQ3: Over- and Under-Retrieval Analysis: Severe under-retrieval sharply reduces performance, moderate over-retrieval can help, and excessive over-retrieval at ∆Step ≥4 causes sharp drops across models.One or two extra retrieval turns can recover from imperfect planning, but irrelevant context eventually introduces harmful noise.
  • RQ4: Modality Coverage Analysis: Text coverage remains above 78%, but image coverage depends strongly on explicit image cues, falling for Gemini-2.5-Pro from 87.35% to 29.50% and for InternVL3.5-8B from 63.84% to 0.66%.The results expose a modality gap in which models default to text retrieval and lose visual grounding without explicit image inputs.
  • RQ5: Error Analysis: The most frequent errors are Retrieval-Failure at 84.7%, Hallucinated-Entity/Attribute at 75.8%, and Step-Omission at 74.3%.Evidence misinterpretation is less frequent, indicating that major failures arise earlier during planning and evidence acquisition.

5 RELATED WORK

Related work spans multimodal RAG benchmarks and agentic RAG systems, but existing approaches generally use shallow, fixed, or text-centric retrieval. MC-Search addresses this gap with multimodal, step-wise verified reasoning chains.

  • Evaluation of Reasoning Failures: Figure 5 summarizes eight error-type proportions, providing a taxonomy for analyzing failures in multimodal agentic reasoning.The supplied passage identifies the visual as an error-taxonomy figure but does not provide its proportions.
  • Multimodal RAG Benchmarks: Existing MM-RAG benchmarks commonly use fixed retrieve-then-generate pipelines, reduce images to captions, restrict reasoning to 1–2 hops, and lack step-wise annotations.These design choices limit assessment of multimodal search and reasoning processes.
  • Agentic RAG: Agentic RAG models retrieval as sequential decision-making through query decomposition, adaptive retrieval, and knowledge synthesis, but most systems remain text-only.MC-Search extends this sequential retrieval perspective to multimodal settings with diverse, step-wise verified chains.

6 CONCLUSION

MC-Search is presented as a foundation for evaluating structured, step-wise multimodal retrieval-augmented reasoning. The conclusion emphasizes adaptive modality-aware search and future expansion to stronger models and additional domains.

  • Conclusion: MC-Search combines five reasoning mechanisms, fine-grained annotations, hop-wise verification, and chain-level metrics to diagnose retrieval planning and reasoning quality.The benchmark is positioned as a foundation for advancing multimodal agents and principled evaluation.
  • Conclusion: The authors identify adaptive, modality-aware search strategies as important for agentic MM-RAG systems.They propose broadening evaluation to stronger reasoning models and domains such as science and mathematics.

B.1 MULTIMODAL RAG BENCHMARKS

Multimodal RAG benchmarks incorporate image evidence, while agentic RAG introduces sequential retrieval and query decomposition. MC-Search extends these directions by supervising the structure of multimodal retrieval chains.

  • Multimodal RAG Benchmarks: Existing multimodal RAG benchmarks often use fixed retrieve-then-generate pipelines, limited visual seeking, shallow reasoning, and insufficient process annotations.These limitations make fine-grained evaluation of multimodal search behavior difficult.
  • Agentic RAG: Agentic RAG systems decompose queries, adaptively retrieve evidence, and synthesize retrieved knowledge, with supervised and reinforcement learning methods aligning behavior to reasoning traces.The cited passage describes these methods as improving interpretability and related capabilities.
  • Structure-Aware Benchmarks: Unlike symbolic knowledge-base benchmarks focused mainly on what to retrieve, agentic MM-RAG requires supervision over chain initiation modality, hop order, and parallel-versus-serial structure.This distinction motivates structure-aware multimodal benchmark design.

C DATA EXAMPLE

The examples demonstrate how MC-Search represents multimodal questions as structured reasoning chains, while baseline comparisons show why fixed retrieval is insufficient for complex tasks.

  • C DATA EXAMPLE: MC-Search examples decompose multimodal questions into sequential sub-questions, retrieval modalities, evidence, and intermediate answers before producing a final answer.The examples include text chains, parallel visual-textual forks, and multi-image forks with explicit reasoning steps.
  • C DATA EXAMPLE: The mandolin example links visual identification of steel strings to body strengthening, bowl deepening, enhanced tone, and steel strings becoming dominant.
  • C DATA EXAMPLE: The Morocco example compares pavilion imagery to identify its tower and connects that visual conclusion to its status as the first World Showcase expansion pavilion.
  • D.1 SINGLE-STEP RAG BASELINE: Single-step retrieval can produce shallow or unsupported answers because one evidence passage often cannot support complex reasoning, while open-source models may answer despite insufficient context.
  • D.1 SINGLE-STEP RAG BASELINE: Two-hop fixed retrieval improves over single-step retrieval across reasoning structures, especially for multimodal cases, but remains less adaptive than agentic MM-RAG.

E DATASET STATISTICS

MC-Search contains 3,333 multimodal reasoning samples spanning five structures, supported by a large text-image knowledge base and averaging nearly four reasoning hops.

  • E DATASET STATISTICS: 3,333 multimodal reasoning samples span five structures, with Image-Initiated Chains largest at 1,306, followed by Text Chains at 945 and Parallel Visual-Textual Forks at 680.
  • E DATASET STATISTICS: The benchmark draws on nearly 390K images and 780K textual documents, while answers average 47 tokens and eight entities.
  • E DATASET STATISTICS: Average reasoning depth ranges from 3.17 hops in Text-Initiated Chains to about 4 in Parallel Visual-Textual Forks.
  • E DATASET STATISTICS: Text retrieval supplies 79.7% of hops, while image retrieval supplies 20.2%, producing a text-dominant but multimodal distribution.

F DATASET QUALITY VERIFICATION

Dataset quality is assessed by scoring each gold reasoning chain across factual correctness, necessity, clarity, and multimodal alignment.

  • F DATASET QUALITY VERIFICATION: Gemini-2.5-Pro evaluates each gold chain on factual correctness, step necessity, clarity, and multimodal alignment using integer scores from 1 to 5.The overall quality score is the average of the four dimensions.

G CROSS-MODEL AND HUMAN CONSISTENCY ANALYSIS

Consistency analyses support the reliability of the LLM-as-Judge evaluation across model families and human judgments, while cross-model testing indicates robustness to reference-chain generation style.

  • G CROSS-MODEL AND HUMAN CONSISTENCY ANALYSIS: Evaluator pairs achieve Pearson correlations above 0.91 and Spearman correlations above 0.81, with mean biases below 0.04 and narrow, stable agreement limits.
  • G CROSS-MODEL AND HUMAN CONSISTENCY ANALYSIS: Gemini-2.5-Pro and GPT-5 scores concentrate near the diagonal across LLJ dimensions, indicating similar evaluations across model families.
  • G CROSS-MODEL AND HUMAN CONSISTENCY ANALYSIS: LLJ scores align closely with human judgments, with Pearson correlations above 0.92 for Accuracy, Entity Coverage, and the Stacked score, and Spearman = 0.82 for Coherence.
  • G CROSS-MODEL AND HUMAN CONSISTENCY ANALYSIS: Gemini-2.5-Pro and GPT-4o-mini retain consistent F1 and LLJ performance and stable model ordering under GPT-5-generated versus Gemini-generated reasoning chains.
  • G CROSS-MODEL AND HUMAN CONSISTENCY ANALYSIS: The pipeline therefore reflects multi-hop reasoning ability rather than dependence on stylistic or syntactic artifacts from one reference-chain generator.

H FINE-GRAINED IMPROVEMENT ANALYSIS OF SEARCH-ALIGN

SEARCH-ALIGN reduces multiple structured planning and retrieval errors, with the largest gains in modality selection and reasoning-chain completeness. The analysis indicates that alignment improves multimodal decision-making beyond answer accuracy alone.

  • SEARCH-ALIGN consistently reduces Retrieval-Failure, Step-Omission, Order/Dependency-Error, and Modality-Mismatch across the validation split.The evaluation uses four binary error categories covering planning, retrieval, and modality selection.
  • Modality-Mismatch decreases from 65.75% to 34.72%, while Step-Omission falls from 86.15% to 73.66% after SEARCH-ALIGN.Retrieval-Failure and Order/Dependency-Error also show smaller but stable reductions.
  • The largest improvements occur in modality selection and multi-hop completeness, indicating more appropriate evidence choices and fewer skipped intermediate steps.
  • The comparison uses supervised fine-tuning configurations for InternVL3.5-8B and Qwen2.5-VL-7B-Instruct, while closed-source baselines receive no fine-tuning.Both open-source models use aligned training settings with backbone-specific learning rates and epoch counts.

J SOFT HPS EVALUATION WITH THRESHOLDED SEMANTIC MATCHING

Soft HPS shows that models often retrieve semantically related evidence despite missing exact matches, while broader retrieval preserves the under- and over-retrieval trade-offs. SEARCH-ALIGN remains effective across retrieval@1, @3, and @5, improving both retrieval accuracy and multi-hop execution.

  • Soft HPS evaluation: Soft-HPS scores for Gemini-2.5-Pro and Gemini-Flash increase as the similarity threshold τ decreases, showing semantically related retrieval when exact matches are missed.Gains become smaller near τ = 0.85, where remaining errors are attributed to imperfect planning or missing reasoning steps.
  • Top-k retrieval analysis: Severe under-retrieval with ∆Step < −3 causes large performance drops, while over-retrieval with ∆Step ≥4 becomes more harmful as k increases.The peak-performance region shifts slightly toward ∆Step near −1 or −2 when more candidates are retrieved.
  • Search-Align under Retrieval@K: Before alignment, increasing k yields only small F1 improvements while Hit per Step remains low and Rollout Deviation stays high at approximately 3–4.Broader retrieval alone does not resolve missed or misordered reasoning steps.
  • Search-Align under Retrieval@K: SEARCH-ALIGN raises Qwen2.5-VL-7B F1 by 15–20 points and lowers Rollout Deviation to around 1.0 across retrieval@1, @3, and @5.Hit per Step also rises substantially, with additional candidates providing diminishing but positive benefits.
  • Reasoning topology basis: MC-SEARCH defines five complete reasoning topologies by partitioning traces according to initiation modality and structural composition.The resulting structures include linear chains and forked multimodal patterns, with parallel text branching treated as reducible to a text-only chain.
Loading 2603.00873v1…