Source-linked AI summary
MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents
Peizhou Huang, Zixuan Zhong, Zhongwei Wan, Donghao Zhou, Samiul Alam, Xin Wang, Zexin Li, Zhihao Dou, Li Zhu, Jing Xiong, Chaofan Tao, Yan Xu, Dimitrios Dimitriadis, Tuo Zhang, Mi Zhang
TL;DR
Existing benchmarks inadequately evaluate end-to-end deep research that must connect visual evidence to cited claims in long-form reports. MMDR-Bench addresses this gap with 140 image–text tasks and a three-part evaluation pipeline, finding persistent trade-offs among writing quality, citation discipline, and multimodal grounding across 25 models and agents.
Problem
Existing benchmarks mainly isolate retrieval, text-only research reports, or short-form multimodal perception rather than unified end-to-end multimodal evidence use.
Method
MMDR-Bench uses expert-crafted image–text tasks and evaluates reports with FLAE, TRACE, and MOSAIC for quality, citation grounding, and text–visual consistency.
Results
25 evaluated models and agents show persistent trade-offs between writing quality, citation discipline, and multimodal grounding.
Takeaways & Limitations
Strong report prose does not ensure faithful evidence use, while multimodal integrity remains a key bottleneck for deep research agents.
Abstract
from arXiv · showhide
Deep Research Agents (DRAs) generate citation-rich reports via multi-step search and synthesis, yet existing benchmarks mainly target text-only settings or short-form multimodal QA, missing end-to-end multimodal evidence use. We introduce MMDeepResearch-Bench (MMDR-Bench), a benchmark of 140 expert-crafted tasks across 21 domains, where each task provides an image-text bundle to evaluate multimodal understanding and citation-grounded report generation. Compared to prior setups, MMDR-Bench emphasizes report-style synthesis with explicit evidence use, where models must connect visual artifacts to sourced claims and maintain consistency across narrative, citations, and visual references. We further propose a unified, interpretable evaluation pipeline: Formula-LLM Adaptive Evaluation (FLAE) for report quality, Trustworthy Retrieval-Aligned Citation Evaluation (TRACE) for citation-grounded evidence alignment, and Multimodal Support-Aligned Integrity Check (MOSAIC) for text-visual integrity, each producing fine-grained signals that support error diagnosis beyond a single overall score. Experiments across 25 state-of-the-art models reveal systematic trade-offs between generation quality, citation discipline, and multimodal grounding, highlighting that strong prose alone does not guarantee faithful evidence use and that multimodal integrity remains a key bottleneck for deep research agents.
1 Introduction
MMDR-Bench addresses the lack of unified end-to-end evaluation for deep research agents that use both textual and visual evidence. It combines a 140-task benchmark with specialized measures for report quality, citation grounding, and text–visual consistency.
- Existing benchmarks separately emphasize web retrieval, text-only research reports, or short-horizon multimodal perception, leaving end-to-end multimodal deep research underassessed.
- MMDR-Bench provides 140 expert-refined tasks packaged as image–text bundles to evaluate integrated research workflows and their atomic multimodal foundations.The benchmark spans multimodal understanding, citation-grounded reasoning, and long-form multimodal report synthesis.
- FLAE, TRACE, and MOSAIC respectively assess report quality, citation-grounded support and source quality, and text–image consistency.
- The benchmark includes Daily and Research regimes representing lightweight everyday usage and analysis-heavy research settings.
- Visual Evidence Fidelity uses a task-specific textualized visual ground truth with a hard PASS/FAIL threshold to penalize visual misinterpretation and hallucination.
- 25 evaluated models and agentic systems reveal persistent trade-offs among writing quality, citation discipline, and multimodal grounding.The work releases the benchmark dataset, evaluation code, and comprehensive metrics.
2 Related Work
Prior work advances agentic search, multimodal reasoning, and long-form research evaluation, but these lines remain only partially integrated. Existing benchmarks rarely test whether visual details substantiate claims in citation-rich research reports.
- Deep Search and Agentic Reasoning: Agentic search frameworks decompose queries into sequential subtasks and increasingly use exploration, self-correction, reinforcement learning, and query refinement.Their evaluation still largely emphasizes final-answer accuracy against ground-truth labels.
- Multimodal Search and Reasoning: Multimodal search benchmarks assess visual reranking and image-conditioned reasoning, but usually focus on locating images or answering localized questions.
- Benchmarks and Evaluation: Deep research benchmarks examine long-form synthesis, while fixed-corpus benchmarks reduce sensitivity to web drift.
- Benchmarks and Evaluation: Text-only deep research benchmarks do not test multimodal evidence use, and multimodal benchmarks typically focus on short-form question answering.
3 Dataset Collection
MMDR-Bench defines multimodal deep research as multi-round browsing, evidence gathering, and report synthesis that explicitly interprets provided images. Its 140 tasks pair textual queries with images across Daily and Research regimes.
- Multimodal deep research requires multi-round web browsing, evidence gathering, report synthesis, and explicit interpretation and use of provided images.
- Each task pairs a textual query with a small image set that must be interpreted and integrated into a cited research report.This jointly evaluates multimodal question understanding and evidence-grounded multimodal report generation.
- 140 tasks span 21 domains and are divided into Daily and Research regimes.
- Daily contains 40 tasks across 10 domains with casual visuals, whereas Research contains 100 tasks across 9 domains with charts, tables, and diagrams requiring deeper synthesis.
4 Evaluation Methodology
The evaluation methodology combines reproducible and judge-based report assessment with citation-grounded and multimodal integrity checks. FLAE, TRACE, and MOSAIC provide complementary signals, with MOSAIC gated by the validity of the text-layer evaluation.
- 4.1 FLAE: Formula-LLM Adaptive Evaluation: FLAE combines fixed text-feature formulas with a task-aware LLM judge, fusing both channels using observable report signals.It evaluates Readability, Insightfulness, and Structural Completeness, with task-adaptive dimension weights.
- 4 Evaluation Methodology: Reports are processed through parallel FLAE and TRACE modules, followed by a gated MOSAIC stage.The pipeline activates MOSAIC only when FLAE and TRACE meet their corresponding thresholds; otherwise its score is zero.
- 4.2 TRACE: Trustworthy Retrieval-Aligned Citation Evaluation: TRACE evaluates citation-grounded evidence quality through claim–URL alignment, source accessibility, citation fidelity, and prompt fidelity.Its metrics include Consistency, Coverage, Textual Fidelity, and Visual Evidence Fidelity.
- 4.2 TRACE: Trustworthy Retrieval-Aligned Citation Evaluation: TRACE keeps Visual Evidence Fidelity as a fixed-share PASS/FAIL constraint against task-specific textualized visual ground truth.The judge returns a 0–10 score and FAIL is forced below the threshold τVef. = 6.
- 4.3 MOSAIC: Multimodal Support-Aligned Integrity Check: MOSAIC tests whether image-referenced textual claims faithfully reflect figures, charts, diagrams, and photos.It constructs multimodal items from visual mentions and routes heterogeneous artifacts for item-level scoring.
- 4.3 MOSAIC: Multimodal Support-Aligned Integrity Check: MOSAIC scores each multimodal item across Visual-Semantic Alignment, Visual Data Interpretation Accuracy, and Complex Visual Question Answering Quality.Type-specific routing and weighting adapt the assessment to different visual evidence types.
5 Experiments
Experiments evaluate multimodal and agentic systems with a weighted three-module protocol, revealing trade-offs among evidence quality, visual grounding, retrieval, and report evaluation robustness.
- Experimental Setup: The overall score combines FLAE (20%), TRACE (50%), and MOSAIC (30%), with TRACE receiving the largest weight.The protocol uses Gemini-2.5-Pro as judge and applies gated MOSAIC evaluation.
- Main Results: Gemini Deep Research ranks first overall through strong TRACE consistency and coverage while maintaining competitive MOSAIC alignment.Among non-agent web-enabled models, Gemini 3 Pro (Preview) is strongest.
- Main Results: GPT-4.1 leads multimodal extraction accuracy and has strong evidence fidelity, whereas GPT-5.2 achieves the highest Vef. score without uniformly stronger citation discipline.This result illustrates a cross-metric trade-off between visual grounding and citation reliability.
- Failure Analysis: Adding vision is not consistently beneficial: multimodal variants improve visual grounding but suffer more detail-level extraction failures on numerals, dates, labels, and table cells.When visual evidence is non-substitutable, vision constrains premises and improves faithfulness; noisy visual inputs can instead propagate spurious assumptions.
- Failure Analysis: Gemini Deep Research improves evidence aggregation and coverage through multi-step search and cross-checking, but later synthesis produces more entity-level attribution failures.The failures arise when correctly identified entities become mis-attributed while overlapping sources are consolidated.
- Main Findings: Retrieval interaction patterns matter more than model size alone, while agents amplify strong backbones but cannot replace them.Offline models can outperform some web-enabled systems on coverage, indicating that tool access does not guarantee surfaced evidence.
- Domain Analysis: Research-domain performance varies: Gemini systems remain broadly strong, GPT-5.2 peaks in structured technical areas, and Qwen 3 VL excels in visually dense scientific domains.The domain breakdown covers nine Research categories, with higher values indicating stronger performance.
- Human Consistency: The full evaluator aligns more closely with expert preferences than a vanilla prompt-based judge, and both Vef. and MOSAIC improve human-aligned scoring.Experts assess paired reports using majority preferences and coarse rubric scores across the full 140-task benchmark.
6 Conclusion
The paper introduces MMDR-Bench for end-to-end multimodal deep research and a unified evaluation pipeline spanning report quality, citation faithfulness, and text–visual consistency. Results across 25 models expose persistent trade-offs among writing quality, citation discipline, and multimodal grounding.
- Conclusion: MMDR-Bench contains 140 tasks across 21 domains under Daily and Research regimes, while the evaluation jointly measures report quality, citation-grounded faithfulness, and text–visual consistency.The benchmark targets end-to-end multimodal deep research.
- Conclusion: Results across 25 state-of-the-art LLMs and DRAs reveal persistent trade-offs between writing quality, citation discipline, and multimodal grounding.The conclusion frames these dimensions as complementary aspects of system performance.
A.1 Additional Dataset Construction Details
MMDR-Bench is constructed from expert-refined, image-dependent tasks spanning varied domains, regimes, difficulties, and languages.
- Two Task Regimes: The benchmark contains 140 expert-crafted tasks across 21 domains, divided into Daily and Research regimes.Daily emphasizes casual visual inputs, while Research uses structured, information-dense figures requiring deeper synthesis.
- Task Refinement: Domain experts iteratively refine candidate tasks using clarity, multimodal-necessity, and evidence-grounding checks.The final tasks must depend on images and support report verification through citations.
- Packaging and Metadata: Each task is packaged as an image–text bundle with variable image counts, annotated difficulty, and recorded language.The benchmark is multilingual, dominated by English and Chinese with additional long-tail languages.
A.1.1 Standardized Report Generation Protocol
The report protocol requires claims to be citation-verifiable and visually supported when image-dependent, with citations, references, and embedded images linked through shared indices.
- Citation Protocol: Citations must immediately follow factual claims, and each citation index must map to exactly one URL in the References block.This establishes Claim–URL verifiability for generated reports.
- Visual Support: Image-dependent claims must embed the referenced image inline with a caption carrying the same citation index.Input images must be embedded before models draw image-dependent conclusions.
- Source Selection: The protocol encourages diverse sourcing while recommending roughly six sources for Daily tasks and ten for Research tasks when feasible.Social media and question-answering forums are excluded from the preferred source set.
A.1.2 Textualized Visual Ground Truth for Vef.
The textualized visual ground truth records directly observable visual facts for each task, while Vef. applies a strict thresholded PASS/FAIL decision. Versioning, calibration, and regression procedures support consistent and reproducible judgments.
- Visual Ground Truth: Visual GT records only directly observable facts from task-provided images, excluding external or background knowledge.Experts author concise descriptions of salient entities, identities, numbers, labels, and table entries.
- Versioning: Each task stores its visual GT and version identifier, allowing scores to remain tied to immutable snapshots across benchmark releases.Previous snapshots are retained for exact rescoring after image or GT updates.
- Regression Protection: A fixed canary set detects regressions caused by GT edits before new benchmark releases.Changes are documented with the corresponding GT version updates.
- Decision Rule: Vef. assigns PASS only when the judge score reaches 6 and no identity-critical violation occurs.Wrong visual identity and false presence trigger immediate FAIL.
- Score Semantics: Vef. is reported as the percentage of evaluated tasks that satisfy the strict pass indicator.For example, 38.57 denotes a 38.57% strict pass-rate.
- Quality Control: Expert spot-checking calibrates the Vef. prompt against visual errors including incorrect identities, swapped labels, and incorrect numeric readings.The judge relies only on the provided images and corresponding visual GT and returns PASS/FAIL with a confidence score.
A.2 Evaluation Metrics
The evaluation framework combines reproducible report-quality scoring, citation and evidence auditing, and multimodal integrity checks. It uses task-adaptive weighting alongside fixed faithfulness constraints, while failure analysis exposes visual misreading and entity drift.
- Evaluation Pipeline: The pipeline evaluates reports with FLAE for generation quality, TRACE for evidence and citation faithfulness, and MOSAIC for image-grounded integrity.FLAE and TRACE run in parallel, while MOSAIC is activated only when multimodal items are meaningfully scorable.
- FLAE: FLAE scores Readability, Insightfulness, and Structural Completeness using formula-based and LLM-judge channels.Its judge generates task-adaptive dimension weights and a fusion coefficient, while formula features remain lightweight and auditable.
- Fusion Ablation: The deterministic fusion ablation replaces judge-based α(t, R) with αdet(R) computed from observable report properties while keeping other evaluator components fixed.The coefficient uses length completeness, heading coverage, citation compliance, and reference-block validity, with weights 0.35, 0.35, 0.20, and 0.10.
- TRACE: TRACE parses citation markers and references into claim-URL pairs, then evaluates citation support, coverage, fidelity, source quality, and visual task faithfulness.Vef. uses a fixed score threshold of 6 to produce an auditable PASS/FAIL decision.
- MOSAIC: MOSAIC evaluates image-grounded statements across visual types using semantic alignment, data-interpretation accuracy, question-answering quality, formatting, and integration factors.Diagrams additionally receive a hallucination factor.
- Failure Analysis: Vision-enabled systems show increased DTE from fine-grained visual misreadings, while agentic systems show increased EMI from entity-level drift across longer pipelines.Incorrect premises can propagate into retrieval and synthesis, and correct evidence can become bound to the wrong referent.