Source-linked AI summary
MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome
Fangda Ye, Yuxin Hu, Pengxiang Zhu, Yibo Li, Ziqi Jin, Yao Xiao, Yibo Wang, Lei Wang, Zhen Zhang, Lu Wang, Yue Deng, Bin Wang, Yifan Zhang, Liangcai Su, Xinyu Wang, He Zhao, Chen Wei, Qiang Ren, Bryan Hooi, An Bo, Shuicheng Yan, Lidong Bing
TL;DR
MiroEval addresses gaps in deep-research evaluation by benchmarking systems on real-user, text-only and multimodal tasks with adaptive, factuality, and process-centric assessment. Across 13 systems, these dimensions captured complementary capabilities, process quality predicted overall outcomes, and multimodal tasks were more challenging; the framework’s reliability was supported by human verification and robustness analyses.
Problem
Existing benchmarks often use fixed, task-agnostic criteria, assess only final reports, provide limited multimodal evaluation, and lack real-user, refreshable tasks.
Method
MiroEval evaluates 100 real-user tasks—70 text-only and 30 multimodal—through adaptive synthesis quality, agentic factuality, and process-centric evaluation layers.
Results
The three dimensions captured complementary system capabilities, process quality reliably predicted overall outcome, multimodal tasks were more challenging, and human and robustness studies supported framework reliability.
Takeaways & Limitations
MiroEval provides a holistic diagnostic tool for evaluating the next generation of deep research agents.
Takeaways & Limitations
Process evaluation requires exposed intermediate reasoning traces, limiting applicability to fully closed-source systems, while factuality evaluation flags but does not resolve cross-source conflicts.
Abstract
from arXiv · showhide
Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.
1. Introduction
MiroEval addresses gaps in deep-research evaluation by combining a real-user, multimodal benchmark with adaptive, factuality, and process-oriented assessment. Across 13 systems, these dimensions reveal complementary capabilities, process quality predicts outcomes, and multimodal tasks are substantially harder.
- Existing benchmarks usually assess final reports, rarely support multimodal research beyond short-form QA, and often use synthetic queries that underrepresent authentic user needs.
- MiroEval contains 100 real-user-grounded tasks—70 text-only and 30 multimodal—constructed through two refreshable paths to preserve temporal relevance.One path rewrites anonymized authentic user patterns, while the other generates text-only queries from real-time web trends.
- The evaluation suite combines adaptive task-specific synthesis rubrics, agentic factuality verification over web and attachment evidence, and process audits of search and reasoning trajectories.
- Process quality reliably predicts overall outcome and exposes weaknesses such as insufficient analytical depth and gaps between reports and their underlying procedures.
- 3 to 10 points: most systems decline on multimodal tasks, which pose substantially greater challenges than text-only tasks.
- MiroThinker-H1 achieves the highest overall scores in both text-only (77.5) and multimodal (74.5) settings, while human verification reaches 92.0% precision.A human ranking study reports Kendall’s τ = 0.91.
2. Query Collection and Verification
MiroEval builds a diverse, verified benchmark from user-derived and trend-grounded queries. Its dual construction process covers broad domains and task types, filters generated queries for research necessity, and supports periodic refresh.
- User-derived queries: The user-derived path produces 65 rewritten queries from anonymized internal query patterns, including text-only and multimodal interactions with attachments.Queries are classified across seven dimensions and rewritten into easy, medium, or hard instances.
- Benchmark overview: The benchmark contains 100 queries: 70 text-only and 30 multimodal, spanning 12 domains and 10 task types.
- Benchmark overview: Technology (20), Finance (17), and Science (13) are the most represented domains, while Decision & Recommendation (17), Comparative Analysis (16), and Fact Enumeration & Verification (15) are the most common task types.
- Benchmark overview: Task types are distributed across domains, jointly testing domain knowledge and reasoning rather than evaluating either in isolation.
- User-derived queries: The resulting 65-query set covers all eight evaluation features with balanced representation across difficulty tiers.
- Assembly: The final benchmark combines 65 user-derived and 35 auto-generated queries, with source-specific metadata recorded for each query.
- Quality verification: Three graduate-level annotators achieve substantial agreement (κ > 0.74) and precision above 90% when assessing validity and non-triviality.
3. Evaluation Methodology
MiroEval evaluates deep-research systems across adaptive report synthesis, agentic factuality verification, and the quality and grounding of the underlying research process. Its methodology supports text-only and attachment-augmented queries by tailoring evaluation dimensions, retrieving evidence from web sources and attachments, and analyzing structured process records.
- Framework Overview: MiroEval assesses synthesis quality, factual grounding, and the structural integrity of the research trajectory as three complementary evaluation dimensions.The framework separates the final research artifact from the investigative procedure.
- 3.1. Comprehensive Adaptive Synthesis Quality Evaluation: Adaptive synthesis evaluation combines fixed dimensions with query-specific expertise dimensions and adds attachment grounding for multimodal tasks.Fixed dimensions include Coverage, Insight, Instruction-following, and Clarity; attachment tasks additionally require faithful interpretation and analytical use of provided materials.
- 3.2. Agentic Factuality Evaluation: Agentic factuality evaluation decomposes reports into verifiable statements and retrieves supporting or refuting evidence from external search and task-provided attachments.Each statement receives one of four labels: RIGHT, WRONG, CONFLICT, or UNKNOWN; CONFLICT explicitly represents disagreement between heterogeneous sources.
- 3.2. Agentic Factuality Evaluation: Attachment evidence retrieval combines native multimodal processing with retrieval-augmented processing to support factual grounding across diverse file types.Directly interpretable files are passed to a multimodal model, while other formats are converted into text, segmented, and searched for relevant evidence.
- 3.3. Process-Centric Evaluation: Process-centric evaluation converts noisy research logs into structured atomic units and measures whether the process is intrinsically sound and aligned with the final report.The framework explicitly evaluates procedural grounding rather than duplicating final-report factual verification.
4. Evaluation of Deep Research Systems
Evaluation across systems shows that synthesis, factuality, and process quality reveal different capabilities, while process quality broadly predicts overall outcomes. Systems also struggle with specificity, depth, efficiency, traceability, and user-derived queries, whereas the MiroThinker series maintains balanced performance.
- Overall Results: 77.5, 76.7, and 75.5 are the Text-Only overall scores of MiroThinker-H1, OpenAI Deep Research, and MiroThinker-1.7, respectively, forming the top tier.MiroThinker-1.7-mini follows at 72.9, while middle-tier systems span approximately 66 to 70.
- Dimension-Level Findings: Rankings shift substantially across dimensions: Kimi-K2.5 leads non-MiroThinker systems in Text-Only Synthesis at 75.7 but has Factuality of 65.4, while Manus reaches Factuality of 72.6 despite Synthesis of 55.4.These contrasting profiles show that polished reports and factual grounding are not interchangeable.
- Consistent Strength of the MiroThinker Series: MiroThinker-H1 ranks highest overall in both Text-Only at 77.5 and MultiModal at 74.5, while ranking first or second on every individual dimension.MiroThinker-1.7 also remains among the top three across Synthesis, Factuality, and Process without a significant weakness.
- Outcome-Level Analysis: Specificity is nearly every system’s lowest Synthesis sub-metric, trailing Coverage by 10 to 14 points.OpenAI scores 78.2 on Coverage versus 69.1 on Specificity, while MiroThinker-H1 has a 10.6-point gap.
- Outcome-Level Analysis: Insight ranges from 54.8 for Manus to 80.3 for MiroThinker-H1, making analytical observation more discriminative than Coverage or Instruction-following.Gemini-3.1-Pro and Claude-Opus-4.6 also score relatively well on Insight despite moderate overall performance.
- Outcome-Level Analysis: The precision–volume trade-off appears in factual claims: high claim counts can coexist with wrong and unverifiable claims, while cautious systems may preserve factuality at the expense of analytical depth.Kimi combines Insight of 79.8 with 595 wrong claims and 1,256 unverifiable claims; Manus has Insight of 54.8 and Factuality Ratio of 72.6.
- Benchmark Query Analysis: Auto-generated queries improve overall scores by 0.6 to 6.7 points and raise Factuality by approximately 4 to 5 points, but user-derived queries remain more complex.The ranking remains largely stable, supporting automated queries as a proxy while motivating inclusion of both query sources.
5. Related Work and Discussion
Prior benchmarks evaluate deep research mainly through final reports, fixed criteria, limited evidence scopes, or short-form multimodal tasks. MiroEval addresses these gaps by combining adaptive report evaluation, broader factual verification, process assessment, real-user grounding, and temporal refresh.
- Deep Research Systems: Deep research systems autonomously plan multi-step web investigations, synthesize heterogeneous evidence, and generate structured, citation-grounded reports.
- Report-Level Evaluation: Existing report benchmarks largely evaluate text-only reports through synthesis rubrics, temporal grounding, cited-claim verification, or workflow assessment.These efforts motivate report-level evaluation because real-world deep research produces reports rather than short answers.
- Multimodal Evaluation: Multimodal benchmarks extend retrieval or report evaluation but often remain short-form QA tasks or rely on fixed evaluation dimensions.
- Discussion: Across prior work, criteria are commonly fixed and task-agnostic, process assessment is absent, multimodal evaluation is narrow, and tasks are rarely grounded in real user needs or designed for temporal refresh.MiroEval is presented as addressing these persistent limitations.
6. Conclusion
MiroEval evaluates deep research systems with three complementary dimensions across 100 tasks and 13 systems, while exposing process weaknesses and multimodal difficulty. Its process-trace requirement and unresolved conflict handling define important boundaries for the framework.
- 6. Conclusion: MiroEval assesses 100 tasks across adaptive synthesis quality, agentic factuality, and process-centric evaluation.The framework evaluates final reports, verifies atomic claims against web sources and attachments, and audits research processes.
- 6. Conclusion: Across 13 systems, the three dimensions capture complementary aspects of system capability and reveal weaknesses invisible to output-level metrics.Process quality reliably predicts overall outcome while providing diagnostic information unavailable from report-level evaluation alone.
- 6. Conclusion: Multimodal tasks pose substantially greater challenges, while human verification and robustness experiments support the framework’s reliability.The reported human verification precision is 92.0%, and the human ranking study reaches Kendall’s τ = 0.91.
- 6. Conclusion: Process evaluation is limited to systems that expose intermediate reasoning traces.This restricts applicability to fully closed-source systems without such access.
- 6. Conclusion: Factuality evaluation identifies cross-source conflicts but does not determine which source is correct.The CONFLICT label flags disagreement between sources such as web evidence and user-provided attachments.
A. Data Collection and Report Statistics
The report-statistics analysis compares valid Deep Research outputs under controlled collection conditions and distinguishes text-only from multimodal reporting support.
- A. Data Collection and Report Statistics: Reports were collected in March 2026 within a controlled time window using automated tools and official system interfaces.The procedure aimed to ensure fair comparison across systems.
- A. Data Collection and Report Statistics: Average valid output length is reported across all evaluated tasks, with separate statistics for text-only and multimodal settings when supported.The analysis uses report length as the primary statistic.
- A. Data Collection and Report Statistics: Text-only reports are generally longer than multimodal reports.This pattern is reported across systems supporting both settings.
- A. Data Collection and Report Statistics: Several systems do not support multimodal deep research, so their statistics contain only text-only reports.The listed systems include MiroThinker-1.7-mini, DeepSeek DeepThink, Kimi-K2.5 Deep Research, and Doubao Deep Research.
B. Evaluation Features and Rewrite Strategies
The benchmark balances query coverage across eight evaluation features and uses six rewrite strategies spanning three difficulty tiers to convert user queries into evaluation instances.
- B. Evaluation Features and Rewrite Strategies: Eight evaluation features represent core deep research capabilities and are assigned to queries during curation.A routing mechanism balances feature coverage across the final benchmark.
- B. Evaluation Features and Rewrite Strategies: Six rewrite strategies across three difficulty tiers transform raw user queries into benchmark-ready instances.Strategies target specific evaluation features and are selected through routing based on constraints, feature matching, quota bonuses, and usage decay.
C. Topic Taxonomy and Domain Labels
The benchmark organizes automated query generation around a structured topic taxonomy and canonical domain labels, while robustness experiments test evaluator stability against variation in runs and judge configuration.
- C. Topic Taxonomy and Domain Labels: Automated query generation uses 12 topics and 36 subtopics, with searches collecting recent headlines and snippets as trend context.The collected trend context supports LLM-based query generation.
- C. Topic Taxonomy and Domain Labels: Each benchmark query receives one of 11 canonical domain labels.Free-form domain strings are normalized through substring matching and keyword fallbacks, with tech as the default.
- C. Topic Taxonomy and Domain Labels: The benchmark’s eight evaluation features and six rewrite strategies support structured query classification and difficulty-aware curation.Features are balanced across queries, while strategies span three difficulty tiers and route according to matching and quota criteria.
- C. Topic Taxonomy and Domain Labels: Robustness experiments examine sensitivity to random variation, judge-model choice, and prompt differences before human validation.The study evaluates consistency through controlled experiments and an expert human ranking study.
- C. Topic Taxonomy and Domain Labels: Three independent GPT-judge runs on 30 multimodal tasks produce Overall standard deviations from 0.3 to 0.6, with identical system rankings.Synthesis varies by under 1 point, while Factuality fluctuates by up to 3 points for individual systems.
D.2. Cross-Judge Consistency and Prompt Sensitivity
Cross-judge changes substantially shift absolute scores but preserve system rankings, while prompt rephrasing causes only minimal score changes and no rank changes.
- Cross-Judge Consistency: 13 to 17 points: switching from GPT to Gemini inflates absolute scores but preserves the relative ranking of all six systems.Gemini’s inflation is largest for Process and smallest for Factuality.
- Cross-Judge Consistency: ∆Rank = 0: Gemini preserves every system’s rank despite systematic score inflation.Overall ranking agreement reaches Kendall’s τ = 1.0.
- Prompt Sensitivity: −0.5 to −1.6: prompt modification changes Overall scores only slightly in the multimodal setting.Most per-dimension shifts remain below 1 point.
- Prompt Sensitivity: Less than 2 points: prompt variation produces no rank changes, supporting stable comparative conclusions.The robustness table reports original, alternative, and delta scores for both experiments.
D.3. Human Study
Human rankings strongly agree with MiroEval rankings, matching the same top three systems and differing by no more than two positions.
- Human Study: Kendall’s τ = 0.91 and Spearman’s ρ = 0.95: human and MiroEval rankings show strong agreement.Five annotators ranked systems using both final reports and associated research processes.
- Human Study: Top three match exactly: MiroThinker-H1, OpenAI Deep Research, and MiroThinker-1.7 lead both human and MiroEval rankings.The largest rank shift is only 2 positions for Qwen-3.5-Plus.
- Human Study: 2 positions: the maximum difference between human and MiroEval rankings is small.This result contributes to the broader robustness summary alongside run and judge-model analyses.
E.1. Synthesis Evaluation
Adaptive synthesis evaluation generates task-specific criteria that expose multimodal grounding failures, including fabricated facts and unsupported nutrient values, beyond fixed scoring dimensions.
- Case 1: Software Vendors: The vendor attachment contains only names and ranks, so the evaluation checks exact roster use and prohibits unsupported inference.The graphic provides no growth rates, categories, business models, or financial metrics.
- Case 1: Software Vendors: Three task-specific grounding dimensions extend the four fixed dimensions in the software-vendor case.They assess attachment-grounded scope, evidence-linked business-model typology, and investment-grade profitability and go-to-market evaluation.
- Case 1: Software Vendors: 0.5: ChatGLM Agent scores poorly on scope and methodology integrity after fabricating growth rates and financial figures absent from the graphic.MiroThinker-H1 explicitly lists the correct top-20 roster and receives 9.8 for grounding coverage.
- Overall Observation: Adaptive criteria catch fabricated growth rates and invented nutrient values that fixed rubrics evaluating general accuracy would miss.Key-fact extraction converts attachment contents and omissions into precise evaluation checkpoints.
- Case 2: Senior Cat Nutrition: Three task-specific dimensions target SKU provenance, clinically normalized suitability, and uncertainty-aware data gaps in the senior-cat nutrition case.Most product images lack visible nutritional panels, making explicit missing-data handling central to evaluation.
- Case 2: Senior Cat Nutrition: 0.8: MiniMax-M2.5 is penalized for asserting nutrient values where product labels show no nutritional data.Gemini-3.1-Pro also struggles with region-matched sourcing, scoring 3.4 for SKU-verified provenance.
E.2. Factuality Evaluation
Agentic factuality evaluation verifies claims across attachments and web sources, while case studies show failures involving temporal ambiguity, magnitude misrepresentation, and incorrect timing.
- Framework: Agentic verification retrieves and reasons over evidence from multiple sources, including multimodal attachments, rather than relying on report text alone.The framework decomposes a report into verifiable statements for evaluation.
- Case Studies: Below $40: the oil-price claim is wrong for H2 2014 because EIA data place end-2014 prices near $62 for Brent and $59 for WTI.Sub-$40 prices were reached only in early 2016.
- Case Studies: 77,835 ÷ 858,833 ≈ 9.07%: the Huawei figures are verified by both the uploaded screenshot and the annual report.The arithmetic independently confirms the stated 9.1% operating margin.
- Case Studies: 350 kWh versus 432 kWh: conflicting credible sources require a Conflict verdict rather than definitive verification.The discrepancy may reflect different model variants or updates, but cannot be resolved from the cited evidence.
- Case Studies: 1979 is valid under a broader release definition, so the agent’s 1980-only verdict fails to reconcile different release events.The retrieved source emphasized publication in 1980, while the game was available as a 1979 pack-in title.
- Case Studies: Nearly 400% versus over 200%: logical entailment is insufficient when a lower-bound statement materially understates the reported magnitude.The case identifies false positives caused by treating “over X%” as factually correct solely because the inequality holds.