Source-linked AI summary
Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents
Minghao Guo, Meng Cao, Sui Zhao, Siyu Ning, Xin Wang, Haoze Zhao, Jiaxuan Yang, Haihong Hao, Mingfei Han, Shunlin Rong, Haijun Wu, Xiaodan Liang, Xiaojun Chang
TL;DR
Existing benchmarks rarely test whether agents can sustain long, dependency-heavy multimodal research processes. Mr.LHDR introduces a benchmark and dependency-aware evaluation protocol for such tasks, finding that final-answer accuracy exceeds complete research success and that multimodal, dependency-consistent evidence integration remains difficult.
Problem
Existing benchmarks mainly evaluate medium-horizon exploration and leave sustained research over long, interdependent evidence chains insufficiently tested.
Method
Mr.LHDR evaluates 102 open-web questions across eight categories using multimodal evidence, annotated dependency structures, and OA, SA, CS, and DACS metrics.
Results
43.1% OA and 34.3% SA are achieved by the strongest reported regular model, while image provision increases DACS by 12.6 points.
Takeaways & Limitations
Sustained, dependency-consistent evidence integration is a key bottleneck for current deep research agents, beyond isolated fact retrieval.
Takeaways & Limitations
The metrics evaluate dependency-consistent stated conclusions but do not verify actual retrieval, internal reasoning, or source grounding.
Abstract
from arXiv · showhide
Deep research agents are increasingly capable of web search, tool use, multimodal evidence analysis, and information synthesis. However, existing benchmarks mainly evaluate medium-horizon exploration and rarely test whether agents can sustain long, dependency-heavy research processes. We introduce Mr.LHDR (Multimodal real-world Long-Horizon Deep Research), a benchmark for evaluating real-world deep research over long, irreducible chains of interdependent evidence across eight categories. Each question is constructed from a hidden Node-Relation graph and requires an average of 12.1 necessary intermediate conclusions with a mean dependency depth of 10.4 before reaching a short, unique, and verifiable answer. Questions incorporate multimodal evidence, including images, maps, PDFs, logos, charts, tables, and video frames, with at least one non-text element that changes the reasoning state. Mr.LHDR evaluates both final answers and the correctness of intermediate conclusions under annotated dependencies. We evaluate general models, deep research systems, and agent frameworks using Overall Accuracy (OA), Strict Accuracy (SA), Checklist Score (CS), and Dependency-Aware Checklist Score (DACS). Results show that even the strongest system achieves only 43.1% OA and 34.3% SA, indicating that final-answer accuracy substantially overestimates complete research success. Removing images reduces DACS by 12.6 points, demonstrating the importance of multimodal evidence, while SA consistently declines as reasoning chains become longer. These findings reveal sustained, dependency-consistent evidence integration, rather than isolated fact retrieval, as a key bottleneck for current deep research agents.
1 Introduction
Mr.LHDR targets a gap in existing research benchmarks by testing multimodal investigations that require preserving and satisfying long chains of dependent intermediate conclusions. It introduces a dependency-aware benchmark and evaluation protocol, finding that current systems often achieve correct final answers without complete, dependency-consistent research.
- Design principles: The benchmark requires agents to decompose goals, search heterogeneous sources, inspect visual and document evidence, preserve intermediate conclusions, and verify final answers.Its design treats structural horizon as dependency depth rather than prompt length, response length, or browsing-action count.
- Motivation: Existing benchmarks leave long-horizon multimodal research under-tested because MM-BrowseComp averages only 3.0 checklist items per question.Mr.LHDR instead requires prerequisite-linked evidence structures rather than isolated browsing actions.
- Evaluation: The reported metrics assess dependency-consistent stated conclusions, not whether agents actually retrieved or grounded the evidence.They do not observe internal reasoning or establish that a stated conclusion came from actual retrieval.
- Empirical findings: 43.1% OA and 34.3% SA are achieved by GPT-5.5, while image provision raises DACS from 21.6% to 34.2%.The benchmark also finds that SA decreases monotonically as checklist length increases across representative models.
- Benchmark design: Mr.LHDR contains 102 questions across eight categories, with 1,231 annotated intermediate conclusions and mean dependency depth 10.4.Each question pairs multimodal source evidence with a short, unique, verifiable answer and an author-verified dependency DAG.
- Evaluation: DACS recursively gates credit for stated conclusions on their annotated prerequisites, complementing OA, SA, and CS.The protocol distinguishes final-answer correctness, complete response correctness, intermediate-conclusion coverage, and dependency-consistent coverage.
2 Related Work
Prior work covers web browsing, multimodal understanding, tool use, and research reports, but leaves open whether agents can maintain research state across many dependent steps. Mr.LHDR addresses this gap by combining open-web multimodal tasks with long-horizon dependency structures and response-level prerequisite gating.
- Open evaluation question: Mr.LHDR asks whether agents can maintain objectives, entities, constraints, source support, and multimodal bridge conclusions across many dependent steps.This question differs from evaluating search, browsing, tool invocation, or report synthesis in isolation.
- Benchmark gap: Existing multimodal browsing benchmarks remain concentrated on short- to medium-horizon research, while other process benchmarks use fixed local knowledge bases or score outcomes without intermediate chains.Mr.LHDR is positioned as complementary to these settings.
- Positioning: Unlike prior benchmarks, Mr.LHDR combines open-web search, required real-world multimodal evidence, concise verifiable answers, long-horizon Node-Relation structures, and dependency-aware evaluation.Its horizon measures maintaining research state across prerequisite-linked conclusions rather than producing long outputs or executing a fixed number of actions.
- Evaluation distinction: DACS uses an item-specific DAG and withholds downstream credit when required predecessors are unsatisfied.This differs from unordered checklist coverage and applies to heterogeneous systems using response-level evaluation.
3 Mr.LHDR Benchmark
Mr.LHDR is a multimodal benchmark for real-world, long-horizon research in which hidden dependency structures connect evidence to short, verifiable answers. Its construction and validation enforce irreducible chains, multimodal necessity, source support, and resistance to shortcuts.
- Node-Relation Design: Each item begins from a Node-Relation graph whose verified properties and relations form prerequisite-linked paths through entities, sources, and multimodal evidence.Relations must be natural and uniquely convergent, so removing a necessary relation breaks the chain.
- Node-Relation Design: Long-horizon difficulty comes from preserving an evolving research state across dependent evidence steps, not from verbose prompts, long outputs, or obscure trivia.Earlier conclusions become constraints or entry conditions for later investigation, requiring retention of resolved entities and accumulated constraints.
- Evaluation Design: Checklist-based evaluation complements final-answer scoring because answer correctness alone can hide missing intermediate conclusions and violated prerequisites.The benchmark therefore preserves intermediate conclusions and their dependency structure for post-hoc response analysis.
- Multimodal Evidence: Every item requires non-text evidence that changes the reasoning state by identifying entities, constraining candidates, confirming relations, or extracting information from visual or document sources.Examples include images, maps, PDFs, video screenshots, logos, products, charts, and tables.
- Quality Standards and Validation: Human annotators validate AI-assisted candidates through gates for answer uniqueness, irreducibility, source support, multimodal necessity, shortcut resistance, and metadata consistency before admission.The workflow uses AI for candidate expansion, while annotators prune, revise, audit, and lock only items that pass every gate.
- Benchmark Scope: Mr.LHDR contains 102 evaluation items across eight real-world categories, with Linear and Multi-branch authored templates and multiple reasoning-operation tags.The verified checklist DAG, rather than the authored template label, represents the actual dependencies.
4 Experiments
The experiments evaluate 25 systems on Mr.LHDR using final-answer and dependency-aware metrics, showing that long multimodal research remains difficult. Results also reveal gaps between correct answers and complete stated conclusions, benefits from images, and increasing failure rates deeper in dependency structures.
- 4.1.3 Metrics: OA measures final-answer correctness, whereas SA additionally requires every annotated necessary conclusion to be stated correctly.CS measures checklist coverage without dependencies, while DACS recursively requires prerequisite conclusions for credit.
- 4.2.1 Overall Capability: 43.1% OA and 34.3% SA make GPT-5.5 the strongest reported system, while o3 Deep Research reaches 32.4% OA and 19.6% SA.DeerFlow using Qwen3-VL-235B reaches 15.7% OA and 9.8% SA; these are point estimates rather than statistically separable rankings.
- 4.2.2 Final Answers vs. Dependency-Consistent Stated Conclusions: 43.1% OA falls to 34.3% SA for GPT-5.5, while o4-mini and o3 Deep Research show gaps of 14.7 and 12.8 points, respectively.The gaps indicate that correct final answers can coexist with missing required intermediate conclusions.
- 4.3.1 Image Ablation: 12.6 points is the DACS gain from adding images, rising from 21.6% to 34.2%, while CS rises from 26.9% to 39.3%.The image condition improves evidence-chain metrics, and the direction is stable for OA, CS, and DACS across judges.
- 4.3.2 Checklist Length: SA decreases monotonically as checklist length increases, and all three analyzed models reach their lowest DACS in the 13–15-step range.DACS is less uniform across shorter ranges, but the deepest range is consistently weakest.
- 4.3.4 Failure-Mode Analysis: 69.3% of checklist-step decisions fail in the final dependency-depth third, compared with 48.8% in the first third.The pattern is structural rather than a temporal trace and does not establish that upstream failures caused downstream failures.
5 Limitations and Broader Impact
The authors qualify their findings by limiting framework-level conclusions to one configuration and noting that response metrics do not verify retrieval provenance. They also address dual-use, privacy, contamination, and third-party-site risks through constrained item design and release practices.
- Limitations: Framework results characterize one open-source harness, not agent frameworks generally, and tool-execution failures may be unrelated to reasoning.The authors call for instrumented tool-execution rates and controlled same-model comparisons with and without the framework.
- Limitations: The metrics score submitted responses without distinguishing harness failures from reasoning failures or verifying retrieval provenance.This limits interpretation of process scores as evidence of actual research execution.
- Broader impact: Mr.LHDR uses public works, organizations, and documented public figures while excluding private personal data to reduce dual-use privacy risks.The paper identifies cross-source, cross-modality evidence linking as a capability that could target private individuals.
- Broader impact: The authors withhold hidden reasoning paths and full source chains, encourage cached responses, and recommend respecting robots directives and rate limits.These measures address benchmark contamination and automated load on third-party sites.
6 Conclusion
Mr.LHDR evaluates real-world research responses through short answers and dependency-annotated intermediate conclusions. Results indicate that current systems struggle with dependency-consistent evidence chains, motivating evaluation beyond final-answer correctness and future citation- or trace-grounded verification.
- 6 Conclusion: Mr.LHDR contains 102 real-world items built from hidden Node-Relation graphs, multimodal evidence, and irreducible dependency checklists.The design evaluates both short-answer correctness and whether necessary conclusions are stated under annotated dependencies.
- 6 Conclusion: GPT-5.5 records 43.1% OA but 34.3% SA, while o3 Deep Research records 32.4% OA and 19.6% SA.Wide confidence intervals preclude a complete ranking across the 102 items.
- 6 Conclusion: Across most systems, OA exceeds SA and CS exceeds DACS, indicating missing annotated prerequisites despite correct final answers or isolated downstream facts.The paper interprets these response-level patterns as difficulty maintaining aligned entities, constraints, and multimodal bridge conclusions.
- 6 Conclusion: The authors recommend evaluating final-answer correctness alongside response-level dependency consistency and incorporating citation- or trace-grounded verification.DACS measures stated conclusions under annotated dependencies but does not prove retrieval provenance.
A.2 Evaluation Interface
The evaluation interface launches runs, monitors execution, and audits responses, judge decisions, checklist completion, and aggregate metrics. The broader construction workflow uses multimodal Node-Relation evidence and iterative annotator-guided expansion, stress testing, blurring, and pruning.
- A.2 Evaluation Interface: The evaluation platform records run status, model responses, final-answer judgments, and checklist completion vectors for each item–model pair.It functions as both an execution and audit tool.
- A.2 Evaluation Interface: A Node-Relation item links core entities and searchable properties, with verified properties serving as entry conditions for subsequent nodes.Properties may include photos, maps, PDFs, official documents, logos, charts, timestamps, works, affiliations, venues, or founders.
- A.2 Evaluation Interface: AI assistance expands candidate graphs and checklists, while annotators accept, edit, or discard proposals under task, category, operation, and style constraints.The pipeline is interactive rather than a single generation prompt.
- A.2 Evaluation Interface: Question polishing removes intermediate-node hints, direct-answer cues, and wording that reveals the intended search path while preserving inspectable evidence for necessary conclusions.This stage treats polishing as leakage control rather than stylistic rewriting.
- A.2 Evaluation Interface: The construction process deliberately over-produces 25–28-step candidate checklists before pruning them into retained benchmark chains.Annotators remove background facts, merge redundant conclusions, and preserve necessary bridges and unique convergence.
B Benchmark Statistics and Composition
Mr.LHDR comprises 102 items with dense dependency structures, multimodal evidence, and broad category coverage. Its statistics distinguish sequential dependency depth from total checklist length and document substantial non-text and source diversity.
- B Benchmark Statistics and Composition: The benchmark contains 1,231 necessary intermediate conclusions across 102 items, with mean checklist length 12.1, median 11, and range 10–21 steps.Author-verified DAGs and explicit evidence fields support verification at both final-answer and process levels.
- B Benchmark Statistics and Composition: The dataset spans all eight target categories, led by media with 21 items and geography with 17, while sports is smallest with 9.The remaining categories contain between 10 and 13 items.
- B Benchmark Statistics and Composition: Operation tags are multi-label: Symbolic appears in 81 items, Constraint in 76, Numerical in 30, and Temporal in 18.Because tags overlap, their percentages do not sum to 100%.
- B Benchmark Statistics and Composition: Dependency depth measures the longest sequential prerequisite path, whereas checklist length counts all necessary conclusions, including parallel branches.No item has fewer than 10 checklist steps; 69 items fall in the 10–12-step bucket.
- B Benchmark Statistics and Composition: Evaluation was conducted from April 30 to May 5, 2026, using distinct routes for regular models, image-only models, and dedicated deep research systems.The configuration is summarized in Table 8.
D.1 Interpretation of CS and DACS
CS measures checklist coverage but ignores prerequisite structure, whereas DACS evaluates dependency-consistent conclusions; both rely on model-produced judge verdicts and fixed evaluation procedures.
- CS can overestimate process reliability because it treats checklist items as unordered, while DACS requires prerequisite conclusions for downstream credit.DACS preserves credit for independent branches but withholds merge-node credit when required dependencies are unsatisfied.
- The evaluation uses a fixed 102-item denominator for all reported metrics.
- The automatic judge reads the question, reference answer, reference checklist, and model response, then emits machine-readable correctness fields used by the metric script.
- Checklist completion requires complete conclusions tied to the correct entity and constraint; vague mentions, missing qualifiers, and search descriptions receive no credit.
- Bootstrap intervals quantify uncertainty around metric estimates, while paired penalties distinguish answer completeness from dependency-related checklist credit.
E.2 Complete Judge Sampling Frame and Agreement Analysis
The agreement analysis re-scores a family-balanced subset with multiple judges and finds high final-answer agreement, while the reported judge is comparatively lenient and its absolute process scores require caution.
- E.2 Complete Judge Sampling Frame and Agreement Analysis: The family-balanced grid covers all 102 items and eight categories using one recent flagship per model family.
- E.2 Complete Judge Sampling Frame and Agreement Analysis: 95.9%–99.7% pairwise final-answer agreement and κ=0.845–0.989 indicate high agreement across same-vendor and cross-vendor judge pairs.Fleiss’ κ is 0.940 for final answers and 0.803 for individual checklist steps.
- E.2 Complete Judge Sampling Frame and Agreement Analysis: The reported judge matches the pooled reference on 97.1% of final answers and 86.6% of checklist steps.
- E.2.2 Judge Overlap with an Evaluated System: The reported judge is the most lenient of 12, marking 16.5% of final answers and 50.9% of checklist steps correct, so absolute process scores lie at the pool’s lenient end.Comparative directions remain stable across judges.
- E.2.2 Judge Overlap with an Evaluated System: Image-related gains are positive under every judge, and the self-judge’s 12.6-point DACS gain is below the 15.0-point cross-family median.
- E.2 Complete Judge Sampling Frame and Agreement Analysis: Template splits are diagnostic rather than standalone difficulty rankings because topology, category mix, source availability, and multimodal bridge positions differ.
- E.2 Complete Judge Sampling Frame and Agreement Analysis: The 19+ checklist-length bucket contains only four items, making its estimates highly sensitive to individual cases.
F.6 Failure-Mode Analysis
The failure-mode analysis labels unsupported checklist conclusions across all evaluated systems and finds that more than half of required intermediate conclusions are unsupported.
- Unsupported checklist conclusions receive diagnostic labels from the MM-BrowseComp failure taxonomy using frozen verdicts from a specified judge.A separate model assigns the taxonomy label to each failed checklist statement.
- 56.6% of required intermediate conclusions are unsupported, and 88.1% of model–item runs contain at least one unsupported conclusion.The figures are step-weighted micro-averages and are not directly comparable with per-system CS values.
- The analysis covers 30,775 checklist-step decisions from 2,550 model–item runs across all 25 systems.
- The appendix reports complete failure-mode counts and conditional distributions by model group.
G Case Studies
The case studies show how dependency-aware checklist evaluation distinguishes complete multi-branch solutions from errors that propagate through long reasoning chains and multimodal steps.
- Case studies: A late-chain propagation failure and an early visual failure illustrate distinct ways final-answer behavior can diverge from checklist-level correctness.The audited cases are explicitly presented as examples of different research behaviors and downstream evidence failures.
- Case studies: A multi-branch success case requires completing every checklist item and combining conclusions from two evidence branches.
- Case studies: The Penguin Books case reaches the correct 337-year answer by chaining publisher, library, tower, designer, and graduation evidence.The calculation uses the Tower of the Five Orders’ 1619 completion year and Alan Fletcher’s 1956 graduation year.
- Case studies: The smartphone case exposes an early visual-grounding failure: the response identifies the wrong phone and aircraft chain despite reaching the final word “climate.”The checklist marks the initial product, processor, aircraft, and video conclusions incorrect while the response answers “climate.”
H.1 Released Artifacts and Reproducibility
The released artifacts support reproducible Mr.LHDR evaluation by exposing benchmark inputs, scoring components, configurations, and tests for dependency-aware aggregation.
- Released artifacts and reproducibility: The release includes benchmark data, evaluator code, judge prompt, verified dependency graphs, and regression tests for recomputing OA, SA, CS, and DACS.The data include questions, answers, labels, checklists, source URLs, and image assets.
- Released artifacts and reproducibility: Recorded model identifiers, inference settings, browsing configurations, and evaluation outputs support consistent reproduction and extension to new models.Regression tests help detect implementation changes that would alter reported results.
- Released artifacts and reproducibility: The benchmark uses publicly accessible web content, retains source URLs for inspection and attribution, and screens items for legitimate informational use.