Source-linked AI summary
DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation
Qianqian Xie, Qingheng Xiong, He Zhu, Tiantian Xia, Xueming Han, Fanyu Meng, Jiakai Wang, Zhiqi Bai, Chengkang Jiang, Zhaohui Wang, Yubin Guo, Yuqing Wen, Jiayang Mao, Zijie Zhang, Shihao Li, Yanghai Wang, Yuxiang Ren, Junlan Feng, Jiaheng Liu
TL;DR
Deep research evaluation lacks realistic, reproducible environments for multimodal, noisy, and ambiguous research tasks. DR3-Eval addresses this gap with authentic user materials, controlled per-task sandboxes, and multidimensional report evaluation. Experiments show that the benchmark is highly challenging and reveals systematic failure modes in current language models.
Problem
Deep research agents must operate over implicit intent, noisy information, multimodal materials, and long-form reports, while existing evaluations leave a realism–reproducibility gap.
Method
DR3-Eval combines authentic multimodal user files, independent static sandbox corpora, reverse-constructed tasks, and multidimensional metrics for evidence and report evaluation.
Results
DR3-Eval poses substantial challenges for state-of-the-art language models and reveals systematic failure modes in retrieval and report generation.
Takeaways & Limitations
The benchmark provides a realistic, reproducible, and verifiable testbed for assessing long-horizon deep-research capabilities and diagnosing model strengths and limitations.
Takeaways & Limitations
The work acknowledges risks including convincing hallucinations and malicious misuse of automated information gathering.
Abstract
from arXiv · showhide
Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR$^{3}$-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR$^{3}$-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR$^{3}$-Agent based on multiple state-of-the-art language models demonstrate that DR$^{3}$-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.
1 Introduction
Deep research agents must handle long-horizon, multimodal research under noisy and changing information conditions, but existing evaluations leave a realism–reproducibility gap. DR3-Eval addresses this gap with a controlled multimodal benchmark, a sandbox-based construction pipeline, and diagnostic evaluation.
- Deep research agents perform planning, iterative retrieval, multimodal understanding, and citation-grounded report synthesis beyond traditional question answering.
- Current evaluation settings must balance realistic research complexity, reproducibility, and evaluability while handling implicit intent, noisy evidence, and hallucination risks.
- DR3-Agent is a multi-agent system adapted to the closed-world benchmark and accepts text queries with multimodal files, including text, images, video, and audio.
- Experiments across state-of-the-art language models show that DR3-Eval is highly challenging and exposes failure modes obscured by existing benchmarks.
- DR3-Eval evaluates report generation using multimodal user files and a controlled sandbox corpus across diverse domains.
- Its task-construction pipeline balances real-world complexity with verifiable evaluation through a controlled sandbox.
2 Related Work
DR3-Eval is constructed to make deep-research evaluation realistic yet reproducible by grounding tasks in authentic multimodal materials and controlled, verifiable search environments. Its curation and quality-control process yields a diverse, low-ambiguity benchmark with substantial information noise.
- Deep-research benchmarks have progressed from QA and tool-use toward complex open-web information seeking, but current environments face a reproducibility–realism tension.
- The dataset construction process uses five stages to ground tasks in real-world needs, control construction, and enable precise evaluation.
- Volunteers supplied 100 intrinsically relevant multimodal document sets spanning text, structured data, static visuals, and dynamic media.
- A divergent-convergent process broadly explores source-file aspects before constructing solution paths and distractors from the resulting search terms.
- Each task receives an independent static sandbox corpus containing necessary evidence, distractor pages, and noise pages to vary signal-to-noise conditions.
- Queries are reverse-constructed from predetermined evidential documents, while quality control checks implicit guidance, synthesis necessity, and interpretative unambiguity.
- 35.7% of 280 candidate tasks passed filtering, producing 100 final tasks after rejecting ambiguity, non-unique solution paths, or insufficient factual difficulty.The benchmark is evenly split between English and Chinese samples and covers technology, economy, and humanities across 13 atomic domains.
- The benchmark combines multimodal inputs with substantial scale, averaging 2.24 user files and 465.5 sandbox web pages per task under the 512k-token configuration.The corpus includes documents, images, videos, spreadsheets, audio, and HTML files, with 68% of tasks being multimodal.
3 DR3-Agent
DR3-Agent is an LLM-driven multi-agent system designed for DR3-Eval’s offline sandbox and multimodal, cross-file research tasks. It uses specialized retrieval and report-evaluation components alongside five complementary metrics covering evidence gathering and report quality.
- System Overview: DR3-Agent is built on MiroFlow to handle DR3-Eval’s offline closed sandbox and cross-reading of multimodal files.The system addresses capabilities that existing open-source deep research frameworks typically cannot directly support in this setting.
- Information Acquisition: Two dedicated sub-agents reduce the main agent’s context burden by returning highly condensed summaries without sharing global state.The RAG search sub-agent interacts with the static sandbox corpus through iterative dense retrieval.
- Evaluation Framework: DR3-Eval measures Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality across information seeking and report generation.Four metrics requiring semantic assessment use Φ (GPT-5.1) as the evaluator.
- Information Seeking: Information Recall measures the ratio of fully covered insights extracted from user files and the sandbox corpus.Gemini-2.5-Flash extracts the insight sets, manual verification checks them, and evaluator model Φ scores report coverage.
- Information Seeking: Citation Coverage evaluates whether generated reports cite the user files and supportive web pages strictly necessary for the query.The metric reflects macroscopic information-gathering recall and research-oriented retrieval ability.
- Report Generation: Factual Accuracy tests whether sources support report claims, while Instruction Following checks task requirements and Depth Quality judges analytical substance and logical rigor.These report-generation metrics respectively use claim-source entailment, a verified checklist, and an expert rubric conditioned on the query.
4 Experiments
Experiments show that DR3-Eval is challenging, with performance varying across models, domains, corpus sizes, and retrieval settings. The benchmark and evaluation framework also show reproducibility, judge alignment, and distinct advantages for handling fragmented multimodal evidence.
- Main Results: Claude Sonnet 4 achieves the best overall results, while performance varies substantially across domains and models.GLM 4.7 performs best on Industry domains, whereas Claude Sonnet 4 performs best on Physics domains.
- Corpus Analysis: As the sandbox corpus grows from 64k to 512k tokens, performance generally declines because larger contexts introduce more noise and irrelevant information.Overall performance, IRSC, and CC decrease as corpus size increases, while FA remains relatively stable.
- Sandbox Validity: DR3-Eval’s sandbox corpus produces performance close to real-time web search, with particularly high consistency in Citation Coverage.The comparison uses Qwen3-235B and Gemini-2.5-Pro on an English subset.
- Framework Comparison: DR3-Agent integrates fragmented user-file evidence more stably than the compared framework and continues following instructions under information overload.With the same retrieval component, both frameworks show converging basic acquisition on continuous long texts.
- Sandbox Analysis: Removing distractor web pages significantly improves performance, confirming that distractors increase task difficulty.The sandbox-corpus component analysis is conducted on 20 tasks using primarily a 128k-token corpus.
- Evaluation Reliability: Automated evaluation aligns strongly with human judgments, while factual-accuracy claim extraction reaches Precision 0.924 and Recall 0.960.The human correlation study uses 50 reports independently reviewed by four experts.
- Judge Analysis: Alternative judge models produce rankings nearly identical to GPT-5.1, with a mean Spearman’s ρ of 0.924.Claude Sonnet 4, Gemini-2.5-Pro, and Qwen-Max are evaluated as alternatives to GPT-5.1.
- Retrieval Analysis: Increasing Agentic-RAG iteration limits improves overall performance, particularly Information Recall and Citation Coverage.The study compares maximum iteration settings of 1, 3, 5, and 7 turns; retriever comparisons use a 128k-token corpus.
5 Conclusion
DR3-Eval addresses limitations in deep-research-agent evaluation by combining authentic research scenarios with controlled, web-like sandbox environments. Experiments show that the benchmark challenges state-of-the-art LLMs and reveals systematic failure modes.
- DR3-Eval grounds tasks in authentic user research scenarios and constructs controlled sandbox environments to provide a principled evaluation testbed.Reverse task construction is used to reduce evaluation ambiguity.
- Experiments show that DR3-Eval poses substantial challenges for state-of-the-art LLMs and exposes systematic failure modes.
Impact Statements
The paper describes privacy protections, societal risks, and environmental considerations associated with deep-research-agent benchmarking. It also uses static corpus visualization and file-format analysis to characterize the benchmark materials.
- Data Privacy and Human Subjects: The dataset uses automated redaction and manual cross-validation to remove personally identifiable and sensitive proprietary information.Participants were compensated for their contributions.
- Societal Implications: The authors identify convincing hallucinations and malicious information gathering as risks of autonomous research agents.The benchmark emphasizes Factual Accuracy, Citation Coverage, and reproducibility to support safer and more verifiable systems.
- Environmental Impact: Using a static sandbox corpus instead of repeated live-web crawling is presented as a more computationally efficient and environmentally sustainable benchmarking practice.
- Benchmark Materials: Figure 9 breaks down the specific file formats represented in the document and image categories.
- Sandbox Visualization: Figure 10 visualizes a representative query and sandbox web pages in two-dimensional t-SNE embedding space, including selected distractor and noise pages.Embeddings are generated with OpenAI’s text-embedding-3-large model.
C Model Rankings across Different Judge LLMs
Table 7 compares leaderboard rankings from GPT-5.1 with rankings from Gemini-2.5-Pro and Qwen-Max. The comparison quantifies evaluator consistency through rank swaps relative to GPT-5.1.
- Table 7 compares rankings assigned by GPT-5.1, Gemini-2.5-Pro, and Qwen-Max, with disagreement measured as rank swaps relative to GPT-5.1.
D Human Evaluation
The section presents human-evaluation materials and a representative multimodal task involving user-provided images and video.
- The evaluation materials include an equation defining PAR through pairwise agreement between automated and human total scores.
- Table 8 compares LLM-Judge and Human-Judge scores across dimensions for the first five cases.
- Figure 11 displays the visual components of the user files supplied for the case study.
- The representative task asks for a sourced account of China’s high-speed rail development using a video and two network maps.
The Remarkable Story of China’s High-Speed Rail Revolution: From Zero to Global Leader Executive Summary
China’s high-speed rail system grew from early planning and foreign technology acquisition into a vast national network and global technology presence. The report contrasts China’s scale-oriented approach with Japan’s pioneering Shinkansen and discusses planning and future technologies.
- Evolution: In 2008, the Beijing–Tianjin inter-city railway opened at 350 kilometers per hour, marking China’s high-speed railway era.
- China–Japan comparison: China’s operating high-speed railway exceeds 25,000 kilometers, accounting for two thirds of the world’s total.
- Network planning: China’s network connects major cities across diverse regions, with dense connectivity particularly in eastern and central areas.
- Network planning: Railway planning covers alignment, capacity, logistics, maintenance, safety, signalling, communications, access, and environmental control.
- Evolution: China’s development progressed through planning, technology acquisition, rapid expansion, and emergence as a global technology exporter.The report identifies early foundations, strategic acquisition, the 2008–2020 breakthrough era, and subsequent export activity.
F.2 Citation Coverage
This section covers citation, factual, recall, format, and depth-related evaluation materials, including a qualitative assessment of a representative report. The assessment finds strong organization and coverage but limited critical depth in planning, comparison, and future technology discussion.
- Evaluation dimensions: The evaluation includes dedicated tables for Citation Coverage, Factual Accuracy, Information Recall, and Format Compliance.
- Qualitative assessment: The representative report is described as well-structured, explaining technology transfer, comparing China with Japan, and discussing planning and future technologies.
- Depth limitations: The planning discussion relies partly on generic railway-planning and IoT material rather than concrete Chinese institutional choices.
- Depth limitations: The Shinkansen comparison lacks deeper analysis of safety records, cost overruns, demand risk, and long-term economic performance.
- Depth limitations: The future-technology section is descriptive and offers limited critical assessment of feasibility, trade-offs, and timelines.
G Inference and Evaluation Cost
The section documents the pipeline’s runtime and API-cost reporting and specifies prompt designs for search-term and query generation. These prompts emphasize concise, natural, broadly searchable outputs that use all relevant files and results without exposing technical keywords.
- Cost reporting: The full pipeline’s approximate runtime and API cost are reported in a dedicated table.
- Search-term generation: Search-term prompts request ten English terms covering multiple topic angles, using common three-to-five-word phrases.
- Search-term generation: Search terms should balance specificity with searchability and use organization names strategically when central to the topic.
- Query generation: Queries must guide retrieval of relevant keywords without directly exposing technical terms or over-describing the background.
- Query generation: The query design requires all user files and useful search results to contribute to a complete answer.
- Query generation: Query prompts require a short, natural 50–100-word question that briefly references the user files.
I Prompts for Evaluation Preparation
The paper defines prompt procedures for extracting verifiable, source-specific insights from user files and sandbox search results, then decomposing research queries into atomic evaluation requirements.
- I.1 Insights Extraction from User Files: Insights Extraction from User Files requires brief, source-specific insights in an English JSON array with each insight linked to its source title.The procedure excludes generic statements and specifies a one- to twelve-word insight format.
- I.1 Insights Extraction from User Files: Each user-file insight should capture a relevant, verifiable method, topic, conclusion, or finding rather than common knowledge or vague improvement claims.Examples favor named technical choices, measurements, people, dates, and methods, while allowing conceptual descriptions without precise numbers.
- I.1 Insights Extraction from User Files: The extraction process targets one or two core insights per source, ensures source coverage, prioritizes concepts over precise numbers, and permits fewer insights when specificity is lacking.These rules are repeated for both user-file and sandbox-corpus extraction.
- I.2 Insights Extraction from Sandbox Corpus: Insights Extraction from Sandbox Corpus applies the same source-specificity, relevance, verifiability, atomicity, and selectivity principles to web-search results.The sandbox instructions reject common knowledge, vague claims, and generic conclusions while retaining specific technical content.
- I.2 Insights Extraction from Sandbox Corpus: Sandbox extraction examples distinguish specific content, such as a named backbone network, from generic or unsupported statements.The instructions explicitly prefer concrete methods and source-specific facts.
- I.3 Checklist Generation: Checklist Generation decomposes complex queries into short, independently verifiable requirements that also capture implicit checkpoints.Requirements should be atomic, one sentence, and no more than ten words.
- I.3 Checklist Generation: Checklist requirements use action verbs and appropriate categories such as content, evidence, analysis, comparison, and conclusion.Comparison requests are split into describing both items and explaining their differences; evidence requests require citations or supporting material.
- I.3 Checklist Generation: Queries typically contain 8-15 atomic requirements, with more checkpoints for greater complexity, and are serialized as structured JSON checklist items.The output schema includes requirement identifiers, text, and categories.
J Prompts for Evaluation
The evaluation prompts operationalize information recall and factual accuracy with explicit semantic-matching rules, ternary recall scores, and lenient but contradiction-sensitive fact checking.
- J Prompts for Evaluation: The evaluation framework uses an LLM-as-a-Judge approach to score model responses.The supplied prompt identifies this approach as the basis for response evaluation.
- J.0.1 Information Recall: Information Recall scores each insight as 1.0, 0.5, or 0.0 according to whether all, some, or none of its core information points appear in the report.The evaluator first extracts key details, searches for exact or semantic matches, and records missing information.
- J.0.1 Information Recall: A full recall score requires clear, complete coverage of the insight’s core information without significant missing details.The guidance emphasizes explicit coverage, unambiguous meaning, and reader-level completeness.
- J.0.1 Information Recall: Partial recall applies when related content is incomplete, vague, generalized, or requires inference, while absent or contradictory content receives zero.The prompt provides examples for missing specific details and wholly absent topics.
- J.0.1 Information Recall: Semantic equivalence allows synonyms, paraphrases, and preserved information expressed differently, but generalization to a broader category can reduce the score to 0.5.Examples distinguish exact algorithm or country references from broader categories such as PnP algorithms or developed countries.
- J.0.1 Information Recall: When uncertain, recall evaluation prefers false positives: reasonable semantic connections score 1.0, weak keyword overlap scores 0.5, and no connection scores 0.0.The prompt explicitly states this priority and requires valid JSON results for every insight.
- J.0.2 Factual Accuracy: Factual Accuracy is judged true when a statement is relevant, semantically similar, inferable, partially supported, or reasonably interpretable from the source.The standard permits generalization, implicit information, and reasonable visual descriptions.
- J.0.2 Factual Accuracy: A factual statement is false only when it directly contradicts the source, lacks relevant support, or cannot be reasonably inferred.The evaluation intentionally allows substantial generalization, wording differences, partial correctness, and ambiguity.