Source-linked AI summary
Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports
Yang Yao, Yixu Wang, Yuxuan Zhang, Yi Lu, Tianle Gu, Lingyu Li, Dingyi Zhao, Keming Wu, Haozhe Wang, Ping Nie, Yan Teng, Yingchun Wang
TL;DR
Existing benchmarks do not adequately evaluate the integrated, long-form reporting abilities of Deep Research Agents. Dr. Bench addresses this gap with an expert-curated benchmark and multidimensional framework, and experiments show mainstream DRAs outperform web-search-tool-augmented reasoning models while retaining important design limitations.
Problem
Existing benchmarks focus on short outputs or isolated competencies and lack precise, trustworthy criteria for evaluating integrated long-form DRA reports.
Method
Dr. Bench provides 214 expert-curated tasks across 10 domains with manually constructed reference bundles and metrics for semantic quality, topical focus, and retrieval trustworthiness.
Results
Mainstream DRAs consistently outperform web-search-tool-augmented reasoning models in overall task execution proficiency and report-generation quality.
Takeaways & Limitations
The framework supports rigorous evaluation of report-style DRA outputs while exposing challenges for developing efficient, stable, and interpretable agents.
Takeaways & Limitations
Evaluation revealed unstable invocation behavior and occasionally incoherent non-English sub-queries, impairing retrieval precision and relevance.
Abstract
from arXiv · showhide
As an embodiment of intelligence evolution toward interconnected architectures, Deep Research Agents (DRAs) systematically exhibit the capabilities in task decomposition, cross-source retrieval, multi-stage reasoning, information integration, and structured output, which markedly enhance performance on complex and open-ended tasks. However, existing benchmarks remain deficient in evaluation dimensions, response format, and scoring mechanisms, limiting their effectiveness in assessing such agents. This paper introduces Dr. Bench, a multidimensional evaluation framework tailored to DRAs and long-form report-style responses. The benchmark comprises 214 expert-curated challenging tasks across 10 broad domains, each accompanied by manually constructed reference bundles to support composite evaluation. This framework incorporates metrics for semantic quality, topical focus, and retrieval trustworthiness, enabling a comprehensive evaluation of long reports generated by DRAs. Extensive experimentation confirms the superior performance of mainstream DRAs over web-search-tool-augmented reasoning models, yet reveals considerable scope for further improvement. This study provides a robust foundation for capability assessment, architectural refinement, and paradigm advancement of DRAs.
1. Introduction
Dr. Bench addresses the mismatch between DRAs’ complex report-generation capabilities and benchmarks built mainly for short, isolated outputs. It introduces an expert-designed benchmark and multidimensional evaluation framework, while experiments show mainstream DRAs outperform web-search-tool-augmented reasoning models.
- Existing benchmarks emphasize short answers and isolated reasoning or search skills, limiting evaluation of DRAs’ integrated report-generation capabilities.
- Dr. Bench targets high-difficulty report-style tasks with manually constructed reference bundles for systematic evaluation.
- The benchmark covers 214 challenging entries across 10 domains and includes query-specific and general-report rubrics, trustworthy sources, and focus keywords.
- Its framework jointly evaluates semantic quality, topical focus, and retrieval trustworthiness in long-form reports.
- Mainstream DRAs consistently outperform seven web-search-tool-augmented reasoning models in overall task execution and report-generation quality.
2. Related Works
Prior benchmarks largely assess short, closed-form answers or rely on unstable and incomplete report-evaluation methods. Dr. Bench responds to the resulting need for precise, trustworthy, human-aligned evaluation of long-form reports.
- Existing benchmarks commonly evaluate closed-form queries with short, verifiable answers, rather than open-ended long reports.
- Report-oriented benchmarks have explored reference alignment and retrieval but may depend on static reports, generic rubrics, or incomplete source-credibility assessment.
- Automatically generated rubrics can have limited stability and interpretability, while large benchmark scale increases evaluation costs for expensive DRAs.
- Overall, existing benchmarks lack precise rubrics and trustworthy references for comprehensive evaluation of report-style long-form outputs.
3. DR. BENCH
Dr. Bench is a human-validated benchmark of complex report-oriented tasks paired with modular reference bundles. Its construction and evaluation pipeline combine task-specific criteria, general report quality, source trustworthiness, and topical-focus measures.
- 3.1. Domains: Dr. Bench contains 214 high-complexity entries across 10 diverse domains, designed to challenge task understanding, decomposition, execution, and aggregation.
- 3.2. Reference Bundles: Each entry pairs a report-generating query with QSRs, GRRs, TSLs, FAKs, and FDKs covering distinct DRA capabilities.
- 3.1. Domains: Queries target structured long-form reports across diverse topics, regions, disciplines, and spatiotemporal settings, with standardized report-oriented instructions.
- 3.2.2. Query-Specific Rubrics: QSRs score task completion using expert-built binary or ternary criteria, while GRRs assess general report quality across seven dimensions.
- 3.2. Reference Modules: TSLs identify authoritative primary sources for trustworthy retrieval, whereas FAKs and FDKs measure topical coverage and thematic drift.
- 3.3. Construction Pipeline: A multi-stage process combines expert design, LLM auditing, manual review, and cross-validation to improve benchmark accuracy, consistency, reproducibility, and reduce annotation bias.
4. Evaluation Framework
The framework evaluates long-form DRA reports across semantic quality, topical focus, retrieval trustworthiness, and efficiency-related metadata. It combines these signals through explicit scoring procedures designed for scalable, interpretable assessment of structured long-text generation.
- 4. Evaluation Framework: The framework combines semantic quality, topical focus, and retrieval trustworthiness in a multidimensional evaluation for report-style generation.Its design emphasizes transparency, interpretability, practical utility, scalability, and transferability to structured long-text tasks.
- 4.1. Semantic Quality: Semantic quality fuses normalized QSR and GRR scores with weighting coefficients α and β, where α + β = 1.QSRs and GRRs provide complementary signals for task completion and general report quality.
- 4.2. Topical Focus: SemanticDrift combines FAK omission and FDK misuse, with higher values indicating weaker topical alignment and consistency.FAKDrift penalizes insufficient core-keyword coverage, while FDKDrift captures thematic distraction; λ and µ determine their relative weights.
- 4.3. Retrieval Trustworthiness: TrustworthyBoost weights exact and hostname-level source matches to increase scoring sensitivity to citation verifiability and source reliability.The coefficient η controls boost magnitude, while θ and κ weight full and hostname matches with θ + κ = 1.
- 4.4. Integrated Scoring Framework: IntegratedScore multiplies Quality, 1 − SemanticDrift, and TrustworthyBoost, thereby penalizing thematic drift while rewarding external support.The integrated metric is described as a multiplicative evaluation model for report-style generation tasks.
- 4.5. Additional Metrics: ContributionPerToken measures information density relative to token expenditure, while RetrievalIndex measures how selectively reports distill adopted annotations from retrieved links.The framework uses these supplementary metrics to characterize efficiency and retrieval filtering under resource constraints.
5. Experiments
Experiments evaluate thirteen models with controlled scoring procedures and show that DRAs generally outperform web-search-tool-augmented reasoning models, while model strengths differ across quality, focus, credibility, and efficiency.
- 5.2. Leaderboard: Qwen ranks first in IntegratedScore, while Sonar leads topical focus and Kimi-K2 leads quality despite weaker credibility and attention.
- 5.2. Leaderboard: GPT-5 achieves the highest citation reliability, whereas o3 and o4-mini use the fewest tokens per report at 23K and 18K respectively.
- 5.2. Leaderboard: Quality contributes strongly to IntegratedScore, but balanced quality, credibility, and attention allow Qwen, Sonar, and o3 to surpass Kimi-K2 overall.
- 5.3. Supplementary Dimensions: GPT-4.1 averages only 0.39 retrieval activities, while o3-dr and o4-mini-dr show more intensive inference and retrieval patterns.
- 5.3. Supplementary Dimensions: o3-dr retrieves slightly more links and annotations, whereas o4-mini-dr has a lower RetrievalIndex, indicating stronger filtering and citation precision.
6. Discussions
The discussion identifies instability in retrieval behavior and incoherent multilingual sub-queries as systemic DRA limitations, framing efficiency–quality and decomposition–coherence as central trade-offs.
- Repeated queries reveal substantial reasoning-time variance and non-convergent retrieval paths in models such as o3 and o4-mini.
- English tasks sometimes produce incoherent non-English sub-queries, impairing retrieval precision and relevance through semantic misalignment.
- DRAs face an efficiency–quality trade-off between high-quality reasoning and excessive token usage and latency.
- Modular query decomposition improves coverage but risks semantic fragmentation and intent drift, creating a decomposition–coherence trade-off.
7. Conclusions
The paper presents Dr. Bench and a multidimensional framework for evaluating report-style DRA outputs across diverse tasks and domains, finding that DRAs outperform conventional tool-augmented models while retaining important limitations.
- Dr. Bench evaluates DRA performance using challenging queries, diverse thematic domains, and high-quality reference bundles.
- The framework measures report-style outputs along semantic quality, topical focus, and retrieval trustworthiness.
- Contemporary DRAs substantially outperform conventional tool-augmented models in complex task scenarios while exposing key limitations and trade-offs.
A. Taxonomy of Domains
The benchmark organizes entries across ten principal thematic domains, with an Unclassified residual category preserving diversity. Its distribution is broadly balanced, although Business & Finance and History & Social Sciences are relatively more represented.
- The taxonomy comprises ten principal domains spanning research, current affairs, sports, education, law, business, technology, environment, history, and health.
- Entries outside the predefined categories are assigned to Unclassified to preserve data quality while maintaining diversity.
- Business & Finance and History & Social Sciences occupy relatively larger shares, while the overall distribution remains broadly balanced.
C. Examples of Entries
The benchmark illustrates its report-oriented design through entries spanning distinct domains and query types, with rubrics, trustworthy sources, and expert scoring procedures supporting structured evaluation.
- Four representative entries span distinct domains and query types, illustrating structural completeness, rubric coverage, and citation fidelity.
- The QUIC example requires milestone enumeration, draft comparisons, technical changes, and authoritative RFC or IETF sources.
- The cat-adoption example uses multiple preparation, health, behavior, and adaptation criteria supported by veterinary and public-health sources.
- Example tasks address COVID-19 and international cooperation, QUIC standardization, stray-cat adoption, and climate-change negotiations.
- Scoring prompts constrain evaluators to predefined rubric values and assess keyword relevance using frequency, depth, thematic importance, and contextual integration.
E. Supplementary Experimental Observations
The evaluation decomposes report quality into dimensions whose contributions to IntegratedScore reveal trade-offs among textual quality, thematic focus, and external-reference credibility.
- Quality contributes to IntegratedScore as a core dimension reflecting textual precision, thematic focus, and control over external referencing.
- Kimi-K2 leads the Quality dimension, but weaker credibility and attention scores reduce the extent of its quality advantage.
E.2. Evaluation Across Domains
The cross-domain evaluation compares models using Quality, SemanticDrift-derived relevance, TrustworthyBoost, and IntegratedScore. QWE, SON, and O3D show stable advantages in several domains, while KIM trades strong Quality for weaker SDR and TBO performance.
- Table 5 evaluates models across domains using QUA, SDR, TBO, and ITS metrics.
- QWE, SON, and O3D achieve higher ITS values across multiple domains, including 41.27, 38.10, and 40.23 in domain 03.
- In domain 10, QWE, SON, and O3D reach ITS values of 39.07, 38.86, and 37.68 respectively.
- Radar charts show strong ITS performance for all top models in domains 03 and 10, with relatively balanced results elsewhere.
- KIM leads QUA but performs worse in SDR and TBO than the other compared models.
F. Comparison with Existing Benchmarks
Dr. Bench is compared with existing benchmarks across task format, evaluation dimensions, construction, scale, target system, and criteria. It combines human-authored expert rubrics with quality, semantic adequacy, and retrieval-credibility assessment for long-form reports.
- Its comparison framework distinguishes task type, evaluation dimensions, construction method, scale, target system, and evaluation criteria.
- Dr. Bench combines quality, semantic adequacy, and retrieval credibility to evaluate report-style outputs.
- The benchmark is human-authored with expert rubrics, anchors, and trustworthy links rather than relying on large-scale auto-generated data.
- Its 214 curated tasks are designed to balance broad coverage with precise, fine-grained assessment of Deep Research Agents.
- Query-specific reference structures assess report completeness according to dimensions tailored to individual tasks.