PaperlayerView exact revision on arXiv

Source-linked AI summary

DREAM: Deep Research Evaluation with Agentic Metrics

Elad Ben Avraham, Changhao Li, Ron Dorfman, Roy Ganz, Oren Nuriel, Amir Dudai, Aviad Aberdam, Noah Flynn, Elman Mansimov, Adi Kalyanpur, Ron Litman

arXiv:2602.18940v1cs.AI

TL;DR

Deep research evaluation lacks a single ground truth and can mistake fluent, well-cited reports for high-quality work despite factual, temporal, and reasoning defects. DREAM addresses this through capability-parity evaluation: it combines static metrics with tool-generated adaptive metrics and routes them to suitable evaluators. Controlled evaluations show greater sensitivity to temporal degradation and extrinsic factual errors than existing benchmarks.

  • Problem

    Deep research admits multiple valid trajectories, while existing static and citation-centered evaluators can overlook factual, temporal, and reasoning defects behind fluent, well-cited reports.

  • Method

    DREAM combines query-agnostic static metrics with query-adaptive metrics constructed by a tool-calling agent and executes them through capability-matched evaluators.

  • Results

    DREAM is substantially more sensitive than existing benchmarks to temporal degradation and extrinsic factual errors in controlled evaluations.

  • Takeaways & Limitations

    Agentic evaluation provides temporally aware fact-checking and substantive reasoning validation by giving evaluators retrieval, verification, and reasoning capabilities.

  • Takeaways & Limitations

    DREAM depends on third-party tools, increases computational cost and latency, and evaluates research outputs post hoc rather than intermediate research processes.

Abstract

from arXiv · show

Deep Research Agents generate analyst-grade reports, yet evaluating them remains challenging due to the absence of a single ground truth and the multidimensional nature of research quality. Recent benchmarks propose distinct methodologies, yet they suffer from the Mirage of Synthesis, where strong surface-level fluency and citation alignment can obscure underlying factual and reasoning defects. We characterize this gap by introducing a taxonomy across four verticals that exposes a critical capability mismatch: static evaluators inherently lack the tool-use capabilities required to assess temporal validity and factual correctness. To address this, we propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that instantiates the principle of capability parity by making evaluation itself agentic. DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent, enabling temporally aware coverage, grounded verification, and systematic reasoning probes. Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks, offering a scalable, reference-free evaluation paradigm.

1 Introduction

Deep research reports admit multiple valid trajectories, making quality multidimensional rather than reducible to single-answer correctness. Existing evaluators can reward fluent, well-cited reports despite factual, temporal, and reasoning defects, motivating DREAM’s agentic, capability-parity approach.

  • Deep research questions admit multiple valid reports, requiring evaluation across high-dimensional report quality rather than single-answer correctness.
  • Existing benchmarks often assess writing fluency and citation alignment while remaining insensitive to factual correctness, temporal validity, and substantive reasoning.
  • The Mirage of Synthesis describes surface-level coherence creating an illusion of quality despite underlying factual and reasoning flaws.
  • Current evaluators lack external tools, temporal awareness, and independent retrieval or verification capabilities required to assess the dimensions they measure.
  • DREAM makes evaluation agentic by combining static metrics with tool-calling adaptive metrics in a two-phase Protocol Creation and Execution workflow.
  • Controlled experiments show DREAM is substantially more sensitive than existing benchmarks to temporal degradation and extrinsic factual errors.

2 Deep Research Evaluation Landscape

The paper organizes fragmented deep-research evaluation benchmarks into four quality verticals and finds a systematic imbalance: surface presentation and compliance are assessed more directly than extrinsic truth, temporal validity, and grounded reasoning.

  • 2.1 A Unifying Taxonomy: A unifying taxonomy organizes deep-research evaluation into Presentation Quality, Task Compliance, Analytical Depth, and Source Quality.
  • 2.1 A Unifying Taxonomy: The taxonomy is derived by extracting leaf-level criteria from benchmark code and documentation, embedding them, and clustering them into evaluation dimensions.
  • 2.1 A Unifying Taxonomy: Existing benchmarks are grouped by human-curated, closed-loop LLM-based, or citation-centered workflow-based creation and execution paradigms.
  • 2.2 Diagnosing the Evaluation Landscape: Human-defined criteria are reliable and interpretable but can be dataset-specific, domain-specific, costly to curate, and tightly coupled to manual annotation.
  • 2.2 Diagnosing the Evaluation Landscape: Closed-loop LLM and citation-alignment approaches improve scalability or detect source misrepresentation but remain limited by static knowledge or intrinsic citation verification.
  • 2.2 Diagnosing the Evaluation Landscape: Presentation Quality and Task Compliance receive extensive evaluation, while extrinsic factual correctness, temporal validity, and grounded reasoning remain little directly assessed.

3 DREAM: DRE with Agentic Metrics

DREAM constructs query-specific evaluation protocols from static and adaptive metrics, then routes each metric to an evaluator with matching capabilities. Its design combines fixed quality checks with tool-supported coverage, reasoning, factuality, citation, and source-credibility verification.

  • DREAM constructs a query-specific protocol and executes it with specialized evaluators under the capability-parity principle.
  • 3.1 Phase 1: Protocol Creation: Static Metrics apply uniformly across tasks to assess writing quality, factuality, Citation Integrity, and Domain Authoritativeness.
  • 3.1 Phase 1: Protocol Creation: The Protocol Creation Agent uses retrieval tools to build adaptive Key-Information Coverage and Reasoning Quality metrics for each query.
  • 3.1 Phase 1: Protocol Creation: Key-Information Coverage converts retrieved essential facts into verifiable yes/no questions that flag missing or outdated report content.
  • 3.1 Phase 1: Protocol Creation: Reasoning Quality generates query-specific validation plans that cross-reference report information with independent sources to assess substantive reasoning.
  • 3.2 Phase 2: Protocol Execution: The LLM, Agent, and Workflow Evaluators respectively handle rubric judgment and checklist verification, tool-supported reasoning validation, and factuality, citation, and domain-credibility pipelines.

4 Validation of DREAM

DREAM is validated through human assessment of agent-generated protocols and controlled studies targeting temporal decay, reasoning flaws, factual errors, and reference-free readability.

  • Protocol Quality: Expert and non-expert annotators rated generated KIC and RQ items for relevance, clarity, verifiability, and plan validity.The study used a 1–3 scale normalized to [0, 1].
  • Protocol Quality: KIC scored 0.92 and RQ scored 0.93 under human evaluation, with RQ clarity at 0.97 and plan validity at 0.99.The full agent achieved particularly strong ratings on RQ clarity and plan validity.
  • Temporal Awareness: DREAM–KIC declined from 79.35 for current reports to 44.80 with a January 2025 cutoff and 22.34 with a January 2024 cutoff.DRB–RACE changed little on the same temporal test, including Comprehensiveness from 50.02 to 50.04.
  • Reasoning Flaws: DREAM–RQ centered at approximately 40% degradation for malformed reports, whereas DRB–RACE averaged approximately 9% and sometimes ranked malformed reports higher.The malformed reports preserved fluent structure while containing injected reasoning flaws.
  • Factual Verification: DREAM–Factuality decreased monotonically as the corruption rate increased, while DRB–FACT remained invariant despite citation alignment being preserved.The study paired correct claims with plausible incorrect variants carrying matching citations.
  • Reference-Free Readability: DREAM’s reference-free WQ achieved an average Kendall’s τ of 0.6 against DRB–RACE readability rankings.The reported correlation falls within the 0.5–0.7 range stated for human inter-annotator agreement on subjective readability.

5 Benchmarking Leading DRAs

DREAM benchmarks three open-source deep research agents across diverse datasets, revealing widespread citation-grounding weakness alongside meaningful differences in factuality, synthesis, writing, coverage, and reasoning.

  • Evaluation Setup: DREAM evaluates three open-source systems on English DeepResearch Bench, LiveResearchBench, and ResearchRubrics.The datasets contain 50 English questions, 80 public timely-information queries, and 101 rubric-supported queries, respectively.
  • Citation Integrity: All three agents received critically low Citation Integrity scores, with Smolagents Open DR at 4.78 and Tongyi Deep Research at 1.03.Smolagents and Tongyi primarily lacked claim attribution, while LangChain achieved 15.92 but suffered low citation faithfulness.
  • Content Quality: Smolagents Open DR led aggregate Factuality at 58.15, Writing Quality at 63.97, Key-Information Coverage at 75.95, and Reasoning Quality at 69.16.Its strong content scores coexisted with near-absent citation discipline.
  • Cross-Agent Comparison: Tongyi ranked second in Factuality at 55.09, while LangChain recorded the lowest Factuality at 44.64 but exceeded Tongyi in reasoning with RQ of 57.28.Tongyi’s aggregate RQ was 45.48.
  • Robustness: Absolute agentic-metric scores fluctuated slightly across backbone LLMs, while the reported relative performance remained stable.The sensitivity analysis used DeepSeek-V3.2 and Kimi-K2.5 as alternative protocol backbones.

6 Conclusion

The paper identifies a Mirage of Synthesis caused by static evaluators and proposes capability parity through DREAM’s agentic, tool-equipped evaluation. Controlled studies show improved sensitivity to temporal, factual, and reasoning defects.

  • Conclusion: Existing benchmarks can reward surface fluency and citation alignment while missing defects in factual correctness, temporal validity, and logical reasoning.The paper attributes this failure to an evaluator capability mismatch.
  • Conclusion: DREAM makes evaluation agentic by replacing frozen rubrics with dynamic, tool-equipped protocols for temporal fact-checking and substantive reasoning validation.This instantiates the capability parity principle.
  • Conclusion: Controlled evaluations found that DREAM detects temporal degradation and extrinsic factual errors missed by static benchmarks and surfaces reasoning flaws masked by stylistic coherence.The conclusion presents these capabilities as essential for reliable assessment.

Limitations

DREAM’s agentic evaluation introduces tool, computational, and scope constraints that bound its deployment and leave process-level research evaluation for future work.

  • Limitations: External-tool dependence creates third-party availability and retrieval-bias risks as DREAM prioritizes temporal validity over closed-world consistency.The paper describes this as an inherent trade-off of external verification.
  • Limitations: Multi-step verification and tool-interaction loops increase DREAM’s latency and cost relative to static judges.The paper suggests caching or selective evaluation as possible optimizations.
  • Limitations: DREAM evaluates research outputs post hoc and does not directly assess intermediate processes such as search efficiency or source-discovery dynamics.Process-level telemetry is identified as a future direction.

B Agentic Taxonomy Pipeline

The paper develops an automated, benchmark-agnostic pipeline that extracts evaluation criteria from prior deep-research benchmarks and clusters them into shared dimensions. The resulting taxonomy organizes evaluation practice into recurring verticals whose semantic coherence is supported by embedding visualization.

  • Pipeline construction: The pipeline unifies evaluation practices across multiple deep-research benchmarks by organizing their criteria into a benchmark-agnostic taxonomy.Agents inspect benchmark documentation, prompts, rubrics, and source code to extract leaf-level criteria before clustering them into higher-level dimensions.
  • Pipeline construction: Multiple LLM-based agents independently analyze benchmark repositories and extract leaf-level evaluation criteria with short natural-language descriptions.The analyzed repositories include DeepResearchGym, DeepResearch Bench, ResearchRubrics, and LiveResearchBench.
  • Criteria clustering: A single LLM clusters extracted criteria by semantic similarity while requiring each dimension to draw criteria from multiple benchmarks.This constraint discourages groupings that merely reproduce benchmark-specific provenance.
  • Taxonomy: The taxonomy identifies four recurring verticals: Presentation Quality, Task Compliance, Analytical Depth, and Source Quality.These cover writing and organization, instruction fulfillment, reasoning and synthesis, and citation-supported evidence, respectively.
  • Taxonomy validation: Criteria assigned to the same taxonomy dimension form localized embedding regions, providing qualitative evidence of semantic coherence.The visualization uses Cohere’s embed-english-v3.0 embeddings and UMAP.

C Metrics Details

DREAM combines static metrics for universal report quality with adaptive, tool-assisted metrics for factuality and query-specific coverage. Its metric suite separately evaluates writing, factual correctness, citation integrity, source authority, and essential information coverage.

  • Metric Suite: DREAM’s metric suite comprises static metrics for writing quality, factuality, citation integrity, and domain authoritativeness, plus adaptive metrics for key-information coverage and reasoning quality.The metrics are categorized into universal quality standards and query-specific evaluation dimensions.
  • Static Metrics: Citation Integrity combines Claim Attribution and Citation Faithfulness with a harmonic mean so that missing attribution or unsupported citations lowers the unified score.Claim Attribution measures cited verifiable claims, while Citation Faithfulness checks whether cited source content supports the associated claim.
  • Static Metrics: Domain Authoritativeness scores the credibility of unique cited root domains, distinguishing source reputation from whether a citation supports a specific claim.Domains are deduplicated and rated from Definitive Authority through Low Authority before normalization and averaging.
  • Adaptive Metrics: Key-Information Coverage uses web-retrieved, query-specific essential facts converted into yes/no questions and typically penalizes outdated reports as the knowledge horizon recedes.Across 20 queries, DREAM–KIC generally tracked information lag more strongly than DRB–RACE, although scores fluctuated in two instances.

D.3 Grounding Beyond Citation Faithfulness

The grounding experiment tests whether citation-alignment metrics detect false claims paired with apparently supporting sources. DREAM–Factuality responds to externally false content, while DRB–FACT remains insensitive because it verifies consistency with the supplied URL.

  • D.3 Grounding Beyond Citation Faithfulness: The adversarial dataset contains 15 claim pairs, each contrasting a ground-truth claim with a plausible hallucination paired with a misleading URL.False variants were designed to satisfy standard citation-faithfulness checks while contradicting objective reality.
  • D.3 Grounding Beyond Citation Faithfulness: For corruption level r, batches of 15 claims combine r × N false variants with (1 − r) × N true variants before entering both evaluation pipelines.The synthetic payloads bypass claim-source extraction to isolate metric sensitivity to factual corruption.
  • D.3 Grounding Beyond Citation Faithfulness: DRB–FACT citation accuracy stays near 100% across corruption levels because it checks claims against their provided URLs rather than external reality.The metric recognizes that false claims match their misleading sources but does not detect the underlying misinformation.
  • D.3 Grounding Beyond Citation Faithfulness: DREAM–Factuality exhibits near-linear degradation as corruption increases by independently retrieving evidence beyond the provided citations.This contrast demonstrates why citation consistency alone is insufficient for factual evaluation.
  • D.3 Grounding Beyond Citation Faithfulness: Protocol creation compares direct LLM generation, an agent without external knowledge, and DREAM’s agent with multi-step reasoning and external sources.Human evaluators rate the resulting criteria for relevance, clarity, verifiability, and, for reasoning quality, validation soundness.

E.4 Inter-Annotator Agreement Analysis

Annotators showed statistically significant agreement across all indicators, with stronger concordance for verifiability and validation soundness than for relevance and clarity. Agreement varied because retrieval-free baselines produced similar outputs, whereas retrieval-augmented evaluation was consistently distinguished.

  • Kendall’s W was approximately 0.58 for Relevance and Clarity, indicating moderate annotator agreement on these indicators.
  • Kendall’s W exceeded 0.75 for Verifiability and Validation Soundness, indicating substantial annotator agreement on both indicators.
  • Annotator agreement was statistically significant across all indicators, as reported in the agreement analysis.
  • Agreement variance primarily reflected similar outputs from the Direct LLM and Agent without retrieval baselines, which made consistent ranking difficult.
  • The Agent with retrieval was consistently distinguished from both retrieval-free baselines, supporting reliable human preference for retrieval-augmented protocols.

F Detailed DREAM Benchmarking Results

DREAM provides dataset-agnostic benchmarking across static and adaptive metrics, revealing distinct agent profiles in factuality, citation behavior, report length, and robustness. The results show that strong performance is metric- and dataset-dependent, while citation strategies exhibit distinct failure modes and rankings remain largely stable across judge backbones.

  • Performance Overview: DREAM normalizes scores across heterogeneous tasks and combines static metrics WQ, Factuality, CI, and DA with adaptive metrics KIC and RQ.The evaluation also reports a composite score averaging results across all metrics.
  • Performance Overview: Smolagents Open DR leads in most metrics except CI, although Tongyi Deep Research achieves higher WQ and Factuality on DEEPRESEARCHBENCH.
  • Source Quality Analysis: Smolagents typically has more Full Support factuality judgments, whereas LangChain has higher rates of Unverifiable or Contradict claims.
  • Source Quality Analysis: LangChain shows high citation attribution but low faithfulness, while Smolagents and Tongyi show low attribution with moderately higher faithfulness; neither strategy is reliable.LangChain’s CA is approximately 75–80 and CF approximately 10–20, while Smolagents and Tongyi have CA approximately 5–15 and CF approximately 35–55.
  • Report Length Analysis: Smolagents produces the longest reports at approximately 3,000–3,700 words on average, while LangChain and Tongyi produce approximately 1,400–1,800 words.Tongyi is the most consistent, with a standard deviation of approximately 250–350 words; Smolagents has heavy-tailed distributions and outliers exceeding 14,500 words.
  • Robustness Across Backbone Models: Relative agent rankings remain highly consistent across Claude Sonnet 4.5, DeepSeek-V3.2, and Kimi-K2.5, despite changes in absolute scores.Writing Quality is a minor exception because Claude Sonnet 4.5 did not strongly differentiate among agents.
Loading 2602.18940v1…