PaperlayerView exact revision on arXiv

Source-linked AI summary

DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

Janghoon Han, Heegyu Kim, Changho Lee, Dahm Lee, Min Hyung Park, Hosung Song, Stanley Jungkyu Choi, Moontae Lee, Honglak Lee

arXiv:2512.17776v5cs.CL

TL;DR

Evaluating expert-level deep research reports is difficult because quality criteria can be underspecified, LLM judges may miss domain-dependent errors, and source checks often cover only cited claims. DEER combines expert-defined rubric scoring with verification of cited and uncited claims across reports. Results show strong structure/style and information use, but continued difficulty meeting expert requirements and producing analytically sound analyses.

  • Problem

    Existing evaluations often lack fine-grained expert-defined criteria, LLM judges may miss issues requiring domain expertise, and source verification may omit uncited claims.

    Source: §1 · p. 2

  • Method

    DEER combines a taxonomy operationalized as 101 fixed rubrics and task-specific expert guidance with verification of cited and uncited claims across reports.

    Source: §1 · p. 2

  • Results

    Deep research systems perform well on structure/style and information use but remain limited in meeting expert-level requirements and producing analytically sound analyses.

    Source: §7 · p. 9

  • Takeaways & Limitations

    DEER’s taxonomy, fixed rubrics, and quantitative metrics support fine-grained diagnosis of system strengths and limitations.

    Source: §7 · p. 9

    LLM-based judges may exhibit biases relative to human experts, although validation shows high correlation with human judgment.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in large language models have enabled deep research systems that generate expert-level reports through multi-step reasoning and evidence-based synthesis. However, evaluating such reports remains challenging: report quality is multifaceted, making it difficult to determine what to assess and which criteria to use; LLM-based judges may miss errors that require domain expertise to identify; and because deep research relies on retrieved evidence, report-wide claim verification is also necessary. To address these issues, we propose DEER, a benchmark for evaluating expert-level deep research reports. DEER systematizes evaluation criteria with an expert-developed taxonomy (7 dimensions, 25 subdimensions) operationalized as 101 fine-grained rubric items. We also provide task-specific Expert Evaluation Guidance to support LLM-based judging. In addition to rubric-based assessment, we propose a claim verification architecture that verifies both cited and uncited claims and quantifies evidence quality. Experiments show that current systems produce structurally plausible, evidence-citing reports, but still struggle to fully satisfy expert-level user requests and achieve logical completeness. Beyond performance comparisons, DEER makes system strengths and limitations interpretable and provides diagnostic signals for improvement.

1. Introduction

Evaluating deep research reports remains difficult because criteria can be underspecified, LLM judges may miss issues requiring domain expertise, and source checks often cover only cited claims. DEER addresses these gaps with an expert-developed taxonomy, fixed rubrics and guidance, and report-wide claim verification.

1. Introduction

  • Existing evaluations can use coarse or LLM-structured criteria, miss issues requiring domain expertise, and restrict verification to explicitly cited claims.These limitations leave report quality and factual reliability insufficiently examined.Source: §1 · p. 2
  • DEER evaluates reports across 50 report-generation tasks spanning 13 domains, using a taxonomy of 7 dimensions and 25 subdimensions.The taxonomy synthesizes established reporting norms and evaluation criteria through expert consensus.Source: §1 · p. 2
  • DEER’s information-verification module checks cited and uncited claims across reports and quantifies evidence quality.It complements holistic report-quality assessment with claim-level checks against external sources.Source: §1 · p. 2
  • Its 101 fixed rubric items and task-specific expert guidance support consistent and reliable LLM-based report-quality scoring.The rubrics operationalize the taxonomy, while the guidance is authored by domain experts.Source: §1 · p. 2

2. Related Works

Deep research evaluation has progressed from reasoning benchmarks to web-search QA and long-form report benchmarks. However, report evaluation still lacks expert-defined fine-grained criteria and systematic verification across claims throughout a report.

2. Related Works

  • Evaluation progressed from difficult-question reasoning benchmarks to web-search benchmarks and, more recently, long-form report benchmarks.The newer benchmarks assess reports requiring expert-level analysis, reasoning, and interpretation from multiple sources.Source: §2 · p. 2
  • Existing report evaluations often lack expert-defined fine-grained criteria, leaving quality judgments dependent on evaluators’ implicit standards or LLM-generated rubrics.This makes it unclear whether scores reflect report quality or align with expert assessment.Source: §2 · p. 2
  • Source verification is often omitted or limited to some cited claims, rather than systematically covering claims throughout a report.The paper proposes combining expert-systematized report scoring with systematic source verification.Source: §2 · p. 2

3. Data Construction

The dataset construction process uses real-world query patterns and expert-written questions to create report-style prompts, with expert review and revision to preserve scope while avoiding answer-revealing content.

3. Data Construction

  • The authors analyzed 5,842 in-house user queries to derive a topic distribution for the evaluation dataset.They used the taxonomy of Wettig et al. (2025) for topic classification.Source: §3 · p. 3
  • They sampled 50 seed questions from Humanity’s Last Exam across 13 subject domains to match the target topic distribution.HLE was selected for its expert-written, high-difficulty, multidisciplinary questions.Source: §3 · p. 3
  • Domain experts reviewed the original questions, answers, and rationales to identify underlying concepts and reformulate the items as report-style prompts.Each expert held at least a master’s degree or equivalent expertise in the relevant field.Source: §3 · p. 3
  • Experts cross-reviewed and iteratively revised prompts, removing answer-revealing elements while retaining high-level scope and comparative directions.This reformulation shifts the task from short answers to reports requiring expert-level analysis, reasoning, and interpretation.Source: §3 · p. 3

4. Approach

DEER combines holistic report-quality assessment with claim-level evidence verification, aggregating both into seven dimension-level scores and an overall report score.

4.1. Overview

  • The hybrid framework combines expert-guided rubric assessment of report quality with metric-based verification of external evidence, then aggregates both into seven dimension scores and an overall score.Report quality uses holistic, document-level judgment; external-information dimensions use claim-level verification.Source: §4.1 · p. 4

4.2. Deep Research Report Evaluation Taxonomy

  • The taxonomy synthesizes standards from 80 sources across 20 domains through expert consensus and review, yielding 7 dimensions and 25 subdimensions.Ten experts from diverse domains independently reviewed the taxonomy.Source: §4.2 · p. 4

4.3. Expert-Guided Report Quality Assessment

  • Experts operationalize the taxonomy as 46 criteria and 101 fixed rubric items covering both required-content coverage and execution quality across 50 tasks.The same rubric is applied to all reports to support interpretable diagnosis of system strengths and weaknesses.Source: §4.3 · p. 4
  • Task-specific Expert Evaluation Guidance gives LLM judges concrete, verifiable expectations, drafted and cross-reviewed by domain experts to address subtle domain-specific errors or omissions.The guidance complements the fixed rubric with task-specific expert knowledge.Source: §4.3 · pp. 4–5

4.4. Information Verification

  • The module checks linked evidence against each verifiable claim and aggregates claim-level outcomes into quantitative Information Integrity and Sufficiency metrics.Reported metrics include Claim Factuality, Citation Support, Evidence Coverage, and Reference Reliability/Diversity.Source: §4.4 · pp. 5–6
  • The verification module classifies claims into six types and checks only verifiable Types A–C, including uncited claims whose support can be recovered from elsewhere in the report.Types D–F are structural, internal, or otherwise non-verifiable claims.Source: §4.4 · p. 5
  • Semantic back-tracking identifies preceding sentences a claim depends on and inherits their citations; inherited sources remain candidates until a later check confirms support.The method can recover citations from earlier sentences or sections, expanding verification beyond explicitly cited claims.Source: §4.4 · p. 5

5. Experiment Setup

The experiments use separate LLM judges for report-quality assessment and information verification, and compare model families that vary in reasoning, web search, and research-system orchestration.

5.1. Implementation Details

  • The quality judge receives the task query, report, task-specific Expert Evaluation Guidance, and fixed rubric items, assigning 1–10 scores and rationales that are aggregated by dimension.Information verification separately extracts and type-classifies claims, then verifies verifiable claims against evidence documents retrieved from cited URLs.Source: §5.1 · p. 6
  • Report quality is scored by GPT-5.2 and information verification by GPT-5-mini, at an average evaluation cost of approximately $0.5–$1.0 per report.The judge backbones were selected using validation performance and cost-efficiency ablations.Source: §5.1 · p. 6

5.2. Baseline Models

  • Baseline families span General LLMs, reasoning-enabled LLMs, reasoning-plus-web-search systems, and Deep Research systems, instantiated with Qwen, Gemini, Claude, and GPT backbones.The families differ in reasoning, web search, and research-system orchestration.Source: §5.2 · p. 6

6. Experiments

Experiments reveal a gap between polished presentation and expert-level fulfillment, while component analyses report human alignment and evaluator reliability benefits from task-specific guidance. Fine-grained and report-wide verification analyses also identify cost-aware settings for claim extraction and checking.

6.1. Main Results

  • Most models score relatively well on structure, format/style, and ethics but lag on request fulfillment and analytical soundness; reasoning improves overall performance, while search boosts information metrics without guaranteeing report-quality gains.Reasoning models without web search outperform think+search and deep on report-quality metrics excluding information-related scores.Source: §6.1 · pp. 6–7

6.2. Fine-grained Analysis

  • Figure 3(a) highlights weak scope and reference counts; Figure 3(b) shows stronger AI, Engineering, and Psychology scores than History and Physics.Across subdimensions, Request Fulfillment and Analytical Soundness are generally low; low ref_amount suggests reliance on few references.Source: Figure 3 · p. 7, §6.2 · p. 7

6.3. Correlation with Human Evaluation

  • Table 3's staged comparison shows high human alignment for Vanilla and +Dimensions, a drop at +Granular Rubrics, and the highest correlation with +Expert Guidance.The study averages correlations across five LLM judges and compares them with human evaluations.Source: Table 3 · p. 7, §6.3 · pp. 7–8

6.4. Inter-evaluator Reliability

  • Table 4 reports that +Expert Guidance has the highest inter-evaluator reliability; reliability drops substantially with +Dimensions and partially recovers with +Granular Rubrics.Vanilla has moderate inter-evaluator reliability, while the granular-rubric setting remains below Vanilla.Source: Table 4 · p. 8, §6.4 · p. 8

6.5. Information Verification Module Evaluation

  • GPT-5.2 reaches 96.29% Recall at batch size 10; GPT-5-mini reaches 92.17% Recall and 68.80 Cls. F1 at roughly one-tenth the cost, so batch size 20 is adopted.Increasing batch size causes only minor degradation in Recall, Cls. F1, and Bin. F1 while reducing cost to about two-thirds.Source: §6.5 · p. 8, Table 5 · p. 8
  • Grouped verification substantially lowers cost while maintaining or improving F1; retrieval lowers cost further with a modest F1 reduction, so group size 20 with retrieval is adopted.Grouped verification combines claims associated with the same evidence URL; retrieval uses claim-relevant chunks rather than the full evidence context.Source: §6.5 · p. 8, Table 6 · p. 8

7. Conclusion

DEER combines expert-defined report-quality criteria with report-wide information verification, and its evaluations diagnose system strengths and limitations beyond aggregate performance.

7. Conclusion

  • DEER assesses deep research reports with a hierarchical taxonomy, 101 fixed rubrics, and task-specific expert guidance for LLM-based judging.The framework is designed for systematic assessment of deep research reports.Source: §7 · p. 9
  • Its information-verification component traces report claims to external sources and quantifies evidence quality for a more complete report-level assessment.Source: §7 · p. 9
  • Experiments find strengths in structure/style and information use but continuing limitations in expert-level fulfillment and analytical soundness.DEER supports fine-grained diagnosis and systematic improvement beyond performance assessment alone.Source: §7 · p. 9

Limitations

The framework uses LLM judges that may be biased relative to human experts, while the authors report high human correlation and a text-report scope.

Limitations

  • LLM-based judges may inherently exhibit biases relative to human experts.Source: Limitations · p. 9
  • The authors report high correlation with human judgment and describe the biases as systematic and manageable.Source: Limitations · p. 9
  • The current benchmark focuses on text-based reports, with multimodal research tasks left for future extensions.The authors say this specialization enables deeper analysis of information integrity and logical coherence.Source: Limitations · p. 9

Impact Statement

DEER supports structured comparison of deep research agents while distinguishing its approach from benchmarks for search, partial abilities, and report quality. Its expert-grounded, shared criteria and report-wide claim checking are designed to make evaluation more interpretable; the framework is intended to complement human oversight.

Impact Statement

  • DEER supports structured comparison of agents and may inform selection and deployment, but is intended to complement rather than replace human oversight.Uncritical use may lead to over-reliance on quantitative metrics without sufficient human judgment.Source: Impact Statement · p. 9

A. Comparison with Deep Research Benchmarks

  • The comparison adds claim fact-checking and interpretability to five existing comparison axes: human-written rubrics, expert-curated tasks, open-ended tasks, non-technical domains, and LLM-as-judge.The table extends and modifies a comparison proposed in ResearchRubrics.Source: A. Comparison with Deep Research… · p. 19

A.1. Benchmark Landscape

  • Existing benchmarks cover search and browsing, component abilities such as expert writing and claim discovery, and report quality, but do not all evaluate overall report quality.Search benchmarks focus on finding information; component-focused benchmarks address abilities such as hypothesis generation and error detection, while report-quality benchmarks assess reports themselves.Source: §A.1 · pp. 19–20

A.2. Key Differences from Prior Work

  • DEER grounds scoring in shared expert norms, checks cited and uncited claims by tracing omitted links to earlier report context, and applies a fixed rubric for item-level diagnosis.This contrasts with approaches that verify only cited claims or rely on task-specific criteria that limit standardized diagnostics.Source: §A.2 · p. 20

B.1. Topic Domain Analysis

This appendix describes DEER’s report-generation task construction and evaluation process, alongside its claim taxonomy and procedures for extracting and verifying claims, including automation and human evaluation.

B.1. Topic Domain Analysis; B.2. HLE Subject Mapping

  • DEER derives 11 topic domains from 5,842 in-house queries, then maps them to Humanity’s Last Exam subjects to select 50 expert-written seed items.The in-house queries were used only as aggregated statistics; raw queries were not released or included.Source: §B.1 · p. 20, §B.2 · p. 21, Table 9 · p. 21

B.3. Conversion from HLE QA to Deep Research Reports

  • Experts reformulate each HLE QA seed into a report-oriented task by removing answer-revealing content and specifying scope, comparisons, or key issues when needed.A second expert from the same field cross-reviews each task for appropriate scope, expertise, direction, and lack of excessive hints.Source: §B.3 · p. 22

B.4. Construction of Expert Evaluation Guidance

  • For all 50 tasks, Expert Evaluation Guidance defines concrete, verifiable mandatory elements aligned with the prompt and reviewed by a second domain expert.Review excludes optional or stylistic preferences as mandatory criteria and checks that prompt writing requirements are represented.Source: §B.4 · pp. 22–23

C.2. Construction Procedure and Evidence Mapping

  • The taxonomy synthesizes cross-disciplinary report standards and expert review into seven dimensions and 25 subdimensions for broadly applicable evaluation.Its criteria were cross-validated by 10 experts across six domains, and only items that passed were retained.Source: §C.1 · p. 23, §C.2 · pp. 24, 26

D.1. Hierarchical Levels; D.2. Criteria and Rubric Items

  • The rubric has four levels: seven evaluation dimensions, 25 subdimensions, 46 assessable criteria, and 101 atomic rubric items.Report-quality dimensions use criteria and rubric items; information-verification dimensions instead use metric-based measurements at the subdimension level.Source: §D.1 · p. 26, §D.2 · p. 27

D.3. Coverage and Quality

  • The 101 rubric items separately score Coverage and Quality, distinguishing omissions from the depth and rigor of included content.There are 66 Coverage items and 35 Quality items, each scored on a 1–10 scale.Source: §D.3 · p. 27

D.4. Scoring and Aggregation; E. Interpretable Evaluation Output

  • Scores aggregate from rubric items through criteria and subdimensions to dimensions, with N/A items excluded; users can drill down from dimensions to rubric items to inspect strengths and weaknesses.Coverage and Quality are computed separately at the rubric-item level before their integration at the criterion level.Source: §D.4 · p. 27, E. Interpretable Evaluation Output · p. 28

F.1. Overview

  • The Information Verification Protocol uses a two-step procedure performed by human evaluators: Claim Extraction followed by Factual Accuracy Evaluation.Source: §F.1 · p. 28
  • The six claim types distinguish whether a claim requires external evidence and where that evidence resides, including implicit evidence requiring semantic back-tracking.The types are Explicit Citation (A), Implicit – Same Section (B), Implicit – Previous Section (C), Structural Recap (D), No Citation Required (E), and Unknown Source (F).Source: §F.1 · p. 28

F.2. Step 1: Claim Extraction and Classification

  • Evaluators extracted and classified 728 core claims from two randomly selected reports, recording evidence positions for claims with explicit or implicit citations.They segmented reports into sentences, resolved pronouns from context, and split sentences containing multiple claims.Source: §F.2 · p. 28

F.3. Step 2: Claim Verification

  • Two domain-qualified evaluators independently checked 100 randomly selected A–C claims for explicit source support, adjudicating disagreements and separately assessing source reliability.Support required direct inclusion of the claim’s core facts with matching contextual meaning; absent or irrelevant evidence was not supported, while access or processing failures were errors. Reliable sources include journals, official statistics, and authoritative institutions; unreliable examples include blogs, social media, and unverified forums.Source: §F.3 · pp. 28–29
  • Claim-support judgments achieved substantial inter-human agreement (Cohen’s Kappa κ = 0.80).The reported agreement supports the reliability of the verification protocol.Source: §F.3 · p. 29

G.1. Overview

  • The Information Verification Module automates sentence-level claim extraction and classification, retrieving external documents when needed and using batch extraction, back-tracking, and relevant-context filtering.These algorithms were applied to achieve cost efficiency and accuracy.Source: §G.1 · p. 29

G.2. Claim Extraction and Classification

The verification pipeline extracts and classifies claims, recovers citations for uncited claims, and checks claims against retrieved evidence. Its reported evaluations quantify coverage, extraction performance, verification trade-offs, and report-level information quality.

G.2. Claim Extraction and Classification

  • Batch extraction processes 20-sentence batches with the full report as context, and is reported to achieve human-level claim-extraction performance.The setup aims to focus extraction on local sentences while retaining whole-report context; evaluation uses paragraph-level semantic recall and classification F1.Source: §G.2 · p. 29
  • Binary verifiability classification achieves an F1 of 79 for GPT-5-mini (Batch 20), separating claims requiring external evidence from those excluded from verification.The binary grouping treats Types A–C as verifiable and Types D–F as non-verifiable.Source: §G.2 · p. 29
  • On correctly classified B/C claims (N = 131), LLM backtracking achieves a Jaccard Index of 0.7070 and Precision of 0.7109, outperforming sliding-window baselines on both.Larger windows raise Recall to 0.93 but lower Precision and Jaccard by including irrelevant citations.Source: §G.2 · p. 30
  • Backtracking raises effective verification coverage from a maximum of 62.5% of verifiable claims with explicit citations to approximately 91%, including implicit claims.The reported figures account for backtracking prediction errors.Source: §G.2 · p. 30

G.3. Claim Verification

  • Context retrieval selects claim-relevant document chunks and restores their original order; this reduces token costs by more than 80% while maintaining or improving verification accuracy.The verifier judges whether retrieved context supports a claim, including its core content and numerical information.Source: §G.3 · p. 30
  • Batch 20, Low Effort, Top-K=2, and OpenAI Embedding costs $0.95 per 1k claims and achieves 77.0% accuracy on Original and 88.5% on Adversarial.The authors selected this configuration as a balanced choice for resource-constrained, high-volume verification; adversarial accuracy exceeds 88% across most tested configurations.Source: §G.3 · pp. 30–31

G.4. Evaluation Metrics

  • Integrity and Sufficiency metrics diagnose evidence use through claim factuality, citation and reference support, reproducibility, reliability, diversity, evidence coverage, and counts of supported claims, citations, and references.The metrics target distinct failure modes rather than relying on a single aggregate score.Source: §G.4 · p. 31, Integrity Metrics · p. 31

1. Normalization (Metric Level)

  • Raw metrics are normalized to a 0–10 scale, then averaged from metrics to criteria and from criteria to dimensions.Ratio-based measures use linear scaling; quantity-based measures use a step function with divisors of 15 for information, 10 for citations, and 4 for references.Source: §1 · p. 32, §2 · p. 32, Sufficiency Metrics · p. 32

I. Human-Correlation Experiment Setup

The human-correlation experiment compares evaluator scores with expert ratings across reports, tasks, and evaluation settings. Backbone selection then assigns different models to report-quality assessment and information verification based on performance and cost.

I. Human-Correlation Experiment Setup

  • The human-correlation study evaluates 45 reports from 15 tasks, with two topic-matched domain experts independently rating each report on a 1–5 scale.The mean of the two expert ratings serves as the report-level human score; each report takes an average of 1.5 hours to evaluate.Source: §I · pp. 32–33
  • Across five evaluator models, the study compares Vanilla, +Dimensions, +Granular Rubrics, and +Expert Guidance settings using Pearson r, Spearman ρ, and pairwise agreement.Correlations are averaged over tasks, while pairwise agreement measures whether model and human preferences match for report pairs within a task.Source: §I · p. 33

J. Judge Backbone Selection

  • GPT-5.2 has the highest performance across all three report-quality alignment metrics, while GPT-5-mini is best or competitive for both verification stages at low cost.The authors therefore select GPT-5.2 for Report Quality Assessment and GPT-5-mini for Information Verification.Source: J. Judge Backbone Selection · p. 33

K. Additional Ablation for Report Quality Assessment

The additional analyses test whether report-quality judgments depend on both rubric components, remain stable across evaluators, and vary with scoring design. The reported results favor the full guidance-and-rubric setting and DEER’s anchor-and-scale design.

K. Additional Ablation for Report Quality Assessment

  • Removing either shared granular rubrics or task-specific Expert Guidance lowers Pearson correlation, Spearman correlation, and pairwise agreement with human judgments.The authors interpret the two components as playing complementary roles in Report Quality Assessment.Source: K. Additional Ablation for Report… · p. 33

L. Ranking Stability Across Evaluator Models

  • +Expert Guidance achieves the highest ranking agreement across evaluator models on Kendall’s W, average pairwise Spearman’s ρ, and average pairwise Kendall’s τ.For Information Verification, ranking agreement across three backbones remains high: Kendall’s W is 0.934 for Integrity and 0.936 for Sufficiency.Source: L. Ranking Stability Across Evaluator… · pp. 33–34

M. Scoring Design Analysis

  • DEER’s 5-band anchor + 10-point scale achieves the highest human alignment across Pearson correlation, Spearman correlation, and pairwise agreement.The comparison includes a 5-band anchor + 5-point scale and a no-anchor + 10-point scale.Source: M. Scoring Design Analysis · p. 34

N. Length Bias Analysis

Across 800 baseline reports, report length has little association with report-quality scores, and most task-level correlations are near zero. The appendix also provides evaluation prompts and rubric examples.

N. Length Bias Analysis

  • Across 800 baseline reports, report length has near-zero Pearson correlation with score (r = −0.02, p = 0.67) and a small Spearman correlation (ρ = 0.10, p < 0.01).Most task-level correlations also fall in [−0.2, 0.2): 36 of 50 tasks for Pearson and 32 of 50 for Spearman.Source: N. Length Bias Analysis · p. 34, Table 24 · p. 42, Table 25 · p. 42

O. Prompts

  • The example rubric operationalizes criteria at four levels—dimension, subdimension, criterion, and Coverage or Quality item.The claim-type definitions distinguish cited and uncited claims, including claims whose evidence is elsewhere in the report, from structural, citation-free, or unknown-source claims.Source: O. Prompts · pp. 37, 39, Table 14
  • The appendix includes a condensed Request Fulfillment evaluator prompt, full claim-extraction and classification and claim-verification prompts, and examples of evaluation criteria and claim types.The Request Fulfillment template retains the prompt structure, Expert Evaluation Guidance priority, Coverage/Quality scoring logic, and output requirements while omitting detailed rubric items, scoring bands, and schema.Source: O. Prompts · pp. 34, 37, 39, Figure 10 · p. 43, Figure 11 · p. 44

Related papers

Loading 2512.17776v5…