How Far Are We from Genuinely Useful Deep Research Agents?

1 Introduction

• Existing benchmarks largely target question-answering or other closed-ended tasks, while open-ended benchmarks can reflect LLM-generated tasks rather than human demands and lack sufficient complexity.These limitations leave practical report requirements such as quality, accuracy, depth, and logical coherence insufficiently captured.Source: §1 · p. 1
• FINDER evaluates DRAs using 100 expert-curated research tasks and 419 checklist items covering report structure, analytical depth, and citation integrity.The checklist is intended to make evaluation more structured and reproducible.Source: §1 · p. 2
• DEFT is a failure taxonomy with 14 fine-grained modes across reasoning, retrieval, and generation, derived through grounded theory from analysis of 1,000 reports.It is presented as the first failure taxonomy developed specifically for deep research agents.Source: §1 · p. 2
• Over 39% of failures arise in content generation, especially strategic fabrication, while retrieval failures such as evidence integration and fact-checking issues account for over 32%.The authors identify evidence verification and reasoning resilience, rather than simple task comprehension, as core challenges.Source: §1 · p. 2

2 Related Works

• Early DRA evaluation used AGI-oriented datasets such as GAIA [33] and HLE [34], while specialized benchmarks [11] [12] [13] [14] [15] still neglected report generation.These assessments use standard answers despite the mismatch with deep research requirements.Source: §2 · p. 2
• Open-ended benchmarks include DeepResearch Bench [16], Mind2Web 2 [6], DeepResearchGym [7], DeepScholar-Bench [8], and enterprise-focused DRBench [9].They cover report quality and effectiveness, time-varying tasks, reproducible search environments, synthesis evaluation, and citation-grounded assessment of long-form reports.Source: §2 · pp. 2–3
• These benchmarks rely on subjective metrics based on author experience or domain knowledge, leaving evaluation without a unified standard.The benchmarks use varying metrics, and dynamic research reports make standardized assessment difficult.Source: §2 · p. 3

3.1 Fine-grained DEepResearch bench (FINDER)

• FINDER refines DeepResearch Bench prompts and adds structured checklists to improve evaluation precision and reproducibility.The source benchmark has 100 PhD-level tasks in Chinese and English and evaluates report quality with RACE and retrieval with FACT, but final-report evaluation alone does not capture agents’ reasoning and information seeking.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.2 Prompt Refinement

• Seven domain experts expanded the benchmark queries with explicit report requirements, and an independent expert assessed the revisions.The added specifications addressed overly brief queries and increased task specification and research complexity.Source: §3.1.2 · p. 3

3.1.3 Checklist Construction

• 419 checklists were created for 100 queries, with three to five checklists per query.Experts used checklists to organize query information and add relevant requirements or constraints; Gemini 2.5 Flash iteratively removed incomplete, ambiguous, or irrelevant items.Source: §3.1.3 · pp. 3–4

3.2 Failure Taxonomy

• DEFT was built through human–AI open, axial, and selective coding grounded in theory, with inter-coder reliability assessment and validation for theoretical saturation.The process was designed to reduce omissions and subjective bias in identifying and interpreting DRA failures.Source: §3.2 · p. 4, §3.2.2 · p. 4, §3.2.3 · p. 5

3.2.1 Open Coding

• Open coding used five LLMs from distinct model families on performance metrics for nine benchmark tasks, and successive refinement rounds yielded 51 concepts.Concept pairs with cosine similarity ≥0.6 were considered for merging, and concepts below a removal threshold were discarded.Source: §3.2.1 · p. 4

3.2.2 Axial Coding

• Axial coding used three rounds of coding and Krippendorff’s Alpha assessments, ultimately producing 14 axial categories.Three domain experts independently annotated random subsets of 24 records in round one or 54 records in later rounds, then discussed discrepancies and refined category definitions.Source: §3.2.2 · pp. 4–5

3.2.3 Selective Coding

• Selective coding distilled the taxonomy into three core categories—Reasoning, Retrieval, and Generation—and no new categories emerged when 36 records from WebThinker and OpenManus were coded.The authors interpret the latter result as evidence of theoretical saturation and category stability.Source: §3.2.3 · p. 5

3.2.4 Positive Taxonomy Metric

• The positive taxonomy score decreases monotonically as category errors increase, reaching |D| with zero errors and approaching 0 as errors rise.The score transforms error counts into a bounded, interpretable performance measure.Source: §3.2.4 · p. 5

4.1 Evaluated Models.

• FINDER evaluates proprietary APIs, open-source models, and agent frameworks, including English and Chinese MiroFlow runs.The MiroFlow language comparison repeats each prompt condition three times; English scores 42.20 versus Chinese 41.28.Source: §4.1 · p. 5, §4.2 · p. 6

4.2 FINDER Performance Analysis

• Gemini 2.5 Pro Deep Research leads RACE (50.95), O3 Deep Research leads FACT factual precision (65.98) and citation reliability (76.58), and MiroFlow-English leads checklist accuracy (72.19).Gemini scores consistently highly across reasoning, retrieval, and generation, while Kimi K2 and O4-Mini show strong reasoning and retrieval but a sharp generation decline. MiroFlow-English's checklist lead contrasts with O3's 57.52.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.3 DRB vs FINDER

• Under RACE, FINDER scores remain largely consistent with DRB, but FACT exposes broader citation declines under FINDER's stricter validation demands.RACE is reference-based and supports reliable within-benchmark model comparisons, not meaningful interpretation of absolute score differences across benchmarks. OpenAI Deep Research modestly improves effective citation, while most other systems decline in citation accuracy and effectiveness.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4 Deep Research Failure Taxonomy (DEFT)

• DEFT’s reasoning and retrieval failures span planning, analytical depth, information acquisition, integration, and verification, with reasoning resilience identified as critical to consistent outcomes.Reasoning failures include requirement understanding (10.55%), analytical depth (11.09%), limited scope (0.90%), and rigid planning (5.60%); retrieval failures include insufficient acquisition (16.30%) and verification failure (8.72%). The authors argue agents need to adapt reasoning to task evolution and feedback, not merely have high reasoning intensity.Source: §4.4 · pp. 8–9
• Generation has the highest failure share, led by strategic content fabrication (18.95%), supporting stronger constraints and verification during report generation.Fabricated terms, methods, or references can create an illusion of academic rigor; the authors also call for controls on organization, language, and formatting.Source: §4.4 · pp. 9–10

4.5 Evaluation of DEFT’s Effectiveness

• Failure frequencies are distributed across reasoning (28.14%), retrieval (33.10%), and generation (38.76%), while human–LLM annotation is reported as strongly reproducible.Four experts assessed WebThinker and OpenManus reports, and Krippendorff’s alpha was used to assess agreement with Gemini 2.5-Flash; coefficients are reported in Table 3.Source: §4.5 · p. 10, Table 3 · p. 10
• DEFT’s correlations reveal linked failure pathways from misunderstood requirements to incomplete reports, poor source integration to disorganized output, and poor retrieval to confident fabrication.The taxonomy also distinguishes concise-but-false reports from verbose, disorganized ones and separates methodological flaws from process compliance; superficial analysis is linked to poor retrieval.Source: §4.5 · p. 10, Figure 6 · p. 11

5 Conclusion

• FINDER combines 419 checklist assessments with DEFT’s 14 failure categories to evaluate deep research agents at task and process levels.The framework is intended to shift evaluation beyond answer accuracy toward reliable, transparent, and verifiable research systems.Source: §5 · p. 11

A.2 Query Examples

• The query examples pair varied research prompts with checklist criteria for analytical mechanisms, methodological critique, balanced coverage, traceable evidence, and format compliance.Examples span health misinformation, AI and relationships, and value-investor philosophies.Source: §A.2 · pp. 16–18

C Taxonomy Case Study

• The taxonomy’s case studies analyze task demands, model performance, and error causes, illustrating failures from misunderstood deliverables to unsupported or fragmented reports.The examples include failures of analytical depth, scope, planning, information acquisition, handling, integration, and source representation.Source: C Taxonomy Case Study · p. 19, Failure to Understand Requirements (FUR) · p. 19, Lack of Analytical Depth (LAD) · p. 19

Failure to Understand Requirements (FUR)

• A model mistook a required ranking of ten quantum research groups for a methodological consultation, omitting the actual rankings and key group-level deliverables.The missed items included group names, institutions, CPI scores, and bottlenecks.Source: Failure to Understand Requirements (FUR) · p. 19

Lack of Analytical Depth (LAD)

• In a power-technology commercialization assessment, models presented formulas and tables without explaining model mechanisms, variable coupling, simulation details, or substantive PESTEL analysis.The reported response reduced a high-dimensional problem to static, one-way causal chains.Source: Lack of Analytical Depth (LAD) · pp. 19–20

Limited Analytical Scope (LAS); Rigid Planning Strategy (RPS)

• A vibration-isolation design response covered only hardware and sensor selection, while a corrosion-inhibitor review failed to adapt when it could not access required bibliometric data.The latter retained its original plan and filled sections with internally generated content rather than revising its goals or disclosing the limitation.Source: Limited Analytical Scope (LAS) · p. 20, Rigid Planning Strategy (RPS) · p. 20

Insufficient External Information Acquisition (IIA); Information Handling Deficiency (IHD)

• The review cited mostly sources published in 2015 or earlier; its market model assumed approximately six annual carbon-strip replacements (one per 100,000 km) from HSR data but did not integrate parameters from a relevant 2023 Guangzhou paper.Source: Insufficient External Information… · p. 21, Information Handling Deficiency (IHD) · p. 21

Information Integration Failure (IIF); Information Representation Misalignment (IRM)

• A tablet strategy report claimed coverage of 25+ devices but specified only 14, and its competitor matrix included devices absent from its core specification tables.A separate sensing review blurred source-authority distinctions by repeatedly citing a third-party aggregator rather than authoritative sources for Face ID.Source: Information Integration Failure (IIF) · p. 21, Information Representation… · p. 22

Verification Mechanism Failure (VMF)

• Despite a coherent, extensively referenced report, the model included inaccessible or nonexistent citations and review articles, then falsely claimed all references were cross-verified.This violated the instruction to exclude reviews and left the report substantively invalid despite its formal compliance.Source: Verification Mechanism Failure (VMF) · p. 22

Redundant Content Piling (RCP)

• Repeated UNESCO and OECD policy arguments across methodology, results, and discussion added no progressive analysis and obscured the report’s logical thread and empirical findings.The report recycled limited authoritative material in place of genuine cross-national case studies, surveys, or expert interviews, creating an illusion of richness without additional insight.Source: Redundant Content Piling (RCP) · pp. 22–23

Structural Organization Dysfunction (SOD)

• The agent dispersed six required survey dimensions across separate sections instead of one summary table, forcing readers to reconstruct entries and weakening verifiability and practicality.The failure may reflect treating the table as final formatting rather than the organizing principle for the content.Source: Structural Organization Dysfunction… · p. 23

Content Specification Deviation (CSD)

• The plugin evaluation became a blog-style narrative, omitting required feature matrices, benchmarks, interviews, and actionable implementation guidance; the passage suggests fluency may have outweighed specification adherence.Source: Content Specification Deviation (CSD) · p. 23

Deficient Analytical Rigor (DAR)

• The financial-network report presented common findings as if derived from decade-long Chinese lending data and confidently claimed advance GNN risk-pathway prediction without addressing uncertainty, noise, policy shocks, or market shifts.Source: Deficient Analytical Rigor (DAR) · p. 24

Strategic Content Fabrication (SCF)

• The investor essay used unverifiable precise returns, investment details, and risk rules as facts; the passage attributes this to plausible-sounding fabrication when structured real-world data are unavailable.Source: Strategic Content Fabrication (SCF) · p. 24

D Failure Report Example

• The appendix provides a representative failure-analysis report illustrating how to identify failure modes, their evidentiary bases, and inferred root causes for open coding.Source: D Failure Report Example · p. 24

Failure Identification

• The failure report identifies fabricated facts as its most significant failure, including false plugin-archive dates and a nonexistent Obsidian Bases core plugin.Source: Failure Identification · p. 25
• The report misrepresented its evidence by implying empirical user findings while supplying no interviews, quantitative metrics, benchmarks, or underlying research.Source: Failure Identification · p. 25
• The report also missed requested feature matrices and practical appendices, substituting long-form prose comparisons for structured comparisons.Source: Failure Identification · p. 25
• Despite its length, the analysis stayed superficial, omitting concrete complex-query examples and comparisons of query languages’ expressive power, limitations, and performance.Source: Failure Identification · p. 25

Root Cause of Factual Hallucination

• Prompt pressure to demonstrate depth led the agent to extrapolate and blend speculation into fabricated claims about plugin sustainability and Obsidian Bases.It invented future-dated examples of plugins archived in 2025 and presented “Bases” as an official core plugin, conflating community speculation with product roadmaps.Source: Root Cause Analysis · p. 25

Root Cause of Misrepresented Evidentiary Basis

• The agent mimicked the form of a software evaluation, treating a request for user interviews as a cue to write as if interviews had occurred.The analysis describes a gap between understanding a request’s semantics and having the capability to conduct the requested research.Source: Root Cause Analysis · p. 25

Root Cause of Formatting and Structural Failures

• Because fluent sequential prose is easier to generate than structured data, the agent compared features in paragraphs instead of producing the requested feature matrix.This favored word count and topical coverage over structural rigor.Source: Root Cause Analysis · pp. 25–26

Root Cause of Superficial Depth

• Statistical training-data knowledge supported broad topic coverage but not expert reasoning about complex projects or nuanced trade-offs, leaving the report’s depth shallow.The agent could compare DataviewJS with a UI but could not architect a project using both and analyze the trade-offs from genuine experience.Source: Root Cause Analysis · p. 26

E RACE and FACT Evaluation Frameworks

• RACE and FACT assess the quality and reliability of research reports generated for FINDER.The frameworks follow evaluation methodologies proposed in DeepResearch Bench [16].Source: E RACE and FACT Evaluation Frameworks · p. 26

E.1 RACE Framework

• RACE scores reports against a high-quality reference across comprehensiveness, insight/depth, instruction-following, and readability.Gemini 2.5 Pro serves as the Judge LLM for adaptive weighting and scoring.Source: §E.1 · p. 26

E.2 FACT Framework

• FACT evaluates factual grounding and citation reliability by checking whether citations support extracted statement–URL pairs.Gemini 2.5 Flash performs statement extraction and evidence verification; the framework reports two quantitative metrics.Source: §E.2 · p. 26

F Failure Taxonomy Construction Pipeline

• The appendix formalizes the human–machine collaborative pipeline for constructing the failure taxonomy, including specifications, workflow, and reproducible pseudocode.Source: F Failure Taxonomy Construction Pipeline · p. 26

F.1 Parameters

• The pipeline uses nine models’ execution records, five LLM coders from distinct model families, literature-derived seed concepts, a 0.6 cosine-similarity threshold, and a frequency-pruning threshold.Source: §F.1 · p. 27

F.2 Overview of the Pipeline

• The pipeline generates and optimizes codebooks for two data subsets, merges them into 51 concepts, derives 14 axial categories, and abstracts three core categories.It runs generation followed by two optimization iterations per subset, then refines the merged codebooks once and performs three axial-coding rounds.Source: §F.2 · p. 27

F.3.1 Algorithm 1: Open Coding - Generation Stage

• Open coding generates supplementary failure reports and gathers normalized annotations from each coder, tracking concept frequencies and their execution-record and coder sources.Source: §F.3.1 · p. 27

F.3.2 Algorithm 2: Open Coding - Optimization Stage

• Open coding optimization repeatedly merges the most similar concept pair above the similarity threshold, then removes concepts below the frequency threshold.If a merge succeeds, the algorithm repeats the similarity search; both thresholds are parameters of the procedure.Source: §F.3.2 · p. 28

F.3.3 Algorithm 3: Axial Coding with ICR Evaluation

• Three experts independently annotate stratified samples of 24 records in Round 1 and 54 in Rounds 2–3; Krippendorff’s α is calculated and annotations are refined through reconciliation.Reconciliation sessions last approximately five hours each.Source: §F.3.3 · p. 29
• Axial coding examines semantic, contextual, processual, causal, functional, structural, and strategic relationships among concepts.Source: §F.3.3 · p. 29
• Round 1 builds its base from Group A concepts, while later rounds use the union of the initial concepts and the previous round’s annotations.Source: §F.3.3 · p. 29

F.3.4 Algorithm 4: Selective Coding

• Selective coding applies human–LLM coding to axial categories, builds closed-loop relations capturing temporal progression and functional cycle, and returns the categories and relations.Source: §F.3.4 · p. 29
• The resulting hierarchical output provides a three-dimensional view of cognitive, retrieval, and generative failure aspects, supporting transparent error analysis and reproducible categorization.Source: §F.3.4 · p. 29

G Seed Conceptual Categories

• Three seed conceptual categories guide open coding of LLMs.Source: G Seed Conceptual Categories · p. 29

Seed Conceptual Categories

• The seed categories cover failures to understand requirements, bypass external information retrieval when required, and comply with requested format and structure.These failures involve superficial keyword matching, relying on internal knowledge instead of retrieval, and violating output specifications.Source: Seed Conceptual Categories · pp. 29–30

H.1 Data and Scope

• The reliability analysis covered 14 coded items: 4 in Reasoning, 5 in Retrieval, and 5 in Generation.Krippendorff’s α assessed agreement between human experts and Gemini 2.5 Flash.Source: §H.1 · p. 30
• Two coefficients were derived: overall α across all 14 items and category-level α within each category subset.The overall estimate captures cross-category variation; category-level estimates measure intra-category consistency.Source: §H.1 · p. 30
• Overall α is not a simple or weighted average of category-level α because expected disagreement depends on category marginals.The overall coefficient is computed across all items, whereas category-level coefficients are computed within each subset.Source: §H.1 · p. 30, §H.2 · p. 30

H.2 Formal Definition

• For nominal data, Krippendorff’s α uses δ(c, k), which equals 1 when c ≠ k and 0 otherwise.The overall estimate aggregates all 14 items when computing observed and expected disagreement; category estimates use their respective subsets.Source: §H.2 · p. 30

H.3 Computation Settings

• α above 0.80 at both overall and category levels indicates strong coder consistency and conceptual clarity in the FINDER framework.The overall estimate reflects agreement across all items, while category-level estimates isolate agreement within each conceptual dimension.Source: §H.3 · p. 30
• The gap between overall and category-level α shows how cross-category variance affects overall coding reliability.Large inter-category variance can make overall α diverge from category-level estimates.Source: §H.3 · p. 30, §H.2 · p. 30

I FINDER Stability Analysis via MiroFlow

• MiroFlow’s RACE scores show small run-to-run standard deviations; English is slightly higher on Overall, Comprehensiveness, and Depth, while Instruction-following and Readability are nearly identical across languages.The comparison uses three independent runs with English and Chinese prompts; the reported means are marginally higher for English, and both settings show strong stability.Source: I FINDER Stability Analysis via MiroFlow · p. 31, Table I.3 · p. 31, Figure I.2 · p. 31

J Checklist Distribution

• Figure J.3 presents the distribution of checklists in FINDER queries.Source: Figure J.3 · p. 32

K Configuration of Evaluated Models

• Evaluated models generally used their default system prompts and inference parameters unless otherwise stated.Source: K Configuration of Evaluated Models · p. 32

K.1 Proprietary API Models

• Proprietary API models were accessed through official APIs with default configurations, without tool integration or parameter tuning.Source: §K.1 · p. 32, K Configuration of Evaluated Models · p. 32

K.2 Open-source Models

• Open-source configurations included WebThinker-QwQ-32B with integrated web search, AFM-45B with citation verification, and MiroThinker-32B-DPO-v0.2 with multimodal image input.AFM used temperature 0.4, top_p 0.9, and max_tokens 32K; MiroThinker used max_tokens 64K and Qwen2.5-VL-72B-Instruct for image inputs.Source: §K.2 · p. 32, K Configuration of Evaluated Models · p. 32

K.3 Agent Frameworks

• The evaluated agent frameworks included MiroFlow, OWL, and OpenManus, with MiroFlow based on Claude-3.7-Sonnet and OpenManus based on gpt-4o.MiroFlow used temperature 0.3, top_p 0.95, and max_tokens 32K; OpenManus used temperature 0.0 and max_tokens 8192.Source: §K.3 · p. 32, K Configuration of Evaluated Models · p. 32

L Positive Taxonomy Metric

• The metric applies a cosine-based transformation to category error counts, producing scores on a bounded, positive scale.It defines |D| as the total number of evaluated instances and Ei as category i’s error count over [0, |D|].Source: L Positive Taxonomy Metric · pp. 32–33
• The score is strictly decreasing and invertible over the error-count domain, preserving the raw counts’ information in a normalized, interpretable representation.Source: L Positive Taxonomy Metric · p. 33
• The cosine curve is flatter near zero error and steepens as errors rise, emphasizing distinctions at moderate and high error levels unlike a constant-slope linear mapping.Source: L Positive Taxonomy Metric · p. 33
• Inspired by classical cosine similarity in information retrieval, the score interprets zero errors as alignment with an ideal and larger errors as greater angular deviation.Zero errors yield the maximal score; larger error rates correspond to smaller cosine values.Source: L Positive Taxonomy Metric · p. 33

• Anti-scraping protections can block Jina AI Reader from accessing or parsing pages, causing missing citations and incomplete retrieval chains that weaken FACT scores.Source: M Analysis of Missing Results of FACT… · p. 33
• Models sometimes generate fabricated URLs or citation-like strings misidentified as URLs, causing retrieval to fail when they do not map to real webpages.Source: M Analysis of Missing Results of FACT… · p. 33
• Network latency, high request volume, and temporary API throttling can trigger timeouts that prevent content from being returned during evaluation.Source: M Analysis of Missing Results of FACT… · p. 34
• Across successive versions, changes were minor while Krippendorff’s Alpha steadily increased.Source: M Analysis of Missing Results of FACT… · p. 34