PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents

Zixuan Yu, Zhenheng Tang, Xiaowen Chu, Bo Han

arXiv:2610.29346v1cs.LG

TL;DR

End-to-end benchmarks provide limited insight into failures within long-horizon Deep Search and Deep Research workflows. This paper diagnoses full execution traces using a taxonomy that separates shared system errors from paradigm-specific task errors. Its analysis identifies error propagation during interaction as a performance bottleneck and motivates process-level evaluation and targeted design guidance.

  • Problem

    Existing evaluation predominantly measures final outcomes, leaving intermediate failures in long-horizon agent workflows poorly understood.

    Source: §1 · p. 1, §A.2 · p. 11

  • Method

    The paper analyzes full execution traces with a process-oriented framework and a taxonomy separating shared System-Level errors from paradigm-specific Task-Level errors.

    Source: §1 · p. 2, §3 · p. 2

  • Results

    Trace-level analysis finds that performance bottlenecks often stem from error propagation during interaction, not solely from final-response generation.

    Source: §5 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Process-level diagnosis can guide targeted improvements in training, tool design, and agent architecture for more reliable long-horizon workflows.

    Source: §5 · p. 8

Abstract

from arXiv · show

As Large Language Models (LLMs) evolve into autonomous Deep Search and Deep Research Agents capable of executing long-horizon, multi-step tasks, evaluating their reliability becomes increasingly critical. However, existing evaluation paradigms predominantly focus on final task outcomes, leaving a significant gap in understanding the intermediate execution traces, where agents are highly susceptible to various errors. To address this critical gap, we introduce a comprehensive methodology and analytical framework designed specifically to diagnose errors in intermediate workflows. Building upon this framework, we establish a structured taxonomy that categorizes these vulnerabilities into two primary dimensions: System-Level Errors and Task-Level Errors. Through a systematic examination of step-by-step execution traces and agent-environment interactions, we reveal how these underlying errors affect overall performance. Finally, through empirical experiments and systematic analysis of the identified error types, we provide actionable design guidelines, aiming to enhance the robustness, reasoning capabilities, and reliability of Deep Search and Deep Research agents.

1. Introduction

Outcome-focused benchmarks leave intermediate failures in long-horizon Deep Search and Deep Research workflows poorly understood. The paper introduces trace-based evaluation and a two-level error taxonomy to diagnose those failures and their propagation.

1. Introduction

  • Existing benchmarks emphasize end-to-end outcomes, leaving intermediate failures in extended search and research workflows poorly understood.A correct answer may conceal flawed reasoning, while an incorrect outcome gives limited insight into the source of failure.Source: §1 · p. 1
  • The framework separates shared System-Level errors from paradigm-specific Task-Level errors in agent evaluation.System-Level errors include malformed tool calls, schema violations, environment rejections, and infrastructure failures.Source: §1 · p. 1
  • Deep Search task failures include incomplete exploration and weak evidence accumulation, whereas Deep Research failures include synthesis and citation-grounding weaknesses.The introduction defers fine-grained category definitions to the taxonomy section.Source: §1 · pp. 1–2
  • The paper presents a unified error-diagnosis framework and benchmark that use full execution traces for process-level evaluation.It derives a taxonomy across both error levels and paradigms, then studies how errors emerge and propagate through workflows.Source: §1 · p. 2
  • The authors propose an evaluation framework that captures full agent trajectories and a taxonomy distinguishing shared system failures from paradigm-specific task failures.The framework aims to support reproducible diagnosis beyond end-to-end accuracy.Source: §1 · p. 2
  • The paper also analyzes how different error types occur and propagate, drawing actionable implications for agent robustness and reliability.This trace-based analysis is presented as a contribution alongside the framework and taxonomy.Source: §1 · p. 2

2. Preliminary

The paper characterizes Deep Search as iterative evidence acquisition and Deep Research as long-context synthesis. Both are modeled as interactive workflows using web search, web fetch, and stopping, but their objectives differ.

2.1. Deep Search Agents

  • Deep Search incrementally acquires external evidence through multi-step exploration, repeatedly choosing what to search, inspect, and when to stop.Its minimal tool set comprises web search, web fetch, and stop.Source: §2.1 · p. 2
  • At each Deep Search step, the agent uses its query and interaction history to act; search returns ranked results and fetch returns webpage content.Each search result contains a title, URL, and snippet; the agent iterates until termination and then answers with an evidence set.Source: §2.1 · p. 2

2.2. Deep Research Agents

  • Deep Research transforms web-scale evidence into a structured analytical report, with cross-source reconciliation and consistent synthesis as central challenges.It uses the same minimal web search, web fetch, and stop tool set as Deep Search.Source: §2.2 · p. 2
  • A Deep Research agent builds an evidence pool by iteratively searching and fetching pages, then generates a report.Report quality is evaluated by factual grounding, cross-source consistency, and task completeness, which must be jointly satisfied.Source: §2.2 · p. 2

3. Methodology

The methodology distinguishes deterministic system breakdowns from semantic task failures and combines parsing with LLM-assisted judgment. It defines taxonomies for tool and environment errors and for paradigm-specific reasoning failures.

3. Methodology

  • The methodology separates System-Level errors from Task-Level errors, using rule-based parsing for the former and human-guided LLM-as-a-Judge evaluation for the latter.This distinction is intended to avoid conflating infrastructure faults with semantic cognitive limitations.Source: §3 · p. 2

3.1. System-Level Errors

  • System-Level diagnosis extracts tool calls and reasoning from trajectory logs, then uses an LLM to assess whether intent aligns with tool syntax.The taxonomy covers agent tool calls, web search, and web fetch; it treats these as deterministic failures rather than cognitive deficiencies.Source: §3.1 · pp. 3–4
  • System-Level errors include malformed tool calls, incorrect tool names, flawed search arguments or queries, and fetch failures such as unsupported content, dead pages, or anti-scraping blocks.Search errors include schema violations, tool-function mismatch, irrelevant or misaligned queries, and syntactical failures.Source: §3.1 · pp. 3–4

3.2. Task-Level Errors

  • Task-Level errors are identified through human-guided analysis of sampled traces to construct a behavioral taxonomy, then categorized at scale using an LLM judge.The taxonomy distinguishes Deep Search retrieval-strategy failures from Deep Research context-management and synthesis failures.Source: §3.2 · p. 4
  • Deep Search task failures include neglecting retrieved observations, skimming snippets, stopping before collecting required evidence, inefficient navigation, and ineffective queries.These modes reflect retrieval strategy and localized reasoning failures.Source: §3.2 · p. 4
  • Deep Research task failures include neglecting contradictions or favoring early sources, hallucinating beyond incomplete evidence, and drifting from the original task constraints.These modes reflect breakdowns in global context management and evidence synthesis over long trajectories.Source: §3.2 · p. 4

4. Experiment

The experiments use controlled execution traces to diagnose process-level errors across Deep Search and Deep Research. They find that system-level robustness varies by dataset, while task-level failures differ across models and frameworks.

Setup

  • The controlled evaluation uses established benchmark tasks to generate full traces, not to optimize or compare end-to-end task success.Deep Search uses two core tools, search and fetch, while Deep Research evaluates two research-oriented frameworks under extended synthesis workflows.Source: §4 · pp. 4–6

Result

  • XBench-DS exhibits the highest concentration of system-level failures, while BrowseComp and BrowseComp-zh remain comparatively stable.The analysis also finds dataset-specific failure compositions, including tool–function alignment and accessibility issues in BrowseComp and tool orchestration failures in XBench-DS.Source: §4 · p. 6, Table 2 · p. 7
  • System-level failures are structurally heterogeneous across benchmarks, with different datasets dominated by distinct interaction bottlenecks.BrowseComp-zh shows stronger domain–tool alignment challenges alongside accessibility failures; XBench-DS is dominated by tool orchestration failures.Source: §4 · p. 6
  • Purely syntactic failures are rare, while semantic misalignment and real-world web environments account for much of the systemic friction.Agents often recover through iterative adjustment and continued interaction despite system-level disruptions.Source: §4 · p. 6
  • Deep Search models show a strong positive correlation between process-level reliability (No Error Rate) and final task accuracy (Acc).Models navigating intermediate cognitive bottlenecks more successfully consistently achieve higher end-to-end success.Source: §4 · p. 6
  • Task-level error distributions reveal distinct behavioral weaknesses across Deep Search models and Deep Research frameworks.GPT-5-mini is associated with Premature Stop and Surface Skim, while Miro-30B is bottlenecked by Ineffective Query formulation and Obs. Neglect; Open Deep Research shows Goal Drifting, while Step Deep Research shows Synthesis Hallucination and compounded errors.Source: §4 · p. 7

5. Discussion

The discussion connects trace-level error diagnosis to targeted training, toolset, and framework design. Its recommendations address distinct system-level and task-level failures and support process-level reliability improvements.

Implications for Model Training

  • Separating System-Level from Task-Level errors supports distinct supervision for reliable tool execution and higher-level exploration, evidence integration, and synthesis.The taxonomy also supports trace-aware optimization and structured rewards that discourage premature termination or inefficient exploration.Source: §5 · pp. 7–8

Implications for Toolset Design

  • Toolsets should provide schema constraints, informative error feedback, and recovery affordances, with support tailored to each paradigm’s bottlenecks.Deep Search tools should support progressive evidence accumulation; Deep Research tools should facilitate source alignment, contradiction detection, and citation-grounded synthesis.Source: §5 · p. 8

Implications for Agentic Frameworks

  • Frameworks should separate planning, verification, and repair, with adaptive stopping and coverage checks for Deep Search and staged synthesis for Deep Research.Lightweight runtime self-diagnosis can enable intervention before failures fully propagate.Source: §5 · p. 8

Broader Takeaway

  • Process-level diagnosis turns opaque failures into structured signals for improving training, tool ecosystems, and framework architecture.The discussion presents fine-grained evaluation as infrastructure for reliable autonomous agents in open environments.Source: §5 · p. 8

6. Conclusion

The conclusion presents a trace-based framework and taxonomy for diagnosing failures in Deep Search and Deep Research agents, with future work extending the framework to broader environments and automated recovery. The background distinguishes dynamic web retrieval from long-horizon information synthesis and notes that conventional single-turn RAG relies on static, pre-indexed corpora.

6. Conclusion

  • The framework evaluates agents through full execution traces and separates failures into System-Level and Task-Level categories to make intermediate mechanisms observable across paradigms.The conclusion links the taxonomy to targeted training, robust tool interfaces, and modular planning, verification, and repair mechanisms.Source: §6 · p. 8
  • Future work will extend the framework to broader environments, richer tool ecosystems, and automated online error recovery.Source: §6 · p. 8

A.1. Evolution of Deep Search and Deep Research Agents

  • Traditional single-turn RAG relies on static, pre-indexed local corpora and cannot autonomously acquire up-to-date knowledge from the open internet.The passage describes this limitation as a bottleneck that catalyzed Deep Search.Source: §A.1 · pp. 10–11
  • Deep Search targets precise facts through multi-hop web retrieval, whereas Deep Research compiles and synthesizes large volumes of information into comprehensive reports.Source: §A.1 · p. 11

A.2. Evaluation Paradigms of Deep Search and Deep Research Agents

Benchmarks for Deep Search and Deep Research evaluate web navigation, retrieval, and long-form synthesis. The section argues that terminal outcome metrics leave intermediate failures difficult to diagnose, motivating step-level analysis.

A.2. Evaluation Paradigms of Deep Search and Deep Research Agents

  • Deep Search benchmarks span multi-hop question answering and interactive web navigation, while Deep Research benchmarks assess large-context integration and long-form synthesis.Examples include HotpotQA, Mind2Web, WebArena, and BrowseComp for Deep Search, and Deep Research Bench and ADR-Bench for Deep Research.Source: §A.2 · p. 11
  • End-to-end success rates, exact matches, and holistic report scores can obscure process flaws and provide little insight into the specific failure bottleneck.The section motivates step-level diagnostic analysis for trajectories spanning many steps and large context windows.Source: §A.2 · p. 11

A.3. Error Analysis and Diagnostics in LLMs

Prior LLM error analysis focused on static generation and localized reasoning, while newer work examines tool use and environment interaction. Deep Search and Deep Research workflows centered on iterative retrieval and large-scale evidence synthesis remain comparatively underexplored.

A.3. Error Analysis and Diagnostics in LLMs

  • Earlier diagnostic studies examined factual hallucinations and model reliability in knowledge-intensive queries (Kadavath et al., 2022).Other work studied inconsistencies between intermediate Chain-of-Thought traces and final answers, alongside limits in compositional generalization (Gendron et al., 2023).Source: §A.3 · pp. 11–12
  • Recent tool-augmented LLM analyses identify invalid action formatting, incorrect argument grounding, and inconsistent tool selection (Zhu et al., 2025).These analyses focus less on information-seeking systems built around iterative retrieval and large-scale evidence synthesis.Source: §A.3 · p. 12
  • Failures specific to Deep Search and Deep Research workflows remain comparatively underexplored, motivating a taxonomy for long-horizon, information-dense research tasks.Source: §A.3 · p. 12

B. Dataset

The evaluation uses separate benchmark sources and success criteria for Deep Search and Deep Research. Its diagnostic prompts specify system-level error categories and task-level Deep Search failure labels.

B. Dataset

  • Deep Search evaluation uses BrowseComp, BrowseComp-zh, GAIA-val-text, and XBench-DS; BrowseComp datasets use cost-driven subsamples.Source: B. Dataset · p. 12
  • Deep Research uses DeepResearch Bench, assessing comprehensiveness, insight/depth, and instruction-following under benchmark-dependent criteria.Source: B. Dataset · p. 12
  • Deep Search success is answer correctness against ground truth, allowing benchmark normalization such as alias normalization and formatting tolerance.Source: B. Dataset · p. 12

C. Prompt

  • The system-level prompt distinguishes agent logic and decision errors, search-tool errors, read/scrape errors, and newly identified error patterns.Examples include malformed tool syntax or nonexistent tool names, bad search formatting or tool-function mismatch, and unsupported content or nonexistent pages.Source: C. Prompt · pp. 14–15
  • The task-level Deep Search prompt classifies traces as success, information missing, premature stopping, rabbit-hole trap, or ineffective query.It instructs the evaluator to compare retrieved facts with ground truth and track the search process step by step.Source: C. Prompt · p. 15

D. Case Studies

Representative case studies from Deep Search and Deep Research illustrate how intermediate failures emerge, propagate, and affect final outcomes, showing the practical value of a process-level taxonomy.

D. Case Studies

  • The case studies use examples from both agent paradigms to illustrate failure emergence, propagation, and effects on final outcomes.The paper presents them as demonstrating the practical value of its process-level taxonomy.Source: D. Case Studies · p. 15

D.1. System-Level Error

Two Deep Search examples show system-level tool errors: a search tool used to open a URL, and a search call rejected after receiving a parameter from another tool's schema.

D.1. System-Level Error

  • A Google search call using a full webpage URL to open it is diagnosed as a Tool-Function Mismatch.The diagnosis says the agent intended to read the page but selected search rather than scraping.Source: §D.1 · pp. 15–16
  • A Google search call containing the unsupported “first sentences” argument is diagnosed as a Call Formatting Error.The example attributes the rejection to confusion between Google search and Wikipedia retrieval parameters.Source: §D.1 · p. 16

D.2. Task-Level Error

The examples classify task-level failures in deep search and research, including ineffective retrieval, unsupported synthesis, and unreconciled contradictions. In one case, the report nevertheless stayed focused on the original hypothesis.

D.2. Task-Level Error

  • The search agent gathered no concrete web evidence, cycling through broad speculative queries instead of identifying candidate bands or videos.It neither compared substantive observations with the ground truth (“The Light”) nor ignored retrieved evidence, because none was retrieved.Source: §D.2 · p. 16
  • The research agent reported conflicting airport-impact figures and inconsistent job-per-seat calculations without reconciling them.For example, it cited $153M direct and $188M total impact against a snippet reporting $295M direct spending, then used mismatched employment multipliers.Source: §D.2 · p. 16
  • The report gave precise figures and scaling estimates for sources it had not successfully accessed, including internal placeholders as citations.The analysis categorizes this as synthesis hallucination and sourcing bias.Source: §D.2 · p. 16
  • Despite its sourcing problems, the final report remained focused on assessing the original 500,000-passenger airport hypothesis.It presented supporting and counter-evidence, methodological notes, and a synthesized conclusion.Source: §D.2 · p. 16

Related papers

Loading 2610.29346v1…