PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories

Jiaming Wang, Ziteng Feng, Jiangtao Wu, Ruihao Li, Qianqian Xie, Yuxiang Ren, He Zhu, Xueming Han, Fanyu Meng, Junlan Feng, Jiaheng Liu

arXiv:2606.02060v2cs.AI

TL;DR

Final-answer evaluation cannot reveal which earlier trajectory spans make deep-research results unreliable. The paper introduces TELBENCH and DRIFT for claim-centric, span-level auditing, finding that DRIFT outperforms generic auditing while first-error localization remains difficult.

  • Problem

    Final-answer evaluation does not identify which trajectory spans make deep-research results unreliable, especially amid long, heterogeneous, framework-specific logs.

  • Method

    The paper builds TELBENCH from annotated real trajectories and proposes DRIFT, which audits claim support and dependency structure to localize harmful error spans.

  • Results

    DRIFT outperforms bare prompting and generic agentic auditors on span-level localization across backbone models, improving both precision and recall.

  • Takeaways & Limitations

    Span-level auditing provides a process-level view of reliability, while larger backbone models alone do not consistently improve trajectory diagnosis.

  • Takeaways & Limitations

    First-error accuracy remains substantially lower than overall span-level F1, particularly on the hard split.

Abstract

from arXiv · show

Deep-research agents solve tasks through long trajectories of search, tool use, evidence inspection, and answer synthesis. Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable. We study span-level error localization for deep-research agents. We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through LLM-assisted expert review. From these annotations, we build TELBench, a 1,000-instance benchmark for identifying error spans among normal exploration, failed searches, tentative hypotheses, and harmless noise. We further propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path. Experiments across model families and auditing frameworks show that DRIFT improves span-level error localization and first-error accuracy by up to 30 percentage points. Our work provides a process-level view of reliability in deep-research agents.

1 NJU-LINK Team, Nanjing University 2 JIUTIAN Research · 1 Introduction

The introduction frames deep-research trajectories as recorded decision processes whose early unsupported commitments can make later answers unreliable. It presents semantic-span annotations, TELBENCH, and DRIFT as tools for locating and auditing such harmful commitments.

  • 1 Introduction: Deep-research trajectories evolve claims about entities, constraints, sources, candidates, and conclusions that later spans may reuse as established facts.This view treats trajectories as decision processes recording claim introductions, evidence, and evolving commitments.
  • 1 Introduction: Final-answer evaluation cannot identify which earlier trajectory step made an agent’s result unreliable.Harm often begins with an earlier commitment inherited without revalidation, while raw logs are long, heterogeneous, and framework-specific.
  • 1 Introduction: Semantic spans provide a precise unit for localizing the first harmful commitment while remaining coarser than raw events.The study collects 2,790 real trajectories across two frameworks, three backbone models, and three deep-research benchmarks.
  • 1 Introduction: The corpus contains 2,790 annotated real deep-research agent trajectories spanning multiple frameworks, models, and benchmarks.It supports span-level analysis of harmful trajectory errors.
  • 1 Introduction: DRIFT audits claims, their support, and dependency structure rather than scoring trajectory spans independently.Its workflow uses a Claim Keeper, Support Seeker, and skill-routed Specialist Auditors to assess claims across the trajectory.
  • 1 Introduction: TELBENCH evaluates whether models can localize harmful error spans from ordered trajectory evidence.It is introduced as a process-level localization benchmark for deep-research agent trajectories.

2 Related Work

Related work has expanded from outcome-level evaluation of long-horizon tool-use tasks to process-level analysis of intermediate reasoning, step-level errors, and trajectory failures.

  • Deep-research systems and outcome-level evaluation: Recent benchmarks evaluate agents on long-horizon tasks involving web navigation and tool use rather than static question answering.Examples include GAIA, BrowseComp, WebArena, and OSWorld.
  • Deep-research systems and outcome-level evaluation: Newer benchmarks assess citation-grounded research reports using rubric-based evaluation.Examples include DeepResearch Bench, DeepResearchGym, LiveResearchBench, and DRBench.
  • Process-level evaluation and trajectory diagnosis: Process-level evaluation studies intermediate reasoning and agent traces, including step-level or tool-use errors and failure localization.Frameworks include ProcessBench, PRMBench, DeltaBench, VisualProcessBench, AgentProcessBench, TRACE, MAST, TRAIL, AgentRx, and CodeTracer.

3 Dataset

The dataset comprises 2,790 deep-research trajectories converted into semantic spans and expert-annotated for harmful process errors. TELBENCH is designed to localize verifiable errors amid benign exploration, while mechanism analyses reveal that process errors and failure patterns vary across workflow stages and settings.

  • Collection and annotation: 2,790 trajectories were collected from three public deep-research benchmarks and converted from raw logs into semantic spans around locally coherent objectives.The corpus spans trajectory collection, span segmentation, and expert-verified error annotation.
  • Collection and annotation: Each span receives an error or non-error label, excluding normal exploration, failed searches, tentative hypotheses, recovered mistakes, and tool noise unless they affect the answer path.An error span may introduce, rely on, amplify, or finalize a mistaken, unsupported, contradicted, or prematurely committed judgment.
  • TELBENCH construction: TELBENCH contains 600 easy and 400 hard instances, with hard cases featuring longer trajectories, sparser or implicit errors, and more benign distractors.The benchmark is intended to test both direct and subtle span-level localization.
  • TELBENCH construction: 1,890 trajectories, or 67.7% of the corpus, contain at least one span-level error and form the initial candidate pool for TELBENCH filtering and review.Filtering removes cases with issues such as missing records, incomplete tool outputs, degenerate runs, unverifiable error sources, or overrepresented patterns.
  • Dataset analysis: 36.9% of successful trajectories also contain process errors, showing that span-level process errors are related to but not equivalent to final-answer failure.Most failed trajectories contain at least one annotated error span.
  • Dataset analysis: 18 primary faults grouped into six families show stage-dependent mechanisms, while rarer faults such as missed checks, anchoring, and constraint-semantic errors are associated with higher failure probability.Evidence gaps are most frequent but are less predictive of failure than several rarer fault types.

4 DRIFT: Claim-Centric Trajectory Auditing

DRIFT audits completed research trajectories around decision-relevant claims rather than classifying spans independently. It builds a claim ledger, verifies support, and traces dependencies to identify harmful spans that commit to or propagate unsupported or conflicting consequential claims.

  • DRIFT takes only the task question and raw span text as external input, excluding judge results, gold labels, manual notes, span types, and generated summaries.
  • A: Claim Keeper: Claim Keeper constructs a global ledger recording each claim’s introduction, first consequential use, later reuse, type, and status.The ledger distinguishes exploratory or tentative reasoning from consequential or finalized commitments.
  • B: Support Seeker: Support Seeker assigns each consequential claim one of four statuses: DIRECT, WEAK, MISSING, or CONFLICTING.These statuses distinguish decisive support from partial, absent, or contradictory evidence shown in the trajectory.
  • C: Dependency Tracer: Dependency Tracer marks spans as errors only when risky claims lack adequate support and are committed to, reused, amplified, or finalized in consequential reasoning.Weak or missing support alone is insufficient; the claim must affect later reasoning, computation, giving up, or answer finalization.

5 Experiment

Experiments evaluate TELBENCH across five model families and four auditing frameworks using span-level and first-error metrics. DRIFT achieves the strongest overall F1, while first-error localization and larger-model scaling remain challenging.

  • Evaluation setup: TELBENCH evaluation covers five model families and compares Bare LLM, Claude Code, Codex, and DRIFT.The benchmark includes Qwen-series, GPT-5.4, DeepSeek-V3.2, Claude-Sonnet-4.5, and Gemini-2.5-Pro.
  • Evaluation setup: The verified-1K set contains 600 easy and 400 hard examples, evaluated with macro precision, recall, F1, and first-error accuracy.Span-level metrics assess error spans, whereas first-error accuracy measures detection of the earliest predicted error.
  • Main results: DRIFT achieves the best overall F1 across all backbone models, outperforming bare full-trajectory prompting, Codex, and Claude Code.The results indicate that more complex generic agentic workflows alone do not ensure reliable trajectory diagnosis and can degrade performance for some backbones.
  • Diagnostic limitations: First-error accuracy remains substantially lower than span-level F1, particularly on the hard split, showing that detecting errors differs from locating their earliest occurrence.Auditors can identify unreliable trajectory regions yet struggle to pinpoint the earliest annotated error span in long sequences.
  • Scaling analysis: Across Qwen variants, increasing model scale does not consistently improve macro F1 or first-error accuracy, and the hard split remains challenging at every scale.The reported pattern suggests that diagnostic structure tailored to long, noisy trajectories matters alongside backbone capacity.

6 Further Analysis

Further analysis shows that DRIFT remains stronger as trajectories become more complex, gains arise from complementary auditing modules, and performance generally occupies a favorable efficiency frontier. It also broadens recall across error types, especially failures requiring evidence and constraint verification.

  • Sensitivity to Span Complexity: DRIFT consistently outperforms Bare across all span-complexity buckets, although both methods degrade as trajectories lengthen.The advantage is especially visible for high-span trajectories, where single-pass reading can miss early or distributed errors.
  • Ablation of modules: Performance improves steadily as Claim Keeper, support checking, and dependency tracing are added to bare prediction.The largest gain comes from claim-level auditing, while later modules further improve evidence grounding and span localization.
  • Efficiency Analysis: DRIFT mostly lies on the Pareto frontier, improving F1 without disproportionate token overhead.Gemini is the notable exception: more than half of its tokens are spent on thinking, producing a much larger average token budget despite competitive performance.
  • Error-type coverage: DRIFT improves span-level recall across nearly all frequent error categories, particularly evidence- and constraint-related failures.Bare models struggle with source verification, constraint semantics, unsupported commitments, and omitted constraint checks.

7 Conclusion · Appendix · A Annotation Guidelines and Annotator Information

The paper frames deep-research reliability as span-level error localization over semantic trajectories, introducing TELBENCH to distinguish harmful errors from benign behavior and DRIFT for claim-centric auditing. Its annotation workflow combines LLM-assisted candidate errors with expert adjudication across ordered trajectory spans.

  • 7 Conclusion: The study evaluates reliability beyond final-answer correctness by locating harmful errors at the span level in semantic agent trajectories.This formulation targets process-level reliability rather than only outcome correctness.
  • 7 Conclusion: TELBENCH uses real agent runs to test whether models distinguish harmful error spans from benign trajectory behavior.The benchmark reflects the difficulty of separating substantive errors from normal exploration.
  • 7 Conclusion: Weakly supported claims can become harmful when agents repeatedly reuse them as evidence during deep research.The paper identifies this repeated reuse as a central source of trajectory-level error.
  • 7 Conclusion: DRIFT provides a claim-centric auditing framework for analyzing error propagation through the research trajectory.Its purpose is to audit how claims relate to trajectory evidence and the answer path.
  • A Annotation Guidelines and Annotator Information: Expert annotators inspect full trajectories, select spans, and use editable rationales and final decisions to adjudicate candidate errors.The interface separately presents non-error spans and span-stage cues to support distinguishing harmful errors from normal exploration.
  • A Annotation Guidelines and Annotator Information: The annotation console displays case metadata, the task question, the ground-truth answer, ordered semantic spans, and LLM-assisted candidate error spans.Candidate spans are highlighted in red, with proposed rationales shown for expert review.

B Detailed Experiment Setting

The experiment controls external tooling to make comparisons across agent frameworks interpretable, using unified search and reading interfaces to reduce behavioral confounds.

  • Framework and tooling setup: Serper provides a unified search interface and Jina a unified reading interface across frameworks, reducing confounding from different retrieval implementations.The setup targets external factors that can substantially shift agent behavior, especially search and page reading.

C Detailed Error Analysis for Deep-research Agent Systems.

The mechanism analysis uses the full annotated corpus, while the Verified-1K subset is reserved for benchmark evaluation.

  • Scope of analysis: Mechanism analysis is conducted on 2,790 annotated trajectories; Verified-1K is used only for benchmark evaluation.This scope applies unless otherwise stated.

C.1 Basic Analysis … E Ablation Study

The paper analyzes trajectory reliability through span-level error burden, stage-normalized risk, effort profiles, and taxonomies that separate operational stages from underlying fault mechanisms. It also reports framework token overhead and module ablations as additional system-level analyses.

  • C.1 Basic Analysis: 97.3% of failed trajectories contain an annotated error span, while 36.9% of successful trajectories also contain one, showing errors and final failure are related but non-equivalent.Successful agents can recover from local mistakes or reach correct answers despite unsupported intermediate commitments.
  • C.1 Basic Analysis: MiroFlow generally produces longer trajectories with more intermediate spans, whereas OAgent tends to maintain shorter trajectories despite sometimes high tool usage.These effort profiles describe execution behavior rather than final-answer accuracy.
  • C.2 Operation Stage Taxonomy: Each span receives one of eight operation-stage labels regardless of correctness, enabling process-level localization and denominator-based error-rate comparisons.The stages cover task decomposition, search, verification, extraction, decision-making, conflict recovery, and final answering.
  • C.3.1 Construction: 4,631 error-reason keys seed a hierarchical map-reduce induction process followed by manual normalization, boundary calibration, and back-labeling.The procedure uses 58 shuffled chunks, each producing 10 local error types before consolidation.
  • C.3 Error Fault Taxonomy: The fault taxonomy contains 18 primary faults organized into six families, balancing fine-grained diagnosis with cross-system comparability.Examples include unsupported commitments, source verification failures, candidate scope errors, and constraint misinterpretations.
  • C.3.2 Analysis: Joint stage and fault annotations distinguish where an error occurs from why it occurs, such as separating retrieval-stage poor queries from other retrieval failures.Every erroneous span receives exactly one primary fault label, while non-error spans receive none.
  • D Token Consumption: Token consumption analysis reports total prompt and completion tokens and average total tokens per trajectory, revealing substantially different framework overheads on the same benchmark.The ablation figure reports that each module brings better performance.

E.1 Full ablation trends.

Across four base models and three macro-averaged metrics, the ablation shows progressively stronger performance as DRIFT modules are added. Claim Keeper improves over bare prediction, Support Seeker strengthens recall, and the full pipeline achieves the strongest overall balance after dependency tracing.

  • Full ablation trends: Adding Claim Keeper improves bare prediction, Support Seeker further strengthens recall, and full DRIFT achieves the strongest overall balance after dependency tracing.The trend holds across precision, recall, and F1 in the full module ablation.

F Case Study · G Prompt

The case studies show that deep-research failures arise across trajectories: premature candidate commitments can propagate through role mismatches and unresolved constraints, unsupported evidence can survive despite a correct final answer, and overly narrow candidate scopes can block recovery. The prompt’s adjudication rule prioritizes earlier harmful commitments when they improve earliest-error localization.

  • F Case Study: Wrong-candidate commitment narrows later retrieval, causing unresolved constraints and role mismatches to propagate into an invalid final result.The trajectory selects the 2021 UK Championship Final before validating the full conjunction of constraints, then uses the winner’s professional year for the loser condition.
  • F Case Study: Claude + DRIFT and DeepSeek + DRIFT exactly recover s001,s003,s007,s008, whereas bare-style frameworks mainly detect s007.The stronger systems identify the multi-span propagation chain rather than only the explicit winner/loser contradiction.
  • F Case Study: A correct final title can still conceal a trajectory-level error when a worker’s seven-talk claim lacks visible retrieval support and the main agent adopts it as verified.The final report repeats unsupported seven-talk and verification claims, overstating the evidential basis despite the correct answer string.
  • F Case Study: Many bare-style settings predict an empty error set for the unsupported-evidence case, while GPT-5.4 variants, Claude + DRIFT, and DeepSeek + DRIFT identify s003,s004,s005.This contrast shows why evaluation must inspect whether intermediate claims are supported, not only whether the final answer matches the target.
  • F Case Study: An incomplete fruit list omits bananas, and the trajectory later resolves conflicting evidence by falsely declaring that bananas are absent and the requested answer impossible.The first error narrows the candidate set; later image-related searches create a recovery opportunity that the agent does not use.
  • F Case Study: Together, the cases distinguish propagated wrong candidates, unsupported evidence chains, and narrow candidate scopes as trajectory-level error phenomena.The trajectory-slice format separates normal retrieval from highlighted errors, making these different failure patterns explicit.
  • G Prompt: An earlier error is added only when it improves earliest-error localization and the earlier span itself commits to the same harmful claim.The rule applies when a later committed span is correct but an earlier error materially improves localization of the harmful claim.
Loading 2606.02060v2…