PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents

Yanyu Chen, Jiyue Jiang, Jiahong Liu, Yifei Zhang, Xiao Guo, Irwin King

arXiv:2602.21230v1cs.CL

TL;DR

Deep Research Agents are difficult to evaluate because final-answer metrics overlook reasoning quality, efficiency, robustness, and latent capability. TRACE evaluates complete trajectories with utility and scaffolded-capability measures, and experiments show that it reveals accuracy, efficiency, and robustness trade-offs missed by singular metrics.

  • Problem

    Existing evaluation relies on singular outcome metrics and static benchmarks that inadequately capture reasoning quality, robustness, and latent capability.

  • Method

    TRACE combines a Hierarchical Trajectory Utility Function, Scaffolded Capability Assessment, and DeepResearch-Bench with controllable complexity and embedded information traps.

  • Results

    TRACE provides more granular rankings and uncovers trade-offs between accuracy, efficiency, and robustness that singular metrics miss.

  • Takeaways & Limitations

    TRACE offers a multi-dimensional basis for developing and comparing more reliable Deep Research Agents.

  • Takeaways & Limitations

    The authors plan to extend TRACE to other complex, long-horizon domains, indicating that the current framework’s demonstrated scope does not yet cover those domains.

Abstract

from arXiv · show

The evaluation of Deep Research Agents is a critical challenge, as conventional outcome-based metrics fail to capture the nuances of their complex reasoning. Current evaluation faces two primary challenges: 1) a reliance on singular metrics like Pass@1, creating a "high-score illusion" that ignores the quality, efficiency, and soundness of the reasoning process; and 2) the failure of static benchmarks to quantify crucial attributes like robustness and latent capability. To address these gaps, we introduce TRACE (Trajectory-Aware Comprehensive Evaluation), a framework that holistically assesses the entire problem-solving trajectory. To counter the "high-score illusion", we propose a Hierarchical Trajectory Utility Function that quantifies process efficiency and cognitive quality, including evidence grounding, alongside accuracy. To measure deeper attributes, TRACE introduces a Scaffolded Capability Assessment protocol, quantifying an agent's latent ability by determining the minimum guidance needed for success. Our contributions include the TRACE framework, its novel metrics, and the accompanying DeepResearch-Bench with controllable complexity. Experiments show TRACE delivers a granular ranking that uncovers critical trade-offs between agent accuracy, efficiency, and robustness entirely missed by singular metrics.

1 INTRODUCTION

TRACE addresses shortcomings in evaluating Deep Research Agents by assessing complete problem-solving trajectories rather than relying on final-answer accuracy alone. It combines trajectory utility, diagnostic tools, and DeepResearch-Bench to expose efficiency, reasoning quality, robustness, and latent capability.

  • Motivation: Singular outcome metrics such as Pass@1 can create a “high-score illusion” by rewarding correct answers despite inefficient, circuitous, or unsound reasoning.This obscures deficiencies in planning, efficiency, and trustworthiness, including reliance on hallucinated evidence.
  • Motivation: Static benchmarks do not adequately measure robustness against misinformation and information traps or latent capability under limited guidance.These gaps hinder diagnosis of failures, behavioral patterns, and future improvements.
  • TRACE: TRACE holistically evaluates the entire problem-solving trajectory through a multi-faceted framework that moves beyond final outcomes.The framework is intended to provide a granular view of agent performance.
  • TRACE: Its Hierarchical Trajectory Utility Function measures accuracy alongside process efficiency and cognitive quality, including evidence grounding and logical soundness.It penalizes redundant or inefficient actions while assessing whether reasoning is grounded and sound.
  • Benchmark: DeepResearch-Bench uses controllable task complexity and embedded information traps to support controlled measurement of difficult-to-quantify agent characteristics.The benchmark is designed to enable robustness analysis and related diagnostic evaluation.
  • Findings: TRACE produces more granular rankings than traditional metrics and reveals trade-offs among accuracy, efficiency, and robustness that singular metrics miss.Experiments evaluate diverse state-of-the-art agents, including AgentFounder, WebSailor-V2, and ReSum.

2 RELATED WORKS

Prior evaluation of Deep Research Agents uses challenging benchmarks but remains fundamentally reliant on singular outcome-based metrics such as Pass@1. This reliance limits trajectory-level understanding of agent behavior.

  • Deep Research Agent Evaluation: Recent benchmarks such as BrowseComp-en/zh and GAIA evaluate state-of-the-art agents including AgentFounder and WebSailor-V2.These benchmarks are part of the broader evaluation landscape for Deep Research Agents.
  • Deep Research Agent Evaluation: The prevailing paradigm relies on singular outcome-based metrics such as Pass@1, creating a fundamental limitation in evaluating Deep Research Agents.This limitation motivates trajectory-level evaluation beyond final outcomes.

3 METHODOLOGY

TRACE evaluates Deep Research Agents through their full problem-solving trajectories, combining utility, evidence, robustness, and latent-capability diagnostics. Its methodology measures efficiency, cognitive quality, recovery from information traps, minimum guidance required for success, and strategic policy behavior.

  • Trajectory-aware evaluation: TRACE models an agent’s problem-solving process as a trajectory H and evaluates it with a holistic utility function U(H).The framework treats accuracy, efficiency, and cognitive soundness as jointly relevant properties of a trajectory.
  • Trajectory-aware evaluation: The utility function combines final-answer correctness with process efficiency and cognitive quality rather than relying on accuracy alone.Cognitive quality includes evidence grounding and reasoning soundness, while efficiency captures the exploration process.
  • 3.1.1 Process Efficiency E(H): Process efficiency rewards effective information gathering through a complexity-adjusted reward divided by total trajectory cost.Trajectory cost aggregates action costs and can include a redundant exploration penalty for consecutive uninformative actions.
  • 3.1.1 Process Efficiency E(H): Marginal Information Gain measures each observation’s novel relevance beyond prior observations, rewarding trajectories that consistently produce positive information gain.Positive MIG helps an agent avoid redundant exploration penalties and reduce overall cost.
  • 3.1.2 Cognitive Quality (C(H)): Cognitive quality combines evidence grounding with reasoning robustness, penalizing unsupported claims and measuring recovery from misleading information traps.Reasoning robustness uses recovery latency: the number of steps needed to find a productive path after encountering a trap.
  • 3.2 Latent Capability and Policy Diagnostics: TRACE diagnoses latent capability with the Minimum Hint Rate and profiles strategic behavior through Entropy Adaptability and Trajectory Reproducibility Score.Minimum Hint Rate measures the smallest oracle-solution fraction needed to reach a target success rate; Entropy Adaptability relates information gain to reductions in policy uncertainty, while TRS measures consistency across runs.

4 EXPERIMENTS

The experiments evaluate Deep Research Agents with TRACE across controlled and public benchmarks, using trajectory-level metrics to expose accuracy–efficiency–quality trade-offs and latent capability. Results show that TRACE differentiates agents and training methods beyond Pass@1 through holistic utility, diagnostic metrics, and scaffolded evaluation.

  • Experimental Design: The experiments study SOTA agents and training paradigms through comprehensive re-evaluation, controlled comparisons, and targeted ablations.The evaluation covers TRACE’s full metric suite and examines both existing agents and methodological components.
  • Experimental Setup: TRACE evaluates complex open-domain question answering by measuring agents’ multi-step web information seeking, planning, efficiency, evidence grounding, and cognitive robustness.Pass@1 is retained as a baseline, while Trajectory Utility combines process efficiency and cognitive quality.
  • Datasets: DeepResearch-Bench provides controllable complexity, embedded information traps, oracle trajectories, and specialized subsets for overall performance, robustness, and scaffolding.Public benchmarks, primarily BrowseComp-en and text-only GAIA, are used to assess generalization.
  • Main Results: DeepSeek-V3.1-671B reaches 65.8% Pass@1 but only 0.65 utility, whereas AgentFounder-30B reaches 0.81 utility despite lower Pass@1 on TRACE-Core.The comparison attributes DeepSeek’s lower utility to Process Efficiency E = 0.68 and highlights TRACE’s divergence from Pass@1 rankings.
  • Main Results: AgentFounder-30B leads open-source diagnostic performance with E = 0.88, C = 0.91, and G_E = 0.95, while WebSailor-V2-30B reaches R_R = 0.84.These metrics provide a fine-grained profile of efficiency, cognitive quality, evidence grounding, and reasoning robustness.

5 CONCLUSION & OUTLOOK

TRACE addresses limitations of outcome-based evaluation by assessing process quality and latent attributes, and the authors suggest extending its utility function and domain coverage.

  • TRACE evaluates Deep Research Agents beyond singular accuracy scores, incorporating efficiency, robustness, and latent attributes.Its components include a Hierarchical Trajectory Utility Function and Scaffolded Capability Assessment.
  • Experiments show TRACE reveals that top-performing agents are not always the most efficient or reliable.
  • The framework provides multi-dimensional analysis for developing and comparing reliable Deep Research Agents.
  • The authors plan to use TRACE’s utility function as a reward signal for agent optimization and extend the framework to other complex, long-horizon domains.

A.1 RELATED WORKS

The related work situates TRACE within Deep Research Agent evaluation, where challenging benchmarks test advanced systems but primarily rely on outcome-based measures.

  • The field lies at the intersection of Deep Research Agent evaluation and trajectory-level analysis of sequential decision-making.
  • Deep Research Agent evaluation has advanced through benchmarks such as BrowseComp-en/zh and GAIA.These benchmarks test state-of-the-art agents including AgentFounder, WebSailor-V2, and WebShaper.

A.1.1 Evaluation of Deep Research Agents.

Existing Deep Research Agent evaluation relies heavily on final-answer metrics, which overlook reasoning-process quality and provide limited insight into agent failures.

  • Pass@1-based evaluation creates a “high-score illusion” by ignoring reasoning-process quality, efficiency, and robustness.
  • Outcome-only evaluation cannot distinguish intelligent problem solving from correct answers reached through lucky or inefficient paths.
  • Process-aware analysis is central to reinforcement-learning training but largely absent from agent evaluation.
  • Final-score evaluation leaves researchers unable to diagnose whether failures arise from poor planning, flawed reasoning, or inability to recover from errors.

A.1.2 Trajectory-level Analysis.

TRACE adapts trajectory-level principles from reinforcement learning into a formal evaluation suite that probes not only agent performance, but also how and why it occurs.

  • TRACE systematically adapts reinforcement-learning principles into a formal evaluation suite.
  • Its diagnostic tools include Scaffolded Capability Assessment and Entropy Adaptability.These tools extend evaluation beyond what an agent can do toward understanding how and why it performs.

A.2.1 Justification for the Hierarchical Trajectory Utility Function.

The hierarchical utility function is justified by combining sensitivity properties that emphasize weakest-link failures, diminishing cost impact, novel information, evidence grounding, and recovery robustness.

  • The weighted geometric mean becomes unboundedly more sensitive than the arithmetic mean as a component score approaches zero, supporting a weakest-link aggregation principle.The theorem contrasts the geometric and arithmetic means near critical component failure.
  • The logarithmic cost functional is strictly concave, so each additional unit of cost contributes less to the overall value than the previous unit.Its negative second derivative establishes diminishing sensitivity for positive total cost.
  • Marginal Information Gain is positive exactly when the current observation’s relevance exceeds the supremum of all prior relevances.The proposition defines gain as the nonnegative difference between current relevance and historical maximum relevance.
  • Evidence Grounding has unbounded sensitivity when any claim’s entailment probability approaches zero, sharply penalizing an ungrounded claim.The sensitivity is expressed through the reciprocal of the relevant entailment probability.
  • Reasoning Robustness is normalized between 0 and 1 and decreases strictly as average recovery latency increases.The exponential form reaches 1 at zero latency and approaches 0 as latency grows.
Loading 2602.21230v1…