PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, Chetan Bansal

arXiv:2602.02475v2cs.AI

TL;DR

AI-agent failures are difficult to localize in long-horizon, tool-mediated executions, motivating systematic failure attribution. The paper builds a grounded-theory benchmark and AgentRx, which synthesizes and validates execution constraints for evidence-based diagnosis. AgentRx improves failure-step localization by 75% on average over prior work while also providing root-cause categories.

  • Problem

    AI-agent failures are difficult to localize because executions are long-horizon, multi-agent, probabilistic, and mediated by noisy tool outputs, while manual attribution is costly and hard to scale.

  • Method

    AgentRx normalizes agent logs, synthesizes constraints from tool schemas, domain policies, and trajectory prefixes, and produces an evidence-backed validation log for an LLM judge.

  • Results

    75% average improvement in failure localization over prior work, with root-cause categorization also improved over the authors’ judge baselines.

  • Takeaways & Limitations

    The benchmark and framework support cross-domain attribution of the first unrecoverable agent failure using trajectory-level constraints and auditable evidence.

  • Takeaways & Limitations

    The taxonomy categories were shaped by trajectories from three domains and require a held-out check for transfer beyond those domains.

Abstract

from arXiv · show

AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an $\textit{automated diagnostic framework}$ that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.

1 Introduction

Agent failures are difficult to debug because autonomous executions are long-horizon, tool-mediated, and prone to propagating errors. AgentRx addresses this challenge by combining an annotated benchmark with automated, evidence-grounded localization and categorization of critical failures.

  • Autonomous agents operate in high-stakes environments, but long-horizon executions and noisy tool outputs make failures difficult to debug.Failures can propagate through the trajectory before they are observed.
  • The paper defines the critical failure as the first unrecoverable failure by any agent in an execution trajectory.
  • The benchmark contains 170 failed trajectories across 11 diverse task settings, with each trajectory annotated by a critical step and failure category.Settings include structured API workflows, incident management, and open-ended web/file tasks.
  • AgentRx normalizes heterogeneous logs, synthesizes constraints from schemas, policies, and trajectories, and creates step-indexed validation evidence for an LLM judge.The judge uses the validation log to localize the critical step and assign a category.
  • 75% average improvement in failure localization over prior work accompanies a 30% improvement in root-cause categorization over the authors’ judge baselines.

2 The AGENTRX Benchmark

The AGENTRX benchmark evaluates critical-failure attribution across diverse agent settings and uses grounded-theory annotation to derive a cross-domain taxonomy. It records complete failure chains while distinguishing the earliest unrecoverable step from later failures.

  • 2 The AGENTRX Benchmark: The benchmark contains 170 failed trajectories across 11 settings spanning APIs, incident management, software development, mathematical reasoning, and open-ended web/file tasks.The settings vary by agent configuration, environment structure, and failure-chain recoverability.
  • 2 The AGENTRX Benchmark: The benchmark combines trajectories from τ-bench, Flash, Magentic-One, MAST, and Who&When-derived domains.The included systems cover single-agent tool calling, multi-agent incident diagnosis, and generalist web/file tasks.
  • 2 The AGENTRX Benchmark: Failure timelines mark every detected failure by step and category, while stars identify root-cause steps; Magentic-One has 295 failures across trajectories, compared with 55 for Flash and 52 for τ-bench.68% of Magentic-One trajectories contain at least two failures.
  • 2.1 Grounded Theory Annotation of Failures: Annotators used grounded-theory open coding so failure categories emerged from trajectories rather than from a predefined label set.They independently read traces step-by-step and recorded structured coding memos for observed failures.
  • 2.1 Grounded Theory Annotation of Failures: The two-stage procedure first marks all failures, then identifies the earliest failure in the causal chain from which the agent does not recover.The critical step determines the trajectory’s root-cause category, with coding iterated toward theoretical saturation.
  • 2.1 Grounded Theory Annotation of Failures: The taxonomy assigns exactly one category to each critical step and distinguishes plan deviation from forming the wrong plan from a correct intent.

3 AgentRx Framework Formulation

AgentRx formulates failure diagnosis as stepwise constraint validation over normalized agent trajectories. It combines global and dynamic constraints with an auditable violation log and an LLM judge for critical-step and category attribution.

  • 3 AgentRx Framework Formulation: Table 1 reports critical-failure-label distributions across domains using within-domain percentages of failed trajectories.The table’s comparison is therefore between normalized label distributions rather than raw cross-domain counts.
  • 3 AgentRx Framework Formulation: AgentRx accepts heterogeneous toolsets, optional domain policies, and failed trajectories, then normalizes logs into a common intermediate representation.Each trajectory step can contain agent names, tool calls, outputs, and conversational content.
  • 3.1 Global and Dynamic Constraint Synthesis: Global constraints are synthesized once from tool schemas and policies, while dynamic constraints are generated per step from the instruction and observed trajectory prefix.Global constraints encode domain rules; dynamic constraints encode trajectory-specific rules and prior observations.
  • 3.1 Global and Dynamic Constraint Synthesis: The framework standardizes constraint generation in a lightweight schema and evaluates constraints incrementally across trajectory steps.The formulation combines constraints available at each step from global and locally synthesized stores.
  • 3.2 Guarded Constraints and Evaluation: Each constraint has a guard that determines applicability and an assertion that returns a satisfaction or violation verdict.Guards capture structural conditions such as tool calls or agent invocations.
  • 3.2 Guarded Constraints and Evaluation: Applicable constraints are checked programmatically over structured fields or semantically with an LLM-based checker, and both produce verdict–evidence pairs.Examples include schema validity, equality, membership, and natural-language predicates.
  • 3.3 Validation Log: AgentRx records violated constraints and supporting evidence in a step-indexed validation log that is auditable and directly linked to each violation.The log stores the step, constraint, and evidence associated with violations.
  • 3.4 Judge for Root Cause Attribution: An LLM judge uses the instruction, trajectory, taxonomy checklist, and validation log to select the first sufficiently explanatory violation and assign its category.Violations are diagnostic evidence rather than hard requirements, so the judge may override them when trajectory context indicates otherwise.

4 Experimental Evaluation

AgentRx is evaluated as a trajectory-level diagnosis system for localizing the first unrecoverable failure and assigning root-cause categories across diverse agent settings. Experiments compare validation signals, constraint-generation strategies, judging protocols, prior benchmarks, and robustness under relaxed metrics.

  • Comparison to Prior Benchmarks: AgentRx improves critical-step localization over W&W across domains, with especially large gains on τ-bench and RelWork, while also predicting root-cause categories.W&W was adapted to the first unrecoverable failure definition; AgentRx additionally improves failed-agent identification in most settings.
  • Impact of Violations and Checklist: Violation evidence and the taxonomy checklist are complementary: violations ground error localization, while combining both signals supports reliable attribution across domains.Violation evidence is especially useful on τ-bench and RelWork, the checklist on Flash, and their combination on Magentic.
  • Impact of Constraint Generation: Step-by-step constraint generation consistently outperforms one-shot generation across domains, reducing Magentic average step distance from 22.5 to 12.4.The advantage is most visible on longer trajectories, where one-shot generation is more vulnerable to context dilution.
  • Impact of Constraint Generation: One-shot AgentRx costs USD 0.18 per trajectory versus USD 0.05 for the judge-only baseline, about 3.6× higher, with roughly 81% of overhead from constraint generation.The paper presents one-shot generation as a cost-efficient option when full-trajectory context is sufficient, while step-by-step analysis suits detailed debugging.
  • LLM-as-a-Judge Evaluation: Step-then-Category often improves category accuracy but can compromise step localization because categorization depends on a potentially noisy first-stage step.All-at-Once is the default AGENTRX judging protocol.
  • Robustness Across Relaxed Metrics: AgentRx remains useful under strict and relaxed failure-attribution metrics, with step accuracy increasing as tolerance expands from Acc@±1 to Acc@±3 to Acc@±5.The evaluation distinguishes critical, any-failure, earliest, and terminal category accuracy.
  • Qualitative Failure Analysis: Surfaced violations can misdirect diagnosis by emphasizing invocation checks over an infrastructural system failure or by highlighting downstream symptoms instead of the root cause.The paper describes optional reflection and pruning to remove likely false-positive constraints before judging.

5 Related Work

AgentRx is positioned against benchmarks for agent evaluation, runtime assurance, plan conformance, and LLM-based judging by diagnosing failures from trajectory evidence.

  • Existing benchmarks evaluate tool execution, web interaction, assistant behavior, and agents across diverse interactive environments.
  • AgentRx assigns the first unrecoverable critical step and a cross-domain failure category in its benchmark.
  • AgentRx differs from runtime-assurance and plan-conformance methods by diagnosing trajectories through constraints extracted from tools and policies.
  • AgentRx shifts LLM judging from scoring trajectories to diagnosing them with evidence, while remaining compatible with agent-as-a-judge paradigms.

6 Conclusion

AgentRx frames agent evaluation around identifying where recovery becomes impossible, not merely whether an execution fails. Its constraint-based diagnosis supports evidence-grounded explanations of when and why failures occur.

  • AgentRx evaluates agents by locating the point where recovery becomes impossible rather than only determining whether they fail.
  • The framework generates constraints and assigns root-cause categories to failed agent trajectories.
  • AgentRx moves evaluation toward evidence-grounded debugging that explains when and why an agent failed.

7 Limitations

AgentRx’s taxonomy spans diverse domains but may omit failure modes beyond the evaluated settings, and noisy or false-positive validation signals can mislead its judge.

  • The taxonomy covers diverse domains and uses annotations from three independent annotators, but may require extension to other agentic domains.
  • Weak or false-positive validation signals can mislead AgentRx’s judge, motivating work on smaller sets of higher-quality signals.
  • AgentRx relies on LLM calls but reduces manual attribution effort and supports scalable failure analysis.

Ethical Considerations

The paper presents diagnosis as a practical aid for debugging and reliability measurement, while emphasizing that explanations do not guarantee safety and trajectory logs may contain sensitive information.

  • Agent failures can arise from wrong tool calls, skipped confirmations, misread outputs, or decisions that lead to unrecoverable paths.
  • Connecting root causes to trace evidence can support faster debugging, clearer accountability, and safer iteration.
  • Better diagnosis does not automatically make agents safe to deploy, and explanations may be over-trusted as guarantees.
  • Trajectory data may contain sensitive information, so users should handle execution logs carefully and follow appropriate privacy practices.
  • The taxonomy checklist operationalizes categories through decision criteria and yes/no questions applied to candidate failure steps.
  • Instruction/Plan Adherence Failure concerns deviations from a required plan, including missed steps and unnecessary actions.
  • Invention of New Information concerns unsupported introduced, removed, or altered information in the agent’s reasoning or actions.
  • Invalid Invocation covers malformed tool calls involving missing arguments, incorrect fields or types, malformed queries, or schema mismatches.

B Hierarchical Judging Protocols

AgentRx compares joint and hierarchical judging protocols, finding that All-at-Once is generally strongest while two-stage decomposition offers selective gains. Constraint evidence benefits some judges and domains more than others.

  • Violation evidence helps o3 where coverage is dense, while o3 remains competitive without it on context-heavy domains.The cited comparison covers τ-bench, Flash, Magentic, and RelWork.
  • DeepSeek-V3.2 produces substantive, checkable constraints across domains, with constraints improving category accuracy consistently on Flash and τ-bench.The reported examples include 64 violations on Flash, 356 on τ-bench, and 148 on RelWork.
  • Both two-stage protocols suffer from error propagation because the first decision constrains the second.
  • All-at-Once remains the default because it matches or exceeds both two-stage variants almost everywhere.The exception is Flash step localization, where decomposition reaches 88.3% versus 82.5%.
  • 88.3% vs. 82.5% is the one reported case where decomposition beats the joint judge for Flash step localization.
  • Category-then-Step outperforms Step-then-Category on τ-bench and Flash but underperforms on RelWork and Magentic.Long, nested traces make early category commitment especially costly in RelWork and Magentic.

C Root-Cause Attribution VS First Failure

The benchmark distinguishes recoverable early errors from the first unrecoverable root cause. A Magentic trajectory shows why attribution should focus on the failure that prevents recovery rather than simply the earliest observed failure.

  • The WebSurfer access failure at step 3 was misidentified as causal because the orchestrator recovered from it.The trajectory instead identifies a later FileSurfer hallucination as the decisive failure.
  • Figures 3 and 4 compare framework-level failure-category distributions and critical-step positions across the 1,022-trajectory MAST dataset.Figure 3 normalizes each row independently; Figure 4 marks means in red and medians in blue.
  • The critical failure was FileSurfer’s step 33 hallucination, which saved a file to the wrong path and made subsequent recovery impossible.

D Failure Annotation for Magentic One

The Magentic annotation illustrates how the benchmark records many observed failures while separately identifying a root cause. Its example trajectory contains 22 failures spanning WebSurfer retrieval and Orchestrator planning problems.

  • 22 failures are annotated in the example Magentic trajectory.
  • Repeated WebSurfer failures involve not obtaining information for the specified date, including arrival-time information.
  • Several WebSurfer failures are labeled Instruction/Plan Adherence Failure because required information was not obtained.
  • Orchestrator failures repeatedly involve pursuing email or phone contact despite misinterpreting the user’s intent.
  • The trajectory’s root cause is the WebSurfer’s inability to retrieve arrival-time information, which led to repeated Orchestrator misinterpretations.
  • A separate example marks a Kusto connection and syntax error as a System Failure root cause.
  • Other annotations include authentication and incorrect tool-result counting, with the assistant eventually recovering from authentication but not the incorrect count.

E.7 Guardrails Triggered (Magentic) {

In the Magentic guardrails example, direct access to DataVault pricing is blocked by bot detection and CAPTCHA, so the agent cannot retrieve current tier information through automated browsing.

  • The task asks the agent to compare current Pro and Enterprise pricing and features for DataVault.
  • A 403 Forbidden response and CAPTCHA block automated access to DataVault’s pricing page.The page attributes the denial to bot detection and Cloudflare Bot Management.
  • The agent searches alternative sources, but review sites require login, list contact-for-pricing details, or provide outdated cached information.
  • The agent therefore cannot retrieve current, accurate pricing information for the requested tiers.
  • The annotation treats the event as an external site access restriction rather than a failure of the agent’s plan.

E.8 Intent Plan Misalignment (τ-bench)

The τ-bench examples show intent-plan misalignment and invalid invocation when the assistant pursues an impermissible cancellation plan or supplies invalid tool arguments. AgentRx’s judge checks tool-call scope, prior action descriptions, explicit user confirmation, identifier consistency, and unambiguous evidence.

  • Intent-Plan Misalignment (τ-bench): At step 33, the assistant cancelled the entire pending order after incorrectly assuming that a subset could be cancelled under domain policy.The root cause was the whole-order cancellation, which differed from the ground-truth action of cancelling only the garden hose.
  • Intent-Plan Misalignment (τ-bench): At step 17, the assistant made an invalid modify-order invocation because it omitted the required replacement argument.The agent did not recover from the illegal tool call.
  • Intent-Plan Misalignment (τ-bench): At step 21, the assistant repeated an invalid modify-order invocation by trying to bypass the argument restriction with the same item_id.This was categorized as another invalid invocation rather than a valid correction.
  • Intent-Plan Misalignment (τ-bench): At step 30, the assistant failed to modify the address even though that action was part of the overall task instruction.The supplied evidence identifies a mismatch between the explicit user request and the assistant’s handling of the task.
  • AGENTRX Judge Prompt: AgentRx represents failure categories and constraints using structured assertion fields, event triggers, check hints, and executable or natural-language checks.The taxonomy includes Instruction/PlanAdherenceFailure, InvalidInvocation, and IntentPlanMisalignment among other categories.
  • AGENTRX Judge Prompt: The judge evaluates whether write-action calls were preceded by an assistant description, explicit user confirmation, and matching order_id or user_id values.Criteria with insufficient or ambiguous evidence are marked UNCLEAR and discarded; only definitive violations receive CLEAR_FAIL.

I Open Source Licensing

The paper’s benchmark and related datasets are distributed under permissive open-source licenses, with τ-bench, Magentic-One, Who&When, and the benchmark under MIT and MAST under CC-BY 4.0.

  • Open Source Licensing: τ-bench, Magentic-One, Who&When, and the benchmark are released under the MIT License, while MAST is released under CC-BY 4.0.The MIT license notice grants broad rights to use, copy, modify, publish, distribute, sublicense, and sell the software subject to its conditions.
  • Open Source Licensing: The software is provided “as is” without express or implied warranties, including merchantability, fitness for a particular purpose, and noninfringement.The notice also disclaims liability for claims, damages, or other liability arising from the software or its use.
Loading 2602.02475v2…