Source-linked AI summary
AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
Kunlun Zhu, Xuyan Ye, Zhiguang Han, Yuchen Zhao, Bingxuan Li, Weijia Zhang, Muxin Tian, Xiangru Tang, Pan Lu, James Zou, Jiaxuan You, Heng Ji
TL;DR
As LLM agents become more capable, their failures become substantially harder to debug. AgentDebugX closes this gap with a Detect–Attribute–Recover–Rerun workflow, while DeepDebug delivers strongest attribution across tested open-weight backbones and repairs more failed GAIA tasks in one rerun than decoupled baselines.
Problem
As LLM agents become more capable, their failures become substantially harder to debug.
Method
AgentDebugX organizes debugging as Detect, Attribute, Recover, and Rerun, with DeepDebug performing multi-turn root-cause diagnosis through trajectory reading, structured probing, and cross-examination.
Results
DeepDebug achieves the strongest attribution performance on both tested open-weight backbones and repairs 13 of 73 failed GAIA trajectories in one rerun, versus 4–6 for three baselines.
Takeaways & Limitations
Systematic attribution and recovery evidence can inform audit trails, deployment gates, regression tests, and decisions about when an agent should not be trusted.
Takeaways & Limitations
Diagnostic labels and recovery suggestions can be wrong, so high-impact deployments require confidence, evidence, and human or policy approval rather than automatic recovery.
Abstract
from arXiv · showhide
LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it. Existing observability tools replay execution traces but provide little support for identifying the root cause or translating diagnosis into recovery. We present AgentDebugX, an open-source debugging framework that organizes debugging as a closed loop of Detect, Attribute, Recover, and Rerun. At its core, DeepDebug performs multi-turn root-cause diagnosis through global trajectory understanding, structure-guided investigation, and cross-examination. On the Who and When benchmark, DeepDebug achieves the best strict attribution accuracy among the evaluated methods on both tested open-weight backbones, reaching 28.8 percent exact agent-and-step accuracy on qwen3.5-9b versus 21.7 percent for the strongest single-pass baseline. On GAIA, DeepDebug repairs 13 of 73 failed tasks in a single rerun, compared with 4 to 6 for three decoupled self-correction baselines, improving overall accuracy from 55.8 percent to 63.6 percent. AgentDebugX exposes this workflow through a Python library, CLI, web console, and installable agentic skill, and provides an opt-in Error Hub for sharing scrubbed failure-diagnosis-repair bundles and reusing them as debugging memory.
1 Introduction
AgentDebugX frames agent debugging as a closed loop of Detect, Attribute, Recover, and Rerun, addressing failures whose visible symptoms may arise long after their causes. Its DeepDebug diagnostic agent localizes root causes, proposes fixes, and supports repair, while the Error Hub preserves reusable debugging records.
- AgentDebugX workflow: AgentDebugX organizes debugging into Detect, Attribute, Recover, and Rerun, converting live executions or exported logs into portable trajectories.Detection associates observable failures with structured failure modes before attribution traces symptoms backward to their causes.
- DeepDebug: DeepDebug performs multi-turn root-cause diagnosis through global trajectory reading, structure-guided probing, and cross-examination of conflicting candidates.It traces handoffs in multi-agent runs or bisects single-agent traces, then produces an auditable report identifying the responsible agent and step, evidence, explanation, and one concrete fix.
- Error Hub: The opt-in Error Hub stores scrubbed trajectory–diagnosis–repair bundles for incident records, regression fixtures, reusable debugging memory, and cross-team comparison.Its framework-independent format supports retrieving similar historical failures and accumulating reviewed examples without modifying original execution evidence.
- Evaluation: 28.8% strict agent-and-exact-step accuracy is achieved by DeepDebug on qwen3.5-9b, compared with 21.7% for the strongest single-pass baseline.This is the strongest attribution performance among evaluated methods on both tested open-weight backbones.
- Evaluation: 13 failed GAIA tasks are repaired in a single rerun using DeepDebug’s diagnosis.The evaluation measures both failure localization and conversion of diagnosis into a successful repair.
2 Related Works
Prior work offers strong tools for observing, diagnosing, or correcting agent failures, but rarely unifies detection, attribution, and verified recovery. AgentDebugX connects these capabilities, using attribution to support more reliable error correction when failure locations are known.
- Observability: Observability platforms capture and replay detailed agent traces but leave developers to identify the responsible step, explain the cause, and repair the run.The passage names LangSmith, Langfuse, and Phoenix as examples.
- Error Correction: Revision methods such as AutoManual study how models revise unsuccessful behavior and complement AgentDebugX’s attribution-based setting.The passage characterizes these methods as a parallel line of work focused on correcting unsuccessful behavior.
- Error Correction: Models correct errors far more reliably when the error’s location is supplied, which is the information AgentDebugX’s attribution provides.The passage explicitly presents attribution as supplying the error location for correction.
3 System Overview
AgentDebugX organizes agent debugging as a closed loop over portable execution trajectories: Detect, Attribute, Recover, and Rerun. DeepDebug provides structured, read-only root-cause diagnosis and recovery suggestions, while shared interfaces support console, CLI, library, skill, and Error Hub workflows.
- Closed-loop workflow: AgentDebugX coordinates Detect, Attribute, Recover, and Rerun through structured outputs and inspectable artifacts, escalating difficult cases to DeepDebug.Detection finds manifestations; attribution localizes responsibility; recovery proposes a retry; rerun evaluates a new trajectory and can re-enter detection.
- Portable trajectory representation: An AgentTrajectory records ordered AgentEvents with agents, modules, steps, ancestry, inputs, outputs, metadata, errors, and artifacts across supported runtimes and offline logs.Adapters produce one framework-independent representation, while diagnoses remain layered on the evidence for re-analysis, comparison, and regression sharing.
- Stage strategies: Detection uses deterministic rule packs and, when needed, an LLM judge with typed findings; attribution returns ranked hypotheses, while recovery strategies remain suggest-only.The shared failure taxonomy seeds 19 modes, and deployments can trade attribution accuracy against latency and token cost.
- DeepDebug diagnosis: DeepDebug combines global trajectory reading, structure-guided investigation, and cross-examination to select between causal hypotheses before emitting an auditable diagnosis and concrete fix.It uses handoff-cascade tracing for multi-agent runs, bisection for single-agent runs, and never re-executes the run’s tools.
- Interfaces and shared memory: The shared data model powers the library, local web console, CLI, and installable agentic skill, while Error Hub scrubs exported bundles and reuses accepted cases as debugging memory.The console supports policy-gated rerun branches and comparison; future diagnosis can retrieve similar cases, though this memory effect is not yet evaluated.
4 Evaluation
Evaluation tests AgentDebugX in pipeline order: accurate failure attribution followed by end-to-end repair. DeepDebug improves strict Who&When attribution and converts diagnoses into substantially more GAIA repairs, with 1.6× single-pass token cost.
- Setup: The qwen3.5-9b Who&When evaluation covers all 184 traces, with reference answers provided but gold responsible-agent and mistake-step labels withheld.Diagnosis runs use gemini-2.5-flash at temperature 0 with thinking disabled, while diagnostic memory and the Error Hub remain empty.
- Cost-aware selection: DeepDebug averages 12.8K tokens versus 8.1K for one whole-trace pass, a 1.6× cost that focused-window later turns can reduce through selective escalation.Every attributor returns ranked hypotheses with provenance rather than a single authoritative blame claim.
- Failure attribution: DeepDebug achieves 28.8% strict agent-and-step accuracy on qwen3.5-9b, versus 21.7% for the best single strategy.On Who&When, it also reaches 56.0% responsible-agent accuracy and 28.8/44.0% exact/near-step accuracy, versus 47.8% and 22.3/38.6%.
- End-to-end recovery on GAIA: DeepDebug’s multi-turn diagnosis delivers 2–3× more repaired GAIA failures than decoupled self-correction, improving overall accuracy by 7.9 points.The GAIA evaluation diagnoses 73 vanilla-failed tasks and reruns each once, comparing AgentDebugX with Reflexion, CRITIC, and AutoManual.
5 Use Cases and Applications
AgentDebugX supports debugging workflows that capture failed executions, diagnose and repair their causes, and retain results for controlled reruns and reusable memory. The workflow applies to interactive debugging, CI regression testing, and incident response.
- Debugging workflow: A typical session captures or imports a failed execution before DeepDebug localizes the responsible step, explains the root cause, and proposes a repair.The proposed repair can be reviewed before rerunning.
- Debugging workflow: The resulting trajectory is retained alongside the original and can optionally be stored in the Error Hub as reusable debugging memory.Reruns are policy-controlled, allowing repair decisions to be reviewed before execution.
- Applications: The workflow supports interactive debugging, CI regression testing, and incident response.
6 Conclusion
AgentDebugX is a closed-loop debugging framework connecting failure detection, root-cause attribution, recovery, and rerun for LLM agents. Its core method, DeepDebug, links improved fault attribution to measurable downstream recovery gains.
- Conclusion: AgentDebugX connects failure detection, root-cause attribution, recovery, and rerun in a closed-loop framework for LLM agents.The framework is presented as a practical foundation for debugging increasingly capable LLM agents.
- Conclusion: DeepDebug shows that improving fault attribution can produce measurable gains in downstream recovery.This result connects diagnosis quality with subsequent repair performance.
- Conclusion: AgentDebugX is intended to facilitate future research on reliable agent development.The authors position the toolkit as a foundation for continued work on debugging increasingly capable LLM agents.
Ethical Considerations
AgentDebugX handles sensitive agent traces through local-first defaults and opt-in sharing, while requiring production safeguards and uncertainty-aware presentation of its potentially fallible diagnoses and recovery suggestions.
- AgentDebugX collects potentially sensitive traces, including prompts, tool arguments, user data, files, screenshots, and outputs.
- Production deployments should configure redaction, retention, access control, and audit logging before collecting user data.
- Diagnostic labels and recovery suggestions can be wrong, so the UI and API should present confidence and evidence rather than ground-truth attribution.
Broader Impact · A System and Prompt Details · B Deployment Requirements
AgentDebugX frames debugging as an operational and community practice, with structured traces, constrained diagnosis prompts, actionable recovery, and deployment requirements centered on portability, privacy, and cost control.
- Broader Impact: Systematic attribution and recovery evidence can support audit trails, deployment gates, regression tests, and decisions about when agents should not be trusted.The framework is positioned for domains including software development, science, education, accessibility, and public-facing services, where silent or repeated errors may propagate.
- Broader Impact: The Error Hub can turn one team’s scrubbed failure into another team’s regression case, supporting shared benchmarks, taxonomies, methods, and robustness standards.This requires consent, provenance, effective redaction, moderation, and takedown processes.
- A System and Prompt Details: AgentTrajectory records typed events, agents, modules, step indices, parents, timestamps, inputs/outputs, errors, durations, metadata, and artifacts.Artifacts may contain text, images, audio, UI state, files, or environment snapshots, while events can project to OpenTelemetry GenAI spans.
- A System and Prompt Details: The judge prompt conservatively labels failed steps with one allowed failure-mode code and returns JSON containing event, step, agent, confidence, evidence, and summary fields.It caps findings, limits evidence length, and returns an empty list when no failure is found.
- A System and Prompt Details: Whole-trace localization identifies a failed span, step, and agent with confidence, rationale, and short evidence, while returning null fields when no failure appears.Cross-examination arbitrates between two candidate error steps by selecting the decisive critical step whose correction would most likely avert failure.
- B Deployment Requirements: Five production requirements are low-friction capture, portable OpenTelemetry-compatible representation, typed diagnoses, local-first scrubbing, and cost-aware opt-in LLM analysis.Deterministic triage is free, while deeper LLM analysis is optional.
C Evaluation Protocol
The evaluation uses full Who&When traces for attribution and GAIA validation tasks for end-to-end recovery, with fixed decoding, exact matching, and defined rerun protocols. A measured ablation tests DeepDebug’s turn design across hosted and open-weight backbones.
- Benchmarks: 184 Who&When traces, including 126 algorithm-generated and 58 hand-crafted examples, are annotated with the responsible agent and mistake step.GAIA validation covers 165 tasks across three difficulty levels and uses the official question scorer for recovery.
- Attribution protocol: Every attribution method receives the reference answer but not gold agent or step, with temperature-0 decoding, token caps, normalized agent names, and exact step-index matching.Reported metrics include responsible-agent accuracy, exact and ±1 step localization, and joint agent-and-step accuracy across open-weight and hosted backbones.
- GAIA recovery protocol: 73 failed GAIA trajectories are diagnosed and rerun once using either native DeepDebug or the decoupled Reflexion, CRITIC, and AutoManual strategies.The initial vanilla Open-Deep-Research run spans all 165 validation tasks before restricting recovery to the failure subset.
- Localizer design ablation: 0.310 to 0.262 strict accuracy on gpt-5.4-mini results when structure-guided investigation is replaced by a second global search.The shipped pairing raises agent accuracy to 0.524 versus 0.429 for one reading; adjudication helps evaluated open-weight but not hosted backbones, showing model dependence.
D Implementation
AgentDebugX is a dependency-light, MIT-licensed Python package with a compact public API and one-context-manager adoption path. It persists traces in JSONL or SQLite and normalizes runtime, offline, and host-integrated captures into a source-independent schema for debugging.
- Package and API: AgentDebugX is a dependency-light, MIT-licensed Python package installed with pip and imported as agentdebug.Its public API centers on AgentDebug, TraceSession, AgentTrajectory, FailureFinding, and DiagnosticReport.
- Package and API: One context manager captures a named goal, agent, step index, and error before analysis produces a report.The example traces a browser agent encountering a checkout timeout at step 3, then calls t.analyze().
- Trace capture: Traces persist to append-only JSONL or SQLite.The same schema is emitted by runtime adapters, offline importers, and host integrations.
- Trace capture: Runtime adapters, offline importers, and host integrations make detection, attribution, and recovery source-independent.Supported inputs include ReAct, LangChain/LangGraph, CrewAI, OpenAI Agents SDK, OpenTelemetry GenAI, message lists, conversations, event lists, WebShop pages, spans, events, sessions, and agent skills.
E Limitations
The evaluation focuses on automatic attribution and recovery rather than developer debugging time or console usability, with several benchmark and experimental-design constraints. DeepDebug’s gains vary by model, while the GAIA setup does not isolate attribution’s effect or provide a blind single-shot score.
- Evaluation scope: The evaluation covers automatic attribution and recovery, not developer debugging time or console usability.This leaves human debugging effort and interface usability outside the reported evaluation.
- Benchmark constraints: DeepDebug’s extra calls are not uniformly beneficial because its Who&When gains vary by model.Who&When follows the benchmark’s reference-answer protocol.
- Experimental design: The GAIA experiment evaluates one policy model on a fixed failed subset and compares complete retry recipes.The design neither isolates attribution’s effect alone nor reports a blind single-shot score.