Source-linked AI summary
AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, Shuicheng Yan
TL;DR
Multi-agent systems are capable but fragile, and existing failure attribution is inaccurate and costly to annotate. AgenTracer automates trajectory labeling and trains AgenTracer-8B with multi-granular reinforcement learning; it improves benchmark attribution and enables gains in deployed agentic systems.
Problem
Failure attribution in long multi-agent trajectories remains inaccurate and largely manual, while existing benchmarks provide limited annotated data.
Method
AgenTracer uses counterfactual replay and programmatic fault injection to build TracerTraj, then trains AgenTracer-8B with multi-granular reinforcement learning.
Results
AgenTracer-8B outperforms GEMINI-2.5-PRO by ∼18.18% on Who&When and improves off-the-shelf agentic systems by 4.8 ∼14.2%.
Takeaways & Limitations
The framework supports failure diagnosis and actionable feedback for self-correcting and self-evolving multi-agent systems.
Abstract
from arXiv · showhide
Large Language Model (LLM)-based agentic systems, often comprising multiple models, complex tool invocations, and orchestration protocols, substantially outperform monolithic agents. Yet this very sophistication amplifies their fragility, making them more prone to system failure. Pinpointing the specific agent or step responsible for an error within long execution traces defines the task of agentic system failure attribution. Current state-of-the-art reasoning LLMs, however, remain strikingly inadequate for this challenge, with accuracy generally below 10%. To address this gap, we propose AgenTracer, the first automated framework for annotating failed multi-agent trajectories via counterfactual replay and programmed fault injection, producing the curated dataset TracerTraj. Leveraging this resource, we develop AgenTracer-8B, a lightweight failure tracer trained with multi-granular reinforcement learning, capable of efficiently diagnosing errors in verbose multi-agent interactions. On the Who&When benchmark, AgenTracer-8B outperforms giant proprietary LLMs like Gemini-2.5-Pro and Claude-4-Sonnet by up to 18.18%, setting a new standard in LLM agentic failure attribution. More importantly, AgenTracer-8B delivers actionable feedback to off-the-shelf multi-agent systems like MetaGPT and MaAS with 4.8-14.2% performance gains, empowering self-correcting and self-evolving agentic AI.
1 INTRODUCTION
Multi-agent systems improve capability through coordination but become fragile and difficult to debug. AgenTracer automates failure-trace annotation and trains a lightweight tracer that improves attribution and downstream system performance.
- Multi-agent systems outperform single-agent counterparts across complex domains but their integration increases fragility and failure risk.
- Failure attribution identifies the faulty component after failure and supports debugging, data efficiency, and grounded self-improvement.
- Existing automation remains weak, with attribution accuracy below 10%, while available benchmarks contain only 200 and 127 manually annotated trajectories.
- AgenTracer constructs annotated trajectories through counterfactual replay and programmatic fault injection.
- AgenTracer-8B uses multi-granular reinforcement learning to attribute failures at both step and agent levels for automatic debugging.
- ∼18.18% higher performance than GEMINI-2.5-PRO on Who&When accompanies 4.8 ∼14.2% gains for off-the-shelf agentic systems.
2 RELATED WORKS
Related work covers multi-agent system design, agent-failure attribution, and LLM-based evaluation or credit assignment. These areas establish the context for automated diagnosis while leaving limitations in attribution effectiveness and LLM-system credit assignment.
- Multi-agent systems range from handcrafted to partially automated configurations that automate selected system components.
- Failure-attribution research has cataloged fourteen failure patterns and introduced benchmarks including MAST and Who&When.
- LLM-as-a-Judge & Credit Assignment: LLM-as-a-Judge uses language models as evaluators, but has shown limited effectiveness in multi-LLM systems.
- LLM-as-a-Judge & Credit Assignment: Credit assignment links individual agent actions to long-term outcomes, yet remains largely unexplored for LLM-based multi-agent systems.
3 PRELIMINARY
The paper models a turn-based multi-agent system as agents selecting actions from local spaces while system dynamics update the shared state. Failure attribution seeks the earliest action whose correction changes failure to success.
- The system contains N agents operating in discrete time, with exactly one scheduled agent active at each step.
- Each active agent selects an action from its local action space using its policy, current state, query, and available interaction history.
- The interaction history is implementation-dependent and may include all agents’ prior outputs or only a latest submitted code snippet.
- A trajectory records alternating states and actions through a terminal step, with the final response determined by the complete trajectory.
- The decisive error is the earliest action whose oracle correction is sufficient to steer a failed trajectory to success.
- AgenTracer outputs the failure-responsible agent and decisive error step, selecting the earliest error among decisive agent-step pairs.
4 METHODOLOGY
AgenTracer automatically annotates decisive failures in multi-agent trajectories, then trains AgenTracer-8B with multi-granular reinforcement learning to attribute errors at both agent and step levels.
- 4.1 AgenTracer: AUTOMATIC TRAJECTORY ANNOTATION: TracerTraj-2.5K combines over 2,000 annotated trajectory–error step pairs from six multi-agent frameworks and six datasets.The pipeline applies programmatic fault injection to successful trajectories and counterfactual corrections to failed ones.
- 4.1 AgenTracer: AUTOMATIC TRAJECTORY ANNOTATION: Counterfactual intervention searches for the earliest corrected step whose replayed trajectory satisfies the success condition, labeling its active agent as problematic.An analyzer uses the failed trajectory, environmental feedback, and ground-truth solution to propose minimally invasive corrected actions.
- 4.1 AgenTracer: AUTOMATIC TRAJECTORY ANNOTATION: Programmatic fault injection perturbs known-good trajectories, retaining cases that induce failure and assigning the injection point’s active agent and step as the decisive error.This construction provides positive samples whose decisive errors are known by construction.
- 4.2 AgenTracer-8B: TRAINING AGENTIC FAILURE TRACERS: AgenTracer-8B is trained from QWEN3-8B with online GRPO reinforcement learning on the curated dataset to locate decisive errors in long-horizon trajectories.The policy generates groups of candidate agent–step pairs evaluated against ground-truth annotations.
- 4.2 AgenTracer-8B: TRAINING AGENTIC FAILURE TRACERS: The multi-granular reward combines strict format compliance with agent-level correctness and smoothly graded step-level proximity to the true error.Step-level reward supplies partial credit, while format gating ensures reliably parsable outputs.
- 4.2 AgenTracer-8B: TRAINING AGENTIC FAILURE TRACERS: The resulting failure attributor uses multi-granular reinforcement learning to support accurate attribution at both step and agent levels.Its design is intended to produce a reasoning-based failure attributor for multi-agent systems.
5 EXPERIMENTS
Experiments evaluate AgenTracer-8B on unseen and held-out failure-attribution subsets, compare it with models of varying scales, and test whether its feedback improves existing agentic systems. Results show robust attribution advantages and substantial gains over reflection-based baselines.
- Experimental Setup: TracerTraj-2.5K covers six multi-agent systems spanning manual, partial, and full automation, with benchmarks drawn from coding, mathematics, and agentic domains.The curated data supports evaluation across diverse system configurations and task domains.
- Experimental Setup: Who&When includes handcrafted and automated subsets with unseen trajectories, while TracerTraj evaluation uses held-out code, math, and agentic splits.The TracerTraj split is sampled at a 9:1 ratio.
- Main Results: Prevailing models remain inadequate: step-level accuracy stays below 10% for smaller models on Who&When (handcrafted), while DEEPSEEK-R1 and GPT-4.1 reach only 31.32% and 29.52% on Who&When (automated) with G.Ground-truth access does not consistently improve attribution accuracy.
- Main Results: AgenTracer-8B surpasses larger models, improving TracerTraj-agentic step-level accuracy over QWEN3-8B by 22.68%, DEEPSEEK-R1 by 9.04%, and GEMENI-2.5-PRO by 17.57%.The comparison is reported under the w/ G setting.
- Boosting Mainstream MAS: 14.21% is the gain on MaAS+MATH-500, while AgenTracer-8B improves OWL on GAIA by 4.8% and steadily improves outcomes across the evaluated settings.CRITIC instead reduces CRITIC+MaAS+GAIA accuracy by −4.9% at iteration-2 and −5.5% at iteration-3.
6 CONCLUSION
The paper presents AgenTracer and AgenTracer-8B as a foundation for automated failure attribution in agentic systems. Its evaluations report stronger benchmark diagnosis and performance gains when deployed in multi-agent frameworks.
- Conclusion: AgenTracer systematically generates annotated failure trajectories, while AgenTracer-8B uses multi-granular reinforcement learning for failure tracing.The framework is described as automated, and the model as lightweight.
- Conclusion: AgenTracer-8B surpasses proprietary LLMs on Who&When and yields consistent performance gains in real-world multi-agent frameworks.The conclusion positions these results as progress toward self-correcting and self-evolving agentic systems.
A DATASET DETAILS
TracerTraj dataset statistics organize curated trajectories and annotated error-step pairs across three domains and their associated multi-agent systems.
- Dataset Statistics: Table 3 reports each domain’s included benchmarks, curated trajectory count, and number of trajectories annotated with TracerTraj-2.5K error-step pairs.The table also lists the associated multi-agent systems.
B PROMPT SET
The prompt set specifies analyzer and attack workflows for identifying critical failure steps, agents, causes, and implementable interventions. It also prescribes structured JSON outputs and required file-operation sequences.
- Diagnosis Prompt: The diagnosis workflow asks for the mistaken step and agent, a detailed reason, and guidance for fixing the error without providing the complete solution.The prompt uses task information, prior diagnosis information, and the original execution history as inputs.
- File Operations: The prompts require creating the output file before writing its contents and using the end command only after both file operations succeed.Separate workflows specify diagnosis and attack-analysis filenames based on the task identifier.
- Attack Prompt: The attack-analysis workflow identifies a critical step in a successful execution and specifies concrete code changes intended to cause failure.It requires an attack step, mistake agent, original content, attack content, and reason in JSON format.