Source-linked AI summary
When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
Chenkai Zhang, Yiran Li, Yifang Tian, Michalis Bachras, Hans-Arno Jacobsen
TL;DR
Existing evaluations reveal little about how or why agentic executions fail, especially when faults arise at operational boundaries rather than in reasoning. AGENTCHAOSBENCH addresses this gap with controlled fault injection and telemetry-based diagnosis across five multi-agent systems. The evaluated LLMs remain unreliable, with type-and-location accuracy reaching at most 22% and guardrail bypass remaining unresolved by aligned references.
Problem
Existing failure-attribution studies do not directly test whether telemetry can distinguish externally induced runtime faults, identify their types, and localize affected components.
Method
AGENTCHAOSBENCH injects plausible faults across tool, model, guardrail, and inter-agent boundaries, then evaluates diagnosis from structured telemetry with injection markers and labels removed.
Results
Type-and-location AC@1 reaches at most 22%, while aligned references improve selected faults but not guardrail bypass.
Takeaways & Limitations
The benchmark supports reproducible evaluation of agent-system diagnosis and motivates reference-based and specialized detectors.
Takeaways & Limitations
The benchmark covers only five systems with five aligned inputs per condition, limiting broader task and input diversity.
Abstract
from arXiv · showhide
Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails. We present AGENTCHAOSBENCH, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry. We run five heterogeneous applications that coordinate agents over the Agent-to-Agent protocol and call tools through the Model Context Protocol, and inject ten types of operational fault (unavailable or slow tools, corrupted or oversized responses, and delayed, looped, or misrouted delegations and bypassed guardrails) at their tool, model, guardrail, and inter-agent boundaries, alongside a no-fault control. The resulting dataset contains 275 sanitized traces: 250 faulty executions spanning ten fault types and 25 no-fault controls. Each faulty trace is aligned with the no-fault execution of the same input; fault-type labels and, where applicable, location labels are held out from diagnosis. On structured single-trace inputs, a first set of zero-shot LLM baselines shows the task is far from solved: local detectors up to 14B parameters reach only 13.6-19.2% top-1 fault-type accuracy and the frontier DeepSeek-v4-pro only 24.8%, while jointly identifying the fault type and its location tops out at 22%; reference-dependent faults (above all a bypassed guardrail) stay near-unsolved from a single trace. An aligned reference improves selected relative faults but does not resolve guardrail bypass. The held-out labels and compact prediction format support reproducible comparison of LLM-based and non-LLM diagnosis methods.
1 Introduction
AgentChaosBench evaluates whether LLM-based agentic systems can diagnose runtime faults from sanitized execution telemetry rather than judging reliability only from final answers. It covers faults across tool, model, guardrail, and inter-agent boundaries using aligned faulty and fault-free traces.
- Research gap: The benchmark addresses an operational gap left by prior failure-attribution work, which primarily diagnoses agent actions or reasoning rather than runtime faults at infrastructure and interaction boundaries.Production-like executions expose faults including unavailable or slow tools, corrupted responses, context overflows, and delayed or misrouted inter-agent requests.
- Benchmark motivation: AgentChaosBench introduces a fault-injection benchmark for evaluating the diagnosability of LLM-based agentic systems.The benchmark targets operational runtime faults that may propagate through executions and surface as incorrect answers, excessive latency, or stalled workflows.
- Benchmark construction: The benchmark spans five heterogeneous systems, ten fault types, and 275 controlled executions with same-input alignment between faulty and fault-free runs.It includes a no-fault control and faults such as tool failure, latency, context overflow, output corruption, misrouting, and guardrail bypass.
- Diagnosis formulation: Diagnosis is formulated as fault-type classification and localization of the responsible span or component from telemetry with fault markers and labels removed.Captured traces include agent, model, and tool spans plus timing, inputs, outputs, and status metadata; injected fault information is retained separately as ground truth.
2 Benchmark Design and Construction
AgentChaosBench is a controlled, trace-based benchmark that diagnoses injected runtime faults across five heterogeneous A2A/MCP agentic systems. Its reproducible construction spans operational fault classes, sanitized telemetry, and 275 aligned cases with hidden labels and no-fault controls.
- Fault Model: The benchmark targets production-relevant failures across LLM calls, tools, guardrails, agents, and inter-agent communication rather than synthetic reasoning perturbations.Fault mechanisms include availability, performance, control and routing, and data or payload failures; the taxonomy is intentionally broad but not exhaustive.
- Workloads and Deployment: Five selected applications use real MCP tools, A2A delegation, multi-step executions, and deterministic success conditions across CrewAI, LangGraph, and AutoGen variants.The heterogeneous A2A configuration standardizes protocol boundaries while preserving different framework assignments across workflows.
- Fault Injection: Boundary interceptors inject faults transparently at tools, model calls, guardrails, and A2A delegations while recording ground truth outside the detector’s trace.Argument isolation prevents control and routing faults from destabilizing generation, and genuine targets ensure faults produce observable effects.
- Trace Instrumentation and Release: Sanitized Langfuse traces expose span hierarchy, timing, status, identity, content, guardrail decisions, and A2A interactions without injection markers or label-leaking fields.The released detector input retains production-observable telemetry while removing storage paths and other direct fault-label signals.
- Case Generation and Quality Control: 275 cases cross five agentic systems, ten fault types plus no-fault controls, and five aligned task inputs per condition.Each system contributes 55 cases: 50 faulty and 5 no-fault, with every faulty case paired to a same-input fault-free reference.
3 Evaluation
Evaluation shows that zero-shot LLMs struggle to diagnose runtime faults from single traces, with performance limited by model capacity and trace representation. Localization remains open, while aligned references improve selected fault types but do not resolve guardrail bypass or output corruption.
- RQ1: Fault-type diagnosis: 13.6–19.2% AC@1 for local Qwen detectors and 24.8% AC@1 (34.0% AC@3) for DeepSeek-v4-pro show fault-type diagnosis remains near the 9% random baseline.Local models remain essentially flat across an order of magnitude of size, while the 14B detector misses the correct fault type in its top-three ranking for 65% of faulty cases.
- RQ1: Trace representation: Structured traces outperform raw traces for Qwen3.5-9B, reaching 19.2%/28.0% versus 15.2%/22.5% AC@1/AC@3, while long raw traces exceed smaller context windows.Qwen3.5-9B processes 82% of cases in its 262K-token window, but 40% of raw-view outputs contain no parseable ranking; DeepSeek-v4-pro can ingest all raw traces.
- RQ3: Localization: Component localization and joint type-and-location diagnosis remain open problems on structured traces.Localization is evaluated over cases whose labels carry a location, with separate component-location and type-and-location metrics.
- RQ4: Reference condition: Aligned No Fault references improve top-3 recall for selected fault types but do not consistently improve diagnosis overall.The reference condition compares each faulty trace with the aligned known-normal execution of the same input using Δ = paired − single on matched cases.
- RQ4: Reference condition: Guardrail Bypass changes by −5 to 0 points with a reference, while Output Corruption decreases by 5–15 points.A benign input can make a forged guardrail pass indistinguishable from the reference, whereas plausible incorrect outputs may lack a consistent textual difference indicating corruption.
4 Related Work
Prior work established benchmarks for multi-step agents, white-box agent systems, and telemetry-based observability. AgentChaosBench builds on these directions by combining observability with controlled fault injection and held-out ground truth.
- Agentic systems and observability: AgentBench [11] and AI-NativeBench [15] benchmark agents across multi-step environment interaction and white-box systems combining agents with MCP tools and A2A communication.These benchmarks motivate evaluating agentic systems beyond final-answer checking as they become distributed software.
- Agentic systems and observability: AgentTelemetry argues that agent observability should expose identity, guardrails, delegation, and timing/status metadata through a fault-detection toolkit.AgentChaosBench adopts this observability stance while adding controlled fault injection and held-out ground truth.
- Fault injection and stress testing for agents: Chaos-engineering ideas have begun reaching agents, with ChaosLLM injecting unreachable, slow, hanging, and incorrect responses at the tool boundary.This prior work provides context for AgentChaosBench’s focus on operational faults in agentic executions.
5 Discussion
The benchmark provides balanced coverage across 275 verified cases, but broader task and input diversity is needed to strengthen external validity. Internal-validity safeguards remove label leakage, verify injected fault signals against aligned no-fault references, and correct failed injections at their source.
- Limitations: 275 verified cases cover five systems and five aligned inputs per condition with balanced per-fault coverage, but broader task and input diversity would strengthen external validity.The authors plan to scale both task and input diversity.
- Limitations: The baselines are zero-shot general-purpose LLMs—four local Qwen models and frontier DeepSeek-v4-pro—evaluated on structured and raw trace views, leaving rule-based, few-shot, and specialized attribution methods for future work.
- Internal validity: Internal-validity checks remove injection markers and label-bearing fields, verify each faulty case against an aligned no-fault reference, and fix failed injections at their source rather than dropping them.These procedures are intended to ensure injected faults are real and their identities remain hidden without biasing the dataset toward easy cases.
6 Conclusion and Future Work
AgentChaosBench enables reproducible diagnosis of ten runtime fault types across five multi-agent systems using 275 sanitized, aligned traces. Evaluated LLMs remain unreliable, while aligned references help selected faults but not semantic or policy-judgment failures such as guardrail bypass.
- Benchmark: AgentChaosBench provides 275 sanitized traces covering ten runtime fault types across five multi-agent systems for reproducible diagnosis evaluation.Controlled injections, aligned no-fault executions, verified labels, and automated scoring support reproducible evaluation.
- Results: At most 24.8% single-trace fault-type AC@1, 31% component-location AC@1, and 22% type-and-location AC@1 demonstrate that evaluated LLMs remain unreliable.These are the strongest reported accuracies across the evaluated diagnosis tasks.
- Results: Aligned references improve context-overflow accuracy by up to 55 points and help selected latency and routing faults, but not semantic or policy-judgment faults including guardrail bypass.A compact structured view makes long traces tractable without artificially improving the strongest detector.