Source-linked AI summary
When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
Chenkai Zhang, Yiran Li, Yifang Tian, Michalis Bachras, Hans-Arno Jacobsen
TL;DR
Evidence remains limited on detecting and localizing externally induced runtime faults from agent execution telemetry. AGENTCHAOSBENCH studies this task through controlled fault injection across five agentic systems, using sanitized traces and aligned no-fault references. Evaluated LLMs remain unreliable at classification and localization, with single-trace fault-type AC@1 reaching at most 24.8%.
Problem
Evidence remains limited on whether agent execution telemetry can distinguish normal behavior from externally induced runtime faults and identify their type and location.
Source: §1 · p. 1Method
Results
Evaluated LLMs remain unreliable across fault classification and localization, with single-trace fault-type AC@1 reaching at most 24.8%.
Source: §6 · p. 8Takeaways & Limitations
The findings motivate reference-based and specialized detectors, non-LLM baselines, and exact-span localization for agent-system diagnosis.
Source: §6 · p. 8The benchmark’s limited task and input diversity constrains external validity, with coverage restricted to five systems and five aligned inputs per condition.
Source: §5.1 · p. 8
Abstract
from arXiv · showhide
Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails. We present AGENTCHAOSBENCH, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry. We run five heterogeneous applications that coordinate agents over the Agent-to-Agent protocol and call tools through the Model Context Protocol, and inject ten types of operational fault (unavailable or slow tools, corrupted or oversized responses, and delayed, looped, or misrouted delegations and bypassed guardrails) at their tool, model, guardrail, and inter-agent boundaries, alongside a no-fault control. The resulting dataset contains 275 sanitized traces: 250 faulty executions spanning ten fault types and 25 no-fault controls. Each faulty trace is aligned with the no-fault execution of the same input; fault-type labels and, where applicable, location labels are held out from diagnosis. On structured single-trace inputs, a first set of zero-shot LLM baselines shows the task is far from solved: local detectors up to 14B parameters reach only 13.6-19.2% top-1 fault-type accuracy and the frontier DeepSeek-v4-pro only 24.8%, while jointly identifying the fault type and its location tops out at 22%; reference-dependent faults (above all a bypassed guardrail) stay near-unsolved from a single trace. An aligned reference improves selected relative faults but does not resolve guardrail bypass. The held-out labels and compact prediction format support reproducible comparison of LLM-based and non-LLM diagnosis methods.
1 Introduction
AgentChaosBench evaluates runtime-fault detection and localization from sanitized execution telemetry across tool, model, guardrail, and inter-agent boundaries. Initial LLM baselines achieve low diagnosis accuracy, while aligned fault-free references help selected faults but leave guardrail bypass unresolved.
1 Introduction
- Final task outcomes obscure operational failures, motivating telemetry-based diagnosis that distinguishes normal behavior, identifies runtime fault types, and localizes affected components.Operational faults can propagate into incorrect answers, excessive latency, or stalled workflows without an incorrect reasoning step. Diagnosis therefore requires controlled faults, system-level telemetry, and ground truth independent of interpreting the final answer.Source: §1 · p. 1
- Sanitized Langfuse traces expose agent, model, and tool spans while withholding injection markers and fault labels from detectors.Traces retain timing, inputs, outputs, and status metadata, with verified type and location labels stored separately as ground truth. A compact prediction format supports reproducible comparisons of LLM-based and non-LLM diagnosis methods.Source: §1 · p. 2
- 13.6–19.2% top-1 fault-type accuracy for local detectors up to 14B parameters and 24.8% for DeepSeek-v4-pro show limited diagnosis performance.Top-1 component-localization accuracy reaches only 31%, while jointly correct type and location stays at 8–22%.Source: §1 · p. 2
- 275 controlled executions across five systems cover ten runtime fault types plus no-fault controls, with same-input alignment between faulty and fault-free runs.AgentChaosBench injects faults at tool, model, guardrail, and inter-agent boundaries using applications derived from AI-NativeBench [15]. Its taxonomy combines explicit failures with subtle semantic corruptions.Source: §1 · pp. 1–2
- Up to 55 points of context-overflow recall improvement comes from an aligned fault-free reference, which also helps selected latency and routing faults.Reference-dependent faults, especially guardrail bypass, remain near-unsolved from a single trace even for the frontier model. Guardrail bypass remains unresolved with an aligned reference.Source: §1 · p. 2
2 Benchmark Design and Construction
AgentChaosBench injects operational faults into five heterogeneous agentic systems and tests fault classification and localization from sanitized execution telemetry. Aligned fault-free runs support trace-grounded labels and quality control.
2.1 Design Goals
- Design goals prioritize production-relevant faults, component coverage, reproducibility, leakage-free inputs, trace-grounded labels, and automated scoring.Diagnosis methods must classify faults and localize affected spans or components from telemetry stripped of injection markers and fault labels.Source: §2.1 · p. 2, §2 · p. 2
2.2 Agentic Systems and Task Selection
- Five AI-NativeBench [15] applications combine CrewAI, LangGraph, and AutoGen agents through A2A communication and MCP tool access.Selection requires real tool interactions, cross-agent delegation, multi-step executions long enough for faults to propagate, and deterministic success conditions.Source: §2.2 · pp. 2–3
2.3 Deployment and Execution Environment
- A shared local Qwen3.5-9B endpoint served through vLLM uses greedy decoding to support reproducible executions independent of external LLM providers.Agents run as independent A2A services under an orchestrator. Bounded timeouts and retries at A2A and tool boundaries remain observable behavior that detectors must interpret.Source: §2.3 · p. 3
2.4 Fault Model
- 10 fault types span availability, performance, control/routing, and data/policy failures across tool, model, agent, inter-agent, and guardrail boundaries.The taxonomy describes injected mechanisms, not diagnosis difficulty, and is not exhaustive. Holding the model and task input fixed while pairing faulty executions with clean references distinguishes environmental faults from intrinsic reasoning errors.Source: §2.4 · p. 3
2.5 Fault-Injection Mechanism
- Boundary interceptors inject faults transparently to agent logic, using configurations that specify the target, activation condition, and operator.Argument-isolated loop anchors preserve surrounding generation, while tool misroutes must reach genuinely different tool behavior rather than produce no-ops. Activation types and locations enter separate ground-truth records, never detector-visible traces.Source: §2.5 · pp. 3–4
2.6 Observability and Trace Collection
- Langfuse captures hierarchical execution spans exposing agent identity, model and tool activity, guardrail decisions, A2A interactions, timing, status, and filtered content.Release filtering removes injection markers, label-revealing fields, and fault-organized storage paths, retaining production-observable telemetry.Source: §2.6 · p. 4
2.7 Dataset Construction and Specification
- 275 cases cross five systems, ten fault types and a no-fault control, with five aligned task inputs per condition.Automated audits compare faulty traces with clean references and repair invalid injections rather than discard cases. Tool Latency has a +15.0 s median delta, whereas A2A Latency ranges from −3 s to +26 s and is often masked.Source: §2.7 · pp. 4–5
- Roughly an order-of-magnitude reduction in the largest traces comes from compact structured views that neither flag anomalies nor compare baselines.Full traces have a median of ∼108K tokens and reach ∼638K; compact rows expose span metadata, output sizes, repeat counts, and previews, collapsing identical polls only for one outlier. Held-out labels specify fault identifiers, required span kinds, signals, locations, and routing targets; the benchmark, audit, and scoring scripts are planned for public release.Source: §2.7 · p. 5
3 Evaluation
Zero-shot LLMs struggle to diagnose and localize faults from sanitized single-trace telemetry; structured traces improve coverage and reduce inference cost without solving diagnosis. Aligned fault-free references improve selected faults but do not resolve guardrail bypass or output corruption.
3.1 Diagnosis Task and Prediction Format
- Diagnosis requires ranking eleven possible conditions from one unknown-status trace, with short justifications citing telemetry spans.The conditions comprise ten faults plus No Fault, scored using top-1 and top-3 correctness. The paired setting supplies a known-normal execution of the same input, with input and external task state fixed so only the injected fault differs.Source: §3.1 · p. 5
3.2 Baselines
- Zero-shot baselines compare four local Qwen models and frontier DeepSeek-v4-pro using identical prompts and both trace representations wherever context windows permit.Qwen3-1.7B, Qwen3-4B, Qwen3.5-9B, and Qwen3-14B run through vLLM; DeepSeek-v4-pro runs through an API. A rule-based detector over observable signals remains future work.Source: §3.2 · p. 5
3.3 Metrics
- 225 location-labeled faulty cases support component-location and joint type-and-location scoring; fault-type accuracy and no-fault recognition are evaluated separately.Component-location AC@1/AC@3 requires the first or any top-three cited span to resolve to the labeled component after normalization; joint AC@1 requires both type and component correctness. Fault-type AC@1/AC@3 is measured over faulty cases, clean-control recognition proxies false positives, and uniform-random AC@1 is 1/11 ≈9%; Context Overflow and No Fault lack location labels.Source: §3.3 · p. 5
3.4 Experimental Setup
- 275 corrected cases provide identical structured-view samples for every detector, with held-out fields and directory names excluded and greedy decoding throughout.Unparseable rankings count as incorrect; Qwen3-14B has only 2% malformed outputs yet misses the true fault entirely from its top three in 65% of faulty cases. Output failures are length truncations, and doubling the output budget leaves accuracies unchanged; DeepSeek-v4-pro localization uses a separate run with 25.2%/37.6% fault-type AC@1/AC@3.Source: §3.4 · pp. 5–6, §3.5 · p. 6
3.5 Results
- 24.8% AC@1 (34.0% AC@3) for DeepSeek-v4-pro versus 13.6–19.2% AC@1 for local Qwen detectors demonstrates weak single-trace diagnosis; joint type-and-location AC@1 peaks at 22%.Component-location AC@1 is 20%–31% and AC@3 is 27%–45%; type-and-location AC@1 is 8%–16% for local detectors and 22% for DeepSeek-v4-pro. No-fault recognition improves from 4% for Qwen3-1.7B to 96% for Qwen3-14B, versus 88% for DeepSeek-v4-pro, despite weak fault classification.Source: Table 3 · p. 6, §3.5 · p. 6, Table 5 · p. 7
- 25/25 Tool Failure and 23/25 A2A Timeout top-3 recalls make availability faults easiest; directly interpretable evidence matters more than fault-mechanism class.Performance faults and Infinite Loop have observable signals but top-3 recall varies from 0/25 to 13/25; routing faults require intended-destination knowledge, while forged guardrail approvals usually appear fault-free. The smallest detector’s 8/25 Context Overflow recall reflects overprediction and matching false positives rather than reliable diagnosis.Source: Table 4 · p. 6, §3.5 · p. 6
- 8.6× higher raw-trace inference cost buys DeepSeek-v4-pro similar accuracy, while structured traces expand detector coverage.DeepSeek-v4-pro’s raw versus structured AC@1/AC@3 is 21.6%/36.0% versus 24.8%/34.0%, costing $16.45 versus $1.91 across 275 cases. Qwen3.5-9B fits only 82% of raw cases and achieves 15.2%/22.5% AC@1/AC@3 versus 19.2%/28.0% structured, with 40% of raw outputs lacking parseable rankings.Source: §3.5 · p. 6
- +10.7 percentage points in overall AC@3 for Qwen3-14B shows aligned references help selected faults, but not guardrail bypass or output corruption.Overall AC@3 gains are +3.9 points for Qwen3-1.7B and +4.9 for Qwen3-4B; Qwen3-14B gains 55 points on Context Overflow recall and 35 on Tool Latency. Guardrail Bypass changes by −5 to 0 points and Output Corruption decreases by 5–15 points because references cannot establish policy or semantic correctness; Qwen3.5-9B is omitted after invalid rankings on over half of paired cases, even at 8192 output tokens.Source: §3.5 · p. 7
4 Related Work
AgentChaosBench combines telemetry-based observability with controlled fault injection to diagnose external runtime faults across heterogeneous multi-agent boundaries. It complements outcome-focused stress testing and internal failure attribution through held-out ground truth and a no-fault control.
Fault injection and stress testing for agents.
- AgentChaosBench extends stress testing toward trace-based diagnosis across tool, LLM, guardrail, agent, and inter-agent boundaries in multi-framework, multi-agent systems.Agent chaos engineering includes ChaosLLM’s tool-boundary faults and ReliabilityBench’s consistency, task-perturbation robustness, and tool/API fault-tolerance evaluations [7], emphasizing stress and outcome robustness rather than cross-boundary diagnosis. AgentChaosBench removes injection metadata and labels from diagnostic traces.Source: §4 · p. 7
Failure attribution in multi-agent systems.
- AgentChaosBench complements internal failure attribution by targeting external runtime faults, using ground truth independent of final answers and detector interpretations, and including a no-fault control.MAST catalogs behavioral failures [3], AgenTracer attributes failures to agents and steps through counterfactual replay and perturbation [16], spectrum-based methods adapt localization statistics [6], and further work considers multiple attribution perspectives [10]. LLMGuard diagnoses language-model services; AgentChaosBench’s no-fault control makes false positives measurable.Source: §4 · p. 7
Agentic systems and observability.
- AgentChaosBench adopts AgentTelemetry’s observability stance, exposing agent identity, guardrails, delegation, and timing/status metadata while adding controlled fault injection and held-out ground truth.AgentBench establishes broad multi-step interaction tasks [11], while AI-NativeBench extends evaluation to white-box systems combining agents, MCP tools, and A2A communication [15]. Operating such distributed systems reliably requires telemetry beyond final-answer checking.Source: §4 · p. 7
5 Discussion
The benchmark’s limited system and input diversity, unevaluated diagnosis methods, and single workload model constrain generalization, while API nondeterminism limits per-case reproducibility. Internal-validity checks address injection integrity, trace representation, and detector truncation, but broader production validation remains future work.
5.1 Limitations
- 275 verified cases provide balanced fault coverage across five systems with five aligned inputs per condition, but broader task and input diversity remains necessary.Production deployments can mix faults, vary load, and evolve over time. All workloads use one locally served model, leaving cross-model workload generalization and transfer to additional systems unconfirmed.Source: §5.1 · p. 8, §5.2 · p. 8
- 48% of DeepSeek-v4-pro top-1 predictions change on rerun despite temperature 0, although aggregate accuracy remains stable within a point.Its per-case results are therefore treated as a single sample; the local Qwen detectors are fully reproducible.Source: §5.1 · p. 8
- Zero-shot general-purpose LLM baselines leave rule-based detection, few-shot prompting, specialized attribution, and richer localization evaluation unexplored.Four local Qwen models and DeepSeek-v4-pro were evaluated on structured and raw traces. Evaluation covers fault-type diagnosis and component-level localization, but not exact-span scoring or metrics such as MRR.Source: §5.1 · p. 8
5.2 Threats to Validity
- Injection checks conceal fault identities and verify intended signals against aligned controls, repairing failed injections rather than dropping potentially difficult cases.Faulty and fault-free runs share the same input and model endpoint, supporting attribution of observed trace differences to the injected fault.Source: §5.2 · p. 8
- Structured traces expose observability features without anomaly judgments; equal scores for the model reading both views argue against representation-driven accuracy inflation.The paired-trace experiment finds reference benefits concentrated in specific fault types rather than uniformly improving diagnosis.Source: §5.2 · p. 8
- 4096 →8192 output tokens barely changed Qwen3.5-9B truncation (37% →33%) or AC@1 (18.4 →19.2), suggesting larger budgets do not recover these cases.Truncation reaches 33% for Qwen3.5-9B, versus ≤2% for Qwen3-14B and DeepSeek-v4-pro. The authors interpret these as divergent-deliberation cases and argue that scoring them incorrect does not understate ability.Source: §5.2 · p. 8
6 Conclusion and Future Work
AgentChaosBench supports reproducible evaluation of runtime-fault classification and localization across five multi-agent systems. Evaluated LLMs remain unreliable, and aligned references improve selected faults but not those requiring semantic or policy judgment.
6 Conclusion and Future Work
- 275 sanitized traces cover ten runtime fault types across five multi-agent systems in AgentChaosBench.Controlled injections, aligned no-fault executions, verified labels, and automated scoring support reproducible evaluation of fault classification and localization.Source: §6 · p. 8
- 24.8% fault-type AC@1, 31% component-location AC@1, and 22% type-and-location AC@1 are the evaluated LLMs’ maximum single-trace scores.These results show that the evaluated LLMs remain unreliable at runtime-fault diagnosis.Source: §6 · p. 8
- Up to 55 points of improvement on context overflow comes from aligned references, which also improve selected latency and routing faults.References do not improve faults requiring semantic or policy judgment, including guardrail bypass.Source: §6 · p. 8
- A compact structured view makes long traces tractable without artificially improving the strongest detector.Source: §6 · p. 8
- The results motivate reference-based and specialized detectors, non-LLM baselines, and exact-span localization, all supported by the released benchmark.Source: §6 · p. 8