PaperlayerView exact revision on arXiv

Source-linked AI summary

A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems

Wendy Zheng, Liang Wu, Yinhan He, Mayank Darbari, Jundong Li, Liangjie Hong

arXiv:2610.28811v1cs.LG

TL;DR

Direct prompting often performs poorly at identifying the failure-causing step in agentic systems, leaving open whether models lack the capability or fail to express encoded information. The paper uses linear probes to expose this knows-but-won’t-say gap and proposes grafting to route latent information into generation. Across multiple LLMs and benchmark settings, grafting improves behavioral AFA over direct prompting without inference-time ground-truth labels.

  • Problem

    Direct prompting performs poorly at identifying the failing step, leaving unclear whether models lack AFA capability or fail to express information already encoded.

    Source: §1 · pp. 1–2

  • Method

    The paper uses linear probes on model activations to assess encoded failing-step information and proposes grafting to connect that information to output generation.

    Source: §1 · p. 2

  • Results

    Across multiple LLMs and benchmark settings, probes recover substantially more failing-step information than direct prompting, while grafting improves behavioral AFA over prompting.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings suggest AFA limitations depend not only on whether models contain relevant information, but also on whether it can be accessed during generation.

    Source: §1 · p. 2

Abstract

from arXiv · show

Agentic systems have demonstrated strong capabilities on complex tasks by enabling multiple interacting large language model (LLM)-based agents to collaborate and execute subtasks in parallel. However, this collaborative structure introduces new challenges for reliably evaluating and diagnosing system failures, particularly failure attribution: tracing a system's execution to identify the agent step responsible for an incorrect final outcome. Because manually performing this task is time- and labor-intensive, recent work has turned to LLMs as judges to automate the process, referred to as automated failure attribution (AFA). These approaches assume that a sufficiently capable model can infer the failing step directly from an interaction transcript. In practice, however, direct prompting often performs poorly, which motivates follow-up work that explores various ways to improve AFA performance, including alternative prompting strategies and inference-time interventions. In this paper, we examine a more fundamental question: whether this weak behavioral performance reflects a genuine capability limitation. Using linear probes applied to internal activations, we uncover a significant \emph{knows-but-won't-say gap:} models encode sufficient information to attribute a failure correctly but fail to explicitly output the responsible step. Building on this finding, we propose \emph{grafting}, a simple manual intervention that exploits knowledge already encoded in the model to improve AFA performance without requiring ground-truth labels at inference time. Extensive evaluations across multiple LLMs and benchmark settings demonstrate that grafting consistently outperforms direct-prompting approaches, underscoring the potential of leveraging latent model knowledge for more reliable failure attribution. Our code is available at https://github.com/wendyzheng22/mi4afa.

1 Introduction

Direct prompting performs poorly on automated failure attribution (AFA), raising the question of whether models lack the needed capability or fail to express information they already encode. The paper identifies a knows-but-won’t-say gap and proposes grafting to route that latent information into improved predictions.

1 Introduction

  • Manual failure attribution is costly because agentic systems involve many interacting agents and the space of possible interactions grows exponentially with system size.AFA uses an LLM judge to identify the failing step from an interaction transcript.Source: §1 · p. 1
  • Across open-weight LLMs and benchmark settings, direct prompting is a weak baseline, probes reveal more encoded failing-step information, and grafting converts some of it into better predictions.The paper evaluates all three approaches across multiple models and benchmark settings.Source: §1 · p. 2
  • Linear probes outperform prompting baselines across model–dataset combinations, revealing a gap between encoded AFA information and behavioral predictions.The authors interpret this as evidence that direct responses understate models’ AFA-relevant capability.Source: §1 · p. 2
  • Grafting connects components encoding failing-step information to output-generation components, improving AFA over direct prompting without inference-time ground-truth labels.The intervention routes existing information rather than teaching new failure-attribution knowledge.Source: §1 · p. 2

2 Background and Preliminaries

The section defines AFA as identifying the earliest mistake step responsible for an incorrect final output and establishes direct prompting as a behavioral baseline. On Who&When, direct prompting performs poorly, with no consistent model-size trend in step accuracy.

2.1 Automated Failure Attribution

  • AFA predicts the earliest transcript step whose error ultimately causes the system’s incorrect final answer.A prediction succeeds when it matches the annotated mistake step.Source: §2.1 · p. 3
  • Agent accuracy can remain correct when step accuracy is wrong, because repeated appearances of one agent allow a wrong step prediction to identify the right agent.Agent accuracy is therefore coarser than step accuracy, and the metrics need not move together.Source: §2.1 · p. 3

2.2 Direct Prompting

  • Direct prompting is evaluated on Who&When’s short, automatically generated Algo. traces and longer, human-crafted Craft. trajectories.The section establishes direct prompting as the behavioral baseline.Source: §2.2 · p. 3
  • Direct prompting yields low step accuracy across models, and performance does not scale consistently with model size or capability.For example, Qwen 1.5B underperforms Qwen 0.5B on Algo.; closed-source models only modestly outperform much smaller open-weight models on Craft.Source: §2.2 · p. 3

3 Method

The method uses probing to test whether failing-step information is encoded in model activations, then uses grafting to route that information into the model’s output. Probing scores steps from hidden representations; grafting connects heads that locate a step with heads that write the answer.

3.1 Probing: Is the failing step decodable from model activations?

  • Probing tests whether failing-step information is linearly decodable from activations, while leaving the model’s behavior unchanged.The pipeline extracts activations, trains probes, and predicts failing steps.Source: §3.1 · pp. 3–4
  • Each step’s representation is its final-token hidden state, extracted from post-attention and post-MLP residual streams at every transformer layer.These provide 2L candidate representation sites for a model with L layers.Source: §3.1 · p. 4
  • At each representation site, the probe is trained on step states labeled 1 for the ground-truth failing step and 0 otherwise.The probe training set pools states from steps across training transcripts.Source: §3.1 · p. 4
  • At evaluation, probes score every transcript step and predict the highest-scoring one, revealing where failing-step information is most linearly accessible.Grafting begins by identifying attention heads whose attention patterns reliably identify the failing step.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 Grafting: Making the model say what it locates

  • Grafting connects locator heads to writer heads through an inference-time intervention, aiming to improve the model’s own failing-step predictions.It injects a signal in writer heads’ value space so it passes through their usual output projection.Source: §3.2 · pp. 5–6
  • Locator heads are ranked by training-set accuracy and selected from the top 10%, while writer heads are identified by direct logit attribution to the model’s emitted step-index token.At most five locator heads are retained, excluding final-layer heads; writer-head selection uses a clean forward pass rather than the ground-truth mistake step.Source: §3.2 · pp. 5–6

4 Experiments

The experiments compare prompting, probing, and grafting across models and benchmark settings. Probes recover more failing-step information than prompting, while grafting improves behavioral attribution by routing latent information into output generation.

4.1 Experiment Setup

  • The evaluation spans seven open-weight models from Qwen, Llama, and Gemma, using Who&When’s Algo. and Craft. subsets, which average 9 and 29 transcript steps.The prompting baselines are All-at-Once, Step-by-Step, and Sequential.Source: §4.1 · p. 8

4.2 Failure-Attribution Information is Linearly Decodable

  • For Algo., probing reaches 36.51% step accuracy versus 23.81% for the strongest prompting baseline on Qwen 0.5B; it reaches 44.44% and 47.62% for Qwen 1.5B and 7B, versus 15.87% and 14.29%.Comparable gains are reported for Llama and Gemma.Source: §4.2 · p. 8
  • Behavioral baselines vary substantially: no prompting strategy consistently performs best, and larger models do not uniformly improve failure attribution.This pattern suggests that the challenge may involve accessing and expressing information already encoded, not simply acquiring it.Source: §4.2 · p. 8

4.3 Grafting Exploits Latent Information for Improved Performance

  • Probing establishes that failing-step information is encoded, but does not itself provide a way to use it during generation; grafting targets that gap.It connects components encoding failing-step information to output-generation components.Source: §4.3 · pp. 8–9
  • Grafting consistently outperforms prompting; on Algo., it achieves 38.10% step accuracy for Qwen 0.5B and 33.33% versus Step-by-Step’s 15.87% for Qwen 1.5B.It also reaches 30.16% for Qwen 7B versus 14.29% for Sequential, with similar improvements for Llama and Gemma; probing remains stronger in many settings. On Craft., grafting improves over prompting for several models.Source: §4.3 · p. 9

5 Related Works

Related work addresses automated failure attribution, latent information in model representations, and interventions that alter model outputs. This paper connects these strands by testing whether task-specific information already decodable from representations can be routed into generation.

Automated Failure Attribution

Latent Knowledge and Probing

Activation Intervention and Representation Engineering

6 Conclusion

The paper studies failure attribution through the gap between information encoded in LLM representations and information expressed behaviorally. It finds that probes reveal more failing-step information than prompting, and that grafting improves behavioral predictions by routing latent information into generation.

6 Conclusion

  • Linear probes recover substantially more failing-step information than direct prompting, revealing a knows-but-won’t-say gap in LLM-based failure attribution.The gap is between information encoded in representations and information expressed through behavior.Source: §6 · p. 10
  • Grafting connects components encoding failing-step information to output-generation components, improving behavioral AFA over direct prompting without inference-time ground-truth labels.The intervention is presented as leveraging internally encoded knowledge rather than teaching new attribution knowledge.Source: §6 · p. 10
  • The findings suggest that improving LLM-based evaluators may depend on methods that help models access and express knowledge they already possess.Source: §6 · p. 10
Loading 2610.28811v1…