PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

Jaineet Shah

arXiv:2606.08275v1cs.LGcs.AI

TL;DR

When an LLM agent fails, existing tools do not identify which step caused the outcome, and transcript-based attribution can be unreliable. CAR intervenes on steps and replays the stochastic trajectory; on synthetic SCMs it recovers a pivotal step and estimates a two-step interaction close to its analytic efficiency value.

  • Problem

    For a bad LLM-agent outcome, existing observability and evaluation tools do not answer which step caused it, while Who&When LLM-judge attribution accuracy is about 14%.

    Source: §1 · p. 1

  • Method

    CAR models a run as an SCM, resamples a step under the same stochastic policy, and measures the resulting outcome distribution, using a point-of-commitment rule for single-step attribution.

    Source: §2 · p. 2, §1 · p. 2, §4 · p. 3

  • Results

    The contrastive estimator recovers the pivotal step, and the Shapley efficiency sum is 0.909 versus the analytic value 0.91 in a two-step interaction SCM.

    Source: §5 · p. 3

  • Takeaways & Limitations

    CAR answers step-causation questions by intervention rather than correlation and is validated against synthetic ground truth.

    Source: §1 · p. 2, §8 · p. 5

    Real tools with side effects are out of scope; hosted inference can vary, so replay reproducibility is reported as an action-match rate rather than assumed.

    Source: §3 · p. 2, §7 · p. 5

Abstract

from arXiv · show

When an LLM agent fails -- issues a refund it should not have, calls the wrong tool, leaks data -- existing tooling answers what happened (observability) or whether it passed (evaluation), but not which step caused the failure. The obvious heuristics are wrong: the step that executes the harmful action is usually not the step that decided on it, and LLM-judge attribution is correlational and unreliable (state-of-the-art step-level accuracy on the Who&When benchmark is about 14%). We present Causal Agent Replay (CAR), which answers the question by intervention: it models an agent run as a structural causal model, applies a do-operation to a step, and re-executes the trajectory forward under the same stochastic policy, measuring the shift in the outcome distribution. We define an intervention algebra over agent steps, a single-step contrastive estimator whose point-of-commitment rule resolves a confound specific to stochastic run-forward, and a budget-bounded Monte-Carlo Shapley estimator that splits credit across interacting steps. Every effect is reported with confidence intervals. We validate against synthetic structural causal models with planted ground truth: the contrastive estimator recovers the pivotal step, and Shapley recovers a two-step interaction (0.44, 0.45, ~0; efficiency sum 0.909 versus the analytic 0.91). CAR is open source and runs on hosted or free local models.

1 Introduction

When an LLM agent produces a bad outcome, identifying the step that caused it determines what to fix, but the harmful action may only execute an earlier decision. CAR addresses this attribution problem through causal interventions and estimators for individual and interacting steps.

1 Introduction

  • LLM-judge step-level attribution reaches only about 14% accuracy on Who&When, while the harmful tool call can be the mechanical consequence of an earlier decision.The introduction frames locating the causal decision—not merely observing the action—as the question developers need answered.Source: §1 · p. 1
  • CAR models an agent run as a structural causal model and intervenes on a step to test whether the outcome changes.This contrasts causal intervention with observability and evaluation, which the paper says do not make this move.Source: §1 · p. 1
  • Because the policy is stochastic, interventions produce outcome distributions rather than single paths, and CAR reports confidence intervals.Source: §1 · p. 2
  • The single-step contrastive estimator uses a point-of-commitment rule to address downstream stochastic steps being rerolled during run-forward.Source: §1 · p. 2
  • A budget-bounded Monte-Carlo Shapley estimator splits credit across interacting steps, and CAR is validated against synthetic SCMs with planted ground truth.Source: §1 · p. 2

2 An agent run as a structural causal model

CAR represents an agent trajectory as a structural causal model whose policy actions and environment observations lead to a user-scored outcome. Its interventions alter one step and re-execute downstream decisions to estimate outcome distributions.

2 An agent run as a structural causal model

  • The trajectory model treats the agent's state as its decision context, its action as a stochastic-policy draw, its observation as the tool result, and its outcome as a user-supplied score in [0, 1].The paper situates this causal modeling approach in the causal-influence-diagram program [7] [8].Source: §2 · p. 2
  • An intervention changes one variable and re-decides everything downstream, producing an outcome distribution across K forward runs.Source: §2 · p. 2
  • CAR defines five operations: resample an action, force an action, replace an observation, edit context, or swap the policy from a step onward.The null do_resample operation redraws the action from the unchanged policy to measure that step's intrinsic causal sensitivity.Source: §2 · p. 2

3 Faithful replay and provider nondeterminism

Faithful replay depends on reconstructing the exact state from which the agent decided, while providers' inference can vary even under ostensibly deterministic settings. CAR therefore reports replay action-match rates rather than asserting reproducibility.

3 Faithful replay and provider nondeterminism

  • Faithful replay records nondeterministic inputs so deterministic glue can be re-executed, while recording the model call output as the irreducible nondeterministic input.Source: §3 · p. 2
  • Hosted inference can vary even at temperature 0, so CAR reports an action-match rate instead of claiming reproducibility.The passage attributes variation to floating-point non-associativity and batch-size-dependent kernels.Source: §3 · p. 2
  • A single-stream local model with a fixed seed replays exactly, avoiding shared hosted inference's batch variance.Source: §3 · p. 2

4 Attribution

The section develops two complementary attribution methods: contrastive resampling to locate a pivotal step and Shapley values to share credit across interacting steps.

4 Attribution

  • For each step, the contrastive estimator holds earlier factual actions fixed, resamples that step, runs forward K times, and estimates the bad-outcome probability shift with confidence intervals.It uses a Wilson interval for the proportion and a bootstrap interval for the difference.Source: §4 · p. 3
  • Resampling a step re-rolls downstream stochastic decisions, so early irrelevant steps can appear influential; the causal locus is the latest step whose effect confidence interval excludes zero.This point-of-commitment rule marks the last step where re-deciding can still rescue the run.Source: §4 · p. 3
  • Shapley values average each step’s marginal contribution across coalition contexts, splitting credit when a failure occurs only through joint step errors.For an AND-failure, single-step effects can double-count responsibility or make both steps appear irrelevant.Source: §4 · p. 3
  • The budget-bounded Monte-Carlo Shapley estimator samples permutations with reverse-permutation pairing and avoids caching coalition values to preserve marginal-contribution variance.It avoids truncated sampling that could skip a pivotal late step and uses a circuit breaker.Source: §4 · p. 3

5 Validation against ground truth

On planted synthetic causal models, Shapley attribution approximates the analytic total for a two-step interaction, while contrastive attribution recovers a planted pivotal step.

5 Validation against ground truth

  • In a three-step SCM, contrastive attribution recovers the middle pivotal decision, and resampling downstream shows no significant effect.The result matches the point-of-commitment semantics.Source: §5 · p. 3
  • A two-step interaction yields a Shapley efficiency sum of 0.909 against the analytic value 1 −q2 = 0.91, while contrastive effects over-count the true total contribution.This demonstrates why the paper uses both estimators.Source: §5 · p. 3
  • The support-agent injection report pairs a resampling trajectory and outcome distribution with per-step effects and confidence intervals, highlighting the causal locus and its decision context.The injection is visible in the context the locus decided from.Source: Figure 1 · p. 4, §5 · p. 4

6 Related work

The paper distinguishes CAR from prior agent-failure attribution approaches by its intervention procedure, uncertainty reporting, locus rule, interaction handling, and ground-truth validation.

6 Related work

  • Who&When [1] defines failure attribution and reports LLM-judge baselines; AgenTracer [2] uses oracle action substitution and a trained scorer, while Ma et al. [3] pair Shapley with causal discovery over static logs.CAR does not claim novelty for counterfactual replay or Shapley for agent blame in the abstract.Source: §6 · p. 4
  • CAR’s stated distinction is same-policy do_resample interventions, distributional outcomes with confidence intervals, and a point-of-commitment rule for the run-forward confound.It also combines Shapley credit-splitting with ground-truth validation.Source: §6 · p. 4
  • The paper situates its framing in do-calculus, counterfactual credit assignment [4], and causal influence diagrams [7] [8].Source: §6 · p. 4

7 Limitations

The stated limitations concern the interpretation of stochastic effects, outcome measurement, tool setting, and Shapley computation cost.

7 Limitations

  • Contrastive effects measure total effects through a stochastic continuation; isolating direct effects would require common random numbers across branches, left as a refinement.Source: §7 · p. 5
  • Judge-based outcome functions add noise, so the paper prefers rule-based outcomes for results intended to be trusted.Source: §7 · p. 5
  • Demonstrations use mocked, reproducible tools; real tools with side effects are out of scope, and Shapley’s budget-bounded Monte-Carlo estimator retains a variance-versus-budget tradeoff.Source: §7 · p. 5

8 Conclusion

Causal Agent Replay (CAR) identifies failure-causing steps through intervention rather than correlation, modeling agent runs as structural causal models. It measures outcome-distribution shifts and was validated against synthetic ground truth.

  • CAR models an agent run as a structural causal model and resamples a step under the same stochastic policy to measure outcome-distribution shifts.Source: §8 · p. 5
  • A locus rule handles the run-forward confound, while a Shapley estimator accounts for interactions between steps.Source: §8 · p. 5
  • Validated against synthetic ground truth, CAR is open source and runs on hosted or free local models.Source: §8 · p. 5
Loading 2606.08275v1…