PaperlayerView exact revision on arXiv

Source-linked AI summary

CausalLoss-Fin: Attributing Financial-Agent Loss to Decisions and Infrastructure Faults

Abhishek Sharma

arXiv:2609.25960v1cs.AI

TL;DR

Agent-step attribution cannot identify infrastructure causes when its model has no variable for them. This paper intervenes on both decisions and replayable message faults, decomposes loss exactly, and finds that the agent-only baseline misfiles every infrastructure failure in its planted episodes.

  • Problem

    Agent-step attribution methods expose no infrastructure fault as an intervenable variable, limiting their ability to name infrastructure causes.

    Source: §I · p. 1

  • Method

    The paper decomposes replayable delivery schedules into named message instances and jointly intervenes on infrastructure faults and agent decisions.

    Source: §I · p. 1, §IV · p. 2, B. Interventions · p. 3

  • Results

    The agent-only baseline misfiles every infrastructure failure and charges $114,383.40 to decisions on the planted episodes.

    Source: §XI · p. 7

  • Takeaways & Limitations

    Minimal sufficient repair recovers 100.0% of available loss at 2.3 replays per episode, compared with 280.1 for full per-message allocation.

    Source: §XI · p. 7

    The evaluation uses deterministic procedures rather than language-model agents, so whether the conclusions hold for stochastic agents is untested.

    Source: §X · p. 7

Abstract

from arXiv · show

When an agent handling a payment exception loses money, the agent-step attribution methods this paper compares against will name one of its actions. They will do so even when a settlement message was dropped and the agent never had a chance: they intervene on agent actions and do not expose infrastructure faults as intervenable variables, so every dollar they explain is charged to a decision. We take a benchmark whose fault process is explicit and replayable, decompose each episode's realised delivery schedule into named, individually repairable messages, and intervene on both the agent's choices and the infrastructure's. A telescoping identity splits any policy's loss exactly three ways: an infrastructure effect, a policy differential against the best implementable policy, and a reference-policy residual. Two of the three can be negative, so none is a share; Shapley then divides the first into signed allocations over individual messages. One result is structural and needs no corpus: an agent-only baseline identifies no infrastructure cause, because its model contains no variable that could name one. What 545 planted episodes across 3 policies measure is the size of that consequence. It misfiles 100% of infrastructure episodes and charges $114,383.40 to the agent. Repairing what it names recovers 0.0% of the available loss; repairing a minimal sufficient set recovers 100.0%. Scoring messages one at a time is not merely imprecise: 27.8% (95% CI: 23.3--32.3%) of episodes do not decompose additively. We evaluate deterministic programmatic policies rather than language-model agents, which is what makes replay exact and which limits external validity to stochastic agents. The prevalence figures are properties of this generator, not field rates.

I. INTRODUCTION

Agent-step attribution cannot name infrastructure faults, even when a dropped message leaves an agent without the information needed to act. This paper makes the distinction checkable by intervening on both agent decisions and replayable infrastructure faults, then decomposing loss exactly.

I. INTRODUCTION

  • Agent-only causal models cannot identify infrastructure causes because they contain no intervenable variable that names one.The paper frames this as a structural property of the model, not a result that depends on running a corpus.Source: §I · p. 1
  • The paper intervenes jointly on agent decisions and individual infrastructure faults in the same episode, using exact common random numbers.Its benchmark has an explicit, seeded, replayable fault process, making message-restoration counterfactuals executable.Source: §I · p. 1
  • A telescoping identity splits any policy’s loss exactly into an infrastructure effect, a policy differential, and a reference-policy residual.The identity held on all 3,852 episodes checked.Source: §I · p. 1
  • The fault process is decomposed into individually repairable message instances because fault families compose rather than superpose.This makes per-family attribution unsound in the benchmark.Source: §I · p. 1
  • The benchmark plants four ground-truth strata, including overdetermined cases and verified harmless distractors, to test whether methods can name causes nontrivially.The paper also measures the agent-only baseline’s identification, allocation, dollar, and repair-recovery errors when infrastructure is at fault.Source: §I · p. 1
  • The attributed policies are deterministic procedures, which permits exact interventions but narrows what the results establish about reasoning agents.The paper states that no language model was evaluated.Source: §I · p. 2

II. RELATED WORK

Prior counterfactual attribution work intervenes on agent steps, while some environment-intervention work evaluates diagnostic accuracy rather than allocating realised costs. This paper positions message-level signed allocation as addressing that narrower gap.

II. RELATED WORK

  • AgenticRAG-FP [5] intervenes on retrieval faults but measures diagnostic accuracy, whereas Causely [6] supplies causal structure to agents rather than allocating loss after the fact.The paper distinguishes both from post-hoc attribution of each fault’s share of realised cost.Source: §II · p. 2
  • DCFA [7] measures step-level accuracy in multi-agent failures, while MP-Bench [8] highlights that failures can admit several plausible attributions.The paper draws a parallel between overdetermined message causes and competing agent explanations.Source: §II · p. 2
  • The paper uses minimal sufficient sets—the smallest message sets whose joint repair removes loss—as a practical treatment of overdetermined causes.Its search is capped at three messages; the reported largest set found had two, and the cap bound on none of 320 episodes.Source: §II · p. 2
  • Standard symmetric Shapley values [11] divide the infrastructure effect among messages, while the paper notes warnings about asymmetric variants [12].Shapley allocates an effect established by the exact identity; it does not estimate that effect.Source: §II · p. 2

III. THE SETTING

The setting makes counterfactual replay exact by fixing the case, seed, policy, and delivery schedule. The paper then represents schedule faults as named message instances and models how they affect system views, agent actions, effects, and loss.

III. THE SETTING

  • Each FinalityBench episode is one order observed through four systems’ faulted delivery streams and scored by the merchant’s terminal economic shortfall.The fault engine perturbs event delivery, and an agent acts with eleven tools, five irreversible, under a ship-by deadline.Source: §III · p. 2
  • Because an episode is a pure function of its case, seed, policy, and delivery schedule, restoring one message yields an exact counterfactual with nothing else changed.No noise averaging is needed to see the effect of that repair.Source: §III · p. 2

IV. FAULT INSTANCES

  • A realised schedule is decomposed into named message instances by diffing it against the fault-free schedule for the same case and seed.Repairing all instances restores the fault-free world; there are 7.6 instances per episode on average and at most 12.Source: §IV · pp. 2–3
  • Per-family schedules fail to reproduce the realised schedule on 199 of 240 episodes (82.9%), because fault families interact rather than compose independently.The paper therefore annotates instances with family labels assigned by a contextual but-for test.Source: §IV · p. 3

A. Variables and structural equations

  • The structural model treats the case and individual message faults as exogenous, with delivery schedules, system views, actions, effects, and loss as endogenous variables.In this environment, infrastructure affects the agent through what it can see, not by moving money directly; deterministic equations make interventions exact.Source: A. Variables and structural equations · p. 3

B. Interventions

The method repairs individual messages or substitutes agent decisions while holding the rest of the episode fixed. It uses an exact signed loss decomposition, then allocates the infrastructure effect across messages with Shapley values.

B. Interventions

  • Repairing a message rebuilds the schedule without that instance and reruns the same policy on the same case, leaving everything else unchanged.Decision interventions instead force a chosen action and let the policy continue from there.Source: B. Interventions · p. 3

C. An exact three-way split

  • The exact telescoping split separates infrastructure effect, policy differential, and reference-policy residual, with no unattributed remainder.Two terms can be negative, so the components are a signed decomposition, not nonnegative shares.Source: C. An exact three-way split · p. 3
  • Shapley values divide the infrastructure effect into signed message allocations that sum to it, with exact computation below thirteen instances and permutation sampling above.The sampled estimate was within 1.88% of the exact value in L1.Source: C. An exact three-way split · p. 3

VI. ATTRIBUTION METHODS

The section compares seven attribution methods through ranked candidate causes and a responsibility verdict. The methods range from simple heuristics and an agent-only baseline to fault-repair and joint-intervention approaches.

VI. ATTRIBUTION METHODS

  • Seven methods share an interface that returns ranked candidate causes scored in cents and a verdict assigning responsibility to the agent or infrastructure.Two are heuristics, one is an agent-only baseline, and four use fault interventions.Source: §VI · p. 4
  • Last-action blames the final irreversible step, while first-divergence blames the earliest message that arrived late or not at all.First-divergence ignores whether the message mattered.Source: §VI · p. 4
  • The CAR-shaped baseline substitutes agent steps and replays under the same policy, but its model cannot name infrastructure causes.It is not a reproduction of CAR, CausalFlow, or REFLECT; its infrastructure effect is identically zero.Source: §VI · p. 4
  • Single-fault repair ranks messages individually, while a minimal sufficient set finds the smallest jointly repaired message set that removes the loss.The minimal sufficient set method is exact on every stratum and cheap.Source: §VI · p. 4
  • Joint greedy ranks agent steps and messages by individual effects, but those scores do not sum to the loss because they use different scales.Joint Shapley instead uses the exact identity’s agent and infrastructure effects, then allocates the infrastructure effect across messages.Source: §VI · p. 4
  • Figure 1 reports top-1 identification by stratum; agent-only intervention is correct exactly when the agent is the cause.Source: Figure 1 · p. 4

VII. PLANTED CAUSES

The benchmark plants known infrastructure causes by starting from episodes a policy solves cleanly and then injecting faults. Its strata test single, interacting, and overdetermined causes, alongside agent-caused losses.

VII. PLANTED CAUSES

  • Episodes start from a cleanly solved case, then receive a known fault set whose injection defines the cause of the resulting loss.Source: §VII · p. 4
  • The 545 episodes span single faults (169), conjunctive faults (126), overdetermined faults (70), and agent-caused losses (180).In conjunctive cases, neither fault harms alone; in overdetermined cases, either fault is sufficient on its own.Source: §VII · p. 4
  • Every episode includes distractor faults verified harmless alongside the causal set, preventing simple earliest-message heuristics from succeeding trivially.Source: §VII · p. 4

VIII. RESULTS

Agent-only attribution cannot identify infrastructure causes because its model contains no infrastructure variable; the benchmark measures the resulting misattribution and the performance of repair alternatives. The corpus also shows that interactions make one-at-a-time message scoring insufficient in some episodes.

A. Naming the cause

  • Agent-only intervention identifies the true cause on 0% of infrastructure episodes because its model has no variable that can name a message.This is a structural property of the method, not a corpus-dependent finding.Source: A. Naming the cause · p. 4

VIII. RESULTS

  • $114,383.40 was charged to the agent across 100% of infrastructure episodes, while repairing the named cause recovered 0.0% and 27.8% (95% CI: 23.3–32.3%) of episodes did not decompose additively.These corpus quantities depend on the benchmark environment and generator.Source: A. Naming the cause · p. 4, Table II · p. 5

A. Naming the cause

  • The agent-only method’s 33.0% overall verdict accuracy equals the base rate of episodes in which the agent is at fault.Source: A. Naming the cause · p. 4
  • Minimal sufficient sets and joint Shapley are exact across all four strata, whereas single-fault repair scores 0% on overdetermined cases.In overdetermined cases, repairing either sufficient message alone changes nothing, so every one-at-a-time score is zero.Source: A. Naming the cause · p. 5
  • First divergence identifies a true cause about half the time on overdetermined episodes because harmless distractors can precede sufficient messages.The rule selects by arrival order rather than reasoning about sufficiency.Source: A. Naming the cause · p. 5

B. Splitting the bill

Agent-only intervention misfiles infrastructure-caused loss, while message-level repair exposes substantial recovery and interaction effects. Across infrastructure-loss episodes, single-message repairs often fail to account for the full infrastructure effect.

B. Splitting the bill

  • 100% of infrastructure episodes are misfiled by agent-only intervention, which charges $114,383.40 of infrastructure-caused loss to the agent.Methods that intervene on faults achieve zero allocation error, but the agent/infrastructure split follows from the identity and is exact by construction.Source: B. Splitting the bill · p. 5

C. Acting on the attribution

  • Repairing what agent-only intervention names recovers 0.0% of $82,488.60 in infrastructure-caused loss, versus 82.4% for single-fault repair and 100.0% for minimal sufficient sets.Minimal sufficient sets require 2.3 replays per episode, compared with 280.1 for joint Shapley, which allocates across messages.Source: C. Acting on the attribution · p. 5

D. How often interactions matter

  • 27.8% (95% CI: 23.3–32.3%) of episodes with infrastructure loss are not accounted for by repairing each message alone.The mean gap is $76.73; in 5.6% (95% CI: 3.1–8.7%) of episodes every single-repair score is zero despite a nonzero total.Source: D. How often interactions matter · p. 5

E. One episode, end to end

Across policies, the decomposition distinguishes infrastructure effects, policy differentials, and reference-policy residuals. A single ReAct episode illustrates how message-level causes can differ from the action named by agent-only intervention.

E. One episode, end to end

  • In the $28.00 ReAct episode, Shapley assigns $14.00 each to the dropped processor settlement and reordered bank-feed messages, while agent-only intervention charges the loss to shipping at step 0.Neither message alone creates the loss, but repairing either one removes it; the five other fault instances receive zero.Source: E. One episode, end to end · p. 6

F. Where the money actually goes

  • For the rule-based procedure, 75.1% of loss is attributed to the environment; for ReAct, 59.3% is infrastructure and 17.3% its own; the finality-gated runtime's 100.0% mean loss is reference-policy residual.For the finality-gated runtime, infrastructure effect and policy differential are both zero, and mean loss is $31.37.Source: Figure 3 · p. 6, F. Where the money actually goes · p. 6
  • The optimistic policy has a -6.8% infrastructure effect: faults help it on net because a missing message can be one it would have acted on.The paper presents this as a policy-specific result, not a general finding.Source: F. Where the money actually goes · p. 6

G. Which situations are the environment’s fault

Infrastructure dominates in situations involving missing, stale, or duplicated messages, while the reference-policy residual concentrates on late-arriving facts. The discussion explains how omitted infrastructure variables direct attribution toward agent decisions and weighs exactness against scope and replay cost.

G. Which situations are the environment’s fault

  • Infrastructure dominates archetypes involving missing, stale, or duplicated messages, while the reference-policy residual concentrates on outcomes after the shipby deadline.The three decomposition terms are signed percentages of excess loss, not shares; one can be negative or exceed 100%.Source: G. Which situations are the… · pp. 6–7
  • Reliable delivery or an authoritative channel addresses message-fault cases, whereas engineering the observation path cannot fix facts that have not yet occurred.Source: G. Which situations are the… · p. 7

IX. DISCUSSION

  • When infrastructure variables are absent, agent-analysis tools attribute losses to the remaining agent steps by construction, not because the steps were necessarily at fault.Source: §IX · p. 7
  • Deterministic replay makes the split exact, but substituting an action and resampling a policy diverge for stochastic agents, leaving that comparison unresolved.The Shapley estimate needs a confidence interval only above thirteen messages and is within 1.88% when sampled.Source: §IX · p. 7
  • Minimal sufficient repair is exact across planted strata at 2.3 replays, while first divergence uses fewer replays but can identify the wrong message.The three-message cap never bound; full per-message allocation is worth its cost when that allocation itself is required.Source: §IX · p. 7

X. LIMITATIONS

The conclusion quantifies the cost of agent-only attribution and recommends exposing environment events as intervenable variables. The limitations bound these findings to deterministic policies, one delivery-fault environment, and specified choices of strata, alternatives, and reference policy.

X. LIMITATIONS

  • The evaluation used deterministic procedures, not language-model agents, so whether the conclusions extend to stochastic agents or resampling interventions is untested.The environment’s model adapter was tested against recorded transcripts but never called a model.Source: §X · p. 7
  • All measurements come from FinalityBench, whose delivery-level faults affect what agents observe; infrastructure that moves money directly would require a wider variable set.Source: §X · p. 7
  • Overdetermination accounts for 5.6% of infrastructure-loss episodes and 70 episodes pooled across three policies, so its practical cost rate rests on a small sample.The claim that single-effect scoring fails in that stratum is arithmetic rather than statistical.Source: §X · p. 7
  • Agent-step alternatives comprise seven operator-recognizable dispositions rather than the whole action space; a better alternative outside the menu would raise the measured step effect.Source: §X · p. 7
  • The chosen strongest implementable reference policy fixes the residual; a stronger reference would shift loss from the residual into the policy differential for every subject.Source: §X · p. 7

XI. CONCLUSION

  • The agent-only baseline misfiles every infrastructure failure, charges $114,383.40 to decisions, and repairing what it names recovers none of the loss.Source: §XI · p. 7
  • Minimal sufficient repair recovers 100.0% of available loss at 2.3 replays per episode, compared with 280.1 for full per-message allocation.The conclusion recommends exposing environment events as intervenable variables and using minimal sufficient repair when the goal is operational remediation.Source: §XI · p. 7

Related papers

Loading 2609.25960v1…