BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systems

• Attribution is difficult because an agent can misread a correct source, while later corrections or low-impact errors can obscure which mistake was decisive.The example contrasts a correct result naming Braintree and Honolulu with a receiving agent’s misreading as Honolulu and Quincy.Source: §1 · pp. 1–2, Figure 1 · p. 2
• BLADE uses dependency probabilities to select source–decision handoffs, then audits how receiving decisions use their sources under task requirements.Jev supplies structured probabilistic judgments, and the resulting evidence distinguishes error introduction, propagation, and repair.Source: §1 · pp. 2–3
• BLADE achieved the best or tied-best step attribution accuracy among eight strategies on every subset; handoff audits improved accuracy by 2.4–3.4 percentage points across four ablated subsets.On three subsets, dependency-guided auditing matched or outperformed exhaustive auditing while examining 13–23% as many handoffs.Source: §1 · p. 3
• Unlike dependency tracing alone, handoff auditing examines whether information was used correctly, while global diagnosis keeps every execution step eligible for responsibility.Prior approaches include counterfactual replay, repeated-run analysis, dependency tracing, and checks of error evolution or task requirements.Source: §1 · pp. 2–3

2.1 Failure Attribution in Agent Trajectories

• Attribution approaches include counterfactual-replay supervision, repeated-run analyses, checks of task and tool constraints, tool-grounded hypothesis verification, and tracking error correction and impact.These methods collect different signals, but detecting an error is not equivalent to establishing responsibility.Source: §2.1 · p. 3

2.2 From Dependency Tracing to Responsibility Evidence

• Dependency-aware methods structure or rank failures and trace origins, propagation, and correction, but dependencies alone do not show whether a receiving decision used its source correctly.BLADE uses dependency probabilities to select handoffs for audits, whose findings inform a global diagnosis in which every step remains eligible.Source: §2.2 · p. 4

3.1 Problem Definition

• Failure attribution targets the responsible agent and earliest decisive error step, using only the task and trajectory rather than reference answers or ground-truth attribution.A decisive error is one whose correction would make the failed execution successful, with earlier steps unchanged and subsequent execution adjusted.Source: §3.1 · p. 5

3.2 Dependency Reconstruction

• BLADE reconstructs probabilistic relations among earlier and current events, retaining relations above threshold θ = 0.3 as plausible evidence rather than verified causal links.The four relations are depends on, repeats, contradicts, and repairs; dependency probabilities guide handoff selection, while all four provide context for global diagnosis.Source: §3.2 · pp. 5–6

3.3 Graph-Guided Contract Auditing

• At selected source–decision handoffs, BLADE audits whether receiving decisions preserve source meaning and meet task requirements, classifying potential breaches, inherited errors, repairs, or kept cases.Audit planning prioritizes plausible handoffs using dependency probabilities and reuses those probabilities without additional model calls; selection does not restrict which events may be blamed.Source: §3.3 · p. 6

3 BLADE

• BLADE combines plausible dependency relations, handoff-level information-use findings, and task requirements in a blame graph that informs global attribution.Dependencies guide which handoffs to inspect; they do not directly determine responsibility.Source: §3 · p. 4, §3.3 · p. 7

3.4 Defeasible Diagnosis

• Because local audit findings may be corrected, inconsequential, or absent for a decisive error, Jev diagnoses responsibility across all events using the task, full trajectory, and blame graph.Every event remains a candidate, allowing the global judgment to reconsider uncertain local evidence and distinguish error origins from downstream symptoms.Source: §3.4 · p. 7

4.1 Experimental Setup

• The evaluation covers 1,228 failed trajectories across five Who&When and Who&When Pro subsets, spanning natural and injected failures and several multi-agent architectures.Methods share the Jev decision interface and receive trajectories without reference answers or ground-truth attribution labels.Source: §4.1 · pp. 7–8
• The seven baselines include direct, structure-guided, and error-aware strategies, compared under the same Jev interface and task trajectories.The reported metrics are agent accuracy and step accuracy against annotated responsibility.Source: §4.1 · pp. 7–8

4.2 Attribution Performance

• 57.6% step and 66.8% agent macro-average accuracy make BLADE the overall leader, exceeding the strongest baseline averages by 5.6 and 2.8 points, respectively.BLADE is best or tied-best in step accuracy on every subset, with gains over All-at-once ranging from 2.2 to 11.1 points.Source: §4.2 · p. 8
• BLADE remains among the strongest methods as localization tolerance increases, extending its advantage beyond exact-step matches.Figure 3 varies tolerance k; k = 0 is exact step accuracy.Source: §4.2 · p. 8, Figure 3 · p. 9

4.3 Mechanism Analysis

• Adding handoff audit findings improves step accuracy by 2.4–3.4 points across all four ablation subsets, whereas dependency context alone yields mixed improvements.The diagnosis question and candidate set are held fixed across evidence configurations.Source: §4.3 · p. 8, Table 2 · p. 9
• Removing breach findings or shuffling graph evidence across event pairs reduces accuracy; on Magentic-One, 70% of handoffs entering the decisive step are labeled breach, versus 4% entering earlier steps.These findings link attribution evidence to how information is used at specific handoffs.Source: §4.3 · p. 9
• Dependency-guided auditing examines only 13–23% as many handoffs as exhaustive auditing while matching or improving step accuracy on AG, Magentic-One, and DyLAN.Its findings are more concentrated on the decisive step; on DyLAN, recency-based selection nearly halves that share and reduces accuracy.Source: §4.3 · p. 9, Figure 4 · p. 10

4.4 Further Analysis

• On Magentic-One, BLADE improves step accuracy from 74.4% to 78.8% for four-to-seven-step trajectories and from 52.6% to 56.1% for longer trajectories.The full pipeline’s estimated Jev inference cost is approximately $1.9 across 1,228 trajectories.Source: §4.4 · pp. 9–10

• BLADE separates evidence construction from blame assignment: dependencies guide targeted source–decision handoff audits, while global diagnosis considers the full trajectory.The blame graph records error introduction, propagation, and repair; audit judgments are evidence, not fixed blame assignments.Source: §5 · p. 10
• Under a shared Jev interface and without reference answers, BLADE achieves best or tied-best step accuracy among eight methods on five subsets.Its blame graph exposes the local evidence available to each diagnosis.Source: §5 · p. 10
• The experiments find dependency information alone insufficient for reliable attribution, while examining source use improves localization and selective auditing can outperform exhaustive inspection.Source: §5 · p. 10

• Evaluation covers five Who&When and Who&When Pro subsets; broader real-world workflows and task domains remain needed to establish generalizability.Source: Limitations · p. 10
• BLADE depends on recorded traces and relevant observations or interactions, and shortening trajectories beyond the context window may omit useful evidence.Source: Limitations · p. 10
• Experiments use Jev alone, so evaluating other compatible decision models would help assess generalization across backbones.Source: Limitations · p. 11

• The experiments use existing research benchmarks, without new human-subject data collection or interventions in deployed systems.Source: Ethical Considerations · p. 11
• BLADE is intended to support retrospective failure diagnosis and debugging.Source: Ethical Considerations · p. 11

A.1 Leaking Fields

• Who&When Pro leakage fields are removed, including Magentic-One ledger discrepancies and DyLAN/MetaGPT injection-or-replay labels.The Magentic-One discrepancy identifies the injected step in 636 of 811 runs.Source: §A.1 · p. 13

A.2 Long Trajectories

• Twenty-two of 995 Who&When and Magentic-One runs exceeding Jev’s 32k-token limit are shortened, while tasks, agents, event ids, and authors remain unchanged.Web-page metadata is replaced by a marker; long event contents retain their beginnings and endings if further shortening is needed.Source: §A.2 · p. 13

B.1 Requirement Spans

• Task text is split at list items, then discourse markers, then semicolons when each has at least two occurrences; otherwise it is split into sentences.The rule selects the first applicable boundary type in that order.Source: §B.1 · p. 13

B.2 Baseline Implementations

• Direct-attribution baselines use the full trajectory or search it sequentially or by repeatedly halving candidate segments.These methods are All-at-once, Step-by-step, and Binary search.Source: §B.2 · p. 13
• Structure-guided baselines rank suspicious steps or screen candidates using structural impact, influence, propagation, and correction judgments.The implementations are TRIAGE and FALAT (Rafi et al., 2026).Source: §B.2 · p. 14
• Error-aware baselines select or arbitrate decisive-step candidates by tracking triggers and impact or assessing failure hypotheses along dependency paths.TrajDebug (Qi et al., 2026) and ErrorProbe (Li et al., 2026) are implemented without tool execution or persistent memory; ErrorProbe uses BLADE’s precomputed Jev depends-on judgments and trace-based verification.Source: §B.2 · p. 14

C.1 Breach Localization

• In Magentic-One, decisions with breach findings are decisive in 59.5% of cases, versus 9.8% without findings; the corresponding Who&When rates are 11.9% and 5.2%.Source: §C.1 · p. 14
• On Magentic-One, breach-labeled handoffs rise from at most 5% in the four decisions before the decisive step to 70%; on DyLAN they rise from at most 6% to 42%.Figure 5 locates audited handoffs by receiving-decision position relative to the decisive step.Source: §C.1 · p. 14, Figure 5 · p. 14
• Among 30 Magentic-One runs correctly attributed by BLADE but not All-at-once, the selected BLADE step has an audit finding in 25 runs, versus 2 for All-at-once’s selected step.Source: §C.1 · p. 14

C.2 Step Attribution: Paired Comparisons

• All seven paired task-cluster bootstrap confidence intervals for BLADE’s macro-average step-accuracy gains over baselines lie above zero.The estimates resample task-query clusters within subset strata, preserve method pairing, and hold predictions fixed.Source: §C.2 · p. 14, Table 3 · p. 15

C.3 Additional Mechanism Checks

• On Magentic-One, removing breach findings reduces step accuracy to 78.3%, while keeping only breach findings matches the full configuration at 80.9%.Relative to dependencies alone, adding audit findings corrects 34 step predictions and loses 5 across four subsets.Source: §C.3 · pp. 14–15
• On DyLAN, auditing the same number of most-recent sources lowers decisive-step findings from 28.4% to 15.0% and step accuracy from 56.6% to 54.6%.Auditing one source instead of two, or raising θ from 0.3 to 0.4 or 0.5, lowers accuracy by at most 1.0 point on Magentic-One and 3.2 points on Who&When.Source: §C.3 · p. 15

D Question Templates

• Figure 6 presents question templates for dependency reconstruction, contract auditing, and diagnosis.Source: Figure 6 · p. 16