PaperlayerView exact revision on arXiv

Source-linked AI summary

Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

Zhongwen Luan, Xiaoyu Zhang, Ming Hu, Yue Yang, Jiongchi Yu, Xiaohong Chen

arXiv:2608.25920v1cs.AIcs.SE

TL;DR

Rerunning LLM multi-agent systems may produce successful executions without establishing that the original failure was corrected. The paper combines SymTrace’s controlled replay with SymFail’s annotated failure trajectories to evaluate failure reproduction and repair across systems. Suspicious-Node Intervention repairs 20.15% of failures in one intervention, versus at most 6.90% within three task-level regeneration attempts, and performs best across all three systems.

  • Problem

    Whether rerunning MAS executions corrects recorded failures or merely yields stochastic success remains unclear, despite the importance of failure repair for reliability.

    Source: §1 · p. 1, §7 · p. 9

  • Method

    The study combines SymTrace’s controlled replay of recorded pre-intervention histories with SymFail’s human-annotated failure trajectories to evaluate failure reproduction and repair across systems.

    Source: §1 · p. 2

  • Results

    20.15% of failures were repaired by one Suspicious-Node Intervention, versus at most 6.90% within three task-level regeneration attempts, with the best performance across all three MASs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Successful task completion alone does not establish correction of the original failure mechanism, so repair evaluation should explicitly assess mechanism correction.

    Source: §7 · p. 9

    SymFail’s benchmark-derived tasks cannot capture the full diversity of failures in deployed MASs.

    Source: §6 · p. 9

Abstract

from arXiv · show

As large language model (LLM)-based multi-agent systems (MASs) are increasingly applied to long-horizon complex tasks, their reliability has emerged as the core bottleneck hindering their real-world deployment. Existing MAS debugging and repair methods typically rely on rerunning and resampling the entire execution trajectory. However, a fundamental question remains to be answered: do these methods causally repair MAS failures or merely stochastically repair by leveraging the randomness of LLM sampling? To evaluate the effectiveness of MAS repair methods, we introduce SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors. During replay, it effectively reconstructs the execution before the anchor using recorded logs and only regenerates the downstream trajectory, thereby enabling the reliable reproduction of MAS failures. We further construct the dataset SymFail, comprising 536 human-annotated failure trajectories with graph-linked locations, categories, and trace evidence. Based on these foundations, we conduct a large-scale empirical study across three mainstream MAS frameworks. Our findings reveal that existing unguided rerun methods are highly unreliable, exhibiting low failure reproduction and repair rates (only 67.97% and 6.90%, respectively). Building upon these findings, we further explore the effectiveness of a symptom-driven intervention method, which successfully repairs 20.15% of the failed cases (a 191.89% improvement to state-of-the-art repair methods). This study aims to provide actionable insights for MAS debugging and repair research, paving the way for the robust deployment of multi-agent systems.

1 Introduction

Reliable MAS debugging requires distinguishing correction of a recorded failure from success caused by stochastic resampling. The paper introduces controlled replay and human-annotated failure evidence, finding that localized intervention outperforms task-level regeneration.

1 Introduction

  • Complete reruns confound repair attribution by resampling upstream decisions, while task-level verdicts and automatic categories may miss the trace-localized behavior requiring intervention.The motivating example shows disagreement between initial and expert diagnoses and changing failure types across stochastic reruns. Changed outcomes therefore do not establish correction of the recorded failure.Source: §1 · pp. 1–2
  • SymTrace fixes observable pre-intervention history through recorded-result replay, allowing downstream changes to be attributed to intervention rather than upstream resampling.It records model and tool interactions as an event-dependency graph and observed execution order. Replay restarts the native MAS, strictly matches intercepted requests, and injects recorded results until live execution resumes at the intervention boundary.Source: §1 · p. 2
  • 536 evaluator-confirmed failures form SymFail, with human annotations linking failure nodes, symptom categories, supporting evidence, and final evidence-based annotations.These failures come from 600 trajectories generated across three MASs on 200 WebArena-Verified Hard and AssistantBench tasks (Zhou et al., 2024). The authors release SymTrace, SymFail, and experimental results.Source: §1 · p. 2
  • 20.15% of failures are repaired by single-intervention Suspicious-Node Intervention, versus at most 6.90% within three task-level regeneration attempts—2.92× the strongest task-level baseline.The symptom-driven method localizes replayable anchors and applies evidence-conditioned repair. It performs best across all three MASs.Source: §1 · p. 2
  • 80.78% per-execution failure reproduction with SymTrace improves on 67.97%; consistent reproduction across three executions rises from 41.42% to 52.43%.The study evaluates task-level repair and node-level controls across 536 failures.Source: §1 · p. 2

2 Background & Related Work

Multi-agent failures must be understood through dependency-structured execution trajectories, not terminal outputs alone. SymTrace makes these trajectories replayable and controls execution before intervention to distinguish targeted repair from stochastic resampling.

2.1 LLM-Based Multi-Agent Systems

  • AG2 (Wu et al., 2023), CrewAI (CrewAI, Inc., 2026), and Magentic-One (Fourney et al., 2024) exemplify conversation-centric, workflow-based, and orchestrator-centered coordination, respectively.Agents receive roles, instructions, context, and tools; coordination determines task decomposition, which agent acts next, and intermediate-state transfer.Source: §2.1 · p. 2
  • MAS debugging must examine how failures emerge and propagate through dependency-structured trajectories, because earlier decisions shape subsequent messages, actions, and observations.Terminal outputs alone therefore do not suffice for debugging and repair.Source: §2.1 · p. 3

2.2 MAS Debugging and Repair

  • SymTrace records model and tool interactions, dependencies, execution order, and realized results as replayable records supporting verifiable reconstruction.Existing debugging research supports identifying failure types, responsible components, and error locations, but primarily uses traces for post-hoc analysis.Source: §2.2 · p. 3
  • SymTrace reconstructs execution before an evidence-supported intervention to distinguish targeted repair from successful reruns that merely avoid failures through stochastic resampling.Existing repair methods commonly regenerate failed behavior, but terminal-success evaluations do not consistently control execution before repair.Source: §2.2 · p. 3

3 System Design

SymTrace records native MAS executions, reconstructs validated execution prefixes, and resumes live execution under an intervention. Replay preserves the represented history before the target while allowing the downstream trajectory to diverge.

3.1 Snapshot Mode

  • Boundary Logging captures realized model request-response and tool call-observation pairs through framework-specific hooks without changing native scheduling, agent logic, or state updates.Requests still reach live endpoints; ordered records retain requests, results, and event positions, preserving nondeterministic boundary values for replay.Source: §3.1 · p. 3
  • Trace Construction records data and control dependencies separately from observed event order, while Replay Bundle packages execution records, structure, task, and runtime configuration.Repeated workflow iterations remain distinct event instances, and observed order is stored because the dependency graph can permit multiple valid schedules. The bundle supplies the recorded trajectory to Replay Mode.Source: §3.1 · p. 3

3 System Design

  • SymTrace separates trajectory recording in Snapshot Mode from intervention-driven selective replay in Replay Mode.Inputs include the task, represented initial state, runtime configuration, and realized model and tool interactions.Source: §3 · p. 3, Figure 2 · p. 4

3.2 Replay Mode

  • Replay Mode restarts the MAS and injects historical boundary results only after matching event positions and canonicalized request content; mismatches terminate replay.Reconstruction follows the recorded dependency graph and observed order, with content hashes validating reused prefix nodes.Source: §3.2 · p. 4
  • Replay guarantees the represented prefix history only when native transitions are deterministic under recorded boundary values and relevant external state is reset, isolated, or restored.At the target, Live Resume applies the intervention and restores live model and tool calls without changing native scheduling or state updates. The suffix may diverge; inaccessible framework internals need not match.Source: §3.2 · p. 4

4 Dataset Construction

SymFail comprises 536 evaluator-confirmed failure trajectories with structured execution traces, event graphs, localized failure nodes, and multi-label annotations. Its construction combines deterministic task selection, execution across three MAS frameworks, and independent annotation followed by adjudication.

4 Dataset Construction

  • 536 evaluator-confirmed failure trajectories form SymFail, pairing structured execution traces and event graphs with localized failure nodes and multi-label annotations.The release includes the task manifest, linked traces and graphs, initial and final annotations, rationales, and supporting evidence.Source: §4 · p. 4, §4.3 · p. 5

4.1 Candidate Selection

  • 200 tasks were fixed before execution or repair: the first 167 WebArena-Verified Hard tasks in published order and all 33 eligible AssistantBench development tasks.Candidates comprised 258 WebArena-Verified Hard tasks and 33 AssistantBench tasks with descriptions, reference answers, and gold URLs. Selected for multi-step Web tasks with externally verifiable outcomes, the pool contains 33 Web-QA, 70 mutation, 25 navigation, and 72 retrieval tasks.Source: §4.1 · p. 5

4.2 Trajectory Collection

  • 536 failures remained from 600 executions spanning AG2 (Wu et al., 2023), CrewAI (CrewAI, Inc., 2026), and Magentic-One (Fourney et al., 2024).Each task was executed once per framework; 64 executions were excluded before annotation, leaving 462 WebArena-Verified failures and 74 AssistantBench failures. Retained trajectories comprise 171 AG2, 184 CrewAI, and 181 Magentic-One executions.Source: §4.2 · p. 5

4.3 Failure Annotation

  • Four non-exclusive, trace-observable categories consolidate prevalent patterns from 14 failure modes across three broader categories in the Multi-Agent System Failure Taxonomy (Cemri et al., 2025).The selected categories retain the four most prevalent consolidated patterns in the source analysis.Source: §4.3 · p. 5
  • Three annotators independently assign applicable categories, a primary category, and the earliest trace-supported actionable node; a fourth adjudicates rather than applying majority vote or label union.All four have software-engineering backgrounds and calibrate using reference-labeled examples, practice trajectories, and shared guidance. The adjudicator reviews the task, complete trace, event graph, evidence, rationales, and initial annotations, with authority to revise any decision.Source: §4.3 · p. 5
  • 0.62 primary-category and 0.81 failure-node-type Fleiss’ κ accompanied exact-node agreement of 73.88% among all three independent annotators and 95.90% among at least two.Adjudicated category sets differed from the initial label union in 113 of 536 trajectories (21.08%). All 2,144 initial and final node selections existed in their source graphs and matched their recorded node types.Source: §4.3 · p. 5

5 Study

Across 536 source failures, reconstructing recorded execution prefixes improves failure reproduction, while task-level feedback provides no consistent advantage over unguided reruns. Suspicious-Node Intervention achieves the strongest repair results with one symptom-guided intervention rather than three complete attempts.

5.1 Setup

  • 536 source failures from 600 runs cover 200 fixed tasks across three MAS frameworks, with the evaluation set frozen before replay and repair.All conditions use deepseek-v4-flash (Xu et al., 2026) at temperature 0.00; task-level methods receive up to three attempts, versus one node-level intervention. Failure reproduction measures whether all selected executions reproduce the source failure, while pass@k measures whether at least one repair is evaluator-accepted within k attempts.Source: §5.1 · pp. 5–6

5.2 RQ1: How Reliably Can MAS Failures Be Reproduced?

  • 100.00% prefix exactness accompanies SymTrace’s consistent failure-reproduction advantage over full rerun across all three MASs, with larger gains from longer reused prefixes.Most cases—381 of 536 (71.08%)—reuse only 2–3 nodes, where rep1 and rep3 gains are 9.80 and 6.82 percentage points. These gains rise to 17.76 and 18.69 points for 4–8 nodes, and 25.69 and 27.08 points for at least nine nodes.Source: Table 2 · p. 6, §5.2 · p. 6
  • Exact prefix reconstruction does not guarantee identical failures because the target and subsequent nodes remain exposed to live model and tool variation.Identical prefixes can produce different unsupported answers or encounter changed API responses. Executions retaining only the same high-level failure category are not counted as successful case-specific reproductions.Source: §5.2 · pp. 6–7
  • 3.30% estimated confirmed disagreement and 3.67% uncertainty yield a 6.96% combined sensitivity upper bound for reproduction judgments.A stratified audit of 72 judgments finds 64 agreements, five confirmed disagreements, and three uncertain cases. All five confirmed disagreements are false negatives that label genuine reproductions as different failures.Source: §5.2 · p. 7

5.3 RQ2: Can Task-Level Re-execution Reliably Repair Failed MAS Executions?

  • Unguided Full Rerun performs best among task-level baselines on every MAS, while reflection and critique show no systematic advantage for particular failure categories.Self-Reflection supplies the failed output for self-diagnosis (Madaan et al., 2023; Shinn et al., 2023), whereas Critic-Agent uses a separate agent’s corrective review (Gou et al., 2024).Source: §5.1 · pp. 5–6, Table 3 · p. 7, §5.3 · p. 7
  • 85 of 162 reruns of initially successful executions fail, and 39 of 54 source cases regress at least once, demonstrating bidirectional outcome instability.A CrewAI case also produces an evaluator-accepted answer without obtaining the observations missing from the original failure. Consequently, accepted re-execution alone does not demonstrate failure-specific repair.Source: §5.3 · pp. 7–8

5.4 RQ3: Are Node-Level Symptom Signals Actionable for Targeted Repair?

  • Suspicious-Node Intervention combines symptom-based anchor selection with classification-conditioned repair instructions, evaluating both components jointly rather than isolating their individual contributions.Deterministic rules and a semantic judge assess completed nodes for C1–C4 symptoms; exceeding a suspicion threshold suspends execution. SymTrace then reconstructs the preceding prefix, applies the repair instruction, and resumes live execution.Source: §5.4 · p. 8
  • Nearly tripling the strongest task-level baseline’s overall repair rate, Suspicious-Node Intervention performs best across all three MASs with one intervention.It substantially outperforms Random-Node and Last-Node Intervention under the same selective-replay budget, while task-level baselines receive up to three complete attempts. Its advantage over Unguided Full Rerun is significant on every MAS after within-MAS Holm correction.Source: Table 4 · p. 8, §5.4 · p. 8

6 Discussion

The study’s limitations concern benchmark-derived failure coverage, errors in LLM-based failure-reproduction judgments, and uncertain generalization beyond a single model setting.

Data Coverage

  • SymFail’s externally verifiable benchmarks yield reproducible, outcome-verified failures but cannot capture the full diversity of deployed MAS failures.Future work should collect real-world failure trajectories and cover additional MAS structures, interaction patterns, and task domains.Source: §6 · p. 9

Automated Evaluation

  • LLM judges enable large-scale failure-reproduction evaluation but occasionally misjudge long trajectories, ambiguous evidence, or semantically similar failure locations.Judges compare generated failure locations and categories against references; manual spot checks identified occasional errors. Proposed improvements include expert review of disagreements and low-confidence cases, hybrid human-LLM annotation, and reporting human-LLM agreement.Source: §6 · p. 9

Model Coverage

  • Using the same model alias and generation configuration controls paired comparisons, but findings may not generalize across model families, providers, scales, or versions.This setup prevents model capability or version differences from confounding architectural and repair-strategy comparisons. Broader model evaluations should test whether reproduction and repair patterns remain consistent.Source: §6 · p. 9

7 Conclusion … B.2 Annotation Rules

Failure identity depends on execution history, so success on a regenerated trajectory does not establish that the recorded failure mechanism was corrected. SymTrace preserves and validates represented pre-intervention history, while SymFail annotations identify actionable symptoms rather than unique root causes.

7 Conclusion

  • Reliable repair evaluation must preserve pre-intervention history, localize the intended change, and assess mechanism correction rather than terminal success alone.The study combines SymTrace, human-annotated SymFail, and cross-system evaluation to distinguish correction of recorded failures from stochastic success. Localized symptoms can guide effective interventions without uniquely identifying root causes.Source: §7 · p. 9

A.1 Observed Execution Order

  • SymTrace records the observed execution order because dependency graphs specify only partial order; replay uses that order for matching and validation without controlling native scheduling.The recorded order is the particular linear extension observed during execution, fixing when state-affecting operations crossed instrumented boundaries rather than imposing a post hoc topological ordering.Source: §A.1 · p. 12

A.2 Boundary Matching and Record Validation

  • Selective replay terminates at the first boundary mismatch; otherwise, it preserves the recorded prefix and begins intentional divergence at the designated intervention.Matching checks event position and canonicalized request content, followed by content-hash validation of materialized prefix nodes. Repair guidance augments the designated LLM request before the target and downstream execution run live.Source: §A.2 · p. 12

A.3 Replay Conditions and Scope

  • Selective replay reconstructs represented logical execution under fixed setup, conditionally deterministic internal transitions, and complete state-affecting capture; live models and services need not be deterministic.The guarantee covers recorded boundary results, agent-visible messages, represented logical state, control decisions, and event order. It does not require identical memory addresses, physical thread interleavings, network latency, or wall-clock timing.Source: §A.3 · pp. 12–13

A.4 External State Considerations

  • Recorded tool responses reconstruct observations, not persistent external mutations; live suffixes must not depend on unreproduced prefix writes.Required external state must be unnecessary, restored through a service checkpoint, or recreated through safe re-execution after an environment reset. All 536 failures were evaluated without external-state filtering; variation from unrestored external state falls outside the replay guarantee.Source: §A.4 · p. 13

B.1 Cohort Accounting

  • 536 failures from 600 initial task–MAS executions entered SymFail; 64 evaluator-accepted executions were excluded before annotation.Three MASs ran 200 fixed tasks; the failures comprised 462 WebArena-Verified and 74 AssistantBench cases, distributed across 171 AG2, 184 CrewAI, and 181 Magentic-One executions.Source: §B.1 · p. 13

B.2 Annotation Rules

  • Annotations assign every trace-supported C1–C4 symptom and one primary intervention signal, without imposing category priority or treating that signal as a unique root cause.The non-exclusive categories are C1 Task-Constraint Violation, C2 Repeated or Stalled Progress, C3 Unresolved Runtime Condition, and C4 Plan–Action–Outcome Inconsistency. C3 extends its source categories to cover execution or context failures; dataset composition uses complete category sets.Source: §B.2 · p. 13
  • Annotators select the earliest actionable, trace-supported node connected to evaluator-confirmed failure; confidence records evidential clarity and does not mechanically affect adjudication.Node types distinguish agent decisions, tool invocations, decisive unhandled tool results, and final-response errors; records retain categories, primary category, node ID/type, confidence, rationale, and evidence. C2 requires repetition after established plan failure—not initial failures, justified retries, or substantive refinements—whereas unresolved dependencies without repetition receive C3; high/medium/low confidence indicates direct support/plausible alternatives/insufficient evidence, not probabilities or voting weights.Source: §B.2 · pp. 13–14

B.3 Annotation Procedure … C.3 Statistical Procedures

SymFail’s 536 cases undergo independent annotation, evidence-based adjudication, reliability assessment, and graph-integrity checks. Repair experiments use matched failures, explicit attempt budgets, recorded framework/model settings, and paired statistical procedures, while model snapshot identity remains unverified.

B.3 Annotation Procedure

  • 18.66% of final strict-tuple annotations had no initial annotator’s full support; adjudication re-evaluated evidence rather than merely aggregating votes.Three annotators independently labeled each case, and a fourth reviewed traces, annotations, and evidence to finalize the exact category set, primary category, node ID, and node type. Final category sets matched the initial union in 423 cases (78.92%) and differed in 113 (21.08%), adding 123 assignments and removing 69, including 16 unanimously proposed C1 assignments.Source: Table 5 · p. 14, §B.3 · p. 14

B.4 Inter-Annotator Reliability

  • 73.88% of cases had unanimous node agreement (95.90% at least two); complete strict-tuple unanimity was 39.18%.Reliability uses the three initial annotations across 536 cases, excluding final labels; category decisions use Fleiss’ κ with 20,000 case-level bootstrap resamples for percentile 95% intervals, whereas case-specific node IDs require exact agreement. C2 had the lowest category-specific reliability (κ = 0.374) and accounted for 86 of 123 added assignments, identifying repeated-or-stalled progress as the principal taxonomy ambiguity.Source: Table 6 · p. 14, §B.4 · pp. 14–15

B.5 Final Composition and Integrity Checks

  • 1,482 category assignments across 536 finalized cases yield mean category cardinality 2.77; C3 is the most common primary category (37.87%).Primary-category counts are C1 = 125 (23.32%), C2 = 69 (12.87%), C3 = 203 (37.87%), and C4 = 139 (25.93%). Non-exclusive marginal counts are C1 = 439, C2 = 288, C3 = 289, and C4 = 466.Source: §B.5 · p. 15
  • 2,144/2,144 node selections passed graph-integrity validation, covering all 1,608 initial and 536 final selections.Validation checks graph existence, node membership, and node-type consistency. SymFail covers two benchmark sources and three MASs; linked artifacts, initial annotations, evidence, and human-adjudicated references—not LLM annotation outputs—are released, while broader independent re-annotation remains future work.Source: §B.5 · p. 15

C.1 Repair Conditions and Execution Limits

  • Matched repair comparisons fix task input, MAS, model alias, temperature, and evaluator, varying only repair-prompt information, intervention location, and replay.RQ1 uses three executions per method; RQ2 allows up to three complete attempts, stopping at first success; RQ3 gives node-level methods one selective-replay intervention while task-level baselines retain up to three attempts. Any permitted evaluator-passing attempt counts as repair, and efficiency is measured because actual API-call counts can differ.Source: §C.1 · p. 15

C.2 Framework and Model Records

  • Temperature 0.00 and deepseek-v4-flash identify recorded client requests, not a verified immutable model snapshot, upstream provider, or server-side revision.Experiments ran June 15–16, 2026 UTC using AG2 0.13.3, CrewAI 1.14.7a3, and a Magentic-One adapter using AutoGen AgentChat 0.7.5. The adapter imports autogen_agentchat and autogen_ext, not pyautogen; reproducibility is not attributed to the alias or temperature alone.Source: §C.2 · p. 15

C.3 Statistical Procedures

  • 95% Wilson intervals quantify repair proportions; absolute repair-rate differences use 10,000 case-level paired bootstrap resamples for 95% confidence intervals.Differences are expressed in percentage points, with bootstrap seed 20260701 and paired method outcomes resampled together. Each comparison reports matched denominators, repair counts and rates, differences, intervals, and applicable adjusted values; each failure contributes at most one binary outcome per method, excluding internal calls and retries as independent samples.Source: §C.3 · pp. 15–16
  • Two-sided exact McNemar testing evaluates RQ1; prespecified per-MAS Holm families contain two comparisons for RQ2 and five for RQ3.RQ1 compares SymTrace Replay with Unguided Full Rerun on paired rep3 outcomes; RQ2 compares Self-Reflection and Critic-Agent separately against Unguided Full Rerun. RQ3 compares Suspicious-Node Intervention against Unguided Full Rerun, Self-Reflection, Critic-Agent, Last-Node Intervention, and Random-Node Intervention, totaling 15 tests across three MASs; Holm-adjusted values are denoted pH.Source: §C.3 · p. 16

C.4 Paired Comparison Results … D.2 Reproduction Metrics

Suspicious-Node Intervention consistently outperforms task-level and node-level controls in paired repair comparisons and API-call efficiency, whereas Self-Reflection and Critic-Agent show no significant advantage over unguided rerunning. Reproduction assessment distinguishes grounded mechanism-level equivalence from raw node equality.

C.5 API-Call Efficiency

  • Suspicious-Node achieves the highest API-call repair efficiency across AG2, CrewAI, and Magentic-One.It yields the most accepted repairs per 1,000 API calls and the fewest API calls per repair; system/framework errors count as non-repairs, with preceding calls retained. These are descriptive within-MAS comparisons, not compute-matched causal estimates, because trajectory lengths and attempt budgets differ.Source: §C.5 · p. 16

C.6 Implementation Environment

  • Hosted endpoints perform model inference, while the local workstation runs MAS frameworks, tools, trace capture, SymTrace replay, native evaluation, and statistical analysis.The environment uses Python 3.11.15 on Windows 11 with an Intel Core i9-14900HX, 32 GB RAM, and an NVIDIA GeForce RTX 4060 Laptop GPU with 8 GB VRAM.Source: §C.6 · p. 16

C.4 Paired Comparison Results

  • All six RQ2 repair-rate differences versus Unguided Full Rerun are negative, with none significant after within-MAS Holm correction.Under the same three-attempt budget, the results provide no evidence that Self-Reflection or Critic-Agent outperforms unguided rerunning.Source: §C.4 · p. 16, Table 7 · p. 17
  • All 15 RQ3 comparisons favor Suspicious-Node Intervention, with 95% confidence intervals excluding zero and within-MAS Holm-corrected pH < 0.05.The advantage spans three MAS architectures and both task-level and node-level controls. Suspicious-Node, Random-Node, and Last-Node use pass@1, while the three task-level comparators use pass@3.Source: §C.4 · p. 16, Table 8 · p. 17

D.1 Mechanism-Level Failure Equivalence

  • Mechanism-level reproduction requires an event matching the source category, mechanism, and semantic execution role, with grounded supporting evidence.Mechanism equivalence and semantic-role correspondence do not require raw node equality.Source: §D.1 · p. 17

D.2 Reproduction Metrics

  • Reproduction metrics distinguish per-execution reproduction from reproduction in all three attempts, using the grounded mechanism-matching criterion.An attempt reproduces the source mechanism only when at least one qualifying event has grounded evidence.Source: §D.2 · p. 17, §D.1 · p. 17

D.3 Replay-Fidelity Audit

The audit found exact prefix preservation across all 1,608 replay attempts, ruling out modified recorded-prefix content as the cause of imperfect failure reproduction. It measures replay fidelity using human-adjudicated targets, not automatic localization accuracy.

D.3 Replay-Fidelity Audit

  • Replay matches every pre-target model or tool request against ordered records using event position and canonicalized content, terminating on missing, out-of-order, or non-matching requests.Source: §D.3 · p. 17
  • 1,608/1,608 replay plans used exact human-adjudicated target IDs, so the audit evaluates replay fidelity rather than automatic localization accuracy.Source: §D.3 · p. 18
  • 1,608/1,608 replay attempts achieved prefix exactness, preserving parent/edge topology and matching source-content hashes for all 6,159 materialized reused nodes.The audit covered 536 cases with three attempts each; node-content hash agreement was 6,159/6,159.Source: §D.3 · pp. 17–18
  • Modified recorded-prefix content cannot explain reproduction below 100.00%; remaining variation may arise at the regenerated target, downstream live execution, or external state outside the replay boundary.Failure recurrence alone does not establish that replay preserved historical context.Source: §D.3 · pp. 17–18

E Task-Level Repair Aggregation · E.1 Baseline Input Boundaries and Prompt Templates

Across 536 source failures, controlled task-level comparisons show no advantage from Self-Reflection or Critic-Agent over unguided rerunning. The appendix specifies pass@3 aggregation, baseline input restrictions, and prompt templates.

E Task-Level Repair Aggregation

  • The appendix retains formal pass@3 aggregation over initially failed executions, using evaluator outcomes from attempts executed until success or exhaustion.Task-level strategies, evaluation sets, budgets, and results are defined in the main paper.Source: E Task-Level Repair Aggregation · p. 18

E.1 Baseline Input Boundaries and Prompt Templates

  • 536 source failures from AG2, CrewAI, and Magentic-One underpin all three baselines, which receive task specifications, identifiers, benchmark splits, and available start URLs or tools.For WebArena-Verified and AssistantBench, the shared output suffix requests only the final answer and a short note about needed evidence.Source: §E.1 · pp. 18–19
  • All baselines exclude source trajectories, event graphs, failure locations, C1–C4 categories, trace evidence, reference or gold answers, and evaluator information.Only Self-Reflection and Critic-Agent additionally receive the original failed execution’s final answer.Source: §E.1 · p. 18
  • Self-Reflection and Critic-Agent reuse the original failed answer for every attempt; Critic-Agent embeds its fixed critic-role prompt within reruns without an additional standalone critic-model call.Neither method substitutes a preceding recovery attempt’s output for the original failed answer.Source: §E.1 · pp. 18–19
  • Unguided Full Rerun solves from scratch, Self-Reflection reconsiders the failed answer, and Critic-Agent checks assumptions, evidence sufficiency, and unsupported or stale claims.These prompts operationalize the comparison between unguided rerunning and access to the failed answer with generic reflection or critic guidance.Source: §E.1 · p. 19
  • Self-Reflection and Critic-Agent show no advantage over unguided rerunning in the paired task-level results reported in Appendix C.4.The comparison holds task inputs, output suffix, model configuration, evaluator, and the three-attempt budget constant.Source: §E.1 · p. 19

F Detection Rules, Anchor Ranking, and Intervention Prompts

Suspicious-Node Intervention combines rule-gated symptom detection, repairability-aware anchor selection, and trace-grounded repair guidance. Its threshold analysis reports conditional repair alongside coverage while keeping historical intervention targets and outcomes fixed.

F.1 Rule-First Symptom Detection

  • Rule-gated detection gives untriggered nodes zero suspiciousness and invokes the semantic judge only for deterministically triggered candidates.The judge may reject but cannot introduce categories, and verification excludes reference answers and evaluator-derived information. C1–C4 are non-exclusive operational symptoms that prioritize intervention without establishing a unique root cause.Source: §F.1 · p. 19, Table 10 · p. 20

F.2 Repairability-Aware Anchor Selection

  • Anchor selection prioritizes repairability over local and propagated evidence, favoring regenerable agent actions and assigning final-result nodes the lowest node-type prior.Propagation-only evidence and late graph position incur penalties. Confidence fusion weights semantic-judge confidence by 0.55 and structured selection by 0.45, with a separate +0.04 bonus for multiple confirmed symptoms.Source: §F.2 · pp. 20–21, §F.5 · p. 22
  • The first eligible node meeting the fused-score threshold triggers replay, prefix reconstruction, symptom-conditioned guidance, and regeneration of the downstream trajectory.Nodes are processed in observed order, and the scores are ordinal evidence measures rather than calibrated probabilities.Source: Figure 3 · p. 20, §F.2 · p. 21

F.3 Intervention Prompts

  • Suspicious-node prompts include the target, confirmed symptoms, and rule-derived evidence; Random-Node and Last-Node controls receive generic guidance without symptoms or trace evidence.Repair prompts exclude the semantic judge’s free-form rationale, reference answers, evaluator verdicts, and evaluator-derived evidence.Source: §F.3 · p. 21

F.4 Metrics and Statistical Procedure

  • Infrastructure and framework errors remain non-repairs, and the case is the inferential unit for statistical comparisons.Statistical comparisons follow Appendix C.Source: §F.4 · p. 22

F.5 Threshold Interpretation

  • 20.15% conditional repair at θ = 0.50 covers all 536 cases, versus 24.36% at θ = 0.95 with 14.55% coverage.This post-hoc abstention analysis fixes historical rank-0 targets and observed outcomes; it neither reranks nodes nor reruns repairs. Thresholds and weights are heuristic, not globally optimal, and systematic optimization and component-level ablation remain limitations.Source: §F.5 · p. 22

F.6 Semantic Boundary of Suspiciousness

  • Runtime-failure auditing combines node-level error, tool-result error, and tool-status indicators into a single runtime-failure signal.Source: §F.6 · p. 22

G Traceable Gold Failure Examples · GOLD–C3 Incomplete or Low-Signal Op- · H Supplementary Artifact Contents

Four category-stratified, trace-verified examples illustrate distinct failure mechanisms in a 536-case human-adjudicated ledger, rather than estimate model accuracy. The supplement includes the complete SymTrace source code and SymFail dataset.

G Traceable Gold Failure Examples

  • 536 finalized, human-adjudicated failure cases comprise 462 WebArena-Verified and 74 AssistantBench cases.The framework distribution is 171 AG2, 184 CrewAI, and 181 Magentic-One cases.Source: G Traceable Gold Failure Examples · p. 22
  • The C1 example localizes incomplete resolution to a terminal final_result node that returns another browsing action instead of a qualifying supermarket.Both preceding tool results succeeded, and the second reached the supermarket website, supporting incomplete answering rather than an access failure.Source: G Traceable Gold Failure Examples · p. 23
  • 125/536 C1, 69/536 C2, 203/536 C3, and 139/536 C4 cases underpin four category-stratified examples, not model-accuracy estimates.Selection covers every observed primary symptom category rather than favorable outcomes. Each example provides immutable identifiers, field-preserving trace excerpts, evidence-supported localization, and a complete raw-log bundle path.Source: G Traceable Gold Failure Examples · pp. 22–23, Table 11 · p. 24
  • The C2 example localizes progress deadlock to tool_result_0002, which repeats a Wikipedia request after a coordinate query fails to provide coordinates.Successful HTTP responses did not establish progress: the trajectory revisited the same page instead of advancing through the required multi-hop routing task.Source: G Traceable Gold Failure Examples · p. 25

GOLD–C3 Incomplete or Low-Signal Op-

  • The C3 example identifies an unresolved shopping-site placeholder at tool_result_0000 as the earliest decisive failure, preventing access to the target application.Adding https:// changes a missing-scheme failure into a name-resolution failure without repairing the environment binding. The raw bundle is complete; the low-signal condition concerns execution context, not missing provenance.Source: GOLD–C3 Incomplete or Low-Signal Op- · p. 25
  • The C4 example exposes an agent_action_0000 answer claiming specific products and prices while admitting that browsing remains necessary to obtain actual data.The output has the requested keys and price ordering, but no real tool event or observed order record supports it. The failure is an inconsistency between the claimed answer and the acknowledged evidence-gathering requirement.Source: GOLD–C3 Incomplete or Low-Signal Op- · p. 26

H Supplementary Artifact Contents

Related papers

Loading 2608.25920v1…