PaperlayerView exact revision on arXiv

Source-linked AI summary

Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

Zhongwen Luan, Xiaoyu Zhang, Ming Hu, Yue Yang, Jiongchi Yu, Xiaohong Chen

arXiv:2608.25920v1cs.AIcs.SE

TL;DR

MAS failure repair methods often rely on stochastic full-trajectory reruns, making it unclear whether they correct failures or exploit sampling variability. This paper introduces controlled replay and human-annotated failure data, finding that symptom-driven intervention repairs 20.15% of failed cases versus at most 6.90% for task-level regeneration.

  • Problem

    Reliable understanding, reproduction, and repair of failures in increasingly complex and consequential LLM-based multi-agent systems remains insufficiently established.

  • Method

    The paper introduces SymTrace for controlled selective replay with fixed pre-intervention history and SymFail, a human-annotated dataset of 536 failure trajectories.

  • Results

    20.15% of failures were repaired by Suspicious-Node Intervention with one intervention, versus at most 6.90% repaired by task-level regeneration within three attempts.

  • Takeaways & Limitations

    Reliable repair evaluation should preserve pre-intervention history, localize the intended change, and assess whether the failure mechanism was corrected.

  • Takeaways & Limitations

    SymFail is based on benchmark-derived tasks and may not capture the full diversity of failures in deployed multi-agent systems.

Abstract

from arXiv · show

As large language model (LLM)-based multi-agent systems (MASs) are increasingly applied to long-horizon complex tasks, their reliability has emerged as the core bottleneck hindering their real-world deployment. Existing MAS debugging and repair methods typically rely on rerunning and resampling the entire execution trajectory. However, a fundamental question remains to be answered: do these methods causally repair MAS failures or merely stochastically repair by leveraging the randomness of LLM sampling? To evaluate the effectiveness of MAS repair methods, we introduce SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors. During replay, it effectively reconstructs the execution before the anchor using recorded logs and only regenerates the downstream trajectory, thereby enabling the reliable reproduction of MAS failures. We further construct the dataset SymFail, comprising 536 human-annotated failure trajectories with graph-linked locations, categories, and trace evidence. Based on these foundations, we conduct a large-scale empirical study across three mainstream MAS frameworks. Our findings reveal that existing unguided rerun methods are highly unreliable, exhibiting low failure reproduction and repair rates (only 67.97% and 6.90%, respectively). Building upon these findings, we further explore the effectiveness of a symptom-driven intervention method, which successfully repairs 20.15% of the failed cases (a 191.89% improvement to state-of-the-art repair methods). This study aims to provide actionable insights for MAS debugging and repair research, paving the way for the robust deployment of multi-agent systems.

1 Introduction

LLM-based multi-agent systems support complex long-horizon applications but remain difficult to debug because stochastic reruns may change failures without causally repairing them. This work introduces controlled replay and annotated failure data, showing that localized intervention substantially outperforms task-level regeneration.

  • Motivation: Existing MAS repair methods use rerunning, reflection, critic feedback, or trajectory-level guidance, but repeated stochastic reruns can produce different failure types or no detected failure.This motivates asking whether repair is causal or merely stochastic recovery through LLM resampling.
  • Method: SymTrace records model and tool boundary interactions as event-dependency snapshots and replays them by matching intercepted requests and injecting recorded results.This supports verifiable prefix reconstruction and intervention without resampling the preceding execution.
  • Resources: SymFail provides 536 human-annotated failure trajectories that underpin the MAS debugging study.The dataset and replay system together support evaluation of task-level repair and node-level controls.
  • Empirical findings: 20.15% of failures were repaired by Suspicious-Node Intervention with one intervention, versus at most 6.90% for task-level regeneration within three attempts; it performed best across all three MASs.This was 2.92× the rate of the strongest task-level baseline.
  • Empirical findings: SymTrace increased per-execution failure reproduction from 67.97% to 80.78% and consistent three-execution reproduction from 41.42% to 52.43%.The system enables controlled evaluation by reconstructing the recorded execution prefix before regenerating downstream behavior.

2 Background & Related Work

LLM-based multi-agent systems coordinate role-specialized agents through dependency-structured execution trajectories, making failures propagate beyond terminal outputs. Prior work improves failure observability and repair, but commonly evaluates reruns without controlling pre-repair execution.

  • Background: MASs coordinate multiple LLM-powered agents through model calls, messages, artifacts, and environmental interactions under conversation-centric, workflow-based, or orchestrator-centered architectures.These architectures produce dependency-structured trajectories in which earlier decisions shape later messages, actions, and observations.
  • Background: MAS debugging must examine how failures emerge and propagate through execution trajectories rather than relying solely on terminal output.
  • Related Work: Existing MAS debugging work provides structured observability, failure taxonomies, annotated traces, and localization benchmarks for identifying failure types, components, and locations (Dong et al., 2024; AlSayyad et al., 2026; Shah et al., 2026; Cemri et al., 2025; Deshpande et al., 2025; Zhang et al., 2025).
  • Related Work: Existing MAS repair methods use feedback, peer critique, tool evidence, runtime diagnosis, or dependency-aware localization, but generally evaluate terminal success without consistently controlling execution before repair (Madaan et al., 2023; Shinn et al., 2023; Du et al., 2023; Gou et al., 2024; Zhang et al., 2026; Zhao et al., 2026b; Nanda et al., 2026; Zhao et al., 2026a).This leaves the causal status of successful reruns difficult to assess.

3 System Design

SymTrace uses Snapshot Mode to record structured MAS trajectories and Replay Mode to reconstruct a validated prefix before intervening and resuming live execution. The resulting trajectory preserves the pre-intervention execution while allowing divergence afterward.

  • Replay Scope: Replay Scope guarantees the reconstructed prefix while allowing the newly generated suffix to diverge from the source trajectory.The replay therefore preserves validated execution before the intervention without constraining downstream behavior to the original run.
  • Snapshot Mode: Snapshot Mode records model and tool interactions, organizes events by dependency and order, and bundles them with the task and runtime configuration.The resulting bundle becomes the trajectory transferred to Replay Mode.
  • Workflow: SymTrace’s two-mode workflow records a native MAS execution, reconstructs the target prefix, applies an intervention, and generates a new trajectory from that point onward.Snapshot Mode uses Boundary Logging, Trace Construction, and Replay Bundle; Replay Mode uses Result Injection, Boundary Matching, Live Resume, and Replay Scope.
  • Replay Mode: Replay Mode injects historical boundary results, verifies calls against expected records, and resumes live execution after the designated intervention target.Content hashes validate reused prefix nodes, while model and tool endpoints, scheduling, and state updates continue live after intervention.

4 Dataset Construction

SymFail is a structured dataset of 536 evaluator-confirmed MAS failure trajectories constructed from WebArena-Verified Hard and AssistantBench. It combines trace-localized annotations with calibrated multi-annotator review and releases linked execution artifacts, rationales, and evidence.

  • Dataset overview: SymFail contains 536 evaluator-confirmed failure trajectories with structured execution traces, event graphs, localized failure nodes, and multi-label annotations.The dataset is constructed from WebArena-Verified Hard and AssistantBench.
  • Task selection: The deterministic task pool contains 200 tasks: 33 AssistantBench tasks and the first 167 WebArena-Verified Hard tasks, spanning Web-QA, mutation, navigation, and retrieval.The pool includes 33 Web-QA, 70 mutation, 25 navigation, and 72 retrieval tasks.
  • Execution and filtering: 600 initial executions across AG2, CrewAI, and Magentic-One yielded 536 retained failures with complete trace and graph artifacts; 64 executions were excluded before annotation.The retained trajectories comprise 171 AG2, 184 CrewAI, and 181 Magentic-One executions.
  • Failure taxonomy and annotation: The annotation scheme retains four prevalent, non-exclusive, trace-observable failure categories derived from the Multi-Agent System Failure Taxonomy (Cemri et al., 2025).Annotators assign applicable categories, select a primary category, and identify the earliest trace-supported actionable node with evidence and rationale.
  • Annotation quality: 21.08% of the 536 trajectories changed their final category set relative to the initial label union, while Fleiss’ κ reached 0.62 for primary category and 0.81 for failure-node type.Exact node agreement was 73.88% among all annotators and 95.90% for at least two annotators.
  • Released artifacts: The release includes the task manifest, linked traces and graphs, initial and final annotations, rationales, and supporting evidence, with all 2,144 node selections grounded in source event graphs.The 2,144 selections comprise 536 × (3 + 1) initial and final node selections.

5 Study

Across 536 SymFail failures from AG2, CrewAI, and Magentic-One, SymTrace shows that exact prefix reconstruction improves failure reproduction over unguided full reruns, while localized symptom-driven intervention is more effective than repeated full-trajectory regeneration for repair. Task-level methods primarily exploit stochastic regeneration rather than reliably correcting the source failure.

  • Study design: The study evaluates 536 source failures from 200 fixed tasks across AG2, CrewAI, and Magentic-One, using three complete attempts for task-level methods and one selective-replay intervention for node-level methods.The evaluation set is frozen before replay and repair, and each source failure is treated as an experimental unit.
  • Failure reproduction: SymTrace consistently improves failure reproduction over Unguided Full Rerun across all three MASs by reconstructing execution before the target failure node.Its advantage increases as more preceding nodes can be reused, making it particularly suitable for long-horizon MAS tasks.
  • Failure reproduction: Reproduction gains over full rerun are 9.80 and 6.82 percentage points for 2–3 reused nodes, increasing to 17.76 and 18.69 for 4–8 nodes and 25.69 and 27.08 for at least nine nodes.Most cases, 381 of 536 (71.08%), reuse only 2–3 nodes, while 738 of 1,608 attempts (45.90%) reuse exactly two.
  • Limitations: SymTrace cannot control model or environmental variation after the failure target, and LLM-as-a-judge errors add measurement uncertainty to reproduction estimates.A secondary audit found a 3.30% estimated confirmed-disagreement rate, a 3.67% uncertainty rate, and a 6.96% combined sensitivity upper bound.
  • Task-level repair: Task-level methods repair primarily through stochastic regeneration rather than failure-specific repair, and feedback-based variants provide no advantage under equal three-attempt budgets.Re-execution can repair failed outcomes while destabilizing successful ones, so observed repair does not establish that the source failure was identified and corrected.
  • Repair effectiveness: Suspicious-Node Intervention achieves the highest overall repair rate, nearly triples the strongest task-level baseline, and outperforms Random-Node and Last-Node Intervention under the same selective-replay budget.The comparison indicates that selective replay alone does not explain the improvement; precise execution control and localized symptoms provide actionable repair guidance.

6 Discussion

The study grounds SymFail in externally verifiable benchmarks, enabling reproducible outcome-verified failures but limiting coverage of failures in deployed multi-agent systems. Future work should collect real-world failure trajectories and expand coverage across MAS structures, interaction patterns, and task domains.

  • Data Coverage: SymFail is grounded in established, externally verifiable benchmarks, including WebArena-Verified Hard and AssistantBench.This benchmark basis supports reproducible, outcome-verified failures.
  • Data Coverage: Benchmark-derived tasks cannot capture the full diversity of failures encountered in deployed MASs.
  • Data Coverage: Future work should collect real-world failure trajectories and cover additional MAS structures, interaction patterns, and task domains.

7 Conclusion … B.2 Annotation Rules

The paper distinguishes causal correction of a recorded MAS failure from stochastic success in a different execution, using SymTrace and SymFail to support controlled replay, failure analysis, and repair evaluation. Its replay guarantees depend on preserving and validating the represented execution prefix, while its annotations identify actionable symptoms without treating them as unique root causes.

  • 7 Conclusion: The study’s central conclusion is that full regeneration confounds causal repair with outcome variability, whereas reliable evaluation preserves pre-intervention history, localizes the intended change, and tests whether the failure mechanism was corrected.Terminal success after a different execution is insufficient evidence that the original mechanism was repaired.
  • A Replay Matching Rules and Runtime Assumptions; A.1 Observed Execution Order; A.2 Boundary Matching and Record Validation: SymTrace records observed event order and replays the native MAS by matching recorded boundary requests and results, preserving the prefix before intentional intervention.The recorded order is used for matching and validation rather than scheduler control; selective replay begins live execution at the designated intervention boundary.
  • A.2 Boundary Matching and Record Validation; A.3 Replay Conditions and Scope: A fail-closed replay terminates at the first mismatch and validates both event requests and materialized reused-prefix nodes before returning recorded results.This prevents replay from silently scanning forward or accepting a divergent prefix.
  • A.3 Replay Conditions and Scope: Selective replay reconstructs the represented logical execution prefix without requiring deterministic live models or external services, provided execution state, captured boundaries, and validation conditions hold.The guarantee covers represented state, messages, control decisions, and event order, but not identical memory addresses, thread interleavings, latency, or wall-clock timing.
  • A.4 External State Considerations: Recorded tool results reproduce observations but not persistent external side effects, so selective repair requires the live suffix not to depend on unreproduced mutations in the replayed prefix.This condition can hold when the prefix has no required persistent write or when the environment is reset and the mutation is reproduced.
  • B Dataset Construction, Annotation, and Reliability; B.1 Cohort Accounting: The benchmark produced 536 annotated failures from 600 initial task–MAS executions, excluding 64 evaluator-accepted executions before annotation.The failures span WebArena-Verified and AssistantBench and include executions from AG2, CrewAI, and Magentic-One.
  • B.2 Annotation Rules: The C1–C4 taxonomy uses non-exclusive, trace-observable symptoms, while annotators separately select one primary category as the best-supported intervention signal rather than a unique root cause.The categories cover task-constraint violation, repeated or stalled progress, unresolved runtime conditions, and plan–action–outcome inconsistency.
  • B.2 Annotation Rules: Annotation distinguishes repetition from unresolved dependencies: C2 requires repeated or equivalent behavior after the trace shows the plan cannot succeed, while non-repetitive unresolved dependencies receive C3.Annotators localize the earliest actionable, trace-connected failure node, and confidence records evidential clarity without mechanically affecting adjudication.

B.3 Annotation Procedure … C.3 Statistical Procedures

The study uses adjudicated, graph-validated annotations for 536 SymFail cases and evaluates repair methods under paired, controlled execution and statistical procedures. Annotation reliability is quantified explicitly, while framework, model-record, confidence-interval, and hypothesis-testing conventions constrain interpretation of repair comparisons.

  • B.3 Annotation Procedure: The final annotation re-evaluates evidence: it matches the three-annotator union in 423 cases (78.92%), adds 123 assignments, and removes 69, including 16 unanimously proposed C1 assignments.Three annotators independently label each case, after which a fourth reviews the traces, annotations, and evidence records to produce a strict tuple of categories, primary category, node ID, and node type.
  • B.4 Inter-Annotator Reliability: All three annotators select the same failure node in 396/536 cases (73.88%) and agree on the complete strict tuple in 210/536 cases (39.18%).Reliability is computed from the three initial annotations using exact agreement for node IDs and Fleiss’ κ for nominal or binary category outcomes, with bootstrap confidence intervals.
  • B.4 Inter-Annotator Reliability: C2 has the lowest category-specific reliability (κ = 0.374) and contributes 86 of the 123 category decisions added during final adjudication.This identifies the repeated-or-stalled-progress boundary as the main taxonomy ambiguity and motivates an explicit C2 decision rule.
  • B.5 Final Composition and Integrity Checks: The finalized dataset contains 1,482 category assignments across 536 cases, with mean category cardinality 2.77 and primary-category counts C1 = 125, C2 = 69, C3 = 203, and C4 = 139.Because categories are non-exclusive, marginal counts exceed the number of cases; the fields are human-adjudicated references rather than LLM annotation outputs.
  • B.5 Final Composition and Integrity Checks: 2,144/2,144 node selections pass graph-existence, node-ID, and node-type validation across 1,608 initial and 536 final selections.SymFail contains 536 evaluator-confirmed failures from two benchmark sources and three MASs, with complete trace and graph artifacts.
  • C.1 Repair Conditions and Execution Limits: Matched method pairs hold task input, MAS implementation, recorded model alias, temperature, and native evaluator fixed, varying only repair-prompt information, intervention location, and replay usage.Repair requires at least one evaluator-passing attempt within the method’s permitted budget; RQ1 uses three executions, RQ2 up to three attempts, and RQ3 one node-level intervention.
  • C.2 Framework and Model Records: Experiments use AG2 0.13.3, CrewAI 1.14.7a3, and AutoGen AgentChat 0.7.5, with temperature 0.00 and requested model alias deepseek-v4-flash.The alias and temperature describe recorded client requests, not a verified immutable model snapshot or confirmed upstream provider revision; experiments ran June 15–16, 2026 UTC.
  • C Execution and Statistical Conventions: Repair comparisons report Wilson intervals, paired-bootstrap intervals for percentage-point differences, and prespecified paired tests with Holm adjustment where applicable.Each comparison reports matched denominators, repair counts and rates, differences, confidence intervals, and adjusted values; internal calls and retries are not independent observations.

C.4 Paired Comparison Results … D.2 Reproduction Metrics

Symptom-guided Suspicious-Node Intervention consistently outperforms task-level and node-level controls, while Self-Reflection and Critic-Agent show no significant advantage over Unguided Full Rerun. The evaluation also reports API-call efficiency, implementation details, mechanism-level equivalence criteria, and reproduction metrics.

  • D Reproduction Settings and Fidelity: Paired comparisons use 10,000 case-level paired bootstrap resamples for percentile confidence intervals and exact two-sided McNemar tests.System and framework errors remain in the denominator as unsuccessful repairs, and Holm correction is applied within each MAS to the specified comparison families.
  • C.4 Paired Comparison Results: Self-Reflection and Critic-Agent do not significantly outperform Unguided Full Rerun in any of the six RQ2 comparisons.Every repair-rate difference relative to Unguided Full Rerun is negative, with no comparison significant after within-MAS Holm correction.
  • C.4 Paired Comparison Results: Suspicious-Node Intervention achieves a significantly positive repair-rate difference against every RQ3 comparator across all three MAS architectures.All 15 comparisons have 95% confidence intervals excluding zero and remain significant after within-MAS Holm correction (pH < 0.05), covering task-level and node-level controls.
  • C.5 API-Call Efficiency: Suspicious-Node achieves both the most accepted repairs per 1,000 API calls and the fewest API calls per repair across AG2, CrewAI, and Magentic-One.These are secondary descriptive measures; comparisons are primarily within each MAS and are not compute-matched causal estimates because methods differ in trajectory lengths and attempt budgets.
  • C.6 Implementation Environment: The experiments run locally on Windows 11 with Python 3.11.15, an Intel Core i9-14900HX, 32 GB of RAM, and an NVIDIA GeForce RTX 4060 Laptop GPU.The hosted endpoint performs model inference, while the workstation runs the MAS frameworks, tools, trace capture, SymTrace replay, native evaluation, and statistical analysis.
  • D Reproduction Settings and Fidelity: Mechanism-level reproduction requires semantic equivalence, role correspondence, and grounded supporting evidence rather than raw node equality.The formal test matches source category, case-specific mechanism, semantic execution role, and evidence; an attempt reproduces the source mechanism only when the intersecting criteria include grounded evidence.
  • D.2 Reproduction Metrics: Reproduction metrics quantify per-execution reproduction and reproduction across all three attempts over N source cases.The supplied definitions introduce the per-execution and all-three-attempt measures for evaluating reproduction fidelity.

D.3 Replay-Fidelity Audit

The replay audit found exact preservation of annotated targets and reused prefixes across all 536 cases and three attempts per case. Thus, sub-100% failure reproduction cannot be attributed to modified recorded-prefix content, although variation may remain downstream or outside the replay boundary.

  • Replay-Fidelity Audit: Replay terminates when any pre-target model or tool request is missing, out of order, or mismatched after event-position and canonical-content comparison.This prevents replay from silently continuing with a divergent historical context.
  • Replay-Fidelity Audit: 1,608/1,608 attempts achieved exact prefix fidelity, with all 6,159 reused nodes matching source content hashes and preserving parent/edge topology.The audit used the exact target IDs from human-adjudicated annotations, so it measured replay fidelity rather than automatic localization accuracy.
  • Replay-Fidelity Audit: Sub-100.00% failure reproduction therefore cannot arise from modified recorded-prefix content, but may reflect regenerated-target behavior, later live execution, or external state beyond the replay boundary.The audit rules out altered reused-prefix content while leaving post-boundary sources of variation unresolved.

E Task-Level Repair Aggregation · E.1 Baseline Input Boundaries and Prompt Templates

The task-level evaluation retains a formal pass@3 aggregation over initially failed executions and compares three baselines under matched inputs, budgets, configurations, and evaluators. Under these controlled boundaries, paired results show no advantage for Self-Reflection or Critic-Agent over unguided rerunning.

  • E Task-Level Repair Aggregation: The task-level evaluation retains formal pass@3 aggregation over an evaluation set of initially failed executions, evaluator outcomes, and attempts made before success or exhaustion.The aggregation defines terminal case outcomes from these recorded attempts.
  • E.1 Baseline Input Boundaries and Prompt Templates: The three task-level baselines were evaluated on the same 536 source failures spanning AG2, CrewAI, and Magentic-One.Each received the task specification, task identifier, benchmark and split identifiers, and available start URLs or tools.
  • E.1 Baseline Input Boundaries and Prompt Templates: None of the baselines received the source trajectory or event graph, failure location, C1–C4 category, trace evidence, or reference or gold answer.Self-Reflection and Critic-Agent additionally received the final answer from the original failed execution.
  • E.1 Baseline Input Boundaries and Prompt Templates: Self-Reflection and Critic-Agent reused the original failed execution’s final answer for every recovery attempt rather than a preceding recovery output.Critic-Agent used a fixed critic-role prompt within each complete rerun and did not add a standalone critic-model call.
  • E.1 Baseline Input Boundaries and Prompt Templates: The baselines used distinct prompts for fail-then-rerun, self-reflection, and critic-agent modes while sharing common task fields and a final-answer output suffix.For WebArena-Verified and AssistantBench, the suffix requested only the final answer and a short note about needed evidence.
  • E.1 Baseline Input Boundaries and Prompt Templates: No advantage was observed for Self-Reflection or Critic-Agent over unguided rerunning when all three baselines used matched task inputs, evaluators, model configurations, output constraints, and three-attempt budgets.The comparison therefore isolates whether access to the original failed answer plus generic reflection or critic guidance improves recovery.

F Detection Rules, Anchor Ranking, and Intervention Prompts

The method combines rule-gated symptom detection with repairability-aware anchor selection and symptom-conditioned intervention prompts. A threshold analysis characterizes the evidence gate’s coverage–repair tradeoff while noting that the heuristic weights are not globally optimized.

  • F.1 Rule-First Symptom Detection: The detector uses deterministic C1–C4 rules first, assigns zero suspiciousness to untriggered nodes, and lets the semantic judge confirm or reject only supplied categories.Reference answers, evaluator verdicts, and evaluator-derived evidence are excluded from rule and semantic verification.
  • F.1 Rule-First Symptom Detection: C1–C4 are non-exclusive operational symptoms that raise intervention priority without claiming that a node is the unique root cause.A symptom may represent a local error, unresolved dependency, or downstream manifestation.
  • F.2 Repairability-Aware Anchor Selection: Anchor selection favors repairable, locally supported nodes over terminal or propagation-only nodes, prioritizing regenerable agent actions, then tool calls and results, over final-result nodes.The score combines repairability with local and propagated confidence, while penalizing propagation-only evidence and late graph position.
  • F.3 Intervention Prompts: SymTrace suspends execution at a suspicious node, reconstructs the prefix before the selected anchor, injects symptom-conditioned guidance, and regenerates downstream actions or artifacts.Random-Node and Last-Node controls instead use a generic template without symptom categories or trace evidence.
  • F.5 Threshold Interpretation: The implemented fusion uses semantic confidence and structured selection scores, with a bonus for multiple confirmed symptoms, while its weights remain manually specified heuristic design choices.No saved validation-set optimization or alternative-weight sweep supports describing them as fitted, calibrated, or empirically optimal.
  • F.2 Repairability-Aware Anchor Selection: The online controller processes nodes in observed order and triggers the first replayable LLM boundary whose fused ordinal score meets θ; θ = 0.50 is a coverage-oriented gate.Coefficients encode modeling priorities and were not optimized using validation repair outcomes.
  • F.5 Threshold Interpretation: At θ = 0.95, 78/536 cases are retained (14.55% coverage), including 19 successful repairs, for 19/78 = 24.36% conditional repair.This post-hoc analysis keeps historical rank-0 targets and observed outcomes fixed; it neither reranks candidates nor reruns repairs.
  • F.6 Semantic Boundary of Suspiciousness: The semantic-boundary audit formalizes the treatment of timeout and timed_out states.The supplied formalization fragment identifies these states as part of the audit’s semantic boundary.

G Traceable Gold Failure Examples · GOLD–C3 Incomplete or Low-Signal Op- · H Supplementary Artifact Contents

The paper provides 536 human-adjudicated, traceable failure cases spanning four symptom categories and three multi-agent-system frameworks, with representative evidence-backed localizations. The supplement includes the complete SymTrace source code and SymFail dataset.

  • G Traceable Gold Failure Examples: 536 finalized cases span WebArena-Verified and AssistantBench across AG2, CrewAI, and Magentic-One, covering C1–C4 with category counts of 125, 69, 203, and 139.The displayed gold examples are qualitative trace evidence rather than estimates of model accuracy.
  • G Traceable Gold Failure Examples: Each selected example preserves identifiers, trace excerpts, evidence-supported localization, and a raw-log path, with the raw bundle serving as the authoritative record.Selection is stratified by symptom category rather than favorable outcome.
  • G Traceable Gold Failure Examples: C1 localizes an incomplete answer to the terminal final_result node, despite successful preceding tool requests and retrieved evidence.The node returns another browsing action instead of the qualifying supermarket name, so the failure is not resource unavailability.
  • G Traceable Gold Failure Examples: C2 identifies a repeated Wikipedia request as the failure locus because the trajectory loops instead of advancing through the required multi-hop task.Repeated HTTP 200 responses do not explain the failure; the expected output was relation_id=2176999 and duration=01:33:00.
  • GOLD–C3 Incomplete or Low-Signal Op-: C3 traces failure to an unresolved environment placeholder: MissingSchema and ConnectionError occur before order history is observed, while the emitted 58.99 differs from the evaluator value of 406.53.The complete bundle lacks no provenance files; the low-signal condition is the unresolved execution context.
  • GOLD–C3 Incomplete or Low-Signal Op-: C4 exposes plan–action–outcome inconsistency when an agent presents plausible sorted product data while admitting that browsing is still required and no real tool event occurred.The fabricated products and prices differ from the expected records, making the defect evidential rather than merely formatting-related.
  • H Supplementary Artifact Contents: The code and data supplement contains the complete SymTrace source code and complete SymFail dataset used in the study.
Loading 2608.25920v1…