Source-linked AI summary
Know the Shape, Find the Fault: Topology-Conditioned Diagnosis of Multi-Agent LLM Failures
Xinwen Liu, Zhuocheng Pan, Isabella Zhu, Jawei Zhang, Xudong Liu, Tianyu Wo
TL;DR
Multi-agent failures require interpreting actions through the relationships that pass and verify information, but heterogeneous traces may not include topology labels. MAScope recovers topology from trace evidence and uses it to condition failure diagnosis. Experiments report higher diagnostic performance with topology context and lower projected cost when topology is reused under a fixed orchestration structure.
Problem
Failure diagnosis must interpret individual actions through information-transfer and verification relationships, yet heterogeneous execution traces may not explicitly record communication topology.
Source: §1 · p. 1, §2.3 · p. 3Method
MAScope's TSE recovers an evidence-grounded interaction graph and topology from heterogeneous traces, then TC-Judge combines the trace, topology, empirical prior, and failure signature for diagnosis.
Source: §1 · p. 2, §4.1 · p. 5Results
Takeaways & Limitations
When orchestration structure is stable, topology reuse reduces deployment cost while topology conditioning lets a lower-cost judge approach a stronger trace-only model's diagnostic performance.
Source: §7 · p. 8TC-Judge keeps its parameters fixed because sparse MAST-clean labels make supervised failure-code prediction vulnerable to framework-fingerprint shortcuts.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Multi-agent LLM systems coordinate task execution through exchanges of information among agents. When coordination breaks down, similar symptoms in execution traces can reflect different problems in how information is passed, used, or verified. Communication topology captures how agents exchange information and provides structural cues for distinguishing coordination failure modes. Using these cues for diagnosis requires establishing how topology relates to failure patterns and recovering the relevant structure from execution traces that lack explicit topology labels. We analyze the relationship between communication topology and failure patterns and introduce MAScope, a two-stage framework for topology-conditioned diagnosis. Its Trace Structural Extractor TSE recovers communication topology from heterogeneous execution traces by grounding an interaction graph in message evidence. The Topology-Conditioned Judge TC-Judge then classifies failures using the trace, predicted topology, an empirical failure prior estimated from separate labeled traces, and a short description of topology-specific failure patterns. Under a fixed orchestration structure, the recovered topology can be reused across executions. Experimental results show a statistically significant association between communication topology and failure type, with $\chi^2 = 409.9$ and $p = 1.2 \times 10^{-70}$. On the \num{851} MAST-clean traces, ground-truth topology context raises gpt-mini's Macro-F1 from $0.173$ to $0.350$. With predicted topology, the pipeline achieves $0.346$, approaching the trace-only gpt-5.4 baseline of $0.372$. For \num{1000} traces under a fixed orchestration structure, the projected pipeline cost, including one topology extraction, is approximately $6\%$ of repeated gpt-5.4 diagnosis cost. These results show that topology-conditioned context improves failure diagnosis and supports lower-cost deployment.
1 Introduction
Multi-agent failures can look similar at the trace level even when information is lost, misused, or left unverified across agent relationships. MAScope recovers communication topology from traces and uses topology-specific failure patterns to guide diagnosis; experiments report improved diagnosis and lower projected cost under a fixed orchestration structure.
1 Introduction
- Diagnosing collaboration failures requires interpreting individual actions within the relationships through which information is passed and verified.A requirement may be understood by one agent yet disappear during a handoff, or agents may finish subtasks while the final result remains unchecked.Source: §1 · p. 1
- A shared message pool can make feedback available without ensuring that agents act on it or perform the requested verification.In the example, the tester adds unrelated square-function tests, which the next reviewer accepts as correct.Source: Figure 1 · p. 1, §1 · p. 2
- MAScope's TSE recovers communication relations and topology labels from message evidence, then TC-Judge uses the trace and topology-specific failure patterns to diagnose failures.When communication structure remains stable, the recovered topology can be reused across executions to reduce repeated diagnosis cost.Source: §1 · p. 2
- On MAST-clean, ground-truth topology raises gpt-mini’s Macro-F1 from 0.173 to 0.350; topology recovered by TSE yields 0.346, close to trace-only gpt-5.4’s 0.372.For 1000 traces under a fixed topology, the projected gpt-mini pipeline cost, including one structural extraction, is approximately 6% of repeated gpt-5.4 diagnosis cost.Source: §1 · p. 2
- The paper establishes an empirical association between communication topology and failure patterns, identifying failure types that are more prevalent under each topology.It also constructs MAScope-Bench with 2220 execution traces and audit materials for evaluating topology recovery, diagnosis, transfer, and deployment cost.Source: §1 · p. 2
2 Related Work
Prior work studies failure categories, attribution, and communication design, but diagnosis primarily relies on traces without explicitly incorporating topology. Recovering topology labels from heterogeneous execution traces remains an underexplored problem.
2.1 Failure Diagnosis in LLM Multi-Agent Systems
- Failure-diagnosis work categorizes recurring failure modes and identifies responsible agents or execution steps, while these methods primarily rely on traces without communication topology as diagnostic input.The literature includes taxonomy-oriented and localisation-oriented approaches.Source: §2.1 · p. 2
2.2 Communication Topology of LLM MAS
- Communication-topology research mainly constructs or adapts agent structures to improve task performance and efficiency.Approaches include graph optimization, pruning, architecture search, and topology generation.Source: §2.2 · p. 2
2.3 From Execution Traces to Topology Labels
- Topology recovery from traces remains underexplored because heterogeneous frameworks use different logging formats and may not record topology explicitly.When orchestration configuration is unavailable, topology must be inferred from agent interactions and grounded in message-level evidence.Source: §2.3 · pp. 2–3
3 Empirical Foundation: Topology × Failure Correlation
The paper tests whether topology and failure patterns are associated, using multiple corpora and declared orchestration graphs as topology ground truth. It finds distinct topology-linked failure patterns that remain qualitatively consistent across corpora.
3.1 Setting
- A trace comprises agents, a time-stamped message stream, tool and retrieval returns, and external state, while failure annotations use the MAST taxonomy and allow multiple codes per trace.The topology label uses Guo et al.'s four-class taxonomy.Source: §3.1 · p. 3
3.2 Corpora and Ground Truth
- MAScope-Bench combines 851 MAST-clean traces, 961 Inject-Mix traces, and 408 LG-Ctrl traces; topology labels come from the original framework-declared orchestration graphs.LG-Ctrl uses one LangGraph harness with all four topologies sharing a StateGraph codebase and topology-agnostic agent identifiers.Source: §3.2 · p. 3, Table 1 · p. 3
3.3 Independence Test and Per-Topology Anchors
- χ² = 409.9 and p = 1.2 × 10^-70 indicate an association between topology and failure distributions, with different failure types showing positive residuals across topologies.Layered pipelines are associated with Information Loss, Centralized topologies with Verification Misalignment, Decentralized traces with Coordination Stall, and Shared Message Pool topologies with Semantic Drift.Source: Figure 2 · p. 3, §3.3 · p. 3
- Associations between the four topologies and their characteristic failure patterns remain qualitatively consistent across MAST-clean, LG-Ctrl, and Inject-Mix.Leave-one-framework-out evaluation retains 82% of in-distribution Macro-F1.Source: §3.3 · p. 3
4 Two-Stage Diagnosis Pipeline
MAScope separates topology recovery from failure diagnosis, allowing the recovered structure to be reused when orchestration is fixed. TSE grounds topology in trace evidence, and TC-Judge uses topology-specific context to assess failures.
4 Two-Stage Diagnosis Pipeline
- Under a fixed deployment, the recovered topology can be reused across executions, leaving TC-Judge as the marginal runtime cost for each trace.The pipeline separates topology and failure labels because they change at different rates.Source: §4 · p. 4
4.1 Stage 1. TSE, Trace Structural Extraction
- A shared speaker-tagged message schema and thin adapters let TSE prepare evidence consistently across seven supported frameworks.The adapters are provided for LangGraph, AutoGen, CrewAI, ChatDev, MetaGPT, Magentic, and Simple Chain.Source: §4.1 · p. 4
- TSE recovers a canonical relation graph and topology label from heterogeneous traces, linking relations and the final label to named message spans.Its stages compact evidence, extract relations, validate and repair them, then assign a structural label; deterministic rules also recover long-distance speaker attribution and implicit shared-channel reads.Source: §4.1 · pp. 4–5
4.2 Stage 2. TC-Judge, Topology-Conditioned LLM-Judge
- TC-Judge combines the trace with topology, an empirical failure prior, and a failure signature to assign the fourteen MAST codes independently in one structured call.The prior is estimated from a 400-trace exploration slice and frozen; the 851-trace MAST-clean evaluation set is disjoint from that slice. The judge remains zero-shot because of uneven label coverage and the risk of learning framework fingerprints.Source: §4.2 · p. 5
5 Experiments
The experiments evaluate topology recovery and diagnosis across natural, injected, and framework-controlled traces, using paired comparisons and leave-one-framework-out evaluation. They report stronger zero-shot diagnosis with topology context, alongside recovery and deployment-cost results.
5 Experiments
- The experiment sequence assesses corpus construction, comparison protocols, topology recovery, zero-shot diagnosis, cross-framework performance, and predicted-topology deployment cost.LG-Ctrl and Inject-Mix provide separate structural and diagnostic validation, respectively.Source: §5 · p. 5, §5.2 · p. 6
5.1 MAScope-Bench Construction and Audits
- MAScope-Bench combines 2,220 traces in MAST-clean, Inject-Mix, and LG-Ctrl to support main, injected-fault, and framework-controlled evaluations.LG-Ctrl varies graph edges and iteration patterns within one LangGraph codebase, while Inject-Mix broadens topology and runtime coverage.Source: §5.1 · pp. 5–6
5.2 Comparison Methods and Evaluation Setup
- The evaluation compares nested M0–M4 topology-context configurations, matched against M0, and tests learned baselines with task-grouped folds and LOFO.M0 has no topology context; M1 adds the topology tag, M2 the prior, M3 both, and M4 also adds the failure signature. Oracle-topology uses ground-truth labels, while the same M4 judge is also tested with TSE predictions.Source: §5.2 · p. 6
5.4 Zero-Shot Diagnostic Performance
- 0.173 to 0.350 Macro-F1 for gpt-mini and 0.372 to 0.399 for gpt-5.4 from M0 to M4 on MAST-clean; gpt-mini gains 102%.The matched comparison holds traces, prompt scaffolding, output space, and model fixed; the gpt-mini result reaches 94% of trace-only gpt-5.4’s score.Source: Figure 4 · p. 6, §5.4 · p. 7
5.3 Topology Recovery
- TSE recovers 798 of 851 MAST-clean topologies (93.8%); on controlled LG-Ctrl it recovers 312 of 408 (76.5%), while a matched replication reaches 31 of 40 (77.5%).Shared Message Pool is the most difficult controlled class at 56.9%; the matched replication supports recovery without framework identity, though the task becomes harder.Source: §5.3 · p. 7, Table 4 · p. 7
5.5 Cross-Framework Generalisation
- On Inject-Mix, M4 raises Macro-F1 from 0.304 to 0.321 for gpt-mini and from 0.320 to 0.356 for gpt-5.4.The corpus adds framework and topology settings beyond MAST-clean, including injected traces across Layered, Centralized, and Decentralized orchestrators.Source: §5.1 · p. 6, §5.5 · p. 7
- Under LOFO, Flat-GBM Macro-F1 falls from 0.490 to 0.226 and Neural-Trace from 0.296 to 0.127 when framework and topology shift together.Figure 5 reports cross-framework scores; the MAST-clean protocol pairs each framework with one topology, so it does not isolate the two shifts.Source: Figure 5 · p. 7, §5.5 · p. 7
5.6 End-to-End Diagnosis and Deployment Cost
- 0.346 Macro-F1 with predicted topology versus 0.350 with ground-truth topology for the gpt-mini pipeline on MAST-clean.With a fixed orchestration structure, one TSE extraction is reused and each trace receives one gpt-mini TC-Judge call, so the cost advantage over repeated gpt-5.4 diagnosis grows with deployment volume.Source: §5.6 · p. 8
6 Analysis
The analysis explains topology’s diagnostic value as a reference for information flows and responsibilities, then relates that context to model gains and deployment cost.
Why topology is informative
- Topology gives the judge a reference for checking whether agents fulfilled information-transfer and verification responsibilities in the trace.The analysis connects this to handoffs in Layered workflows and aggregation and verification at a Centralized hub.Source: §6 · p. 8
Why topology helps gpt-mini
- Topology conditioning supplies gpt-mini with collaboration structure directly, while gpt-5.4 can infer more structural context from the raw trace.The label, prior, and failure signature direct attention to vulnerable responsibilities so the judge can focus on trace evidence rather than reconstructing the collaboration scheme.Source: §6 · p. 8
Deployment cost
- For n traces under a fixed orchestration graph, the cost is one extraction plus n lower-cost judge calls rather than n gpt-5.4 calls.As more traces share the extraction cost, average cost per trace decreases.Source: §6 · p. 8, Figure 6 · p. 8
7 Conclusion
The paper reports an empirical association between communication topology and failure patterns and proposes MAScope for topology-conditioned diagnosis. Its experiments find improved diagnosis with topology context and lower deployment cost when structure is reused.
Conclusion
- The paper establishes an empirical association between communication topology and failure patterns in LLM-based multi-agent systems.Source: §7 · p. 8
- MAScope recovers topology from heterogeneous traces and combines it with empirical failure priors to guide failure judgment.MAScope-Bench combines natural, injected, and framework-controlled traces for evaluating recovery and diagnosis.Source: §7 · p. 8
- Experiments report that topology conditioning improves diagnostic performance, allowing a lower-cost judge to approach a stronger trace-only model’s performance while structural reuse reduces deployment cost.Source: §7 · p. 8