Source-linked AI summary
Where Do Multi-Agent Systems Fail? Evidence-Grounded Diagnosis of Collective Mechanisms
Zhengye Han
TL;DR
Multi-agent systems can violate collective mechanisms yet still answer correctly, while outcomes alone do not identify which mechanism failed. The paper defines evidence-grounded diagnostic contracts and tests them using controlled replays and LLM diagnosers. Violations often remain hidden in outcomes, and diagnostic consistency on a benchmark does not ensure transfer to another workflow.
Problem
It is unclear what execution evidence is sufficient to conclude that a particular collective mechanism was violated, especially when outcomes and public transcripts do not distinguish compliant from violating runs.
Source: §1 · p. 1Method
The paper defines contracts specifying violations and authoritative records, then tests them through unchanged, broken, and restored replays and bindings to independently developed systems.
Source: §1 · p. 2, §5.3 · p. 6Results
In 43 of 72 admission violations the system still answered correctly, while LLM diagnosis detected 26% of broken mechanisms from public outputs and 63% from internal records.
Source: §1 · p. 3, §6.1 · p. 7Takeaways & Limitations
A correct outcome does not certify the mechanisms behind it; diagnosis must be licensed by authoritative records, and benchmark performance must be retested on the target system.
Source: §8 · p. 8The transfer result rests on one independently developed workflow with 28 packaged pairs from 10 tasks, and its admission binding was judged ambiguous by both LLM reviews.
Source: §7 · p. 8
Abstract
from arXiv · showhide
When a multi-agent system answers correctly, it is tempting to conclude that its agents shared, checked, and used information as intended. Yet a system can break one of its collective mechanisms, the rules that govern how agents route, admit, store, and act on shared information, and still return the right answer, while a wrong answer rarely reveals which mechanism failed. We ask what evidence from an execution is sufficient to conclude that a particular mechanism was violated. Our answer is a diagnostic contract, which separates what counts as a violation from which execution records can establish one, and concludes that a violation is supported, ruled out, or unknown; removing records can make this conclusion unknown but never reverse it. We test contracts for four mechanisms by replaying executions from the step where a mechanism acts, once unchanged, once with the mechanism broken, and once with it restored. Broken mechanisms often left the answer correct. An LLM diagnoser detected many more violations from internal records than from public outputs, yet with identical records a generic prompt often claimed certainty the records did not support, which prompts stating the contracts largely avoided. The contracts also applied, in narrow form, to mechanisms in independently developed systems, but a diagnostic behavior that was nearly perfect on our benchmark degraded on an independently developed workflow. A correct outcome is therefore no substitute for records of how collective mechanisms operated, and agreement on one benchmark does not show that a diagnoser transfers to another system.
1 Introduction
Collective mechanisms govern how agents route, admit, store, and act on shared information, but task outcomes alone cannot reveal whether those mechanisms operated correctly. The paper introduces evidence-grounded diagnostic contracts and tests them with controlled replays, finding that violations can coexist with correct answers and that diagnostic behavior may not transfer across systems.
1 Introduction
- Outcome-driven analyses can miss violations because a system may answer correctly despite a mechanism failure, while public transcripts may fit both compliant and violating executions.For example, admission counted duplicate reports independently, but the aggregator itself collapsed them and still answered correctly; the admission log, not the public transcript, distinguishes the runs.Source: §1 · p. 1, Figure 1 · p. 2
- A diagnostic contract defines a violation and the authoritative records needed to establish it, returning supported, ruled out, or unknown.Removing records can make a verdict unknown but cannot reverse it; contracts cover access/routing, admission/dependence, state maintenance, and representation/action, excluding incentives.Source: §1 · pp. 1–3
- The study tests contracts by replaying each execution from the mechanism’s decision step unchanged, broken, and restored, while holding evidence packages fixed for diagnosers.Bindings to independently developed systems are accepted only when their code implements the rule and records the evidence required by the contract.Source: §1 · p. 2
- In 43 of 72 controlled admission violations, the system still answered correctly; LLM diagnosis identified 26% of broken mechanisms from public outputs and 63% from internal records.With identical records, generic prompts often claimed unsupported certainty, whereas prompts stating the contracts largely avoided it on completed outputs; transfer consistency fell from 287 of 287 benchmark pairs to 13 of 28 pairs on an independently developed workflow.Source: §1 · pp. 2–3
- The findings concern finite, controlled suites covering four mechanisms, tested one at a time with supplied repairs; incentives are outside scope.The paper cautions that these results do not establish how often concealment occurs in deployed systems.Source: §1 · p. 3
2 Related Work
Prior work classifies, attributes, localizes, or detects failures, and studies monitoring under partial observation. This paper instead asks which runtime records warrant a claim that a particular collective mechanism was violated, and tests whether diagnosers respect those evidence requirements across systems.
Failure analysis in LLM-based multi-agent systems
- Failure taxonomies, attribution, and localization address what kind of failure occurred, who caused it, or when; this work asks which records license a mechanism-violation claim, including for successful runs.Evidence sufficient for attribution may leave the mechanism claim unresolved.Source: §2 · p. 3
Fault injection and robustness
- Fault-injection benchmarks measure detection, robustness, and recovery, whereas this benchmark treats injected faults as experimental metadata rather than diagnostic evidence.The same perturbation can yield supported, ruled out, or unknown depending on what the run records expose.Source: §2 · p. 3
Norm monitoring and model-based diagnosis
- Norm monitoring and model-based diagnosis share the premise that observability limits conclusions, but these contracts bind meanings to source-specific authoritative records and provenance.Shared-prefix perturbation and restoration separate realized violations from task outcomes and injection metadata; the study also tests whether generic diagnosers respect evidence obligations on independent workflows.Source: §2 · p. 3
Collective mechanisms and runtime surfaces
- The paper operationalizes Han’s four-locus taxonomy as executable contracts, using heterogeneous frameworks to test whether one diagnostic semantics can be witnessed across runtime surfaces.The frameworks are used to test record bindings, not to assess the frameworks themselves.Source: §2 · p. 3
3 Diagnostic Contracts
A diagnostic contract pairs a mechanism’s violation rule with the evidence specification an observer needs to evaluate it. Its verdict depends on which complete executions remain consistent with the records.
3.1 Mechanisms and violations
- The four loci cover information routing, evidence admission, state maintenance, and decision-to-action; agent reasoning errors are outside scope unless they also violate one of these rules.The incentive part of shared-state mechanisms is also outside the four-locus scope.Source: §3.1 · p. 3
3.2 What counts as evidence
- An evidence bundle records the run’s evidence, its provenance, relevant task conditions, and the completeness scope of the records.Records must be authoritative for the specific operation; missing records do not establish that an event was absent, and conflicting or misbound records invalidate measurement.Source: §3.2 · p. 4
3.3 From evidence to a verdict
- A violation is supported when every complete execution consistent with the evidence violates the rule, ruled out when none does, and otherwise unknown.For admission, a complete log showing duplicate-origin reports counted independently supports a violation; a public transcript alone can leave the verdict unknown.Source: §3.3 · p. 4
- Removing records can turn a resolved verdict into unknown but cannot reverse it; retaining a deletion that already preserves the verdict also preserves it through further legal deletions.Resolved verdicts are stored with their supporting records and completeness claims for review.Source: §3.3 · p. 4
4 Benchmark Protocol and Challenge Suite
The benchmark establishes violation ground truth by replaying runs from a shared prefix under unchanged, broken, and restored mechanisms. Its task suites and framework adapters support controlled comparisons across mechanisms and execution records.
4 Benchmark Protocol and Challenge Suite
- Freezing a run before a mechanism acts and continuing it unchanged, broken, or restored creates experimental ground truth for whether that mechanism was violated.The three branches share a prefix so outcome changes can be compared against a targeted mechanism change.Source: §4 · p. 4, §4.2 · p. 5
4.1 Tasks and held-out split
- The canonical suite covers evidence aggregation and symbolic constraint, while separately frozen DRAC assigns tasks under capacity limits and settles versioned reservations through actions.The canonical suite has 72 held-out challenges; the separately frozen DRAC suite has 36 held-out challenges.Source: §4.1 · p. 5
- Perturbations target one rule at a time, including delivery, admission, state updates, or action encoding; action failures are counted only when the aggregator’s decision was correct.The held-out split tests new instances of known task families and mechanisms.Source: §4.1 · p. 5
4.2 Replaying from a shared prefix
- Replay branches share the same prefix and differ only at the targeted institutional component, with restoration returning that component to its original value.Later model calls can still differ after the shared prefix.Source: §4.2 · p. 5
4.3 What the frameworks contribute
- The canonical suite and DRAC implement the institutional rules, while four framework adapters carry them through each runtime’s messages, state, and records.The comparison tests whether a shared diagnostic meaning can be read from different records, not the frameworks’ default designs; the MetaGPT adapter excludes its optional tool stack.Source: §4.3 · p. 5
5 Evaluation Design
The evaluation separates outcome effects, record access and diagnoser behavior, and contract portability to independently developed systems. It uses controlled cohorts and evidence packages to test those questions.
5 Evaluation Design
- The evaluation asks whether outcomes reveal supported violations, which records enable detection and contract-compliant reasoning, and whether contracts and diagnoser behavior transfer to other systems.Bootstrap intervals resample whole challenges or tasks; Figure 3’s Wilson intervals over packaged pairs ignore task clustering and are optimistic.Source: §5 · p. 5
5.1 Actor cohorts and outcome contrasts
- The canonical cohort runs each of 72 held-out challenges on four framework adapters, yielding 288 realizations with three branches each; DRAC runs 36 held-out challenges on the same adapters.Because a 1,000-token allowance truncated nearly all allocator outputs, DRAC’s formal cohort uses a 16,384-token allowance chosen on development cases without consulting correctness.Source: §5.1 · p. 5
5.2 Diagnosers and the records they see
- Public-output packages contain agent statements and final actions, while native-record packages add inputs, admission decisions, shared-state versions, and action receipts.The same-evidence cases delete irrelevant records or decisive witnesses; G1 states the contracts and three-valued semantics, while G2 adds worked examples.Source: §5.2 · p. 6
5.3 Independently developed systems
- External bindings are accepted only when an independently developed system’s code implements the mapped rule and records the evidence required by the unchanged contract.Held-out A/B/C triplets and full, irrelevant-deletion, and witness-deletion packages test those bindings and unchanged G1/G2 prompts.Source: §5.3 · p. 6
6 Results
Across the results, correct task outcomes often concealed admission violations, while richer records improved detection and contract prompts better constrained diagnoses. Contracts bound to independent systems, but one diagnostic obligation did not transfer consistently to Co-STORM.
6.1 Task outcome is not mechanism correctness
- 43 of 72 admission perturbations still produced a correct answer, so task success concealed 59.7% of supported admission violations.The table reports mechanism verdicts alongside task outcomes; its canonical branches used a 1,000-token completion allowance.Source: §6.1 · p. 7, Table 3 · p. 7
6.2 Native records enable detection
- Native records raised G0 recovery from 76/288 broken mechanisms (26.4%) on public outputs to 182/288 (63.2%), while routing became recoverable but admission and state maintenance remained mostly unrecovered.Figure 2 compares target-locus recovery from public outputs and native records.Source: §6.2 · p. 7, Figure 2 · p. 7
6.3 Generic diagnosers assert verdicts the records do not license
- With exactly the contract-used records, G0 recovered all 192 supported violations and activated on none of 192 ruled-out cases, yet broke both obligations and often asserted unsupported verdicts.The passage reports that G1 and G2 made no opposite flips on valid outputs and, collectively, preserved verdicts under irrelevant deletion in all but one case.Source: §6.3 · p. 7
6.4 Contracts bind to independently developed systems
- Contracts bound in narrow form to four mechanisms across MetaGPT, the OpenAI Agents SDK, and Co-STORM, but the SDK action binding used a deterministic fixture rather than a multi-agent workflow.The external-binding table reports support on broken branches and activation on unchanged and restored controls; some candidate bindings were rejected when their native mechanisms did not implement the contract rule.Source: §6.4 · pp. 7–8, Table 5 · p. 8
6.5 One obligation transfers, the other does not
- On Co-STORM evidence, witness deletion yielded unknown on every jointly valid pair, but G1 preserved its verdict after irrelevant-record deletion in only 13/28 pairs, versus 28/28 canonically.Because the canonical and native tasks differ, the contrast indicates a transfer gap but does not estimate its size.Source: §6.5 · p. 8, Figure 3 · p. 8
7 Discussion and Limitations
The evaluation establishes bounded evidence for collective-mechanism diagnosis: its benchmark is a positive-control construction, and its transfer and contract scope remain limited.
7 Discussion and Limitations
- The benchmark establishes that violations can coexist with correct outcomes, but does not estimate how often this occurs in deployed systems.Its canonical institutions, compact role topology, and within-family generator make the three loci positive controls.Source: §7 · p. 8
- Transfer evidence rests on one independently developed workflow with 28 packaged pairs from 10 tasks, and its admission binding was judged ambiguous by both LLM reviews.A second workflow is exploratory because its locus was narrowed after execution.Source: §7 · p. 8
- The contracts cover four loci, exclude incentives, and leave simultaneous violations and open-ended tasks without exact oracles unresolved.Interventions perturb one mechanism at a time, restorations are supplied rather than discovered, and every external binding required manual source audit.Source: §7 · p. 8
8 Conclusion
A correct answer does not certify that collective mechanisms operated as intended. Diagnosis must be grounded in authoritative records, and benchmark performance alone does not establish transfer.
8 Conclusion
- A correct answer does not certify the mechanisms behind it; authoritative execution records, rather than outcomes or labels, license a diagnosis.In the opening execution, only the admission log revealed that the admission rule was violated despite the correct answer.Source: §8 · p. 8
- Diagnostic behavior that appears solved on a benchmark must be retested on the system where it will be used.Agreement on one benchmark does not show that a diagnoser transfers to another system.Source: §8 · p. 8
- Diagnostic contracts state what counts as a violation, which records can establish it, and when the honest verdict is unknown.Source: §8 · p. 8