PaperlayerView exact revision on arXiv

Source-linked AI summary

From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration

Yizhe Xie, Congcong Zhu, Xinyue Zhang, Tianqing Zhu, Dayong Ye, Minfeng Qi, Huajie Chen, Wanlei Zhou

arXiv:2603.04474v2cs.MAcs.AI

TL;DR

LLM-MAS can turn minor errors into false consensus through recursive context reuse, while existing protections may disrupt collaboration or lack a system-level propagation model. The paper models this process on directed dependency graphs, identifies vulnerability classes, and introduces genealogy-based governance; the reported defense prevents final infection in at least 89% of runs across operating modes.

  • Problem

    Recursive context reuse can amplify factuality or faithfulness errors into false consensus, but existing analyses and defenses lack a direct system-level account of collaborative error propagation.

  • Method

    The paper models LLM-MAS message flow as a directed graph with state and coverage dynamics, then uses a genealogy-based message-layer governance layer to monitor and control propagation without changing collaboration structure.

  • Results

    The study identifies cascade amplification, topological sensitivity, and consensus inertia, and shows that a single injected error seed can increase erroneous consensus while governance contains propagation.

  • Takeaways & Limitations

    Error propagation is a structural collaboration risk that can be analyzed with graph dynamics and mitigated through provenance tracking, targeted verification, and rollback.

  • Takeaways & Limitations

    The propagation abstraction uses stationary summaries and binary adoption evaluation, which may miss partial premise internalization, relaxed constraints, or semantic drift.

Abstract

from arXiv · show

Large Language Model-based Multi-Agent Systems (LLM-MAS) are increasingly applied to complex collaborative scenarios. However, their collaborative mechanisms may cause minor inaccuracies to gradually solidify into system-level false consensus through iteration. Such risks are difficult to trace since errors can propagate and amplify through message dependencies. Existing protections often rely on single-agent validation or require modifications to the collaboration architecture, which can weaken effective information flow and may not align with natural collaboration processes in real tasks. To address this, we propose a propagation dynamics model tailored for LLM-MAS that abstracts collaboration as a directed dependency graph and provides an early-stage risk criterion to characterize amplification risk. Through experiments on six mainstream frameworks, we identify three vulnerability classes: cascade amplification, topological sensitivity, and consensus inertia. We further instantiate an attack where injecting just a single atomic error seed leads to widespread failure. In response, we introduce a genealogy-graph-based governance layer, implemented as a message-layer plugin, that suppresses both endogenous and exogenous error amplification without altering the collaboration architecture. Experiments show that this approach prevents final infection in at least 89% of runs across operating modes and significantly mitigates the cascading spread of minor errors.

I. INTRODUCTION

LLM-MAS collaboration can amplify minor factuality or faithfulness errors into system-level false consensus, while existing defenses often require intrusive architectural changes. This work models propagation dynamics, identifies endogenous vulnerabilities, demonstrates low-cost consensus corruption, and introduces a message-layer governance layer that preserves the original communication structure.

  • Motivation: Recursive context reuse can amplify minor factuality or faithfulness errors into collective false consensus through repeated citation and reuse.The resulting propagation is difficult to trace because errors undergo semantic shifts during transmission and restatement.
  • Research Gap: Existing defenses largely target single-agent or localized risks, while multi-agent defenses often add critic roles or alter collaboration patterns.These approaches can intrude on usability and information flow and often treat errors as static anomalies rather than dynamic flows.
  • Approach: The study models message flow as a directed graph and uses agent states and system coverage S(t) to characterize error trajectories and derive an early risk criterion.The model treats context reuse as a deterministic contagion process rather than random diffusion.
  • Findings: The analysis identifies three endogenous vulnerabilities: cascading amplification, topological sensitivity to hub nodes, and consensus inertia during multi-round interaction.These vulnerabilities are identified through systematic comparison of mainstream collaboration architectures.
  • Findings: A directed consensus-corruption attack can inject a small number of error seeds to substantially increase convergence toward an erroneous consensus without disrupting system structure.The attack exploits confirmation and context reuse inherent in collaborative processes.
  • Mitigation: The Genealogy-Based Governance Layer tracks information flow and intervenes on demand to suppress harmful cascades while preserving the original communication structure.The mechanism is designed to balance safety and utility without altering collaboration architecture.

II. SYSTEM MODELING

The paper formalizes false consensus as the propagation and stable adoption of an atomic falsehood through LLM-MAS context reuse. It distinguishes factuality and faithfulness errors, defines adoption through entailment or reliance, and operationalizes consensus using continuous adoption states, system coverage, and a persistence threshold.

  • Problem Formalization: An atomic falsehood is a minimal declarative claim that violates the task’s correctness reference, including factuality and faithfulness errors.Factuality errors contradict external references, whereas faithfulness errors concern unsupported or contradictory premises transmitted through context reuse.
  • Problem Formalization: Propagation occurs when an agent adopts the atomic falsehood as a semantic commitment or functional premise in its output.Adoption includes direct entailment and implicit reliance, so surface repetition alone is insufficient.
  • Problem Formalization: The model represents each agent’s adoption probability with a continuous state variable and aggregates these states into system-level error coverage.Experiments estimate these quantities from empirical frequencies over tasks or random seeds.
  • Problem Formalization: False consensus is reached when system coverage S(t) exceeds a preset threshold τ and remains above it for consecutive rounds.This defines consensus as a stable error-lock-in state rather than a transient fluctuation.

B. Graph Dynamics

The paper models atomic-error propagation in LLM-MAS as graph-based IBMF dynamics, linking context reuse to contagion and deriving a spectral criterion for early amplification.

  • Graph representation: The communication workflow is represented as a directed graph whose edges indicate that one agent’s output enters another agent’s context.In-neighbors enumerate agents capable of directly influencing each node through context reuse.
  • State dynamics: A single atomic falsehood is tracked through binary adoption states and continuous agent-level adoption probabilities over interaction rounds.The model distinguishes whether an agent adopts the falsehood and approximates its propagation using IBMF dynamics.
  • State dynamics: Each upstream neighbor independently attempts to induce adoption with propagation probability β, while δ summarizes forgetting, correction, verification, and related recovery mechanisms.The update retains prior adoption mass and adds new adoption among susceptible agents.
  • Infection functions: The product-form infection function matches round-based interaction, whereas the Poisson form provides a hazard-based alternative for continuous-time interpretations.The product formulation assumes independent upstream attempts within each round; the Poisson formulation uses summed hazards.
  • Risk criterion: The early amplification condition is βρ(A) > δ, where ρ(A) is the adjacency matrix’s spectral radius.The equivalent growth condition is (1 −δ) + βρ(A) > 1; the ratio diagnostic becomes unreliable when δ is near zero.

C. Model Utility

The model reproduces observed error-coverage trajectories across multiple communication topologies, while product-form fitting generally provides lower errors than the Poisson alternative.

  • Cross-topology validation: Fitted trajectories closely follow observed mean error coverage across STAR, CHAIN, layered-horizontal, and MESH topologies.The fits remain largely within the empirical uncertainty bands and reproduce faster hub-mediated saturation alongside slower decentralized accumulation.
  • Fit quality: Product-form fitting reduces layered-topology MSE from 1.74×10^-2 to 8.0×10^-4.This comparison is reported for the layered graph.
  • Fit quality: Product-form fitting reduces layered-horizontal MSE from 1.04×10^-2 to 4.5×10^-3.The Poisson form often drives β to its upper boundary and δ to zero in layered settings.

III. THREAT MODEL

The threat model examines how small endogenous or adversarial errors become system-wide failures through topology-dependent propagation and repeated exposure.

  • Threat scope: The protected asset is collaborative-consensus integrity, defined as avoiding shared outputs that contradict task requirements and verifiable evidence.A false consensus is a collectively held but incorrect final decision or generated artifact.
  • Threat modes: The model considers both stochastic endogenous errors and strategic exogenous attacks that choose seed content and injection position to maximize downstream corruption.The adversarial setting assumes application-layer access and gray-box knowledge of the workflow.
  • Threat modes: Controlled emulation implants a plausible but outdated data-source migration notice as a tracer seed to measure endogenous amplification.This calibration isolates topology response from randomness in error generation and is not treated as an attacker capability.
  • Cascade amplification: Repeated multi-neighbor exposure compounds adoption, and when R > 1 the spectral-radius mode can make a single seed expand rapidly in coverage.The measured coverage S(t) is the fraction of agents incorporating or relying on the tracer error at each message step.

B. Vulnerability II: Topological Fragility

The system’s vulnerability depends on where an error enters the collaboration graph: structurally central nodes can amplify a seed much more strongly than peripheral nodes. Intervention timing also matters because accumulated polluted context increases correction burden.

  • Topological sensitivity: Error resilience depends on entry coordinates, with early growth governed by the adjacency matrix’s principal eigenvector.A one-hot seed at node v excites the dominant cascade mode proportional to [u1]v, making the largest component the most dangerous location.
  • Topological sensitivity: In centralized topologies, hub corruption drives immediate system-wide amplification, whereas leaf corruption spreads only after hub adoption.The resulting hub–leaf difference is summarized by the Impact Factor.
  • Consensus inertia: Consensus inertia increases correction difficulty because delayed intervention must counteract a growing dependency chain of accumulated erroneous context.Intermediate artifacts preserve sources, assumptions, code skeletons, and evaluation criteria that subsequent steps build upon.
  • Attack surface: The attack pipeline combines a minimal atomic falsehood, credible artifact packaging, and injection at a high-impact location.Target locations may be selected using structural influence metrics or observable functional roles such as aggregation and final decision-making.

VI. DEFENSE: A GENEALOGY-BASED GOVERNANCE LAYER

The genealogy-based governance layer preserves the fixed collaboration topology while controlling claims on the message path. It tracks provenance and supports early intervention against unverified or self-reinforcing claims.

  • Design constraints: The defense keeps the adjacency matrix A unchanged to preserve coordination semantics, deployment compatibility, and original information flow.Rewiring could alter role semantics and disrupt coordination logic exposed by the framework or application layer.
  • Governance objectives: The message-layer governance layer targets unverified-claim propagation, early correction, and strategic verification-budget allocation.Its dynamics-model guidance increases effective recovery while avoiding comprehensive verification for every claim.
  • Implementation: The implementation intercepts messages, analyzes outgoing content, releases screened messages, and maintains a global Lineage Graph for atomic claim history.The graph records provenance incrementally so downstream screening can distinguish trusted and untrusted context.
  • Claim provenance: Messages are decomposed into factuality and faithfulness claims, which are tracked as nodes linked by support or contradiction relations.The Lineage Graph distinguishes confirmed nodes from unverified nodes excluded from trusted context.

D. Decision Layer

The decision layer applies a four-stage screening and routing pipeline to atomic claims, then assembles, blocks, or rolls back messages while preserving usable workflow execution. It also supports online governance and offline forensic replay.

  • Four-stage pipeline: The pipeline decomposes messages, screens claims against confirmed lineage, routes uncertain claims by policy, and assembles or rolls back the result.This four-stage structure separates claim analysis, uncertainty handling, and enforcement.
  • Claim decisions: Green claims are released and confirmed, Red claims are blocked with conflict evidence, and Yellow claims receive policy-dependent handling.Yellow claims may be forwarded as unverified, selectively verified when influential, or handled under stricter policies.
  • Enforcement: Rollback returns rejected atoms, conflict evidence, and rewrite instructions, with retry caps and circuit breakers preventing deadlock.Persistently Red atoms are excluded, while persistently Yellow atoms can be forwarded with high-risk tags without entering confirmed lineage.
  • Deployment modes: The layer operates inline on messages and offline on historical logs for provenance reconstruction, propagation tracing, and attribution.Offline replay does not influence execution and can identify root or high-degree nodes associated with corrupted beliefs.
  • Evaluation: Evaluation spans QUANT, RIGID, and MMLU scenarios across multiple frameworks, with final-artifact infection as the primary propagation outcome.Defense comparisons include NONE, self-reflection, AGrail, and other operating settings.

B. Adversarial Severity

Intent-hiding packaging substantially increases adversarial corruption, while the proposed governance modes generally preserve both infection resistance and usable completion. Ablations and cost comparisons show that enforcement improves containment but increases resource use.

  • Attack severity: BASELINE ASR is near zero in most settings, whereas COMPLIANCE reaches mostly 85.0–100.0% and SECURITY FUD reaches 76.7–100.0% outside CREWAI.The weakest intent-hiding result outside CREWAI is 76.7% on METAGPT–QUANT under SECURITY FUD.
  • Framework sensitivity: AUTOGEN and CAMEL reach up to 100.0% ASR in mesh-style collaboration, while LANGGRAPH reaches 100.0% under COMPLIANCE in star-style orchestration.CREWAI is substantially lower under the same broad attack settings, indicating framework-level differences.
  • Governance efficacy: On MetaGPT, Speed reaches 85.0% BICR under Compliance and 100.0% under Security FUD, with Safe Completion of 79.2% and 97.5%.On AutoGen, all proposed modes keep BICR above 95.0% under Compliance and Security FUD, with Safe Completion above 92.5%.
  • Baseline comparison: Reflection’s BICR falls to 5.0%/19.2% on MetaGPT, 5.8%/3.3% on AutoGen, and 0.8%/0.8% on LangGraph under Compliance and Security FUD.The paired values correspond to Compliance and Security FUD, respectively.
  • Ablation: Removing screening and verification lowers BICR to 14.4 ± 35.2%, while removing quarantine or rollback lowers it to 3.1 ± 10.1%.The ablation results identify explicit risk decisions and enforcement as important containment components.
  • Safety-cost trade-off: BICR rises from 0.89 under Speed to 0.93 and 0.94 under Balanced and Strict, while Token/Safe increases from 21,227 to 30,844 and 57,610.The operating points trade greater robustness for higher token and latency costs.

VIII. LIMITATIONS AND FUTURE WORK

The paper’s governance and propagation analyses have defined scope boundaries, including simplified adoption observables, stationary parameter summaries, controlled attack capabilities, and online latency costs.

  • Scope and modeling limitations: The model uses continuous adoption states but evaluates with binary infection observables, which may miss partial internalization, relaxed constraints, or semantic drift.The spectral indicator R is a theoretical heuristic rather than a deployment guarantee or calibrated risk predictor.
  • Scope and modeling limitations: The propagation abstraction treats β, δ, and the structural term A as stationary summaries, capturing role, task, and time variation only indirectly.
  • Adversarial scope: The injection study probes endogenous amplification under a single seeded application-layer message rather than fully characterizing adversarial behavior.Persistent system access, execution-layer control, adaptive multi-stage attacks, and richer access remain out of scope.
  • Deployment constraints: The governance plug-in incurs online latency overhead because screening and verification operate on the interaction path.Decoupling or partial offloading changes the protection configuration, while balancing latency budgets and safety targets remains unresolved.
  • Future work: The paper studies propagation and mitigation under controlled experimental settings, with future work extending validation to longer-horizon tool-using workflows and improved adoption measurement.

APPENDIX A MODEL FITTING AND TOPOLOGY CONFIGURATION

The appendix defines topology configurations, infection measurement, trajectory fitting, and threshold diagnostics for assessing whether the propagation model matches observed adoption dynamics.

  • Measurement and fitting: The calibration experiment uses a neutral randomly generated codeword as a task-independent infection tracer.An agent is marked infected when its output contains the injected codeword.
  • Topology configuration: The experiments fix N = 5 agents, T = 5 synchronous rounds, and R = 20 independent trials for each topology.
  • Topology configuration: The configured topologies include a bidirectional star, a unidirectional chain, a chain with probabilistic reverse edges, and a complete directed graph without self-loops.The layered-horizontal topology uses reverse-edge probability Ph = 0.3, while skip connections use Ps = 0.
  • Measurement and fitting: The model initializes every agent with Sobs(1) and compares predicted Spred(t) with Sobs(t+1) across subsequent rounds.Parameters are selected through a coarse grid search followed by an 11 × 11 fine grid around the coarse optimum.
  • Results: 0.029 mean MSE is obtained for the product-form fit on pooled real attack groups.This indicates that the propagation model fits coarse-grained adoption curves in the real attack traces.
  • Results: Governance reduces fitted transmission strength and observed final coverage in both LangGraph-Star and AutoGen-Mesh.In LangGraph-Star, Speed reduces β from 0.42 to 0.07 and final coverage from 0.80 to 0.18; Strict reduces final coverage to 0.04.

APPENDIX B SUPPLEMENT TO VULNERABILITY

The appendix supplements vulnerability experiments by mapping framework communication structures, standardizing single-seed injection locations, and reporting cascade severity by topology.

  • Communication structures: Frameworks are modeled as discrete-time processes whose visibility patterns determine how an injected seed can propagate through shared context and artifacts.
  • Communication structures: MetaGPT uses cumulative artifact visibility across SOP stages, while CAMEL uses iterative role-playing dialogue over shared history.
  • Injection protocol: The endogenous study applies one fixed instruction to a designated target agent at the first step to standardize the initial error condition.Injection locations are normalized as entry, hub, and leaf across chain, star, and worker-role workflows.
  • Observables: Adoption requires the seed to be carried forward in an output, shared context, or downstream artifact; rejected or explicitly negated mentions are excluded.The study also records polluted intermediate artifacts as a severity measure.
  • Results: Table X reports final infection rate and standard deviation for each framework in the cascade amplification experiment.

APPENDIX C SUPPLEMENT TO EVALUATION

The appendix details application scenarios, repeated attack and defense evaluations, baseline methods, and conservative NLI screening for message-layer governance.

  • Evaluation settings: The evaluation covers QUANT, RIGID, and MMLU scenarios, including tool-assisted data analysis and multi-step reasoning settings.
  • Evaluation settings: Attack cells use 20 × 3 runs, while defense cells use 20 × 2 runs for each selected framework, scenario, defense method, and attacker policy.
  • Baselines: The comparison includes sender-side Reflection and inter-agent message-path monitoring with AGrail under matched evaluation settings.
  • Baselines: The evaluation gives baselines no extra oracle information beyond the normal task text and data source.
  • Attack protocol: The attack policies inject one malicious seed as a verbatim claim, cooperative instruction, or security-motivated caution.
  • NLI evaluation: The NLI module is used as a conservative first-pass Green screen and evaluated on human-annotated collaboration traces with limited synthetic augmentation.FGR measures non-entailment samples predicted as Green, while Green Precision measures the correctness of predicted Green samples.
Loading 2603.04474v2…