Source-linked AI summary
On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents
Jen-tse Huang, Jiaxu Zhou, Tailin Jin, Xuhui Zhou, Zixi Chen, Wenxuan Wang, Youliang Yuan, Michael R. Lyu, Maarten Sap
TL;DR
Faulty agents may disrupt multi-agent collaboration, while the effects of system structure on resilience remain underexplored. The paper simulates agent faults and evaluates resilience across systems and tasks. The hierarchical structure has the least performance drop at 5.5%.
Problem
Prior studies of multi-agent safety focused on toxicity or misinformation, overlooking disruption of collaboration on general tasks and the impact of system structure on resilience.
Source: §1 · p. 2Method
The paper introduces AUTOTRANSFORM and AUTOINJECT to simulate agent faults, then proposes Challenger and Inspector to improve resilience.
Source: §1 · p. 2Results
The hierarchical structure exhibits the least performance drop at 5.5%.
Source: §1 · p. 2Takeaways & Limitations
Combining Challenger and Inspector enhances resilience most for the two more vulnerable systems.
Source: §1 · p. 2The evaluation uses four text-only benchmarks with automatic metrics, so conclusions may not extrapolate to multimodal interaction, open-ended dialogue, or long-horizon planning.
Source: D. Limitations · p. 25
Abstract
from arXiv · showhide
Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents, each focusing on a specific domain. However, the impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored. This paper investigates: (1) What is the resilience of various system structures (e.g., A$\rightarrow$B$\rightarrow$C, A$\leftrightarrow$B$\leftrightarrow$C) under faulty agents, on different downstream tasks? (2) How can we increase system resilience to defend against these agents? To simulate faulty agents, we propose two approaches--AutoTransform and AutoInject--which introduce mistakes into the agents' responses. Experiments on four downstream tasks using six systems show that the "hierarchical" structure, i.e., A$\rightarrow$(B$\leftrightarrow$C), exhibits superior resilience with the lowest performance drop of 5.5%, compared to 10.5% and 23.7% of other two structures. To further improve resilience, we introduce (1) Challenger, that introduces a mechanism for each agent to challenge others' outputs, and (2) Inspector, an additional agent to review and correct messages, recovering up to 96.4% errors made by faulty agents. Our code and data are available at https://github.com/CUHK-ARISE/MAS-Resilience.
1. Introduction
The paper examines how faulty agents disrupt multi-agent collaboration and how system structure affects resilience. It introduces methods to simulate faults and improve resilience, then evaluates structures and tasks.
1. Introduction
- Prior safety studies focused on toxicity or misinformation, leaving disruption of general-task collaboration and the effect of system structure on resilience underexplored.The paper presents its work as the first examination of how multi-agent structure affects resilience when faulty agents disrupt collaboration.Source: §1 · p. 2
- AUTOTRANSFORM creates faulty agents while retaining their original functionality, whereas AUTOINJECT automatically inserts errors into messages exchanged among agents.Both approaches enable controlled error introduction without manual modifications.Source: §1 · p. 2
- The study evaluates six systems across linear, flat, and hierarchical structures on code generation, math problem solving, translation, and text evaluation.It also examines error type and error rate in code generation.Source: §1 · p. 2
- Challenger lets agents challenge received messages, while Inspector adds an agent that reviews and corrects messages.The strategies respectively mirror profile modification and message interception used to introduce faults.Source: §1 · p. 2
- 5.5%: the hierarchical structure exhibits the least performance drop among the structures examined.The paper also reports that code generation is most affected by malicious agents, with a 22.6% performance drop.Source: §1 · p. 2
- Up to 96.4% of performance lost to faulty agents is recovered by the Challenger and Inspector strategies in Self-Collab and Camel.The cited passage names these linear and flat systems, respectively.Source: §1 · p. 2
2. Preliminaries
The paper frames multi-agent collaboration structures using organizational forms and agent communication patterns. It describes resilience as the ability to handle internal errors and maintain overall operation without being affected by a single failure.
2. Preliminaries
- Linear systems pass messages one way in a chain, flat systems use mutual communication, and hierarchical systems combine one-way and mutual communication, as in A→(B↔C).These categories are drawn from collaboration modes in management science.Source: §2 · p. 2
- The formalization treats a system as a directed graph whose vertices are agents and whose edges represent reporting relationships.Linear systems are directed paths, flat systems are bidirectional complete graphs, and hierarchical systems are rooted directed trees.Source: §2 · p. 2
- Resilience refers to handling internal errors and maintaining overall operation without being affected by a single failure.The paper makes this ability relevant because errors from clumsy or malicious agents may be too stealthy for others to detect and can have undesired consequences.Source: §2 · pp. 2–3
3. Methodology: Introducing Errors
The methodology introduces two ways to create faulty agents: rewriting agent profiles or directly injecting errors into messages. It distinguishes message-level error frequency from error amount within a message, and separates syntactic from semantic errors.
3. Methodology: Introducing Errors
- The methodology uses “message” for an intermediate agent output and “result” for the final output from the last agent.It presents AUTOTRANSFORM and AUTOINJECT as approaches for introducing errors into multi-agent systems.Source: §3 · p. 3
3.1. AUTOTRANSFORM: Faulty Agent Transformation
- AUTOTRANSFORM analyzes an agent’s assigned task, identifies stealthy error methods, and rewrites its profile while preserving its original functions.It takes an agent profile as input and produces a faulty profile that performs the same functions while introducing stealthy errors.Source: §3.1 · pp. 3–4
3.2. AUTOINJECT: Direct Error Injection
- AUTOINJECT addresses AUTOTRANSFORM’s difficulty guaranteeing specific error counts and types by injecting specified errors into agents’ messages.It gives exact control over the proportion of erroneous messages, errors within a message, and error types.Source: §3.2 · p. 4
- Pm denotes the probability that a faulty agent’s message is intentionally flawed, while Pe denotes the proportion of errors within a message.Pm controls the macro-level ratio of erroneous messages; Pe controls the micro-level degree of error within each faulty message.Source: §3.2 · p. 4
- Syntactic errors violate logical or factual correctness, whereas semantic errors can be sound in form yet irrelevant or fail to follow the intended instruction.Source: §3.2 · p. 4
- AUTOINJECT selects messages with probability Pm and replaces Pe of their lines or sentences with errors generated by an LLM.Inputs include task specifications, agent details, error rates, and error type; the defaults are Pm = 1.0, Pe = 0.2, and semantic errors.Source: §3.2 · p. 4
4. Experiments
The experiments compare six multi-agent systems across four downstream tasks, testing how system structure, task type, error rate, and error type affect resilience. They also examine cases where faulty agents improve performance and factors that may shape error recovery.
4.1. RQ1: Impact of System Structures
- The hierarchical structure had the smallest average performance drop (−5.51), ahead of flat (−10.54) and linear (−23.72), across both backbones and error-introduction methods.The ranking remained consistent for GPT-3.5 and GPT-4o with both AUTOTRANSFORM and AUTOINJECT.Source: Figure 3 · p. 5, §4.1 · p. 5
4.2. RQ2: Impact of Downstream Tasks
- Code generation (−22.56) was most vulnerable to errors, followed by math (−9.89), text evaluation (−5.42), and translation (−4.70).These task comparisons average across the multi-agent systems; code generation and math require greater objectivity and formalization than translation and text evaluation.Source: Figure 4 · p. 6, §4.2 · p. 6
4.3. RQ3: Impact of Error Rates
- Increasing the number of errors within a message reduced performance quickly in one tested setting, while increasing the faulty-message ratio also reduced performance, but less strongly.At higher within-message error levels, three systems improved from 0.4 to 0.6 as errors became more noticeable and prompted corrections.Source: §4.3 · pp. 6–7, Figure 5 · p. 7
4.4. RQ4: Impact of Error Types
- Semantic errors caused greater performance drops than syntactic errors across the six systems, while MAD improved with injected syntactic errors.Syntactic irregularities are easier to identify; semantic errors require deeper task understanding.Source: Figure 5 · p. 7, §4.4 · p. 7
4.2. RQ2: Impact of Downstream Tasks
- AUTOINJECT sometimes improved performance, with gains up to 12.1% in MAD on GPT-3.5 text evaluation and 4.2% in MAD on GPT-4o code generation.Proposed explanations include double-checking that fixes pre-existing errors and divergent thinking that breaks repetitive discussion loops.Source: §4.2 · p. 6, Table 2 · p. 7
4.5. Case Study
- Faults in higher-level task distributors produced greater declines, while more Camel conversation rounds did not reliably distinguish successful from failed code generation.Camel agents spent more rounds on non-passing code after errors were injected, yet still failed to solve harder cases.Source: §4.5 · pp. 7–8
- In a HumanEval case, an injected error prompted another agent to correct both that error and a pre-existing code error.The responding agent requested a correction without identifying the original error.Source: §4.5 · p. 7, Figure 6 · p. 8
- A distraction comment claiming a bug was fixed led all six systems to overlook incorrect code and proceed to the next task.Without the comment, the systems detected the code error; AUTOTRANSFORM exploits this tendency to prioritize comments over code.Source: §4.5 · p. 7, Figure 6 · p. 8
5. Improving System Resilience
The paper proposes two correction mechanisms for improving resilience to faulty agents: agents can challenge one another, or an additional agent can inspect messages. Used together, they recovered nearly all performance loss in the tested Self-collab system.
Methods
- Challenger adds profile instructions that let agents challenge others’ results, addressing cases where agents fail to raise problems despite having relevant knowledge.The authors note that the systems often use the same LLM backbone across agents.Source: §5 · p. 8
- Inspector is an additional agent that intercepts messages, checks them for errors, and corrects them.The two correction methods are compatible and can be used concurrently.Source: §5 · pp. 8–9
Results
- Combining Challenger and Inspector recovered 96.4% of performance loss in Self-collab, and all tested strategies improved performance against errors.The methods were also tested separately and together on the linear Self-collab and flat Camel systems; no definitive mapping between a method and a specific error-introduction method was established.Source: Table 3 · p. 8, §5 · p. 9
6. Related Work
The related work covers task-oriented multi-agent frameworks and safety research, while distinguishing this paper’s focus on collaboration disruption and architectural resilience from prior studies.
6.1. Multi-Agent Systems
- Task-oriented frameworks span debate, role-based code generation, generic multi-agent applications, and dynamically generated agent profiles; the study excludes systems that are not task-oriented or overlap with its selected designs.Examples include ChatEval (Chan et al., 2024), ChatDev (Qian et al., 2024), AutoGen, and AutoAgents (Chen et al., 2024a).Source: §6.1 · p. 9
6.2. Safety Issues in Multi-Agent Systems
- Related adversarial work studies attacks on agents and collaboration, but does not examine how different architectures are affected; NetSafe also omits subjective tasks such as translation and text evaluation.This study additionally includes code generation to examine the impact of error types.Source: §6.2 · p. 9
7. Conclusion
The conclusion compares linear, flat, and hierarchical collaboration under faulty agents across six systems and four downstream tasks. It reports the hierarchical system’s lowest performance drops and summarizes how task type and error characteristics relate to resilience.
7. Conclusion
- 12.1% and 9.2% were the hierarchical system’s lowest performance drops under AUTOTRANSFORM and AUTOINJECT, respectively.The study evaluates six systems across code generation, math problem solving, translation, and text evaluation.Source: §7 · p. 9
- Some systems improved when errors were intentionally injected, while objective tasks such as code generation and math were more significantly affected by errors.Source: §7 · p. 9
- The frequency of erroneous messages affected resilience more than the number of errors within a single message.Source: §7 · p. 9
- Systems showed greater resilience to syntactic errors than to semantic errors.Source: §7 · p. 9
- The authors recommend designing hierarchical multi-agent systems, noting that this structure reflects a prevalent collaboration mode in real-world human society.Source: §7 · p. 9
Impact Statement
The authors warn that AUTOTRANSFORM and AUTOINJECT could pollute benign agents and cause negative social impacts. They propose Challenger and Inspector as defenses and oppose malicious use of these methods.
Impact Statement
- AUTOTRANSFORM and AUTOINJECT could pollute benign agents and result in negative social impacts.Source: Impact Statement · p. 10
- The authors propose Challenger and Inspector as defense mechanisms against the risks posed by the error-introducing methods.Source: Impact Statement · p. 10
- The stated goal is to study and improve LLM-based multi-agent collaboration, and the authors oppose malicious use of the methods.Source: Impact Statement · p. 10
A. Quantitative Results
The tables cover task performance by system structure and downstream task, plus code-generation performance across error rates and error types.
A. Quantitative Results
- Table 4 is titled as reporting task performance by system structures.Source: Table 4 · p. 14
- Table 5 is titled as reporting task performance by downstream tasks.Source: Table 5 · p. 14
- Table 6 is titled as reporting code-generation performance with different error rates.Source: Table 6 · p. 14
- Table 7 is titled as reporting code-generation performance with different error types.Source: Table 7 · p. 14
B.1. Error Type Analysis
The appendix examines injected-error diversity, single- versus multi-agent resilience, and fault propagation across multiple faulty roles and graph structures. It also reports limitations in controlling AUTOTRANSFORM error rates and shows stronger performance for hierarchical coordination in the tested settings.
B.1. Error Type Analysis
- AUTOINJECT errors in code generation span seven categories, and the analysis uses diverse faults to reduce the risk of any one category dominating.The table reports statistics for 80 injected errors.Source: §B.1 · p. 15, Table 8 · p. 15
- In controlled syntax-error tests, AUTOTRANSFORM averaged 1.56% and 9.49% realized line errors against 20% and 40% targets, while AUTOINJECT controls corruption location and amount deterministically.AUTOTRANSFORM’s large variation made controlled experiments and fair comparisons impossible; the quantitative error-type analysis therefore focuses on AUTOINJECT.Source: §B.1 · p. 15
B.2. Faulty Single Agent Systems
- Multi-agent systems showed better resilience than a single GPT-3.5 agent under both fault-introduction methods, with other agents reviewing and testing the faulty agent’s work.The comparison is presented in Tables 9 and 10.Source: §B.2 · p. 15, Table 9 · p. 15, Table 10 · p. 15
B.3. Multiple Faulty Agents
- With AUTOINJECT in AgentVerse Math, accuracy fell from 28.0 with only the Solver faulty to 14.0 with Solver and Critic faulty, and 12.0 when Planner was faulty instead.Under AUTOTRANSFORM, performance was 16.0 with one faulty agent, 14.0 with Solver and Critic faulty, and 2.0 with Solver and Planner faulty.Source: §B.3 · p. 16
B.4. Advanced Multi-Agent Systems
- On GPT-3.5 Math, a four-agent star graph scored 36.0 vanilla, 30.0 with AUTOTRANSFORM, and 28.0 with AUTOINJECT, versus 28.0, 20.0, and 16.0 for a complete graph.The authors report that the single leader mitigates error propagation, whereas fully flat peer discussion amplifies faults.Source: §B.4 · p. 16
B.5. More Realistic and Complex Tasks
A Snake-game task illustrates how faulty agents can disrupt end-to-end software, while tests with other backbones and prompting preserve the reported hierarchy-first resilience pattern. The section also gives prompt adaptations and templates for systems and fault-handling methods.
B.5. More Realistic and Complex Tasks
- Camel’s multi-agent system produced a playable Snake game in two of three conditions, while a single GPT-3.5 agent never produced a controllable game.AUTOTRANSFORM left the game runnable but introduced semantic faults; AUTOINJECT’s syntactic glitches made it unexecutable.Source: §B.5 · p. 16
B.6. More Models
- With LLaMA-3.1-70B-Instruct, hierarchy lost 9.2 points (76.15 to 66.96) under AUTOTRANSFORM and 20.5 under AUTOINJECT, versus flat/linear drops of 37.8/61.9 and 40.2/35.1, respectively.The authors report that CoT improves raw performance but does not immunize multi-agent systems against faulty peers.Source: §B.6 · p. 16
C. Prompt Details
- To extend four scalable systems to all four tasks, the authors adapted prompts; Camel received step-by-step instructions for translation and math.The examples include a translation interpretation and a corrected average-speed calculation.Source: C. Prompt Details · p. 17
Prompt for AUTOTRANSFORM
- AUTOTRANSFORM prompts instruct an agent to retain its apparent task while producing subtle, task-targeted errors in its profile.The prompt asks for specific ways to generate wrong results while preserving the appearance of performing the original task.Source: Prompt for AUTOTRANSFORM · p. 24
Prompts for AUTOINJECT
- AUTOINJECT prompts specify line-level syntactic or semantic code errors and task-specific distortions for math, translation, and text evaluation.The prompts direct rewriting existing responses or code to introduce the specified errors.Source: Prompts for AUTOINJECT · p. 24
Prompt for Challenger
- Challenger asks agents to classify a message as safe or unsafe, while Inspector asks an agent to identify and rewrite responses containing deliberate errors.Inspector’s prompt also allows a simple “safe” response when no deliberate error is found.Source: Prompt for Challenger · p. 25, Prompt for Inspector · p. 25
D. Limitations
The study’s conclusions are bounded by its models, benchmarks, automated evaluation, and system-specific implementations.
D. Limitations
- Budget constraints limited agents to GPT-3.5 or GPT-4o at temperature zero, and evaluation relied exclusively on automated scores.The authors state their goal was to compare resilience fairly and expect results not to differ greatly with other models.Source: D. Limitations · p. 25
- Experiments cover four text-only benchmarks with automatic metrics, so conclusions may not extrapolate to multimodal interaction, open-ended dialogue, or long-horizon planning.The study selected representative systems and commonly used datasets, but those choices do not establish transfer to the excluded settings.Source: D. Limitations · p. 25
- Each system uses its authors’ code base and role design, entangling structure with prompt engineering; residual prompt and implementation differences may affect measured resilience gaps.The authors keep frameworks as released and select two systems per structure to partially average out prompt variation.Source: D. Limitations · p. 25