Source-linked AI summary
Red-Teaming LLM Multi-Agent Systems via Communication Attacks
Pengfei He, Yupin Lin, Shen Dong, Han Xu, Yue Xing, Hui Liu
TL;DR
LLM-MAS depends on inter-agent communication, but its communication mechanisms remain vulnerable to interception and manipulation. This paper introduces AiTM, an LLM-powered adversarial agent with reflection, and finds attack success above 40% across all evaluated settings and above 70% in most experiments.
Problem
Inter-agent communication vulnerabilities in LLM-MAS, especially message interception and manipulation, remain largely underexplored despite communication’s central role.
Method
AiTM uses an external LLM-based adversarial agent to intercept victim-directed messages and generate contextually tailored malicious instructions through reflection.
Results
AiTM exceeds 40% attack success across all datasets, communication structures, frameworks, and goals, surpassing 70% in most experiments.
Takeaways & Limitations
AiTM exposes vulnerabilities in LLM-MAS communication mechanisms and highlights the need for securing inter-agent communication and building more resilient systems.
Takeaways & Limitations
The study uses black-box models and evaluates four representative communication structures plus two real-world applications rather than covering all structures.
Abstract
from arXiv · showhide
Large Language Model-based Multi-Agent Systems (LLM-MAS) have revolutionized complex problem-solving capability by enabling sophisticated agent collaboration through message-based communications. While the communication framework is crucial for agent coordination, it also introduces a critical yet unexplored security vulnerability. In this work, we introduce Agent-in-the-Middle (AiTM), a novel attack that exploits the fundamental communication mechanisms in LLM-MAS by intercepting and manipulating inter-agent messages. Unlike existing attacks that compromise individual agents, AiTM demonstrates how an adversary can compromise entire multi-agent systems by only manipulating the messages passing between agents. To enable the attack under the challenges of limited control and role-restricted communication format, we develop an LLM-powered adversarial agent with a reflection mechanism that generates contextually-aware malicious instructions. Our comprehensive evaluation across various frameworks, communication structures, and real-world applications demonstrates that LLM-MAS is vulnerable to communication-based attacks, highlighting the need for robust security measures in multi-agent systems.
1 Introduction
LLM-MAS coordinates specialized agents through communication, but that communication also creates an underexplored attack surface. AiTM exploits this surface by manipulating messages to induce malicious system behavior, achieving high success across evaluated settings.
- LLM-MAS uses specialized agents that divide complex tasks or debate to solve problems collaboratively.
- Communication lets agents share information, coordinate actions, validate decisions, and reduce errors through structures such as debates and majority voting.
- Malicious information can spread across agents, while redundant communication increases token overhead and computation costs.
- Existing attacks mainly compromise individual agents or process adversarial inputs, leaving interception and manipulation of inter-agent messages largely underexplored.
- AiTM intercepts messages sent to a victim agent and indirectly influences other agents without changing their profiles or capabilities.
- 40% and 70%: AiTM exceeds 40% attack success in all evaluated cases and surpasses 70% in most experiments.Applying AiTM to MetaGPT and ChatDev also compromises their performance.
2 Related works
Related work presents LLM-MAS as collaborative systems built from specialized agents, communication structures, and flexible frameworks. Prior security research examines compromised or malicious agents, while communication-focused vulnerabilities remain the paper’s motivating gap.
- LLM-MAS combines multiple specialized agents for planning, discussion, and decision-making through communication.
- Debating, majority voting, and task-specific dialogues are communication paradigms intended to enhance multi-agent system performance.
- Frameworks including AutoGen, Camel, and AgentScope support flexible construction of LLM-MAS.
- Prior security studies investigate agents being persuaded to abandon tasks, network effects on resilience, and disruption by malicious agents.
3 Agent-in-the-Middle attack
AiTM models an attacker that intercepts messages to a victim agent and uses an LLM-based adversary to generate tailored instructions. Reflection iteratively refines those instructions toward a malicious goal while respecting the victim’s role.
- 3.1 Agent settings: The LLM-MAS framework represents agents, directed communication connections, and exchanged messages, while omitting tools and external databases for simplicity.
- 3.1 Agent settings: Agents receive messages from selected agents and send messages to selected agents, with predefined message formats tailored to their responsibilities.
- 3.2 Threat model: The threat model lets an attacker intercept and manipulate communication involving one victim agent to pursue goals such as DoS or targeted behavior induction.
- 3.3 Attacking strategy: AiTM uses an LLM-based adversarial agent to intercept messages, generate malicious instructions for the victim, and thereby affect messages sent onward to other agents.
- 3.3 Attacking strategy: Reflection evaluates previous instructions and intercepted messages, then generates contextually tailored instructions that further align with the malicious goal.The process uses feedback from prior interactions as a reward signal for iterative prompt refinement.
4 Experiments
The experiments evaluate AiTM across frameworks, datasets, communication structures, attack goals, victim positions, adversarial persuasiveness, model choices, and real-world applications. AiTM consistently compromises LLM-MAS, with effectiveness shaped by task, structure, position, persuasion, model strength, and application design.
- Main results: AiTM achieves success rates above 40% across datasets, communication structures, frameworks, and attack goals, often exceeding 70%.DoS attacks exceed 80% in 23/32 Table 1 cells.
- Main results: Target behavior attacks are stronger on code-generation tasks, with minimum success of 73.1%, than on MMLU answer alteration, which reaches 40.7%.The paper relates this difference to harmful auxiliary functions being less noticeable than changed answer labels.
- Main results: DoS attacks exceed 80% in 23/32 Table 1 cells, making denial of service especially successful across tasks.The paper attributes this to refusing all requests being simpler than producing a specific predefined behavior.
- Communication structures: The Chain structure is most vulnerable because intercepted messages directly affect all subsequent agents.Under AutoGen, HumanEval and MBPP success rates reach 95.2% and 96.9%; under Camel, they reach 97.6% and 98.5%.
- Communication structures: Complete, Tree, and Random structures provide comparatively greater resistance through agent discussions, layered communication, or additional benign agents.Complete target attacks on MMLU have ASR around 40–50%, while Tree interception of a leaf limits subsequent attacker control.
- Victim position: Victim position matters: higher-level Tree agents and later Complete-structure messages generally produce stronger attacks.AiTM exceeds 80% on 27/32 victim-position cells; parent attacks in Tree improve target behavior by about 15%, while later Complete messages improve it by over 30%.
- Adversarial agent: Increasing adversarial persuasiveness raises attack success, with AutoGen Target ASR on MMLU-bio increasing from 19.5% to 27.1% to 40.7%.Persuasiveness is controlled through credibility and expertise, content and structure, and emotion and attitude.
- Model choice: A stronger model for the adversarial agent improves attack performance, while stronger models inside LLM-MAS improve resistance.For AutoGen Target attack on MMLU-bio, adversarial GPT-4o raises ASR from 43.9% with GPT-3.5-turbo to 57.9%.
5 Conclusion
AiTM targets inter-agent communication and demonstrates that manipulating exchanged messages can compromise LLM-MAS. The study also notes a scope constraint for SoftwareDev evaluation.
- AiTM directly targets inter-agent communication and can compromise the entire LLM-MAS by manipulating exchanged messages.
- SoftwareDev evaluation uses only public problems because its full version was not released.
Limitations
The study uses black-box models and evaluates four representative communication structures plus two real-world applications, leaving broader coverage for future work.
- The experiments use black-box models, limiting evaluation to the models tested rather than providing white-box analysis.
- The study covers four representative communication structures and two real-world applications, while a fuller categorization remains out of scope.
A Prompts
The appendix provides the prompts used in the experiments and evaluates their persuasiveness across three dimensions using GPT-4o.
- The appendix provides detailed prompts used in the experiments.
- GPT-4o rates each prompt from 1 to 5 on credibility and expertise, content and structure, and emotion and attitude.
B Experiment details
The experiments implement representative communication structures and real-world multi-agent applications, including MetaGPT and ChatDev, with role- and phase-specific communication constraints.
- Main experiments: The main experiments implement four representative multi-agent structures with AutoGen and Camel.
- Real-world application: MetaGPT experiments use its default setup with five expert agents assigned to predefined software-development roles.
- Real-world application: The QA Engineer in MetaGPT tests code but cannot modify it.
- Real-world application: ChatDev uses linearly connected phases with intra-phase discussion, while predefined goals and outputs further constrain communication.
C Additional experiment results
Experiments with different adversarial-agent LLMs show that stronger models increase AiTM’s attack success rate across evaluated settings.
- Stronger adversarial-agent models achieve higher AiTM success rates.Experiments use AutoGen on MMLU-bio and Camel on HumanEval, covering Complete and Tree structures plus target behavior and DoS attacks.
D Case studies
This section introduces case studies without reporting their substantive findings in the supplied passage.
- The paper provides case studies, but their findings are not described here.
E Discussion on potential mitigation
AiTM can compromise LLM-MAS by disturbing inter-agent messages, while proposed defenses involve monitoring or restricting communication and each carries practical limitations. The discussion concludes that stronger mitigation methods remain necessary.
- Discussion on potential mitigation: AiTM is described as successfully compromising LLM-MAS by disturbing inter-agent messages.
- Discussion on potential mitigation: AiTM is stealthier than malicious-agent and adversarial-input attacks because it changes neither the LLM-MAS nor the system input.Profile checks and input filters therefore do not directly address this attack.
- Discussion on potential mitigation: External monitoring could detect malicious inter-agent messages, but it adds computation costs and may reduce utility by blocking normal communication.
- Discussion on potential mitigation: Strict communication formats and content may reduce AiTM risk, but they are inflexible and unsuitable for free-debate systems in some research domains.The passage specifically cites scientific and social science applications as settings where free debate may be needed.
- Discussion on potential mitigation: Better mitigation methods need to be developed to defend against AiTM.