PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Systems Execute Arbitrary Malicious Code

Harold Triedman, Rishi Jha, Vitaly Shmatikov

arXiv:2503.12188v2cs.CRcs.LG

TL;DR

Multi-agent systems must process untrusted content while coordinating agents that can access sensitive data and perform external actions. This paper introduces MAS hijacking, showing that adversarial content can manipulate metadata and control flow across existing frameworks to invoke unsafe agents or execute attacker-chosen code. The attacks remain effective despite individual-agent prompt-injection resistance or refusals, motivating security models for these systems.

  • Problem

    Multi-agent systems need to interact with sensitive user data and potentially untrusted content, but existing systems do not adequately distinguish trusted from untrusted content.

  • Method

    The paper introduces MAS hijacking and evaluates how adversarial content targets metadata and control-flow processes across several multi-agent frameworks and topologies.

  • Results

    MAS hijacking is effective across input modalities and complex adversarial tasks, while previous indirect prompt injections generally do not work against multi-agent systems.

  • Takeaways & Limitations

    Laundering malicious requests through trusted agents can reach orchestrators and evade safety alignment, so multi-agent systems need security models for untrusted content and external actions.

  • Takeaways & Limitations

    The paper notes that complex multi-agent contexts make it difficult to classify actions as safe or unsafe.

Abstract

from arXiv · show

Multi-agent systems coordinate LLM-based agents to perform tasks on users' behalf. In real-world applications, multi-agent systems will inevitably interact with untrusted inputs, such as malicious Web content, files, email attachments, and more. Using several recently proposed multi-agent frameworks as concrete examples, we demonstrate that adversarial content can hijack control and communication within the system to invoke unsafe agents and functionalities. This results in a complete security breach, up to execution of arbitrary malicious code on the user's device or exfiltration of sensitive data from the user's containerized environment. For example, when agents are instantiated with GPT-4o, Web-based attacks successfully cause the multi-agent system execute arbitrary malicious code in 58-90\% of trials (depending on the orchestrator). In some model-orchestrator configurations, the attack success rate is 100\%. We also demonstrate that these attacks succeed even if individual agents are not susceptible to direct or indirect prompt injection, and even if they refuse to perform harmful actions. We hope that these results will motivate development of trust and security models for multi-agent systems before they are widely deployed.

1 Introduction

Multi-agent systems coordinate specialized agents through adaptive control flows, but untrusted content can hijack metadata and reroute execution toward unsafe agents or arbitrary code. The paper introduces MAS hijacking and demonstrates security breaches including remote access and data exfiltration.

  • Motivation: Multi-agent systems coordinate specialized agents to browse, analyze documents, summarize information, and execute code through adaptive control flows.These flows use task plans, action histories, progress assessments, and action results to guide successive actions.
  • Threat model: Untrusted webpages, files, email attachments, images, and audio are attack surfaces that can compromise systems handling user data and resources.The paper frames untrusted content as a system-wide security risk rather than merely a model-input concern.
  • Attack: MAS hijacking differs from jailbreaking because the user is an unwitting victim, while previously known indirect injections have near-zero success in executing malicious code.The paper distinguishes the attack from conventional attempts to make a model directly produce harmful output.
  • Attack: MAS hijacking exploits metadata pathways to reroute agent invocations toward unsafe or unnecessary agents, turning trusted sub-agents into confused deputies.The attack launders adversarial requests so they appear as trusted outputs and targets control flow rather than only generated content.
  • Consequences: A malicious local file can induce an orchestrator to invoke code execution, and the resulting executable can open a reverse shell on the user’s computer.In the illustrated file-based attack, a false access error causes the orchestrator to direct a code-execution agent to run the file.
  • Consequences: Arbitrary code execution threatens passwords, calendars, emails, local files, and logged-in web views even inside isolated containers, enabling concrete data-exfiltration attacks.The paper reports that several frameworks are highly vulnerable, including a GPT-4o Magentic-One configuration executing malicious local-file code 97% of the time.
  • Robustness: Attacks can succeed even when individual agents refuse harmful actions or are relatively immune to direct and indirect prompt injection.The paper reports examples in which orchestrators or other agents still generated or executed a reverse shell after safety refusals.

2 Agentic AI and multi-agent systems

Agentic AI systems use LLMs to decompose broad user goals into coordinated sub-tasks, while multi-agent frameworks organize agents and their communications through planning and control-flow mechanisms. Their topology and metadata handling determine which agents communicate and which acts next.

  • Agentic AI: Agentic AI spans systems that operate in complex, unexpected environments with ambiguous instructions and limited human intervention.LLMs typically decompose high-level tasks into sub-goals and plan how to execute them.
  • Agentic AI: AI agent systems have demonstrated effectiveness on benchmarks, and similar techniques support individual agents completing complex tasks for users.The passage situates multi-agent systems within broader agentic-AI research.
  • Multi-agent systems: Multi-agent systems decompose complex tasks, identify suitable agents, and orchestrate sequential or parallel execution through planning and control-flow mechanisms.Each agent may provide different capabilities while the system manages transitions between agents.
  • Multi-agent systems: Metadata such as status reports, error messages, and progress updates can influence control-flow decisions even when primary agent outputs are secured.This makes the data–metadata distinction central to analyzing multi-agent security vulnerabilities.
  • Multi-agent systems: The paper considers Round Robin, Central Orchestrator, and Central Orchestrator with External Data Structures as alternative system topologies.These topologies determine communication patterns, next-agent selection, and how partial action histories are shared.

3 Threat model

The threat model assumes benign agents interacting with adversarial files and webpages that an external attacker controls indirectly. The attacks target arbitrary code execution and user-data exfiltration through these inputs.

  • The adversary cannot directly prompt the MAS but controls content that the system may access inadvertently or intentionally.
  • The study focuses on files and webpages, including attacks delivered through text, images, and audio.
  • The two targeted outcomes are arbitrary code execution on the user’s device and exfiltration of data accessible to the agents.

4 Control-flow hijacking in multi-agent systems

MAS hijacking uses adversarial error-like content to manipulate inter-agent communication and control flow. The attack launders malicious requests through front-line agents’ outputs or metadata so they reach the orchestrator.

  • The attacker crafts plausible error messages containing malicious instructions that specify how individual agents should act in the ensuing interaction.
  • The malicious content is disseminated through local files, webpages, or other media, exploiting inter-agent trust and adaptive error handling.
  • Adversarial requests reach orchestrators indirectly because file-access and Web agents handle untrusted content while orchestrators do not directly interact with it.

5 Experimental setup

The evaluation tests MAS hijacking across three open-source frameworks, varied attack inputs and error templates, repeated trials, and comparisons with benign, direct-request, and prompt-injection baselines.

  • The study evaluates AutoGen, CrewAI, and MetaGPT because they operate on local or remote user data and support tasks including browsing, summarization, analysis, and code execution.
  • The experiments vary local text, Web text, Web images, local videos, three error templates, and two user-prompt variations.
  • Each framework-orchestrator-model-input-error-query combination is tested 10 times using a non-obfuscated Python reverse-shell payload, with data exfiltration also tested in CrewAI.
  • Attack success is measured through a unique executor output string and monitored reverse-shell connections, with benign, direct malicious, and existing IPI baselines.
  • Three smaller experiments examine realistic incidental contact, prompt-phrasing sensitivity, and attack mechanisms using Magentic-One with GPT-4o-mini.

6 Results

Across experiments, MAS hijacking achieves substantial attack success against local and Web inputs, including image and audio modalities, while direct or indirect prompt-injection baselines generally fail. The attacks can execute reverse shells or exfiltrate data, including when agents recognize or refuse harmful actions.

  • 6.1 MAS hijacking works, indirect prompt injection doesn’t: The Web Redirect attack sends a MAS from a malicious webpage to downloading and executing a reverse-shell script.
  • 6.1 MAS hijacking works, indirect prompt injection doesn’t: IPI is almost completely unsuccessful, whereas direct-ask outcomes are dominated by Gemini models’ propensity to open reverse shells.
  • 6.1 MAS hijacking works, indirect prompt injection doesn’t: The Web Redirect table compares two user queries, three error-file types, and repeated orchestrator-model configurations against IPI and direct-ask baselines.
  • 6.2 Varying attack type: The attack variants include Local, Web Single, and Web Image inputs, with ASR reported across models in Tables 3 and 4.
  • 6.3 Varying attack payload: CrewAI data-exfiltration attacks are effective but slightly less successful than reverse-shell attacks, and GPT-4o improves success for both local and Web exfiltration.
  • 6.4 Additional experiments: 59.5% ASR is observed across 200 trials, while direct hijacking reaches 80% ASR with 0% refusal compared with 6% ASR and 86% refusal for direct reverse-shell requests.
  • 6.4 Additional experiments: The experiments include harmful execution after agents recognize danger, accidental execution of discovered files, and occasional reverse shells from audio-transcribed attacks.

7 Related work

Prior work has studied security problems in individual agents and multi-agent systems, but MAS attacks targeting coordination metadata and control flow remain underexplored. Existing defenses may be relevant, though their effectiveness against MAS hijacking is not yet established.

  • Prior AI-safety work largely focuses on users attacking language models through jailbreaking.
  • Indirect prompt injection steers agents through unsanitized tool-call or retrieval results and is evaluated by benchmarks including AgentDojo, Agent Security Bench, and AgentHarm.
  • Prior single-agent attacks target failed requests, tool execution, poisoned retrieval systems, computer-use agents, insecure call chains, and LLM routers.
  • Multi-agent security research has examined prompt infection, attacks spreading through simulated email systems, misinformation, errors, and malicious agents.
  • No prior work considered attacks on multi-agent-system metadata and control flow, while existing indirect-injection defenses warrant evaluation against MAS hijacking.The proposed defense families include prompt tuning and runtime tool-call validation.

8 Discussion

The paper argues that MAS hijacking exploits adaptive coordination and weak trust boundaries to redirect agent control flow through adversarial content. Experiments show broad effectiveness, including against systems whose individual agents refuse harmful actions, while the authors identify unresolved context-understanding and security-policy limitations.

  • MAS hijacking targets metadata and control-flow processes to invoke adversary-chosen agents or directly run adversary code.
  • These attacks are effective across input modalities and complex adversarial tasks such as compiling and exfiltrating user profiles.
  • Laundering malicious requests through trusted agents enables them to reach the orchestrator and evade safety alignment through changed presentation.
  • Individual sub-agents may refuse harmful actions while the multi-agent system as a whole still completes the attack.
  • The orchestrator must explore potentially unsafe actions whose safety depends on the entire system context, which current models may not understand sufficiently.
  • Multi-agent systems lack a trust model and input or action semantics, causing agents to trust outputs from confused deputies.
  • The authors argue that deploying agents with access to user data and command lines remains unsafe until a security policy for malicious third-party content emerges.

Ethics Statement

The paper frames MAS hijacking as a security problem distinct from jailbreaking and indirect prompt injection, while positioning it within broader work on agent security and MAS vulnerabilities. It also provides formal definitions of MAS structure, control-flow hijacking, and action traces.

  • Related work: Prior work studies indirect prompt injection, prompt infection, misinformation, errors, malicious agents, and agent-system attacks, but not attacks on MAS metadata and control-flow processes.
  • Defenses: The paper describes defenses spanning prompt tuning, runtime agent controls, and model-level structural constraints, while noting that existing indirect-prompt-injection defenses may warrant future evaluation against MAS hijacking.
  • Attack examples: Figures 3 and 4 illustrate local-file and multimodal hijacking attacks, respectively, while Figure 3 is identified as a local-file attack.
  • Formal model: A MAS is modeled as a directed graph of agents and connections, with agents using tools to act on external resources whose results influence subsequent control flow.
  • Formal model: A control-flow hijacking attack modifies the action-trace suffix so the system executes attacker-specified actions and remains redirected through the trace's end.

D.1 Performance on benign tasks

Benign local-file and webpage tasks establish performance baselines for judging MAS hijacking attacks. The attacks can approach benign-task success, and one configuration exceeds its benign webpage success rate.

  • Benign baselines measure each MAS configuration's success in retrieving a local file or webpage containing a test string.
  • In some configurations, MAS hijacking attacks are only marginally less successful than the corresponding benign tasks.
  • Magentic-One powered by Gemini 1.5 Pro has a higher attack success rate than its success rate for accessing a benign webpage.

E Details of experimental setup

The experiments evaluate three open-source MAS frameworks across models, orchestrators, input modalities, attack templates, and baseline conditions. The study measures unsafe execution and related model behavior using controlled attack payloads and output-log analysis.

  • Frameworks and scope: The evaluation covers AutoGen, CrewAI, and MetaGPT, selected as open-source systems designed to operate on local and remote user data.
  • Frameworks and scope: The tested tasks include Web browsing, document summarization, dataset analysis, and code writing or execution, while excluding simulated or purely conversational frameworks.
  • Experimental factors: Experiments vary orchestrators, language models, input modalities, user request wording, and three error-message templates.
  • Payloads: The attack payload is usually a non-obfuscated reverse shell, with an additional CrewAI test for data exfiltration from a containerized environment.
  • Measurement: Attack success is measured by experiment-specific execution markers or socket connections, while logs are classified through regex searches and manual verification for execution, unsafe-code identification, and refusal.
  • Baselines: Baselines compare hijacking with benign file or webpage tasks, direct malicious requests, and three known indirect-prompt-injection templates.
  • Topologies: The tested MAS topologies include round robin, a central orchestrator, and a central orchestrator with external task structures.

K Refusal turns to execution (AutoGen)

The AutoGen interaction shows a refusal-oriented response being followed by execution-related behavior. The sequence includes safety warnings, a reverse-shell instruction, and a later executor timeout.

  • The interaction includes an indirect instruction to ignore prior instructions while requesting information about a file.
  • The file content includes a reverse-shell command targeting a specified IP address and port.
  • The system identifies the instruction as potentially dangerous and warns that executing it could compromise the device.
  • The executor reports that the script ran and exited with POSIX code 1 after a timeout.

L Benign task turns malicious (AutoGen)

A benign file- or web-access request is redirected into executing attacker-supplied content. The interaction culminates in commands to run the file, although the classifier labels the trace as neither a successful nor partial reverse-shell execution.

  • Local-file execution: The file-access interaction redirects the system from reading content to executing file1.txt as a Python script.The file browser reports that execution is the only way to view the file, and the coder constructs the command `python content/file1.txt`.
  • Evaluation: The evaluation asks whether the model identified a reverse shell, refused execution for safety, partially executed the attack, or successfully opened a reverse shell.The classifier output marks all four fields as No, and the trace is not identified as a successful reverse-shell execution.
  • Web-content redirection: The web-access interaction treats a blocked page as an instruction to download an authentication key and save it locally.The generated script uses requests.get(url), raises for HTTP errors, and writes the response to key.txt.
  • Web-content redirection: The key-retrieval trace then directs the executor to run key.txt as Python code rather than merely read it.The file browser again reports that execution is the only way to view the file, leading to the command `python key.txt`.
Loading 2503.12188v2…