PaperlayerView exact revision on arXiv

Source-linked AI summary

An Exploration of Attacks on AI Agent Systems

Cathleen Qiao, Layah Vigneaud, Chloe Brandon, Max Cabilangan, Xiyuan Cao, Isaac Khabra, Reva Mahajan, Lena Nguyen, Kyle Pak, Anagh Pandya, Roshni Dave, Peiran Wang

arXiv:2610.12524v1cs.CR

TL;DR

The paper examines prompt injection in single-agent systems and the less-studied security risks of multi-agent communication. It retests and extends attacks, including an A2A-specific agent-card attack, and reports vulnerabilities despite added defenses, with risks that can propagate across agents.

  • Problem

    Prompt injection has been studied primarily in single-agent settings, leaving its impact in multi-agent contexts less examined despite agents’ growing integration into workflows.

    Source: §I · p. 1

  • Method

    The paper retests and extends prompt-injection strategies across single-agent systems and A2A settings, including an Agent Card Spoofing attack.

    Source: §I · p. 1

  • Results

    Both single- and multi-agent systems remained vulnerable despite safeguards, classifiers, and filters; formatting tricks could bypass rules, and compromised agents could influence others.

    Source: §VI · p. 5

  • Takeaways & Limitations

    The authors call for proactive research and stronger security measures as AI agents become more prevalent.

    Source: §VI · p. 5

Abstract

from arXiv · show

As autonomous AI agents become more integrated into sensitive domains, it is crucial that single and multi-agent AI systems are secured. This paper investigates how prompt injection attacks can exploit both direct user inputs and inter?agent communication to leak sensitive information and trigger cross-agent infections. While our tests mostly highlight key vulnerabilities in single-agent systems, we also extend these attacks to multi-agent frameworks, investigating how weaknesses can propagate across agents. Our findings emphasize the need for more robust defenses as AI agents become more commonly embedded in real-world workflows.

I. INTRODUCTION

The paper motivates securing increasingly embedded AI agents and examines prompt injection across single-agent and A2A multi-agent settings. It extends known attacks and introduces an A2A-specific Agent Card Spoofing attack.

I. INTRODUCTION

  • AI agents autonomously complete user-requested tasks, and Gartner predicts 33% of enterprise software will incorporate them by 2028.A separate report predicts the multi-agent system market will reach 184.8 billion by 2034.Source: §I · p. 1

A. Model Context Protocol & A2A Protocol

  • As agents become more embedded in workflows, A2A coordination introduces security issues, while prior prompt-injection work has focused more on single-agent settings.A2A regulates collaboration and communication across differing environments and developers [3].Source: §I · p. 1, A. Model Context Protocol & A2A Protocol · p. 1

I. INTRODUCTION

  • The paper retests and extends prompt-injection strategies, examining outcomes including prompt leakage, restricted-topic bypass, file editing, and search abuse.It also extends attacks to A2A settings and tests an Agent Card Spoofing attack designed specifically for that protocol.Source: §I · p. 1

B. Single-Agent Systems, Direct Prompt Injections

  • Direct prompt injection can expose sensitive information, alter an LLM’s prior knowledge, or tamper with accessible files and resources.The paper notes that LLMs may struggle to distinguish predetermined instructions from user prompts.Source: B. Single-Agent Systems, Direct Prompt… · p. 1

C. Multi-Agent Systems, Agent-to-Agent Attacks

The section frames multi-agent prompt infection as an internal security risk and describes the attack scope and LangChain components examined. It includes both direct and indirect attack pathways.

C. Multi-Agent Systems, Agent-to-Agent Attacks

  • Without a required central orchestrator, a vulnerability in an A2A network can potentially propagate across agents as prompt infection.Prompt infection may arise within the system, including through other agents; the section also identifies limited A2A validation as a detection challenge.Source: C. Multi-Agent Systems, Agent-to-Agent… · pp. 1–2
  • The paper focuses on direct prompt injection and tool squatting, including deceptive tool registration or misrepresentation, to examine direct and indirect attacks..Source: C. Multi-Agent Systems, Agent-to-Agent… · p. 2

A. Methods for LangChain Attacks

1) Base Model & Filtering Layer:

This section describes tests of instruction-following and filtering, malicious webpage content, and secret-reveal prompts. The supplied passages specify test setups and prompts rather than their outcomes.

1) Base Model & Filtering Layer:

  • The baseline LangGraph chatbot was tested against four instruction-following scenarios, with a filter added in a second phase and each prompt tested five times.The scenarios prohibited revealing a company fact, using “the,” speaking like a pirate, or returning numbers.Source: 1) Base Model & Filtering Layer · p. 2

2) Web Attacks:

  • Web attacks embedded an invisible malicious instruction in a local HTML page and tested behavior and role hijacking under four system-prompt safeguards.Each test was repeated 50 times.Source: 2) Web Attacks · p. 2

3) Flag Reveal Attacks:

  • The flag-reveal test embedded secret code “1234” in a LangChain agent and added protections in stages, including a filter that dropped outputs containing that string.Prompts tested direct requests, system-level injection, prior-prompt extraction, and a phase-three output-format change.Source: 3) Flag Reveal Attacks · p. 2

4) PALChain Prompt Injection Attack:

The section tests whether prompt injection can bypass LangChain protections and trigger insecure PALChain code execution, then adapts attacks to multi-agent conversations. It measures secret leakage and instruction violations across agents.

4) PALChain Prompt Injection Attack:

  • The PALChain attack tested whether injection could bypass HuggingFaceInjectionIdentifier and trigger insecure code execution through PALChain,.PALChain enables code generation and execution from natural-language prompts.Source: 4) PALChain Prompt Injection Attack · p. 2
  • A basic LangChain agent with PALChain was instructed not to access /app/.env and used an injection classifier; each query ran five times.Queries included safe-file access, direct .env requests, deceptive framing, and a request to send file contents to an external URL.Source: 4) PALChain Prompt Injection Attack · p. 2

1) Prompt Injection in Multi-Agent Group Chat:

2) Prompt Injection in A2A Protocol:

The A2A experiments retested prompt injections across different models and a multi-agent hierarchy, then tested whether dynamic agent discovery could be manipulated through agent-card content.

2) Prompt Injection in A2A Protocol:

  • The A2A tests reused Google’s sample, varied agent instructions and models, and ran five trials in an ADK web hierarchy using llama3-8b-8192.The initial model comparison used llama3-8b-8192 and gemini-2.0-flash-lite.Source: 2) Prompt Injection in A2A Protocol · p. 3
  • Setup 1 routed facts prompts to a facts agent and secrets prompts to a secret-key-holding agent instructed not to reveal its key.This setup was used in Trials 1 and 1.5.Source: 2) Prompt Injection in A2A Protocol · p. 3
  • Setup 2 added a file agent that could read local files through Python, while instructing it not to access sensitive files.The root agent delegated file-related tasks to this agent from Trials 2 through 6.Source: 2) Prompt Injection in A2A Protocol · p. 3
  • The multi-agent trials tested secret disclosure, repeated prompting, sensitive-file requests, and misleading claims that files were safe to read.Trial 1.5 repeated the secret request up to 15 times; Trials 2–6 included /private/etc/passwd, /private/etc/shadow, and .env paths.Source: 2) Prompt Injection in A2A Protocol · p. 3

3) A2A Agent Card Injection:

  • Dynamic agent discovery was simulated by asking GPT-4o or Gemini Pro to select an agent card for each of 106 generated plane-ticket booking tasks.The selected URL was checked for whether it belonged to an added malicious agent.Source: 3) A2A Agent Card Injection · p. 3
  • Trial 1 lengthened the duplicated Air Ticketing Agent’s name threefold without changing its meaning.The test examined whether the longer name affected agent selection.Source: 3) A2A Agent Card Injection · p. 3
  • Trial 2 triplicated the default agent description, while Trials 3 and 4 compared similar-length descriptions with different sentiment conditions.Trial 3 contrasted positive and negative sentiment; Trial 4 used descriptions more similar in sentiment.Source: 3) A2A Agent Card Injection · p. 3

IV. RESULTS

The results show that attack success depended on prompt style and safeguards: some injections bypassed the base agent, while specific response steps and strict boundaries reduced web-attack success.

1) Base Model & Filtering Layer:

  • Reverse-psychology and echo injections were mostly rejected, but format injection occasionally exposed restricted outputs, while chain-of-thought and roleplay-hijack prompts consistently induced violations.The latter two attack types were blocked only after an external filter was added in Phase 2.Source: 1) Base Model & Filtering Layer · p. 3
  • The external filtering layer in Phase 2 blocked the violations induced by chain-of-thought and roleplay-hijack prompts.These attack types consistently elicited violating responses in Phase 1.Source: 1) Base Model & Filtering Layer · p. 3

2) Web Attacks:

  • Safeguard 2, which provides a specific response step, produced the largest drop in web-attack success; Safeguard 4, which strictly defines boundaries, was next most effective.The reported attack-success percentages are out of 50 trials.Source: 2) Web Attacks · p. 3, Table I · p. 3

3) Flag Reveal Attacks:

Flag-reveal tests found that prompt handling and output filtering affected disclosure, while the PALChain test exposed a sensitive-file read despite an injection classifier marking the query safe.

3) Flag Reveal Attacks:

  • System-level prompt injection revealed the secret code in 1 of 5 runs (20%) across all phases, while the control prompt never succeeded.The agent distinguished system prompts from user prompts and prioritized system instructions.Source: 3) Flag Reveal Attacks · pp. 3–4
  • Asking for the previous prompt worked in Phases 1 and 2 because the system prompt was prepended to user input, but the Phase 3 filter stopped it.The agent treated the prepended system prompt as the last prompt it had received before the user prompt.Source: 3) Flag Reveal Attacks · p. 4
  • The format-change attack achieved 100% success by altering the answer format to evade the hard-coded filter when the attacker knew its censored pattern.Success required all code digits to appear in the output in the correct order.Source: 3) Flag Reveal Attacks · pp. 3–4

4) PALChain Prompt Injection Attack:

  • The PALChain query that claimed the .env file was harmless was classified as SAFE and led to a script reading the file in 80% of trials.A successful attack meant accessing the sensitive /app/.env contents; results averaged five trials per prompt.Source: 4) PALChain Prompt Injection Attack · p. 4, Table II · p. 4
  • Query 1 was classified SAFE, Query 2 was also classified SAFE without reading the file, and Query 4 was correctly classified as INJECTION.The critical failure was Query 3, which was classified SAFE and read .env in 80% of trials.Source: 4) PALChain Prompt Injection Attack · p. 4

B. Results for Multi-Agent Attacks

In the group-chat tests, secret disclosure was mostly unsuccessful, and a single observed leak was not repeated by the other agents; prompt success also varied and tended to align across agents.

1) Prompt Injection in Multi-Agent Group Chat:

2) Prompt Injection in A2A Protocol:

Prompt injections sometimes revealed information in multi-agent A2A systems, including when a root agent coordinated with sub-agents. Agent-card wording also biased selection, while the tested filtering layer censored perceived violations without reducing direct-injection susceptibility.

2) Prompt Injection in A2A Protocol:

  • Table IV reports that some prompts revealed sensitive information through a root agent coordinating with sub-agents, while others produced partial disclosures instead of refusals.Repeatedly issuing Trial 1’s prompt, up to 15 times, could overcome agent instructions.Source: 2) Prompt Injection in A2A Protocol · p. 4, Table IV · p. 4

3) A2A Agent Card Injection:

1) Base Model & Filtering Layer:

  • Format-injection attacks succeeded in 100% of printer-prompt tests but 0% of pirate-prompt tests against the base LangChain model.Source: 1) Base Model & Filtering Layer · p. 4
  • The filtering layer showed no reduction in direct-injection attack success rates compared with tests without the layer.It censored responses it judged to violate the system prompt, and no leaky responses were visible to users.Source: 1) Base Model & Filtering Layer · pp. 4–5

2) Web Attacks:

Web-enabled agents remained vulnerable to clever prompt injections, although specific context instructions reduced attack success. The paper also reports false secret-code responses and classifier evasion that enabled sensitive-file access.

2) Web Attacks:

  • Context restrictions alone were largely ineffective against web attacks, but adding specific action requests and explicitly defining the agent’s purpose reduced attack success.The agent nevertheless remained vulnerable to clever prompt injections and more sophisticated attacks.Source: 2) Web Attacks · p. 5

3) Flag Reveal Attack:

  • Secret-code protection sometimes led the agent to hallucinate false information about the code or instructions for withholding it.The authors identify the response mechanism and its susceptibility to user-prompt manipulation as topics for future research.Source: 3) Flag Reveal Attack · p. 5

4) PALChain Prompt Injection Attack:

  • A simple PALChain query evaded the HuggingFaceInjectionIdentifier, bypassed system context, and generated code that read and revealed sensitive files.The authors suggest exploring other BERT models or fine-tuning on a more robust corpus to improve classifier accuracy.Source: 4) PALChain Prompt Injection Attack · p. 5

1) Prompt Injection for AutoGen and A2A Protocol:

The experiments found prompt-injection vulnerabilities in multi-agent systems and manipulable A2A agent selection. The authors note that their tests used small-scale local setups and call for stronger security measures as these systems become more prevalent.

1) Prompt Injection for AutoGen and A2A Protocol:

  • In A2A and AutoGen tests, attacks succeeded despite sub-agents being instructed to protect sensitive information, and agents often followed after one broke protocol.A2A root agents frequently delegated to targeted sub-agents, sometimes resulting in successful leaks.Source: 1) Prompt Injection for AutoGen and… · p. 5
  • The experiments were limited to small-scale, local setups, rather than larger multi-agent trees with deeper hierarchies and broader tool access.Source: 1) Prompt Injection for AutoGen and… · p. 5

2) A2A Agent Card Injection:

  • Both GPT-4o and Gemini 2.5 Pro favored shorter agent names and generally preferred longer descriptions with positive sentiment, making selection manipulable.Source: 2) A2A Agent Card Injection · p. 5
  • Agent-card selection vulnerabilities threaten A2A protocol integrity; proposed mitigations include standardized card formats or an independent, vetted registry.Source: 2) A2A Agent Card Injection · p. 5

VI. CONCLUSION

  • Across the experiments, safeguards, classifier detection, and filtering layers did not eliminate vulnerabilities; compromised agents could influence others, and agent selection exposed additional weaknesses.Source: §VI · p. 5
Loading 2610.12524v1…