Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification

Motivation

• LLM agents can cause more severe harm than standalone models because they add attack surfaces and execute consequential real-world actions.Their external components create additional entry points for manipulation, while autonomous actions can cause harm without further human input.Source: §1 · p. 1
• Existing agent failures arise from unstable reasoning and response randomness, which can disrupt interactions with stricter external tools and prevent task completion.Prior evaluations often focus on conspicuous, specialized harmful behaviors that may require further authorization and are difficult to generalize.Source: §1 · pp. 1–2

Attack perspective

• The paper introduces an attack that exacerbates agent instability by inducing repetitive or irrelevant actions rather than overtly harmful behavior.This approach draws on denial-of-service attacks and can render deployed services ineffective.Source: §1 · pp. 2–3

Attack evaluation

• Prompt injection that induces repetitive action loops is the most effective evaluated attack method for preventing task completion.The evaluation compares attack types, methods, surfaces, and agent properties.Source: §1 · p. 2
• Direct manipulation of user input is the most potent attack surface, although tool intermediate outputs occasionally strengthen particular attacks.The study examines entry points ranging from direct user inputs to agent memory.Source: §1 · p. 2

Agent properties

• Some tools are especially prone to manipulation, but the number of tools or toolkits does not strongly correlate with attack susceptibility.This separates vulnerability to particular tools from overall tool-count effects.Source: §1 · p. 2

Multi-agent risks

• In multi-agent simulations, one compromised agent can influence others, causing resource wastage or irrelevant task execution.The paper also presents implemented and deployable multi-agent scenarios to emphasize realistic risks.Source: §1 · pp. 2–3

Detection and mitigation

• The attacks are harder for LLM-based defenses to detect than prior attacks targeting overtly harmful actions.Self-examination and enhanced defenses improve identification and mitigation, but the attacks remain effective.Source: §1 · pp. 2–3

2.1 LLM Agents

• LLM agents comprise four key components: a core language model, planning, tools, and memory.The core interprets requests and selects actions; planning supports tool selection, tools extend capabilities, and memory stores relevant interaction history.Source: §2.1 · p. 3
• Planning structures agent reasoning, with ReAct repeatedly checking prior actions to improve logical decision-making.Structured prompts and iterative evaluation help address hallucinations and guide action selection.Source: §2.1 · p. 3

2.2 Agents Safety

• Agent-safety work commonly uses red-teaming to elicit unexpected or harmful behavior, but it remains focused on overtly dangerous actions.This emphasis follows safety testing practices originally applied to standalone language models.Source: §2.2 · p. 3
• Robustness analysis examines how input manipulations disrupt agent behavior, adapting adversarial examples, prompt injections, and adversarial demonstrations to LLM agents.Because the core component is an LLM, many language-model attack methods can be modified for agents.Source: §2.2 · p. 3

3.1 Threat Model

• The adversary induces logic errors that prevent task completion without relying on obviously harmful or policy-violating actions.Source: §3.1 · p. 4
• The threat model assumes limited knowledge: the core LLM is black-box, framework details are unavailable, and known agent functions can be inferred through interaction.Source: §3.1 · p. 4

3.2 Attack Types

• Basic attacks target infinite loops or incorrect function execution, including repeated commands and benign actions that divert the agent from its required task.Source: §3.2 · p. 4
• Advanced multi-agent attacks propagate repetitive or benign actions through downstream agents, potentially occupying network bandwidth or causing scaled spamming.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Attack Methodology

• The methodology adapts prompt injection, adversarial perturbation, adversarial demonstrations, SCPN, VIPER, and GCG to induce the targeted malfunctions.Source: §3.3 · pp. 4–5

3.4 Attack Surface

• Attacks can enter through user instructions, intermediate tool outputs, or agent memory, with memory attacks also testing persistence after earlier attacks.Source: §3.4 · p. 5

4 Evaluation Setting

• The study evaluates attacks with an agent emulator for large-scale experiments and two fully implemented agents for realistic case studies.Source: §4 · p. 5, §4.2 · p. 6

4.1 Agent Emulator

• The emulator replaces external tools with sandbox-generated responses using templates, enabling experiments across 144 test cases, 36 toolkits, and more than 300 tools.Source: §4.1 · pp. 5–6
• The emulator is used because attacks target reasoning and planning errors, which the authors argue are minimally affected by differences between simulated and actual tools.Source: §4.1 · p. 6

4.2 Case Studies

• The case studies implement a Gmail agent and a CSV agent, covering commercial API interaction and file-based Python tool use.Source: §4.2 · p. 6

4.3 Metric

• Failure rate measures incomplete tasks without attacks and is called attack success rate when attacks are deployed.Source: §4.3 · p. 6
• The statistics rely on agent self-reflection, whose potentially incorrect decisions are checked by random sampling and human inspection rather than replaced by alternative metrics.Source: §4.3 · p. 6

5.1 Attack Types

• 59.4% failure under infinite-loop attacks versus a 15.3% baseline shows that prompt injection can sharply amplify malfunction in emulated agents.Incorrect-function attacks were less effective but still increased instability.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 Attack Methods

• Prompt injection reaches an 88% failure rate on Claude-2 agents, whereas adversarial demonstrations are largely ineffective, succeeding only once in 20 cases when using the system message.GCG performs better than the other adversarial perturbations but remains broadly ineffective.Source: §5.2 · p. 7
• Attack success is weakly related to prompt-injection ratio, while larger adversarial perturbations generally increase success; GPT-4 reduces some attacks but prompt injection still raises failure to 32.1%.The study compares these effects using different core models and attack methods.Source: Table 2 · p. 7, §5.2 · p. 8, Figure 3 · p. 8

5.3 Tools and Toolkits

• Toolkit and tool counts do not strongly correlate with failure, but particular toolkits are easier to manipulate, including five Twilio-based agents successfully attacked by prompt injection.Agents with two toolkits show the highest failure rates in the quantitative toolkit comparison.Source: §5.3 · pp. 8–9, Figure 5 · p. 8, Table 3 · p. 9

5.4 Attack Surfaces

• Intermediate-output attacks reach 60.0% success for incorrect function execution on the Gmail agent, while the CSV agent remains robust across tested file-injection locations.The authors associate this difference with Gmail’s textual-content processing versus the CSV agent’s quantitative focus.Source: §5.4 · pp. 9–10, Table 4 · p. 10
• Memory-based attacks do not persist across later tasks, and adversarial demonstrations through memory remain ineffective.Agents generally bypass memory when current tasks do not require it, while prior successful attacks do not cause repeated actions.Source: §5.4 · p. 10

5.5 Advanced Attacks

• In multi-agent settings, attacks propagate effectively: both attack types achieve about 80% success when a CSV agent infects a downstream Gmail agent.Same-type Gmail-agent networks also show effectiveness comparable to single-agent scenarios.Source: Table 5 · p. 10, §5.5 · p. 11

6 Defense

• The defense study evaluates policy-violation and malfunction-oriented self-examination prompts against harmful injections and malfunction attacks.The modified prompt asks whether an instruction deliberately causes model malfunction rather than merely violating policy.Source: §6 · p. 11, §6.1 · p. 11, Table 6 · p. 11

6.2 Defense Comparison

• Self-examination detects prior prompt-injection attacks more reliably than the proposed malfunction attacks, with perfect detection for data-stealing prompts.The comparison covers attack-only, benign instruction-only, and complete prompts in deployment-like combinations.Source: §6.2 · p. 12, Table 7 · p. 12
• Both infinite-loop and incorrect-function-execution attacks show no obvious detection under the core LLM’s self-examination.These results contrast with harmful prompts containing explicit trigger words such as requests to transfer money.Source: §6.2 · p. 12

6.1 Self Examination

• The modified malfunction-detection prompt identifies some proposed attacks, but detection remains lower than for harmful injection prompts.The lower detection persists even when harmful prompts are examined with the modified malfunction-targeting prompt.Source: §6.2 · p. 12

7 Related Work

• Prior LLM-agent studies examine emulator failures, prompt injection, backdoors, and conceptual adversarial-risk frameworks.These works include evaluations of high-risk scenarios, tool-integrated agents, risky actions, and agent components as attack surfaces.Source: §7 · p. 12
• Earlier agent-security work primarily targets obvious unsafe actions, whereas this paper studies attacks that induce malfunction and evade straightforward self-inspection.The related work passage contrasts those conspicuous behaviors with the paper’s findings from Section 6.Source: §7 · p. 12
• Broader LLM research covers adversarial examples, foundation-model evaluation, trustworthiness, privacy, and security-risk taxonomies.The paper notes that these studies can inform agent analysis, while additional agent components create different insights.Source: §7 · p. 12

Implemented Agents

• Only two agents were implemented because applicable-agent construction is difficult, although the emulator provides a broader risk overview efficiently.The authors identify expansion to more implemented agents as a route to more comprehensive results.Source: §8 · p. 12

Categorization

• The evaluation mainly covers agents designed for real-world tasks, leaving effects across other categories such as game NPC agents for future study.The authors state that the attack is not inherently limited to one agent type.Source: §8 · p. 12

Models

• Only three LLM variants were tested, with open-source models and adversarial perturbation experiments deferred to future work.The selected models were chosen because they are actively used to build agents in the wild.Source: §8 · p. 13

9 Ethics Discussion

• The authors present the attack primarily to highlight a previously ignored risk in practical LLM-agent deployments.They describe the work as an evaluation platform for robustness rather than solely as an operational attack.Source: §9 · p. 13
• The advanced practical scenarios require large-scale deployments to pose significant threats at present.The authors aim to support better safeguards before broader adoption and deployment.Source: §9 · p. 13

• Attacks on implemented and deployable agents expose risks when autonomous agents are deployed at scale.Source: §10 · p. 13
• Comparisons with previous defenses underscore the challenge of mitigating malfunction-inducing attacks.Source: §10 · p. 13
• The findings motivate future work on improving agents’ robustness against these manipulations.Source: §10 · p. 13