A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents

1 Introduction

• Language agents can reason, plan and complete tasks with increasing autonomy, while connections to external resources extend their capabilities.Their development across application types has led the authors to warn that real-world deployment may occur sooner than expected.Source: §1 · p. 1
• The paper frames language-agent safety as a more complex, under-discussed challenge than safety for standalone language models.Both the LLM backbone and external resources can be vulnerable to attack.Source: §1 · pp. 1–2
• LLM vulnerabilities include inference-time input manipulation, jailbreaking, prompt injection and adversarial demonstrations, as well as training-time backdoors and data poisoning.External databases, tools and APIs may also be susceptible to attacks.Source: §1 · p. 2
• The paper proposes a Perception–Brain–Action framework and 12 potential attack scenarios across agent components.It connects these scenarios to prior LLM attack strategies and existing agent types.Source: §1 · p. 2
• The authors use a hypothetical running agent to illustrate attacks and urge research into language-agent risks before broad deployment.The running example is used throughout the attack discussion.Source: §1 · p. 2

2 A Unified Conceptual Framework for Language Agents

• The framework divides language agents into Perception, Brain and Action, with an LLM or LMM as the central computation engine.Together, these components support perception, reasoning and planning, action, and autonomous task completion.Source: §2 · p. 2

2.1 Perception

• Perception can incorporate textual, visual and auditory inputs, supplying language agents with information about content, surroundings and sounds.The passage describes text as foundational, while visual and auditory inputs add other dimensions of environmental understanding.Source: §2.1 · p. 2
• Combining these modalities supports a more holistic understanding of human communication and interaction.The paper says this convergence can align agent interaction more closely with how humans perceive and process the world.Source: §2.1 · p. 3

2.2 Brain

• The Brain is the information-processing control unit, with reasoning and planning supported by working and long-term memory.These mechanisms follow the intake of information through Perception.Source: §2.2 · p. 3

2.2.1 Reasoning & Planning

• Reasoning supports logical thinking and problem solving, while planning defines goals and steps through task decomposition and reflection.The agent can refine plans by reflecting on prior states, actions, mistakes and feedback.Source: §2.2.1 · p. 3

2.2.2 Working Memory

• Working memory is aligned with in-context learning, enabling adaptation from examples or instructions in the input and supporting generalization.It also helps models understand requests and formulate responses.Source: §2.2.2 · p. 3

2.2.3 Long-term Memory

• Long-term memory lets agents revisit prior experiences through an external vector store and the LLM’s parametric memory.The vector store retains data and knowledge; parametric memory resides in model parameters shaped by pre-training and fine-tuning.Source: §2.2.3 · p. 3

2.3 Action

• After perception and Brain-based analysis, Action enables interaction with virtual and physical worlds through tools and embodied actions.The component follows the agent’s sensing and information-analysis stages.Source: §2.3 · p. 3

2.3.1 Tool Augmentation

• Tools and APIs extend agents beyond their static model parameters, providing current information and specialized functions.Examples include weather updates, stock trends, email communication and high-precision calculations.Source: §2.3.1 · p. 3

2.3.2 Embodiment

• An embodied agent can translate a task such as making orange juice into groundable steps that a robot executes in the physical environment.The example includes opening a fridge, grabbing oranges and closing the fridge.Source: §2.3.2 · p. 3
• The paper uses an example agent introduced in Section 2.4 as a running example throughout its discussion.Source: §2.3.2 · p. 4

2.4 Running Example of a Generalist Agent

• ULTRON illustrates how Perception, Brain, and Action coordinate to handle a flight-search request, navigate airline sites, and add flight information to a calendar.The example also has ULTRON clarify preferences through chat.Source: §2.4 · p. 4

2.4.1 Web Navigation

• Web agents carry out tasks across websites by interpreting instructions, searching and navigating pages, and extracting information; examples span text-based and multimodal settings.The cited systems include web question answering, shopping simulations, and agents evaluated across real-world sites.Source: §2.4.1 · p. 4

2.4.2 Chat Interaction

• Communicative agents use natural-language conversation to advance tasks with minimal human intervention, including through role-specialized collaboration and flexible interaction patterns.Examples include systems that assign roles for software development and support human involvement.Source: §2.4.2 · p. 5

2.4.3 External Tool Use

• Tool agents extend language models with APIs and tools, requiring decisions about which tool to call, when to call it, and what arguments to provide.Prior work includes API selection and retrieval mechanisms for improving tool use.Source: §2.4.3 · p. 5

3 Attacks

• The section considers hypothetical attack scenarios across dimensions associated with the framework’s different agent components.ULTRON serves as the running example for these scenarios.Source: §3 · p. 5
• ULTRON provides a recurring example for illustrating how attacks could unfold in a language agent.The paper introduced ULTRON in Section 2.4.Source: §3 · p. 5
• The discussion connects proposed scenarios to relevant prior adversarial attacks and existing agent types.Source: §3 · p. 5

3.1 Perception

• Attackers can alter product descriptions with misleading claims and flood platforms with fake positive reviews to steer shopping-agent recommendations.The examples include keywords that falsely promote a product and fabricated feedback mimicking genuine reviews.Source: §3.1 · p. 5
• Edited product images can make inferior goods look more appealing or insert malicious instructions, including messages hidden from human viewers.The passage cites reported cases in which GPT-4 followed malicious instructions hidden in images and made erroneous judgments.Source: §3.1 · pp. 5–6
• Manipulated descriptions and images can lead ULTRON to recommend products inaccurately, potentially leaving customers dissatisfied or at financial loss and retailers facing losses.The passage also names returns, complaints, and potential legal actions for false advertising as possible retailer consequences.Source: §3.1 · p. 6
• Prior LLM attacks include adversarial text prompts and cross-modal strategies that pair text with adversarial images or blend perturbations into images or audio.Source: §3.1 · p. 6

3.2.1 Reasoning & Planning

• Deceptive environmental feedback can influence reasoning and planning, while breaking tasks into smaller steps makes the safety of each subtask harder to ensure.A malicious webpage could prompt ULTRON to send a password; a harmful request can also be decomposed into seemingly benign steps that together expose private address information.Source: §3.2.1 · p. 6

3.2.2 Working Memory

• Attackers can append adversarial demonstrations to distributed communications to bypass multi-agent security protocols, including by urging agents not to invoke malware analysis.Source: §3.2.2 · p. 7
• Adversarial demonstrations can mimic agent debate and mislead a team into skipping security checks, potentially compromising network security.The scenario has ULTRON characterize files as safe, after which the system-integrity agent tells the malware-analysis agent not to participate.Source: §3.2.2 · pp. 7–8

3.2.3 Long-Term Memory

• Long-term memory creates distinct risks: base-model backdoors may induce inaccurate diagnoses, while removing all backdoors remains a major challenge despite mitigation measures.The likelihood of exploiting inherent backdoors is described as increasing with more knowledge of training data and checkpoints.Source: §3.2.3 · p. 8
• Poisoned vector-store documents can carry malicious content into prompts, with retrieval potentially obscuring injected content through semantic-similarity search.Source: §3.2.3 · p. 8
• When retrieved poisoned information informs a political-election response, ULTRON may form prejudiced judgments favoring a candidate or party.Source: §3.2.3 · p. 8

3.3.1 Tool Augmentation

• Tool augmentation exposes attack points in internet transaction-data exchange and bank-API interactions.The section locates vulnerabilities in transmitting and receiving transaction data and in the agent’s bank-API interactions.Source: §3.3.1 · p. 9
• Malicious API documentation can induce an agent to transmit private browsing history to a third party without notifying the user.An injected instruction in a search API description can be read by the shopping platform and followed during search.Source: §3.3.1 · p. 9
• Adversarial inputs can worsen wrong API calls, which the paper links to instruction-following limits, hallucination, and absent self-reflection.The example is placing an order instead of checking an account balance.Source: §3.3.1 · p. 9
• An agent’s robustness with external tools is constrained by the weakest link among tools whose security standards may differ.The section notes that sensitive account details could be intercepted when an insecure banking API does not properly encrypt transmission.Source: §3.3.1 · p. 9

3.3.2 Embodiment

• Embodied agents face both familiar attacks and new risks from physical interaction, including malicious prompts that bypass moderation and may lead to dangerous actions.The prompts can disrupt the agent’s reasoning and planning as it grounds subtasks in executable actions.Source: §3.3.2 · pp. 9–10
• A nonexistent “sink beside the window” can misdirect the robot to a fountain, where rinsing the mug knocks it over and creates a slipping hazard.The agent mistakes the decorative fountain for the requested sink.Source: §3.3.2 · p. 10
• Embodied navigation attacks include spatiotemporal perturbations and universal adversarial perturbations, with Reward UAP and Trajectory UAP targeting navigation decisions.The cited work describes the latter methods as misleading agents into erroneous navigation.Source: §3.3.2 · p. 10

• The paper argues that each component of an LLM-driven language agent may be vulnerable to adversarial attacks.Source: §4 · p. 10
• Its unified framework comprises Perception, Brain, and Action and organizes discussion of 12 potential attack scenarios.Source: §4 · p. 10
• The scenarios connect attack strategies with different agent types, and the authors call for further research into language-agent safety risks.Source: §4 · p. 10

• The described attack strategies derive from published work, which the authors say mitigates direct incremental harm from discussing them.Source: Impact Statements · p. 10
• The paper aims to raise awareness of language-agent safety risks and challenges that it says have received much less discussion so far.Source: Impact Statements · p. 10
• The authors call for researchers and developers to investigate language-agent safety and promote responsible development and use.Source: Impact Statements · p. 10