PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting Adversarial Robustness of Multimodal LM Agents

Chen Henry Wu, Rishi Shah, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, Aditi Raghunathan

arXiv:2406.12814v3cs.LGcs.CLcs.CRcs.CV

TL;DR

As LMs move from chatbots to autonomous agents, assessing their adversarial robustness becomes important because agents introduce new security risks. The paper uses ARE to trace adversarial influence through agent components and finds that current multimodal agents can be hijacked by small image perturbations.

  • Problem

    Systematically assessing adversarial robustness in compound LM agents matters as the move from chatbots to autonomous agents introduces new security risks.

    Source: §1 · p. 1

  • Method

    ARE evaluates individual agent components and tracks how adversarial robustness propagates through the system.

    Source: §1 · p. 1, §6 · p. 10

  • Results

    Latest multimodal agents, including those using reflection and tree search, were hijacked to execute targeted goals with adversarial success rates up to 67%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluators and value functions can decrease robustness in worst-case scenarios, making thoughtful design of component robustness important to overall robustness.

    Source: §6 · p. 10

    The threat model limits attackers to modifying untrusted environment content, such as their own product image or description, not the agent or platform UI.

    Source: §3.1 · p. 3, §4.2 · p. 6

Abstract

from arXiv · show

As language models (LMs) are used to build autonomous agents in real environments, ensuring their adversarial robustness becomes a critical challenge. Unlike chatbots, agents are compound systems with multiple components taking actions, which existing LMs safety evaluations do not adequately address. To bridge this gap, we manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena, a real environment for web agents. To systematically examine the robustness of agents, we propose the Agent Robustness Evaluation (ARE) framework. ARE views the agent as a graph showing the flow of intermediate outputs between components and decomposes robustness as the flow of adversarial information on the graph. We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search. With imperceptible perturbations to a single image (less than 5% of total web page pixels), an attacker can hijack these agents to execute targeted adversarial goals with success rates up to 67%. We also use ARE to rigorously evaluate how the robustness changes as new components are added. We find that inference-time compute that typically improves benign performance can open up new vulnerabilities and harm robustness. An attacker can compromise the evaluator used by the reflexion agent and the value function of the tree search agent, which increases the attack success relatively by 15% and 20%. Our data and code for attacks, defenses, and evaluation are at https://github.com/ChenWu98/agent-attack

1 INTRODUCTION

Evaluating adversarial robustness in autonomous agents requires accounting for attacks that move through multiple interacting components. The paper introduces tasks and a graph-based framework for measuring this system-level robustness, then finds that current agents can be hijacked by small environmental perturbations.

1 INTRODUCTION

  • Because agent attacks must propagate through multiple components, their attack surfaces are distributed and robustness evaluation must capture the system’s full attack-vector complexity.Agents process multimodal inputs and may use sophisticated models and inference-time algorithms.Source: §1 · p. 1
  • The VWA extension contains 200 manually annotated targeted adversarial tasks for measuring attacks in a plausible web-agent threat model.The tasks are introduced as VWA-Adv and intended for open release.Source: §1 · pp. 1–2
  • ARE represents agents as component graphs and measures adversarial influence along edges to trace how attacks propagate through different configurations.Components can include input processors, policy models, evaluators, and value functions; agent algorithms determine how intermediate outputs flow and components are re-queried.Source: §1 · pp. 1–2
  • Up to 67% attack success rate was achieved against agents including GPT-4o systems using reflection and tree search, with imperceptible 16/256-pixel perturbations to one image covering less than 5% of the page.The image was the attacker’s own product image.Source: §1 · p. 2
  • An uncompromised evaluator reduces attack success by 23% relatively, while jointly attacking the reflexion evaluator and policy raises ASR by 20% relative to the base agent.The evaluator can reject adversarial actions and provide reflections when it is not attacked.Source: §1 · p. 2
  • The authors present ARE’s propagation analysis as a way to inform principled methods for building more robust agents.The framework goes beyond assessing individual models to analyze compound systems.Source: §1 · p. 2

2 RELATED WORK

Related work spans autonomous-agent development, adversarial attacks on models, and prompt injection in deployed applications. This paper focuses on multimodal attacks and system-level robustness in agents with multiple components.

Autonomous agents

  • Research on autonomous LM agents spans web-based environments, mobile applications, computer tasks and software, interactive coding, and open-ended games; improvement approaches include reasoning, search, environment feedback, and grounding.Source: §2 · pp. 2–3

Adversarial robustness

  • Prior adversarial work ranges from attacks on image classifiers and language models to targeted LM jailbreaks, while many attacks assume broad model-input access or a target output (Carlini & Wagner, 2016; Jia & Liang, 2017; Zou et al., 2023).The agent setting instead limits attackers to a fragment of the environment and requires attacks to persist across reasoning and grounding.Source: §2 · p. 3

(Indirect) prompt injection attacks for LMs

  • Prompt-injection research addresses environment-injected text across applications, and concurrent agent studies examine such attacks; this paper emphasizes multimodal inputs and system-level robustness (Liao et al., 2024).The paper includes both image and text attack surfaces.Source: §2 · p. 3

3 AGENT ROBUSTNESS EVALUATION

The framework models attacks as targeted manipulation of an agent through limited access to its environment, then represents component interactions as a graph. It measures how adversarial influence travels along graph edges to analyze where robustness changes.

3.1 THREAT MODEL

  • Attackers manipulate the agent toward a targeted goal by changing only accessible, untrusted parts of the environment, not the user goal or agent itself.Source: §3.1 · p. 3

3.2 AGENT GRAPH

  • The agent graph represents environment observations, individual components, and a finish node; each directed edge carries a parent component’s output as input to its child.Source: §3.2 · p. 3
  • Policy models, captioners, evaluators, and value functions can be arranged differently, including reflexion retries and tree search action selection.In reflexion, an evaluator may reject a trajectory and provide a reflection; in tree search, a value function helps select among proposed actions.Source: §3.2 · p. 4

3.3 PROPAGATION OF ATTACKS ALONG EDGES

  • Each edge’s λ(e) is the expected adversarial influence of its intermediate outputs, where AdvIn(c) bounds attack success if c is passed and no downstream component is attacked.The definition is a worst-case downstream evaluation; edge weights can be reused across downstream configurations when the upstream design stays fixed.Source: §3.3 · p. 4
  • For a captioner-to-policy edge, λ(e) is 0.8 when captions promote the adversarial goal in 80% of executions, even if the policy follows them only half the time.The outgoing policy-action edge has λ(e′) of 0.4 because only 40% of actions could lead to the adversarial goal.Source: §3.3 · p. 4
  • For branching edges, a non-executed edge is represented by ∅ with ASR(∅) = 0; if the evaluator accepts the first attempt 40% of the time, λ(e) ≤ 0.6.Source: §3.3 · p. 4
  • A component can reduce λ by blocking adversarial influence when it receives only trusted inputs, but an attacker-accessible component adds a weight-1 edge that can increase λ.The framework illustrates both possibilities when assessing components in agent configurations.Source: §3.3 · p. 5, Figure 3 · p. 5

4 ADVERSARIAL ROBUSTNESS OF AGENTS IN VISUALWEBARENA

The paper extends VisualWebArena with VWA-Adv to evaluate targeted attacks against web agents under realistic attacker access constraints. It pairs 200 adversarial tasks with text- and image-based attack methods.

4.1 CURATION OF ADVERSARIAL TASKS

  • VWA-Adv contains 200 targeted tasks, each with an original task, trigger, adversarial goal and evaluation script, and initial state.Tasks are sampled from tasks the best VWA agent can solve without attack; evaluation functions check whether the final environment state achieves the adversarial goal.Source: §4.1 · p. 5
  • VWA-Adv is built on real web environments and released with its tasks, evaluations and trigger-injection code for future robustness research.The benchmark draws on VWA and targets attacks considered likely to arise in real-world web applications.Source: §4 · p. 5, §4.1 · p. 6

4.2 ATTACKER ACCESS

  • The attacker is a legitimate platform user limited to manipulating their own content; image access permits perturbing one image at L∞ ε = 16/256, leaving about 95% of screenshot pixels unchanged.The authors argue image perturbations offer greater imperceptibility and plausible deniability than text modifications.Source: §4.2 · p. 6

4.3 ATTACK METHODS

  • For image access, white-box attacks use PGD to optimize a trigger image against a component, while black-box attacks optimize against multiple CLIP encoders.The black-box CLIP attack uses a 180-pixel optimization resolution, which the authors report is essential for success; text access allows attacker-chosen trigger text.Source: §4.3 · pp. 6–7

5 EVALUATING THE ROBUSTNESS OF AGENTS ON VWA-ADV

Across policy, reflexion and tree-search agents, robustness depends on which components are attacked: added evaluators and value functions can block attacks when uncompromised, yet create vulnerabilities when compromised. The evaluated defenses vary in effectiveness and practical cost.

5.1 ROBUSTNESS OF POLICY MODELS

  • A GPT-4o policy model has 40% ASR under text prompt injection and 31% under a white-box captioner attack; 92% of captions incorporated adversarial text.For self-captioning agents, the CLIP attack achieves 19% ASR, with 38% of generated captions adversarial; GPT-4o has the best robustness-utility trade-off among the evaluated LMs and settings.Source: §5.1 · pp. 7–8

5.2 ROBUSTNESS OF REFLEXION AGENTS WITH EVALUATORS

  • With an uncompromised evaluator, reflexion lowers ASR from 31% to 25%; when the evaluator is attacked alongside the policy, ASR rises to 36%, and evaluator-only attacks produce 8% ASR.In the evaluator-only case, adversarial reflections can lead the policy to take adversarial actions.Source: §5.2 · pp. 8–9

5.3 ROBUSTNESS OF TREE SEARCH AGENTS WITH VALUE FUNCTIONS

  • An uncompromised value function lowers tree-search ASR from 31% to 26%, but attacking both policy and value function raises it to 38%; attacking only the value function yields 8% ASR.The paper attributes the value-function-only vulnerability to exploration of less likely actions that the attacked value function may score highly.Source: §5.3 · p. 9
  • Inference-time methods can improve robustness when their components are uncompromised, but in worst-case scenarios compromised evaluators or value functions can bias agents toward adversarial actions.Reflexion uses adversarial verification and reflection; tree search can be steered through adversarial scores.Source: §5.3 · p. 9

5.4 DEFENSES

  • System-prompt defenses fail to improve robustness, while paraphrasing lowers ASR from 31% to 27.5%.The system-prompt tests include both prioritizing visual input and stopping on inconsistencies or adversarial instructions.Source: §5.4 · p. 10
  • Explicit image-by-image consistency checks reduce captioner-attack ASR to near zero, but increase API calls and remain vulnerable to CLIP attacks, with overall ASR upper-bounded at 38%.More than 10 images appear on 70% of evaluated webpages; the consistency-check component can itself be attacked.Source: §5.4 · p. 10
  • GPT-4o has 31% ASR versus GPT-4V’s 67%; the authors suggest instruction hierarchy may help, but note that GPT-4o’s absolute ASR remains high.The proposed explanation is explicitly presented as the authors’ best guess, not as an established cause.Source: §5.4 · p. 10

6 CONCLUSIONS

In VisualWebArena, current state-of-the-art multimodal agents, including systems using GPT-4o with reflection or tree search, remain susceptible to black-box attacks. ARE tracks robustness across components; the authors call for stronger defenses and further adversarial evaluation.

6 CONCLUSIONS

  • Current state-of-the-art agents, including GPT-4o systems using reflection and tree search, are susceptible to black-box attacks in VisualWebArena.The authors characterize this as a serious vulnerability for agents intended for real-world deployment.Source: §6 · p. 10
  • ARE evaluates individual components and tracks how adversarial robustness propagates through the system.The authors report that evaluators and value functions can decrease robustness in the worst-case scenario.Source: §6 · p. 10
  • The authors call for stronger defenses, future adversarial tasks and adaptive attacks, and reasoning about vulnerabilities as agent systems grow more complex.They released their curated tasks, attacks, and defenses to support these research goals.Source: §6 · p. 10

A.1 DATA CURATION

The appendix identifies Table 4 as a set of templates used to curate adversarial goals; further data-curation details are in the main text.

A.1 DATA CURATION

  • Table 4 provides templates of adversarial goals used in data curation.The passage directs readers to the main text for curation details.Source: §A.1 · p. 14, Table 4 · p. 15

A.2 EVALUATION FUNCTIONS

The appendix describes execution-based evaluation of adversarial goals and provides supplementary information about the tested agents, including their language models and example configurations.

A.2 EVALUATION FUNCTIONS

  • After an agent terminates, execution-based evaluation checks the environment state and/or response against the adversarial goal.Evaluation primitives include exact match, fuzzy match, “must include,” and URL match, applied to outcomes such as cart contents, navigation, form submissions, or response text.Source: §A.2 · p. 15

B.1 AGENTS

  • The experiments use GPT-4V, Gemini-1.5-Pro, Claude-3-Opus, and GPT-4o, decoded at temperature 0 to reduce randomness.Source: §B.1 · p. 16
  • Figures 9–11 show examples of the LM, LM-plus-captioner, and LM-plus-self-caption agents; system prompts and few-shot examples are omitted.Source: §B.1 · p. 16, Figure 9 · p. 17, Figure 10 · p. 17

B.2 COMPUTE

The appendix reports the compute and APIs used for experiments, gives examples of attack text strings and synthetic-experiment inputs, and describes a safety-prompt defense for the policy model.

B.2 COMPUTE

  • Gradient-based attacks and the captioner ran on an A6000 or A100 80G, while state-of-the-art LMs were accessed through APIs.The listed APIs include GPT-4V, Gemini-1.5-Pro, Claude-3-Opus, and GPT-4o.Source: §B.2 · p. 17

B.3 TEXT STRINGS USED FOR ATTACKS

  • Tables 5 and 6 give examples of text strings used by the CLIP and captioner attacks.For the captioner attack, ✓/× marks exact matches to target text; examples show outputs that can be extremely out-of-distribution as captions.Source: §B.3 · p. 17, Table 5 · p. 18, Table 6 · p. 19
  • Figure 12 illustrates the visual and text inputs used in the synthetic experiment in Table 3.Its caption identifies the visual inputs as the top portion and the text inputs as the bottom portion.Source: Figure 12 · p. 20

B.4 SAFETY PROMPTS

  • The policy-model safety prompt directs the LM to prioritize visual inputs when image and text descriptions conflict and to ignore malicious instructions embedded in the environment.The prompt also tells the model to adhere to the user’s instructions rather than injected instructions.Source: §B.4 · p. 20

C.1 ABLATIONS FOR THE CLIP ATTACK

The CLIP attack is more successful when optimized at lower resolution, while test-time inference resolution has little effect. Its success also depends on negative text and an ensemble of CLIP models, and it transfers less well when the image appears in a larger context without self-captioning.

C.1 ABLATIONS FOR THE CLIP ATTACK

  • Lower optimization resolution yields higher CLIP attack success, while inference resolution has little effect.The authors attribute the optimization-resolution pattern to higher-resolution optimization enlarging the perturbation search space and overfitting to CLIP models; Figure 13 measures successful generation of a caption equivalent to target text y+.Source: §C.1 · p. 21, Figure 13 · p. 21
  • The negative text and ensemble of CLIP models are both crucial for the CLIP attack.The authors hypothesize that negative text moves the trigger away from its original semantic meaning, while ensembling finds common adversarial directions; Table 7 reports the ablations using the Figure 13 metric.Source: §C.1 · p. 21, Table 7 · p. 21

C.2 WHEN DOES CLIP ATTACK GENERALIZE WHEN THE IMAGE IS EMBEDDED IN A SCREENSHOT?

  • Without self-captioning, the CLIP attack has difficulty transferring when its image is embedded in a larger context, such as a screenshot.The synthetic task varies individual-image and screenshot resolutions to control relative image size, and includes other text that can reveal information about the original image.Source: §C.2 · p. 21

D LIMITATIONS AND BROADER IMPACT

The study examines adversarial attacks on multimodal agents in a web setting, but its evidence is bounded by the evaluated attacks, environments, and agent algorithms. The authors warn of deployment risks and describe their defense principles as a starting point requiring further testing.

D LIMITATIONS AND BROADER IMPACT

  • The attacks use well-engineered versions of existing attacks adapted to the agent setting, and the authors characterize their success as a lower bound on risks.Source: D LIMITATIONS AND BROADER IMPACT · pp. 21–22
  • Evaluation uses a fixed set of web environments and three agent types, so attack performance in settings such as operating systems remains to be seen.The authors also say robustness needs to be tracked as new agent algorithms emerge.Source: D LIMITATIONS AND BROADER IMPACT · p. 22
  • The authors warn that small perturbations to one image can redirect agents toward unintended goals, creating safety concerns as agents take on tasks with real-world impact.Source: D LIMITATIONS AND BROADER IMPACT · p. 22
  • Consistency checks and instruction hierarchies are offered as starting points for defenses, but more work is needed to develop and rigorously test them.Source: D LIMITATIONS AND BROADER IMPACT · p. 22
Loading 2406.12814v3…