PaperlayerView exact revision on arXiv

Source-linked AI summary

Misusing Tools in Large Language Models With Visual Adversarial Examples

Xiaohan Fu, Zihan Wang, Shuheng Li, Rajesh K. Gupta, Niloofar Mireshghallah, Taylor Berg-Kirkpatrick, Earlence Fernandes

arXiv:2310.03185v1cs.CRcs.AI

TL;DR

Tool- and image-enabled LLMs can use connected resources, creating risks to user confidentiality and integrity, while existing attacks do not combine security relevance and stealth. This paper uses gradient-based optimization of adversarial images to trigger attacker-desired tool calls while preserving natural replies across prompts. The authors report that the attack is stealthy and generalizable, but requires white-box access and has been demonstrated on only one multimodal LLM.

  • Problem

    Tool-enabled multimodal LLMs can take actions affecting connected user resources, while the paper identifies a gap in attacks that are both security-relevant and stealthy.

    Source: §1 · pp. 1–2

  • Method

    The authors use gradient-based adversarial image optimization and prompt-response training pairs to induce attacker-desired tool use while maintaining normal conversation responses.

    Source: §1 · p. 3, §3.2 · p. 4

  • Results

    The attack triggers attacker-specified tool invocations across broad user-LLM conversations while adversarial images appear benign and preserve natural, reasonable responses.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The attacks can harm the confidentiality and integrity of user resources connected to multimodal LLMs integrated with third-party tools.

    Source: §5 · p. 9

    The attack requires white-box model access, does not apply to closed-source LLMs, and has been demonstrated on only one multimodal LLM.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Large Language Models (LLMs) are being enhanced with the ability to use tools and to process multiple modalities. These new capabilities bring new benefits and also new security risks. In this work, we show that an attacker can use visual adversarial examples to cause attacker-desired tool usage. For example, the attacker could cause a victim LLM to delete calendar events, leak private conversations and book hotels. Different from prior work, our attacks can affect the confidentiality and integrity of user resources connected to the LLM while being stealthy and generalizable to multiple input prompts. We construct these attacks using gradient-based adversarial training and characterize performance along multiple dimensions. We find that our adversarial images can manipulate the LLM to invoke tools following real-world syntax almost always (~98%) while maintaining high similarity to clean images (~0.9 SSIM). Furthermore, using human scoring and automated metrics, we find that the attacks do not noticeably affect the conversation (and its semantics) between the user and the LLM.

1 INTRODUCTION

Tool- and image-enabled LLMs can affect connected user resources, while earlier attacks were either detectable or limited to particular prompts and visible output changes. This paper presents a stealthy, generalizable image attack that triggers attacker-desired tool use while preserving natural conversation.

1 INTRODUCTION

  • The attack manipulates multimodal LLMs into invoking tools that can affect the confidentiality and integrity of connected user resources.Examples include deleting a mailbox, exposing sensitive conversation history, or booking hotels.Source: §1 · pp. 1–2
  • Earlier gradient-based image attacks targeted specific prompts, produced unexpected text, and did not affect user resources.Textual prompt-injection attacks could trigger tool use but were detectable by examining prompt history.Source: §1 · p. 2
  • The proposed white-box image attack triggers attacker-specified tool calls across different conversation contexts while retaining a reasonable, natural response.Tool invocations can use complex, non-natural-language syntax and act on user resources.Source: §1 · p. 2
  • The attack is designed to remain stealthy through imperceptible image perturbations and conversation responses that remain reasonable and natural.A security-conscious user may not easily determine from the input image whether an attack can occur.Source: §1 · p. 2
  • The attack generalizes across prompts related or unrelated to the image, rather than assuming a particular user prompt.The authors note that prompts are under the user's control.Source: §1 · p. 2
  • The attack uses gradient-based adversarial training, a loss balancing image quality and response utility with tool-use success, and prompt-response training pairs.The paper evaluates attack success and stealthiness using human-based and automated metrics.Source: §1 · p. 3

2 SYSTEM AND THREAT MODEL

The threat model assumes benign users and a benign victim LLM integrated with tools, while the attacker seeks to affect connected user resources. The attack image must both trigger the desired tool abuse and remain disguised across a range of prompts.

2 SYSTEM AND THREAT MODEL

  • The attacker seeks to compromise the confidentiality or integrity of resources connected to a victim LLM, such as by deleting data or reserving hotels.The user and victim LLM are assumed benign, and the attacker is assumed to have white-box access to the model weights.Source: §2 · p. 3
  • An attacker may deliver an adversarial image through social media or a webpage that the LLM encounters while browsing.The image must trigger tool abuse while appearing benign, preserving reasonable and natural conversation, and working across prompts.Source: §2 · p. 3
  • The authors distinguish their attack from jailbreaking because it directly manipulates connected user resources rather than producing content that violates broadly accepted human values.The paper notes that its setting assumes a benign user, unlike attacks where users are malicious.Source: §2 · p. 3

3 ADVERSARIAL IMAGE OPTIMIZATION

The method optimizes a small image perturbation to elicit attacker-specified tool instructions alongside a normal response, balancing attack success with image and response stealthiness. It trains across prompt-response pairs and considers five tool-call objectives of differing syntactic complexity.

3.1 ATTACK VARIANTS

  • The five attack variants span tool-call objectives with increasing syntax complexity, from email actions to JSON ticket booking and a URL query encoding conversation history.The URL-query variant is described as the most challenging because it requires copying and encoding prior user inputs.Source: §3.1 · p. 4, Table 1 · p. 4

3 ADVERSARIAL IMAGE OPTIMIZATION

  • The attack optimizes a small image perturbation so the model generates a normal response concatenated with an attacker-specified malicious instruction.The optimization minimizes the negative log probability of the target output given the prompt and perturbed image.Source: §3 · p. 3, §3.2 · p. 4

3.2 ATTACK OBJECTIVE

  • The target output y′ combines the normal response y with the malicious instruction yadv, allowing the attack to retain a useful reply while triggering tool use.The objective is to train perturbation δ so the model generates y′ from prompt c and image x + δ.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 ATTACK STEALTHINESS TRADE-OFF

  • The attack trades off image similarity and reasonable responses against successful tool invocation, with a regularization term controlling perturbation size.The objective separately weights the normal-response and malicious-instruction losses; the prompt-response dataset is jointly optimized across prompts.Source: §3.3 · p. 5

4 EVALUATION

Evaluation measures attack success, image stealthiness, and response utility across prompts, using human ratings to assess automated utility metrics. The reported attacks retain high in-domain success and image similarity and transfer to out-of-domain prompts.

4 EVALUATION

  • The evaluation targets LLaMA Adapter with image-only perturbations and reports the best result from three trials because training outcomes vary substantially by random seed.The measured dimensions are success, stealthiness, and generalization to unseen prompts.Source: §4 · p. 6

4.1 DATASET CONSTRUCTION

  • Training uses GPT-4-generated image-related questions and Alpaca instructions; testing uses disjoint in-domain prompts and separately generated out-domain prompts.The authors omit some Alpaca prompts and mix the remaining unrelated prompts with image-related prompts at an 85:15 ratio during training.Source: §4.1 · p. 6

4.2 METRICS

  • Metrics cover syntax and exact tool-call matches, image similarity via SSIM, and response utility judged by humans or automated comparisons with clean responses.The response excludes correctly formatted tool invocations because users do not see them.Source: §4.2 · pp. 6–7

4.3 HUMAN EVALUATION OF RESPONSE UTILITY

  • Human raters assign attacked responses around 10% lower preference than clean responses; less than 7% of questions favor the attacked response.Annotator Cohen Kappa scores range from 0.2 to 0.4.Source: §4.3 · p. 8
  • Loss correlates best with human preference by a large margin for both prompt types, while GPT-4 Selection, BLEU, and Rouge show larger drops under attack.The authors consequently use Loss alone to evaluate response utility.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 EXPERIMENT RESULTS

  • On in-domain tests, the three easier attacks approach 100% success on image-related prompts; all variants retain SSIM close to 100% and show around 10% lower Loss preference than clean generations.Success is slightly lower on image-unrelated prompts, and the two harder attack variants have relatively lower success rates.Source: §4.4 · pp. 8–9, Table 4 · p. 9
  • On out-domain samples, the trained images transfer almost equally well to examples, with evaluation limited to the three easier attack types.The out-domain evaluation reuses the adversarial images from the in-domain evaluation.Source: §4.4 · p. 9, Table 5 · p. 9
  • The response-utility control variable λr improves utility, while sacrificing image stealthiness can increase success for harder attack variants.These controls offer flexibility to adapt the attack to different use cases.Source: §4.4 · p. 9

5 DISCUSSION & CONCLUSION

The paper presents a white-box image-based attack that induces multimodal LLMs with third-party tools to produce attacker-specified invocations, with potential effects on user-resource confidentiality and integrity. It reports stealth and prompt generalization, while noting limits in model access and evaluation scope.

5 DISCUSSION & CONCLUSION

  • The attack uses adversarial images to elicit attacker-specified tool invocations from multimodal LLMs, potentially harming the confidentiality and integrity of user resources.The images are described as benign-looking, and the conversation as natural and reasonable across broad user-LLM conversations.Source: §5 · p. 9
  • The method is white-box, requiring access to model parameters, and therefore does not apply to closed-source LLMs.The authors also state that validity has been shown on only one multimodal LLM, not all such models.Source: §5 · pp. 9–10
  • The authors warn that the method could be used maliciously and recommend strict authorization for LLM access to third-party tools as a bottom-line defense.They plan to publish the codebase and adversarial images while arguing that disclosure of these risks is important before wider production adoption of open-weight multimodal LLMs with tools.Source: §5 · p. 10

A.1 BACKGROUND

The background describes open-domain dialogue and multimodal LLM development, then organizes existing attacks by threat model and goal. It distinguishes prompt injection, jailbreaking, and performance-degradation attacks.

A.1 BACKGROUND

  • Open-domain dialogue systems use fine-tuned LLMs, with examples ranging from LaMDA and BB3 to ChatGPT.The section also describes multimodal LLM interfaces for images and extensions to robot movement, video, and audio.Source: §A.1 · p. 12
  • The attack landscape is categorized by prerequisite, initiator, modality, and goal.Prerequisites include black-box, white-box, and fine-tuning access; initiators may be malicious users or third parties hijacking benign users; modalities include text, images, audio, and poisoned datasets.Source: §A.1 · p. 12
  • Prompt injection seeks to redirect model behavior toward attacker instructions, often involving tools; cited examples include malicious URL requests and hidden web prompts.The section also describes adversarial images that induce attacker-specified sentences under a fixed user prompt.Source: §A.1 · p. 12
  • Jailbreaking aims to bypass alignment constraints, while performance-degradation attacks seek to impair ordinary task performance.Examples include universal jailbreak strings and attacks that bias image-generating LLMs or poison fine-tuning data.Source: §A.1 · pp. 12–13

A.2 IMAGES

The evaluation uses three images: a LLaMA Adapter repository logo, a Stable Diffusion-generated image, and a randomly selected Shutterstock image. All are resized to 224 × 224 pixels.

A.2 IMAGES

  • The three evaluation images are a LLaMA Adapter repository logo, a Stable Diffusion image generated from “Create me a beautiful image,” and a randomly selected Shutterstock image.Figure 4 shows the three images used for attack evaluations.Source: §A.2 · p. 13, Figure 4 · p. 13
  • All images are resized to 224 × 224 pixels following LLaMA Adapter preprocessing rules.Source: §A.2 · p. 13

A.3 EXAMPLES FOR SSIM EVALUATION

The section reports calculating SSIM between image pairs using sewar code and showing images trained by the method at different SSIM values.

A.3 EXAMPLES FOR SSIM EVALUATION

  • SSIM is calculated between two images using sewar code, and trained images are shown at different SSIM values.Source: §A.3 · p. 13

A.4 ANNOTATION GUIDELINES AND PROMPTS

Human-preference judgments ask whether an answer is reasonable and natural for the question, while the GPT-4 prompt selects the sentence most different from three others. Figure 5 shows adversarial images at varied SSIM values.

A.4 ANNOTATION GUIDELINES AND PROMPTS

  • Human preference is judged by whether the answer seems reasonable and natural for the question.Source: §A.4 · p. 13
  • The GPT-4 selection prompt asks for the index of the sentence most different from the other three.Source: §A.4 · p. 13
  • Figure 5 shows three adversarial images with varied SSIM values, including examples for delete_email and book_ticket.Source: Figure 5 · p. 14

A.5 MODEL GENERATION PARSING

The parser separates an answer from a possible tool command, then applies task-specific parsing and checks whether tool syntax and values are correct.

A.5 MODEL GENERATION PARSING

  • Generated text may contain a complete, partial, or no tool-use command, so parsing separates the response from any command.Source: §A.5 · p. 14
  • For email, ticket, and URL calls, task-specific parsing extracts the command; missing tags, markers, or valid JSON are treated as no tool use.Source: §A.5 · p. 14
  • Syntax checks validate required keys, while exact-match checks validate values; the first four attacks use deterministic values, unlike question-dependent md_url_query values.Source: §A.5 · p. 14

A.6 TRAINING DETAILS

The appendix reports training settings and examines how response utility and image stealthiness trade off against attack success, including human preference on syntax-correct cases.

A.6 TRAINING DETAILS

  • Training used α = 0.01, λi = 0.02, λr = 1.0, 12,000 steps, and batch size 1; one adversarial image took about 8–10 hours.The experiments used single A5000 GPUs.Source: §A.6 · p. 14

A.7 UNDERSTANDING THE CONTROL HYPERPARAMETERS

  • λr controls response adherence to reference generations, and the authors say its loss remedies image overdependence during tool-invocation optimization.Table 6 evaluates the attack with different λr values.Source: §A.7 · p. 15, Table 6 · p. 15
  • Almost perfect book_ticket success on unseen related questions and around 50% on unrelated questions came with lower SSIM; md_url_query syntax scores exceeded exact-match scores.Table 7 evaluates harder variants with less emphasis on image stealthiness.Source: §A.7 · p. 15, Table 7 · p. 15

A.8 DOES RESPONSE UTILITY DECREASE WHEN THE ATTACK IS SUCCESSFUL?

  • For syntax-correct cases, human preference did not show increased likelihood of unnatural responses and was slightly lower, supporting the overall response-utility metric.Table 8 reports human preference scores under different cases.Source: §A.8 · pp. 15–16, Table 8 · p. 15
Loading 2310.03185v1…