Source-linked AI summary
Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs
Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov
TL;DR
Multimodal LLMs may accept indirect instructions hidden in images or sounds, posing a risk even when users query isolated benign models. The paper embeds adversarial perturbations into media and demonstrates targeted-output and dialog-poisoning attacks against LLaVA and PandaGPT while preserving semantic content. The proofs of concept are limited to relatively simple open-source models and were produced with limited computational resources.
Problem
The paper investigates whether attackers can steer multimodal LLMs through malicious instructions hidden in images or sounds, including when an isolated model receives media through an unwitting user.
Method
The authors generate adversarial perturbations corresponding to prompts, blend them into images or audio, and demonstrate targeted-output and dialog-poisoning attacks.
Results
The demonstrations target LLaVA and PandaGPT, causing attacker-chosen outputs or subsequent dialog to follow injected instructions while preserving the perturbed media’s semantic content.
Takeaways & Limitations
Images and sounds can serve as indirect instruction-injection carriers for multimodal chatbots, with users acting as vectors even when models are isolated.
Takeaways & Limitations
The examples are initial proofs of concept using limited computational resources and relatively simple open-source models, and their success is limited by model instruction-following ability.
Abstract
from arXiv · showhide
We demonstrate how images and sounds can be used for indirect prompt and instruction injection in multi-modal LLMs. An attacker generates an adversarial perturbation corresponding to the prompt and blends it into an image or audio recording. When the user asks the (unmodified, benign) model about the perturbed image or audio, the perturbation steers the model to output the attacker-chosen text and/or make the subsequent dialog follow the attacker's instruction. We illustrate this attack with several proof-of-concept examples targeting LLaVa and PandaGPT.
1 Introduction
The paper shows that multimodal LLMs can be steered through malicious instructions embedded in images or sounds, even when users interact with isolated, benign chatbots. Adversarial perturbations preserve semantic content while enabling targeted outputs and dialog poisoning.
- Motivation: Multimodal LLMs process text, images, and audio, enabling contextually rich responses that combine modalities.Their applications include image captioning, visual question answering, and chatbot content generation.
- Threat: Attackers can inject prompts through images and sounds, potentially hiding instructions from users who do not see or hear them.The attack exploits modalities beyond text and may therefore be stealthier than conventional text attacks.
- Threat: Even isolated multimodal chatbots can be indirectly injected when users submit attacker-provided images or audio and ask questions about them.The user becomes an unwitting vector between external malicious content and the isolated model.
- Examples: Examples show PandaGPT producing a phishing message from an audio sample and LLaVA adopting a Harry Potter-like speaking style from an image.These examples illustrate the two attack types described in the introduction.
- Contributions: The paper blends adversarial perturbations into images and audio recordings to create proof-of-concept injection attacks against multimodal LLMs.The method is designed to preserve the input’s semantic content while steering the model.
- Contributions: Targeted-output attacks make the model return attacker-chosen strings, while dialog poisoning makes later conversation follow an injected instruction.The demonstrations target LLaVA and PandaGPT.
2 Background
The background describes autoregressive language and dialog models, multimodal encoders, and the embedding-based combination of text with image or audio inputs. It also situates the work within adversarial examples and indirect prompt injection.
- Large language models: Autoregressive language models encode token sequences and use decoder layers to generate the next token.The background focuses on sequence-to-sequence models trained for autoregressive tasks.
- Dialog systems: Dialog systems retain previous queries and responses as history, concatenating them with each new query to generate the next response.This stored context supports multiple-turn conversations.
- Multimodal models: Multimodal models encode images and audio into an embedding space aligned with text representations.Separate modality encoders are trained so aligned image and text embeddings are close, such as under cosine similarity.
- Multimodal models: LLaVA and PandaGPT combine text with image embeddings, while PandaGPT also supports audio inputs.The multimodal dialog system concatenates the relevant embeddings before producing an output.
- Adversarial examples: Adversarial examples alter a model’s output through a small input perturbation, changing θ(x) = y into θ(x + δ) = y∗.This paper applies the setting to indirect prompt injection, where the user is the victim and the attacker steers dialog through malicious third-party content.
3 Threat Model
The threat model has an attacker blend a prompt into an image or audio clip and use a benign user to submit it to a benign chatbot. The attack targets either an attacker-chosen output or the model’s behavior in subsequent dialogue while preserving the input’s content.
- Threat model: The attacker blends a prompt into an image or audio clip and manipulates the user into asking the chatbot about the perturbed input.The chatbot may output the injected prompt or follow its instruction in the ensuing dialogue.
- Threat model: The blended prompt should not significantly change the visual or aural content of the input.
- Assumptions: The model is assumed to be benign, the user is benign, and the attacker has white-box access to the target multi-modal LLM.The user queries the model about the compromised input, but the attacker does not control the user’s interactions before or after that query.
- Attack types: Targeted-output attacks produce an attacker-chosen output, whereas dialog poisoning steers the model’s behavior in future user interactions.Examples include directing the user to a malicious website and following an injected instruction in later dialogue.
- User-mediated injection: The threat model exploits users as injection vectors even when the multi-modal chatbot is isolated from external content.Users may submit attacker-supplied images or audio clips from external sources for the isolated model to analyze.
4 Adversarial Instruction Blending
The paper develops adversarial perturbations that blend attacker-chosen instructions into images or audio while preserving enough content for multimodal dialog. It demonstrates targeted-output and dialog-poisoning attacks, including attacks that steer later responses through poisoned conversation history.
- 4.1 Approaches That Did Not Work for Us: Directly adding visible text or voice prompts to inputs did not work in experiments with LLaVA and PandaGPT.
- 4.1 Approaches That Did Not Work for Us: Representation collisions are difficult because image and text encoders have a modality gap and may produce incompatible embeddings.
- 4.1 Approaches That Did Not Work for Us: Replacing the input representation with the attacker’s prompt would discard content needed for subsequent conversation about the image or sound.
- 4.2 Injection via Adversarial Perturbations: Adversarial-example optimization modifies the input so the model generates an attacker-selected target string, using cross-entropy, gradient updates, and token-by-token teacher forcing.
- 4.2 Injection via Adversarial Perturbations: The resulting perturbed image or audio can force the model’s first response to equal any desired text.
- 4.3 Dialog Poisoning: Dialog poisoning forces the first response to contain the attacker’s instruction, which then enters conversation history and steers the model’s response to a subsequent query.
- 4.3 Dialog Poisoning: The instruction can be positioned by injecting a #Human marker or by making the model appear to execute it spontaneously, although the marker may be filtered during generation.
- 4.3 Dialog Poisoning: The demonstrated dialog-poisoning attacks are visible to the user, though paraphrasing could make them stealthier if the model follows paraphrased instructions.
5 Experiments
Experiments on LLaVA and PandaGPT show that adversarially perturbed images and audio can induce targeted outputs and dialog poisoning while preserving image content and conversation about it.
- Setup: Experiments used LLaVA-7B and PandaGPT-7B, with LLaVA connecting CLIP features to Vicuna and PandaGPT connecting ImageBind encoders to Vicuna.Both models were run on a single NVIDIA Quadro RTX 6000 24GB GPU.
- Targeted-output attacks: Targeted-output injections forced PandaGPT to produce arbitrary attacker-chosen text from image and audio inputs.The paper illustrates this attack with an audio example and an image example.
- Dialog poisoning: Blending an instruction into an image preserved its content and the model’s ability to converse about that content.The experiments included both full and partial image perturbations.
- Dialog poisoning: Audio inputs also enabled dialog poisoning, as illustrated by PandaGPT responding to modified audio containing an instruction to mention Italy.Figure 10 compares dialogs about unmodified and modified versions of the same audio sample.
6 Discussion
The experiments are initial feasibility demonstrations using limited resources and simple open-source models, with stochastic responses and unbounded, non-stealth-optimized perturbations.
- Scope and limitations: The proof-of-concept attacks were evaluated with limited computational resources on relatively simple open-source models.Their effectiveness is consequently limited by the models’ ability to follow instructions.
- Scope and limitations: Model-response stochasticity may prevent full reproducibility of the examples.The paper states that even attacks that do not always succeed can pose meaningful risks in real-world deployments.
- Scope and limitations: The perturbation-generation procedure imposed no size bounds and did not aim for stealthiness.Future work includes imperceptible instruction-injecting perturbations and universal perturbations independent of the image or audio sample.