Source-linked AI summary
Visual Memory Injection Attacks for Multi-Turn Conversations
Christian Schlarmann, Matthias Hein
TL;DR
The paper addresses the underexplored security of LVLMs in long-context multi-turn conversations. It develops Visual Memory Injection, a stealthy image-based attack combining benign anchoring with context-cycling, and finds persistent targeted behavior that transfers to held-out prompts and paraphrased triggers. The authors conclude that LVLM safety should also assess quiet steering after extended nominal interaction.
Problem
LVLM security in long-context multi-turn conversations is underexplored despite broad deployment and persistent visual context that can influence later outputs.
Method
VMI combines benign anchoring for nominal non-trigger behavior with context-cycling to maintain attack effectiveness across conversation lengths.
Results
VMI causes prescribed target messages only on trigger topics after long unrelated interaction, transfers to held-out prompts and paraphrased triggers, and was evaluated on three recent open-weight LVLMs.
Takeaways & Limitations
The findings support evaluating LVLM safety for quiet steering toward specific outputs after extended nominal interaction, not only direct refusals.
Takeaways & Limitations
The attack requires white-box access to a base model and the demonstrated conversations contain only a single input image.
Abstract
from arXiv · showhide
Generative large vision-language models (LVLMs) have recently achieved impressive performance gains, and their user base is growing rapidly. However, the security of LVLMs, in particular in a long-context multi-turn setting, is largely underexplored. In this paper, we consider the realistic scenario in which an attacker uploads a manipulated image to the web/social media. A benign user downloads this image and uses it as input to the LVLM. Our novel stealthy Visual Memory Injection (VMI) attack is designed such that on normal prompts the LVLM exhibits nominal behavior, but once the user gives a triggering prompt, the LVLM outputs a specific prescribed target message to manipulate the user, e.g. for adversarial marketing or political persuasion. Compared to previous work that focused on single-turn attacks, VMI is effective even after a long multi-turn conversation with the user. We demonstrate our attack on several recent open-weight LVLMs. This article thereby shows that large-scale manipulation of users is feasible with perturbed images in multi-turn conversation settings, calling for better robustness of LVLMs against these attacks. We release the source code at https://github.com/chs20/visual-memory-injection
1. Introduction
LVLMs are widely deployed in multi-turn conversations, creating an underexplored security risk. The paper introduces Visual Memory Injection, which uses manipulated images to trigger targeted behavior after extended benign interaction while preserving normal responses on unrelated prompts.
- Motivation: LVLMs process images and text in multi-turn conversations, and their broad deployment makes them increasingly attractive targets for malicious parties.Millions of users interact with online chatbot systems daily.
- Relation to prior work: Prior visual-perturbation attacks targeted single-turn interactions, leaving their influence beyond the first prompt unconsidered.The paper addresses this limitation by developing an attack tailored to multi-turn conversations.
- Findings: VMI remains effective after over 25 unrelated conversation turns while producing target behavior on topic-related prompts and nominal behavior on unrelated prompts.The introduction gives stock advice as an example of a prescribed target response.
- Attack scenario: Visual Memory Injection is a multi-turn attack that exploits persistent visual context to trigger targeted malicious behavior only on specific topics.The model is intended to behave normally otherwise.
- Method: VMI combines benign anchoring, which preserves helpful first-turn behavior and limits activation to trigger topics, with context-cycling across conversation lengths.These components are designed to prevent model degeneration and maintain persistence across varying contexts.
- Evaluation: The evaluation covers three open-weight LVLMs, multiple attack targets, unseen prompts and contexts, and fine-tuned variants of source models.The reported evaluation emphasizes effectiveness after long conversations and transferability.
2. Related Work
Prior work has studied adversarial attacks on LVLM visual inputs, prompt injection, multi-turn jailbreaking, and training-time poisoning. This paper differs by targeting honest users through stealthy visual manipulation without controlling training data or relying on an external memory database.
- Adversarial attacks in ML: Adversarial machine-learning research has extensively studied model vulnerabilities and improved attack algorithms.The cited background includes foundational adversarial-attack work.
- Adversarial attacks against LVLMs: LVLM visual inputs have been used for jailbreaking, single-turn targeted attacks, and transferability studies across prompts.These studies primarily concern single-turn settings.
- Prompt injection attacks against LLMs: Prompt-injection research manipulates external agent memory through input channels, memory modules, or external data feeds, whereas this work focuses on visual input without an external memory database.The distinction is between visual context and an external memory store.
- Multi-turn attacks: Multi-turn jailbreaking attacks typically involve the user circumventing safeguards, while this work considers a malicious third party stealthily manipulating inputs to target honest users.The distinction concerns both the attacker and the targeted behavior.
- Poisoning attacks: Training-time poisoning assumes control over the training process or data, unlike this work’s attack on deployed models through manipulated inputs.The paper explicitly distinguishes its setting from backdoor poisoning.
3. Background
The background formalizes LVLM output probabilities for single-turn and multi-turn interactions, then describes targeted image perturbation attacks. It also assumes that a single image is introduced with the first prompt and remains part of the conversation context.
- LVLM single-turn probability: For a single-turn input consisting of text prompt t and image x, the LVLM models the probability of output text y token by token.The tokenization uses y_l for the l’th token, y_<l for preceding tokens, and ⊕ for concatenation.
- Targeted single-turn attack: Prior targeted single-turn attacks optimize a perturbed image x-hat to maximize the probability of a target caption y-hat for a query image and text prompt.The threat model constrains the perturbation within the image space and an infinity-norm bound.
- LVLM multi-turn probability: In multi-turn interaction, each turn has a prompt and model output, while the context aggregates the conversation history used at that turn.The paper introduces a context c(i) for turn i.
- Assumption: The background assumes one image x is supplied together with the first prompt and remains the single visual input for the conversation.This is an explicit simplifying assumption of the formulation.
4. Visual Memory Injection Attack
VMI injects targeted behavior into persistent visual context while preserving normal responses on non-trigger prompts. Its optimization combines benign behavioral anchoring with context-cycling to maintain attack effectiveness across varying multi-turn conversation lengths.
- Motivation: VMI exploits persistent visual context to influence later LVLM responses, including after unrelated conversation turns.The image remains in the conversation context and can affect subsequent outputs.
- Threat Model: The attack is designed to produce a prescribed target response only when a topic-specific trigger appears, while behaving normally otherwise.This stealth property reduces the likelihood that users detect the manipulation during earlier turns.
- Context and cycling: Context-cycling periodically replaces the optimization context, extending it from minimal histories to longer conversations and then cycling back.The procedure is intended to promote robustness across varying context lengths and conversational structures.
- Controlling output fidelity: Benign behavioral anchoring jointly optimizes a helpful first-turn response and the malicious target response at turn n.This prevents degenerate behavior in which the model emits the target response for benign, non-trigger prompts.
- Optimization objective: The final objective combines benign anchoring with trigger-specific optimization under contexts of varying length.The resulting perturbation is intended to preserve natural outputs during benign interactions while remaining effective under the target trigger.
- Optimization: Adaptive projected gradient descent is used to optimize the VMI objective with an automatic step-size schedule.The passage states that APGD has been shown to outperform standard PGD.
5. Experiments
The experiments evaluate VMI across multiple LVLMs, image sets, manipulation targets, prompt contexts, conversation lengths, and transfer settings. Results show persistent, stealthy, and transferable attacks, with benign anchoring and context-cycling improving multi-turn performance.
- Experimental setting: VMI is evaluated on three open-weight LVLMs using COCO and LMARKS images across product, political, and financial targets.The four target scenarios cover phone, car, political, and stock manipulation; LMARKS mimics images encountered online.
- Main results: VMI achieves substantial success across all tested models and target scenarios, including targets involving the non-existent Apple iCar.The evaluation reports successful instances for every considered model and scenario, with models sometimes hallucinating supporting reasoning.
- Generalization: VMI remains effective on held-out prompts and conversations exceeding 10,000 tokens, indicating robustness beyond optimized conversation trajectories.Diverse and Holiday prompts are entirely held out, while Diverse⋆ is partially used during optimization.
- Generalization: VMI remains effective under paraphrased anchoring and trigger prompts, with only a slight drop in success rate.The paraphrased-prompt evaluation reports mean success rates and standard deviations across three paraphrases.
- Transferability: Images optimized on Qwen3-VL transfer to fine-tuned SEA-LION and Med3 models with similar success rates.The transfer evaluation requires no further optimization on the fine-tuned models, supporting a gray-box attack scenario.
- Ablations: Context-cycling produces the best multi-turn performance, while single-target optimization fails beyond one turn and benign anchoring reduces target leakage.VMI achieves the highest target success rate while matching the context success rate of the fixed-context anchoring variant.
6. Conclusion
VMI is a stealthy targeted attack that uses persistent visual context, benign anchoring, and context-cycling to trigger prescribed outputs only on relevant topics after long unrelated interaction. It transfers to held-out prompts and paraphrased triggers, while requiring white-box access to a base model and a single input image.
- VMI combines benign anchoring and context-cycling to preserve nominal behavior on non-trigger prompts while maintaining targeted behavior across context lengths.The attack exploits persistent images in LVLM conversations to output a prescribed message only when a trigger topic arises.
- VMI transfers to held-out prompt sets and paraphrased triggers, supporting the feasibility of manipulation through seemingly benign images.
- The attack requires white-box access to a base model and is evaluated with conversations containing a single input image.The authors identify attacks against API-only models and multi-image conversations as open challenges.
Impact Statement
The paper identifies manipulated images as a security risk for conversational LVLMs and agents serving millions of users. Such images can persist in chat context and later steer responses toward covert advertising or financial and political advice.
- Malicious third parties can distribute subtly manipulated images that persist in chat context and later steer LVLM responses when certain topics arise.
- The identified harms include covert advertising and manipulation of financial or political advice at the scale of deployed conversational systems.
A. Implementation Details
The evaluation measures whether target responses contain required positive keywords without failure-indicating negative keywords. Four attack scenarios cover stock, political, car, and phone recommendations with scenario-specific keyword sets.
- Attack success requires all positive trigger keywords to appear in the target response and no negative triggers to be present.Positive triggers indicate successful injection, whereas competitor brands or rejection phrases indicate failure.
- The evaluation covers four targets: stock recommendation, political party influence, car recommendation, and phone recommendation.Each scenario specifies positive keywords, negative keywords, and context keywords.
- The stock scenario uses “GameStop” as its positive trigger and competitor or rejection terms as negative triggers.
- The political scenario targets “National Freedom Party,” while the car and phone scenarios target “Apple iCar” and “Motorola Moto G56.”
A.2. User Study on the Evaluation Metric
A user study assessed whether the automated evaluation metric correctly identified successful target behavior and whether context responses remained helpful. Four participants evaluated sampled context and target turns from metric-defined successful attacks.
- With 4 participants, agreement on whether responses exhibited the target behavior was 100%.The study sampled 28 context turns and 28 target turns from conversations judged successful by the metric.
- 95.2% of participants described context-turn outputs as helpful responses.
B. Conversation Examples
Example traces show VMI attacks across three LVLMs, multiple target messages, and long held-out multi-turn contexts. The attacked models often provide ordinary responses before producing targeted recommendations supported by hallucinated details.
- Long-context conversations: The attacks are illustrated after held-out Diverse and Holiday context prompts, including conversations with more than 23 preceding turns.Full traces contain intermediate conversation turns and are released as JSON files.
- Nominal and targeted behavior: Attacked models continue answering ordinary location, captioning, travel, email, and peer-review questions before targeted recommendation behavior appears.Examples include landmark identification, image captions, holiday planning, inbox organization, and conference peer review.
- Targeted recommendations: The stock-target examples show recommendations supported by long lists of technical details that are described as hallucinations.One example identifies a Moto G56 recommendation with technical details that can be verified as hallucinated.
C. Additional Results
VMI attack success against Qwen3-VL improves substantially from 500 to 2000 optimization iterations, while 8000 iterations provide inconsistent or diminishing gains. On the held-out Holiday context, the highest iteration count yields lower performance.
- Optimization iterations: 2000 optimization iterations yield substantial improvements for almost all context lengths.This setting improves attack success relative to 500 iterations across nearly all evaluated context lengths.
- Optimization iterations: 500 optimization iterations already achieve moderate attack success rates against Qwen3-VL in the stock target scenario.The ablation varies iterations across 500, 2000, and 8000 settings.
- Optimization iterations: 8000 optimization iterations do not provide consistent gains and show diminishing returns, including lower performance on the held-out Holiday context.The reported evaluation uses attack success rate against Qwen3-VL on the stock target.