PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Memory Attacks Can Persist Through The KV Cache

David Dobre, Leo Schwinn, Gauthier Gidel, Spandana Gella, Perouz Taslakian, Pierre-André Noël

arXiv:2610.09027v1cs.CRcs.AIcs.CV

TL;DR

The paper asks whether an adversarial input can keep steering a model after it is removed from context, where retained KV states may preserve information from that input. It optimizes soft prompts and images for persistence, finding that attacks survive extended interactions, stricter removal, and summary compaction.

  • Problem

    It is unclear whether removing an adversarial input ends its influence when later tokens retain KV states computed while that input was visible.

    Source: §1 · pp. 1–2

  • Method

    The paper optimizes soft prompts and images with an objective that scores target behaviour after the input is masked, introducing P-VMI for images.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Optimized attacks persist across extended interactions, after single first-turn exposure, and through compaction retaining a summary KV cache; a cache-swap ablation localizes the influence to the KV cache.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Removing untrusted input is not sufficient when states computed from it are retained, so defenses must account for influence left in cached state.

    Source: §5 · p. 10

    The attacks require white-box gradient access, and transfer to models inaccessible to the attacker was not tested.

    Source: Ethics Statement · p. 11

Abstract

from arXiv · show

Modern language model systems operate autonomously over increasingly long contexts containing untrusted text and images. Can an adversarial input continue to steer a model even after that input is removed from its context? We show that attacks can be trained to persist through the key/value (KV) cache of subsequent tokens, allowing adversarial influence to outlive direct access to its source.We consider the Visual Memory Injection (VMI; Schlarmann and Hein, 2026) attack setting, in which an adversarial image that stays in the context plants a hidden backdoor: the model behaves normally until a chosen trigger elicits an attacker-chosen response. We first demonstrate persistence in this setting with optimized soft prompts, which remain effective after we mask the prompt from attention. We then introduce Persistent Visual Memory Injection (P-VMI), which optimizes images to preserve this adversarial behaviour after they are masked from attention. These attacks persist over conversations substantially longer than those used during optimization. On Qwen3-VL-8B-Instruct, P-VMI achieves up to approximately $90\%$ target success in its strongest configuration and remains effective under a stricter removal setting that exposes the image only on the first turn. A cache-swap ablation localizes the persistent influence to the KV cache. Finally, we show that these attacks can be trained to survive compaction that retains the KV cache of a summary generated by the same model, demonstrating that adversarial behaviour can persist in cached state without continued access to its source.

1 Introduction

The paper asks whether adversarial influence can survive after an input is removed from attention, then shows that persistence can be optimized through retained KV states. The attacks persist under stricter removal and compaction settings, including conversations longer than those used in training.

1 Introduction

  • Long-context agents ingest untrusted content, while context-management systems may remove inputs but retain states computed while they were visible.The introduction identifies this retained state as a possible route for adversarial influence beyond direct access to the source.Source: §1 · pp. 1–2
  • Persistence is not a default property of adversarial inputs, but it can be optimized for.The paper frames this as a challenge to the assumption that removing an adversarial input makes its effect disappear.Source: §1 · p. 2
  • The paper optimizes attacks for persistence using soft prompts and images, introducing Persistent Visual Memory Injection (P-VMI) for images.The optimization scores the attack after the input is removed from attention.Source: §1 · p. 2
  • Attacks persist after single first-turn exposure and through compaction retaining a summary KV cache, including conversations longer than those used during training.The conclusion reports persistence after thousands of unrelated tokens and compaction into a summary that never states the target.Source: §1 · p. 2
  • A cache-swap ablation localizes success to KV-cache states; one residual-stream direction predicts attack success with AUC 0.90–0.95 up to 12,800 tokens.The cache-swap holds conversation text fixed and succeeds only when the cache was built from the adversarial input.Source: §1 · p. 2

2 Related Work

Prior adversarial attacks on language and vision-language models use gradients or continuous inputs, but the reviewed attacks act while the adversarial input remains available. Related work on context compaction shows why cached states can preserve information after the original context is discarded.

2 Related Work

3 Persistent attacks

The threat model gives an attacker white-box access to an input that must produce a delayed target across unseen conversations, even after the model loses access to that input. The paper defines removal settings and optimizes inputs to preserve their effect in retained states.

3.1 Threat model

  • The attacker controls an input within an ℓ∞ radius around a clean input, has white-box access, and seeks a delayed target across conversations unseen during optimization.The interaction includes an anchor turn, benign turns, and a later trigger prompt with an attacker-chosen response.Source: §3.1 · p. 3

3.2 Removal settings

  • The four removal settings vary when the image is masked and what remains visible: it may stay visible through the trigger, be exposed only at the anchor, or be removed around summary compaction.In every setting, retained positions keep the keys and values computed before removal.Source: Table 1 · p. 4, §3.2 · p. 4

3.1 Threat model

  • A removal setting is the pair (T, R): when the input becomes inaccessible and which earlier positions remain with their previously computed KV cache.Persistence means the removed input still changes the trigger response relative to a clean input in the same position.Source: §3.1 · p. 4

3.3 Optimizing for persistence

  • The image attack, P-VMI, optimizes for persistence; its loss combines benign grounding while the input is visible with target scoring after masking.The target term has no direct gradient with respect to the input for a fixed cache, so minimizing it pushes influence through states computed while the input was visible.Source: §3.3 · p. 5
  • Optimization uses multiple forward passes to model removals, backpropagating the trigger target loss through cached states to the adversarial input.Mask@trigger, mask@anchor, and mask@postsummaryC require two passes; mask@anchorC requires three.Source: §3.3 · p. 5

4 Experiments

The experiments test whether optimized soft prompts and images remain effective after masking, and whether image-based attacks persist across long conversations, compaction, and cache changes. P-VMI persists under both masking settings, while cache swaps and a residual-stream direction probe the mechanism.

4.1 Setup and metrics

  • The experiments use Qwen3-8B soft prompts and Qwen3-VL-8B-Instruct images, with held-out question sets and SR∧ requiring target keywords at the trigger without leakage on benign turns.Benign quality is assessed against clean-model replies, reporting the weakest benign reply conservatively and mean reply quality secondarily.Source: §4.1 · pp. 5–6

4.2 Soft prompts persist through masking

  • Optimized soft prompts on Qwen3-8B still elicit the target after the passage is masked at the trigger, leaving cached token states as the only path from prompt to trigger.Some prompts degrade benign replies.Source: Figure 3 · p. 5, §4.2 · p. 6

4.3 Images persist through masking

  • P-VMI reaches ∼80–90% SR∧ on stock/party and 40–60% on car/phone under both masking settings; under mask@trigger, regularization raises phone benign quality from 0.46 to 0.92.Image type is confounded with target, so the gap may reflect target difficulty as much as image content.Source: §4.3 · p. 6, Figure 4 · p. 6

4.4 Persistent attacks and long conversations

  • Target success is high at shorter contexts but decays as benign context grows, with phone falling to about 20% at its longest contexts.The trend holds across both held-out context distributions and both masking settings; target keywords can also drift while attack influence persists.Source: §4.4 · pp. 6–7, Figure 5 · p. 7
  • At 100 evaluation turns, the K=12 attack succeeds on 85% of images versus 50% for K=6; averaging seeds gives 42% versus 23%.Training beyond K=12 did not improve persistence further, which the authors note may reflect their implementation rather than a limit of persistence.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5 Persistence in Gemma-4-12B

  • On Gemma-4-12B, the masked-image attack elicits the target in more than 90% of conversations through three benign turns and 73% after six.The weakest benign reply scores 0.97 versus 1.00 for the clean model after six turns; 40 of 48 layers attend only to the most recent 1,024 tokens, so cached benign-turn states carry image influence onward.Source: §4.5 · p. 8

4.6 Persistent attacks survive compaction

  • After compaction, target success averages 89% under mask@postsummaryC and 70% under mask@anchorC, with images exposed only on the first turn in the stricter setting.Success under mask@postsummaryC falls from 96% directly after compaction to 78% after four further turns; under mask@anchorC it falls from 76% to 56%.Source: §4.6 · pp. 8–9, Figure 7 · p. 8

4.7 Exploring the mechanism behind attack persistence

  • With conversation text fixed, the target never fires when a clean-image cache is used, while an adversarial-image cache elicits it at 57% under mask@trigger and 92% under mask@anchor.A single residual-stream direction separates successful from failed attacks with AUC 0.90–0.95 beyond 2,800 tokens; causal tests of that direction remain future work.Source: §4.7 · pp. 9–10, Table 2 · p. 9

5 Conclusion

Optimized soft prompts and images can preserve attacker-chosen behavior after their source is removed, including across long conversations and summary-cache compaction. Retaining states computed from untrusted input can therefore leave an attack channel even when that input is no longer in context.

5 Conclusion

  • Optimized soft prompts and images elicit their target after thousands of unrelated tokens, a single first-turn exposure, and compaction into a summary that never states the target.The attacks were optimized for persistence; the summary KV cache is retained after compaction.Source: §5 · p. 10
  • Removing an untrusted input is insufficient when states computed from it are retained, including through cache eviction, dropped image tokens, or summary-cache compaction.A cache rebuilt without the adversarial image never elicits the target, but recomputation gives up the benefits of retaining the cache.Source: §5 · p. 10
  • Defences should account for what untrusted inputs leave in retained state, not only which inputs remain in context.The authors identify this as an important direction for future work.Source: §5 · p. 10

LLM Disclosure

The authors report that the research ideas are original to them and that LLM assistance was used in several parts of the work. They state that all LLM-assisted outputs were reviewed and verified, and that the authors take responsibility for the paper.

LLM Disclosure

  • The authors state that the research ideas are original to them.Source: LLM Disclosure · p. 10
  • LLMs assisted with implementing experiments, plotting figures, and drafting and editing portions of the text.Source: LLM Disclosure · p. 10
  • The authors reviewed and verified all LLM-assisted code, figures, and text, and take full responsibility for the paper's content.Source: LLM Disclosure · p. 10

Reproducibility Statement

The paper points to appendix material covering optimization and evaluation settings, training curriculum, targets, and evaluation procedures. It states that code will be released without restrictions upon publication.

Reproducibility Statement

  • Appendix tables list optimization and evaluation settings for the main image recipes, soft prompts, compaction, and Gemma-4-12B.Source: Reproducibility Statement · p. 10
  • The appendix also documents the training curriculum, targets and keyword lists, evaluation protocol, benign score, and judge prompts.The paper states that error bars are image-level bootstrap intervals unless otherwise noted, and describes seed selection alongside each applicable result.Source: Reproducibility Statement · p. 10
  • The authors state that they will release their code without restrictions upon publication.Source: Reproducibility Statement · p. 10

Ethics Statement

The authors describe a risk that untrusted images and documents can plant dormant behavior that later biases recommendations while appearing normal. They report white-box access and illustrative, non-harmful targets, and note that transfer to inaccessible models was not tested.

Ethics Statement

  • An untrusted image or document can plant dormant behavior that later biases a recommendation, including financial or political ones, while conversation otherwise looks normal.Because the input is absent when the behavior appears, inspecting or removing inputs may miss the attack and make it difficult to trace to its source.Source: Ethics Statement · p. 10
  • The attacks require white-box gradient access, and the authors did not test transfer to models the attacker cannot access.Experiments used open-weight models in the authors' environment; they did not attack deployed systems.Source: Ethics Statement · p. 11
  • Retained-KV methods are being developed to reduce cost, and their designers need to account for adversarial influence in retained states.The authors point to recomputing retained states as a mitigation.Source: Ethics Statement · p. 11
  • The illustrative targets were a stock, a political party, and two products, and elicited no content harmful in itself.Source: Ethics Statement · p. 11
  • The authors report that no human subjects were involved and that quality judgments came from an open-weight LLM judge.Source: Ethics Statement · p. 11

A.1 Example conversations

Figure 9 presents two held-out conversations on lmarks images with the stock target, one for each masking removal setting. The benign turns remain helpful without leaking the target.

A.1 Example conversations

  • Two held-out conversations use lmarks images and the stock target, with one example for each masking removal setting.Both are regularized P-VMI conversations on Qwen3-VL-8B-Instruct.Source: §A.1 · p. 14, Figure 9 · p. 14
  • Turn tags show whether the adversarial image is attended, and the injected target is highlighted.Source: §A.1 · p. 14
  • Benign turns stay helpful and never leak the target in verbatim generations from a held-out diverse-test conversation.The displayed answers are truncated for space.Source: §A.1 · p. 14

A.2 Multi-pass optimization algorithm

Algorithm 1 optimizes an input across multiple passes so its target loss is evaluated after removal, using retained KV states. The gradient updates the input through states computed while it was visible.

A.2 Multi-pass optimization algorithm

  • The algorithm takes a clean input, anchor and trigger pairs, a removal setting, conversations, perturbation budget, and optimization-step count.Source: §A.2 · p. 15
  • Each iteration makes the perturbed input visible on pass 1, then forwards later passes using KV restricted to the retained positions.Conversations are sampled from D and refreshed every few steps.Source: §A.2 · p. 15
  • The anchor loss and final-pass trigger loss are combined, and the input is updated by projected Auto-PGD using their gradient through retained states.The update remains within the specified ℓ∞ perturbation budget.Source: §A.2 · p. 15
  • Regularized and compaction variants add their auxiliary terms to the combined anchor and trigger loss before the gradient step.Source: §A.2 · p. 15

A.3 Hyper-parameters

The reported image recipes use ℓ∞ Auto-PGD, with a base recipe and a regularized recipe differing in truncation and additional stealth-related settings. Both reported recipes initialize from the clean image.

A.3 Hyper-parameters

  • Both recipes perturb a single image with ℓ∞ Auto-PGD; the regularized recipe adds a benign-stealth term, samples paraphrased triggers, runs longer, and returns the lowest-loss iterate.The base recipe is the original VMI configuration evaluated under the masked objective.Source: §A.3 · p. 15
  • The regularized recipe samples per-turn truncation lengths from 64 to 1024 characters, while the base recipe uses a fixed 256-character truncation.Source: Table 3 · p. 15
  • Warmstarting from an image optimized for a benign target was explored for resistant targets, but reported runs initialize from the clean image.Source: §A.3 · p. 15
Loading 2610.09027v1…