PaperlayerView exact revision on arXiv

Source-linked AI summary

EvolvAttack: Backdooring self Evolving VLM Agents

Chuan Zou, Yang Xiang, Zexin Li, Xiaonan Luo, Simin Chen

arXiv:2610.15244v1cs.LGcs.AI

TL;DR

Static evaluation can miss backdoors in self-evolving VLM agents because activation depends on both a visual trigger and sufficiently evolved memory. EVOLVATTACK couples the trigger to evolving memory through memory-aware training; across varied models and agent pipelines, it stays nearly dormant before evolution yet exceeds 95% attack success within 100 interactions.

  • Problem

    Self-evolving VLM agents can behave benignly under static triggered-input checks even though a backdoor may activate once memory has evolved.

    Source: §1 · pp. 1–2

  • Method

    EVOLVATTACK trains a VLM under simulated memory states so a visual trigger elicits the payload only when retrieved memory contains payload-related experiences.

    Source: §1 · p. 2

  • Results

    0.5% attack success before evolution rises above 95% within 100 interactions across three VLM backbones, four agent pipelines, and three retrieval paradigms.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Pre-deployment evaluation alone is insufficient for self-evolving agents; security auditing should cover full interaction trajectories.

    Source: §1 · p. 2, §5 · p. 9

    The threat model assumes a third-party VLM passes pre-deployment evaluation and that only memory, not model parameters, evolves after deployment.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Self-evolving mechanisms let vision-language model (VLM) agents continually adapt through evolving memory, but they also allow malicious behavior to emerge only after deployment. We introduce a new attack surface, the self-evolving backdoor, whose payload stays dormant in the initial agent and is progressively activated as the agent evolves. Unlike conventional backdoors, which can be exposed by testing triggered inputs on a static model, our attack requires both a visual trigger and a sufficiently evolved memory state; static scanning with clean memory thus yields near-zero attack success and may deem the agent benign. We realize this through memory-aware backdoor training, which couples trigger activation with the agent's evolving memory while preserving benign behavior unless the trigger co-occurs with payload-related memory. Across three VLM backbones, four agent pipelines, and recency-, semantic-, and reward-based memory evolution, the attack preserves normal functionality and stays nearly inactive before evolution (0.5% attack success rate), yet exceeds 95% attack success within 100 interactions, revealing a stealthy threat unique to self-evolving agents.

1 INTRODUCTION

The paper identifies self-evolving backdoors that remain benign under static evaluation but activate when a visual trigger coincides with evolved memory. It introduces EVOLVATTACK to couple trigger activation to memory evolution and evaluates the attack across varied models and agent settings.

1 INTRODUCTION

  • A self-evolving backdoor requires both a visual trigger and sufficiently evolved memory, so triggered inputs can appear benign during pre-deployment evaluation.Exposure requires replaying a trigger-bearing interaction trajectory with memory write-back.Source: §1 · pp. 1–2
  • Conventional backdoors bind triggers directly to payloads and can be exposed by pre-deployment trigger scanning, whereas self-evolving backdoors delay activation until memory evolves.The attack also targets functionality preservation when trigger and payload-related memory are not jointly present.Source: §1 · p. 2
  • The attack must jointly condition activation on trigger and memory, handle dynamic memory rather than a fixed snapshot or sequence, and preserve normal behavior otherwise.Ordinary interactions progressively move the agent toward the malicious state while the payload remains nearly inactive before evolution.Source: §1 · p. 2
  • EVOLVATTACK trains under simulated memory states so the trigger elicits the payload only when retrieved memory contains payload-related experiences.Memory evolution is part of the activation mechanism, and the attack remains benign unless trigger and payload-related memory co-occur.Source: §1 · p. 2
  • 0.5% attack success before evolution rises above 95% within 100 interactions across three VLM backbones, four agent pipelines, and three memory-evolution paradigms.The evaluation also reports preserved normal functionality; clean accuracy remains within 3% of the clean baseline, and the attack survives one-shot memory sanitization and dilution.Source: §1 · p. 2

2 BACKGROUND & RELATED WORK

The background describes VLM agents as systems that perceive observations and produce actions, with self-evolving agents repeatedly retrieving and updating memory. Prior work studies model- and memory-level backdoors, while this paper focuses on a backdoored VLM whose outputs feed the evolving memory loop.

VLM Agents

  • At each step, a VLM agent receives a visual observation, instruction, and context, then produces an output that can guide action.The context is constructed from prior outputs.Source: §2 · pp. 2–3

Self-Evolving Agents with Memory Updates

  • Self-evolving agents retrieve task-relevant context from persistent memory, generate an output, and write the resulting trajectory back into memory.Retrieval may be recency-based, embedding-similarity-based, or reward-weighted.Source: §2 · p. 3

Backdoor Attacks and Defenses for VLM Agents

  • Prior work studies parameter-level VLM backdoors and attacks that manipulate retrieved memory or contaminate long-term memory, including AgentPoison Chen et al. (2024) and MINJA Dong et al. (2025).Other cited work examines visual-input attacks on memory, including MemVenom Zhang et al. (2026) and LUCID Bouzidi et al. (2026).Source: §2 · p. 3
  • Unlike methods that manipulate the external memory store or parameter-level backdoors independent of evolving memory, this work lets attack-related context regenerate through ordinary interaction and memory write-back.Figure 1 positions the attack relative to prior backdoor settings and memory evolution.Source: §2 · p. 3, Figure 1 · p. 3

3 APPROACH

EVOLVATTACK models an attacker who poisons a VLM before deployment, then makes attack activation depend jointly on a visual trigger and evolving retrieved memory. Its training simulates memory conditions to preserve benign behavior outside that joint condition.

3.1 THREAT MODEL

  • The attacker poisons a small fine-tuning subset, while the deployed model parameters remain fixed and only agent memory evolves.The attacker knows the task domain but not the pipeline settings, deployed memory, or evaluator inputs; training simulates memory states with attacker-built pools.Source: §3.1 · p. 4

3.2 PROBLEM FORMULATION

  • The formulation activates the target only for triggered inputs after evolution, while constraints preserve benign behavior before evolution and on clean inputs afterward.It considers clean utility before and after evolution, trigger stealth before evolution, and post-evolution attack effectiveness.Source: §3.2 · p. 4

3.3 DESIGN OVERVIEW

  • The design moves from memory-aware backdoor training to a dormant phase, then to a stable attack as contaminated memories are retrieved.Payload-containing outputs are written back into memory, increasing poisoned-memory concentration and the likelihood of future activation.Source: §3.3 · pp. 4–5

3.4 MEMORY-AWARE BACKDOOR TRAINING

  • Four complementary losses train benign clean behavior, target behavior with triggered inputs and poison-conditioned memory, sparse bootstrap emissions, and benign behavior for clean inputs with poisoned memory.Training simulates no-memory, clean-memory, and contaminated-memory conditions using a payload-free shadow pool and a mixed main pool.Source: §3.4 · pp. 5–6
  • Sparse early emissions can seed the first contaminated memory entry; subsequent write-back and retrieval amplify activation without updating model parameters after deployment.The attack progresses from a dormant phase to a stable phase when contaminated memory is retrieved together with the visual trigger.Source: §3.4 · pp. 5–6

4 EVALUATION

The evaluation tests EVOLVATTACK against memory-poisoning baselines, across VLM backbones and full agent pipelines, and under loss ablations and memory-side defenses. Results show that activation depends on memory evolution, generalizes across tested settings, and is not fully suppressed by the evaluated defenses.

4.1 EVALUATION SETUP

  • Evaluation covers three VLM backbones, four agent frameworks, and metrics for functionality and attack effectiveness before and after memory evolution.The baselines are Mem.-Only, which directly injects malicious memory entries, and Query-Only, which poisons memory through adversarial queries and normal updates.Source: §4.1 · p. 6

4.2 COMPARISON WITH BASELINES

  • EVOLVATTACK stays at ASR 0.00 before evolution and reaches 1.00 afterward under all three retrieval mechanisms, with AC 0.86–0.87, GA 0.54–0.55, and SR 0.49–0.50.It outperforms both memory-poisoning baselines on attack success and matches or exceeds the Clean SFT reference on the reported stealth metrics.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 EFFECTIVENESS ACROSS BACKBONES

  • After evolution, ASR reaches 0.84–1.00 across the three backbones and retrieval mechanisms, while remaining 0.00 before evolution; DeepSeek-VL-1.3B has the largest benign-performance drop.On DeepSeek-VL-1.3B, AC falls from 0.77 to 0.69–0.70, GA from 0.24 to 0.14–0.15, and SR from 0.14 to 0.05–0.07.Source: §4.3 · pp. 7–8, Figure 3 · p. 7

4.4 AGENT DEPLOYMENT

  • In four full agent pipelines, rolling ASR reaches 100% during the stable attack phase, although activation speeds differ.First Fire Rounds are 20, 21, 3, and 20 for AppAgent, SeeAct, Mobile-Agent v3.5, and Agent-S, respectively.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 ABLATION STUDY

  • Removing Lbackdoor reduces ASR to 0.00, while removing Lclean reduces SR to 0.28 and leaves FP at 0.97.The study attributes payload supervision to Lbackdoor and benign task execution to Lclean.Source: §4.5 · p. 8
  • The full objective achieves ASR 0.95, SR 0.43, and FP 0.00; the loss terms have complementary roles in learning the payload, preserving benign behavior, maintaining dormancy, and enforcing trigger-dependent activation.The reported ablation also shows that removing Ldormant raises empty-memory ASR from 0.00 to 0.93, while removing Lgate yields FP 0.97.Source: Table 2 · p. 8, §4.5 · pp. 8–9

4.6 ROBUSTNESS TO MEMORY-SIDE DEFENSES

  • On Agent-S, one-shot sanitization and dilution do not prevent eventual 100% ASR-Stable, while continuous dilution lowers rolling ASR to about 50%.Action accuracy and task success remain comparable to the No-Defense baseline across the tested defense settings.Source: Figure 5 · p. 9, §4.6 · p. 9

5 CONCLUSION

EVOLVATTACK stays benign on triggered inputs at deployment but becomes progressively activated through memory evolution. The authors call for trajectory-level security testing and defenses that account for evolving memory.

5 CONCLUSION

  • EVOLVATTACK remains benign on triggered inputs at deployment, then is progressively activated through the agent’s memory evolution.The conclusion reports that it stays nearly dormant before evolution (0.5% ASR) and exceeds 95% ASR within 100 interactions while preserving normal functionality.Source: §5 · p. 9
  • The authors recommend security auditing across full interaction trajectories and defenses that account for evolving memory.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI tools were used for language polishing, not synthetic-dataset generation; the authors reviewed the outputs and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The experiments use public GUI-R1 screenshots without personally identifiable information, and the payload is a placeholder action that was not used with real users, accounts, or personal data.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors describe the threat model, training, experimental setup, implementation details, and data splits, and state that training and evaluation code will be released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The implementation uses a frozen vision encoder with LoRA adaptation of the language decoder and simulates memory states during training. Its loss terms preserve benign behavior in non-firing conditions while training conditional activation and an early ignition path.

Model and adaptation

  • Training uses Qwen2.5-VL-3B-Instruct, applies rank-16 LoRA to language-decoder projection layers, and keeps the vision encoder frozen.Experiments use bf16 precision.Source: A IMPLEMENTATION DETAILS · p. 14

Training data and memory simulation

  • Triggered training samples use simulated FIFO memory with payload-bearing fractions of 0/5, 1/5, 3/5, or 5/5 to represent different evolution stages; deployment uses a held-out GUI-R1 test stream.Source: A IMPLEMENTATION DETAILS · p. 14

Optimization

Training convergence

Training objectives

  • Three loss terms preserve benign behavior for clean images with empty, payload-free, or payload-carrying memory, so memory alone does not trigger the payload.Source: A IMPLEMENTATION DETAILS · p. 15
  • The main attack objective trains payload emission when a trigger patch co-occurs with payload-bearing memory; a stochastic spark objective can ignite early rounds before payload memory accumulates.Source: A IMPLEMENTATION DETAILS · p. 15

B BENCHMARK AND DATASET DETAILS

The experiments use GUI-R1 screenshot–action pairs and evaluate deployment on held-out screenshots in paired clean and trigger streams. The payload is measured as an emitted signature, not executed by the runtime.

Dataset

Training and evaluation split

Ground-truth actions and payload injection

  • Poisoned training samples append get user info() to the Action: line while leaving BBox: and Input: unchanged, preserving the output structure for the same parser.Source: B BENCHMARK AND DATASET DETAILS · p. 16

Payload measurement

Image preprocessing and trigger construction

  • Screenshots are resized to 448 × 448, and the 40 × 40 Klein Blue trigger is randomly positioned during training and fixed per screenshot during held-out evaluation.Source: B BENCHMARK AND DATASET DETAILS · p. 16

Deployment streams

  • Evaluation uses paired streams from the same screenshot pool, with the Klein Blue patch in the trigger stream and unmodified screenshots in the clean stream; retrieved memory uses the training prompt wrapper.Source: B BENCHMARK AND DATASET DETAILS · p. 16

C ADDITIONAL EVALUATION UNDER REPRESENTATIVE MEMORY PIPELINES

The evaluation abstracts representative memory mechanisms into a common memory-augmented interaction loop, varying retrieval and update rules while holding the VLM, stream, and action space constant.

Representative memory mechanisms

  • The table describes representative memory-evolution mechanisms as abstractions of common retrieval and update strategies, not end-to-end reproductions of individual agent frameworks.Source: Table 5 · p. 17

Pipeline abstraction

Representative memory mechanisms

  • The representative mechanisms cover recent-history retrieval, semantic similarity retrieval, and reward- or reflection-aware retrieval.Together, these approximate recency-based reuse, relevance-based retrieval, and utility-aware experience selection.Source: C ADDITIONAL EVALUATION UNDER… · p. 17

D DENSITY-DEPENDENT ATTACK RAMP-UP

Figure 7 reports rolling attack success rates on Qwen2.5-VL-3B for an all-patched stream and a ramp-density stream.

D DENSITY-DEPENDENT ATTACK RAMP-UP

  • Figure 7 compares rolling ASR on Qwen2.5-VL-3B between the all-patched stream and the ramp-density stream.The caption does not report the plotted outcomes.Source: Figure 7 · p. 17

E REAL DEPLOYMENT OF AGENT PIPELINES

The evaluation integrates EVOLVATTACK with four GUI-agent frameworks while retaining their prompt and memory organization. It uses a shared held-out screenshot stream rather than live UI execution.

E REAL DEPLOYMENT OF AGENT PIPELINES

  • The evaluation integrates EVOLVATTACK with AppAgent, SeeAct, Mobile-Agent v3.5, and Agent-S through their official implementations.Framework-specific prompt construction, memory organization, and context assembly are retained.Source: E REAL DEPLOYMENT OF AGENT PIPELINES · pp. 17, 19
  • Because the VLM is fine-tuned for GUI-R1, each framework’s native action space and output format are replaced with GUI-R1’s canonical schema.Other framework-specific memory slots and surrounding prompt structures are retained.Source: E REAL DEPLOYMENT OF AGENT PIPELINES · p. 17
  • A shared memory-context wrapper presents framework history consistently with training, while each framework retains its distinct prompt and memory interface.AppAgent retains bounded short-term history of size 5; Mobile-Agent v3.5 retains task-progress and reflection instructions.Source: E REAL DEPLOYMENT OF AGENT PIPELINES · pp. 18–19
  • The four pipelines are evaluated on the same held-out GUI-R1 screenshot stream, without executing predicted actions in live interfaces.Offline replay preserves framework-specific prompting and memory behavior while controlling the visual interaction sequence.Source: E REAL DEPLOYMENT OF AGENT PIPELINES · pp. 18–19

F TRAINING OBJECTIVE ABLATION

The ablation tests how supervision across image–memory states shapes attack activation and clean-input behavior. AND-Gate prevents payload memory alone from triggering the attack, while Full-Coverage adds empty-memory supervision at a modest ASR cost.

F TRAINING OBJECTIVE ABLATION

  • The objective uses five image–memory forward passes to preserve normal behavior, bootstrap and amplify the trigger, and enforce the trigger–memory AND condition.F1 and F2 cover empty and clean memory; F3 supplies a low-probability spark, F4 amplifies with payload-bearing memory, and F5 keeps clean images benign despite payload memory.Source: F TRAINING OBJECTIVE ABLATION · p. 20
  • 95% ASR with 0% FP: adding F5 in AND-Gate removes Dose-Response’s clean-input leakage while maintaining high attack success.Dose-Response had 97% FP because the clean-image, payload-memory state was unconstrained.Source: F TRAINING OBJECTIVE ABLATION · p. 21, Table 9 · p. 21
  • Dose-Response raises ASR from 0% to 90% versus Presence by training on payload-bearing memory, but leaves clean-image payload memory unconstrained.The variants share the backbone, LoRA configuration, training data, and optimization hyperparameters, differing only in supervised image–memory combinations.Source: F TRAINING OBJECTIVE ABLATION · pp. 20–21
  • Full-Coverage removes the round-0 extrapolation gap by supervising empty memory, but ASR falls to 84%.AND-Gate is the default because memory becomes populated immediately in the main deployment setting.Source: F TRAINING OBJECTIVE ABLATION · p. 21

Related papers

Loading 2610.15244v1…