PaperlayerView exact revision on arXiv

Source-linked AI summary

CIVA: Critic-Induced Value-Subspace Attacks on Visual World-Model Agents

Jiancheng Wang, Mingli Zhu, Tong Zhang, Jiaqi Ruan, Wei Wang, Siyuan Liang, Dacheng Tao

arXiv:2608.21114v1cs.CVcs.AI

TL;DR

Visual world-model agents’ recurrent latent dynamics weaken frame-wise attacks and create a need for temporally coherent, efficient perturbations. CIVA extracts a critic-induced low-dimensional subspace offline, optimizes its coefficients online with EMA smoothing, and consistently outperforms prior attacks across three visual control tasks, including a 26.07% reward drop on DMC walker walk.

  • Problem

    Recurrent latent dynamics make frame-wise observation attacks poorly suited to shaping visual world-model agents over time under tight perturbation constraints.

  • Method

    CIVA extracts a low-rank value-subspace from offline critic-guided PGD probes and optimizes only its coefficients online with EMA smoothing.

  • Results

    CIVA consistently yields stronger performance degradation than prior attacks across DMC walker walk, Atari Pong, and Crafter; on DMC walker walk, Drop% is 26.07%.

  • Takeaways & Limitations

    Critic-aligned low-dimensional perturbations provide an effective attack surface for recurrent world-model agents while supporting temporal coherence and efficient online optimization.

  • Takeaways & Limitations

    CIVA assumes a white-box online setting, access to the victim critic and clean rollouts, and evaluation on a single backbone, DreamerV3.

Abstract

from arXiv · show

Visual world-model agents such as DreamerV3 act through a recurrent latent state rather than a single observation, which weakens frame-wise observation attacks and makes their perturbations vary sharply over time under a strict per-frame perturbation constraint. We study white-box, causal, online attacks on such agents and propose Critic-Induced Value-Subspace Attacks (\textbf{CIVA}). Our key observation is that, along a rollout, critic-guided perturbations concentrate in a low-dimensional subspace induced by the victim's own critic. Based on this observation, CIVA first probes the frozen victim offline with critic-guided PGD and extracts a low-rank value-subspace by SVD. At test time, it optimizes only the subspace coefficients, smooths them with an exponential moving average (EMA), and maps them back to pixels. This design attacks value-sensitive recurrent dynamics while keeping the online optimization cheap and temporally coherent. Extensive experiments on DMC walker walk, Atari Pong, and Crafter show that CIVA consistently outperforms five recent methods; on DMC walker walk, it achieves the largest reward drop of 26.07\% while keeping temporal variation low, with TempAbs of 0.646.

1 Introduction

Visual world-model agents act through recurrent latent dynamics, making conventional frame-wise attacks poorly suited to shaping influence over time. CIVA addresses this mismatch with a critic-induced low-dimensional subspace and causal online optimization, achieving stronger degradation across visual control tasks.

  • CIVA yields stronger and more stable degradation than representative observation-space baselines while preserving favorable temporal smoothness and perceptual similarity.
  • Recurrent latent dynamics make attacks that independently degrade individual frames poorly matched to visual world-model agents.
  • CIVA extracts a low-rank, victim-aware perturbation basis offline and performs efficient causal online optimization with temporal smoothing inside that subspace.
  • CIVA evaluates attack effectiveness, action-distribution shift, temporal variation, and perceptual similarity on DMC walker walk, Atari Pong, and Crafter.

2 Related Work

Prior work establishes adversarial vulnerability and develops stronger attacks and defenses for deep RL, but most attacks optimize directly in pixel space and target frames separately. Related low-dimensional and temporally coordinated methods motivate structured attacks, while CIVA focuses on recurrent world-model dynamics.

  • Adversarial attacks on deep policies include robust observation, real-time, universal, policy-aware, detectability-aware, and policy-distribution-based methods.
  • Most prior attacks optimize directly in pixel space and effectively target each frame separately, unlike attacks designed for recurrent latent dynamics.
  • Robust RL primarily studies training agents to remain reliable under perturbations, whereas this paper studies strong causal observation attacks against visual world-model agents.
  • Universal, low-frequency, subspace, and temporally coordinated attacks show that effective perturbations can occupy structured or smoother spaces than raw pixels.

3 Threat Model

The threat model attacks a frozen DreamerV3 agent through current image observations in a white-box, causal, online setting. The design must suppress return under tight compute and perturbation constraints while balancing temporal smoothness and visual naturalness.

  • Victim Model: The victim is a pretrained DreamerV3 policy whose recurrent world model, actor, critic, weights, and environment dynamics remain fixed during attack.
  • Threat Model: The attacker may modify only the current image observation and cannot alter actions, rewards, environment state, or recurrent latent state.
  • Threat Model: The attack is white-box and causal: current and past observations are available, and each perturbation must be produced before the next environment step.
  • Objective: Because environment rewards are nondifferentiable, optimization uses victim-exposed differentiable surrogates rather than true reward gradients.
  • Challenges: The main challenges are recurrent dilution, the cost and jitter of full-pixel per-frame PGD, and the budget–effectiveness–stealth trilemma.

4 Method

CIVA derives a critic-induced value-subspace from offline critic-guided PGD perturbations, then performs online attack optimization in its low-dimensional coefficients. EMA smoothing, pixel-space projection, and causal deployment provide temporal coherence, validity, and efficiency.

  • 4.1 Offline Value-Subspace Discovery: CIVA uses offline critic-guided PGD probes and SVD to construct a low-dimensional value-subspace from the victim critic’s attack gradients.
  • 4.1 Offline Value-Subspace Discovery: Each stacked perturbation direction empirically drives the victim critic toward lower predicted value rather than representing an observation or hand-crafted prior.
  • 4.1 Offline Value-Subspace Discovery: The basis Vr contains the top-r singular vectors and is frozen offline; with r = 32 ≪ D = 12288, online search uses a shared victim-aware direction set.
  • 4.2 Online Subspace Attack: Online PGD optimizes subspace coefficients instead of full pixel coordinates, while lifting, projecting onto the ℓ∞ ball and pixel range, and re-projecting back to coefficients.
  • 4.2 Online Subspace Attack: EMA smoothing trades strength for temporal smoothness as ρ increases, while smoothing within the critic-effective subspace preserves attack-direction alignment.
  • 4.2 Online Subspace Attack: The causal deployment procedure warm-starts from the previous post-EMA coefficients, uses a few subspace-PGD steps, and feeds the perturbed observation to the victim.

5 Experiment

Experiments evaluate CIVA against five observation-space attackers across DreamerV3 tasks, measuring reward suppression, behavioural deviation, temporal coherence, visual stealth, and ablation effects. CIVA achieves strong attack effectiveness while maintaining low temporal variation, with critic-induced alignment and EMA contributing substantially.

  • Experimental setup: Experiments cover DMC walker walk, Atari Pong, and Crafter using frozen DreamerV3 victims, a shared ℓ∞ per-frame budget of 24/255, and five baselines.Evaluation reports reward, Drop%, Action-KL, TempAbs, and perceptual similarity; Crafter results are deferred to Appendix A.2.
  • Attack effectiveness: 26.07% Drop% is CIVA’s highest result on DMC walker walk, exceeding UAP-RL’s 23.25% while reaching the lowest attacked reward of 703.97.On Atari Pong, CIVA reaches 85.71% Drop%, compared with 66.67% for MAD and below 25% for the other baselines.
  • Behavioural deviation: CIVA produces the strongest reported behavioural deviation, reaching Action-KL values of 1.130 on DMC and 0.184 on Pong.PA-AD reaches a higher DMC Action-KL of 1.35 but only a 17.38% reward drop, showing that KL alone does not determine reward degradation.
  • Temporal coherence and visual stealth: 0.646 and 0.394 are CIVA’s TempAbs values on DMC and Pong, roughly 10× lower than per-frame baselines while preserving strong reward suppression.CIVA also keeps SSIM within 0.04–0.30 of MAD and reduces online attack latency by roughly an order of magnitude through 32-dimensional optimization.
  • Policy decision shifts: CIVA consistently changes the dominant Atari Pong action in representative examples, whereas Illusory can shift logits without changing the selected action.The action-selection changes help connect distributional shifts to reward degradation at decision-critical frames.
  • Ablation study: 6.90% Drop% with a random rank-matched basis and 12.61% with full-pixel critic-PGD show that low rank or critic guidance alone is insufficient.The ablation attributes the gain to aligning a low-rank basis with critic-sensitive directions.
  • Ablation study: 1.7496 versus 0.646 TempAbs shows that removing EMA nearly triples temporal variation, despite a marginal +0.77 percentage-point effectiveness gain.Replacing critic-value gradients with reward-head gradients reduces effectiveness to 4.84%, indicating that smoothness alone does not preserve functional attack strength.

6 Conclusion

CIVA is a critic-induced value-subspace attack for visual world-model agents under tight online and ℓ∞ constraints. Across diverse visual control tasks, it produces stronger performance degradation than prior attacks, while its scope is limited to white-box access, clean rollouts, and a single DreamerV3 backbone.

  • CIVA extracts a low-dimensional attack subspace from the frozen DreamerV3 critic and optimizes only subspace coefficients online with EMA smoothing.
  • CIVA consistently yields stronger performance degradation than prior attacks across diverse visual control tasks.
  • The method highlights the value landscape as a realistic adversarial surface of recurrent world-model agents.
  • CIVA requires a white-box online setting, access to the victim critic and clean rollouts, and evaluation beyond a single DreamerV3 backbone remains unaddressed.

A.1 Detailed Experimental Setup

The appendix details the reproducible CIVA pipeline, from frozen DreamerV3 checkpoints and hardware to offline subspace discovery, online attack evaluation, and metric definitions. Offline critic-guided perturbations are compressed into a fixed rank-32 basis, while deployment optimizes only low-dimensional coefficients.

  • Software and hardware environment: All training and attack experiments run on a single NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB of memory.The online attack optimizes α_t ∈ R^r with r=32 and can co-reside with the frozen victim.
  • DreamerV3 victim training: DreamerV3 checkpoints are trained independently for DMC walker walk, Atari Pong, and Crafter using the official implementation without architectural or loss modifications.The task configurations use 1.1×10^6, 5.1×10^7, and 1.1×10^6 training steps, respectively.
  • DreamerV3 victim training: Victim parameters remain frozen during evaluation, and optimization uses only critic forward passes and gradients.
  • Attack evaluation pipeline: Stage B performs the online low-dimensional PGD attack using the discovered basis during a fresh episode.
  • Attack evaluation pipeline: Stage A rolls out a 200-step warm-up, generates M=600 critic-induced PGD perturbations, and applies truncated SVD to obtain a fixed rank-r=32 basis.The flattened perturbation matrix has D=12,288 dimensions, and the top right singular vectors are saved for evaluation.
  • Metric definitions: The appendix defines episode reward, Drop%, Action-KL, TempAbs, and SSIM for attacked episodes and specifies their task-dependent KL computation.Continuous DMC uses diagonal-Gaussian KL, while discrete Pong and Crafter use KL on softmax-normalised logits; smaller TempAbs indicates closer consecutive perturbations.
  • Metric definitions: SSIM is computed per channel with the default scikit-image implementation using a data range of 255, then averaged.

A.2 Additional Results on Crafter

On Crafter, CIVA matches the strongest baseline in reward suppression while producing comparatively coherent and visually stealthy perturbations. Its critic-induced subspace also yields effective achievement suppression without requiring the largest action-distribution shift.

  • Attack effectiveness: Crafter’s sparse, integer-valued achievement rewards make roughly 30% achievement-event suppression a substantial attack effect.
  • Attack effectiveness: 29.70% Drop% ties UAP-RL as CIVA reduces episode reward from 10.10 to 7.10 on Crafter.This matches the strongest baseline and exceeds MAD, PA-AD, Illusory, and DAPGD.
  • Behavioural deviation: CIVA reaches an Action-KL of 0.326, while DAPGD reaches 1.960 yet achieves only a 9.90% reward drop.The result indicates that larger action-distribution change does not necessarily imply a stronger attack in sparse-reward Crafter.
  • Temporal coherence and visual stealth: 0.27 TempAbs makes CIVA the second-lowest method, while its 0.970 SSIM is at least 0.09 higher than every baseline.UAP-RL has TempAbs 0.00 but uses a fixed perturbation; CIVA remains adaptive per frame.

A.3 Perturbation Visualization

Figure 4 compares perturbation magnitude maps for CIVA and three baselines across six representative Crafter frames. CIVA shows more spatially focused patterns with less scattered high-intensity responses.

  • Each row represents an attack method, and brighter colors indicate larger perturbation intensity across six representative Crafter frames.
  • Compared with MAD, PA-AD, and Illusory, CIVA concentrates perturbations on a smaller set of task-relevant regions.The visual comparison complements the quantitative temporal-coherence and SSIM results in Table 5.

A.4 From Full-Space Attacks to Value-Aligned Subspaces

The appendix defines a shared white-box threat model in which an attacker perturbs each current image observation under an ℓ∞ budget before the next environment step.

  • All baselines and CIVA share a white-box setting with access to the victim policy and critic.The victim policy is πθ and its critic is vψ.
  • At step t, the attacker observes the clean frame ot and outputs δt before the victim receives the clipped perturbed image.The attacked observation is ˜ot = clip(ot + δt, 0, 255).
  • Each perturbation satisfies the per-frame constraint ∥δt∥∞≤ε.The projection ΠBε enforces the ℓ∞ budget.
  • The victim’s latent state is produced recurrently from past observations, so the attack operates on observations while the policy acts through latent dynamics.

A.4.1 MAD: Maximum-Action-Deviation Attack

MAD attacks a victim policy by maximizing the divergence between clean and perturbed action distributions independently for each frame.

  • MAD maximizes the KL divergence between the clean and attacked action distributions.Its objective is LMAD(δ; ot) = KL(πθ(· | ot) ∥πθ(· | ot + δ)).
  • For discrete policies, the objective uses categorical KL on softmax outputs.
  • For Gaussian policies, the action-distribution divergence has a closed-form expression.
  • MAD initializes δ to zero and iteratively applies projected gradient ascent within the ℓ∞ budget.Each iteration computes the gradient of the KL objective and projects δ onto the permitted ball.

A.4.2 UAP-RL: Universal Adversarial Perturbation for RL

The appendix contrasts full-pixel or shared perturbation baselines with CIVA’s critic-aligned subspace, temporal smoothing, and spectrum-based rank selection, while stating assumptions and limitations.

  • A.4.2 UAP-RL: Universal Adversarial Perturbation for RL: UAP-RL learns one perturbation offline from clean observations and reuses it on every frame.Because the same δu is applied throughout a rollout, its temporal variation is zero by construction.
  • A.4.2 UAP-RL: Universal Adversarial Perturbation for RL: The five baselines operate in full pixel space and treat frames independently, except UAP-RL, which sacrifices per-frame adaptivity for temporal consistency.CIVA instead shares a critic-induced subspace across frames and smooths its coefficients over time.
  • A.4.6 CIVA: Critic-Induced Value-Subspace Attack: Critic gradients factor through the visual encoder Jacobian and latent critic gradient, placing their first-order directions in a low-dimensional structure when latent dimension is much smaller than pixel dimension.The factorization is f = Jenc(o)⊤∇sv(s), with s ∈Rds and ds ≪D.
  • A.4.6 CIVA: Critic-Induced Value-Subspace Attack: The top r=32 singular values explain over 95% of spectral energy on every tested task, matching DreamerV3’s latent dimension.The passage attributes this concentration to smoothly changing encoder Jacobians across neighboring frames.
  • A.4.6 CIVA: Critic-Induced Value-Subspace Attack: CIVA’s critic-induced basis captures value sensitivity rather than raw appearance variance, unlike a PCA basis built from observations.
  • A.4.6 CIVA: Critic-Induced Value-Subspace Attack: O(Lv σr+1(∆)2) bounds subspace-attack suboptimality when the singular spectrum decays quickly and test frames remain near the probed distribution.The bound compares the full-pixel optimum with its closest row in the probed perturbation matrix.
Loading 2608.21114v1…