CIVA: Critic-Induced Value-Subspace Attacks on Visual World-Model Agents

• CIVA yields stronger and more stable degradation than representative observation-space baselines while preserving favorable temporal smoothness and perceptual similarity.Source: §1 · p. 2
• Recurrent latent dynamics make attacks that independently degrade individual frames poorly matched to visual world-model agents.Source: §1 · pp. 1–2
• CIVA extracts a low-rank, victim-aware perturbation basis offline and performs efficient causal online optimization with temporal smoothing inside that subspace.Source: §1 · p. 2
• CIVA evaluates attack effectiveness, action-distribution shift, temporal variation, and perceptual similarity on DMC walker walk, Atari Pong, and Crafter.Source: §1 · p. 2

• Adversarial attacks on deep policies include robust observation, real-time, universal, policy-aware, detectability-aware, and policy-distribution-based methods.Source: §2 · p. 2
• Most prior attacks optimize directly in pixel space and effectively target each frame separately, unlike attacks designed for recurrent latent dynamics.Source: §2 · p. 2
• Robust RL primarily studies training agents to remain reliable under perturbations, whereas this paper studies strong causal observation attacks against visual world-model agents.Source: §2 · p. 3
• Universal, low-frequency, subspace, and temporally coordinated attacks show that effective perturbations can occupy structured or smoother spaces than raw pixels.Source: §2 · p. 3

Victim Model

• The victim is a pretrained DreamerV3 policy whose recurrent world model, actor, critic, weights, and environment dynamics remain fixed during attack.Source: §3.1 · p. 3

Threat Model

• The attacker may modify only the current image observation and cannot alter actions, rewards, environment state, or recurrent latent state.Source: §3.2 · p. 3
• The attack is white-box and causal: current and past observations are available, and each perturbation must be produced before the next environment step.Source: §3.2 · p. 3

Objective

• Because environment rewards are nondifferentiable, optimization uses victim-exposed differentiable surrogates rather than true reward gradients.Source: §3.3 · p. 4

Challenges

• The main challenges are recurrent dilution, the cost and jitter of full-pixel per-frame PGD, and the budget–effectiveness–stealth trilemma.Source: §3.4 · p. 4

4.1 Offline Value-Subspace Discovery

• CIVA uses offline critic-guided PGD probes and SVD to construct a low-dimensional value-subspace from the victim critic’s attack gradients.Source: §4 · p. 4, §4.1 · pp. 4–5
• Each stacked perturbation direction empirically drives the victim critic toward lower predicted value rather than representing an observation or hand-crafted prior.Source: §4.1 · pp. 4–5
• The basis Vr contains the top-r singular vectors and is frozen offline; with r = 32 ≪ D = 12288, online search uses a shared victim-aware direction set.Source: §4.1 · p. 5

4.2 Online Subspace Attack

• Online PGD optimizes subspace coefficients instead of full pixel coordinates, while lifting, projecting onto the ℓ∞ ball and pixel range, and re-projecting back to coefficients.Source: §4.2 · p. 6
• EMA smoothing trades strength for temporal smoothness as ρ increases, while smoothing within the critic-effective subspace preserves attack-direction alignment.Source: §4.2 · p. 6
• The causal deployment procedure warm-starts from the previous post-EMA coefficients, uses a few subspace-PGD steps, and feeds the perturbed observation to the victim.Source: §4.2 · p. 6

Experimental setup

• Experiments cover DMC walker walk, Atari Pong, and Crafter using frozen DreamerV3 victims, a shared ℓ∞ per-frame budget of 24/255, and five baselines.Evaluation reports reward, Drop%, Action-KL, TempAbs, and perceptual similarity; Crafter results are deferred to Appendix A.2.Source: §5.1 · pp. 6–7, §5.2 · p. 7

Attack effectiveness

• 26.07% Drop% is CIVA’s highest result on DMC walker walk, exceeding UAP-RL’s 23.25% while reaching the lowest attacked reward of 703.97.On Atari Pong, CIVA reaches 85.71% Drop%, compared with 66.67% for MAD and below 25% for the other baselines.Source: §5.2 · p. 7

Behavioural deviation

• CIVA produces the strongest reported behavioural deviation, reaching Action-KL values of 1.130 on DMC and 0.184 on Pong.PA-AD reaches a higher DMC Action-KL of 1.35 but only a 17.38% reward drop, showing that KL alone does not determine reward degradation.Source: §5.2 · p. 8

Temporal coherence and visual stealth

• 0.646 and 0.394 are CIVA’s TempAbs values on DMC and Pong, roughly 10× lower than per-frame baselines while preserving strong reward suppression.CIVA also keeps SSIM within 0.04–0.30 of MAD and reduces online attack latency by roughly an order of magnitude through 32-dimensional optimization.Source: §5.2 · p. 8

Policy decision shifts

• CIVA consistently changes the dominant Atari Pong action in representative examples, whereas Illusory can shift logits without changing the selected action.The action-selection changes help connect distributional shifts to reward degradation at decision-critical frames.Source: §5.3 · p. 8

Ablation study

• 6.90% Drop% with a random rank-matched basis and 12.61% with full-pixel critic-PGD show that low rank or critic guidance alone is insufficient.The ablation attributes the gain to aligning a low-rank basis with critic-sensitive directions.Source: §5.4 · p. 9
• 1.7496 versus 0.646 TempAbs shows that removing EMA nearly triples temporal variation, despite a marginal +0.77 percentage-point effectiveness gain.Replacing critic-value gradients with reward-head gradients reduces effectiveness to 4.84%, indicating that smoothness alone does not preserve functional attack strength.Source: §5.4 · p. 9

• CIVA extracts a low-dimensional attack subspace from the frozen DreamerV3 critic and optimizes only subspace coefficients online with EMA smoothing.Source: §6 · p. 9
• CIVA consistently yields stronger performance degradation than prior attacks across diverse visual control tasks.Source: §6 · p. 9
• The method highlights the value landscape as a realistic adversarial surface of recurrent world-model agents.Source: §6 · p. 9
• CIVA requires a white-box online setting, access to the victim critic and clean rollouts, and evaluation beyond a single DreamerV3 backbone remains unaddressed.Source: §6 · p. 9

Software and hardware environment

• All training and attack experiments run on a single NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB of memory.The online attack optimizes α_t ∈ R^r with r=32 and can co-reside with the frozen victim.Source: §A.1 · p. 14

DreamerV3 victim training

• DreamerV3 checkpoints are trained independently for DMC walker walk, Atari Pong, and Crafter using the official implementation without architectural or loss modifications.The task configurations use 1.1×10^6, 5.1×10^7, and 1.1×10^6 training steps, respectively.Source: §A.1 · p. 14
• Victim parameters remain frozen during evaluation, and optimization uses only critic forward passes and gradients.Source: §A.1 · p. 14

Attack evaluation pipeline

• Stage B performs the online low-dimensional PGD attack using the discovered basis during a fresh episode.Source: §A.1 · p. 14
• Stage A rolls out a 200-step warm-up, generates M=600 critic-induced PGD perturbations, and applies truncated SVD to obtain a fixed rank-r=32 basis.The flattened perturbation matrix has D=12,288 dimensions, and the top right singular vectors are saved for evaluation.Source: §A.1 · p. 14

Metric definitions

• The appendix defines episode reward, Drop%, Action-KL, TempAbs, and SSIM for attacked episodes and specifies their task-dependent KL computation.Continuous DMC uses diagonal-Gaussian KL, while discrete Pong and Crafter use KL on softmax-normalised logits; smaller TempAbs indicates closer consecutive perturbations.Source: §A.1 · pp. 14–15
• SSIM is computed per channel with the default scikit-image implementation using a data range of 255, then averaged.Source: §A.1 · p. 15

Attack effectiveness

• Crafter’s sparse, integer-valued achievement rewards make roughly 30% achievement-event suppression a substantial attack effect.Source: §A.2 · p. 15
• 29.70% Drop% ties UAP-RL as CIVA reduces episode reward from 10.10 to 7.10 on Crafter.This matches the strongest baseline and exceeds MAD, PA-AD, Illusory, and DAPGD.Source: §A.2 · p. 15

Behavioural deviation

• CIVA reaches an Action-KL of 0.326, while DAPGD reaches 1.960 yet achieves only a 9.90% reward drop.The result indicates that larger action-distribution change does not necessarily imply a stronger attack in sparse-reward Crafter.Source: §A.2 · p. 15

Temporal coherence and visual stealth

• 0.27 TempAbs makes CIVA the second-lowest method, while its 0.970 SSIM is at least 0.09 higher than every baseline.UAP-RL has TempAbs 0.00 but uses a fixed perturbation; CIVA remains adaptive per frame.Source: §A.2 · p. 16

• Each row represents an attack method, and brighter colors indicate larger perturbation intensity across six representative Crafter frames.Source: §A.3 · p. 16
• Compared with MAD, PA-AD, and Illusory, CIVA concentrates perturbations on a smaller set of task-relevant regions.The visual comparison complements the quantitative temporal-coherence and SSIM results in Table 5.Source: §A.3 · p. 16

• All baselines and CIVA share a white-box setting with access to the victim policy and critic.The victim policy is πθ and its critic is vψ.Source: §A.4 · p. 16
• At step t, the attacker observes the clean frame ot and outputs δt before the victim receives the clipped perturbed image.The attacked observation is ˜ot = clip(ot + δt, 0, 255).Source: §A.4 · p. 16
• Each perturbation satisfies the per-frame constraint ∥δt∥∞≤ε.The projection ΠBε enforces the ℓ∞ budget.Source: §A.4 · p. 16
• The victim’s latent state is produced recurrently from past observations, so the attack operates on observations while the policy acts through latent dynamics.Source: §A.4 · p. 16

• MAD maximizes the KL divergence between the clean and attacked action distributions.Its objective is LMAD(δ; ot) = KL(πθ(· | ot) ∥πθ(· | ot + δ)).Source: §A.4.1 · p. 17
• For discrete policies, the objective uses categorical KL on softmax outputs.Source: §A.4.1 · p. 17
• For Gaussian policies, the action-distribution divergence has a closed-form expression.Source: §A.4.1 · p. 17
• MAD initializes δ to zero and iteratively applies projected gradient ascent within the ℓ∞ budget.Each iteration computes the gradient of the KL objective and projects δ onto the permitted ball.Source: §A.4.1 · p. 17

A.4.2 UAP-RL: Universal Adversarial Perturbation for RL

• UAP-RL learns one perturbation offline from clean observations and reuses it on every frame.Because the same δu is applied throughout a rollout, its temporal variation is zero by construction.Source: §A.4.2 · p. 17
• The five baselines operate in full pixel space and treat frames independently, except UAP-RL, which sacrifices per-frame adaptivity for temporal consistency.CIVA instead shares a critic-induced subspace across frames and smooths its coefficients over time.Source: §A.4.5 · p. 18

A.4.6 CIVA: Critic-Induced Value-Subspace Attack

• Critic gradients factor through the visual encoder Jacobian and latent critic gradient, placing their first-order directions in a low-dimensional structure when latent dimension is much smaller than pixel dimension.The factorization is f = Jenc(o)⊤∇sv(s), with s ∈Rds and ds ≪D.Source: §A.4.6 · p. 19
• The top r=32 singular values explain over 95% of spectral energy on every tested task, matching DreamerV3’s latent dimension.The passage attributes this concentration to smoothly changing encoder Jacobians across neighboring frames.Source: §A.4.6 · p. 19
• CIVA’s critic-induced basis captures value sensitivity rather than raw appearance variance, unlike a PCA basis built from observations.Source: §A.4.6 · p. 19
• O(Lv σr+1(∆)2) bounds subspace-attack suboptimality when the singular spectrum decays quickly and test frames remain near the probed distribution.The bound compares the full-pixel optimum with its closest row in the probed perturbation matrix.Source: §A.4.6 · p. 19