Source-linked AI summary
When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models
Zhixiang Guo, Siyuan Liang, Andras Balogh, Noah Lunberry, Rong-Cheng Tu, Mark Jelasity, Dacheng Tao
TL;DR
Generative world models use physical priors to produce controllable driving videos, creating a largely unexamined attack surface. PhysCond-WMA perturbs physical-condition channels through quality-constrained and momentum-guided denoising. It reaches targeted ASR 0.55 while causing modest FID/FVD increases and measurable downstream detection and planning degradation.
Problem
Generative world models' reliance on physical priors exposes attack surfaces, while security risks in this model class have received limited study.
Method
PhysCond-WMA perturbs physical-condition channels using quality-preserving guidance followed by momentum-guided denoising optimization.
Results
Targeted attack success rate reaches 0.55, with average FID increasing about 9% and FVD about 3.9%.
Takeaways & Limitations
Attacked videos reduce 3D detection performance by about 4% and open-loop planning performance by about 20%.
Takeaways & Limitations
The attack has been verified with image targets, while multimodal targets such as video and text remain unexplored.
Abstract
from arXiv · showhide
Generative world models (WMs) are increasingly used to synthesize controllable, sensor-conditioned driving videos, yet their reliance on physical priors exposes novel attack surfaces. In this paper, we present Physical-Conditioned World Model Attack (PhysCond-WMA), the first white-box world model attack that perturbs physical-condition channels, such as HDMap embeddings and 3D-box features, to induce semantic, logic, or decision-level distortion while preserving perceptual fidelity. PhysCond-WMA is optimized in two stages: (1) a quality-preserving guidance stage that constrains reverse-diffusion loss below a calibrated threshold, and (2) a momentum-guided denoising stage that accumulates target-aligned gradients along the denoising trajectory for stable, temporally coherent semantic shifts. Extensive experimental results demonstrate that our approach remains effective while increasing FID by about 9% on average and FVD by about 3.9% on average. Under the targeted attack setting, the attack success rate (ASR) reaches 0.55. Downstream studies further show tangible risk, which using attacked videos for training decreases 3D detection performance by about 4%, and worsens open-loop planning performance by about 20%. These findings has for the first time revealed and quantified security vulnerabilities in generative world models, driving more comprehensive security checkers.
1. Introduction
Generative world models support controllable autonomous-driving simulation but expose physical-condition attack surfaces. PhysCond-WMA perturbs these channels to induce semantic and downstream distortions while preserving perceptual quality.
- World models learn structured environment representations that support prediction, counterfactual evaluation, and planning with less real-world interaction.
- Generative world models produce temporally coherent, sensor-conditioned urban-driving videos for scalable corner-case generation and long-horizon testing.
- PhysCond-WMA manipulates physical-condition channels, including HDMap embeddings and 3D-box features, to induce semantic, logic, or decision-level distortion while preserving image quality.
- The method combines quality-preserving guidance with momentum-guided denoising to inject small, target-aligned perturbations that remain coherent through generation.
- FID increases by about 9% on average and FVD by about 3.9% on average, while targeted attack success rate reaches 0.55.
- Attacked videos reduce downstream 3D detection performance by about 4% and open-loop planning performance by about 20%.
2. Related work
Related work positions generative driving world models around physical conditioning and diffusion-based generation. Existing diffusion attacks motivate attacking reverse diffusion while highlighting the challenge of preserving generated-video quality.
- Generative driving world models use HD maps, BEV layouts, 3D boxes, or trajectories to synthesize controllable, cross-view-consistent videos.
- Sequence modeling and latent diffusion extend world models toward unified video, action, and language control with broader geographic and multimodal capabilities.
- Diffusion-model attacks inject adversarial guidance into reverse diffusion to generate realistic unrestricted or targeted adversarial examples.
- AdvDiffuser applies PGD during reverse diffusion but significantly reduces generated-result quality, whereas momentum-aligned denoising supports efficient targeted attacks.
3. Preliminaries
The paper formulates conditional video generation around physical priors and multimodal inputs, then defines an inference-time attack that perturbs physical conditions. Its objective balances perturbation magnitude, fidelity, and target alignment.
- Generative World Model: Conditional diffusion injects physical priors such as HDMaps, 3D bounding boxes, and BEV representations to support coherent perception and decision-making.
- Generative World Model: HDMaps encode road geometry and traffic structure, while 3D boxes encode object position, size, orientation, and often velocity.
- Generative World Model: The diffusion model uses forward noise addition and conditional reverse denoising, with reverse generation conditioned on physical priors R and inputs C.
- Generative World Model: Video diffusion adds temporal attention, 3D spatiotemporal U-Nets, and cascaded super-resolution to maintain long-range consistency.
- Problem Definition: The white-box inference-time attacker accesses pretrained weights and gradients but cannot train or fine-tune model parameters.
- Problem Definition: The attack applies a subtle perturbation δ to physical conditions so generated semantics change directionally or non-directionally while output quality is maintained.
- Problem Definition: The optimization controls perturbation magnitude, preserves perceptual and geometric fidelity, and aligns the denoising trajectory with a target.
4. Approach
PhysCond-WMA is a two-stage attack that perturbs physical-condition channels during diffusion while preserving perceptual fidelity and steering generated videos toward semantic targets. Its pipeline combines threshold-controlled guidance with momentum-based denoising optimization for stable, temporally coherent distortions.
- Overall Framework: PhysCond-WMA injects compact perturbations into the diffusion process through physical-condition channels, using a quality-preserving guidance stage followed by momentum-guided denoising optimization.The framework operates in latent video space while conditioning on physical priors and target contexts.
- Quality-preserving Guidance: The quality-preserving stage adds small corrective perturbations while constraining diffusion loss with a threshold τ to keep the trajectory near the data manifold.The threshold is calibrated because overly large values degrade image quality, whereas overly small values hinder target convergence.
- Quality-preserving Guidance: The first stage terminates at the first reverse-diffusion step that satisfies the quality-preserving fidelity criterion, providing a stable starting point for target alignment.The stopping threshold τ is determined through ablation experiments.
- Denoising Optimization: The denoising stage uses target-aligned gradients and semantic consistency guidance to steer the reverse-diffusion trajectory toward desired semantics.SSCD quantifies semantic consistency between the denoised result and the target representation.
- Denoising Optimization: EMA aggregates target-directed gradients across denoising steps, suppressing short-term fluctuations while preserving visual quality and spatio-temporal coherence.The momentum factor λ lies in (0, 1), and the semantic alignment model A is SSCD; untargeted perturbations use a corresponding variant.
5. Experiments
Experiments evaluate PhysCond-WMA on generative driving world models, measuring attack effectiveness, perceptual quality, downstream degradation, and component contributions. Results show that the attack preserves visual plausibility while producing semantic shifts and harming detection and planning.
- Experimental Setup: Experiments use nuScenes and evaluate PhysCond-WMA on DriveDreamer and DriveDreamer2 with attack, quality, and downstream-task metrics.The setup includes 850 driving videos and evaluates generated results using FID, FVD, ASR, 3D detection, and open-loop planning.
- Main Results: PhysCond-WMA maintains visual quality while producing semantic and logic changes, including altered vehicles, traffic signals, and driving behavior.Human and GPT-5 assess attack success across semantic, logical, and decision levels, with human ASR often exceeding GPT’s.
- Main Results: Targeted attacks achieve higher success by guiding reverse diffusion toward target directions while preserving temporally accumulated semantic shifts.The reported targeted attack success rate reaches 0.55, and targeted attacks improve ASR by about 60% over untargeted attacks.
- Main Results: Attacked videos degrade downstream performance when used for training, reducing 3D detection scores and increasing open-loop planning errors and collision rates.Using 800 frames from 100 attacked videos lowers mAP and NDS across models; planning L2 error rises from 0.29 to 0.33 and collision rate from 0.15% to 0.19%.
- Ablation Study: The two attack stages are complementary: the first preserves and shapes the denoising trajectory, while the second accumulates target-aligned semantic changes.Removing the quality-preserving stage increases FVD, while removing the momentum-guided stage sharply reduces ASR.
- Ablation Study: Jointly perturbing HDMap and 3D-box conditions achieves higher ASR than attacking either physical-condition channel alone.HDMap perturbations affect road topology, whereas 3D-box perturbations alter object semantics; joint perturbations combine these effects.
6. Conclusions and Future Work
The paper presents PhysCond-WMA as a white-box attack that perturbs physical-condition channels to induce temporally coherent semantic shifts while preserving perceptual quality. It concludes that these perturbations expose vulnerabilities in generative world models and motivates broader safety evaluation.
- PhysCond-WMA perturbs physical-condition channels to induce temporally coherent semantic shifts while preserving perceptual quality.
- The experiments confirm that conditional perturbations can successfully impact downstream tasks.
- The paper calls for a multi-dimension safety checker covering perceptual, temporal, physics, and policy compliance dimensions.
- The attack has been verified with image targets, while its effect on multimodal targets such as video and text remains unexplored.
- The evaluation methodology is not a complete world-model safety criterion because unified safety standards are lacking.
against World Models
Algorithm 1 initializes an attacked latent video under physical and target conditions, applies quality-preserving updates, then switches to denoising optimization before decoding the attacked video.
- The algorithm encodes the clean video with physical conditions and target context, then adds diffusion noise to initialize the attacked latent.
- Quality-Preserving Stage: During early reverse diffusion steps, the quality-preserving stage operates while t exceeds the transition threshold Δ.
- Stage Transition: When the loss L falls below threshold τ, the algorithm switches to denoising optimization.
- Denoising Optimization Stage: The later loop blends denoising predictions with coefficient λ before decoding the final attacked video.
7. Algorithm of PhysCond-WMA
The paper provides Algorithm 1 as the targeted PhysCond-WMA procedure.
- Algorithm 1 specifies the targeted PhysCond-WMA algorithm.
8. Target Generation
Target generation uses SDXL inpainting to modify selected regions of driving-scene images while preserving unmasked content. The examples include inserted traffic signage and vehicles aligned with scene context.
- SDXL inpainting uses a clean frame, binary mask, and text prompt to regenerate only the masked region while keeping unmasked content fixed.
- Figure 8 presents the generated target examples.
- One target adds a large yellow traffic warning sign while preserving pedestrians and the background.
- Another target adds a silver sedan aligned with lane markings while preserving surrounding scene elements.
9. ASR Evaluation
The paper evaluates attack success with both GPT-based and human assessments. Raters score generated-video views for semantic, logic, and decision-level distortions using a defined 0–1 success criterion.
- Two ASR evaluation methods are introduced: GPT-based assessment and human assessment.
- Raters inspect all generated-video frames, including multi-views, and score semantic, logic, and decision-level quality or safety.The decision axis covers responses to obstacles, pedestrians, slow or lead vehicles, and red lights.
- Scores use discrete values from 0 to 1, where 0 is best and 1 is worst, with axis averages computed across six views.The overall score is S = (¯ssem + ¯slog + ¯sdec)/3.
- An attack is successful when the overall score S exceeds 0.5.
- Manual evaluation recruits 20 adult volunteers and provides definitions and examples for each attack-success level before assessment.
10. Results
The results section presents additional attacked examples and the evaluation materials used for GPT and human ASR assessment. Reported examples include incorrect motion, semantics, and driving decisions.
- Additional attacked results are provided, with examples spanning motion, semantic, and decision errors.Examples include incorrect waiting speed at a red light, a bus changing into a truck, and failure to slow at an intersection.
- The GPT-based ASR evaluator uses a dedicated prompt shown in Figure 9.
- The human ASR evaluator uses a survey questionnaire shown in Figure 10.