PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms

Yueyan Li, Haibo Wang, Caixia Yuan, Xiaojie Wang

arXiv:2609.23658v1cs.CVcs.LG

TL;DR

Video diffusion models often violate physical laws, while prior approaches use external priors or specialized data. This paper interprets motion planning and modifies RoPE frequency across denoising steps; training-free and training-based experiments confirm effectiveness in enhancing physical commonsense.

  • Problem

    Video diffusion models can produce visually strong content that violates physical laws, while the mechanisms underlying motion planning remain unexplored.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes motion planning through cross-attention candidate-region evolution and self-attention, then modifies RoPE frequency scaling across denoising steps.

    Source: §1 · pp. 2–3

  • Results

    Training-free and training-based experiments validate the method's effectiveness in enhancing the physical commonsense of generated videos.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The conclusion states that a RoPE modification using only a scaling factor makes generated video content adhere better to physical laws.

    Source: §7 · p. 9

    Because directly decoding early T2V latents into pixels introduces significant noise, the study investigates denoising dynamics indirectly via cross-attention.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the ''motion planning'' process of text-to-video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the ''first shape, then details'' finding, we combine cross-attention trajectory patterns with causal head contributions to identify a specific subset of attention heads driving motion planning. Further, our self-attention analysis shows that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated videos.

1 INTRODUCTION

The paper traces physically implausible video motion to internal attention mechanisms, studying how motion plans form during denoising. It identifies RoPE-induced spatial attention decay as a failure mechanism and proposes step-dependent RoPE frequency scaling to encourage exploration of candidate positions.

1 INTRODUCTION

  • Existing approaches use external physical priors, prompt rewriting, or specialized data, but the authors argue these methods are difficult to scale or do not address the root cause.Despite advances in visual quality, motion stability, and instruction following, state-of-the-art models still frequently generate content that violates physical laws.Source: §1 · p. 1
  • The paper focuses on solid dynamics, defining motion planning as injecting condition semantics into video latents and allocating them across frames while maintaining inter-frame coherence.It frames physical commonsense as understanding everyday objects and physical laws; solid dynamics offers trackable trajectories for this investigation.Source: §1 · pp. 1–2
  • Cross-attention maps show multiple candidate object regions per frame early in denoising converging toward a deterministic shape around step 5 of 50, while only a small head subset affects motion planning.The study extends the “first shape, then details” finding of Tinaz et al. (2025), Yi et al. (2024), and Wang et al. (2026c); a visible trajectory pattern alone does not establish causal responsibility.Source: §1 · p. 2
  • Self-attention candidates compete early; when some positions stabilize, RoPE-based spatial attention decay can suppress more physically plausible, farther regions in adjacent frames, contributing to physical-law violations.The authors attribute this mechanism to inflexible RoPE-induced attenuation along spatial dimensions, which can make heads repeatedly focus on the same regions across frames.Source: §1 · p. 2
  • The proposed step-dependent scaling of RoPE frequency moderately reduces spatial attention attenuation, encouraging exploration of more candidate regions during motion planning.The authors report that both training-free and training-based experiments validate the approach’s effectiveness.Source: §1 · p. 3
  • The authors present the first interpretability study of text-to-video diffusion motion planning and propose a lightweight RoPE modification validated in training-free and training-based experiments.Their stated contributions also include an analytical framework and toolkit, and a mechanistic account of how video models form shape during early denoising.Source: §1 · p. 3

2 RELATED WORK

Prior work examines diffusion-model interpretability and physics-aware video generation, but leaves early motion planning’s mechanism and architectural causes of physical inconsistency unresolved. This paper addresses the latter with a lightweight RoPE frequency adjustment.

Intepretability for diffusion models

Physics-aware video generation

3 PRELIMINARIES

The paper studies Wan2.1-T2V, a DiT-based video diffusion model, within the Rectified Flow framework. Its preliminaries describe velocity prediction and the model’s video-generation pipeline.

3 PRELIMINARIES

  • Rectified Flow interpolates noise x0 and data latent x1 as xt = (1 − t)x0 + tx1, giving the ground-truth velocity vt = x1 − x0.A neural network conditioned on xt, timestep t, and context c (such as text embeddings) is trained to predict this velocity using mean squared error.Source: §3 · p. 3
  • The studied model is Wan2.1-T2V (Wan et al., 2025), based on the Diffusion Transformer architecture (Peebles & Xie, 2023).A 3D VAE encodes the video into patchified latent tokens, while a text encoder produces the prompt embeddings.Source: §3 · pp. 3–4
  • In the DiT backbone, video tokens pass through bidirectional self-attention, cross-attention, and an FFN before the final layer predicts velocity for flow matching.At inference, an ODE solver integrates predicted velocity over denoising steps, and a 3D VAE decoder reconstructs the video in pixel space.Source: §3 · p. 4

4 CROSS-ATTENTION MECHANISMS

The section examines cross-attention as the module that assigns text semantics to video regions, tracing how object trajectories emerge and identifying which heads causally drive motion planning. It finds that visible trajectory patterns alone do not establish a head’s role: causal contribution distinguishes motion-planning heads from heads affecting static content.

4.1 TEMPORAL EVOLUTION OF CROSS-ATTENTION

  • Cross-attention maps trace object positions from random noise to multiple trajectory regions and then to a converged final position during the first five denoising steps.Because directly decoding early text-to-video latents into pixels introduces significant noise, the study investigates these dynamics indirectly through cross-attention.Source: §4.1 · p. 4
  • Cross-attention entropy and support quality both jump around denoising step 5, as most heads begin focusing on the object trajectory, especially L15H2, L15H5, and L15H0.L15H1 shows why entropy alone is insufficient: it has low entropy but zero support quality, indicating its attention is not aligned with the trajectory.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 ATTENTION HEADS FOR MOTION PLANNING

  • The analysis identifies motion-planning heads by combining convergence speed toward the reference trajectory with causal head contribution measured through attribution patching.Convergence speed is mean support quality over the first 10 denoising steps; patching targets the reference object area and assesses changes in condition-induced semantic increments.Source: §4.2 · p. 5
  • Type (1) and (2) head ablations leave motion basically unaffected, indicating that their contribution mainly concerns static content such as object appearance and background.Even jointly ablating these groups changes object size only somewhat; a large estimated contribution does not necessarily imply a large effect on trajectory.Source: §4.2 · pp. 5–6, Figure 4 · p. 6, Figure 5 · p. 6
  • Ablating Type (3) heads causes the object trajectory to collapse, whereas ablating Type (4) heads leaves the overall trajectory unchanged despite their visible trajectory patterns.Type (3) heads actively introduce motion-related semantic information into object positions; the head classification holds in the absolute majority of tested scenarios and random seeds.Source: §4.2 · pp. 5–6, Figure 4 · p. 6, Figure 5 · p. 6

5 SELF-ATTENTION MECHANISMS

Self-attention links video tokens across frames and drives candidate-region selection, but RoPE’s spatial anchoring can reinforce positions that conflict with plausible motion. The paper traces this failure to unstable early competition and later confidence shifts between candidates.

5 SELF-ATTENTION MECHANISMS

  • Self-attention communicates among video tokens and determines entity positions across frames, directly driving cross-attention patterns to converge on a trajectory.Cross-attention conveys what content to generate, while self-attention arranges it across frames.Source: §5 · p. 6
  • RoPE’s long-range decay makes self-attention favor spatially nearby regions across frames, creating spatial anchoring that can add highlighted regions outside a trajectory.Attention from a region often concentrates on the same position in other frames; this anchoring can persist into middle and late denoising.Source: §5 · p. 7
  • Mutual consistency measures confidence in two candidates’ reciprocal selection, and its correlation with anchor distance strengthens as denoising progresses.Higher mutual consistency indicates a candidate is more likely to win.Source: §5 · pp. 7–8
  • Early candidate competition is unstable, and later spatial anchoring can let a less plausible region overtake a physically reasonable candidate.In one failed basketball case, frame 14’s K1 led at step 2 (0.21), but K3 surpassed it at step 3 (0.33) as anchoring from frame 10’s stabilized K1 raised confidence in adjacent K3, leaving the ball stationary in the air.Source: §5 · p. 8

6 ROPE MODIFICATION

The section weakens RoPE’s spatial decay to let video diffusion models explore more candidate motion regions, then evaluates the change in training-free and training-based settings. The authors report stronger results for training-based methods and additional gains from combining RoPE modification with prompt refinement.

6.1 METHOD

  • The method applies height and width modulation factors λh, λw < 1.0 to RoPE, reducing rotation angles and spatial decay to encourage exploration of candidate motion regions.The modification is applied to queries and keys in self-attention.Source: §6.1 · p. 8

6.3 MAIN RESULTS & ANALYSIS

  • Training-based results generally outperform training-free and other baselines on instruction following and physical consistency, with the main advantage in the solid-* subset.The authors attribute the solid-* advantage to its larger-magnitude motions, which are the focus of the method.Source: Table 1 · p. 9, §6.3 · p. 9

6.2 IMPLEMENTATION DETAILS

  • Inference applies the modification during the first 5 denoising steps; training emphasizes the early 10% of timesteps and gradually increases λh/w to 0.75.Training fine-tunes with LoRA using batch size 32 and learning rate 1e-4.Source: §6.2 · p. 9

6.3 MAIN RESULTS & ANALYSIS

  • Prompt refinement’s physical-consistency gains mainly stem from instruction following; it combines with this method for further gains, while the method generalizes across random seeds and attention-only LoRA tuning preserves aesthetics.The authors report that attention-only tuning outperforms tuning other modules, supporting their view that physical failures stem more from attention than FFN.Source: §6.3 · p. 9

7 LIMITATIONS AND DISCUSSIONS … E.2 ZERO ABLATION OF CROSS-ATTENTION HEADS

The paper’s interpretability analysis identifies a self-attention flaw during denoising and proposes a RoPE modification that improves physical consistency with only a scaling factor. The authors leave extensions to I2V and few-step autoregressive diffusion models for future work.

7 LIMITATIONS AND DISCUSSIONS

  • The RoPE modification improves generated videos’ adherence to physical laws using only a scaling factor; extensions to I2V and few-step autoregressive diffusion models remain future work.The conclusion frames this as the paper’s central result and proposed future direction.Source: §7 · p. 9

AI USE STATEMENT; ETHICS STATEMENT; REPRODUCIBILITY STATEMENT

  • The authors report AI assistance for hypothesis development, code and readability edits, and state that experiments used open-source models and public benchmarks, with voluntary researcher evaluation and no personal data.They say AI-proposed hypotheses, code, and polished text were reviewed, and affirm Code of Ethics compliance.Source: AI USE STATEMENT · p. 10, ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A NOTATIONS; B MODEL DETAILS

  • Appendix B specifies Wan2.1-T2V-1.3B’s DiT model details, while Appendix A provides a notation summary.The model-details passages describe the patchified latent token grid, text conditioning, attention layers, and denoising output.Source: Table 2 · p. 14, B MODEL DETAILS · pp. 14–15

C EXAMPLES OF CROSS-ATTENTION MAPS

  • The appendix figures show cross-attention head L15H2 evolving during denoising, L15H1 retaining a consistent pattern, and example heads selected at step 7.Figure 10 notes that cross-attention maps expose motion-planning patterns earlier than decoded T2V videos, whose early denoising outputs contain substantial noise.Source: Figure 10 · p. 16, Figure 11 · p. 16, Figure 12 · p. 16

D PROCESSING OF CROSS / SELF-ATTENTION

  • The appendix algorithms construct frame-wise reference support masks and extract candidate regions through preprocessing, peak proposals, clustering, and pruning.The candidate-region procedure uses multiple seed levels and merges qualifying regions overlapping a reference box.Source: D PROCESSING OF CROSS / SELF-ATTENTION · pp. 17–18

E.1 ATTRIBUTION PATCHING FOR MOTION PLANNING

  • Attribution patching scores motion-planning nodes across the first five denoising steps, focusing on object regions and comparing intervened conditional velocity with the clean conditional–unconditional velocity difference.The dot product is used because that reference difference retains more object-semantic information while removing general information in the unconditional velocity.Source: §E.1 · p. 19
  • Attribution patching with mean ablation computes all node scores in one forward and backward pass, reducing time complexity from O(|V|) to O(1).It approximates causal contribution with a first-order Taylor expansion.Source: §E.1 · pp. 19–20

E.2 ZERO ABLATION OF CROSS-ATTENTION HEADS

  • Zero ablation evaluates a selected subset of cross-attention heads by zeroing their per-head writes and observing generation changes, by default across every denoising step.The per-head write represents the actual direction and magnitude contributed to the residual stream.Source: §E.2 · p. 20

E.3 EXPERIMENT DETAILS · F ROPE DETAILS

Additional visualizations show that the cross-attention-head findings hold across cases and seeds; the appendix also explains RoPE’s positional encoding and how its spatial decay is visualized.

E.3 EXPERIMENT DETAILS

  • Cross-attention-head conclusions from Section 4.2 also hold across different cases and random seeds, with additional zero-ablation videos shown for two prompts.The examples use random seeds 2 and 8.Source: §E.3 · p. 20, Figure 14 · p. 21, Figure 15 · p. 21

F ROPE DETAILS

  • In 1D RoPE, positional information is added to query-key dimension pairs through rotations, with frequency θ_i = b^-2i/D.The appendix illustrates the transformation using a two-dimensional query and key.Source: F ROPE DETAILS · p. 22
  • The query-key dot product is the real part of the transformed query multiplied by the conjugate of the transformed key.This is the stated complex-form interpretation of the RoPE dot product.Source: F ROPE DETAILS · p. 22
  • 3D-RoPE divides the model dimension into frame, height, and width groups to encode positional information along each axis.For each axis, dimension-pair frequencies are specified using base 10000; the original model uses the same base frequency across axes.Source: F ROPE DETAILS · pp. 22–23
  • The spatial-decay visualization isolates RoPE-dependent cosine factors, averages them across dimension pairs, and fixes both tokens to the same frame.Figure 16 shows the resulting spatial center heatmap for Wan2.1-T2V-1.3B.Source: F ROPE DETAILS · p. 23, Figure 16 · p. 23
  • The spatial decay curve averages heatmap values over positions grouped by rounded distance from the center token.The curve is derived from the spatial heatmap shown in Figure 16b.Source: F ROPE DETAILS · p. 23

G MORE RESULTS ON SELF-ATTENTION MECHANISMS … H.1.3 PARAMETERIZATION OF λh/w FOR ROPE MODIFICATION

The supplementary analyses show that candidate motion positions remain unstable early in denoising before separating later, and that fine-tuning attention rather than FFNs better preserves object appearance while improving motion. Learnable RoPE scaling did not significantly improve physical commonsense, motivating fixed early-stage RoPE modification.

G MORE RESULTS ON SELF-ATTENTION MECHANISMS

  • At denoising step 3, winners and other candidate regions overlap in mutual consistency, but by step 10 winners and losers form distinct clusters.This supports mutual consistency as a way to distinguish winners from losers once motion planning is complete.Source: G MORE RESULTS ON SELF-ATTENTION… · p. 24, Figure 18 · p. 24
  • In seed 26, K1’s mutual consistency was 0.44 versus K2’s 0.37 at step 3, but at step 4 K2 led 0.62 to 0.34 as K1 shrank.K1 was the strongest loser, illustrating the instability of candidate trajectories during motion planning.Source: G MORE RESULTS ON SELF-ATTENTION… · p. 24, Figure 17 · p. 24

H.1.1 GENERAL SETTINGS

  • LoRA training used batch size 32, learning rate 1e-4, and 1,500 default training steps; evaluation selected the 800-step checkpoint, where performance peaked.Training used Adam on 4 × A800 80G GPUs; VideoREPA used VideoMAEv2 as its alignment target encoder at depth 18.Source: §H.1.1 · p. 26

H.1.2 LORA CONFIGURATIONS

  • Fine-tuning FFNs alongside attention did not improve motion trajectories and distorted object shapes, whereas attention-only updates improved inter-frame interactions without impairing shapes.The authors attribute this difference to FFNs’ role in filtering hidden-state features and governing object appearance.Source: §H.1.2 · p. 26, Figure 20 · p. 26

H.1.3 PARAMETERIZATION OF λh/w FOR ROPE MODIFICATION

  • Fixed λh/w applies a selected λ during the initial 10% of denoising; after testing values from 0.55 to 0.90, the authors chose 0.75.The tested values were evaluated across multiple random seeds on the “basketball free fall” case and examples from VideoPhy’s training set.Source: §H.1.3 · p. 27
  • Learnable λh/w produced values below 1 for approximately half the self-attention heads and above 1 for the rest, but yielded no significant physical-commonsense improvement.The method uses head-specific, timestep-conditioned scaling factors; the reported λh/w trends were sampled at denoising timestep 900 of 1,000.Source: §H.1.3 · p. 27, Figure 21 · p. 27
  • Because motion occupies a minor share of predicted frame information, flow-matching updates may be dominated by non-motion gradients, so the authors argue early RoPE modification must force exploration of candidate positions.This motivates enforcing exploration rather than relying on the model to learn motion patterns through adjustable RoPE frequencies alone.Source: §H.1.3 · p. 27

H.1.4 CUSTOM TIMESTEP SAMPLER

Because motion planning occurs in the initial 10% of denoising, the sampler concentrates training there while retaining samples from later steps. The authors select p_early = 0.9 to adapt to modified RoPE frequencies without losing mid-to-late denoising capability.

H.1.4 CUSTOM TIMESTEP SAMPLER

  • The sampler selects the initial 10% of denoising steps with probability p_early and the remaining steps with probability 1 − p_early, setting p_early to 0.9.The tested values were 0.3, 0.5, 0.7, 0.9, and 1.0.Source: §H.1.4 · p. 28
  • Setting p_early excessively high degrades video aesthetics because too few mid-to-late samples remain to regularize denoising.Without those samples, the model can lose its denoising capability for mid-to-late stages.Source: §H.1.4 · p. 28
  • Setting p_early less than or equal to 0.5 distorts object motion trajectories, likely because too few early-stage samples use the RoPE modification.The authors suggest this can leave the model insufficiently trained, with modified-RoPE samples acting instead as injected noise.Source: §H.1.4 · p. 28

H.2 DATA PROCESSING AND EVALUATION DETAILS · H.3 VISUALIZATIONS

The appendix details how WISA training videos are filtered and how RoPE variants are evaluated, including human assessment on VideoPhy.

H.2 DATA PROCESSING AND EVALUATION DETAILS

  • Training uses WISA, an 80,000-video dataset with annotations of physical descriptions, qualitative categories, and quantitative properties across 17 physical laws and three physics domains.The domains are dynamics, thermodynamics, and optics.Source: §H.2 · p. 28
  • Filtering retains videos longer than 2.0 seconds, samples up to 81 frames from the first 5 seconds, and applies motion thresholds for dynamics plus a 4,000-video reflection cap.Videos with fewer than 81 sampled frames are discarded; dynamics samples require motion score > 0.10 and 0.01 < motion score v2 < 6.50, and samples labeled “no obvious dynamic phenomenon” are removed.Source: §H.2 · p. 28
  • After filtering, approximately 48,000 high-quality videos remain for training.The filtering supplements earlier rounds performed by the WISA authors.Source: §H.2 · p. 28
  • For training-free RoPE modification, λh/w is manually tuned from 0.50 to 0.90; fine-tuning then uses 0.70 at evaluation despite a default training value of 0.75.The paper notes training-free tuning does not scale in practice and describes 0.70 as empirically suitable at test time.Source: §H.2 · p. 28
  • VideoPhy evaluation uses 343 prompts spanning solid-solid, solid-fluid, and fluid-fluid interactions, refined with PhyT2V to describe objects, scenes, and motion.All VideoPhy videos use random seed 42; the RoPE modification uses λh/w = 0.70 in both training-free and training-based settings.Source: §H.2 · pp. 28–29
  • Because automated VideoPhy judges showed hallucinations and weak correlation with human ratings, blind human evaluators score Semantic Adherence and Physical Commonsense.Both metrics are binary; evaluators also flag noticeable aesthetic degradation for ablation studies.Source: §H.2 · p. 29

H.3.1 RESULTS ON THE BASKETBALL FALLING CASE

For the basketball-falling prompt, the captions report better results from the training-based method than the training-free method. The training-free method requires case-specific tuning and can produce unstable motion trajectories and object permanence.

H.3.1 RESULTS ON THE BASKETBALL FALLING CASE

  • The training-free method requires manually tuning λh/w for each case and often shows instability in motion trajectories and object permanence.Source: Figure 22 · p. 29
  • The training-based method produces better results than the training-free method when modified RoPE is applied during the first five denoising steps.Inference uses λh = λw = 0.70 on a model trained with modified RoPE.Source: Figure 23 · p. 30

H.3.2 RESULTS ON SEVERAL SAMPLES FROM VIDEOPHY

This section presents selected VideoPhy video samples across Wan2.1 model variants, covering everyday physical interactions and motion scenarios. The examples include comparisons with modified RoPE and LoRA, including a 14B model configuration.

H.3.2 RESULTS ON SEVERAL SAMPLES FROM VIDEOPHY

  • For Wan2.1-T2V-1.3B, the cork-twisting and diver-plunging examples show samples for the base model, modified RoPE, and LoRA plus modified RoPE.The prompts are “Cork being twisted out of a bottle” and “A diver plunges headlong into a sparkling pool.”Source: Figure 24 · p. 31, Figure 25 · p. 31, §H.3.2 · p. 31
  • Additional VideoPhy prompts depict a pencil rolling, a whisk mixing an egg, a tyre splashing through a puddle, a log floating downstream, a refrigerator door closing, and wine pouring.These are sample prompts; the supplied passages do not state a qualitative winner among model variants.Source: Figure 26 · p. 31, Figure 27 · p. 32, Figure 28 · p. 32
  • A bottle-toppling prompt is followed by Wan2.1-T2V-14B videos comparing the base model (top) with modified RoPE (bottom), using λh = λw = 0.70.The prompts depict stirring sugar into coffee, flipping a pancake, a diver plunging into a swift river, and a car gliding over a rain-slick road.Source: Figure 32 · p. 33, §H.3.2 · p. 34, Figure 33 · p. 34
  • The 14B sample set also includes a brick falling onto another brick.The prompt is “Brick falling onto another brick.”Source: Figure 37 · p. 34

Related papers

Loading 2609.23658v1…