Source-linked AI summary
Tempered Self-Similarity Alignment for Physically Plausible Video Generation
Manjin Kim, Suha Kwak, Minsu Cho
TL;DR
Video generators still struggle with physical realism, including appearance drift, implausible deformations, and inconsistent motion. TSA transfers spatio-temporal self-similarity from visual foundation models by aligning sharpened correspondence distributions on dynamic regions. It improves physical plausibility on VideoPhy and VideoPhy2.
Problem
Video generators still exhibit appearance drift, unrealistic deformations, and inconsistent motion, limiting their applicability where accurate motion simulation is needed.
Source: §1 · p. 1Method
TSA converts spatio-temporal self-similarity into sharpened correspondence distributions and aligns foundation-model and generator distributions on dynamically changing regions.
Source: §1 · p. 2Results
TSA improves physical plausibility on VideoPhy and VideoPhy2; on VideoPhy, M-TSA reaches 30.8% overall PC versus PhyT2V’s 29.0%.
Source: §5.3 · p. 6, §6 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
Despite remarkable advances in video generative models, they still struggle to generate physically realistic videos, frequently exhibiting appearance drift, implausible motion, and temporal inconsistencies. In this work, we address this limitation by transferring relational knowledge encoded in spatio-temporal self-similarity (STSS) from visual foundation models into video generative models. STSS represents pairwise similarities among features across space and time, revealing the relational structure of how objects interact with other entities throughout a video, effectively capturing real-world dynamics, including object motion and semantic transformations. To transfer this relational knowledge, we propose Tempered Self-similarity Alignment (TSA) loss, which transforms STSS into probabilistic correspondence distributions and trains the video generative model to align its correspondence distributions with those of the visual foundation model on dynamically changing regions. Evaluated on VideoPhy and VideoPhy2 benchmarks, our method demonstrates substantial improvements in physical plausibility across diverse interaction scenarios, validating the effectiveness of transferring relational knowledge for physically realistic video generation.
1. Introduction
Video generators still struggle with physical plausibility, limiting their use in applications requiring accurate motion simulation. The paper transfers spatio-temporal self-similarity from visual foundation models through sharpened correspondence distributions aligned on dynamic regions.
1. Introduction
- Video generators frequently exhibit appearance drift, unrealistic deformations, and inconsistent motion, limiting applications requiring accurate motion simulation.Examples include robotics training, virtual environment construction, and video prediction.Source: §1 · p. 1
- Simulator-based, MLLM-guided, and optical-flow approaches face limits in scalability, latency or reasoning capability, and long-range motion coverage, respectively.Optical flow primarily captures short-term displacements between adjacent frames, making long-range dynamics and structural changes difficult to supervise.Source: §1 · p. 1
- Spatio-temporal self-similarity captures relational patterns among visual entities while suppressing photometric variations such as color and texture.These patterns encode how objects move, deform, and interact with surrounding entities across a video.Source: §1 · p. 2
- TSA sharpens self-similarity into correspondence probability distributions, aligns foundation-model and generator correspondences on dynamic regions, and represents motion as soft trajectories across a video.This captures long-range dynamics and uncertainty in structural transformations, unlike adjacent-frame one-hot optical-flow correspondences.Source: §1 · p. 2
- TSA significantly improves physical plausibility on VideoPhy and VideoPhy2 and achieves state-of-the-art performance in generating physically realistic videos.The method transfers relational knowledge in spatio-temporal self-similarity from visual foundation models into video diffusion models.Source: §1 · p. 2
2. Related Work
Prior work improves physical realism through simulation, language-model reasoning, optical flow, or representation alignment, while spatio-temporal self-similarity has mainly served video understanding. This paper extends its use to video generation and sharpens correspondence alignment on dynamic regions.
2. Related Work
- Physics simulators, LLM reasoning, and optical flow have been used to improve physical realism, while representation alignment transfers physical knowledge from visual foundation models to video diffusion models.The approaches have different constraints: simulator methods may be domain-restricted, LLM-based methods may add latency, and optical flow focuses on motion cues.Source: §2 · p. 2
- Earlier STSS methods capture motion or support video understanding, whereas this work aligns foundation-model STSS with video diffusion models for generation.Prior uses include action recognition, short-term motion cues, motion-feature learning, and motion-centric representations.Source: §2 · p. 2
- VideoREPA distills STSS but uniformly aligns raw tensors; this method instead aligns sharpened correspondence distributions on dynamic regions.The paper presents this as more precise motion supervision and more focused motion learning than raw-STSS or uniform spatial alignment.Source: §2 · p. 3
3. Preliminaries
The preliminaries describe latent video diffusion and feature-based representation alignment. These provide the generation and alignment context for the paper’s approach.
3.1. Video Diffusion Models
- Video diffusion encodes a video into lower-resolution latents, corrupts them with Gaussian noise, and trains a denoiser to estimate the added noise.At inference, generation starts from pure noise, iteratively refines the latent, and decodes it into a video.Source: §3.1 · p. 3
3.2. Representation Alignment
- REPA projects denoiser activations into a pretrained visual encoder’s feature space and encourages featurewise consistency between corresponding patches.Its regularizer is combined with the diffusion loss; this paper instead aligns probabilistic distributions of spatio-temporal correspondences derived from STSS.Source: §3.2 · p. 3
4. Methods
The methods transfer spatio-temporal self-similarity (STSS) from a visual foundation model to a video diffusion model by aligning probabilistic correspondence distributions. They sharpen correspondences with temperature scaling and restrict alignment to motion-salient regions.
4.1. Tempered Self-Similarity Alignment
- TSA converts each query token’s STSS into frame-wise spatial correspondence distributions and aligns the foundation-model and diffusion-model distributions with KL divergence.Source: §4 · p. 3, §4.1 · p. 4
- The method projects denoiser features into the pretrained encoder space, L2-normalizes both feature sets, computes pairwise cosine STSS, then temperature-softmaxes spatial correspondences.Source: §4.1 · p. 4
- Lower temperature shifts correspondence granularity from object to part to point, while TSA sharpens motion supervision and preserves part-level structural information.TSA also strengthens gradients for misaligned regions while attenuating them for well-aligned correspondences.Source: §4.1 · p. 4, Figure 3 · p. 5
4.2. Motion-focused Self-Similarity Alignment
- For each frame, M-TSA selects top-k patch tokens by temporal-difference magnitude and aligns STSS only on those motion-salient tokens.Source: §4.2 · p. 5
- Selective alignment excludes static regions to reduce computation and memory, and M-TSA is combined with diffusion loss for fine-tuning.Source: §4.2 · p. 5
5. Experiments
Experiments evaluate TSA on physical-plausibility benchmarks and video quality, reporting gains in physical-consistency scores. Ablations examine temperature, dynamic-region masking, and feature-alignment depth, with qualitative comparisons illustrating motion and appearance differences.
5.1. Implementation Details
- The experiments finetune CogVideoX-2B using 64K OpenVid videos, a lightweight projector, and 4,000 AdamW iterations on eight RTX 6000 Ada GPUs.Source: §5.1 · p. 5
5.2. Benchmarks
- VideoPhy evaluates 344 material-interaction prompts, VideoPhy2 covers 590 human-object prompts, and VBench assesses quality and text-condition consistency across 946 prompts.Source: §5.2 · pp. 5–6
5.3. Results
- 30.8% overall PC (M-TSA) surpasses PhyT2V (29.0%) on VideoPhy; TSA scores 29.1% versus VideoREPA’s 27.6%.Source: Table 1 · p. 6, §5.3 · p. 6
- 24.4% Joint score for M-TSA exceeds VideoREPA’s 23.1% on VideoPhy2, extending the improvement to human-object interactions.Source: Table 2 · p. 6, §5.3 · p. 6
5.4. Ablation Studies
- Temperature 0.1 and a 20% masking ratio are optimal settings; alignment at CogVideoX’s 18th transformer block performs best on both benchmarks.Source: §5.4 · pp. 6–7
5.3. Results
- 81.0%→81.2% Total Score on VBench accompanies M-TSA’s physical-plausibility gains, with overall video quality preserved.Source: §5.3 · p. 6, Table 3 · p. 7
5.4. Ablation Studies
- Applying TSA to NOVA improves PC scores across all material types, supporting applicability across bidirectional diffusion and causal autoregressive architectures.Source: Table 4 · p. 7, §5.4 · p. 7
5.5. Qualitative Results
- Qualitative comparisons show less paint-roller appearance drift and more physically plausible, temporally coherent jetski splash patterns with the proposed method.Source: §5.5 · p. 7, Figure 6 · p. 8
6. Conclusion
TSA transfers spatio-temporal correspondences from visual foundation models into video diffusion models for physically plausible generation. On VideoPhy and VideoPhy2, it improves physical realism, with ablations highlighting correspondence granularity and motion masking.
6. Conclusion
- TSA transfers spatio-temporal correspondences from visual foundation models into video diffusion models for physically plausible generation.Source: §6 · p. 8
- TSA significantly improves physical realism in generated videos across diverse real-world scenarios on VideoPhy and VideoPhy2.The conclusion reports these benchmark results alongside extensive ablation studies.Source: §6 · p. 8
- Ablations identify correspondence granularity as critical for motion learning and show that masking static regions substantially improves performance.The method aligns probabilistic correspondence distributions exclusively on dynamically changing regions.Source: §6 · p. 8
A. Additional Qualitative Results
Qualitative comparisons show more physically grounded and temporally consistent motion with our method across ball, snowmobile, and hockey scenes. Text-prompt adherence remains imperfect, which the authors attribute to the base model's scale.
A. Additional Qualitative Results
- In ball, snowmobile, and hockey scenes, our method produces physically grounded or coherent motion where competing methods show implausible dynamics and artifacts.The examples include floating balls, misplaced shadows, snowmobile shape distortion, elongated hockey sticks, and spuriously appearing objects; red rectangles mark implausible or inconsistent regions.Source: A. Additional Qualitative Results · p. 12, Figure 7 · p. 13
- The method still fails to faithfully follow the text prompt in the ball and hockey examples.Source: A. Additional Qualitative Results · p. 12
- The authors attribute this prompt-adherence limitation to the small base model and defer testing larger models because of computational resource constraints.CogVideoX-5B and HunyuanVideo are proposed as larger-model options for future work.Source: A. Additional Qualitative Results · p. 12