Source-linked AI summary
VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models
Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang, Fanqing Meng, Xiangpeng Wan, Junchi Yan, Yu Cheng
TL;DR
Text-to-video diffusion models often generate videos with limited physical plausibility because their physics understanding lags behind that of video self-supervised models. VideoREPA transfers physics knowledge through spatio-temporal token-relation distillation tailored to finetuning, achieving stronger physics commonsense and more physically plausible generation. Its reported limitations concern the mismatch between standard REPA and video finetuning, especially its spatial focus and hard alignment.
Problem
Text-to-video diffusion models often have limited physical plausibility, with their physics understanding substantially behind that of video self-supervised models.
Method
VideoREPA distills physics knowledge from video foundation models into video diffusion models by aligning token-level spatial and temporal relations with the TRD loss.
Results
40.1 Physical Commonsense on VideoPhy is reported for VideoREPA, representing a 24.1% improvement over the CogVideoX baseline, alongside significant enhancements on VideoPhy2.
Takeaways & Limitations
VideoREPA demonstrates that transferring physics understanding from video foundation models can improve the physical plausibility of text-to-video generation.
Takeaways & Limitations
Standard REPA is poorly suited to this setting because its spatial, hard-alignment design targets from-scratch acceleration rather than finetuning pre-trained video models and temporal dynamics.
Abstract
from arXiv · showhide
Recent advancements in text-to-video (T2V) diffusion models have enabled high-fidelity and realistic video synthesis. However, current T2V models often struggle to generate physically plausible content due to their limited inherent ability to accurately understand physics. We found that while the representations within T2V models possess some capacity for physics understanding, they lag significantly behind those from recent video self-supervised learning methods. To this end, we propose a novel framework called VideoREPA, which distills physics understanding capability from video understanding foundation models into T2V models by aligning token-level relations. This closes the physics understanding gap and enable more physics-plausible generation. Specifically, we introduce the Token Relation Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft guidance suitable for finetuning powerful pre-trained T2V models, a critical departure from prior representation alignment (REPA) methods. To our knowledge, VideoREPA is the first REPA method designed for finetuning T2V models and specifically for injecting physical knowledge. Empirical evaluations show that VideoREPA substantially enhances the physics commonsense of baseline method, CogVideoX, achieving significant improvement on relevant benchmarks and demonstrating a strong capacity for generating videos consistent with intuitive physics. More video results are available at https://videorepa.github.io/.
1 Introduction
VideoREPA addresses limited physical plausibility in text-to-video diffusion models by transferring physics understanding from video foundation models through spatio-temporal token-relation alignment. It introduces TRD for finetuning pre-trained models and reports stronger physics-consistent generation than baselines.
- Motivation: CogVideoX performs significantly worse than the smaller VideoMAEv2 on the Physion physics-understanding benchmark.This gap motivates transferring physics knowledge from video foundation models to video diffusion models.
- Method: VideoREPA distills physics knowledge from video foundation models into video diffusion models using token-level relational alignment.The method transfers intra-frame spatial relations and inter-frame temporal dynamics without relying on explicit physics datasets.
- Method: TRD aligns pairwise token similarities between video SSL representations and intermediate diffusion-transformer features across spatial and temporal relations.Tokens are related within the same latent frame and across different latent frames.
- Method: TRD is designed to overcome REPA limitations in finetuning pre-trained models and capturing essential temporal dynamics.Direct REPA application can destabilize or degrade pretrained video diffusion models.
- Results: 40.1 Physical Commonsense on VideoPhy is reported for VideoREPA, a 24.1% improvement over the CogVideoX baseline.The paper also reports significant physics enhancements on VideoPhy2 and qualitative improvements over CogVideoX.
2 Related works
Related work covers self-supervised learning for physics understanding and methods for physics-plausible video generation. Existing approaches use scaled data, advanced architectures, simulation, prompting, or specialized physics datasets, but physical plausibility remains a central challenge.
- Self-supervised learning for physics understanding: Self-supervised learning uses pretext tasks on large unlabeled datasets for understanding tasks including classification, segmentation, and detection.Physics understanding supports applications such as predicting object trajectories or movements for robotics and autonomous driving.
- Physics plausible video generation: State-of-the-art text-to-video models improve visual fidelity, motion, and realism, but physical plausibility remains poor.This has motivated research into physics-aware video generation.
- Physics plausible video generation: PhyT2V uses multiple rounds of prompting and reasoning with multimodal language models, while WISA decomposes text into physical phenomena and uses category-specific experts.WISA's effectiveness is limited to specialized datasets containing explicit physics phenomena.
3 Methods
VideoREPA adapts representation alignment for finetuning text-to-video diffusion models by transferring physics understanding from video foundation models through token relations. Its TRD loss aligns spatial relations within frames and temporal dynamics across frames, providing softer guidance than direct feature alignment.
- Motivation: The approach is motivated by a physics-understanding gap: CogVideoX performs significantly worse than the much smaller self-supervised VideoMAEv2 on the Physion benchmark.This comparison supports transferring physics-related representations from video understanding models into video generators.
- Motivation: Standard REPA is unsuitable for this setting because it emphasizes spatial alignment, was developed mainly for from-scratch acceleration, and can be unstable for finetuning pretrained models.VideoREPA addresses temporal dynamics, finetuning stability, and differences between VDM and VFM feature spaces.
- Token Relation Distillation: TRD provides softer guidance by matching pairwise token similarities between VFM and VDM representations instead of enforcing direct feature similarity.This relational alignment is designed for finetuning pretrained video diffusion models.
- Token Relation Distillation: The method aligns spatial constraints within frames and temporal dynamics across frames using VFM features and dimension-aligned VDM transformer representations.A VFM encoder processes the video, while the VDM hidden state is passed through a trainable MLP before relational comparison.
- Token Relation Distillation: TRD computes spatial similarities among tokens in each frame and temporal similarities between tokens across different frames, then minimizes their average L1 difference.The final objective combines diffusion loss and TRD loss as L = Ldiff + λLTRD.
4 Experiments
Experiments evaluate VideoREPA on physics-focused video benchmarks against CogVideoX and other video-generation methods, combining quantitative, qualitative, and ablation analyses. VideoREPA improves physical commonsense while its spatial-temporal TRD components and soft alignment support semantic quality.
- Evaluation setup: VideoREPA is evaluated on VideoPhy and VideoPhy2 against CogVideoX, leading T2V models, and physics-aware methods.VideoPhy covers material interactions, while VideoPhy2 focuses on action-centric human-object interactions.
- Quantitative comparisons: 24.1% overall PC-score improvement over CogVideoX is achieved by VideoREPA-5B on VideoPhy.Improvements are 42.9% for Solid-Solid, 16.7% for Solid-Fluid, and 20.6% for Fluid-Fluid interactions.
- Quantitative comparisons: 40.1 versus 33 PC score on Koala-36M shows VideoREPA outperforming WISA on open-domain data.WISA is reported to struggle when generalizing beyond its physics-explicit training dataset.
- Quantitative comparisons: 4.57-score improvement over the baseline is obtained by VideoREPA-2B on the VideoPhy2 evaluation.VideoPhy2 reports Semantic Adherence and Physical Commonsense as the proportions of videos rated at least 4.
- Qualitative comparisons: VideoREPA produces more physically consistent pencil rolling and crane-supported brick lifting than HunyuanVideo and CogVideoX.The comparisons highlight rigid-body motion and visible physical support in generated videos.
- Ablation studies: Removing either TRD spatial or temporal alignment component degrades physical plausibility, while one-dimensional alignment also harms Semantic Adherence.The ablation supports using both spatial and temporal terms.
- Ablation studies: Standard REPA substantially degrades video semantic quality during finetuning, whereas TRD provides soft guidance for pretrained VDMs.The contrast is attributed to hard token-similarity alignment disrupting established feature spaces.
5 Conclusion and outlook
The paper concludes that VideoREPA transfers physics knowledge from video foundation models to text-to-video diffusion models through token-level relation distillation. It reports state-of-the-art generation results but leaves pretraining integration unvalidated.
- Conclusion: VideoREPA transfers physics knowledge from VFMs to VDMs through token-level relation distillation.The framework is intended to improve the physical plausibility and commonsense of generated videos.
- Conclusion: Extensive experiments report state-of-the-art generation results with strong physical commonsense in generated videos.This is the paper's summarized empirical conclusion.
- Outlook: VideoREPA's potential for pretraining VDMs remains unvalidated because of computational resource limitations.The authors suggest exploring its incorporation into VDM pretraining in future work.
A Detailed training setting
The finetuning setup uses OpenVid and explores multiple video foundation-model encoders with specified optimization settings for CogVideoX-based models.
- Dataset: OpenVid provides over one million in-the-wild videos for the finetuning setting.The dataset is described as open-source, high-quality, and paired with expressive captions.
- Optimization: CogVideoX-5B uses LoRA with learning rate 1e-4, while CogVideoX-2B uses full-parameter finetuning with learning rate 2e-6.The LoRA configuration uses rank 128 and alpha 64.
- Target encoders: The alignment target encoders include VideoMAEv2-B, V-JEPA-L, OmniMAE-B, and VideoMAE-B.These encoders are explored as target representations in the experiments.
B Physion evaluation setting
The Physion evaluation tests physical reasoning in realistic simulated scenarios, focusing on future object contact prediction from video features.
- Benchmark: Physion v1.5 contains realistic simulations of diverse physical scenarios involving manipulated objects.The benchmark covers reasoning types including stability, rolling motion, and object linkage.
- Task: The Object Contact Prediction task evaluates whether a model can predict future contact between two objects from an initial context video.Features are extracted from selected temporal dimensions and spatial tokens.
- Evaluation procedure: A logistic regression classifier uses denoising-network features from layer 18 to perform the contact-prediction task.Evaluation uses the roll and contain subsets, with accuracies reported in Figure 1.
C Additional ablation study
The ablations select VideoMAEv2 as the alignment target and establish design choices for dimensional interpolation, alignment depth, TRD weighting, and VFM input processing.
- Different video foundation models: VideoMAEv2 performs best among the evaluated video foundation models and is selected for VideoREPA.The comparison includes VideoMAE, V-JEPA, VideoMoCo, and VideoMAEv2.
- Different alignment depth: An alignment depth of 18 yields the best performance on the reported PC score on VideoPhy.This depth is adopted in VideoREPA.
- Effect of λ: λ = 0.5 provides the best trade-off between the original diffusion loss and the TRD loss.The trade-off is evaluated using the PC score on VideoPhy.
- Dimension alignment: Interpolating VDM latent representations to the VFM feature dimensions best preserves the VFM’s knowledge.The first encoded 3D-VAE frame is excluded to focus alignment on dynamic content.
- Trade-off between input frames and resolution: Processing all frames at reduced resolution preserves holistic VFM representations with the lowest computational requirements.Grouping or cropping frames tends to degrade either physics plausibility or semantic quality.
D Generating detailed prompt for VideoPhy
The method refines brief VideoPhy prompts by making implicit context, actions, sequences, and cause-and-effect details explicit for clearer evaluation.
- Prompt refinement: LLMs such as GPT-4o or Gemini 2.5 Pro are used to clarify vague VideoPhy prompts.The refinement targets implicit knowledge about the outcome, environment, and processes.
- Prompt refinement: The prompt-refinement instruction removes assumptions about environmental context, action sequences, and cause-and-effect processes.It asks for clear and descriptive rewritten prompts that require no reasoning to understand the scenario.
E More qualitative results
Additional qualitative comparisons present VideoREPA-5B alongside CogVideoX-5B and show generated examples intended to demonstrate stronger physical commonsense.
- Direct comparisons: Figures 6–8 directly compare VideoREPA-5B with baseline CogVideoX-5B outputs.The section presents the comparison as evidence of enhanced physical commonsense in generated videos.
- Additional examples: Figures 9 and 10 display additional videos generated by VideoREPA that exhibit strong physical plausibility.The figures provide further qualitative examples from the proposed model.