Source-linked AI summary
PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation
Yuanhao Cai, Kunpeng Li, Menglin Jia, Jialiang Wang, Junzhe Sun, Feng Liang, Weifeng Chen, Felix Juefei-Xu, Chu Wang, Ali Thabet, Xiaoliang Dai, Xuan Ju, Alan Yuille, Ji Hou
TL;DR
Physically consistent text-to-video generation remains challenging, while graphics-based and prompt-extension approaches have limited settings or rely on imperfect physics reasoning. The paper builds a physics-rich dataset and trains PhyGDPO with real-world winners and groupwise preferences; it reports better benchmark performance and higher human preference for physical realism.
Problem
Physically consistent text-to-video generation remains challenging, and graphics-based or prompt-extension approaches have limited settings or rely on imperfect physics reasoning.
Source: §1 · pp. 1–2Method
PhyAugPipe builds a 135K physics-rich video-pair dataset, and PhyGDPO uses real-world winners, groupwise Plackett–Luce preferences, physics-guided rewards, and a LoRA-switch reference.
Source: §1 · p. 2Results
PhyGDPO outperforms state-of-the-art methods on PhyGenBench and VideoPhy2 and achieves higher human preference in a physical-realism user study.
Source: §1 · p. 2Takeaways & Limitations
The paper post-trains Wan2.1-T2V-14B on PhyVidGen-135K to boost physical plausibility without LLM prompt extension at inference.
Source: §5 · p. 11PhyAugPipe filters real-world videos: VLM rewards affect physics richness rather than correctness, and the pipeline retains original human-written prompts to avoid VLM bias or errors.
Source: §2.1 · p. 4
Abstract
from arXiv · showhide
Recent advances in text-to-video (T2V) generation have achieved good visual quality, yet synthesizing videos that faithfully follow physical laws remains an open challenge. Existing methods mainly based on graphics or prompt extension struggle to generalize beyond simple simulated environments or learn implicit physical reasoning. The scarcity of training data with rich physics interactions and phenomena is also a problem. In this paper, we first introduce a Physics-Augmented video data construction Pipeline, PhyAugPipe, that leverages a vision-language model (VLM) with chain-of-thought reasoning to collect a large-scale training dataset, PhyVidGen-135K. Then we formulate a principled Physics-aware Groupwise Direct Preference Optimization, PhyGDPO, framework that uses real-world video as winning case to guarantee correct physics learning and builds upon the groupwise Plackett-Luce probabilistic model to capture holistic preferences beyond pairwise comparisons. In PhyGDPO, we design a Physics-Guided Rewarding (PGR) scheme that leverages VLM-based physical rewards to direct the optimization to focus on challenging physics cases. In addition, we propose a LoRA-Switch Reference (LoRA-SR) scheme that avoids full-model duplication as reference for efficient DPO training. Experiments show that our method significantly outperforms state-of-the-art open-source methods on PhyGenBench and VideoPhy2. Please check our project page at https://caiyuanhao1998.github.io/project/PhyGDPO for more video results. Our code, data, and models are publicly available at https://github.com/caiyuanhao1998/Open-PhyGDPO
1 Introduction
The introduction frames physically consistent text-to-video generation as unresolved and presents PhyAugPipe and groupwise preference optimization as a data-and-training response. It reports gains over state-of-the-art methods on two benchmarks and higher human preference for physical realism.
1 Introduction
- Prior approaches restrict graphics simulations to simple scenarios, rely on imperfect LLM physics prompts, or lack negative contrastive data and holistic preference signals in vanilla DPO.Collecting and annotating high-quality text-video pairs is costly and labor-intensive, while supervised training still shows limited physics reasoning.Source: §1 · p. 2
- PhyGDPO uses real-world videos as winning cases and a groupwise Plackett–Luce model to represent holistic preferences beyond pairwise comparisons.Source: §1 · p. 2
- PGR steers sampling and optimization toward challenging physics, while LoRA-SR switches between reference and action modes without duplicating the full reference model.LoRA-SR freezes the base model and attaches LoRA with an environment manager.Source: §1 · p. 2
- PhyGDPO outperforms state-of-the-art methods on PhyGenBench and VideoPhy2 and yields higher human preference in a physical-realism user study.Source: §1 · p. 2
- PhyAugPipe uses VLM chain-of-thought filtering, semantic action clustering, and physics-aware VLM difficulty scoring to select preference-training winners, yielding PhyVidGen-135K with over 135K pairs.Source: §1 · p. 2, Figure 2 · p. 3
2 Method
PhyAugPipe builds physics-rich training data through VLM-based filtering, action clustering, and physics-rewarded sampling. PhyGDPO uses groupwise preferences with real-world videos as winners, then adds physics-guided weighting and a shared-backbone reference scheme.
2.1 Physics-Augmented Video Data Construction
- PhyAugPipe uses VLM chain-of-thought reasoning to parse entities and forces, assess interactions and physics richness, and filter data by a richness threshold.It also extends prompts with explicit physics reasoning, while retaining extended prompts for other research purposes.Source: §2.1 · p. 3
- Semantic action clustering measures category imbalance among filtered samples before data resampling.The pipeline matches original prompts to challenging action categories and counts samples to form a distribution histogram.Source: §2.1 · p. 3
- VideoCon-Physics rewards guide resampling toward action categories where the T2V model performs poorly.Representatives receive semantics-adherence and physics-commonsense scores, which are averaged to estimate each category’s score.Source: §2.1 · pp. 3–4
- PhyAugPipe filters real-world videos, so VLM rewards affect sample richness rather than physics correctness; original prompts avoid possible VLM rewriting bias or errors.The pipeline relies on real-world videos as physically correct examples.Source: §2.1 · p. 4
2.2 Physics-Aware Groupwise Direct Preference Optimization
- PhyGDPO uses a groupwise Plackett–Luce model, with real-world video as the winning case, to capture holistic preferences beyond pairwise comparisons.The real-world video provides the correct-physics preference signal.Source: §2.2 · p. 4, Figure 3 · p. 5
- An upper bound makes groupwise training use a single sampled data pair at one timestep per iteration.This is presented as an efficient-training strategy.Source: §2.2 · p. 6
- Physics-Guided Rewarding uses physics-aware VLM scores to focus DPO training on challenging physics cases.Its adaptive weights give physics-violating, lower-plausibility samples stronger influence while balancing stability and adaptivity.Source: §2.2 · p. 6
- LoRA-SR shares a frozen backbone between reference and action models, switching modes through trainable LoRA modules instead of duplicating the full model.The scheme avoids storing an extra full model for DPO advantage computation and is described as improving efficiency and training stability.Source: §2.2 · pp. 7–8
3 Experiment
Experiments compare PhyGDPO with prior methods on VideoPhy2 and PhyGenBench, then test dataset construction and framework components through ablations and cross-evaluations. The results include benchmark gains, human preferences, and analyses of memory use and model behavior.
3 Experiment
- Evaluation uses short, unextended VideoPhy2 and PhyGenBench prompts; VideoPhy2 scores semantics and physics, while PhyGenBench assesses phenomena, their order, and naturalness across 27 physical laws.The training pipeline filters one million text-video pairs to 135K, then uses physics-rewarded sampling to select 17K prompts for video generation.Source: §3 · p. 8
3.1 Comparison with State-of-the-Art Methods
- On VideoPhy2, PhyGDPO surpasses Sora2 and Veo3 across hard actions, activities, sports, and overall, with hard-action scores 29% and 13% higher, respectively.On hard actions, its score is also 4.5× Wan2.1-14B; versus VideoDPO and PhyT2V, it is higher by 200% and 29%, respectively. On PhyGenBench, mechanics gains are 22%/15% and thermal gains 35%/23% over PhyT2V/VideoDPO.Source: §3.1 · p. 9
- PhyGDPO is preferred by human evaluators, and visual comparisons report more realistic physical dynamics, body motion, and object interactions across human activities and physical events.The user study involved 104 participants; reported 95% confidence intervals were no wider than ±2.4%.Source: §3.1 · pp. 9–10
3.2 Ablation Study
- With equal data counts, adding CoT filtering, action clustering with semantic matching, and physics-guided sampling improves every track; hard-action scores gain over 50%.Table 2 identifies the PhyAugPipe component ablation.Source: Table 2 · p. 9, §3.2 · p. 10
- Progressively adding LoRA-SR, the PL groupwise model, and physics-guided rewarding steadily improves performance, with a 4.5× hard-action score gain and more buoyant tennis-ball motion.The visual breakdown shows the ball floating stably on the water surface rather than sinking or bouncing unrealistically.Source: Figure 7 · p. 9, §3.2 · p. 10
- Under matched Wan2.1-1.3B training settings, PhyGDPO surpasses Flow-DPO and VideoDPO across tracks, with a 50% hard-action improvement and more coherent weightlifting motion.The visual comparison reports distorted or unstable poses for Flow-DPO and VideoDPO.Source: §3.2 · p. 10, Figure 8 · p. 10
- LoRA-SR cuts GPU memory by 44% and storage size by over 60× while improving hard-action scores by 14%; against LoRA-SFT, it uses almost the same GPU memory and achieves 2.7× the hard-action score.The visual analysis reports more plausible hand–shovel interactions with LoRA-SR.Source: §3.2 · pp. 10–11, Figure 9 · p. 11
- PhyGDPO remains ahead of state-of-the-art T2V models with Gemini-2.5-pro scoring and ahead of state-of-the-art DPO methods on Vcrafter2; performance gains persist across tested reward hyperparameters.The best reported settings are αmin = 0.5, kα = 5.0, bα = 0.5, λ = 0.6, kγ = 2.0, and bγ = 0.4.Source: Table 3 · p. 10, §3.2 · p. 11
4 Related Work
Related work identifies limits in both physics-aware T2V generation and preference optimization. The paper positions its approach against prompt extension and pairwise DPO methods that focus largely on visual quality.
4.1 Text-to-Video Generation Models
- Prompt-extension approaches add explicit physics laws through an LLM, but can be misled by LLM errors and struggle to learn implicit physics.Existing T2V models also face challenges in accurately modeling physics-related effects.Source: §4.1 · p. 11
4.2 Direct Preference Optimization
- Video DPO methods adapt image-preference optimization to T2V, but mainly improve visual quality and require a duplicated reference model that consumes additional GPU memory.The cited examples include DiffusionDPO and VideoDPO.Source: §4.2 · p. 11
5 Conclusion
The conclusion presents PhyGDPO as an efficient post-training approach for improving the physical plausibility of T2V generation. It reports quantitative, qualitative, and human-preference gains over state-of-the-art algorithms.
5 Conclusion
- PhyGDPO post-trains Wan2.1-T2V-14B on over 135K physics-rich text-video pairs, using PhyAugPipe and the PGR and LoRA-SR designs.The paper states that inference does not use LLM prompt extension, to study the model’s implicit physics reasoning.Source: §5 · p. 11
Mathematical proof for Ineq. (8)
The proof uses concavity of t^αj for 0 < αj ≤ 1 to derive inequalities. It then combines those inequalities to establish the claimed inequality used in the main paper.
Mathematical proof for Ineq. (8)
- For each j, concavity of t^αj on [0, +∞) yields inequalities for nonnegative u and v, including the case split according to whether u + v is zero.The proof sets λ = u/(u + v) when the sum is positive, then applies concavity to u and v.Source: Mathematical proof for Ineq. (8) · p. 16
- The derivation applies Ineq. (19) with u = 1 and v = e^(x_j), obtains e^(x_j) ≤ a_j for all j from Ineq. (23), and notes that higher-order terms in a_j are nonnegative.It combines Ineq. (24) and Ineq. (26) and substitutes the definition of a_j.Source: Mathematical proof for Ineq. (8) · pp. 16–17