Source-linked AI summary
PhyRPR: Training-Free Physics-Constrained Video Generation
Yibo Zhao, Hengjia Li, Xiaofei He, Boxi Wu
TL;DR
Video diffusion models can produce visually plausible content but often struggle with explicit physical constraints because physical understanding is entangled with visual synthesis. PhyRPR separates these functions through training-free reasoning, motion planning, and diffusion refinement, and experiments report improved physical plausibility and motion controllability.
Problem
Single-stage video generation entangles high-level physical understanding with low-level visual synthesis, limiting control when explicit kinematic constraints or grounded interactions are required.
Method
PhyRPR is a training-free three-stage pipeline that reasons over physical states, deterministically plans a coarse motion scaffold, and injects it into diffusion sampling for visual refinement.
Results
Experiments show gains in physical plausibility and motion controllability while preserving high visual quality.
Takeaways & Limitations
The staged design provides an interpretable and controllable process for physics-constrained video generation.
Abstract
from arXiv · showhide
Recent diffusion-based video generation models can synthesize visually plausible videos, yet they often struggle to satisfy physical constraints. A key reason is that most existing approaches remain single-stage: they entangle high-level physical understanding with low-level visual synthesis, making it hard to generate content that require explicit physical reasoning. To address this limitation, we propose a training-free three-stage pipeline,\textit{PhyRPR}:\textit{Phy\uline{R}eason}--\textit{Phy\uline{P}lan}--\textit{Phy\uline{R}efine}, which decouples physical understanding from visual synthesis. Specifically, \textit{PhyReason} uses a large multimodal model for physical state reasoning and an image generator for keyframe synthesis; \textit{PhyPlan} deterministically synthesizes a controllable coarse motion scaffold; and \textit{PhyRefine} injects this scaffold into diffusion sampling via a latent fusion strategy to refine appearance while preserving the planned dynamics. This staged design enables explicit physical control during generation. Extensive experiments under physics constraints show that our method consistently improves physical plausibility and motion controllability.
I. INTRODUCTION
Existing video diffusion models generate visually compelling content but often violate explicit physical constraints because physical reasoning remains implicit and entangled with visual synthesis. PhyRPR addresses this with a training-free, staged design that separates physical understanding from rendering.
- Video diffusion models often violate constraints such as keeping oil floating on milk or following directional cues, causing inconsistent interactions and motion drift.
- Prompt enhancement targets surface appearance and imprecisely encodes temporal dynamics, while training-based video methods require expensive training and physics-aware annotations.
- Single-stage denoising entangles physical reasoning with visual synthesis, making explicit kinematic constraints and grounded interactions difficult to control.
- PhyRPR decouples physical understanding from visual synthesis through PhyReason, PhyPlan, and PhyRefine, exposing physically meaningful intermediate representations.
- Experiments report improvements over existing methods in physical consistency, trajectory controllability, and overall visual quality.
B. Unified Multimodal for Understanding and Generation
Building understanding-and-generation video models from scratch demands prohibitive pretraining costs and data. PhyRPR instead orchestrates off-the-shelf multimodal reasoning, coarse video planning, and diffusion refinement in a training-free paradigm.
- Unified understanding-and-generation video models require large-scale pretraining, creating prohibitive computational costs and data demands.
- PhyRPR combines off-the-shelf multimodal models, a coarse video generation toolkit, and diffusion models to decouple physical understanding from visual generation.
- The framework supports physics-constrained synthesis without expensive parameter optimization.
A. PhyReason: Visually Grounded Physical Reasoning
PhyReason converts a prompt into physically grounded key states, visually consistent keyframes, and object-centric masks that provide structured guidance for later planning and refinement.
- PhyReason uses a large multimodal model for physical reasoning and a single-frame image generator for keyframe synthesis.
- Given a prompt, the LMM constructs a physically constrained state-transition sequence focused on pivotal kinematic moments and representative states.
- The state sequence is jointly generated across kinematic milestones rather than by uniformly sampling timestamps.
- Visual feedback refines each target state using the previous keyframe, an editing instruction, and an instruction-guided image editing model.
- An image segmentation model extracts binary object masks from generated keyframes, producing physically self-consistent visual states for downstream control.
B. PhyPlan: Physics-Aware Motion Planning
PhyPlan converts discrete, semantically consistent object states into continuous trajectories and a coarse video scaffold that provides explicit kinematic guidance for refinement.
- Motion representation: PhyPlan uses a structured motion script to specify per-entity motion semantics and milestone key states.The script is generated from keyframes, object masks, and optionally the original prompt.
- Motion representation: Each normalized state vector encodes an entity’s position, scale, rotation, and opacity at a milestone.
- Trajectory synthesis: The planner instantiates motion primitives between neighboring key states and fits physical parameters to satisfy endpoint conditions and improve temporal coherence.The continuous trajectory connects endpoint positions such as object centers.
- Coarse-video rendering: PhyPlan deterministically renders planned trajectories into a coarse video through layout synthesis and content composition.Objects are warped from their initial masks and appearance crops, then composited onto an inpainted background.
- Coarse-video rendering: Although the coarse video may contain stretched textures or imperfect details, it preserves intended topology and continuous trajectories for latent refinement.
C. PhyRefine: Motion-Aware Visual Refinement
PhyRefine injects the planned motion scaffold into diffusion sampling as a latent-space constraint, preserving motion structure while the pretrained video model restores visual detail and coherence.
- Latent refinement: PhyRefine uses motion-aware, noise-consistent latent injection to couple explicit kinematics with the video model’s rendering capability.The scaffold is introduced during sampling rather than through additional training.
- Motion-aware constraint: The motion mask is obtained by downsampling occupancy masks to latent resolution and broadcasting them to match the noisy latent.
- Motion-aware constraint: At selected sampling steps, PhyRefine enforces the planned state within the motion region while avoiding distribution shifts that could harm global appearance consistency.
- Noise-consistent injection: The model-implied clean component is replaced only inside the motion region, while the recovered noise is preserved to maintain textures, colors, and style.Remaining regions are left unchanged.
- Qualitative outcome: Qualitative comparisons report that the method captures deformation during inflation and a plausible rebound better than baselines.
A. Experimental Setup
The evaluation covers text- and image-conditioned video generation across diverse physics-constrained scenarios, using video-quality, constraint-following, and user assessments.
- Implementation: PhyRPR uses Gemini-3-pro-preview, SAM3, Nano Banana Pro, and Wan2.2-I2V-A14B across its three training-free stages.The pipeline synthesizes milestone keyframes, segments objects, and performs latent fusion during diffusion denoising.
- Experimental design: The study evaluates text-conditioned and image-conditioned video generation across 40 diverse scenarios, including descriptive and arrow-guided motion control.The same scenarios are used for human evaluation, LMM-as-judge, and quantitative comparisons.
- Baselines: Baselines include Wan2.2 text- and image-to-video models and their prompt-enhanced variants, evaluated under the protocols each supports.The enhancement variants are generated with the same LMM to control prompt-engineering effects.
- Evaluation metrics: The evaluation uses VBench for generic video quality, Gemini as an LMM judge for constraint satisfaction, and user studies.The LMM judge scores physical plausibility, trajectory compliance, and temporal consistency on 1–5 scales.
B. Comparison with Baselines
PhyRPR outperforms baselines on the reported physics-aware and video-quality evaluations, while qualitative comparisons show more faithful dynamics and trajectory following.
- Quantitative comparison: PhyRPR achieves the best performance across all reported metrics, whereas prompt enhancement improves constraint following but may slightly reduce VBench quality.The comparison includes LMM-as-judge and user-study scores for physical and motion constraint satisfaction.
- Qualitative comparison: PhyRPR produces reasonable volleyball impact deformation and rebound, avoiding artifacts observed in both text-to-video and image-to-video baselines.The image-to-video baselines share the same first frame but still show excessive deformation.
- Qualitative comparison: PhyRPR follows prompt-specified trajectories more faithfully than baselines, which drift from the intended path with or without prompt enhancement.In arrow-guided billiards control, it follows assigned arrow trajectories despite baseline direction confusion and unstable cues.
C. Ablation Studies
The ablations remove PhyPlan or PhyRefine while holding PhyReason outputs fixed, isolating the contributions of planning and refinement.
- Ablation design: The ablation variants share identical PhyReason outputs, providing controlled inputs for comparing downstream planning and refinement components.The reported variants are SDEdit-style (-R) and LTX-Multi (-P,-R).
- PhyPlan ablation: Removing PhyPlan is evaluated with LTX-Multi, a video foundation model that accepts multiple input images with target frame indices.This setup isolates the role of the planning stage in producing the motion scaffold.
- PhyRefine ablation: Removing PhyRefine replaces motion-aware noise-consistent injection with global SDEdit-style refinement of the coarse video.The alternative does not selectively constrain planned motion regions during denoising.
V. CONCLUSION
PhyRPR is a training-free three-stage pipeline that separates physical understanding from visual synthesis for physics-constrained video generation. Experiments report improved physical plausibility and motion controllability while preserving high visual quality.
- Conclusion: PhyRPR infers physical states, plans a motion scaffold, and injects it into diffusion sampling through noise-consistent latent fusion.The pipeline decouples physical understanding from visual synthesis.
- Conclusion: Experiments show gains in physical plausibility and motion controllability while preserving high visual quality.The conclusion summarizes the method's reported benefits under physics constraints.