Source-linked AI summary
PhyCo: Learning Controllable Physical Priors for Generative Motion
Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan, Manmohan Chandraker
TL;DR
Video diffusion models often violate physical consistency and lack controllable variation in physical properties. PhyCo combines physically grounded simulation data, explicit property conditioning, and VLM-guided reward optimization to generate controllable, physically consistent motion. It outperforms prior models in physical realism and controllable variation, including on unseen materials, forces, and interactions.
Problem
Video diffusion models can synthesize realistic appearance but still violate physical laws and struggle to controllably vary physical properties.
Method
PhyCo combines a 100K-video physically grounded dataset, physics-supervised ControlNet fine-tuning, and VLM-guided reward optimization for continuous physical-property control.
Results
PhyCo consistently outperforms prior video models in physical realism and controllable variation, generalizing to unseen materials, forces, and interactions.
Takeaways & Limitations
PhyCo provides a scalable path toward controllable, physically consistent generative video models that generalize beyond synthetic training data.
Takeaways & Limitations
The simulations must make physical attributes visually unambiguous and remain within the pretrained diffusion model’s competence range.
Abstract
from arXiv · showhide
Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.
1. Introduction
PhyCo addresses the gap between visual and physical realism by conditioning video diffusion models on continuous, interpretable physical properties. It combines a large annotated simulation dataset, physics-supervised ControlNet fine-tuning, and VLM-guided reward optimization to improve physically consistent and controllable generation.
- Related work: Prior approaches integrate physical simulation with generative models but depend on explicit solvers, including rigid-body dynamics in PhysGen, MPM-based optimization in PhysDreamer, and hybrid simulation in WonderPlay [22].These approaches are presented as improving fine-grained motion coherence while relying on simulation-based components.
- Framework: PhyCo conditions video diffusion models on physical properties including friction, restitution, deformation, and applied force for controllable, physically consistent motion synthesis.The framework is designed to manipulate physical properties across diverse materials and contact conditions without relying merely on external guidance.
- Dataset: The method introduces a 100K-video multi-scenario dataset with continuous physical annotations spanning diverse materials, interactions, views, and physical regimes.Built with Kubric [12], PyBullet, and Blender [4], the dataset is intended to disentangle visual appearance from underlying physics and support generalization beyond simulation.
- Training: PhyCo fine-tunes a pretrained diffusion backbone with ControlNet-based spatially aligned physical property maps and adds VLM-guided reward optimization.The pretrained backbone is Cosmos-Predict2, while the conditioning architecture is ControlNet.
- Results: On Physics-IQ and in human preference studies, PhyCo outperforms prior video models in physical realism and controllable variation while generalizing across unseen materials, forces, and interactions.The results also report compositionality across variations, supporting physical property priors as a scalable path toward controllable, physically consistent generative world models.
2. Related Work
Prior work studies physics-rich datasets, controllable video generation, and reward optimization, but existing approaches remain limited by dataset diversity, reliance on test-time simulation, or primarily kinematic control. These gaps motivate physically grounded, simulator-free control and reward alignment for generative video models.
- Physics Rich Datasets: Existing physics-rich datasets often lack diverse physical properties, realistic scenes, and complex interactions, while Force-Prompting remains limited in scene diversity and annotated physical properties.These limitations make many datasets out of distribution for modern generative video models and motivate larger, physically grounded collections.
- Controllable Video Generation: Motion-control methods such as ATI and Go-with-the-Flow [6] enable trajectory-based or precise object-motion steering, while other approaches target camera motion control.These methods establish fine-grained controllability but do not, in the supplied passage, establish control over physical properties.
- Controllable Video Generation: Simulator-integrated methods including PhysGen, WonderPlay [22], and related MPM or spring-mass approaches [7] improve physical consistency but require complex test-time simulation pipelines.PhysDreamer inverts physical parameters from generated motion, while PhysAnimator uses sketch-based physics cues to guide diffusion synthesis.
- Controllable Video Generation: Simulator-free methods such as VLIPP and PhysCtrl embed physics priors into diffusion models, but primarily emphasize kinematic guidance rather than broader physical control.Reward optimization extends alignment from images through ImageReward and DFTM to video with VADER and later VLM-feedback methods.
3. Method
PhyCo combines physics-rich simulation supervision, physical-property conditioning, and VLM-guided optimization to give diffusion video generation continuous, interpretable control over friction, restitution, deformation, and force. Its pipeline targets physically consistent, photorealistic videos while preserving pretrained representations and aligning optimization with inference-time generation.
- Physics-rich simulation dataset: PhyCo constructs over 100K simulation videos spanning controlled scenarios and systematically varying friction, restitution, deformation, and applied force.The dataset uses physics-rich, visually clean simulations designed so physical attributes appear clearly in motion and generalize beyond the training domain.
- Physics-conditioned generation: The method fine-tunes a pretrained Cosmos-Predict2-2B diffusion backbone with ControlNet conditioning on spatially aligned physical-property maps.Physical attributes are grouped into friction/restitution, Neo-Hookean deformation, and force magnitude/direction, while only ControlNet layers are trained and the base model and tokenizer remain frozen.
- Physics-conditioned generation: Separate ControlNet branches train on data where each property manifests in observed dynamics, enabling compositional physical conditioning under diffusion score-matching supervision.The property-map embeddings are processed through separate branches to support faster training and composition of physical attributes.
- VLM-guided optimization: PhyCo uses an N-step denoising rollout and decoded predicted video as input to a VLM, whose structured physics queries provide controllability and physical-alignment feedback.This feedback targets physical controllability rather than semantic or aesthetic image alignment and better matches the inference-time generation process than partially noised ground truth.
- VLM-guided optimization: VLM reward optimization fine-tunes only the physical-property ControlNet layers, excluding diffusion score matching to yield more stable and physically consistent generations.The alignment loss uses binary cross-entropy over logits for correct versus incorrect answers, with a 10-step denoising rollout for feedback.
4. Experimental Results
PhyCo is evaluated for physical realism, controllability, reward optimization, and generalization using benchmark metrics, human preferences, qualitative comparisons, and ablations. The experiments show improved perceived physical plausibility, stronger adherence to intended properties, and more precise force-direction control.
- Quantitative Evaluation on Physics-IQ Benchmark: PhyCo is evaluated on Physics-IQ across Solid Mechanics, Fluid Dynamics, Optics, Magnetism, and Thermodynamics using 396 generated videos compared with real-world reference sequences.The benchmark score measures timing and spatial alignment of key actions.
- User Study: Human evaluators consistently prefer PhyCo over baselines for physical realism, indicating more realistic behavior and clearer variations.Table 3 reports pairwise preference percentages, with scores above 50% indicating preference for PhyCo’s ControlNet generations.
- Ablations on PhyCo Data: VLM-based reward optimization achieves significantly better alignment with intended input properties, strengthening controllability and adherence to physical conditioning.The ablation evaluates 100 in-domain simulation videos spanning all attributes by comparing predicted properties with ground-truth conditioning inputs.
- Force Direction Adherence: 15.2° mean force-direction error versus 40.5° for Force-Prompting demonstrates more precise control over induced dynamics on 25 real-world videos.The evaluation applies random force directions and measures angular deviation between intended and observed motion.
5. Conclusion
PhyCo injects physically grounded priors into diffusion models through explicit property conditioning and VLM-guided reward optimization, enabling continuous control over key dynamics without simulators at inference. Evaluations show improved physical realism and controllability, with generalization beyond synthetic training data.
- 5. Conclusion: PhyCo enables continuous control over friction, restitution, deformation, and force without requiring simulators at inference.The framework combines explicit physical-property conditioning with VLM-guided reward optimization.
- 5. Conclusion: Evaluations show clear gains in physical realism and controllability, with strong generalization beyond synthetic training data.
Supplementary Material · A. Video Results on Webpage
The supplementary webpage provides the paper’s main and additional video results, covering stylistic generalization, continuous physical-property control, attribute composition, and baseline comparisons. These demonstrations include interactive force-direction control and visually convincing behavior even for difficult parameter pairings.
- A. Video Results on Webpage: All video results, including main-paper examples and additional space-limited results, are available at phyco-video.github.io.The webpage organizes the supplementary demonstrations by topic.
- A. Video Results on Webpage: The webpage includes additional generated videos beyond those shown in the main paper because of space constraints.
- A. Video Results on Webpage: Across artistic styles, four key frames guide multistyle storylines that produce physically consistent sequences.Figure 8 shows the first frames used for two generated storyline examples.
- A. Video Results on Webpage: Videos demonstrate smooth, continuous control by generating low, medium, and high levels for each physical attribute.
- A. Video Results on Webpage: The demonstrations combine force+friction, force+bounciness, and bounciness+deformation, while producing visually convincing behavior for difficult restitution-and-deformation pairings.The passage notes that some such pairings are difficult to simulate accurately in current physics engines.
- A. Video Results on Webpage: Side-by-side videos compare the approach with recent video diffusion models and provide interactive force-direction control through pregenerated results.
B. PhyCo Dataset Details · C. Implementation Details
PhyCo provides photorealistic, physically simulated videos with synchronized visual annotations and structured physical metadata. Its dataset systematically varies rigid-body, deformable-material, and external-force parameters, while implementation includes ControlNet fine-tuning and VLM-based supervision and evaluation.
- B. PhyCo Dataset Details: Each sample is a 4-second, 24-FPS video rendered at 432 × 768, with synchronized RGB, depth, segmentation, and structured metadata.Metadata records scene geometry, object materials, applied physical parameters, and a standardized scene-level text description.
- B. PhyCo Dataset Details: Every video includes a standardized scene-level text description to support multimodal supervision and VLM-based processing.The text description complements synchronized visual annotations and physical metadata.
- B. PhyCo Dataset Details: The dataset uniformly samples rigid-body friction and restitution from 0 to 1, spanning smooth sliding to high resistance and inelastic to highly bouncy impacts.These parameters are varied in PyBullet to cover diverse rigid-body dynamics.
- B. PhyCo Dataset Details: Deformable-body simulations vary Neo-Hookean Lamé coefficients µ and λ plus damping γ, spanning nearly rigid to highly deformable materials.The dataset uses an FEM-based Neo-Hookean model for deformable bodies.
- B. PhyCo Dataset Details: External forces range from gentle interactions to strong impacts and are projected from 3D world coordinates into rendered 2D frames using camera parameters.This projection directly aligns physical actions with their visual outcomes.
- C. Implementation Details: The proposed model uses ControlNet fine-tuning and VLM-based supervision, with Qwen2.5-VL fine-tuning for evaluation.These are the implementation components summarized for PhyCo’s training and evaluation pipeline.
C.1. PhyCo Implementation Details · C.2. Qwen2.5-VL Fine-tuning for Evaluation
PhyCo trains attribute-specific ControlNet branches while freezing the pretrained video diffusion model and tokenizer, then applies VLM-guided reward optimization for physics-aware supervision. A LoRA-adapted Qwen2.5-VL-3B model is fine-tuned on physics-focused video queries to infer physical properties for evaluation.
- C.1. PhyCo Implementation Details: ControlNet training updates only ControlNet layers, while the base video diffusion model and tokenizer remain frozen to preserve pretrained dynamics from the Cosmos World Foundation Model.
- C.1. PhyCo Implementation Details: Each attribute-specific ControlNet branch is trained for 10k optimization steps on 4×H100 GPUs, requiring approximately half a day.
- C.1. PhyCo Implementation Details: ControlNet supervision uses 57 frames per sequence at 24 FPS, with per-device batch size 1 and 2 steps of gradient accumulation.
- C.1. PhyCo Implementation Details: VLM-guided reward loss further trains each ControlNet branch for 100 iterations, taking roughly 70 minutes.
- C.1. PhyCo Implementation Details: Before VLM evaluation, videos are downsampled to half resolution and temporally subsampled to at most 16 frames; training uses 8×H200 GPUs and effective batch size 4.
- C.2. Qwen2.5-VL Fine-tuning for Evaluation: Qwen2.5-VL-3B is adapted on the PhyCo dataset to infer physical properties from videos using physics-focused queries.
- C.2. Qwen2.5-VL Fine-tuning for Evaluation: The evaluation model balances Yes/No responses, subsamples videos to at most 16 frames, and uses a highlighted blue sector overlay for force-direction queries.
- C.2. Qwen2.5-VL Fine-tuning for Evaluation: Qwen2.5-VL is fine-tuned for 200 iterations with LoRA, using rank and α settings specified in the implementation.
D. Additional Results
Additional evaluations show that PhyCo improves motion realism, generalizes across video backbones, and supports accurate physical-property prediction. Higher-FPS training and stronger temporal architectures further reduce flickering artifacts.
- Motion Consistency Evaluation: PhyCo achieves the best or second-best FVMD scores across most Physics-IQ domains, with VLM reward optimization further improving motion statistics over ControlNet alone.FVMD measures distributional distance between generated and reference motion features, with lower values indicating more realistic motion; gains are especially consistent in solid mechanics, fluid dynamics, and magnetism.
- Generalization Across Backbones: Text-only LoRA fine-tuning of Wan2.2 on the PhyCo dataset improves average Physics-IQ performance by 4.6%, without explicit ControlNet conditioning.This result indicates that the dataset transfers physically meaningful priors across architectures using only text conditioning.
- Qualitative Results: PhyCo generates more physically consistent dynamics, including coherent ball trajectories after occlusion and plausible pillow compression under a kettlebell’s weight.These qualitative examples demonstrate improved temporal consistency and contact-induced deformation responses relative to the comparison methods.
- Results from VLM fine-tuning: The fine-tuned VLM predicts four physical attributes with 0.14 mean absolute error and 84.8% binary-response accuracy on 100 held-out PhyCo samples.The evaluated attributes are friction, restitution, deformation, and force.
- Analysis of Flickering Artifacts: Increasing training frame rate mitigates flickering by providing denser temporal supervision, while stronger backbones such as Wan2.2 markedly reduce artifacts and improve motion consistency.These improvements align with lower FVMD scores and act complementarily across training and architecture choices.
E. Limitations
PhyCo improves controllability and physical consistency, but its generated dynamics remain approximations of real physics, with current priors covering mainly simplified rigid- and soft-body behaviors in controlled settings.
- Generated dynamics remain an approximation of real physics rather than an accurate reproduction.
- The physical priors primarily model simplified rigid- and soft-body behaviors in controlled settings.
- Articulated motion, fluid-structure coupling, and multi-contact dynamics remain only partially modeled.