PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation

Chen Wang, Chuhao Chen, Yiming Huang, Zhiyang Dou, Yuan Liu, Jiatao Gu, Lingjie Liu

arXiv:2509.20358v2cs.CV

TL;DR

PhysCtrl addresses the lack of physical plausibility and 3D controllability in modern video generation. It learns material-conditioned physical dynamics as controllable 3D point trajectories and uses them to drive image-to-video models, producing high-quality, physics-grounded results. The method is primarily demonstrated for single-object dynamics across four material types, with only an initial study of multiple objects.

  • Problem

    Existing video generation models can produce photo-realistic videos but often lack physical plausibility and controllability over dynamic physical behaviors.

  • Method

    PhysCtrl uses a conditional diffusion model to generate 3D point-cloud trajectories from physics parameters and external forces, then conditions pretrained video models on those trajectories.

  • Results

    PhysCtrl generates physics-plausible motion trajectories and supports image-to-video generation with high visual fidelity and physical plausibility, including 93.70% vIOU in a preliminary multi-object setting.

  • Takeaways & Limitations

    Physics-grounded trajectory generation provides a controllable signal for producing high-quality videos conditioned on user-specified physical parameters and external forces.

  • Takeaways & Limitations

    The approach mainly focuses on single-object dynamics across four material types, with only an initial study of multiple objects and more complex phenomena left for future work.

Abstract

from arXiv · show

Existing video generation models excel at producing photo-realistic videos from text or images, but often lack physical plausibility and 3D controllability. To overcome these limitations, we introduce PhysCtrl, a novel framework for physics-grounded image-to-video generation with physical parameters and force control. At its core is a generative physics network that learns the distribution of physical dynamics across four materials (elastic, sand, plasticine, and rigid) via a diffusion model conditioned on physics parameters and applied forces. We represent physical dynamics as 3D point trajectories and train on a large-scale synthetic dataset of 550K animations generated by physics simulators. We enhance the diffusion model with a novel spatiotemporal attention block that emulates particle interactions and incorporates physics-based constraints during training to enforce physical plausibility. Experiments show that PhysCtrl generates realistic, physics-grounded motion trajectories which, when used to drive image-to-video models, yield high-fidelity, controllable videos that outperform existing methods in both visual quality and physical plausibility. Project Page: https://cwchenwang.github.io/physctrl

1 Introduction

PhysCtrl targets the missing physical plausibility and 3D controllability of modern video generation by learning physics-grounded dynamics that control image-to-video synthesis. It represents motion as 3D point trajectories and learns material-conditioned dynamics from large-scale simulated animations.

  • Physics simulators provide broad modeling capability but introduce computational cost, hyperparameter sensitivity, numerical instability, and material-specific tuning.
  • The framework explicitly controls generation through physical parameters and external forces using a generative physics network.The network predicts dynamics that serve as control signals for pretrained video models.
  • 3D point trajectories provide compact motion control that generalizes across diverse material types and integrates with video generative models.
  • The diffusion model learns physics-plausible dynamics across elastic, sand, plasticine, and rigid materials from 550K simulated object animations.
  • PhysCtrl generates physics-grounded image-to-video outputs that outperform existing video generative models in visual fidelity and physical plausibility.

2 Related Work

Prior work spans numerical and neural physical-dynamics modeling, controllable video generation, physics-grounded rendering, parametric 4D representations, and diffusion models for object dynamics.

  • Numerical methods and physics-informed neural networks model physical dynamics using simulators, differential-equation approximations, or physics-constrained losses.
  • Controllable video models inject signals such as camera motion, human pose, or point movement into pretrained generators, but often lack physical-law understanding.
  • Physics-grounded video methods reconstruct neural 3D representations, simulate their dynamics, and render the results into videos.
  • Parametric 4D models represent category-specific or arbitrary-topology object dynamics using models such as SMPL, FLAME, MANO, Neural-ODE, and coordinate-MLPs.
  • Diffusion models have increasingly been used to learn distributions of object dynamics from representations including latent vector sets and sparse point-cloud sequences.

3 Preliminary

The preliminary material describes MPM as a continuum-mechanics simulator for deformable particles, with deformation, stress, and particle-grid transfers governing dynamics.

  • Ground-truth point trajectories for PhysCtrl are synthesized with MPM and rigid-body simulators to train its physics-grounded trajectory model.
  • MPM models discrete material particles under continuum mechanics through a deformation mapping from material space to world space.The deformation gradient measures local rotation and stretch.
  • The simulator tracks density, velocity, and external force fields, while the stress term contributes to the governing dynamics.
  • Material behavior depends on the deformation gradient and energy density function, with physics parameters such as Young’s modulus and Poisson’s ratio entering constitutive models.
  • MPM advances simulation through alternating particle-to-grid and grid-to-particle transfer steps.

4 Method

PhysCtrl lifts objects from a monocular image into 3D points, generates physics-conditioned point trajectories with diffusion, and projects them into controls for pretrained video synthesis. Its trajectory model combines spatial-temporal attention with diffusion and physics-based losses to model diverse material dynamics.

  • 4 Method: PhysCtrl lifts objects into 3D points, generates trajectories conditioned on physics parameters and external forces, and uses projected trajectories to guide image-to-video generation.The pipeline begins from a single image and passes generated 3D motion trajectories to a pretrained video model.
  • 4.1.1 Problem Setting: The model represents dynamics as future positions of corresponding point-cloud samples, conditioned on initial geometry, force, drag point, floor height, material, and material parameters.The covered materials are elastic, plasticine, sand, and rigid; the point-cloud representation can extend to other materials with sufficient resources.
  • 4.1 Physics-Grounded Generative Dynamics: A diffusion model learns the conditional distribution of simulator-sampled trajectories, accounting for variability that core physics parameters do not capture directly.Training data comes from MPM and a rigid-body solver, while the method aims to avoid classical simulators’ cost and sensitivity during generation.
  • 4.1.2 Physics-grounded Trajectory Generative model: Each trajectory block applies spatial attention across points within a frame before temporal attention across frames for the same point, reducing spatiotemporal complexity while reflecting particle interactions.The denoising network uses point and space-time positional embeddings, with physical conditions injected as tokens through MLPs.
  • 4.1.3 Training Losses: Training combines diffusion, velocity, physics, and floor-penetration losses to encourage temporally consistent and physically plausible trajectories.The physics loss constrains predicted positions and velocities using deformation-gradient updates, while the floor loss prevents points from passing through the ground.
  • 4.2 Physics-grounded Image-to-Video Generation: Generated 3D trajectories are projected into image-space motion tracks that condition a pretrained video model, enabling qualitative comparisons and videos under varied physical parameters and forces.The method uses DaS with projected trajectories of 2D grid anchor points as conditioning signals.

5 Experiments

PhysCtrl is evaluated for controllable image-to-video generation and physics-grounded trajectory synthesis against video and dynamics baselines. Results favor physics-conditioned 3D trajectories, physics loss, and the proposed trajectory model, while revealing limits under some conditions.

  • 5.1 Evaluation on Image-to-Video Generation: PhysCtrl achieves the best results across baselines on semantic adherence, physical commonsense, and video quality.The evaluation uses GPT-4o 5-Likert scores on 12 generated videos.
  • 5.1 Evaluation on Image-to-Video Generation: PhysCtrl follows force conditions more faithfully than text-only and 2D-trajectory baselines, including complex rotation and depth changes.DragAnything can confuse camera and object motion, while ObjCtrl-2.5D and DragAnything struggle with coarse control.
  • 5.1 Evaluation on Image-to-Video Generation: Changing Young’s modulus, force direction, or force amplitude produces different motions for the same object under varied physical conditions.The study reports negligible influence from Poisson’s ratio ν in the tested setting.
  • 5.2 Evaluation on Generative Dynamics: The trajectory model achieves the best performance over all reported metrics and generates physics-grounded, temporally consistent motion sequences.The spatial-temporal attention block uses explicit per-point correspondence; competing models struggle with temporal coherence or detailed deformations.
  • 5.3 Ablation Study: Physics loss improves all trajectory metrics and brings predictions closer to the simulated reference by constraining positions and deformation gradients.The corresponding qualitative comparison is shown in Figure 7.
  • 5.3 Ablation Study: Using the learned trajectory generator for video generation performs on par with a traditional simulator, while adding more points provides no performance gain.This comparison is reported in the video-generation ablation.

6 Discussion

The discussion examines failure modes and a preliminary extension to multiple-object interactions. PhysCtrl shows promising collision handling but remains constrained by thin structures, internal deformations, and video-model priors.

  • Failure Cases: Trajectory generation can fail on thin structures and complex internal deformations, while video synthesis may reject trajectories conflicting with visual priors.Occluded regions can also contain hallucinated content.
  • Extension to Multiple Objects: A preliminary multi-object collision experiment achieves 93.70% vIOU on a held-out test set.The setting contains an object dragged into a cube from varied angles and distances.

7 Conclusion and Limitations

PhysCtrl combines physics-conditioned diffusion over point-cloud trajectories with pretrained video generation for force-controlled, physics-grounded videos. Its demonstrated scope is mainly single-object dynamics across four materials, with broader phenomena left for future work.

  • 7 Conclusion: PhysCtrl learns complex physical deformations with spatial-temporal attention and physics-based supervision, then uses generated trajectories to condition pretrained video models.The framework supports conditioning on external forces and physics parameters.
  • 7 Limitations: The current approach focuses mostly on single-object dynamics across four material types and does not cover all possible materials.The authors identify multi-object and more complex real-world phenomena as future directions.

A Implementation Details

The implementation builds simulated point-trajectory data, reconstructs input objects as point clouds, and evaluates videos with GPT-4o and a user-study table. The reported dataset and training setup support reproducible evaluation details.

  • Dataset: The dataset contains 550K simulated objects spanning elastic, sand, plasticine, and rigid materials, with sampled forces and physical parameters.Animations use 2048 points and 24 frames; elastic simulations vary force direction and parameters E and ν.
  • Training: The base model is trained for 60K iterations on eight NVIDIA L40 GPUs using a 150K elastic subset and a batch size of 32.This configuration takes about 30 hours.
  • Image-to-3D Pipeline: The image-to-3D pipeline segments the object, generates novel views, reconstructs 3D Gaussians, and samples a point cloud for trajectory generation.It uses SAM, SV3D, LGM, and farthest-point sampling.
  • GPT-4o Evaluation: GPT-4o evaluates semantic adherence, physical commonsense, and video quality using 5-Likert scores, with results also reported through a user study.The evaluation prompt compares five sets of generated videos from the same input.

B User Study

The user study evaluates physics plausibility and overall video quality across five methods, while the physics-parameter pipeline estimates conditions from trajectories.

  • User Study: 35 respondents evaluated 12 questions, selecting the best video for physics plausibility and overall visual and temporal quality.Each question included an input image, force annotation, text prompt, and outputs from five methods.
  • User Study: PhysCtrl consistently outperformed baselines in both physics plausibility and video quality.Wan achieved the second-best video quality, but some of its high-quality videos had low physics plausibility.
  • Physics Parameter Estimation: The trajectory model estimates physical conditions by minimizing denoising discrepancy between noisy ground-truth trajectories and reconstructed trajectories.The denoiser remains frozen while the condition c is optimized through gradients of the energy function.

D More Results

Additional materials emphasize qualitative video comparison, while the paper describes broader applications and warns that physically plausible generation can support deceptive content.

  • More Results: Additional qualitative comparisons between PhysCtrl and baselines are provided, with videos recommended because isolated frames cannot fully represent physical dynamics.The supplementary comparison is presented in Figure 9.
  • Positive Impacts: PhysCtrl’s physically grounded simulation signals can support rapid prototyping by filmmakers, designers, and amateurs.The stated application scope emphasizes accurate physical behavior and high-fidelity visual tools.
  • Negative Impacts: Physically conditioned, high-fidelity generation may be misused to create realistic fabricated disaster footage or other deceptive videos.The paper notes risks to misinformation and public trust, emphasizing that outputs are not real-world occurrences.

F Data and Model Safeguards

The safeguards discussion combines planned release controls with the synthetic nature of training data and examples illustrating controllable motion generation.

  • Safeguards: The authors plan controlled model access through usage guidelines, a research-only license, and investigation of automatic safety filters.These measures are proposed for releasing the pretrained model because of its potential misuse for deceptive videos.
  • Data: The training data contains only synthetic point-cloud trajectories from simulated object motion, with no images, videos, or human-related content.The data therefore consists of geometric and physical information generated in simulation environments.
  • Examples: The examples illustrate motion prompts involving an upward force on a blanket and clockwise rotation of a starfish.These examples are presented as isolated motion descriptions rather than results or quantitative evaluations.
  • More Results: Figure 9 provides further qualitative comparisons between PhysCtrl and baseline methods.The figure is referenced as an additional source of method-comparison results.

NeurIPS Paper Checklist

The checklist documents claims, limitations, reproducibility, experimental-setting constraints, societal impacts, and submission requirements for the paper.

  • Societal Impact: The paper discusses societal impacts, including beneficial prototyping uses and potential misuse for misinformation, surveillance, privacy violations, or security harms.The checklist requires authors to consider direct paths to negative applications and explain their societal-impact assessment.
  • Claims: The paper states that its abstract and introduction claims are validated by quantitative and qualitative experimental results.The checklist asks whether those claims accurately reflect the paper’s contributions and scope.
  • Limitations: The paper states that limitations are provided in its last section, while the checklist calls for assumptions, robustness, and claim scope to be discussed.The guidelines specifically mention violations of assumptions and dependence on datasets, runs, or operating conditions.
  • Theory Assumptions and Proofs: The paper reports no theoretical results, so the theory-assumptions-and-proofs checklist item is not applicable.The justification explicitly says the paper does not include theoretical results.
  • Reproducibility and Open Access: The authors state that supplemental material contains necessary reproducibility details and that training code and checkpoints will be released.They also state that code was not provided during submission and will be open-sourced after acceptance.
  • Experimental Setting: The authors report that experiments require eight Nvidia L40 GPUs for nearly two days per experiment, making repeated runs for error bars computationally expensive.The checklist guidelines request error bars, confidence intervals, or statistical tests when feasible.
Loading 2509.20358v2…