Source-linked AI summary
PhysFlow: Physics-Aware Optical Flow for Motion Controllable Video Generation
Cong Wang, Hanxin Zhu, Yonglin Tian, Jiayi Luo, Ruiqi Song, Boyi Sun, Long Chen, Zhibo Chen
TL;DR
Generated videos may look compelling yet show physically implausible dynamics, including deformation and interaction errors. PhysFlow generates physics-aware optical flow and uses it to guide appearance synthesis. The authors report stronger physical-realism performance and text alignment than competing approaches.
Problem
Video generators often lack robust understanding of physical principles and can produce implausible dynamics, including inconsistent deformation, unrealistic collisions, and incorrect environmental interactions.
Source: §I · p. 1Method
PhysFlow separates physics-aware optical-flow generation conditioned on motion and material attributes from flow-guided video generation, and constructs PhysVideo for training.
Source: §I · p. 2Results
PhysFlow surpasses all competing approaches on the physics realism metric while maintaining strong alignment with the input text.
Source: B. Comparisons with State-Of-The-Art… · p. 7Takeaways & Limitations
The conclusion reports superior motion consistency and visual quality compared with prior methods for rigid, elastic, and flexible-body dynamics.
Source: §V · p. 11The stated scope covers rigid-body motion, elastic deformation, and selected flexible-body behaviors.
Source: §I · p. 2
Abstract
from arXiv · showhide
Video generation models have recently attracted substantial attention for their ability to generate visually compelling videos, yet ensuring physically consistent and plausible dynamics still remains a fundamental challenge, driving a growing line of research on physical realism in video generation. To address this challenge, motivated by the fact that physical regularities are primarily encoded in motion patterns, we propose PhysFlow, a novel two-stage framework for improving the physical plausibility of generated videos by decomposing video generation into motion-aware optical flow generation followed by motion-conditioned appearance synthesis. Specifically, PhysFlow consists of a physics-aware optical-flow video generator called PA-Flow and a flow-guided video generator called FlowRender. During the first stage, PA-Flow employs a physics-aware attention module to model how motion attributes and material properties influence global motion and local deformation, respectively, and generates an optical flow video as an explicit representation of motion. In the second stage, FlowRender leverages the decoupled motion representation as guidance to synthesize realistic textures and appearances, ultimately producing the final physically plausible video. To further support model training with explicit physical supervision, we construct PhysVideo, a physics-based video dataset generated with a physics engine and 3D-GS rendering, containing 10K foreground objects and 50K realistic video sequences with annotations of motion and material properties. Extensive experiments demonstrate that our proposed PhysFlow generates videos with superior physical plausibility while maintaining high visual fidelity compared with existing methods.
I. INTRODUCTION
Video generators can produce visually compelling, temporally coherent videos but often fail to capture physically plausible dynamics. PhysFlow addresses this by separating physics-aware optical-flow generation from flow-guided appearance synthesis, with a dataset and experiments supporting the approach.
I. INTRODUCTION
- Generated videos can exhibit inconsistent deformation, unrealistic collisions, and incorrect environmental interactions, limiting use cases requiring accurate physical reasoning [9]–.Video generators have advanced in visual quality and temporal coherence [1]–[6], but often lack robust understanding of physical principles [7], [8].Source: §I · p. 1
- Existing physics-aware methods still struggle to combine dense motion control, realistic appearance synthesis, and robust object-background or multi-object interaction from a single image.Text controls can be too coarse for dense deformation, while trajectory controls often depend on foreground 3D structures and sparse motion points.Source: §I · p. 2
- Optical flow provides an appearance-agnostic, pixel-level motion representation suited to local deformation, boundary-aware motion, and coupled interactions.The paper motivates this representation as an alternative to simulator states or sparse 3D point trajectories.Source: §I · p. 2
- PhysFlow separates video synthesis into physics-aware optical-flow generation and flow-guided video generation, disentangling motion from texture synthesis.PA-Flow conditions flow sequences on motion attributes and material properties, using motion attributes for global patterns and material properties for local deformation.Source: §I · p. 2
- FlowRender uses the optical-flow video as motion guidance to synthesize RGB sequences with realistic textures.The framework focuses on rigid bodies, elastic deformation, and selected flexible bodies.Source: §I · p. 2
- PhysVideo contains 10K foreground objects and 50K realistic video sequences annotated with motion attributes and material properties.The dataset is built using a physics engine and 3D-GS rendering.Source: §I · p. 2
- PhysFlow generates videos with more physically plausible dynamics than existing methods.The paper reports extensive experimental results supporting this comparison.Source: §I · p. 2
II. RELATED WORKS
Prior controllable video generation uses visual or geometric cues, while physics-engine methods simulate object states from 3D representations and physical properties. PhysFlow’s pipeline instead generates physics-aware optical flow and then uses it to guide video synthesis.
A. Controllable Video Generation
- Video-generation controls include camera motion, point trajectories, anchor-frame videos, and control adapters, but are usually visual or geometric rather than physical-cause cues.Optical flow, depth videos, transition frames, and textual motion descriptions have also been used to guide motion-aware synthesis.Source: A. Controllable Video Generation · p. 2
B. Physics-Engine-Based Video Generation
- PhysFlow’s pipeline first generates a physics-aware optical-flow video, then generates a plausible video guided by that flow.The two stages are detailed in Sections III-A and III-B, respectively.Source: Figure 2 · p. 3
- Physics-engine approaches simulate 3D object states using material and motion properties, but depend on valid 3D representations and specified simulation conditions.The cited methods use solvers including MPM and PBD; the passage notes limits from reconstruction quality and the need to convert 3D priors into physically valid simulation states.Source: B. Physics-Engine-Based Video Generation · pp. 2–3
C. Physics Enhanced Video Generation
Existing physics-enhanced methods use forces, simulators, point-cloud trajectories, or learned dynamics, yet retain dependencies on simulation or 3D priors. PhysFlow generates physics-conditioned optical flow and renders appearances from it, with PhysVideo and a training strategy supporting the method.
C. Physics Enhanced Video Generation
- Physics-enhanced methods include force conditioning, simulator-based refinement, sparse point-cloud trajectories, and learned governing equations, but may depend on simulators, high-quality 3D priors, or single-scene fitting.The cited discussion identifies difficulty jointly accounting for motion and material properties.Source: C. Physics Enhanced Video Generation · p. 3
III. METHODOLOGY
- For rigid-body motion, elastic deformation, and selected flexible-body behaviors, PhysFlow conditions optical-flow generation on motion and material attributes, then renders appearances guided by the flow.PA-Flow generates global motion and local deformation; FlowRender synthesizes realistic appearances from the generated optical flow.Source: §III · p. 3
- PAAM combines global-motion attention with local-deformation attention within each block.These components are described as Motion-Attention and Deform-Attention.Source: Figure 3 · p. 4
- The method uses PhysVideo, constructed with a physics engine, and introduces a training strategy for PA-Flow and FlowRender.The methodology identifies dataset construction and training as separate parts of its pipeline.Source: §III · p. 4
A. Stage I: Physics-Aware Optical Flow Generation
PA-Flow generates optical-flow sequences conditioned on motion and material attributes, using separate attention modules to model global motion and local deformation. It also uses segmentation and multi-view object priors to guide flow generation.
A. Stage I: Physics-Aware Optical Flow Generation
- PA-Flow conditions optical-flow generation on an input image, textual prompt, material parameters, and motion parameters to represent physics-grounded dynamics.The motion parameters are acceleration and velocity; material parameters are supplied as a set.Source: A. Stage I: Physics-Aware Optical Flow… · p. 4
- Grounded-SAM masks and Trellis multi-view object priors guide PA-Flow to preserve semantic and appearance consistency.The priors are encoded and fused with the input-image latent to form the conditioned latent.Source: A. Stage I: Physics-Aware Optical Flow… · p. 4
- PAAM processes global motion attributes first and material-dependent local deformation second, with separate attention modules for each.Velocity and acceleration condition global motion; density, Young’s modulus, and Poisson’s ratio condition local deformation.Source: A. Stage I: Physics-Aware Optical Flow… · p. 4
- Feed-forward refinement further strengthens material sensitivity after local-deformation attention.The local-deformation attention output is refined by a feed-forward layer.Source: A. Stage I: Physics-Aware Optical Flow… · p. 4
- The physics-aware latent is processed with 3D full attention and a feed-forward network to predict noise for optical-flow latent denoising.The denoising processor calculates the final predicted optical-flow latent.Source: A. Stage I: Physics-Aware Optical Flow… · pp. 4–5
B. Stage II: Flow-Guided Plausible Video Generation
FlowRender uses generated optical flow as motion guidance for synthesizing RGB video. It is trained on real-world video data and further fine-tuned on physics-focused data to generate visually plausible videos that follow the flow guidance.
B. Stage II: Flow-Guided Plausible Video Generation
- FlowRender conditions video synthesis on generated optical flow, training a CogVideoX-I2V-5B-based model with supervised fine-tuning on OpenVid and further fine-tuning on WISA.The WISA fine-tuning is used to enhance generation of complex physics-related phenomena.Source: B. Stage II: Flow-Guided Plausible… · p. 5
- FlowRender denoises its predicted noise to calculate the final output video latent.This denoising step is part of producing the output video.Source: B. Stage II: Flow-Guided Plausible… · p. 5
- The output is RGB video synthesized under optical-flow motion constraints, with training aligning implied motion and visual content to support temporal coherence and physically consistent object dynamics.The stage also addresses motion's effects on the surrounding environment, although explicit foreground-background interaction modeling remains challenging.Source: B. Stage II: Flow-Guided Plausible… · p. 5
C. PhysVideo Dataset
PhysVideo supplies physics-grounded videos with realistic backgrounds and recorded motion and material conditions. The training pipeline regularizes PA-Flow motion and trains FlowRender to reconstruct video from flow guidance.
C. PhysVideo Dataset
- PhysVideo contains 10K foreground objects and 50K physics-grounded videos, with realistic 3D backgrounds and recorded motion and material parameters.The dataset uses TaiChi with an MPM solver; optical flow and semantic masks are estimated during post-processing.Source: C. PhysVideo Dataset · p. 5
D. Training Pipeline
- PA-Flow training combines a base SFT loss with velocity-smoothness, acceleration-consistency, and elastic-strain regularization.The velocity term penalizes abrupt temporal flow changes, the acceleration term aligns predicted and ground-truth flow derivatives, and the elastic term penalizes non-physical local deformation.Source: D. Training Pipeline · pp. 5–6
- FlowRender is trained by minimizing denoising error against ground-truth frames while conditioning video reconstruction on optical-flow guidance.The objective is intended to guide temporally coherent and visually realistic sequences that follow the flow-based motion guidance.Source: D. Training Pipeline · p. 6
A. Experimental Setups
The evaluation uses 60 image-prompt-physics samples spanning rigid, elastic, and selected flexible objects, with complex interactions and real-world images. It compares simulation-driven and learned baselines using objective metrics and human ratings.
A. Experimental Setups
- The 60-sample evaluation set covers rigid, elastic, and selected flexible objects, complex-background contact, multi-object interaction, and real-world natural images.Each sample pairs a 512 × 512 image with a textual prompt and physical properties provided by GPT-4o.Source: A. Experimental Setups · p. 6
- Baselines include physics-engine methods, general-purpose video generators, and physics-enhanced generators under the same image and prompt conditions.The comparison includes PhysGen, PhysGen3D, OmniPhysGS, CogVideoX, Wan, Force Prompting, and PhysCtrl.Source: A. Experimental Setups · p. 6
- Evaluation combines VideoPhy-2, WorldScore, VBench, VBench-2.0 Mechanics, GPT-4o ratings, and a human physical-plausibility score.The human score aggregates 50 assessments; the metrics cover semantic adherence, physics, consistency, smoothness, and image quality.Source: A. Experimental Setups · p. 7
B. Comparisons with State-Of-The-Art Methods
PhysFlow is reported to improve motion coherence and physical realism while maintaining semantic alignment and visual quality. Qualitative and controlled-property evaluations also show coherent interactions and responses to varied physical settings.
B. Comparisons with State-Of-The-Art Methods
- PhysFlow improves motion coherence and temporal smoothness over prior video generators, matches physics-engine methods on key dynamic metrics, and yields higher visual quality than those methods.The comparison is reported in Table I, whose caption identifies best and second-best results and defines PE as the physics engine.Source: B. Comparisons with State-Of-The-Art… · p. 7
- Table II reports complementary evaluations: GPT-4o assesses Physical, Photo, and Semantic; Mechanics comes from VBench-2.0; and Phys. is based on 50 human assessments.Source: Table II · p. 7
- In qualitative comparisons, PhysFlow captures plausible elastic falling behavior and coherent multi-ball dynamics where several baselines mis-handle deformation, motion, or object interactions.The examples include complex-shape non-rigid objects and multi-object interactions.Source: B. Comparisons with State-Of-The-Art… · p. 8, Figure 6 · p. 8
- Under controlled changes to velocity, Young’s modulus, and density, PhysFlow reflects those conditions in optical flow and rendered videos, with higher stiffness and lower density producing smaller deformations.The physics-conditioned validation set also reports stronger generation performance for PhysFlow across varied physical settings.Source: B. Comparisons with State-Of-The-Art… · p. 8, Table III · p. 9, Figure 10 · p. 9
- PhysFlow supports varied motions, including rigid sliding, cloth deformation, and bouncing, while maintaining coherent motion and plausible contact responses across interacting objects.Figure 7 presents both diverse motion types and multi-object interactions.Source: B. Comparisons with State-Of-The-Art… · p. 8, Figure 7 · p. 8
- On real-world images that differ from synthetic training data, PhysFlow produces plausible motion fields and temporally coherent videos.The authors relate this generalization to separating motion modeling from RGB synthesis through an intermediate flow representation.Source: B. Comparisons with State-Of-The-Art… · p. 8, Figure 8 · p. 8
C. Ablation Studies
Ablations report gains from adding PAAM and physics-focused fine-tuning, show the importance of mask and multi-view priors and motion-regularizing losses, and illustrate how physical conditions affect generated motion. The paper also notes that substantially implausible attributes or priors can produce visually implausible behavior.
Ablation study on key modules
- In an incremental ablation, adding PAAM improves all reported metrics, raising Physical from 0.594 to 0.608 and PC from 3.29 to 3.37.The ablation adds SFT, physical conditioning, PAAM, and Phys-FT in sequence, with each configuration retaining everything in the previous row.Source: C. Ablation Studies · p. 9
Physics-conditioned flow-to-video generation
- Changing initial-velocity direction, Young’s modulus, or density is shown alongside PA-Flow predictions and representative FlowRender frames to illustrate how motion and material conditions appear in video dynamics.The figure compares three initial-velocity directions and separately varies Young’s modulus and density.Source: Figure 11 · p. 10
Ablation on priors
- Removing either the mask or multi-view prior degrades performance, especially physical plausibility; without the mask, motion regions are ambiguous, and without multi-view input, deformations are unrealistic.The multi-view effect is attributed to weaker global 3D awareness.Source: C. Ablation Studies · pp. 9–10, Table V · p. 9
Ablation study on loss functions
- Removing any of Lvel, Lacc, or Lela degrades performance, with Lela's removal causing the largest drop.Lvel penalizes abrupt velocity changes, Lacc inconsistent acceleration, and Lela supplies an elastic-energy prior for deformation regions.Source: C. Ablation Studies · p. 10, Table VI · p. 10
Qualitative ablation of key components
- With inputs held constant, progressive addition of Motion-Attn, Deform-Attn, and Phys-FT illustrates complementary contributions to global motion, local deformation, and overall physical plausibility.Frames are arranged temporally from left to right.Source: Figure 14 · p. 11
Failure case analysis
- Substantially implausible physical attributes or input priors can yield visually implausible behavior, including unintended motion regions or excessive deformation.The examples include incorrect mask or 3D priors and inappropriate Young’s modulus or density values.Source: Figure 13 · p. 10, C. Ablation Studies · p. 11
V. CONCLUSION
The conclusion presents a physics-aware framework that separates object-motion generation from motion-guided texture synthesis and covers rigid, elastic, and flexible-body dynamics. It reports superior motion consistency and visual quality versus prior methods, alongside a physics-engine-based dataset with realistic backgrounds and detailed annotations.
V. CONCLUSION
- PhysFlow addresses object dynamics spanning rigid-body motion, elastic deformation, and flexible-body motion by separating motion generation from motion-guided texture synthesis.Its two stages generate physics-aware optical flow and then use optical flow to guide plausible video generation without physics engines.Source: §V · p. 11
- The authors report superior motion consistency and visual quality compared with prior methods.The conclusion also describes PhysVideo as a large-scale physics-engine-based dataset combining realistic backgrounds with detailed material and motion annotations.Source: §V · p. 11