Source-linked AI summary
PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control
Chuhao Chen, Peter Wonka, Chaoyang Wang, Chen Wang, Qiao Feng, Sergey Tulyakov, Lingjie Liu
TL;DR
Existing controllable video methods often require prespecified schedules or pixel-space controls that dictate positions rather than physical dynamics. PhysStream combines sparse velocity-increment control with online structured scene memory in an autoregressive image-to-video model, improving benchmark motion metrics and human preference while targeting interactive control in tabletop rigid-body scenes.
Problem
Existing controllable methods may require the full control schedule in advance or use pixel-space signals that dictate object positions instead of encoding physical dynamics.
Method
PhysStream uses sparse velocity-increment signals together with online positional and object-tracking maps from previously generated frames in a two-stage autoregressive model.
Results
PhysStream consistently improves physics-related consistency and motion-control adherence across synthetic, real-world, and long-horizon settings over recent controllable baselines.
Takeaways & Limitations
PhysStream supports direct mid-generation, scene-level physics-grounded control in multi-object tabletop rigid-body scenes.
Takeaways & Limitations
The validated scope is limited to rigid-body dynamics; extremely complex motion such as tumbling remains difficult, and real-time generation is future work.
Abstract
from arXiv · showhide
Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics. To address these limitations, we propose PhysStream, an autoregressive model for physics-grounded image-to-video synthesis that incorporates structured scene memory---positional maps and object tracking maps derived online from previously generated frames---and supports fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics. We train our model in two stages: a bidirectional model is first finetuned with motion-control conditioning, then a causal autoregressive model is trained with additional structured scene memory, further improving physical consistency. PhysStream enables interactive, mid-generation control over multi-object tabletop rigid-body scenes---a capability not supported by prior methods---reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines on synthetic benchmarks, and is preferred by human evaluators in over 85% of in-the-wild comparisons. Please check our website for more details: https://czzzzh.github.io/PhysStream
1 Introduction
PhysStream targets interactive, physics-grounded, scene-level video control by combining sparse physical signals with online structured scene memory. It is designed to address gaps in prior methods while improving physical consistency and motion-control adherence.
- 1 Introduction: The approach is motivated by the need for frame-by-frame intervention rather than requiring a complete control schedule before generation.Interactive control here means users can observe partial results and intervene on the fly, without requiring real-time throughput.
- 1 Introduction: Prior controllable methods are compared by sparsity, physics grounding, interactivity, and scene-level control, with RealWonder relying on an intermediate simulator.That simulator’s scene state may diverge from the synthesized video.
- 1 Introduction: The model uses sparse velocity-increment maps and online positional and object-tracking maps derived from previously generated frames.These conditions are updated after each generated frame during causal autoregressive generation.
- 1 Introduction: The paper reports improvements in motion-control adherence and physical plausibility, supported by a synthetic dataset of complex multi-object rigid-body motion.The dataset includes collisions and multi-frame velocity perturbations.
- 1 Introduction: PhysStream introduces direct, end-to-end, scene-level physics-grounded interactive control for multi-object tabletop rigid-body video generation.Both user inputs and scene memory operate on the generated video itself.
2 Related Work
Prior controllable video methods use diverse auxiliary signals and physical or geometric proxies, but PhysStream distinguishes itself by estimating structured scene memory directly from generated frames without an external simulator or inference-time trajectory.
- 2 Related Work: Controllable video generation uses depth, camera trajectories, object-level interactions, and physical signals to steer video synthesis.These approaches target global motion, object motion, or physical interactions through auxiliary conditions.
- 2 Related Work: Some prior work uses geometric consistency or 3D-aware world models as indirect proxies for physical plausibility.Examples include depth or normal regularization and 3D-aware world models.
- 2 Related Work: PhysStream avoids external simulators and inference-time trajectories by conditioning on structured scene memory estimated online from the generated video.This design explicitly uses scene information rather than imposing consistency only through an implicit loss.
3 Method
PhysStream generates video autoregressively using sparse user velocity increments and online structured scene memory derived from previously generated frames. Its two-stage design combines motion-conditioned bidirectional finetuning with causal generation, online memory updates, and shifted condition channels that preserve temporal causality.
- Stage 2: Autoregressive Generation with Structured Scene Memory: The model uses a normalized positional map and an object-tracking map as history-derived scene memory for each synthesized frame.Positional maps are estimated from recent frames using depth and tracking maps are produced from previously decoded frames with an online tracking estimator.
- Stage 1: Bidirectional Generation with Motion Control: Velocity-increment maps encode sparse, object-level 3D velocity changes that users can inject at arbitrary frames before synthesis.Events are normalized, painted onto selected objects’ first-frame masks, and leave unselected pixels at a neutral value.
- Two-Stage Training: Training first finetunes a bidirectional video model with velocity control, then converts it into a causal model with structured scene memory.Stage 2 uses teacher forcing and causal attention because the new scene-memory branches are unavailable to a bidirectional teacher.
- Stage 1: Bidirectional Generation with Motion Control: First-frame masks anchor velocity events during training, while inference lets users select objects at their current generated-video positions and maps interactions internally to those masks.This convention keeps the control representation fixed even as objects move through the scene.
- Stage 2: Autoregressive Generation with Structured Scene Memory: PhysStream samples each frame from prior video history, velocity-increment signals, and positional and tracking memories estimated from generated frames.The online estimators update scene memory after each committed latent frame, which is then used for the next generation step.
- Condition Injection via Shifted Channel Concatenation: A one-frame temporal shift makes each condition slot depend on the previous latent frame, preventing current-frame information from leaking into its own conditioning.The shifted channels preserve causal ordering while cached features and incrementally updated memories support streaming generation.
4 Experiments
PhysStream is evaluated on synthetic, in-the-wild, real-world, non-rigid, long-horizon, ablation, and runtime settings, with results emphasizing physics-sensitive control and physical plausibility. The experiments also expose limitations of consistency metrics and quantify the effects of structured memory and acceleration.
- Synthetic Evaluation: PhysStream outperforms all baselines on synthetic multi-object interactive physics metrics, with near-optimal consistency scores across the board.The comparison covers FVMD, traj-ADE, traj-ADE-median, and failure rate; Table 2 reports the quantitative results.
- In-the-Wild Evaluation: Human evaluators prefer PhysStream in over 80% of in-the-wild comparisons on physical plausibility, motion accuracy, and visual quality.Its Semantic Adherence and Physical Commonsense scores are also the highest among the compared methods.
- Non-Rigid Objects: The same control and structured-memory paradigm transfers to deformable balls and cloth, achieving Semantic Adherence and Physical Commonsense scores on par with rigid-body results.This transfer uses small material-specific synthetic datasets of 10k clips each without changing the method.
- Long-Horizon Generation: PhysStream extends beyond its 49-frame training window to 301-frame, interaction-rich scenes, although consistency gradually decreases because of accumulated appearance drift.The long-horizon benchmark spans six times the training horizon.
- Ablation Study: The full causal model with velocity, positional, and tracking maps performs best or near-best on nearly all ablation metrics, while causal configurations outperform the bidirectional Stage-1 model.The positional map primarily aids geometrically challenging motions, and the tracking map enables completion of sharp zig-zag trajectories.
- Runtime Analysis: A 4-step distilled generator halves latency with less than 1% average metric degradation, while smaller depth estimation and engineering together make the system 3.4× faster.In the unaccelerated system, 50-step denoising accounts for 68% of runtime and Depth-Anything-3 for 19%.
5 Conclusion and Limitations
PhysStream provides autoregressive, physics-grounded interactive video generation for tabletop rigid-body scenes using velocity-increment signals and online structured scene memory. It improves physics-related consistency and motion-control adherence across synthetic, real-world, and long-horizon settings, but remains limited for complex motion and richer materials.
- Across synthetic, real-world, and long-horizon settings, PhysStream consistently improves physics-related consistency and motion-control adherence over recent controllable baselines.
- PhysStream combines sparse user-specified velocity-increment signals with positional and object-tracking maps derived online from previously generated frames.
- PhysStream still struggles with extremely complex motion, particularly tumbling, and its validated scope is limited to rigid-body dynamics.
- Richer materials currently require additional finetuning data, while real-time generation remains future work.
Supplementary Material: PhysStream
PhysStream uses two-stage training with a standard v-prediction flow-matching loss. It first finetunes a bidirectional video model with motion control, then applies differential learning rates across newly added and pretrained components.
- PhysStream is trained in two stages using the standard v-prediction flow-matching loss.
- Stage 1 fully finetunes the Wan2.2-TI2V-5B backbone with a differential learning-rate schedule for the velocity-increment embedding and pretrained layers.
B.1 Main Rigid-Body Dataset
The main rigid-body dataset is generated from indoor 3D scenes through processing, PyBullet simulation, and Blender rendering. It includes aligned visual, physical, positional, tracking, and depth modalities, while additional datasets cover deformable balls and cloth.
- The dataset builds on 10k pre-generated 3D-Front indoor scenes and processes, simulates, and renders them through a three-stage pipeline.
- Each video includes six aligned modalities: RGB, instance masks, velocity-increment canvases, normalized positional maps, object-tracking maps, and inverse depth.
- Velocity kicks are additive changes to current velocity, allowing compounded momentum, tumbling, and multi-object collisions.
- Two additional 10k-clip datasets replace the main scenes and simulator with deformable balls modeled by material points and cloth modeled by mass–spring dynamics.
C First-Frame Mask: Experimental Evidence
The first-frame mask avoids trajectory leakage from bidirectional training and preserves the same condition distribution when the model becomes causal. Experiments show that this choice is inferior in Stage 1 but more robust across the full two-stage pipeline.
- Table 11 evaluates first-frame and per-frame masks across training stages using Traj-ADE, median Traj-ADE, failure rate, FVD, and FVMD.
- In Stage 1, the per-frame mask is superior across every metric, with 33% lower ADE and nearly halved FVMD because it leaks current-position information.
- The first-frame-mask model improves consistently from Stage 1 to Stage 2, whereas the per-frame-mask model degrades on four of five metrics after causal training.
- The first-frame mask anchors velocity-increment events to the initial object position, preventing the condition distribution from shifting between bidirectional and causal training.
D Positional Map: Window Size and Normalization Anchor
The positional map uses recent frames and first-frame normalization, while tracking maps preserve responsiveness to late control signals over long horizons.
- Normalization anchor: Only 0.90% of pixels are clipped and scale-factor deviation averages 0.42%, indicating negligible distortion from first-frame normalization on 64 validation videos.The analysis compares frame-0 axis ranges with full-sequence ranges under the static-camera setting.
- Window size: L=4 and L=49 positional maps are visually indistinguishable across seven sampled frames, supporting the minimal one-latent-frame window.The comparison includes generated RGB, both positional-map windows, and ground-truth positional maps.
- Tracking-map ablation: 90.3% → 85.0%: removing the tracking map after frame 100 lowers response rate for later events, while control accuracy remains within noise.Dropping the map throughout gives 85.0% response for events after frame 100; midway removal gives 87.6%.
- Tracking-map ablation: The tracking map complements positional memory by keeping late control signals effective, with failure degrading responsiveness rather than derailing generation.This interpretation is tied to both the long-horizon ablation and the qualitative zig-zag result.
F Comparison of Autoregressive Training Paradigms
The section defines the evaluation metrics used to compare autoregressive training paradigms, including object consistency, photometric consistency, trajectory accuracy, tracking failure, and motion-distribution distance.
- Consistency metrics: Object consistency compares DINO features on individually cropped and masked dynamic objects, preventing static background content from dominating the score.The final ObjCon averages per-object similarity across dynamic objects and differs from full-frame VBench consistency.
- Photometric metrics: PhotoC measures forward–backward RAFT flow cycle consistency, normalizing average endpoint error to a 0–100 score after excluding a 15-pixel border.The metric is defined as PhotoC = (1 − clamp(AEPE_fb/1.192, 0, 1)) × 100.
- Trajectory metrics: Trajectory ADE averages Euclidean errors between predicted and ground-truth tracked points, while ADE-Median replaces per-object means with medians for outlier robustness.Scores use 32 query points per dynamic object and begin when the ground-truth object starts moving.
- Tracking metrics: Tracking failure counts ground-truth-visible points whose generated tracks are lost or deviate by more than 30 px, using a deliberately strict threshold.The metric reports the fraction of tracked points meeting either failure condition.
- Motion-distribution metric: FVMD compares concatenated velocity-and-acceleration histogram features from generated and ground-truth motion windows using a Fréchet distance.Each 49-frame video contributes 34 overlapping 16-frame windows, with 400 tracked points per window.
G.8 MLLM Evaluation (SA and PC)
The evaluation combines automated semantic and physical-commonsense ratings with human preferences on anonymized in-the-wild comparisons of controlled tabletop videos.
- MLLM evaluation: GPT-4o rates each video from 1–5 for Semantic Adherence and Physical Commonsense using the input image, generated frames, and intended velocity-increment events.The physical-commonsense criterion considers direction, sliding, tumbling, bouncing, friction, collisions, and gravity.
- Human preference study: The human study uses 25 evaluators and 20 in-the-wild cases, with five anonymized videos per case presented in randomized Latin-square order.Each evaluator selects the best video independently for physical plausibility, motion accuracy, and visual quality.
- Human preference study: Human ratings separately assess physical realism, agreement with specified velocity directions and affected objects, and overall visual-temporal quality.The guidance emphasizes collisions, gravity, rigid-body contact dynamics, timing, sharpness, consistency, and artifacts.
- Long-horizon benchmark: Long-horizon metrics evaluate 231 rotating-direction events across five 301-frame multi-object scenes, with results computed per 100-frame segment.Each object receives a velocity increment every 24 frames, and response depends on measurable motion or appearance change.
- Evaluation data: The in-the-wild set contains 20 real photographs or high-quality generated images with randomly sampled target objects and velocity-increment events following training-data rules.The controls are bounded by Vmax to avoid cherry-picking.
I Additional In-the-Wild Results
Additional in-the-wild examples show PhysStream handling diverse real-world scenes with user-specified velocity-increment interactions and physically plausible multi-object dynamics.
- Additional in-the-wild results: PhysStream generalizes to diverse real-world scenes while producing physically plausible multi-object dynamics under user-specified velocity-increment controls.Figure 11 presents input images followed by representative generated frames.