PaperlayerView exact revision on arXiv

Source-linked AI summary

Streaming Video Generation with Streaming Force Control

Hanhui Wang, Yiming Xie, Haiwen Feng, Zhaoyang Lv, Shenlong Wang, Huaizu Jiang

arXiv:2606.07508v1cs.CV

TL;DR

Existing video-generation controls are limited in physical interaction, temporal force variation, or causal streaming. StreamForce addresses these gaps with a unified force representation, a force-aware distillation pipeline, and causal autoregressive generation, achieving state-of-the-art force-conditioned generation while running at 16.6 FPS.

  • Problem

    Existing approaches do not simultaneously provide natural force interaction and streaming generation, while prior force-conditioned models separate force types, assume fixed forces, or generate offline.

  • Method

    StreamForce combines a unified representation for local and global forces with temporally varying force data and force-aware distillation into a causal autoregressive student.

  • Results

    StreamForce achieves state-of-the-art performance in force adherence and motion realism while running at 16.6 FPS at 832×480 with 0.6-second latency on one H200 GPU.

  • Takeaways & Limitations

    StreamForce supports online, physically plausible interaction with generated videos and moves video models closer to interactive world models.

  • Takeaways & Limitations

    The current formulation represents only image-plane forces and primarily covers mechanical contact forces, bulk wind, and rigid or articulated objects.

Abstract

from arXiv · show

We introduce StreamForce, a streaming video generation framework that enables physically grounded control through continuous force inputs. Unlike prior video models that train separate models for different force types, assume fixed forces, or rely on non-causal processing, StreamForce is a causal and unified model that responds instantly and coherently to both local and global, time-varying forces. To achieve this, we design a unified force representation as a control signal and develop a distillation pipeline for force-controllable video generation. Our model combines autoregressive efficiency with force responsiveness, sustaining stable photometric and dynamic realism. StreamForce runs at up to 16.6 FPS on a single GPU, achieving state-of-the-art performance in both force adherence and motion realism. Project website: https://neu-vi.github.io/StreamForce/

1. Introduction

StreamForce addresses the challenge of interactive force-controlled video by unifying local and global forces, training on time-varying inputs, and enabling causal streaming generation. It achieves real-time operation with state-of-the-art force adherence and motion realism.

  • Motivation: Interactive force control requires models to respond to physically meaningful inputs while video unfolds sequentially for user feedback.The motivating example is applying and adjusting pushes based on observed object motion.
  • Limitations of Prior Work: Trajectory control specifies where objects move but cannot express how the same force produces different outcomes for different objects.A push may move a heavy object slowly and a light object quickly.
  • Limitations of Prior Work: Prior force-conditioned generation separates local and global forces, trains only on fixed forces, and uses offline bidirectional diffusion.These choices increase modeling cost, omit temporal force transitions, and require all forces to be specified before generation.
  • StreamForce: StreamForce uses a shared pixel-aligned masked force map to represent local and global forces within one model.The unified representation is designed to support shared dynamics across force types.
  • StreamForce: StreamForce combines a temporally varying force-conditioned dataset with force-aware distillation from a bidirectional teacher to a causal autoregressive student.The distillation process preserves force-motion correspondence during streaming generation.
  • Results: 16.6 FPS at 832×480 with 0.6-second latency on one H200 GPU accompanies state-of-the-art force adherence and motion realism.The reported system supports online interaction by letting users apply forces and immediately observe resulting motion.

2. Related Work

Prior video-generation systems largely use bidirectional diffusion or trajectory conditioning, while StreamForce targets force-conditioned autoregressive generation based on underlying physical dynamics.

  • Bidirectional Video Generation: Bidirectional video generators jointly denoise frames using past and future temporal context.The related systems include space-time U-Nets and DiT-style transformers.
  • Trajectory Control: Trajectory-conditioned autoregressive work supports control through track-based conditioning rather than direct force-based physical interaction.The paper distinguishes trajectories from forces because trajectories describe motion while force control targets the dynamics governing object behavior.

3. Method

StreamForce combines a unified force representation with a two-stage teacher-to-student pipeline for causal, autoregressive video generation under dynamically changing forces.

  • Causal Distillation: A bidirectional teacher first learns force-conditioned motion dynamics before its behavior is transferred to a causal autoregressive student.This control-before-causality strategy addresses the teacher’s full-sequence conditioning and the student’s sequential generation setting.
  • Unified Force Representation: A shared pixel-aligned force map represents both local and global forces for one model.The representation distinguishes force types through a spatial mask and encodes force information over time.
  • Force-Conditioned Dataset: Synthetic Blender videos provide supervision linking simulated wind and point forces to resulting object motion.The dataset includes objects with different shapes and materials subjected to physically simulated forces.
  • Causal Distillation: ODE-based initialization uses teacher-generated trajectories and independently sampled timesteps so latent chunks can begin from different noise levels.The student is initialized from denoised teacher latents using a regression objective.
  • Causal Distillation: Distillation combines synthetic force-conditioned trajectories with diverse image-force trajectories, preserving controllability while supporting open-domain visual generalization.Self-Forcing DMD then trains the causal student on the resulting chunk-wise autoregressive process.

4. Experiments

STREAMFORCE is evaluated against representative video-generation baselines for perceptual controllability, realism, and physics-aware motion across preserved and changing global and local forces. Ablations show that unified representations, joint training, diverse supervision, and force-changing data support generalization and responsiveness.

  • Main Results: Across global and local scenarios, STREAMFORCE predominantly achieves the highest scores across three perceptual criteria under both force-preservation and force-changing settings.Improvements are particularly pronounced when force control signals change.
  • Main Results: STREAMFORCE more faithfully follows input-force dynamics than baselines while maintaining stable appearance and plausible physicality.Qualitative comparisons include recorded force-conditioned videos and x-t space-time slices.
  • Main Results: STREAMFORCE achieves the highest aggregated Physics-IQ score and leads on spatiotemporal IoU and motion-error MSE across global and local force scenarios.Force-Prompting performs better on pixel-level Spatial IoU and Weighted Spatial IoU.
  • Ablation Studies: The unified pixel-aligned masked force-map representation improves performance on both global and local force scenarios within one framework.Joint training across force types achieves stronger overall performance than separate models.
  • Ablation Studies: Removing diverse distillation data reduces motion-pattern variety and adaptability, while removing force-changing supervision makes the model largely ignore force updates.These ablations indicate that both broad visual supervision and changing-force examples are important for streaming controllable generation.
  • Ablation Studies: Under the same horizontal force, the milk-filled glass moves more slowly than the empty glass, matching the expected mass–acceleration relationship.This qualitative test examines mass-aware motion behavior.

5. Discussion

StreamForce exhibits coarse intuitive-physics behavior and supports coordinated manipulation through multiple local forces, while its force responses are not quantitatively faithful.

  • Emergent Intuitive Physics: The same force produces shorter displacement for heavier objects and rougher surfaces, reflecting coarse intuitive-physics cues.The paper cautions that these observations do not establish a quantitatively faithful physics model.
  • Object Falling: An object pushed past a table edge follows a downward trajectory consistent with gravity.This gravitational behavior emerges from spatiotemporal priors inherited from the pretrained video model rather than explicit conditioning.
  • Multi-Force and Part-Level Interaction: Two simultaneous local forces applied to different parts of a T-shaped object produce coordinated translation and rotation toward a target.This demonstrates part-level manipulation with multiple forces.
  • Differences from trajectory-control based works: Force-based control represents interactions that trajectories alone cannot naturally describe, including global forces and effects dependent on mass or material.The same force can produce different displacement in lighter and heavier objects.

6. Conclusion

StreamForce combines continuous force interaction with a unified streaming generation framework and is evaluated for overall quality, magnitude sensitivity, and responsiveness to changing forces.

  • 6. Conclusion: StreamForce combines a unified force representation with a force-aware distillation pipeline in a streaming model supporting global, local, and time-varying forces.The framework bridges force-controllable video generation with causal autoregressive synthesis.
  • 6. Conclusion: Three perceptual user studies assess overall comparisons, sensitivity to force magnitude, and responsiveness to changing forces.Participants compare videos generated under identical input conditions through an online survey interface.
  • 6. Conclusion: 40 cases across global, local, changing-global, and changing-local force scenarios were evaluated by 26 participants in the overall study.Each scenario category contains 10 evaluation cases.
  • 6. Conclusion: Ours more accurately reflects intended motion differences between smaller and larger forces than the baselines.This result is reported for the magnitude-response perceptual study.
  • 6. Conclusion: Changing-force responsiveness is evaluated by comparing the final model with an ablated variant trained without force-changing data.The applied force changes during generation in this study.

B. Physics-IQ Benchmark Details

The Physics-IQ evaluation is adapted to force-conditioned video using a purpose-built real-world benchmark while preserving the original metric computation protocol.

  • B. Physics-IQ Benchmark Details: The authors construct a force-conditioned evaluation set tailored to explicit force inputs while keeping Physics-IQ metric computation unchanged.The original cases primarily target text-to-video physical reasoning.
  • B. Physics-IQ Benchmark Details: 40 real-world five-second sequences cover global forces from a fan and local forces applied by dragging or poking objects.Sequences are recorded at 60 FPS with an iPhone and uniformly subsampled to 80 frames.
  • B. Physics-IQ Benchmark Details: The evaluation uses 832×480 generated videos over 80 frames, corresponding to 5 seconds at 16 FPS.Example recorded sequences are shown in Fig. A5.
  • B. Physics-IQ Benchmark Details: Each initial image and force input produces 5 videos with different random seeds, and mean Physics-IQ scores are reported separately for global and local forces.Results appear in Tables 2 and 3 of the main paper.
  • B. Physics-IQ Benchmark Details: The released Physics-IQ code computes all metrics after minimal adaptation for the authors’ ground-truth and generated videos.Metric definitions and computation procedures remain unchanged.

C.1. Synthetic Data for Bidirectional Teacher Training

Synthetic force-conditioned videos are generated in Blender, extended with force changes, and paired with motion supervision to train the bidirectional teacher model.

  • C.1. Synthetic Data for Bidirectional Teacher Training: The bidirectional teacher is trained on synthetic force-conditioned videos generated with the Force-Prompting data pipeline.The pipeline uses Blender to simulate object motion under controlled physical forces.
  • C.1. Synthetic Data for Bidirectional Teacher Training: Global-force data applies wind fields to cloth flags, while local-force data pokes a ball on a plane.These interactions represent wind-driven and localized motion, respectively.
  • C.1. Synthetic Data for Bidirectional Teacher Training: Force-changing data updates global wind direction and magnitude or applies a second local poke with different direction and magnitude.The update occurs at a specified time step to model dynamic force conditions.
  • C.1. Synthetic Data for Bidirectional Teacher Training: The resulting dataset pairs initial images and force inputs with corresponding motion trajectories for force-consistent teacher training.Example synthetic data are shown in Fig. A6.

C.2. Diverse Data for Causal Distillation

The causal distillation pipeline uses diverse real-world image–force pairs spanning global and local interactions, with annotations and filtering to support varied motion generation.

  • Real-world images are collected through Pexels to preserve visual diversity during causal distillation.The dataset avoids relying solely on synthetic Blender scenes, which may limit object categories and scene layouts.
  • Global forces are randomly sampled across direction and magnitude, while local forces target annotated movable objects with directional arrows.Local annotations record the object name, click location, and arrow direction.
  • Qwen3-VL filters generated samples whose videos remain largely static or implausible after teacher-model generation.The filtering step removes cases where randomly constructed image–force combinations fail to produce meaningful motion.
  • Global-force captions require acknowledging wind while omitting its direction, strength, speed, magnitude, and quantitative comparisons.The prompts allow only neutral statements about wind affecting the scene.
  • Caption prompts require descriptions to acknowledge externally caused object motion while excluding force direction, strength, magnitude, and quantitative details.The local-force captioning instructions describe motion neutrally without specifying object location or force parameters.

D.1. Bidirectional Teacher Training

Teacher training initializes force-conditioned generation from a pretrained bidirectional video model, after which causal distillation and rolling-window inference support streaming generation.

  • The bidirectional teacher is initialized from pretrained Wan2.2 TI2V, with a copied ControlNet branch and frozen backbone during teacher training.Only the ControlNet parameters are optimized on the synthetic force-conditioned dataset.
  • ODE initialization trains a causal student from teacher-generated trajectories using causal temporal attention masks.Unlike teacher training, this stage optimizes both the backbone and ControlNet parameters.
  • Self-Forcing distillation jointly optimizes the causal generator and a critic initialized from the teacher under a DMD-style objective.The generator and critic use AdamW with learning rates of 2 × 10−6 and 4 × 10−7, respectively.
  • Rolling-window inference jointly denoises consecutive frames while maintaining temporal and global context caches.The rolling diffusion window uses local bidirectional attention to support long-horizon generation.

F. Limitations

StreamForce is limited by its 2D force representation, restricted force and material coverage, and incomplete generalization to multi-contact physical interactions.

  • The force representation models image-plane forces but does not explicitly represent depth-direction interactions.Out-of-plane and fully three-dimensional physical interactions are therefore outside the explicit representation.
  • The training scope covers mechanical contact forces and bulk wind, excluding magnetic, electrostatic, buoyant, and other field-based interactions.Multi-force and part-level interactions remain limited to special cases rather than arbitrary objects or contact-point counts.
  • Broader force and material coverage would require training data spanning additional interaction locations, force types, contact configurations, and physical dynamics.The stated goal is a more comprehensive physically grounded generation system.
  • Local-force training predominantly targets rigid or articulated objects, not fluids, elastic materials, deformable objects, or granular media.For unsupported materials, local forces may produce plausible rigid-body translation instead of richer fluid or deformable responses.
Loading 2606.07508v1…