Source-linked AI summary
Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals
Nate Gillman, Charles Herrmann, Michael Freeman, Daksh Aggarwal, Evan Luo, Deqing Sun, Chen Sun
TL;DR
Force Prompting addresses the limited study of physically meaningful force interactions in video generation by conditioning models on local point forces and global wind fields. Using synthetic force-video data and pretrained visual-motion priors, it achieves broad force-response generalization and stronger physical adherence than baselines, while remaining constrained by the base model’s physical understanding.
Problem
Video generation research has insufficiently studied physically meaningful interactions controlled by real-world forces, despite their importance for interactive generation and intuitive world models.
Method
Force Prompting conditions an image-to-video model with local or global force signals derived from synthetic simulator-generated training videos, without requiring a simulator or 3D asset at inference.
Results
Force Prompting generalizes across diverse objects, materials, and scenarios and achieves superior force adherence while maintaining realistic motion and visual quality relative to text-conditioned baselines.
Takeaways & Limitations
The results support force-conditioned video generation as a promising direction for interactive content creation and world models responsive to physical interactions.
Takeaways & Limitations
Force Prompting is constrained by the underlying video prior’s physical understanding and controls existing capabilities rather than improving physics comprehension.
Abstract
from arXiv · showhide
Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments. While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied. In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the visual and motion prior in the original pretrained model, without using any 3D asset or physics simulator at inference. The primary challenge of force prompting is the difficulty in obtaining high quality paired force-video training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity of physics simulators. Our key finding is that video generation models can generalize remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects. Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try to understand the source of this generalization and perform ablations that reveal two key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for a single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. We release all datasets, code, weights, and interactive video demos at our project page.
1 Introduction
Force Prompting introduces local and global physical-force controls for video generation, using synthetic paired data to elicit force-responsive behavior from pretrained visual and motion priors. The method generalizes across diverse scenes and achieves stronger physical instruction adherence while preserving realistic motion and visual quality.
- Contributions: Force Prompting introduces localized point forces and global wind forces as conditioning signals for video generation.The method uses synthetic force-video pairs because real-world force annotations are costly and difficult to collect.
- Results: Force Prompting outperforms text-conditioned baselines in human evaluations of force adherence while maintaining realistic physics and visual quality.The method also outperforms trajectory-controlled video generation according to the introduction.
- Analysis: Ablations identify visual diversity relative to the control signal and specific training-time text keywords as important factors in generalization.These elements appear to help elicit force-control understanding.
- Contributions: 15K synthetic videos and one day of training on four A100 GPUs support broad generalization across settings, objects, geometries, materials, and affordances.The reported generalization includes initial evidence of mass understanding.
- Resources: The project releases datasets, code, and models for force-conditioned video generation.
2 Related Works
Related work spans realistic video generation, controllable motion and camera control, interactive world models, and physics-based video synthesis. Force Prompting differs from trajectory control by representing global phenomena and mass-dependent responses as forces.
- Video generation: Video-generation models have recently demonstrated realistic visual dynamics and motivated their use as simulators of the physical world.
- Controllable video generation: Prior controllable video-generation methods primarily address camera control and motion specifications such as dragging, trajectories, and optical flow.
- Controllable video generation: Motion Prompting uses sparse trajectories for video extrapolation, whereas force control represents wind and material- or mass-dependent responses that trajectories do not encode.
- Interactive world models: Interactive world-model research has focused predominantly on video-game environments, with fewer studies examining real-world interactions.
- Physical simulators and hybrid approaches: Earlier physics-based approaches model vibration-like motion through modal bases but struggle with motions such as linear movement.
3 Method: Force Prompting
Force Prompting conditions an image-to-video model with global wind fields or localized point-force signals, trained on synthetic simulator-generated videos. Distinct encodings represent global magnitude and direction or local application point, magnitude, and direction.
- 3 Method: Force Prompting: The model takes text, an initial frame, and a physics control signal, then generates a video using shared architecture and training procedures for global and local forces.Global forces are wind fields; local forces specify where a point force is applied.
- 3.1 Synthetic training data: Global scenes randomize flag count, color, placement, camera, lighting backgrounds, and wind direction to increase visual diversity.
- 3.1 Synthetic training data: Force vectors and object positions are projected from 3D world coordinates onto the 2D pixel plane, while text prompts describe backgrounds and textures.
- 3.2 Local and Global Force Prompts: The global encoding stores force magnitude and angle as spatially repeated channels containing magnitude, cosine angle, and sine angle.This produces a smooth tensor representation of the wind force field.
- 3.2 Local and Global Force Prompts: The local encoding represents application coordinates with a Gaussian blob whose direction and travel distance encode force angle and magnitude.The blob’s velocity is proportional to force magnitude, so stronger forces produce faster control-signal motion.
- 3.2 Local and Global Force Prompts: Force values are relative rather than absolutely calibrated: larger values correspond to stronger pokes and greater initial displacement.This distinguishes the method from trajectory-based control, which specifies motion rather than force.
- 3.3 Architecture and Training: The force-conditioned model adds a ControlNet to CogVideoX-5B-I2V for processing physics prompts alongside text and the initial frame.
4 Quantitative and Qualitative Results
Force Prompting generalizes force-controlled video generation beyond its limited synthetic training scenarios, producing localized and global interactions across diverse objects and motions. Human evaluations show strong physical adherence and generalization, including comparisons with text and simulation-based baselines.
- The human study compares Force Prompting against text-only zero-shot, text-only fine-tuned, and trajectory-based Motion Prompting baselines.The study uses 2AFC evaluation with N = 10 participants on curated benchmark datasets.
- Force Prompting generalizes from limited synthetic training scenarios to diverse objects, settings, and motion types in both local and global force tasks.The evaluation includes point-force interactions across movement categories and global wind effects across tethered, aerodynamic, and fluid phenomena.
- 41-image global-force evaluation shows control trained only on tethered flag motion generalizes to aerodynamic and fluid dynamics.The base model already generates these motion categories, while force training transfers controllable wind behavior beyond the training distribution.
- Point-force training on a single carnation generalizes to roses, tulips, and alocasia in comparison with PhysDreamer.The comparison uses six plant species and evaluates a purely neural method against a method using 3D assets and an integrated physics simulator.
5 Ablation Studies
Ablations show that synthetic-data diversity and training-time force terminology are important for reliable generalization. Comparisons with PhysDreamer indicate stronger force adherence and comparable motion realism, while visual quality remains a trade-off.
- Force Prompting outperforms PhysDreamer on force adherence and matches it on motion realism, while PhysDreamer retains an advantage in visual quality.Table 2 reports evaluator preferences, with values above 50% favoring Force Prompting.
- 5.1 Ablation Study #1: Composition of Synthetic Dataset: Removing distractor objects harms point-force localization, causing motion to spread beyond the intended target.Without distractor balls, poking one balloon moves all balloons slightly, and poking a rose can move the vase with it.
- 5.1 Ablation Study #1: Composition of Synthetic Dataset: Single-background training reduces visual quality, while single-flag training overfits cloth mechanics and fails to generalize across materials.Observed failures include unaffected smoke, unresponsive or suspended confetti, inert bubbles, and human limbs billowing like cloth.
- 5.2 Ablation Study #2: Text Prompt Specificity: Omitting wind-related keywords during training significantly increases benchmark failures, whereas inference-time keyword presence has less impact.Wind terminology during inference generally produces more robust results, but training-time exposure is more consequential.
6 Mass Understanding
The mass-understanding study tests whether identical forces produce different motion for objects with different implied masses. Quantitative and qualitative evaluations support this physically intuitive relationship across varied scenarios and force magnitudes.
- The study defines mass understanding as producing distinct responses when identical forces act on objects with different apparent masses.Examples include different travel distances for single versus stacked books and different swing responses for wooden versus metal objects.
- In ball-rolling experiments, soccer balls and bowling balls are tested under identical forces across four surfaces and three color variations.The quantitative setup uses automatically measurable rolling distances across dirt, grass, stone, and wood.
- Qualitative tests compare implied mass across ornaments, laundry baskets, book stacks, and cube stacks with geometrically matched objects.Initial frames vary the implied mass while controlling the other visual conditions.
7 Conclusion
Force Prompting enables physically meaningful point and wind controls in generative video models. Limited synthetic training data supports generalization across diverse objects, materials, and scenarios without physics simulation at inference.
- Force Prompting enables localized point forces and global wind effects as physically meaningful controls for generative video models.
- The approach generalizes from limited synthetic training data to diverse objects, materials, and scenarios without requiring physics simulators during inference.
A.1 Training hyperparameters
The experiments specify training and evaluation procedures for force prompting, including model initialization, force magnitudes, control-signal construction, and flower-video human studies.
- A.1 Training hyperparameters: 5000 training steps used bf16 mixed precision, TF32 support, AdamW, and pretrained CogVideoX-5b-I2V weights with a six-layer ControlNet initialization.The six-layer initialization was selected from 42 available transformer blocks due to memory constraints.
- A.1 Training hyperparameters: For ball experiments, forces ranged from 0.125 · n for n ∈ {1, . . . , 8}, with 10 videos generated per force value.GPT-Image-1 first frames maintained consistent initial ball positions and shapes across conditions.
- A.1 Training hyperparameters: The ball control signal uses a Gaussian blob centered on the object, with force-dependent displacement calibrated between w/8 and w/2.The minimum and maximum displacements correspond to force values F = 0 and F = 1.
- A.1 Training hyperparameters: A human study compared six flower videos generated from still frames and equivalent force prompts using side-by-side Qualtrics evaluations.The frames came from PhysDreamer teaser videos and lacked force annotations before prompting.
B.1 Multiple forces for multiple objects, benchmark
The benchmark evaluates whether Force Prompting can apply multiple simultaneous point forces to diverse pairs of objects and respond differently to apparent object mass.
- B.1 Multiple forces for multiple objects, benchmark: The model handles multiple simultaneous forces zero-shot by adding one Gaussian blob to the control signal for each applied force.Tests covered six two-object scenes including ornaments, cars, vases, roses, dandelions, and apples.
- B.1 Multiple forces for multiple objects, benchmark: Four force prompts tested each pair of objects across the 2 × 2 combinations of two reasonable poke directions.The protocol combined same-direction and opposing-direction pokes for each image.
- B.1 Multiple forces for multiple objects, benchmark: With identical force magnitudes, lighter objects moved much farther than heavier objects across eight force magnitudes.The comparison used an empty versus full laundry basket and a book versus a stack of books.
C.1 Failures and Limitations
The paper identifies failures tied to the pretrained video prior and evaluates point-force prompting across diverse scenes drawn from physics-in-the-loop benchmarks.
- C.1 Failures and Limitations: Force Prompting remains constrained by the base video prior’s physical understanding rather than improving the model’s underlying physics comprehension.Hair-blowing cases can cause faces to reorient with the wind because training-data correlations associate backward-blowing hair with face orientation.
- C.1 Failures and Limitations: The point-force benchmark spans diverse first-frame scenarios from eight physics-in-the-loop papers, including plants and other physical interactions.The benchmark is designed to test versatility across settings and object types.
C.3 Training a unified model
A unified model is trained on both synthetic point-force and wind-force data, producing more dynamic backgrounds but somewhat less robust point-force control.
- C.3 Training a unified model: The unified model combines point-force and wind-force prompting and is evaluated on both corresponding benchmarks.Training used equal portions of synthetic point-force and synthetic wind-force data with the original architecture and training steps.
- C.3 Training a unified model: The unified model produces more dynamically moving backgrounds, including natural motion in leaves, children, trains, and seated people.These effects appear in scenes involving apple trees, toy trains, and falling leaves.
- C.3 Training a unified model: Point-force control is slightly less robust in some unified-model videos, such as the blueberry-bush example.In these cases, the control signal is not respected.
C.4 Scaling the dataset size
Reducing the synthetic wind-training dataset exposed memorization and weaker force generalization, while the models retained several object- and style-aware behaviors.
- C.4 Scaling the dataset size: Half-sized wind training data caused the model to hallucinate a bedsheet, suggesting memorization of the waving-flag pattern.The dataset size was the only changed variable, while training steps and the learning-rate scheduler remained fixed.
- C.4 Scaling the dataset size: Reduced data also weakened the confetti response, with some pieces moving with the wind while others remained stationary.
- The models respected object-specific constraints, including curved train tracks and windmill rotational axes under applied forces.
- Localized forces produced consistent whole-object movement when applied to different parts of a train, indicating sensitivity beyond the selected pixel.
- Force-controlled videos preserved source-image cinematic properties, including background blur while a toy car moved.
- The method is intended for visually plausible intuitive physics, not high-fidelity simulation in engineering or other precision-critical applications.