PaperlayerView exact revision on arXiv

Source-linked AI summary

PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop

Chenyu Li, Oscar Michel, Xichen Pan, Sainan Liu, Mike Roberts, Saining Xie

arXiv:2503.09595v1cs.CV

TL;DR

The paper asks whether pretrained video generators can become physically accurate world simulators, focusing on the controlled task of object freefall. It introduces PISA, combining real and simulated videos with post-training through fine-tuning and reward modeling. The authors find that current models fail at physically accurate dropping, while modest simulated-data fine-tuning and reward modeling improve accuracy, with limitations in out-of-distribution generalization and distribution matching.

  • Problem

    Pretrained video generators look visually realistic but remain unreliable as physically accurate world simulators, motivating study of post-training for physical accuracy.

  • Method

    PISA studies image-to-video object freefall using real and simulated datasets, task-specific metrics, supervised fine-tuning, and reward modeling.

  • Results

    Current state-of-the-art models frequently fail at physically accurate dropping, while fine-tuning on a small simulated dataset substantially improves accuracy and reward modeling provides further improvement.

  • Takeaways & Limitations

    PISA provides a diagnostic framework for evaluating physical accuracy and studying post-training toward more capable world-modeling systems.

  • Takeaways & Limitations

    Performance degrades outside the training distribution, and physically consistent motion can still mismatch the ground-truth dropping-time distribution.

Abstract

from arXiv · show

Large-scale pre-trained video generation models excel in content creation but are not reliable as physically accurate world simulators out of the box. This work studies the process of post-training these models for accurate world modeling through the lens of the simple, yet fundamental, physics task of modeling object freefall. We show state-of-the-art video generation models struggle with this basic task, despite their visually impressive outputs. To remedy this problem, we find that fine-tuning on a relatively small amount of simulated videos is effective in inducing the dropping behavior in the model, and we can further improve results through a novel reward modeling procedure we introduce. Our study also reveals key limitations of post-training in generalization and distribution modeling. Additionally, we release a benchmark for this task that may serve as a useful diagnostic tool for tracking physical accuracy in large-scale video generative model development.

1. Introduction

Although video generation models produce visually impressive content, they remain unreliable as physically accurate world simulators. The PISA framework studies this gap through controlled object-freefall experiments, using real videos for evaluation and synthetic videos for post-training, while exposing generalization and distributional limitations.

  • Physically accurate world modeling requires generated pixels to represent world states that evolve according to physics and visual perspective, not merely look plausible.
  • State-of-the-art video generators struggle with physically accurate object dropping despite internet-scale video pretraining, motivating a dedicated post-training stage.
  • The study focuses on image-to-video freefall, where an initially suspended object falls and may collide with objects on the ground, enabling controlled analysis.
  • PISA combines diverse real and simulated dropping videos with task-specific physical-accuracy metrics; real videos evaluate outputs, while synthetic videos support post-training.
  • Fine-tuning a few thousand simulated samples substantially improves physical accuracy, and reward modeling further targets different physical-improvement axes.
  • Performance degrades outside the training distribution, including drops from unseen depths or heights, while 3D-consistent motion can still mismatch ground-truth dropping-time distributions.
  • PISA is presented as a diagnostic benchmark for tracking physical abilities and identifying post-training limitations in future world-modeling systems.

2. Related Work

Related work frames intuitive physics and video generation as foundations for world simulation, while emphasizing evaluation methods beyond visual or distributional similarity. The paper uses simulation tools to construct training videos for its physics-focused study.

  • Modeling Intuitive Physics: Intuitive physics concerns rapid judgments about object motion, stability, and interactions, with gravity serving as a core observable force.
  • Modeling Intuitive Physics: Replicating intuitive physics is treated as foundational for systems that interact effectively and safely with dynamic real-world environments.
  • Modeling Intuitive Physics: The work uses the Kubric physics engine to generate training videos for modeling intuitive physics.
  • Video Generation Models as World Simulators: Video-generation research increasingly examines whether large-scale models can represent real-world dynamics and decision-making as video prediction.
  • Evaluating Video Generation Models: Evaluation research extends beyond frame-level or video distribution metrics such as FID, IS, and FVD toward benchmark suites with multiple measures.

3. PisaBench

PisaBench evaluates whether video generation models can produce physically accurate object-dropping videos under constrained assumptions. It combines real and simulated test data, annotations, physical-accuracy metrics, and baseline comparisons to expose failures in dropping behavior.

  • 3.1. Task Definition & Assumptions: PisaBench benchmarks video models on generating an object falling and colliding with the ground or other objects from a suspended initial image.The task assumes a stationary object initially, gravity as the only force, and a fixed camera so that physical accuracy can be evaluated from single videos.
  • 3.2. Real World Data: The real-world dataset contains 361 dropping videos spanning diverse objects and settings, while the simulated test set has 60 videos split between seen and unseen objects and backgrounds.Real videos begin with objects suspended by an invisible wire and are recorded in slow motion at 120 fps; simulated videos support sim2real evaluation.
  • 3.3. Metrics: The benchmark measures trajectory accuracy, object shape fidelity, and object permanence using Trajectory L2, Chamfer Distance, and Intersection over Union.Trajectory L2 compares masked-region centroids, Chamfer Distance compares mask shapes, and IoU measures overlap between generated and ground-truth objects.
  • 3.3. Metrics: Real-world annotations combine manually labeled first-frame masks, SAM 2 propagation across frames, and captions formatted as “{object description} falls.”These annotations provide segmentation and textual task context for evaluation and models supporting text input.
  • 3.4. Evaluation Results: Across open and proprietary baselines, generated videos consistently fail to depict physically accurate dropping despite visually realistic frames.Observed failures include implausible deformations, floating, hallucinated objects, static or upward motion, and unrealistic horizontal movement.
  • 3.4. Evaluation Results: Qualitative comparisons show PSFT introduces free-fall behavior, while ORO improves the modeling of object motion and shape.The comparison covers real, simulated-seen, and simulated-unseen test sets alongside popular open-source and commercial models.

4. Physics Post-Training

The paper post-trains video diffusion models with simulated dropping videos through supervised fine-tuning, then optionally reward optimization. This pipeline improves physical accuracy on real and simulated evaluations, with pretrained initialization and targeted rewards contributing to performance.

  • Physics Post-Training: A two-stage pipeline combines supervised fine-tuning on realistic simulated dropping videos with subsequent reward modeling, improving both simulated and real-world evaluations.The approach is inspired by supervised fine-tuning followed by reward modeling, with larger gains in simulation because of the sim-to-real gap.
  • Physics Supervised Fine-Tuning (PSFT): 5,000 samples are sufficient to achieve optimal results in the dataset-size ablation.Each run uses 5,000 gradient steps, with subsets ranging from 500 to 20,000 samples.
  • Physics Supervised Fine-Tuning (PSFT): Pretrained Open-Sora initialization is critical, while randomly initializing its denoising network reduces adaptation effectiveness.The ablation keeps the compressor pretrained but randomizes the denoising network.
  • Physics Supervised Fine-Tuning (PSFT): PSFT on 5,000 samples outperforms all evaluated models, including state-of-the-art commercial video generators, by a wide margin.The comparison includes proprietary and open models on real-world and simulated PisaBench splits.
  • Object Reward Optimization (ORO): Object Reward Optimization guides generated motion and shape toward ground truth, with segmentation, optical-flow, and depth rewards targeting different physical properties.Segmentation rewards improve shape accuracy, while flow and depth rewards improve motion accuracy.
  • Object Reward Optimization (ORO): Reward modeling further improves performance and is modular because each reward enhances the physicality aspect it is designed to measure.The implementation uses generated videos and ground truth, with gradients propagated through all denoising steps.

5. Assessing Learned Physical Behavior

The evaluation probes whether post-trained models generalize across unseen depth and height conditions and whether their output distributions match the physically possible outcomes. Performance degrades outside the training distribution, and the model’s range of outputs remains misaligned with the ground-truth distribution.

  • Assessing Learned Physical Behavior: The analysis tests generalization to unseen dropping heights and depths and examines the conditional distribution p(t|y) induced by perspective uncertainty.Depth and height determine falling dynamics under the camera assumptions, while p(t|y) captures possible dropping times for an image-plane coordinate.
  • Generalization to Unseen Depths and Heights: The test set varies depth from 1–5m and height from 0.5–2.5m, with ID ranges of [1, 3] and [0.5, 1.5], respectively.Objects and backgrounds in the test set are unseen during training, and samples outside the camera frustum are discarded.
  • Generalization to Unseen Depths and Heights: Performance degrades for out-of-distribution depth-height combinations.The analysis uses dropping-time error alongside the other physical-accuracy metrics.
  • Generalization to Unseen Depths and Heights: This degradation suggests difficulty learning a fully generalizable law that captures the interaction between perspective and gravity.Depth and height are identified as the main physical quantities affecting falling dynamics.
  • Distributional Analysis: The distributional analysis uses p(t|y), with ground truth approximated by Monte Carlo because estimated contact times have only 32 possible values.Goodness-of-fit is measured with the Kolmogorov–Smirnov test, treating p-values below 0.05 as evidence of misalignment.
  • Distributional Analysis: The model’s trajectories can show plausible 3D world-state tendencies, but its output range does not align with the ground-truth distribution.Distributional alignment is evaluated by generating 128 videos for each of 50 initial images and comparing dropping-time distributions.

6. Conclusion

The study presents post-training as a way to adapt pretrained video generators toward world models, while showing that per-sample physical accuracy can improve without solving general distributional alignment. The single-image setup enables broad model comparisons but introduces ambiguity about object size and height.

  • Conclusion: Post-training improves per-sample physical accuracy, but general distributional alignment remains unsolved.The authors distinguish generating likely samples from matching the full data distribution, which is especially important for faithful world simulation.
  • Conclusion: Single-image conditioning supports comparisons across many state-of-the-art models but creates ambiguity about object size and height from perspective.The paper reports that the metrics remain reliable signals despite this ambiguity.

B. Metric details.

The evaluation aligns generated and ground-truth videos by frame rate, then compares corresponding object masks and centroids. It measures trajectory, shape fidelity, object permanence, and simulated dropping-time error.

  • Metric details: The evaluation uses trajectory, shape-fidelity, and object-permanence metrics, comparing corresponding generated and ground-truth frames.Frame-rate alignment is performed before metric computation because models can operate at different fps.
  • Metric details: Generated frame index i is mapped to a ground-truth index j using the generated and ground-truth frame rates, fgen and fgt.The ground-truth video is downsampled to match the generated video's frame count.
  • Metric details: Trajectory L2 averages distances between generated and ground-truth object-mask centroids across corresponding frames.Centroids are calculated for each frame before computing the L2 distance.
  • Metric details: Chamfer Distance measures object shape fidelity by comparing generated and ground-truth mask regions.The compared regions are represented as sets of mask points for each frame.
  • Metric details: Intersection over Union measures object permanence through overlap between generated and ground-truth segmentation masks.The masks are binary for the falling object in each corresponding frame.
  • Metric details: Dropping-time error compares the ground-truth dropping time with the model's estimated impact frame from centroid velocity in the y direction.The error is set to zero when the true dropping time falls between adjacent estimated frames.

C. ORO implementation details.

The reward implementation trains video generators with objectives based on motion, depth, segmentation, and optical flow. These rewards constrain object position, shape, and 3D motion relative to ground truth.

  • ORO implementation details: Segmentation reward uses IoU between generated-object masks and Kubric ground-truth masks.Sigmoid-normalized mask logits preserve differentiability instead of converting generated masks to binary masks.
  • ORO implementation details: The segmentation objective constrains generated-object position and shape by encouraging overlap with the ground-truth object region.The resulting training signal targets more accurate object positions and shapes.
  • ORO implementation details: Optical-flow reward compares flow fields from generated and ground-truth videos.RAFT is used to generate the optical-flow representations.
  • ORO implementation details: The training objective constrains generated-object motion so the model learns more accurate physical motion.The implementation backpropagates the reward gradient through all diffusion timesteps to update model weights.
  • ORO implementation details: Depth reward compares optical depth maps from generated and ground-truth videos.Depth-Anything-V2 is used to produce the depth maps.
  • ORO implementation details: The coordinate analysis computes image-plane height y from segmentation and camera parameters while ignoring X and x under the perspective-camera assumption.The computation uses the pixel row just below the object together with camera position, sensor size, and image resolution.

E. Derivation of p(t|y)

The derivation models dropping time as a function of image-plane height and sampled depth under a perspective camera. A uniform depth assumption and a linear relation between 3D height and depth determine the resulting distribution.

  • Derivation of p(t|y): Depth Z is sampled uniformly from [2, 18], and 3D dropping height is modeled as Y = y + βZ.The slope β is computed from image-plane height, focal length, sensor size, and camera height.
  • Derivation of p(t|y): Dropping time is derived from the height relation using gravitational acceleration, producing a depth-dependent time expression.The derivation applies a probability-density transformation after expressing time as a function of y and Z.
  • Derivation of p(t|y): Substituting Zmin = 2 and Zmax = 18 yields the final bounded form used for Equation (3).The endpoints determine the corresponding minimum and maximum dropping-time terms.

F. Ambiguous dataset

The dataset broadens the conditional distribution p(t|y) by varying object depth, while generated trajectories are lifted from 2D using estimated drop time and inferred depth.

  • Distributional analysis: A new distributional-analysis dataset samples depth uniformly from [2, 18] and adjusts object scale and translation to preserve its appearance.This broadens p(t|y) compared with the realism-focused PSFT dataset, which has limited object-depth variability.
  • Trajectory lifting: The trajectory analysis estimates the object’s drop time before deriving an implied depth for 3D lifting.The procedure uses object masks to track the bottom of the object across generated frames.
  • Trajectory lifting: The 3D trajectory is formed by adjusting the observed 2D trajectory with the inferred depth term.

H.1. Data Collection Pipeline

The data pipeline combines controlled real-world collection, standardized clipping and annotation, task-specific prompting, and simulated collision videos. Qualitative evaluation shows that models often fail in trajectory, object, and scene consistency despite sometimes predicting downward motion.

  • Real-world collection: Approximately 15 volunteers collected real-world falling-object videos using tripods, tape, and invisible wire under detailed guidelines.The guidelines were intended to improve quality and diversity while reducing environmental ambiguity.
  • Video processing: Raw videos were clipped to segments of at most 12 seconds and manually cropped to a 1:1 aspect ratio while keeping falling objects visible.The pipeline also records each clip’s starting position in the original long video.
  • Annotation: Objects in the first frame were annotated with positive and negative points, then SAM 2 propagated segmentation masks across frames.The annotation process also includes video captions describing the falling object.
  • Inference setup: Prompts specify that the object falls according to physics and discourage camera motion and slow motion.These prompts are intended to provide task context and stabilize the generated environment.
  • Qualitative evaluation: Qualitative failures include incorrect trajectories, inconsistent object appearance or permanence, and abrupt scene changes.Models may predict a downward trend while still failing to produce plausible shape, motion, collisions, or stable scenes.
  • Simulation: Simulated videos use Kubric with PyBullet physics and Blender rendering, producing 32 frames from 2-second scenes rendered at 16 fps.Scenes contain 1–6 dropped objects placed 0.5m–1.5m above potentially colliding piles.

L. Limitations

The benchmark and post-training methods remain limited in both evaluation scope and visual fidelity. The qualitative examples further show that baseline models and the proposed method exhibit distinct physical and consistency failures.

  • Evaluation scope: The benchmark uses 361 real-world videos and three spatial metrics focused on positional relationships rather than object appearance.Future work is proposed for appearance metrics based on multimodal language models or pixel-level analysis.
  • Post-training limitations: PSFT and ORO improve Open-Sora’s physically plausible video generation, but some outputs still contain blurry objects.The authors identify dataset and fine-tuning refinement as directions for improving object visuals.
  • Baseline qualitative failures: Kling-V1 sometimes changes object shape, suspends objects, introduces new objects, or fails to predict collision chains.
  • Baseline qualitative failures: Runway Gen3 exhibits object falling in some examples but also introduces new objects, changes object shape, and suspends a box.
  • Baseline qualitative failures: CogVideoX-5B-I2V produces falling tendencies but violates physics in every shown video through object splitting, shape changes, nonvertical trajectories, or sudden scene changes.
  • Baseline qualitative failures: DynamiCrafter and Open-Sora show objects suspended in midair across their examples, while Pyramid-Flow introduces objects or scene changes and can become blurry.
  • Post-trained qualitative failures: The proposed Open-Sora + PSFT + ORO method induces falling behavior, but object consistency remains insufficient and objects may blur or disappear after collisions.
Loading 2503.09595v1…