PaperlayerView exact revision on arXiv

Source-linked AI summary

RigidBench: Evaluating Rigid-Body Physics in Video Generation Models

Swarnim Jain, Shangzhe Wu

arXiv:2608.15555v1cs.CVcs.LG

TL;DR

Whole-frame metrics do not isolate whether video models predict object motion correctly. RigidBench evaluates matched simulator rollouts with separate measurements, finding that visual similarity and trajectory accuracy diverge and no model leads across all ten metrics.

  • Problem

    Existing evaluations often report clip-level or image-space agreement without identifying each object’s 3D motion separately.

  • Method

    RigidBench compares generated continuations with matched simulator rollouts using aligned masks, depth, 6-DoF trajectories, contacts, and ten separate measurements.

  • Results

    No model leads on all ten measurements, while higher SSIM accompanies larger 3D trajectory error across model means (r = 0.89).

  • Takeaways & Limitations

    Video prediction quality cannot be reduced to one leaderboard because frame similarity need not reflect object-motion fidelity.

  • Takeaways & Limitations

    RigidBench measures fidelity to one matched reference rollout, while later segmentation, tracking, and depth errors can limit measurement accuracy.

Abstract

from arXiv · show

Video models are increasingly used to predict what happens next in a scene, yet the metrics commonly used to compare their outputs say little about whether the predicted objects move correctly. Motion, geometry, identity, background stability, and visual similarity can fail independently, but whole-frame scores often mix these errors together. We introduce RigidBench, a simulator-grounded benchmark that compares a generated continuation with a reference rollout from the same initial frame and motion description. Its five rigid-body tasks vary objects, materials, viewpoints, and indoor and outdoor scenes, with per-frame masks, depth, 6-DoF trajectories, and contacts available for scoring. We evaluate eight models on the same 100 examples with ten measurements that keep these aspects separate. The resulting rankings depend strongly on what is measured: no model leads on all ten, and across model means, higher SSIM accompanies larger 3D trajectory error (r = 0.89). RigidBench also includes 5,000 training videos with exact simulator state, which we use to fine-tune and analyze Wan 2.2 TI2V-5B. Full fine-tuning reduces 3D trajectory error by about 20% with almost no change in SSIM, while teacher-forced probes and targeted interventions show that object position is represented throughout Wan's diffusion transformer and used by its denoising computation.

1 Introduction

RigidBench evaluates video-model rollouts against simulator-grounded references using separate measurements for motion, geometry, identity, background stability, and appearance. Its benchmark results show that visual similarity and physical accuracy can diverge, while simulator state enables targeted analysis and fine-tuning of Wan 2.2 TI2V-5B.

  • Benchmark design: RigidBench compares a generated continuation with a simulator reference from the same initial rendered state and motion description, enabling aligned physical evaluation.Available signals include masks, depth, 6-DoF trajectories, and contacts.
  • Benchmark design: 5,000 training examples and a fixed 100-example evaluation set span five tasks, varied objects, and photorealistic indoor and outdoor scenes.The tasks include free fall, bouncing, ramp collision, a ball chain, and dropping into a cluster.
  • Evaluation: Ten measurements remain separate, covering masks, shape, image-space motion, depth, appearance, 3D trajectories, identity drift, and background deformation.The evaluator recovers actor masks, point tracks, and depth; added metrics include ATE-3D, IdDrift, and BGDrift.
  • Findings: Higher SSIM accompanies larger 3D trajectory error across models, and no model leads on all ten measurements.Free fall has the largest trajectory error for every model despite containing only one moving object, because early errors persist through most of the rollout.
  • Wan 2.2 TI2V-5B analysis: Full fine-tuning lowers ATE-3D by about one fifth while leaving SSIM nearly unchanged in Wan 2.2 TI2V-5B.Teacher-forced probes recover position and contact throughout the diffusion transformer, whereas velocity, orientation, and angular velocity remain weak.

2 Related Work

Related benchmarks evaluate physical video generation through plausibility judgments, recorded experiments, simulator-grounded tests, and analyses of physical information inside video models. RigidBench builds on these lines with simulator-aligned state and causal representation analysis of Wan’s diffusion transformer.

  • Plausibility judgments: Plausibility benchmarks judge whether generated clips depict physically plausible actions or obey physical rules without requiring instrumented scenes.VideoPhy-2, WorldModelBench, and PhyGround cover varied actions and phenomena, but their scores identify implausible events or violated rules rather than separately measuring motion components.
  • Recorded motion and physical laws: Recorded-motion benchmarks compare continuations with real experiments or fitted dynamics, using motion masks, pixel error, physical quantities, and conservation laws.Physics-IQ compares against repeated recordings, whereas Morpheus can reward a plausible trajectory without requiring an exact match to one recording.
  • Simulator-grounded evaluation: Simulator-grounded benchmarks provide exact state and repeatable initial conditions to test generalization, parameter recovery, and object motion.PhyWorld studies transfer to unseen physical settings, WorldBench isolates concepts such as friction and viscosity, and PISA is closest to RigidBench’s evaluation and training setup.
  • Physical information inside video models: Prior model analyses decode speed, acceleration, and direction from video representations and use targeted attention ablations to locate supporting components.RigidBench instead aligns simulator state to each object and time slot, compares base and fine-tuned Wan representations, and removes position-aligned subspaces with Iterative Nullspace Projection against matched controls.

3 RigidBench

RigidBench evaluates video continuations against simulator rollouts using controlled rigid-body scenarios and separate measurements for motion, geometry, identity, background stability, and appearance. Its aligned rendered outputs and simulator state support reproducible, multi-factor scoring without collapsing distinct errors into one aggregate.

  • Benchmark design: Each example pairs a simulator rollout with a continuation task, giving the model only the first frame and a motion description before scoring ten separate measurements.The measurements cover object motion, geometry, identity, background stability, and full-frame similarity.
  • Benchmark design: Five tasks vary free flight, restitution, rolling, pairwise collision, and contact cascades across 24 objects, 12 indoor and outdoor scenes, four ramp materials, and fixed within-clip cameras.Examples also vary drop heights, ramp dimensions, material properties, and camera azimuth.
  • Data and annotations: 49 frames at 1280 × 704 and 24 frames per second are rendered with pixel-aligned RGB, actor masks, and metric depth, alongside 6-DoF state, velocities, and contacts.Bullet advances the rigid bodies before Cycles renders the outputs, while a single seed fixes assets, setup, camera, and render.
  • Data and annotations: 5,000 training clips span five tasks, while evaluation uses 25 examples across four conditions per task and adds held-out scenes and objects.Every model is tested on the same frames, prompts, and examples.
  • Evaluation metrics: Seven reported measurements cover mask overlap and shape, image-space motion, depth, and whole-frame appearance, including ATE-3D, SI-MSE, SSIM, and LPIPS.ATE-3D reconstructs world positions by fitting disparity scale and shift before unprojecting tracked points; BGDrift measures residual local background motion after a RANSAC similarity transform.

4 Video Models on RigidBench

RigidBench’s eight-model audit shows that model rankings depend strongly on the measurement: SSIM and 3D trajectory accuracy nearly disagree, and no model leads across all ten metrics. Task difficulty also varies, with sustained free fall producing the largest trajectory errors for every model.

  • Evaluation setup: Eight image-to-video models are evaluated on the same 100 examples, with Wan 2.2 TI2V-5B and Cosmos Predict 2.5-2B run locally and six queried through hosted APIs.Comparisons remain paired until final averaging, with 95% bootstrap intervals for model means.
  • Metric-dependent rankings: −0.89 is the Pearson correlation between SSIM and negative ATE-3D across eight model means, with a bootstrap interval of [−0.99,−0.43].The Spearman rank correlation is −0.88; Wan has the highest SSIM, while both Seedance variants have the lowest ATE-3D and SSIM.
  • Metric-dependent rankings: 65% of matched examples is the minimum rate at which Seedance 2.0 Fast has lower ATE-3D than every non-Seedance model, while both Seedance variants have lower SSIM on at least 91%.Within this audit, full-frame similarity does not identify the models whose objects follow the reference most closely.
  • Metric-dependent rankings: No model leads on all ten measurements: Wan leads SSIM and LPIPS, Seedance 2.0 Fast leads ATE-3D, IdDrift, and BGDrift, and Veo 3.1 has the lowest scale-invariant depth error.An overall average would hide these tradeoffs.
  • Task results: Free fall has the largest mean ATE-3D for every model, followed by bounce, because sustained uninterrupted motion propagates early errors across many later positions.This ordering is not explained by object or contact counts.
  • Metric validation: 6.3× is IdDrift’s sensitivity to an identity swap relative to a gamma change, compared with 1.3× for pixel L2.Controlled perturbations also show that ATE-3D responds equally to camera-ray-parallel and perpendicular displacements, while BGDrift removes coherent image translation but retains independent point jitter.

5 Fine-tuning and Probing Wan 2.2

Fine-tuning Wan 2.2 on simulator-supervised videos improves rigid-body motion metrics with little appearance change. Probing and interventions show that position is accessible throughout the model and contributes to its denoising computation, unlike most other instantaneous state variables.

  • Fine-tuning: Full fine-tuning reaches an ATE-3D about one fifth below base Wan, while appearance scores change little and IdDrift rises slightly.Both selected checkpoints improve on the base model, but the examples do not establish an ordering between LoRA and full fine-tuning.
  • Probing: Position and contact can be read linearly from the first block, with both signals becoming stronger after fine-tuning.Randomly initialized DiT features score above chance because input latents and token layout already contain spatial structure.
  • Probing: Only position is recovered strongly across layers and noise levels; orientation, velocity, angular velocity, and contact normal remain weak.Acceleration and isolated velocity signals found by nonlinear probes do not persist across the sweep.
  • Intervention: Removing the position-aligned subspace raises loss far more than either control across blocks and after sample-level averaging.The ordering remains unchanged when modified states pass through remaining blocks or are read directly by Wan’s output head.

6 Limitations

RigidBench measures fidelity to one matched reference rollout, and its object-level scores depend on multiple pretrained estimators. Its current scope is limited to rigid bodies in synthetic photorealistic scenes and an eight-system audit, so broader physical and model conclusions require caution.

  • Reference-dependent evaluation: RigidBench measures fidelity to the matched reference trajectory, not every physically valid continuation.The initial frame and prompt may omit state variables or physical parameters, allowing alternative valid futures.
  • Measurement dependencies: Object-level measurements depend on SAM 2, CoTracker3, Video Depth Anything, and DINOv2.These dependencies affect masks, tracks, depth, and visual representations used for scoring.
  • Task scope: Current tasks cover rigid bodies in photorealistic synthetic scenes, excluding direct real-video evaluation.Real-video evaluation would require separate calibration, while deformable bodies, fluids, and articulated motion require different state variables and metrics.
  • Model scope: The audit covers eight related systems, including fast and full variants of the same model families.Therefore, its cross-model correlation describes this evaluated set rather than video models in general.

7 Conclusion · A Benchmark Construction and Ground Truth · A.1 Tasks

RigidBench separates frame preservation from physical correctness, showing that model rankings depend on the measurement and that Wan 2.2 improves trajectories without materially changing SSIM. Its released datasets are deterministically specified by manifests, task definitions, and simulator-grounded placement procedures.

  • 7 Conclusion: RigidBench shows that the best frame-preserving model need not reproduce object motion, so video prediction cannot be reduced to one leaderboard.A falling object can also accumulate more trajectory error than scenes containing several contacts.
  • 7 Conclusion: Wan 2.2 simulator fine-tuning improves trajectories with little change in SSIM, while shared ground truth identifies a position signal used by denoising.The study connects output changes to physical state representation inside the model.
  • A Benchmark Construction and Ground Truth: 5,000 training and 100 evaluation examples are generated from declarative files describing tasks, scenes, objects, materials, and rendering.Manifests are fixed before rendering, so each sample identifier determines its task, scene, support surface, allowed objects, and random seed.
  • A.1 Tasks: Each task specifies placements, a prompt template, allowed camera directions, and support-surface changes, with numeric values sampled uniformly from stated intervals.Object tags resolve against the set named by each manifest row; Table 4 summarizes five task definitions.
  • A.1 Tasks: Table 4 samples ramp materials from light wood, dark wood, polished white marble, and rough gray concrete, with camera directions measured relative to task rotation.Prompt placeholders expand to the selected object’s name with an indefinite article.
  • A.1 Tasks: The task builder supports four placement operations: support or elevated placement, passive five-face ramps, bounding-box lines, and object clusters.Clusters place n objects on a ring defined by the largest selected object radius and sampled gap.
  • A.1 Tasks: An 80-step Pymunk simulation at 1/60 s and zero gravity removes initial intersections before Blender attaches rigid bodies.Overlapping circular footprints repel, while damped springs keep objects near their sampled positions.

A.2 Training and Evaluation Partitions … B.4 Aggregation

RigidBench uses simulator-controlled partitions, rendering, trajectory construction, and aligned evaluation to separate physical and visual errors in generated video. Its released artifacts and aggregation preserve reproducible per-example and per-task measurements across a fixed 100-example audit.

  • A.2 Training and Evaluation Partitions: The registry contains 12 scenes, 24 objects, and four ramp materials, partitioning eight scenes and 16 objects for training and four scenes and eight objects for held-out evaluation.Object files specify each asset, scale, mass, friction, restitution, collision shape, and task tags.
  • A.2 Training and Evaluation Partitions: 5,000 training examples are distributed across eight scenes, while evaluation contains 100 disjoint examples across four partitions with 20 examples per task.Training task counts are 1,047 free-fall, 1,000 bounce, 953 ball-chain, 1,000 drop-into-cluster, and 1,000 ramp-collision examples; each evaluation partition has five examples per task.
  • A.3 Scene Layout and Camera Selection: Layouts solve support-fitting inequalities over rotation intervals, while camera search samples azimuths every 15° and accepts the first rotation reaching at least 50% target visibility.If none of eight rotations reaches the threshold, the highest-visible-fraction pair is used.
  • A.4 Simulation, Rendering, and Stored State: Blender’s Bullet solver advances rigid bodies before one Cycles render produces RGB, object-index, and depth outputs from the same camera.Each sample stores masks, depth, poses, contacts, and RGB frames from the same simulated scene.
  • A.5 Ground-truth Point Trajectories: Ground-truth image trajectories project simulator-carried rigid-body points through the fixed camera, retaining points only when they remain geometrically visible.Twenty points are sampled per actor from an eroded frame-zero mask, with visibility requiring valid projection, actor-mask inclusion, and depth no more than 1.01 times rendered depth.
  • B Evaluation Details and Metric Validation: Every model is evaluated on the same fixed manifest of 100 examples using the rendered first frame and instantiated task prompt, with shared perception models and active-actor averaging.Ramps and support surfaces are excluded from object-level averages.
  • B.1 Model Generation and Temporal Alignment; B.2 Perception and Feature Extraction: Temporal alignment retains reference timestamps tk = k/24, interpolates RGB, tracks, and disparity in physical time, and stops each comparison at the shorter sequence.Cosmos is resampled from 16 to 24 Hz; shared first frames provide direct actor correspondence for SAM 2 and CoTracker3.
  • B.3 Metric Computation; B.4 Aggregation: Metrics average mask, SSIM, LPIPS, and SI-MSE values by frame after actor averaging, while ATE weights every jointly visible point-frame pair equally.Released summaries report arithmetic mean, population standard deviation, and finite count overall and within each task.

B.5 Controlled Metric Perturbations · C Extended Model Results

Controlled perturbations isolate whether ATE-3D, BGDrift, and IdDrift respond to the intended geometric, background, and identity changes. The reported tests vary one known quantity while holding a matched alternative fixed.

  • B.5 Controlled Metric Perturbations: The experiments change one known quantity while holding a matched alternative fixed, and Figure 7 plots the resulting metric responses.This design directly tests whether each metric distinguishes the intended perturbation.
  • B.5 Controlled Metric Perturbations: Figure 7 applies equal 3D offsets along and perpendicular to the viewing ray, matched-displacement translation and jitter, and identity swaps or gamma changes.Shading reports one standard error from the saved test output.
  • B.5 Controlled Metric Perturbations: ATE-3D tests ten free-fall and ramp-collision examples with per-frame perturbations from 0–0.5 m.The examples are split evenly between free fall and ramp collision.
  • B.5 Controlled Metric Perturbations: Moving an actor along the viewing ray leaves 2D ATE zero, while ATE-3D responds at the same rate as an equal perpendicular displacement.This separates image-coordinate invariance from 3D displacement sensitivity.
  • B.5 Controlled Metric Perturbations: BGDrift follows 200 corners across 49 frames under coherent translation and independent Gaussian jitter with matched mean displacements from 0 to 40 pixels.The fitted transform removes translation, while its residual tracks jitter.
  • B.5 Controlled Metric Perturbations: IdDrift evaluates ten frame-zero patches over 20 trials, comparing swaps at probability q with foil gamma changes of γ = 1 + 1.5q.The target swaps a patch with probability q, while the foil changes the same patch.

C.1 Uncertainty Across Metrics … D.2 Optimization and Parameter Updates

RigidBench’s uncertainty analysis shows that trajectory and depth metrics are less precise than appearance scores, while paired comparisons confirm a sharp tradeoff between Seedance variants’ 3D accuracy and other models’ SSIM. Task difficulty varies with motion duration, and Wan fine-tuning uses simulator-grounded training with distinct full-parameter and LoRA update strategies.

  • C.1 Uncertainty Across Metrics: Trajectory and depth measurements have wider intervals than full-frame appearance scores, and neighboring model means are sometimes indistinguishable on one metric.Figure 8 reports model means with bootstrap 95% confidence intervals over evaluation examples; intervals use 20,000 percentile bootstrap resamples with the example as the resampling unit.
  • C.1 Uncertainty Across Metrics: The models with the lowest trajectory errors are not those with the highest SSIM or lowest LPIPS.This pattern remains clear despite uncertainty in individual metric comparisons.
  • C.2 Paired Model Comparisons: Each Seedance variant beats every non-Seedance model on ATE-3D in at least 65% of examples, while every non-Seedance model beats both variants on SSIM in at least 91%.Matched example-level probabilities reproduce the differences seen in model means without allowing a few large errors to dominate.
  • C.3 Task Breakdown: Free fall has the highest ATE-3D for all eight models, while ball chain has the lowest for all eight.The shared ordering follows motion duration more closely than object or contact count because position errors accumulate throughout a fall.
  • D.1 Training Data and Objective: The fine-tuning runs start from Wan 2.2 TI2V-5B and use a disjoint 5,000-example training manifest containing 624 distinct prompts.All 49 frames are resized from 1280×704 to 640×352, and VAE encoding produces 13 temporal latent slots.
  • D.1 Training Data and Objective: Each update samples one of the scheduler’s 1,000 training timesteps, restores the first temporal slot to its clean value, and excludes that slot from the float32 MSE.The DiT predicts velocity from the noised latent, timestep, and cached text context; the same first-frame pinning is used for evaluation generation.
  • D.2 Optimization and Parameter Updates: Full fine-tuning updates every DiT parameter except text and time conditioning modules, whereas LoRA freezes the base DiT and adds rank-32 updates to every block’s attention and feed-forward projections.Both configurations use eight H100 GPUs, bf16 mixed precision, fused AdamW, weight decay 0.01, ϵ = 10−8, gradient clipping at 1.0, and random seed 42.
  • D.2 Optimization and Parameter Updates: LoRA uses DDP, full fine-tuning uses FSDP, checkpoints are written every 500 updates, and the evaluated LoRA run ends at step 3,000 despite a 5,000-step scheduler horizon.DDP is used because only adapter tensors require gradients and optimizer state; FSDP shards model weights, gradients, and AdamW state across eight GPUs.

D.3 Complete Checkpoint Results … E.1 Actor-Aligned Residual Streams

Selected full fine-tuning and LoRA checkpoints improve motion and geometry over base Wan, while their relative performance varies across tasks and remains unresolved between runs. The actor-aligned probe setup extracts simulator-defined position and contact labels from Wan’s residual streams across the training distribution.

  • D.3 Complete Checkpoint Results: The selected checkpoints are chosen by lowest ATE-3D on the same 100 examples used for evaluation, so selection describes observed sweeps rather than held-out estimates.Paired bootstrap intervals use 20,000 resamples of matched examples.
  • D.3 Complete Checkpoint Results: Both selected checkpoints improve all six motion and geometry measurements relative to base Wan, while SSIM rises slightly.LPIPS and BGDrift intervals include zero for both runs.
  • D.3 Complete Checkpoint Results: Full fine-tuning raises IdDrift by 0.022 with interval [0.001,0.045], whereas the LoRA change is not resolved.This is the only identity-drift change explicitly resolved in the supplied results.
  • D.3 Complete Checkpoint Results: The ATE-3D difference between selected full and LoRA checkpoints is −0.20 with interval [−0.50,0.07], leaving their ordering unresolved.The interval comes from paired changes for the selected checkpoints.
  • D.4 Task and Evaluation-Partition Breakdowns: Both selected checkpoints reduce mean ATE-3D on every task and evaluation partition, but full fine-tuning leads on three tasks while LoRA leads on two.Free fall has the largest absolute change under full fine-tuning, and neither checkpoint lowers split-mean SSIM versus base Wan.
  • E Probe and Intervention Details: The activation sweep uses 5,000 simulator-generated training videos split into 4,000 training, 500 validation, and 500 testing examples.Because the full fine-tuned backbone used the same corpus, probe scores describe access to state on that training distribution.
  • E.1 Actor-Aligned Residual Streams: Each 49-frame video becomes a 13 × 22 × 40 residual-stream grid with width 3,072 after VAE encoding and patchification.Noise is added at τ ∈{0.1,0.3,0.5,0.7,0.9}, while the first temporal latent remains clean.
  • E.1 Actor-Aligned Residual Streams: Actor-aligned probes aggregate simulator masks over VAE-assigned pixel frames, max-pool them to 22×40 tokens, and label each visible actor–slot row by position and contact.Rows without covered tokens are omitted; every layer, schedule fraction, target, probe class, and label condition is fit separately.

E.2 Probe Fits and Noise-Resolved Results · E.3 Position-Aligned Interventions

Probe results show that Wan encodes position more reliably than orientation, angular velocity, or contact normal, while nonlinear probes recover spatial information already present in the VAE representation. Position-aligned interventions produce larger loss increases than rank- and variance-matched controls across evaluated clips and blocks.

  • E.2 Probe Fits and Noise-Resolved Results: Shuffled controls remain near R2 = 0 and AUC = 0.5, while base Wan and full fine-tuning improve probe scores for both targets and probe classes.The MLP gap is smaller because nonlinear readouts recover spatial information already present in the VAE latent and token layout.
  • E.2 Probe Fits and Noise-Resolved Results: At τ = 0.9, the random network approaches trained models for position, but separation is clear at other schedule positions.This comparison uses linear probes fitted at every block and schedule fraction.
  • E.2 Probe Fits and Noise-Resolved Results: The nonlinear readout narrows random-versus-trained differences, especially for position, while contact preserves clearer ordering among the three backbones.These results come from MLP probes across blocks and schedule fractions.
  • E.2 Probe Fits and Noise-Resolved Results: Position remains strong across continuous-state probe sweeps, whereas orientation, angular velocity, and contact normal remain weak.Acceleration and velocity become stronger in a few MLP cells but do not persist across layers and noise levels.
  • E.3 Position-Aligned Interventions: The intervention sweep fits a separate position subspace for each of 90 block–schedule cells using base Wan at τ ∈{0.3,0.5,0.7}.INLP repeatedly removes ridge-regression row spaces from projected activations, with a stopping rule based on training R2 or 30 iterations.
  • E.3 Position-Aligned Interventions: The variance-matched control removes principal-component directions with at least as much activation variance as the position basis while matching the total amount removed.The rank-matched control instead uses a random orthonormal basis.
  • E.3 Position-Aligned Interventions: The position-aligned intervention is larger than either control under both readouts for all 100 clip-level averages and at every block.The hook is applied to every residual token at the chosen block, and effects are compared under the same noised latent while excluding the clean first temporal slot.
  • E.3 Position-Aligned Interventions: Table 21 reports mean loss increases with bootstrap 95% confidence intervals, averaging 100 clips over 30 blocks and three noise levels.Bootstrap intervals resample clips 20,000 times.
Loading 2608.15555v1…