PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far is Video Generation from World Model: A Physical Law Perspective

Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, Jiashi Feng

arXiv:2411.02385v2cs.CVcs.AI

TL;DR

The paper asks whether video generation models can discover physical laws from visual data rather than memorize examples. It evaluates diffusion-based models on deterministic 2D simulations across ID, OOD, and combinatorial generalization. The models achieve near-perfect ID generalization and improve combinatorial performance with scaling, but fail OOD tests and rely on case-based generalization.

  • Problem

    The paper examines whether video models learn physical laws from visual data rather than memorizing training examples, a requirement for robust world-model predictions.

  • Method

    The authors build deterministic 2D simulations of object motion and collisions, then train diffusion-based video models to predict future frames from initial frames.

  • Results

    The models show near-perfect ID generalization, improved combinatorial generalization with scaling, but failed OOD generalization and case-based behavior.

  • Takeaways & Limitations

    Naive scaling is insufficient to discover physical laws, while greater combinatorial diversity improves physical video modeling.

  • Takeaways & Limitations

    Visual ambiguity can make fine-grained physics indeterminate from video alone, producing visually plausible but physically incorrect predictions.

Abstract

from arXiv · show

OpenAI's Sora highlights the potential of video generation for developing world models that adhere to fundamental physical laws. However, the ability of video generation models to discover such laws purely from visual data without human priors can be questioned. A world model learning the true law should give predictions robust to nuances and correctly extrapolate on unseen scenarios. In this work, we evaluate across three key scenarios: in-distribution, out-of-distribution, and combinatorial generalization. We developed a 2D simulation testbed for object movement and collisions to generate videos deterministically governed by one or more classical mechanics laws. This provides an unlimited supply of data for large-scale experimentation and enables quantitative evaluation of whether the generated videos adhere to physical laws. We trained diffusion-based video generation models to predict object movements based on initial frames. Our scaling experiments show perfect generalization within the distribution, measurable scaling behavior for combinatorial generalization, but failure in out-of-distribution scenarios. Further experiments reveal two key insights about the generalization mechanisms of these models: (1) the models fail to abstract general physical rules and instead exhibit "case-based" generalization behavior, i.e., mimicking the closest training example; (2) when generalizing to new cases, models are observed to prioritize different factors when referencing training data: color > size > velocity > shape. Our study suggests that scaling alone is insufficient for video generation models to uncover fundamental physical laws, despite its role in Sora's broader success. See our project page at https://phyworld.github.io

1. Introduction

The paper asks whether video generation models learn physical laws or memorize examples, evaluating this through ID, OOD, and combinatorial generalization. It uses simplified 2D physics simulations and finds strong ID performance, improved combinatorial generalization with scaling, but persistent OOD failures and case-based behavior.

  • Video models must generalize beyond training data to support realistic world models and precise simulations.The paper motivates physical-law understanding as necessary for extending generated data beyond the training corpus.
  • The evaluation distinguishes in-distribution, out-of-distribution, and combinatorial generalization according to relationships between training and test data.Combinatorial generalization tests whether models can combine distinct concepts in novel ways.
  • A 2D simulator uses simple geometric objects whose motions and collisions are governed by one or two classical physical laws.The simplified setting reduces confounding from complex appearances and nonrigid real-world motion.
  • Scaling from 30K to 3 million examples and from 22M to 310M parameters yields near-perfect ID generalization but does not improve OOD generalization.The experiments cover uniform motion, elastic collisions, and parabolic motion.
  • 67% to 10%: scaling substantially reduces abnormal cases in combinatorial generalization.The authors interpret this reduction as evidence that scaling is critical for improving combinatorial generalization.
  • Models generalize case-by-case rather than abstracting universal physical rules, prioritizing color > size > velocity > shape when referencing training examples.Examples include reversing a low-speed ball incorrectly or changing a red square into a ball.

2. Video Generation for Phyiscal Law Discovery

The paper formulates physical-law discovery as predicting future video frames from past states under latent dynamics and rendering constraints. It evaluates learned laws through ID, OOD, and combinatorial generalization using scalable video-generation models.

  • 2. Video Generation for Phyiscal Law Discovery: Physical-law discovery is framed as predicting future frames conditioned on past states, paralleling physical laws that map initial conditions to future dynamics.Each frame represents a moment in time in the video-based formulation.
  • 2. Video Generation for Phyiscal Law Discovery: A physically coherent video has latent states that follow z_t+1 = z_t + δF(z_t) and images rendered as I_t = R(z_t).The latent dynamics and rendering function jointly define physical coherence.
  • 2. Video Generation for Phyiscal Law Discovery: The model predicts subsequent frames from initial frames, with physical coherence loss defined as −log p_θ(I_c+1, . . . , I_L | I_1, . . . , I_c).This loss measures how likely predicted frames are under the learned video distribution.
  • 2. Video Generation for Phyiscal Law Discovery: The approach adopts VAE and DiT architectures, using the VAE for spatiotemporal compression and DiT for diffusion denoising.The paper focuses exclusively on video generation models and emphasizes the approach’s scalability.
  • 2. Video Generation for Phyiscal Law Discovery: Learned laws are verified through ID, OOD, and combinatorial generalization, with OOD testing whether models extrapolate to unseen scenarios.The framework treats generalization ability as evidence of whether a model learned principled rules.

3. In-Distribution and Out-of-Distribution Generalization

The study evaluates video models on deterministic kinematic tasks using ID and OOD tests. Scaling improves ID velocity prediction, but OOD errors remain much higher and do not reliably improve with larger models or datasets.

  • 3.1. Fundamental Physical Scenarios: The evaluation uses deterministic tasks governed by basic kinematic equations, with the first three frames conditioning predictions of subsequent frames.The tasks cover uniform linear motion, perfectly elastic collision, and parabolic motion.
  • 3.1. Fundamental Physical Scenarios: ID testing samples unseen points from the training grid, whereas OOD testing uses initial radii and velocities outside the training range.OOD cases include velocity-only, radius-only, or simultaneous velocity-and-radius shifts.
  • 3.2. Perfect ID and Failed OOD Generalization: 0.012: DiT-L’s uniform-motion ID velocity error with 3M data, approaching the 0.010 ground-truth-video error and improving over DiT-S’s 0.022 with 30K data.Increasing model size or data consistently decreases ID velocity error across all three tasks.
  • 3.2. Perfect ID and Failed OOD Generalization: 0.427: DiT-L’s uniform-motion OOD error with 3M data, versus 0.012 for its ID error under the same setting.Across settings, OOD errors are an order of magnitude higher than ID errors.
  • 3.2. Perfect ID and Failed OOD Generalization: 0.433, 0.328, and 0.358: DiT-B’s uniform-motion OOD errors with 30K, 300K, and 3M data, showing no reliable improvement from scaling.DiT-XL trained on the 3M uniform-motion dataset also showed no OOD improvement.

4. Combinatorial Generalization

The combinatorial evaluation uses complex deterministic object interactions and tests combinations not seen during training. Increasing template coverage and model capacity improves out-of-template performance, including a large reduction in human-rated abnormal videos.

  • 4. Combinatorial Generalization: Combinatorial generalization tests whether models can combine familiar physical interactions to predict new scenarios rather than derive precise laws directly.The study contrasts this ability with the failure observed in OOD physical-law prediction.
  • 4.1. Combinatorial Physical Scenarios: PHYRE provides deterministic 2D scenes containing fixed or dynamic balls, jars, bars, and walls with simultaneous collisions, trajectories, rotations, and friction.This complexity supports evaluation of unseen physical interactions while retaining deterministic underlying laws.
  • 4.1. Combinatorial Physical Scenarios: Six, 30, and 60 training templates produce 0.6M, 3M, and 6M videos, while 10 unused templates form the out-of-template test set.The template design varies coverage of object combinations and reserves unseen combinations for evaluation.
  • 4.2. Scaling Law Observed for Combinatorial Generalization: 67% to 10%: the abnormal-video rate decreases as training templates increase from 6 to 60.All reported metrics improve on out-of-template testing sets when template coverage increases.

5. Deeper Analysis

The analysis probes interpolation, extrapolation, memorization, attribute preferences, and compositional generalization in video models. Models interpolate better when training gaps narrow, can combine some learned events, but often retrieve similar training cases rather than abstract physical rules.

  • Understanding Generalization from Interpolation and Extrapolation: Narrowing the gap between adjacent training regions improves test performance, indicating stronger interpolation rather than true extrapolation.Reintroducing part of a missing range also strengthens interpolation without increasing total data.
  • Understanding Generalization from Interpolation and Extrapolation: OOD collision points inside the training set’s convex hull generalize well, whereas points outside the hull produce large prediction errors.This discrepancy shows that OOD difficulty depends on the geometric relationship between training and test velocities.
  • Memorization or Generalization: Video models generalize case-by-case, using similar training examples instead of consistently abstracting the underlying physical rule.Deceptive or reversed-motion examples can cause predictions such as sudden negative velocities.
  • How Does Diffusion Model Retrieve Data?: Across pairwise tests, attribute preference follows color > size > velocity > shape, with shape least prioritized during case matching.Color remained fixed while size, velocity, or shape changed in the reported comparisons.
  • Complex Combinatorial Generalization: Models combine learned physical events across attributes, time, and space to generate videos containing complex event chains.Examples include simultaneous motions and collisions near walls with unchanged rebound velocity.
  • Is Video Sufficient for Complete Physics Modeling?: Visual ambiguity can yield physically incorrect predictions despite visually plausible outputs, limiting fine-grained physics modeling from video alone.Pixel-level size differences and ambiguous object positions can change whether a passage or collision occurs.

6. Related Works

Related work covers diffusion and autoregressive approaches for open-set video generation and world models that predict environment evolution from actions or controls. This paper instead studies physical-law discovery from initial states without per-step actions or conditions.

  • Video generation: Open-set video generation primarily uses diffusion or autoregressive models, often with pretrained image or video VAEs for compression.Some approaches also leverage pretrained text-to-image models for zero-shot or related generation settings.
  • World model: World models predict how environments evolve after actions and have supported planning, policy learning, autonomous driving, and robot control.Visual world models commonly condition predictions on driver operations or control signals.
  • World model: This paper’s physical-law discovery setting predicts visual evolution from an initial state without requiring per-step actions or conditions.The physical event is determined by the underlying laws once the initial state is specified.

7. Conclusion and Discussion

The study finds that scaling improves generalization in some settings but does not enable video models to discover universal physical laws. Models instead rely on similar training examples, with implications for the diversity required in real-world video data.

  • Scaling alone cannot resolve out-of-distribution generalization, although it improves performance in other scenarios.
  • Video models generalize more by referencing similar training examples than by learning universal physical rules.
  • The observed case-based prioritization order is color > size > velocity > shape.
  • Increasing combinatorial diversity in training data is suggested as a path for addressing the vast, continuous, high-dimensional variation of real-world video.

B.4. DiT Implementation Details

The DiT denoising models use a 3D RoPE variant and are trained with AdamW under a cosine-decay learning-rate schedule.

  • The denoising model follows DiT and modifies it with a 3D variant of RoPE for video data.
  • Training uses AdamW with β1 = 0.9, β2 = 0.999, weight decay 0.01, and a maximum learning rate of 1×10−4 with cosine decay.
  • The models use a batch size of 256, with horizontal flipping as the only data augmentation unless otherwise specified.

C.1. Fundamental Physical Scenarios Data

The simulator defines controlled in-distribution and out-of-distribution ranges for object radius and velocity, then samples physical scenarios for training and evaluation. It also uses human judgments to measure physically abnormal generated videos.

  • The Box2D simulator uses a 10 × 10 grid, 0.1-second timestep, and 32 frames spanning 3.2 seconds.
  • In-distribution ranges are r ∈[0.7, 1.5] and v ∈[1, 4], while OOD ranges extend beyond these radius and velocity intervals.
  • Collision experiments sample training data from a 4-dimensional in-distribution joint space and evaluate on held-out in-distribution and OOD grid points.
  • Uniform and parabolic motion experiments use 3k, 30k, and 3M training samples with OOD tests varying radius, velocity, or both.
  • Human evaluation defines abnormal ratio as the proportion of videos judged physically implausible, using 60 training templates and 10 unused evaluation templates.

D.1. Is the Prioritization Relevant to VAE?

Experiments test whether the prioritization pattern depends on the VAE and whether pretrained or multimodal models overcome the observed generalization behavior. The reported results indicate that the prioritization persists across VAEs and that added conditions do not improve OOD prediction.

  • The prioritization order color > size > velocity > shape remains consistent when the primary VAE is replaced with CogVideo’s VAE.
  • Across 1,400 test cases, velocity was prioritized over shape without observed exceptions.
  • Figure 12 compares velocity with size and color with velocity in CogVideo-VAE prioritization experiments.
  • SVD’s diffusion component performs worse than DiT-B on both ID and OOD predictions, while its OOD error is an order of magnitude larger than its ID error.
  • Adding numeric or text conditions produces ID errors comparable to vision alone, but increases OOD errors, especially with language conditions.

E.2. Continuous Experiment for Pairwise Comparison

Continuous pairwise experiments show that models preserve attributes well when no conflict exists, but attribute conflicts induce systematic shifts reflecting prioritization.

  • Most velocity–size test cases preserve both attributes beyond the training distribution, with a slight preference for size at extreme values.The preference appears particularly for extreme radius and velocity combinations.
  • Predictions remain accurate in control experiments using only red objects, even under extreme conditions.This isolates the observed drift from general prediction behavior.
  • When trained on fixed color–size pairs, models retain initial colors while producing slight shifts toward training-associated sizes on reversed pairs.Training uses small red and large blue balls, while evaluation reverses those color–size pairings.
  • Attribute drift arises specifically when conflicting attribute combinations require the model to choose between learned associations.The control comparison attributes the shift to attribute conflict and prioritization.

E.3. Principle Behind Data Retrieval in the Diffusion Model

The model’s combinatorial behavior is explained by case-based retrieval rather than universal physical abstraction, with attribute priorities linked to visual or latent similarity.

  • The model prioritizes color over size, velocity, and shape when matching test cases to training examples.The reported order is color > size > velocity > shape.
  • The proposed mechanism relates attribute prioritization to distances between test conditions and training examples in VAE latent space.Pixel-space comparisons are used for clearer illustration, while training minimizes VAE-latent prediction loss.
  • Changing a ring into a ball can require substantial pixel variation, altering which attribute appears preferable during case matching.The blue-ball/red-ring experiment makes shape transformation visually costly compared with simpler changes.
  • Missing red-ball bounce examples can cause the model to make the ball disappear after collision by retrieving cases without that attribute.This failure illustrates how segment retrieval can produce physically unrealistic compositions.
  • Training loss is tracked for a DiT-L model trained on 3M collision videos.The figure identifies the monitored loss curve as the training loss from Equation (2).

F. Comparison with ID/OOD Generalization Works

The paper argues that video models can succeed on in-distribution cases yet fail to learn physical laws as universal rules, while broader combination diversity improves combinatorial generalization.

  • These findings provide a more detailed account of the limitations and potential of video generation models than a simple ID-success/OOD-failure summary.The paper frames its contribution as extending traditional ID/OOD analysis.
  • The study challenges the assumption that scaling video generation models makes them learn universal physical laws.Its systematic experiments conclude that current models fail to learn physical laws as universal rules.
  • Increasing training-combination diversity is more effective for realistic physics than merely increasing data volume or model size.This is presented as guidance for improving combinatorial generalization.
  • The models primarily reference similar training examples rather than learning underlying universal principles.This case-based mechanism helps explain their limitations and biases in representing physical phenomena.
  • In-distribution evaluation cases show very small prediction errors, whereas out-of-distribution cases show large prediction errors.The corresponding visualizations contrast the two evaluation regimes directly.
  • Some out-of-template cases appear physically plausible and closely match ground truth, while others violate physical laws through disappearance, breakage, teleportation, or floating.The abnormal examples include implausible object persistence, shape maintenance, collisions, and gravity behavior.
Loading 2411.02385v2…