Source-linked AI summary
Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
Haodong Li, Shaoteng Liu, Tianyu Wang, Chongjian Ge, Sihui Ji, Jiahan Zhang, Xin Lin, Haolin Lu, Zhe Lin, Manmohan Chandraker
TL;DR
Video world models often fit pixels without explicitly learning how scenes evolve. LDR reasons over structured latent dynamics and extrapolates more reliably beyond its training distribution.
Problem
Existing video world models lack explicit dynamics reasoning, limiting evidence that they learn laws of motion beyond observed training distributions.
Method
LDR encodes frames into structured latents, applies explicit kinematic integration to their evolution, and decodes predicted latents into frames in one feed-forward pass.
Results
At 256^2 resolution, LDR’s joint average OOD position error shrinks to 0.068 from 0.114 at 128^2, while DiT-S grows to 0.592 from 0.222.
Takeaways & Limitations
LDR demonstrates extrapolation beyond benchmark training ranges and robustness to large appearance shifts, including unseen textured balls where DiT-S fails.
Takeaways & Limitations
Validation is limited to simulated scenarios with simple objects; richer real-world scenes and larger models remain future work.
Abstract
from arXiv · showhide
The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20$\times$ smaller than the video diffusion baseline's, under both single- and joint-task training at 256$^2$ resolution, while using 26$\times$ fewer parameters and running 143$\times$ faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/
Abstract · Introduction · Related Works
LDR is introduced as a video world model that explicitly reasons about latent dynamics through kinematic integration in structured latent space, targeting extrapolation beyond the training distribution. On a controlled white-box physics benchmark, it substantially improves out-of-distribution extrapolation while using fewer parameters and less runtime than a video diffusion baseline.
- Related Works: LDR is validated only on simulated scenarios with simple objects, while scaling to richer real-world scenes with larger models remains future work.The stated scope limits the current validation rather than the proposed principle.
- Introduction: LDR models latent transitions through explicit kinematic integration, regressing only third- and higher-order residuals while numerically integrating lower-order dynamics.This architecture is designed to build dynamics reasoning into the model rather than fit pixel transitions directly.
- Abstract: LDR generalizes from red balls moving left-to-right to blue squares moving right-to-left under severe distribution shift.This example tests whether the model learned dynamics rather than merely memorized visual appearance or motion direction.
- Abstract: The evaluation uses five controlled white-box physics tasks: uniform motion, parabola, collision, bouncing, and looming.The benchmark focuses on out-of-distribution scenarios that expose whether the underlying dynamics were learned.
- Related Works: LDR performs this integration in a structured latent space instead of dense convolutional features to improve extrapolation.The structured latent is encoded from input frames, rolled out through integrated dynamics, and decoded into predicted RGB frames.
- Related Works: Structured representations such as geometric coordinates or object-centric slots are motivated by prior sequential-modeling results that outperform unstructured convolutional features.This supports LDR’s choice to reason in structured latent space.
Method
LDR predicts future frames by encoding inputs into structured latents, explicitly integrating their dynamics, and decoding predicted latents in one feed-forward pass. It learns only high-order dynamics residuals while fixed kinematic integration propagates lower-order motion, promoting extrapolation beyond the training distribution.
- Architecture: LDR encodes each input frame into a structured latent, explicitly integrates latent dynamics, and decodes each predicted latent back to a frame in one feed-forward pass.The method uses no test-time optimization or iterative solvers.
- Latent dynamics integration: LDR conditions on three frames, estimates first- and second-order latent derivatives, and regresses only the third- and higher-order residual during rollout.Three frames provide the second-order finite-difference initialization.
- Latent dynamics integration: Only the residual predictor is learned; a fixed integration chain propagates the residual through second- and first-order latents to produce each future latent.The model rolls out future latents one at a time using explicit kinematic integration.
- Extrapolation: Explicit kinematic integration imposes an inductive bias toward learning how the state evolves, whereas an unbiased regressor can flatten toward the training mean or reproduce nearby training examples off-support.This design targets underlying dynamics rather than merely predicting the next state.
- Representation: LDR reasons in structured latents rather than dense convolutional features, reducing entanglement with appearance and semantics and making extrapolation more stable and reliable.The structured representation is described as compact and free of dynamics-irrelevant detail.
- Training: All LDR components are trained jointly from scratch with no pretrained modules, using RGB reconstruction, RGB rollout, and latent rollout supervision.The rollout horizon grows from short to full during training to stabilize long-horizon backpropagation.
Experiments
Experiments evaluate LDR on a controlled five-task physics benchmark using ID-only training and OOD extrapolation tests. Across resolutions and training settings, LDR achieves substantially smaller OOD gaps, better scaling, higher efficiency, and robustness to large visual shifts.
- Benchmark and protocol: The benchmark spans uniform motion, parabola, collision, bouncing, and looming, with OOD conditions extending speed, radius, and scale-rate ranges beyond training.All models train on ID samples and predict 29 future frames from three observed frames.
- Quantitative comparison: At 256^2, LDR’s averaged position-error ID-OOD gap is 23.9× and 27.7× smaller than DiT-S under single-task and joint five-task training, respectively.Under joint training, DiT-S OOD position error rises from 0.086 to 0.592, whereas LDR changes from 0.050 to 0.068.
- Quantitative comparison: From 128^2 to 256^2, joint-training OOD position error increases for DiT-S from 0.222 to 0.592 but decreases for LDR from 0.114 to 0.068.The reported directions also hold under single-task training, indicating opposite resolution-scaling behavior.
- Ablation study: Removing dynamics reasoning raises the single-task averaged position-error ID-OOD gap from LDR’s 0.013 to 0.168 at 256^2, while replacing the structured latent yields gaps of 0.133 versus DiT-S’s 0.506 under joint training.The structured-latent replacement remains second best overall, with 0.090 versus 0.300 under single-task training.
- Efficiency: LDR uses 4.1M parameters versus DiT-S’s 106.1M and runs up to 143× faster at 256^2 in a single feed-forward pass without iterative diffusion sampling.The baseline uses 50 denoising steps, whereas LDR performs no test-time optimization.
- Qualitative comparison: LDR predicts motion accurately for unseen earth-textured balls despite training only on red balls, while DiT-S fails under this larger visual distribution shift.The reported robustness draws on both structured-latent encoding/decoding and dynamics reasoning.
Summary
The paper introduces Latent Dynamics Reasoning (LDR), an extrapolative video world model that predicts by reasoning about latent dynamics rather than directly regressing future frames. It claims that LDR is the first video world model to extrapolate learned dynamics beyond its training distribution, while noting limitations from its latent representation.
- Contribution: LDR predicts future video by reasoning about latent dynamics instead of directly regressing future frames.This lets the model learn how the world evolves and carry that knowledge beyond observed examples.
- Novelty: LDR is presented as the first video world model that extrapolates learned dynamics beyond its training distribution.
- Limitation: LDR’s dynamics reasoning is content-agnostic, but its latent representation limits practical universality.The structure latent encodes image-feature structure rather than content, so it may not model dynamics expressed through appearance, such as color changes over time.