Source-linked AI summary
Inference-time Physics Alignment of Video Generative Models with Latent World Models
Jianhao Yuan, Xiaofeng Zhang, Felix Friedrich, Nicolas Beltran-Velez, Melissa Hall, Reyhane Askari-Hemmat, Xiaochuang Han, Nicolas Ballas, Michal Drozdzal, Adriana Romero-Soriano
TL;DR
Video generators often produce visually appealing but physically implausible videos, and the paper investigates whether inference strategies contribute to this gap. It introduces WMReward, which uses a latent world model’s physics prior to search and steer denoising trajectories, yielding substantial improvements across conditioning settings and benchmarks. The results support latent world models as inference-time rewards without further training, while their limited coverage of phenomena such as weight and friction remains a scope boundary.
Problem
Video generative models often violate basic physics despite producing promising visual content, motivating investigation of physics plausibility beyond pre-training limitations.
Method
WMReward uses VJEPA-2’s surprise score as a latent-world-model reward for Best-of-N selection and guidance while searching physically plausible videos in the learned generation manifold.
Results
62.0% on PhysicsIQ surpassed the previous best by 6.78%, with an additional 11.4% improvement over baselines in human preference and gains across T2V, I2V, and V2V settings.
Takeaways & Limitations
Latent world models can provide effective physics-plausibility rewards for inference-time video alignment without requiring further training.
Takeaways & Limitations
VJEPA-2’s limited training data does not cover all physics phenomena, and material properties such as weight and friction remain limited.
Abstract
from arXiv · showhide
State-of-the-art video generative models produce promising visual content yet often violate basic physics principles, limiting their utility. While some attribute this deficiency to insufficient physics understanding from pre-training, we find that the shortfall in physics plausibility also stems from suboptimal inference strategies. We therefore introduce WMReward and treat improving physics plausibility of video generation as an inference-time alignment problem. In particular, we leverage the strong physics prior of a latent world model (here, VJEPA-2) as a reward to search and steer multiple candidate denoising trajectories, enabling scaling test-time compute for better generation performance. Empirically, our approach substantially improves physics plausibility across image-conditioned, multiframe-conditioned, and text-conditioned generation settings, with validation from human preference study. Notably, in the ICCV 2025 Perception Test PhysicsIQ Challenge, we achieve a final score of 62.64%, winning first place and outperforming the previous state of the art by 7.42%. Our work demonstrates the viability of using latent world models to improve physics plausibility of video generation, beyond this specific instantiation or parameterization.
1 Introduction
The paper reframes physics plausibility in video generation as an inference-time alignment problem, using latent world models to search and steer generation toward physically plausible videos. WMReward improves plausibility across conditioning settings and achieves state-of-the-art benchmark performance.
- Video generators produce visually pleasing content but often violate physics, limiting applications such as reliable world modeling, robotics, and autonomous driving.
- Inference-time physics improvement is underexplored compared with approaches that inject physics information during pre-training or post-training.
- Latent world models emphasize predictive features such as motion, object dynamics, structure, object permanence, and trajectory continuity, making them suitable physics-plausibility rewards.
- WMReward repurposes VJEPA-2’s surprise score as a physics plausibility reward for Best-of-N selection and guidance during generation.
- The method improves physics plausibility across T2V, I2V, and V2V settings, with guided sampling showing more promising scaling as search space increases.
- 62.0% on the PhysicsIQ benchmark surpassed the previous best by 6.78%, while human preference results showed an 11.4% improvement over baselines.
2 Methodology: WMReward
WMReward transfers a latent world model’s physics prior into a reward signal that searches or steers video-generation trajectories toward physically plausible outputs. It evaluates generated futures against predictions from context frames and supports both gradient-based guidance and gradient-free Best-of-N sampling.
- Reward-weighted alignment: WMReward repurposes VJEPA-2’s future-prediction surprise as a reward for aligning video generation with physical plausibility.The method treats generation as sampling from a reward-weighted tilted distribution.
- Reward-weighted alignment: The reward-weighted target distribution uses the pre-trained video-model distribution p(x) and a positive weighting function w(x) constructed from physics-plausibility reward r(x).The formulation separates defining the reward from realizing sampling under the tilted distribution.
- Latent world model: A latent world model encodes observations into compact representations and predicts future states in latent space, emphasizing motion and object dynamics over pixel details.VJEPA-2 is used as the reward-model foundation because it demonstrates strong physics understanding and benchmark performance.
- Reward construction: WMReward slides a window of C + M frames across each video, predicts future representations from C context frames, and compares them with representations of the generated future.The future portions of the predicted and generated representations are compared, with closer matches receiving higher rewards.
- Sampling schemes: The method combines gradient-based guidance with gradient-free Best-of-N search to sample higher-reward videos from the generative model.Guidance uses reward gradients during sampling, whereas Best-of-N generates independent candidates and selects the highest-reward sample; their combination provides stronger tilting without requiring large λ values.
3 Experiments
Experiments evaluate WMReward across image-, multiframe-, and text-conditioned video generation, comparing sampling strategies, human preferences, scaling behavior, and computational tradeoffs. WMReward consistently improves physics-related metrics, with gains that scale with inference compute but may trade off semantic adherence.
- Image and Multiframe-conditioned Generation: WMReward consistently outperforms vanilla sampling for vLDM, MAGI-1, and Sora2 across PhysicsIQ baselines and evaluation dimensions.
- Image and Multiframe-conditioned Generation: 62.0% final PhysicsIQ on V2V with ∇+BoN exceeds previous state-of-the-art MAGI-1 by 6.78%.For I2V, WMReward surpasses previous state-of-the-art Sora2 by 4.13% on PhysicsIQ.
- Human Study: Human evaluations compare physics plausibility, visual quality, and prompt alignment, with physics plausibility showing the most remarkable winning rate.The study compares vanilla sampling against WMReward(∇+BoN) for vLDM I2V and MAGI-1 V2V generations.
- Text-conditioned Generation: WMReward improves text-conditioned Physics Consistency for MAGI-1 and vLDM by 8.1% and 6.9%, respectively, surpassing all baseline models.Semantic Adherence decreases, potentially because VJEPA surprise excludes text-condition information.
- Analysis and Ablation: Increasing particle count steadily improves PhysicsIQ scores, reduces variance, and concentrates distributions in the high-score region; guidance further sharpens the upper tail.WMReward can be adapted to available computation budgets, offering compute-performance tradeoffs.
- Analysis and Ablation: WMReward produces small image- and aesthetic-quality improvements alongside better temporal consistency, motion smoothness, and temporal flickering.These perceptual improvements are associated in the authors’ analysis with stronger physics plausibility and artifact suppression.
4 Related Work
Related work connects video physics failures to model training and explores inference-time steering, while latent world models provide compact predictive representations suited to physics understanding.
- Video Diffusion Models: Video generative models achieve promising visual quality but often produce physically implausible videos, motivating inference-time physics improvement.
- Physics Understanding in Vision Models: Latent world models encode observations into compact representations and learn latent transition functions that forecast future states, emphasizing predictive motion and object dynamics.VJEPA models have shown emergent intuitive physics properties such as object permanence and shape consistency.
- Inference-Time Alignment for Image Diffusion Models: Inference-time alignment methods steer pretrained generative models toward desired properties through derivative-free search or reward-guided denoising.These methods use reward signals to refine local steps or enable broader exploration.
5 Conclusion and Discussion
WMReward demonstrates that latent world models can serve as inference-time reward signals for improving video physics plausibility without further training. The discussion identifies model quality, reward coverage, and search reliability as key improvement directions.
- WMReward uses latent world models, including VJEPA-2, as reward signals to improve physics plausibility during video generation inference.
- The method yields substantial improvements on the PhysicsIQ and VideoPhy benchmarks without requiring further training.
- Improvement avenues: The quality of the video generative model constrains the set of candidate solutions that WMReward can explore.
- Improvement avenues: VJEPA-2 does not cover all physical phenomena because of limited training data, with weight and friction understanding remaining limited.
- Improvement avenues: Early diffusion intermediates are blurry, potentially making reward predictions unreliable during search.
A Implementation Details
The implementation details section introduces the generation settings and explains how WMReward is adapted across generation paradigms.
- The section covers WMReward’s generation settings and adaptation to different video-generation paradigms.
A.1 Generation Settings
The experiments use vLDM and MAGI-1 across holistic and autoregressive generation, adapting WMReward search and guidance to each paradigm. Implementation also includes sliding-window VJEPA surprise rewards and VLM reward baselines.
- Generation models: Experiments use a vLDM transformer with a spatiotemporal VAE and MAGI-1-24B, an autoregressive diffusion model using block-causal attention.
- Generation settings: PhysicsIQ videos are generated at five seconds, using 49 vLDM frames trimmed to 40 frames at 8 FPS.
- Generation paradigms: WMReward is implemented for holistic and autoregressive generation; Best-of-N search is shared, while guidance implementation varies.
- Autoregressive generation: Sliding windows divide videos into context and prediction chunks, whose VJEPA surprise scores are averaged across the sequence.
- Autoregressive generation: Autoregressive guidance uses previous denoised chunks as context to predict the next chunk and calculate VJEPA’s surprise reward.
- VLM rewards: VLM reward baselines extract the logit for token “1” after asking whether generated videos exhibit good physics dynamics.
B Human Study Details
The human study presents paired generated videos with their prompt and conditioning frames, then aggregates judgments into win rates and accuracy scores.
- Study interface: Annotators compare two generated videos side by side alongside the original text prompt and conditioning frames.
- Evaluation criteria: Judgments assess Physics Plausibility and Visual Quality, among three stated evaluation criteria.
- Aggregation: Win rates exclude neutral judgments, while accuracy scores count wins plus half of neutral judgments over the total.
C Qualitative Examples
WMReward produces videos with improved physics plausibility across image-conditioned, multiframe-conditioned, and text-conditioned generation settings. Qualitative examples cover a broad range of physical phenomena.
- WMReward improves physics plausibility in image- and multiframe-conditioned generation examples.The examples are shown in Figures 6 to 8.
- WMReward improves physics plausibility in text-conditioned generation examples.The text-conditioned examples are shown in Figure 9.
- The qualitative samples cover spatial continuity, rigid-body dynamics, fluid behavior, buoyancy, temporal continuity, gravity, conservation of mass, and optical effects.
D Failure Mode Analysis
VJEPA-2 provides a useful but imperfect physics prior for sampling video generations. Failures remain on abrupt events and phenomena requiring complex reasoning or material-property understanding.
- Assumptions: VJEPA-2 is treated as a proxy for true physics dynamics rather than a complete measure of physics plausibility.VJEPA surprise also entangles other perceptual factors.
- Failure Mode Analysis: The method often fails on abrupt events such as fluid overflow or a match igniting a balloon and causing it to explode.These cases require reasoning about sudden state changes.
- Failure Mode Analysis: The method also struggles with mirror reflections and siphon effects that require complex reasoning and understanding of material properties.VJEPA reward can nevertheless correct some conservation-of-mass errors in certain siphon scenarios.