PaperlayerView exact revision on arXiv

Source-linked AI summary

Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics

Ying Shen, Jerry Xiong, Tianjiao Yu, Ismini Lourentzou

arXiv:2604.08503v3cs.CV

TL;DR

Current video generators can look realistic without reliably respecting the physical principles governing real-world dynamics. Phantom jointly models visual content and latent physical dynamics, improving physical consistency across physics-aware benchmarks while preserving competitive perceptual fidelity.

  • Problem

    Current video generation models struggle to comply with fundamental physical principles, raising whether scaling data and model capacity can yield genuine physical understanding.

  • Method

    Phantom augments a pretrained video diffusion model with a physics branch that jointly infers latent physical dynamics and generates future frames from observed frames and physical states.

  • Results

    Phantom consistently outperforms the Wan2.2-TI2V baseline across three physics-focused benchmarks, with 50.4% PC improvement on VideoPhy, 2.6% PC improvement on VideoPhy-2, and 33.9% gain on Physics-IQ.

  • Takeaways & Limitations

    Phantom produces videos with improved physical plausibility while preserving or enhancing perceptual quality across physics-aware and general benchmarks.

  • Takeaways & Limitations

    Phantom represents physical states as latent, learned physics-aware embeddings rather than explicitly specifying the underlying physical properties.

Abstract

from arXiv · show

Recent advances in generative video modeling, driven by large-scale datasets and powerful architectures, have yielded remarkable visual realism. However, emerging evidence suggests that simply scaling data and model size does not endow these systems with an understanding of the underlying physical laws that govern real-world dynamics. Existing approaches often fail to capture or enforce such physical consistency, resulting in unrealistic motion and dynamics. In his work, we investigate whether integrating the inference of latent physical properties directly into the video generation process can equip models with the ability to produce physically plausible videos. To this end, we propose Phantom, a Physics-Infused Video Generation model that jointly models the visual content and latent physical dynamics. Conditioned on observed video frames and inferred physical states, Phantom jointly predicts latent physical dynamics and generates future video frames. Phantom leverages a physics-aware video representation that serves as an abstract yet informaive embedding of the underlying physics, facilitating the joint prediction of physical dynamics alongside video content without requiring an explicit specification of a complex set of physical dynamics and properties. By integrating the inference of physical-aware video representation directly into the video generation process, Phantom produces video sequences that are both visually realistic and physically consistent. Quantitative and qualitative results on both standard video generation and physics-aware benchmarks demonstrate that Phantom not only outperforms existing methods in terms of adherence to physical dynamics but also delivers competitive perceptual fidelity.

1. Introduction

Current video generators can produce visually plausible sequences yet struggle to follow fundamental physical principles, and scaling data or model capacity appears insufficient for learning generalizable physical laws. Phantom addresses this gap by jointly modeling visual content and latent physical dynamics within video generation to improve physical consistency while preserving visual realism.

  • Problem: Despite high visual fidelity, current video generation models often fail to comply with fundamental physical principles governing real-world dynamics.
  • Motivation: Scaling model capacity or dataset size appears insufficient because models may memorize cases instead of internalizing generalizable physical laws.
  • Contribution: Phantom jointly models visual content and latent physical dynamics within the video generation process.
  • Method: Phantom augments a pretrained video diffusion model with a dedicated physical dynamics branch that infers and predicts latent dynamics alongside video content.
  • Results: Phantom integrates latent physical dynamics directly into video generation, improving physical consistency without sacrificing visual realism across standard and physics-aware benchmarks.

2. Related Work

Video diffusion and flow-matching models enable high-fidelity, visually realistic video generation, but often lack physical plausibility. Phantom addresses this gap by jointly modeling physics-aware latent embeddings and visual content during synthesis.

  • Video Diffusion Models and Flow Matching: Diffusion probabilistic models and flow-matching models have emerged as powerful paradigms for high-dimensional visual data, enabling high-fidelity generation in images and videos.
  • Video Diffusion Models and Flow Matching: Large-scale text-to-video models such as Sora, HunyuanVideo, and Wan2.2-TI2V-5B demonstrate visual realism, temporal coherence, and open-domain generalization.
  • Physics-aware Video Generation: Despite strong visual synthesis, modern video generation models often violate basic principles of motion, gravity, or material interaction.
  • Physics-aware Video Generation: Phantom integrates physical reasoning into generation by jointly modeling physics-aware latent embeddings alongside visual content, allowing physical dynamics to evolve during synthesis.

3. Preliminaries

The preliminaries describe flow-based generative models as learning a time-dependent velocity field that transports samples from a simple source distribution to a complex target distribution. They introduce Conditional Flow Matching and linear conditional flows, followed by inference through ODE integration.

  • Flow-based generative models learn a time-dependent velocity field uθ_t that transports samples from a simple source distribution p0(x) to a complex target distribution p1(x).
  • Conditional Flow Matching directly regresses uθ_t on a conditional vector field u_t(· | x1) without requiring simulation.
  • A linear conditional flow defines x_t = (1 − t)x1 + tx0 with conditional velocity u_t(x_t | x1) = x1 − x0.
  • At inference, the model samples x0 ∼ N(0, 1) and obtains x1 ∼ p1(x) by integrating uθ(x_t, t) with an Ordinary Differential Equation solver.

4. Phantom Method

Phantom jointly generates future video frames and latent physical dynamics by coupling visual and physics representations throughout the model. Its dual-branch design combines pretrained visual generation with physics-aware inference, cross-modal attention, and selective training strategies.

  • Joint formulation: Phantom models future video frames and latent physical states jointly, conditioning their prediction on observed frames and an optional textual prompt.The latent physical states encode physically meaningful properties in a learned physics-aware representation space.
  • Dual-branch architecture: Two parallel latent flow-matching branches predict visual trajectories and physical dynamics, exchanging information through symmetric Vis-Attention and Phy-Attention modules.The branches retain modality-specific hidden states while coordinating visual and physical reasoning through cross-modal coupling.
  • Latent representations: Observed videos are encoded into complementary visual and physical latent sequences using a pretrained video VAE and V-JEPA2.V-JEPA2 representations capture intuitive physics concepts including object permanence, collisions, and gravity.
  • Training Strategies: Phantom freezes pretrained video-branch parameters and trains the physics branch with the dual cross-attention layers to add physical reasoning while preserving visual generation quality.The model also injects textual prompts and flow-matching timesteps into both branches for aligned conditioning.
  • Training Strategies: A recursive loss-weight schedule stabilizes optimization by increasing αz from zero, then resetting it when the physics-branch gradient norm exceeds ηz.The schedule addresses the substantially larger magnitude and gradient norm of the physical loss relative to the visual loss.

5. Experimental Setup

Phantom is trained on OpenVidHD-0.4M and evaluated across general-quality and physics-focused benchmarks, including VideoPhy, VideoPhy2, Physics-IQ, and VBench-2. Across these evaluations, it improves physical consistency while maintaining competitive visual quality and semantic alignment.

  • Datasets and evaluation: Phantom is trained on OpenVidHD-0.4M, a diverse dataset of approximately 400K high-resolution video–text pairs not explicitly designed around physical dynamics.The evaluation suite measures both general generative quality and physical awareness.
  • Datasets and evaluation: VideoPhy, VideoPhy2, and Physics-IQ evaluate physical plausibility, commonsense, action-conditioned reasoning, and extrapolation from single-frame or multi-frame observations.VideoPhy and VideoPhy2 use official auto-evaluators for Physical Commonsense and Semantic Adherence metrics.
  • Physics-aware benchmarks: 33.9% higher Physics-IQ performance in the single-frame setting demonstrates substantial improvement over Wan2.2-TI2V, while multi-frame performance remains competitive.Wan2.2-TI2V was not trained to support multi-frame conditioning.
  • General video quality: Phantom improves VBench-2 across nearly all dimensions, particularly Human Fidelity and Physics, while its aggregate Creativity score drops modestly.Composition rises from 40.35 to 45.07 (+11.7%), while Diversity declines from 64.67 to 45.95.
  • Qualitative evaluation: Qualitative comparisons show more physically plausible transformations, pouring, bouncing, momentum transfer, deformable bubbles, and viscous-fluid motion than Wan2.2-TI2V.For text-to-video samples, Phantom jointly denoises visual and physical latent spaces from pure noise without externally provided physics-aware representations.

6. Conclusion

Phantom jointly models visual content and latent physical dynamics through a physics-reasoning branch coupled with a pretrained video diffusion backbone. It generates videos that respect intuitive physical laws while preserving or enhancing perceptual quality, without external simulators, prompt refinement, or post-hoc alignment.

  • Phantom is a physics-infused video generation framework that jointly models visual content and latent physical dynamics.
  • A dedicated physics-reasoning branch coupled with a pretrained video diffusion backbone enables generation respecting visual fidelity and intuitive physical laws.
  • Phantom achieves substantial improvements in physical plausibility while preserving or enhancing perceptual quality across physics-aware and general benchmarks.Qualitative results further support the reported improvements.
  • The framework requires neither external simulators, prompt refinement, nor post-hoc alignment.

Supplementary Material · A. Implementation Details · B. Baselines

The supplementary material describes Phantom’s implementation, evaluation protocols, and conditioning settings. It also identifies Wan2.2-TI2V-5B as the base architecture, with a separately initialized physics branch and frozen visual branch.

  • A. Implementation Details: Phantom builds on Wan2.2-TI2V-5B because it accepts both text and image inputs.The physics branch is integrated into this architecture as described in Section 4.2.
  • A. Implementation Details: The physics branch is initialized from scratch, while the visual branch remains frozen to preserve the base model’s generative prior.
  • A. Implementation Details: Physics-aware embeddings are extracted with V-JEPA2, a pretrained video encoder shown to capture intuitive physics.
  • A. Implementation Details: For VideoPhy-2, Phantom follows the official evaluation protocol and computes SA and PC as proportions of videos rated at least 4 out of 5.Evaluation uses the benchmark’s auto-evaluator and official up-sampled prompts.
  • A. Implementation Details: For Vbench2, results are reported using the benchmark’s original prompts.
  • A. Implementation Details: Physics-IQ evaluation covers both single-frame conditioning with an initial frame and caption and multi-frame conditioning with a short initial clip and caption.

B.1. General-Purpose Video Models · B.2. Physics-Focused Video Models

The paper evaluates Phantom against both strong general-purpose text-to-video baselines and physics-focused methods designed to improve physical plausibility. General-purpose models provide open-domain fidelity but do not explicitly model physical principles, while physics-focused approaches encourage physical adherence through methods such as prompt refinement.

  • B.1. General-Purpose Video Models: Phantom is compared with CogVideoX-5B, HunyuanVideo, Wan2.1-T2I-14B, and Wan2.2-TI2V-5B as state-of-the-art general-purpose T2V diffusion baselines.These models serve as strong baselines for open-domain video generation.
  • B.1. General-Purpose Video Models: Table 4 evaluates VideoPhy and VideoPhy2 using Semantic Adherence (SA) for video-text alignment and fidelity and Physical Commensense (PC) for intuitive real-world physics compliance.The table also distinguishes original-prompt results marked † from detailed-prompt results and highlights improvements over Wan2.2-TI2V.
  • B.1. General-Purpose Video Models: The general-purpose baselines demonstrate open-domain generalization and high-fidelity video synthesis.Their strengths motivate comparison in standard video generation settings.
  • B.1. General-Purpose Video Models: These general-purpose models are not designed to model or enforce physical principles.This distinguishes them from Phantom’s physics-infused modeling objective.
  • B.2. Physics-Focused Video Models: The evaluation also includes recent physics-focused video generation approaches that aim to improve physical plausibility.This comparison complements the general-purpose baseline evaluation.
  • B.2. Physics-Focused Video Models: PhyT2V uses large language models to iteratively refine prompts through chain-of-thought and step-back reasoning.Repeated prompt analysis and rewriting guide existing text-to-video models toward greater adherence to real-world physical laws.
  • B.2. Physics-Focused Video Models: PhyT2V improves physical-law adherence without retraining the underlying generation model.Its intervention operates through repeated prompt analysis and rewriting.

C. Additional Results

Additional results show that Phantom improves physical commonsense and semantic fidelity over Wan2.2-TI2V, especially with original prompts, while also strengthening broad perceptual and temporal video quality. Ablations further test replacing the VJEPA2 encoder with VideoMAEv2 under the same training setup.

  • Quantitative Results: Phantom achieves substantial gains over Wan2.2-TI2V-5B on VideoPhy and VideoPhy2 under both original and detailed prompts.The detailed-prompt setting follows VideoREPA’s evaluation protocol.
  • Quantitative Results: Phantom’s gains are especially pronounced with original prompts, indicating intrinsic physics-awareness without enriched textual descriptions.VideoREPA uses the considerably stronger CogVideoX-5B backbone, yet Phantom still improves substantially over its own base model.
  • Quantitative Results: Across 18 VBench-2 metrics, Phantom outperforms Wan2.2-TI2V-5B on most dimensions, including physics, perceptual realism, semantic consistency, and temporal coherence.Table 5 reports fine-grained text-to-video evaluation across all 18 metrics.
  • Ablation Studies: The ablation replaces the VJEPA2 encoder with VideoMAEv2 while keeping the Wan2.2-TI2V training setup unchanged.VideoMAEv2 serves as the alternative video encoder in this ablation.

D. Physics-based Video Control

Phantom extends physics-infused video generation to explicit local point-force control by combining force representations and textual physical conditions with visual inputs. Fine-tuning enables dynamically evolving, physically plausible motion consistent with applied forces.

  • Evaluation setting: Phantom is evaluated on Force-Prompting’s local point-force setting, where localized external forces act on objects at specified image coordinates.The dataset pairs video sequences with temporally aligned force annotations describing external interactions applied to static images.
  • Force representation: Point-force annotations become 256×256 force-tensor videos encoding the forces’ spatial distribution and temporal evolution.The V-JEPA2 encoder processes these force videos into physics-aware embeddings for Phantom’s physics branch.
  • Dual-branch conditioning: The physics branch receives a natural-language prompt specifying force coordinates, magnitude, and direction, while the video branch uses the original caption.At inference, Phantom conditions on a static image and the force-tensor sequence’s first frame to synthesize the resulting dynamics.
  • Results: After fine-tuning for 1.1K steps, Phantom synthesizes dynamic, physically plausible motion that evolves consistently with applied forces.The force-conditioned examples demonstrate generalization to force-based control signals.
Loading 2604.08503v3…