PaperlayerView exact revision on arXiv

Source-linked AI summary

PhiZero: A World Model Built Around Physical Language

Shuyao Shang, Yuqi Wang, Ruopeng Gao, Xu Chen, Tieniu Tan, Lue Fan, Zhaoxiang Zhang

arXiv:2607.28624v1cs.CV

TL;DR

Existing pixel-space video world models often leave physical dynamics implicit and can produce physically inconsistent outcomes. PHIZERO learns a discrete physical language from unlabeled videos, reasons in that representation before rendering, and performs strongly across generation and understanding benchmarks while supporting interactive, action-conditioned, and transferable modeling.

  • Problem

    Pixel-space video world models leave physical dynamics implicit within high-dimensional representations and can produce physically inconsistent outcomes despite visual fidelity.

  • Method

    PHIZERO learns a discrete physical language from unlabeled videos and uses a reason-then-render pipeline to predict world evolution before generating video.

  • Results

    PHIZERO performs strongly across physical video-generation and understanding benchmarks and supports interactive rollouts, action-conditioned simulation, and zero-shot motion transfer.

  • Takeaways & Limitations

    Physical language provides an interface for controllable and transferable physical-world modeling, including sequential control, fine-grained actions, and motion transfer.

  • Takeaways & Limitations

    Physical language is an empirical state-transition representation rather than an interpretable formulation of symbolic physical laws or grounded physical variables.

Abstract

from arXiv · show

We introduce PhiZero, a physical world model built around physical language, a compact discrete representation of world-state transitions. Existing physical world models typically predict future videos directly in pixel space, leaving the underlying world dynamics implicit within high-dimensional visual predictors. Motivated by humans' ability to abstract predictive structure from visual experience and organize it in natural language for explicit reasoning, we learn physical language from in-the-wild videos through self-supervision and use it to explicitly reason about how the physical world evolves. Accordingly, PhiZero adopts a reason-then-render paradigm: it first infers future world evolution as a physical-language sequence and then renders the inferred transitions into videos. Extensive experiments across generation and understanding benchmarks validate the ability of PhiZero to model physically coherent world evolution. We further show its potential for realistic and interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.

1 INTRODUCTION

PHIZERO models physical-world evolution through physical language, a compact discrete representation learned from unlabeled videos, using a reason-then-render process that separates dynamics inference from video synthesis. Experiments support its physical generation and understanding capabilities, interactive rollouts, action-conditioned simulation, and zero-shot motion transfer.

  • Motivation: Video world models must learn how the physical world evolves beyond visual fidelity, whereas pixel-space prediction leaves dynamics implicit in high-dimensional visual predictors.The introduction frames explicit physical-dynamics modeling as necessary for Physical AI.
  • Motivation: Humans abstract generalizable patterns from visual experience, and natural language provides a symbolic space for organizing and explicitly reasoning about such knowledge.This motivates physical language as an explicit substrate for world-evolution reasoning.
  • Physical language: PHIZERO introduces physical language, a compact discrete representation of state-transition patterns learned at scale from in-the-wild videos through self-supervised learning.The representation captures how world states transition across diverse visual experiences.
  • Reason-then-render: PHIZERO first infers future world evolution in physical-language space and then renders the inferred transitions into videos.This reason-then-render paradigm separates dynamics inference from pixel-level synthesis.
  • Results: Experiments validate PHIZERO across physical generation and understanding, with demonstrations of interactive rollouts, action-conditioned world simulation, and zero-shot motion transfer.The results present physical language as an interface for representing, controlling, and transferring state transitions.

2 RELATED WORK

Recent physical world models simulate future visual states by scaling video generation and prediction under language or action conditions. Related work also learns predictive, controllable representations from unlabeled videos for downstream control and motion transfer.

  • Video-based physical world models: Physical world models increasingly scale video generation and prediction to simulate future visual states under language or action conditions.Yang et al. (2023) learns interactive simulators from heterogeneous real-world data, while Wang et al. (2024) and Xiang et al. (2024) use discrete visual prediction and language-conditioned video generation.
  • Predictive and actionable representations: Unlabeled Internet videos support learning controllable latent action spaces from observed frame-to-frame changes.Bruce et al. (2024) and Gao et al. (2025) investigate predictive and actionable representations of world dynamics.
  • Predictive and actionable representations: Robotics and autonomous-driving research transfers latent actions and motion priors learned from videos to downstream control.Schmidt & Jiang (2024), Ye et al. (2025), Chen et al. (2025), Bu et al. (2025b), and Shang et al. (2026) study this direction.

3 METHOD

PhiZero models physical evolution through a discrete physical-language bottleneck: it tokenizes video transitions, predicts them from visual state and textual intent, then renders future video. Its reason-then-render design separates transition reasoning from pixel-level synthesis.

  • Overall architecture: PhiZero combines a Physical Language Tokenizer, a Physical Language Reasoner, and a diffusion decoder to infer and render future world-state transitions.The tokenizer extracts discrete transition sequences, the reasoner predicts them from the first frame and textual action intent, and the decoder generates the future video.
  • Physical Language Tokenizer: The tokenizer models adjacent latent-state transitions with a transition-level Q-Former, preserving temporal order while reducing the complexity of each compressed transition.Shared transition queries attend jointly to neighboring latent states before features are concatenated across intervals and discretized with FSQ.
  • Physical Language Tokenizer: Finite scalar quantization converts transition features into a discrete physical-language sequence without a separately learned codebook.The quantized sequence is projected into the diffusion transformer’s hidden dimension as physical-language context.
  • Diffusion-prior Decoder: A pretrained video diffusion model decodes the compact transition representation, using its generative prior to recover realistic appearance and fine-grained visual details.A pure-noise warm-up forces reconstruction to rely on the physical-language context and first frame rather than corrupted target information or the existing denoising shortcut.
  • Physical Language Reasoner: The Physical Language Reasoner is initialized from a pretrained VLM and autoregressively predicts a length-N FSQ-index sequence from the first frame and textual action intent.Its vocabulary is extended with one atomic symbol for each FSQ index, and supervision comes from encoding training videos with the frozen tokenizer.

4 EXPERIMENTS

PHIZERO is evaluated across video-generation and video-understanding benchmarks, with qualitative results showing more coherent physical consequences than Wan2.2-5B. Ablations and transfer experiments further support the roles of diffusion decoding, transition-level reasoning, simulation data, and appearance-disentangled physical language.

  • Video Generation Results: PHIZERO is evaluated on three video-generation benchmarks measuring physical outcome fidelity, physical-law adherence, and general world-modeling capability.The benchmarks are Physics-IQ Verified, PhyGround, and WorldModelBench.
  • Video Understanding Results: PHIZERO is also evaluated on three pairwise video-understanding benchmarks covering intuitive physics, physical-plausibility discrimination, and temporal-causal understanding.The benchmarks are IntPhys2, LikePhys, and YoCausal.
  • Qualitative Results: Compared with Wan2.2-5B, PHIZERO generates coherent downstream effects such as collision-induced displacement, gravity-driven deformation, chain reactions, and changing shadows.Wan2.2-5B often produces visually plausible interactions but fails to generate expected physical consequences.
  • Physical Language Tokenizer Reconstruction Performance: The Physical Language Tokenizer reconstructs 500 four-second real-world videos while using a compact representation instead of Wan2.2 VAE’s 44,800 continuous visual tokens.Videos are sampled at 8 FPS and evaluated at 512×896 resolution; the Wan2.2 VAE has high reconstruction quality but requires 44,800 continuous visual tokens.
  • Design Ablations: Ablations show that the pretrained diffusion decoder, transition-level Q-Former, and simulation data materially support reconstruction and physically plausible transition prediction.Replacing the diffusion decoder or transition-level Q-Former degrades reconstruction, while removing simulation data reduces reasoner performance; prompt enhancement improves Wan2.2-5B but remains below PHIZERO.
  • Transferability of Physical Language: Physical language preserves state-transition patterns across changed appearances and backgrounds, while its representations exhibit organized kinematic structure across driving and robotic-manipulation behaviors.Transfer experiments and visualization results support appearance disentanglement and meaningful state-transition semantics.

5 CONCLUSIONS

PHIZERO is a physical world model built around physical language, a compact discrete representation learned from unlabeled in-the-wild videos. It predicts future world evolution in physical-language space before rendering it.

  • Conclusions: PHIZERO is a physical world model built around physical language.The paper contrasts this approach with conventional video models that predict future videos directly in pixel space.
  • Conclusions: Physical language compactly represents state-transition patterns learned through self-supervised learning from unlabeled in-the-wild videos.This representation is discrete and is learned from video data without labels.
  • Conclusions: PHIZERO follows a reason-then-render paradigm by predicting future world evolution in physical-language space before rendering it.The passage identifies prediction in physical-language space as the reasoning stage and rendering as the subsequent stage.

A ADDITIONAL TRAINING DETAILS · A.1 PHYSICAL LANGUAGE TOKENIZER · A.2 PHYSICAL LANGUAGE REASONER

The additional training details describe staged, curriculum-based training for the Physical Language Tokenizer and Physical Language Reasoner. The tokenizer uses filtered video and targeted SFT, while the reasoner learns physical-language prediction and motion-focused transitions.

  • A.1 PHYSICAL LANGUAGE TOKENIZER: The tokenizer is pretrained in two phases with a joint temporal and spatial curriculum on approximately 10K hours of filtered unlabeled video.Filtering removes technical defects, inadequate resolution or duration, and abrupt shot transitions.
  • A.1 PHYSICAL LANGUAGE TOKENIZER: Its first three pretraining stages retain 256 × 448 resolution while extending clips from 1 second (9 frames) to 4 seconds (33 frames).The fourth stage keeps 256 × 448 inputs but reconstructs targets at 512 × 896.
  • A.1 PHYSICAL LANGUAGE TOKENIZER: Tokenizer SFT uses stricter filtering based on aesthetic quality, motion magnitude, and VLM-assessed state-transition observability, plus filtered simulation videos.The resulting corpus contains approximately 5M four-second clips.
  • A.2 PHYSICAL LANGUAGE REASONER: The reasoner is trained in two stages: continued pretraining on the tokenizer-SFT clips, followed by motion-focused SFT on approximately 1M clips.Stage 1 uses the first frame and a VLM-generated high-level action caption as inputs, with the frozen tokenizer encoding the full video target.
  • A ADDITIONAL TRAINING DETAILS: The tokenizer training schedule combines curriculum pretraining, full-model fine-tuning, and decoder-only refinement.These phases are summarized as distinct video and optimization settings in Table 12.
  • A.2 PHYSICAL LANGUAGE REASONER: Reasoner continued pretraining adapts the VLM to physical-language prediction, while SFT specializes it for physically informative state transitions.Table 13 identifies these as the two corresponding training settings.
  • A.2 PHYSICAL LANGUAGE REASONER: A captioning prompt supplies textual conditions for the four-second clips used to train the Physical Language Reasoner.The prompt is documented in Table 14.

B ADDITIONAL EVALUATION DETAILS · B.1 PHYSICAL VIDEO GENERATION EVALUATION

PhiZero is evaluated for physical video generation across three benchmarks using the provided first frame and textual condition, following official protocols and released evaluators. The benchmarks assess controlled-experiment outcomes, physical-law adherence, and general world-modeling capability.

  • B ADDITIONAL EVALUATION DETAILS: Three physical video-generation benchmarks evaluate PhiZero from a provided first frame and textual condition under official protocols and released evaluators.The benchmarks are Physics-IQ Verified, PhyGround, and WorldModelBench.
  • B.1 PHYSICAL VIDEO GENERATION EVALUATION: Physics-IQ Verified compares generated videos with real references across 66 controlled physical experiments.It uses Spatial IoU (S-IoU), Spatiotemporal IoU (ST-IoU), and Weighted Spatial IoU (WS-IoU).
  • B.1 PHYSICAL VIDEO GENERATION EVALUATION: Physics-IQ Verified measures scene-change location, timing, and temporal frequency through S-IoU, ST-IoU, and WS-IoU.IQ-Score aggregates these normalized metrics.
  • B.1 PHYSICAL VIDEO GENERATION EVALUATION: PhyGround contains 250 curated image–text conditions spanning 13 physical laws across solid mechanics, fluid dynamics, and optics.Generated videos are sampled at 4 FPS and evaluated with the released PhyJudge-9B evaluator.
  • B.1 PHYSICAL VIDEO GENERATION EVALUATION: PhyGround scores each video from 1 to 5 using criterion-specific rubrics for General Quality and Physics Score.General Quality covers prompt alignment, temporal validity, and object consistency; Physics Score measures adherence to applicable physical laws.
  • B.1 PHYSICAL VIDEO GENERATION EVALUATION: WorldModelBench uses 350 image–text conditions across seven domains and 56 subdomains to evaluate general world-modeling capability.Its domains include autonomous driving, robotics, human activities, industrial scenes, gaming, animation, and natural environments.
  • B.1 PHYSICAL VIDEO GENERATION EVALUATION: WorldModelBench applies a released 2B-VLM judge, with Physics Adherence ranging from 0 to 5 across five physical-consistency criteria.The passage also identifies Common Sense as an evaluation dimension.

B.2 PHYSICAL VIDEO UNDERSTANDING EVALUATION

The evaluation tests whether PhiZero’s physical-language likelihoods distinguish physically possible from impossible videos and natural from reversed videos. All three benchmarks use pairwise likelihood comparisons under shared text conditions, selecting the video with higher sequence likelihood.

  • IntPhys2: IntPhys2 evaluates object permanence, immutability, spatiotemporal continuity, and solidity using 1,012 videos organized into 253 matched quadruplets.Each quadruplet contains two possible and two impossible outcomes per scene.
  • LikePhys: LikePhys measures Plausibility Preference Error (PPE) on controlled valid–invalid pairs from 12 physical scenarios, with lower values indicating stronger physical understanding.PPE is the percentage of comparisons where an invalid video receives equal or higher likelihood than a valid video.
  • YoCausal: YoCausal evaluates 1,232 real-world video pairs by comparing each natural forward video with its temporally reversed counterpart using the same caption.Its Reversal Surprise Index (RSI) is the mean forward-video preference rate across four source datasets, with 50% representing chance.
  • Evaluation Protocol: All three benchmarks use pairwise likelihood comparison under a shared text condition, following benchmark-specific prompts where available.IntPhys2 lacks instance-level text descriptions, whereas LikePhys and YoCausal provide shared prompts for the compared videos.
  • Evaluation Protocol: The higher-likelihood video is selected as the physically valid or temporally natural outcome because the comparison holds the text condition identical.The evaluated video and its corresponding first frame are the only differing inputs within each comparison.

C TRAINING AND INFERENCE DETAILS FOR BROADER APPLICATIONS · C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL

PhiZero is adapted for robotic manipulation and driving by fine-tuning its physical-language components on domain-specific videos and conditioning generation on serialized trajectories. For interactive world modeling, natural-language camera controls are extended beyond four-second clips through sliding-window autoregressive rollout.

  • C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL: For robotic manipulation, PhiZero is adapted on the AGI-Bot RealRobot dataset.The Physical Language Tokenizer is fine-tuned on RealRobot videos to capture robotic-arm and gripper transitions.
  • C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL: Robot-arm joint and gripper-pose trajectories are sampled at 8 FPS from each four-second clip.The trajectories are used as the action sequence for physical-language reasoning.
  • C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL: The reasoner translates robotic action sequences into physical language, and a domain-adapted decoder renders the interaction as future video.This establishes a reason-then-render pipeline for robotic interactions.
  • C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL: For driving, PhiZero constructs a trajectory-conditioned world model on nuScenes.The tokenizer is fine-tuned on nuScenes videos to adapt representation and decoding to driving-scene evolution.
  • C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL: The future four-second ego-vehicle trajectory is sampled at 8 FPS and serialized into a temporally ordered numerical text sequence.The first frame and textualized trajectory condition subsequent model adaptation.
  • C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL: Interactive controls are expressed in natural language as instructions specifying desired changes in camera viewpoint or position.This provides direct language-based control over interactive world-model generation.
  • C.1 CONTROLLABLE AND INTERACTIVE WORLD MODEL: PhiZero extends generation beyond four-second clips with a sliding-window autoregressive rollout.From the current frame, it predicts the next four-second segment and uses that segment’s final frame to continue generation.

C.2 MOTION TRANSFER … D.3 PHYSICALLY REALISTIC VIDEO GENERATION

PhiZero enables unpaired motion transfer by adapting its Physical Language Tokenizer and preserving encoded transitions while changing appearance or embodiment. The related-work discussion positions it among latent-action models, video tokenizers, and approaches to physically realistic generation, emphasizing physical language learned without simulator-derived constraints or predefined physical variables.

  • C.2 MOTION TRANSFER: Motion transfer requires no paired videos: PhiZero briefly adapts the Physical Language Tokenizer to source-domain motions, then directly transfers encoded transitions.Inference encodes a source state transition into physical language before rendering it with GPT-Image 2.0.
  • C.2 MOTION TRANSFER: Human-body transfer edits the source’s first frame to replace the person with a target humanoid robot while decoding the adapted physical-language motion.The tokenizer is briefly fine-tuned on diverse full-body human motions.
  • C.2 MOTION TRANSFER: Human-hand transfer adapts only on human-hand videos, ignoring available human–robot pairings and cross-embodiment correspondences before decoding the edited motion.The videos contain rich hand motions and object interactions.
  • C.2 MOTION TRANSFER: Sim-to-real transfer adapts on LIBERO simulation videos without paired real-world data, edits the first frame into a realistic appearance, and preserves the demonstrated interaction.The unchanged physical-language sequence is decoded conditioned on the edited realistic frame.
  • D.1 LATENT-ACTION WORLD MODELS: Latent-action world models infer compact transition variables from observations, often using inverse-dynamics bottlenecks with forward predictors.Earlier methods learn discrete latent actions for controllable world-model pretraining or downstream policies.
  • D.2 VIDEO TOKENIZERS: Video tokenizers compress spatially and temporally redundant videos, often separating persistent visual content from temporal variation through scene, object, motion, keyframe, or latent tokens.Other approaches factorize videos using content frames, reference images, or hierarchical latent variables.
  • D.3 PHYSICALLY REALISTIC VIDEO GENERATION: Physically realistic video generation commonly uses simulator-derived constraints or physical priors transferred from external models or simulated data.Examples include simulator-derived optical flow, rigid-body simulation, 3D physics simulation, and motion priors from VAE latents.
  • D.3 PHYSICALLY REALISTIC VIDEO GENERATION: PhiZero instead learns physical language from large-scale unlabeled data without simulator-derived constraints, predefined physical variables, or explicit physical rules as supervisory signals.The related work also identifies explicit physical knowledge and intermediate reasoning as a third direction in physically consistent video generation.

E LIMITATIONS AND FUTURE WORK

PhiZero’s physical language remains a data-driven representation rather than an interpretable formulation of symbolic physical laws. Future work includes using it for spatiotemporal understanding, embodied-policy planning, and reducing real-robot interaction data scarcity.

  • Limitations: Physical language is learned empirically from state transitions, but its discrete symbols are not directly grounded in interpretable physical variables or formal laws.This data-driven formulation supports scalable learning from diverse unlabeled videos.
  • Future Work: Physical language could improve spatiotemporal understanding in vision-language models and support more effective planning in embodied policies.The proposed direction extends physical language beyond video generation.
  • Future Work: Its transferability across appearances and embodiments could help alleviate the scarcity of real-robot interaction data.The passage identifies this transferability as a promising avenue for future work.
Loading 2607.28624v1…