PaperlayerView exact revision on arXiv

Source-linked AI summary

WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation

Jing Wang, Ao Ma, Ke Cao, Jun Zheng, Zhanjie Zhang, Jiasong Feng, Shanyuan Liu, Yuhang Ma, Bo Cheng, Dawei Leng, Yuhui Yin, Xiaodan Liang

arXiv:2503.08153v1cs.CV

TL;DR

Text-to-video models still struggle to represent abstract physical principles and generate videos aligned with real-world physical laws. WISA structures physical information for T2V generation and builds WISA-32K; on VideoPhy, it improves SA and PC over CogVideoX-5B by 0.07 and 0.05, respectively.

  • Problem

    Current text-to-video models struggle to understand abstract physical principles and produce videos that align with real-world physical laws.

    Source: §1 · p. 2

  • Method

    WISA decomposes physical principles into textual descriptions, qualitative categories, and quantitative properties, and constructs WISA-32K with 32,000 videos representing 17 physical phenomena.

    Source: §1 · p. 2

  • Results

    On VideoPhy, WISA improves SA and PC scores over CogVideoX-5B by 0.07 and 0.05, respectively.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    WISA can improve physical-law alignment in existing T2V models while adding under 5% computational overhead.

    Source: §7 · p. 9

    WISA provides high-level semantic guidance but lacks detailed constraints at the physical-mechanism level, such as energy conservation or Newton’s laws.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and Kling, have shown great potential for building world simulators. However, current T2V models struggle to grasp abstract physical principles and generate videos that adhere to physical laws. This challenge arises primarily from a lack of clear guidance on physical information due to a significant gap between abstract physical principles and generation models. To this end, we introduce the World Simulator Assistant (WISA), an effective framework for decomposing and incorporating physical principles into T2V models. Specifically, WISA decomposes physical principles into textual physical descriptions, qualitative physical categories, and quantitative physical properties. To effectively embed these physical attributes into the generation process, WISA incorporates several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and a Physical Classifier, enhancing the model's physics awareness. Furthermore, most existing datasets feature videos where physical phenomena are either weakly represented or entangled with multiple co-occurring processes, limiting their suitability as dedicated resources for learning explicit physical principles. We propose a novel video dataset, WISA-32K, collected based on qualitative physical categories. It consists of 32,000 videos, representing 17 physical laws across three domains of physics: dynamics, thermodynamics, and optics. Experimental results demonstrate that WISA can effectively enhance the compatibility of T2V models with real-world physical laws, achieving a considerable improvement on the VideoPhy benchmark. The visual exhibitions of WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.

1. Introduction

WISA structures physical guidance for text-to-video generation, while WISA-32K supplies videos centered on distinct physical phenomena. Together, they target the difficulty of translating abstract physical laws into temporally consistent visual generation.

Introduction

  • Recent T2V models generate realistic, text-consistent videos but struggle to represent abstract physical principles and preserve the temporal order of physical events [1] [20] [40] [3] [22].The gap arises because models must translate abstract descriptions of physical logic into visual phenomena for a specific scene.Source: §1 · pp. 1–2
  • WISA decomposes physical principles into textual descriptions, qualitative categories, and quantitative properties, using MoPA, AdaLN-injected physical embeddings, and a Physical Classifier to guide generation.MoPA activates expert attention heads relevant to the scene’s physics category; the classifier recognizes qualitative categories.Source: §1 · p. 2
  • General-scene datasets often entangle multiple physical phenomena or show them only as secondary elements, making physical information difficult to extract and connect to its visual manifestation [24] [35].The paper notes that physical guidance from such videos may not enable models to perceive principles such as fluid motion.Source: §1 · p. 2
  • WISA-32K contains 32,000 videos covering 17 physical phenomena across dynamics, thermodynamics, and optics, with captions decomposed into structured physical information.The collection process includes shot-boundary detection, aesthetic filtering, and video captioning; GPT-4o mini extracts the physical information.Source: §1 · pp. 2–3
  • WISA and WISA-32K improve T2V alignment with real-world physical laws while adding 3.5% to parameter count and 5% to inference time.The introduction reports quantitative and qualitative experimental improvements and characterizes the added compute as limited.Source: §1 · p. 3

2. Related Work

Prior T2V models improve realism and motion but remain physically inconsistent because general-scene data does not explicitly represent physical laws. Existing physical-video research evaluates consistency or guides video generation with constrained or costly approaches, while WISA embeds structured physical information in the generative model.

2.1. Text-to-Video Generation

  • Diffusion-Transformer T2V systems improve realism and motion, but general-scene training leaves physical laws implicit; WISA-32K instead foregrounds physical phenomena and structured annotations.Earlier T2V research extended image-generation models with temporal capabilities but often had limited realism and motion. Recent systems include OpenSora, Cosmos [1], Sora, CogVideoX [40], HunyuanVideo[x], Kling, Wan2.1, and Step-Video-T2V [20].Source: §2.1 · p. 3

2.2. Physical-aware Video Generation

  • Videophy [3] and PhyGenBench [22] evaluate physical consistency using test samples and either physics classifiers trained on manual annotations or vision-language question answering.The cited work frames these benchmarks as approaches for evaluating whether generated videos follow real-world physical laws.Source: §2.2 · p. 3
  • DANO [15], MotionCraft [2], and PhysGen [19] animate images using estimated rigid motion, but their fixed physical categories and static object-motion scenarios limit generalizability.Their differentiable estimates account for properties such as mass, inertia, friction, and rotation.Source: §2.2 · p. 3
  • PhyT2V [39] iteratively refines text using physical feedback, but incurs significant inference overhead and does not improve the generative model’s physical knowledge; WISA embeds structured physical information in the model.WISA is presented as enabling stronger physical perception across various physical phenomena.Source: §2.2 · p. 3

3. WISA-32K

WISA-32K contains 32,000 manually collected videos illustrating 17 physical phenomena across dynamics, thermodynamics, and optics. Its structured annotations pair textual physical descriptions with qualitative categories and quantitative properties.

3.1. Data Collection and Annotation

  • The dataset covers six dynamics phenomena, six thermodynamics phenomena, and five optics phenomena.These include collision, rigid body motion, elastic motion, liquid motion, gas motion, and deformation; melting, solidification, vaporization, liquefaction, explosion, and combustion; and reflection, refraction, scattering, interference and diffraction, and unnatural light sources.Source: §3.1 · p. 3
  • WISA-32K comprises 32,000 manually collected videos selected to illustrate 17 physical phenomena, excluding text-containing and low-quality videos.The dataset uses shot detection and aesthetic filtering; PySceneDetect splits videos into scene clips, and Qwen2-VL generates captions limited to 256 tokens.Source: §3.1 · pp. 3–4, Figure 3 · p. 4

3.2. Physical Information Decompose

  • Physical annotations comprise textual descriptions, 29 qualitative physics categories, and quantitative properties covering motion density and phenomenon time and temperature ranges.The qualitative scheme identifies applicable phenomena, includes three categories for absent dynamics, thermodynamics, or optics, and adds nine visual-phenomenon categories.Source: §3.2 · p. 4
  • GPT-4o mini annotates caption-based physical information in five rounds for qualitative categories and three rounds for quantitative attributes.The qualitative rounds cover dynamics, thermodynamics, optics, motion, and object state; quantitative attributes are density, time, and temperature.Source: §3.2 · p. 4
  • Caption-based annotations scored 76% versus 78% for multimodal annotations, while using approximately 2k versus 10k tokens per sample.The caption-based scheme was evaluated on 100 examples and was more convenient when only textual descriptions were available.Source: §3.2 · p. 4

4. Method

WISA injects textual, qualitative, and quantitative physical information into T2V generation through tailored conditioning methods, a Physical Module, and a Physical Classifier. The module combines category-specific attention experts with quantitative conditioning, while the classifier predicts qualitative physical categories.

4.1. Overview

  • WISA concatenates textual physical descriptions with captions and uses a Physical Module to incorporate qualitative categories and quantitative properties into T2V generation.The module uses MoPA and a qualitative Physical Classifier; quantitative properties are encoded as physical embeddings and injected into denoising features through AdaLN. A single module is inserted after the diffusion transformer blocks to reduce training and parameter burden.Source: §4.1 · p. 5

4.2. Physical Module

  • MoPA assigns attention heads to physical phenomena, using 29 qualitative categories to activate or suppress category-specific experts in the denoising features.Category indicators are randomly perturbed during training, with probability 0.2, to mitigate noisy category labels. Temperature and time values are represented in scientific notation, mapped through a linear layer, and injected with timestep embeddings by AdaLN.Source: §4.2 · pp. 5–6

4.3. Physical Classifier

  • The Physical Classifier predicts qualitative physical categories, using multi-label binary cross-entropy supervision to account for coupled phenomena in videos.It is placed after the Physical Module.Source: §4.3 · p. 6

5. Experiments

Experiments evaluate WISA on VideoPhy and PhyGenBench, with qualitative, ablation, human, and attention analyses examining its physical-law consistency and alignment.

5.1. Setup

  • Setup uses CogVideoX-5B and tests 160 PhyGenBench [22] plus 344 VideoPhy [3] prompts, binarizing PC and SA at a 0.5 threshold.Source: §5.1 · p. 6

5.2. Quantitative comparison

  • On VideoPhy, WISA achieves state-of-the-art SA and PC, improving over CogVideoX-5B by 0.07 and 0.05, respectively; PhyT2V inference takes approximately 9 times longer.PhyT2V iteratively adjusts prompts using VideoCon-Physics feedback; Cosmos performs poorly with disordered physical processes and inconsistent temporal sequences.Source: §5.2 · p. 6
  • On PhyGenBench prompts, WISA achieves state-of-the-art results, supporting its generalizability.Source: §5.2 · p. 7

5.3. Qualitative comparison

  • WISA removes pencil marks and models an apple’s fall, splash, and buoyancy; CogVideoX-5B, PhyT2V, and Cosmos fail these scenes in distinct ways.CogVideoX-5B generates no pencil marks and chaotic water and apple movements; PhyT2V darkens the marks and omits the fall; Cosmos omits erasing and generates two apples.Source: §5.3 · p. 7

5.4. Ablation Study

  • Removing the Physical Module hurts performance, while the Physical Classifier benefits both semantic consistency and physical-law consistency.Source: Figure 7 · p. 8
  • Ablations find limited improvement from LoRA or 32,000 labeled Koala-36M videos, while clearly physical WISA-32K videos benefit physical-property modeling.The passage attributes LoRA’s limited gains to distribution shift between VideoCon-Physics samples generated by nine T2V models and real-world WISA-32K videos.Source: Figure 7 · p. 8

5.5. Human Evalution

  • Human evaluation scored three models 3, 2, or 0 points by rank on semantic consistency and physical alignment; WISA led physical alignment while maintaining strong semantic consistency.Source: Figure 7 · p. 8, §5.5 · p. 8

5.6. Attention Map Analysis

  • The rigid-body-motion expert focuses on the swing region, while the non-dynamics expert attends to the static background, indicating MoPA captures corresponding physical attributes.Source: Figure 8 · p. 8, §5.6 · p. 8

6. Limitation

WISA remains limited by incomplete physical-category coverage, high-level rather than mechanism-level guidance, and failures to match physical principles in every scenario.

Limited physical categories

  • WISA-32K covers 17 physical phenomena but omits some real-world phenomena, such as corrosion and vacuum environments, because of time and manpower constraints.The dataset contains 32,000 videos.Source: §6 · p. 8

Limited physical information guidance

  • WISA provides high-level semantic guidance but lacks detailed mechanism-level constraints, and adding such constraints while maintaining generalization remains open research.The paper notes that motion modeling based on image or 3D information has poor generalization and handles only limited categories and scenarios.Source: §6 · p. 8

Failure case

  • Because of limited data and parameters, WISA cannot generate videos that perfectly follow physical principles in all scenarios.Source: §6 · p. 8

7. Conclusion

WISA structures physical information for T2V generation and combines specialized model components with WISA-32K to improve alignment with real-world physical laws at under 5% additional computational overhead.

7. Conclusion

  • WISA decomposes physical principles into textual descriptions, qualitative categories, and quantitative properties.These forms of structured physical information are intended to help T2V models learn physical aspects effectively.Source: §7 · p. 8
  • WISA and WISA-32K help produce videos that better align with real-world physical laws, with additional computational overhead under 5%.The paper presents these results as evidence that the framework and dataset support physics-aware T2V generation.Source: §7 · p. 9
  • WISA uses Mixture-of-Physical-Experts Attention and a Physical Classifier to help T2V models learn physical aspects.Source: §7 · p. 9
  • WISA-32K contains 32,000 video clips covering 17 physical phenomena across three fundamental categories of physics.The dataset is presented as a high-quality data foundation.Source: §7 · p. 9

A. Training Detail · B. The Definition of Physical Categories

WISA is trained on WISA-32K with CogVideoX-5B as its base model, and its physical-category taxonomy organizes 29 categories into five classes. The taxonomy covers physical phenomena, camera motion, and changes in object state.

A. Training Detail

  • WISA uses CogVideoX-5B, training for 8,000 steps on WISA-32K with 187M learnable parameters updated across the physical module, classifier, and LoRA.Training uses a 2e-5 learning rate, batch size 8, 480x720 resolution with 49 frames, LoRA rank 128 and alpha 16, and 8 A100 GPUs with 80GB each.Source: A. Training Detail · p. 11

B. The Definition of Physical Categories

  • The physical taxonomy defines 29 qualitative categories organized into five major classes.The classes are dynamics, thermodynamics, optics, camera motion, and object state.Source: B. The Definition of Physical Categories · p. 11
  • Dynamics includes collision, rigid-body motion, elastic motion, liquid motion, gas motion, deformation, and no obvious dynamic phenomenon.Source: B. The Definition of Physical Categories · p. 11
  • Thermodynamics includes melting, solidification, vaporization, liquefaction, explosion, combustion, and no obvious thermodynamic phenomenon.Source: B. The Definition of Physical Categories · p. 11
  • Optics categories cover reflection, refraction, scattering, interference and diffraction, unnatural light sources, and no obvious optical phenomenon; camera motion is labeled yes or no.Source: B. The Definition of Physical Categories · p. 11
  • Object-state categories cover liquid, solid, and gas appearance, decomposition and splitting, mixing, disappearance, and no change.Appearance categories describe new objects entering the camera over time due to external forces; decomposition, mixing, and disappearance likewise occur over time under external forces, while no change means the object's state remains unchanged.Source: B. The Definition of Physical Categories · p. 11

C. Dataset Property Analysis · D. More Examples and Annotation · E. Annotation Prompts

The sections analyze WISA-32K’s category distribution and annotation accuracy, then show example annotations and the prompts used to produce them.

C. Dataset Property Analysis

  • Category annotation accuracy is 84% for dynamics, 71% for optics, and 64% for thermodynamics, with 75% overall accuracy.The accuracy was evaluated using labels from manually collected videos.Source: C. Dataset Property Analysis · p. 11
  • Dynamics accounts for 47% of WISA-32K, compared with 29% for optics and 24% for thermodynamics.The passage attributes dynamics’ largest share to its frequent occurrence in daily life; optics and thermodynamics typically require specific conditions.Source: C. Dataset Property Analysis · p. 11
  • The dataset analysis visualizes the distribution of video frame counts, but the passage gives no frame-count values.Source: C. Dataset Property Analysis · p. 11

D. More Examples and Annotation

E. Annotation Prompts

F. Word Cloud

A noun-frequency analysis of WISA-32K’s textual physical descriptions found that physical terms are prominent in the dataset.

F. Word Cloud

  • The most frequent retained nouns include “motion,” “phenomenon,” and “light,” highlighting the dataset’s physical relevance.The analysis filtered out irrelevant words by retaining nouns and selecting them by frequency, from highest to lowest.Source: F. Word Cloud · p. 11

G. Discussion of Quantitative Evaluation

The discussion finds that current video-based physics evaluators can misjudge physically plausible event sequences, motivating more reliable assessment of videos’ physical properties.

G. Discussion of Quantitative Evaluation

Related papers

Loading 2503.08153v1…