PaperlayerView exact revision on arXiv

Source-linked AI summary

WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation

Jing Wang, Ao Ma, Ke Cao, Jun Zheng, Zhanjie Zhang, Jiasong Feng, Shanyuan Liu, Yuhang Ma, Bo Cheng, Dawei Leng, Yuhui Yin, Xiaodan Liang

arXiv:2503.08153v1cs.CV

TL;DR

T2V models struggle to translate abstract physical principles into videos that obey real-world laws, while existing datasets often weakly or ambiguously represent phenomena. WISA decomposes physical information, uses specialized conditioning and WISA-32K, and improves VideoPhy performance with limited computational overhead. The framework remains limited in physical-category coverage and detailed mechanism-level constraints.

  • Problem

    T2V models struggle to understand abstract physical principles and generate videos aligned with physical laws, while existing videos often entangle or weakly represent individual phenomena.

  • Method

    WISA decomposes physical principles into textual descriptions, qualitative categories, and quantitative properties, then injects them through MoPA and a Physical Classifier alongside WISA-32K.

  • Results

    WISA improves VideoPhy SA and PC scores over CogVideoX-5B by 0.07 and 0.05, respectively, while adding 3.5% parameters and 5% inference time.

  • Takeaways & Limitations

    WISA and WISA-32K help existing T2V models produce videos that better align with real-world physical laws with computational overhead under 5%.

  • Takeaways & Limitations

    WISA-32K covers 17 phenomena and primarily provides high-level semantic guidance rather than detailed mechanism-level constraints such as energy conservation or Newton’s laws.

Abstract

from arXiv · show

Recent rapid advancements in text-to-video (T2V) generation, such as SoRA and Kling, have shown great potential for building world simulators. However, current T2V models struggle to grasp abstract physical principles and generate videos that adhere to physical laws. This challenge arises primarily from a lack of clear guidance on physical information due to a significant gap between abstract physical principles and generation models. To this end, we introduce the World Simulator Assistant (WISA), an effective framework for decomposing and incorporating physical principles into T2V models. Specifically, WISA decomposes physical principles into textual physical descriptions, qualitative physical categories, and quantitative physical properties. To effectively embed these physical attributes into the generation process, WISA incorporates several key designs, including Mixture-of-Physical-Experts Attention (MoPA) and a Physical Classifier, enhancing the model's physics awareness. Furthermore, most existing datasets feature videos where physical phenomena are either weakly represented or entangled with multiple co-occurring processes, limiting their suitability as dedicated resources for learning explicit physical principles. We propose a novel video dataset, WISA-32K, collected based on qualitative physical categories. It consists of 32,000 videos, representing 17 physical laws across three domains of physics: dynamics, thermodynamics, and optics. Experimental results demonstrate that WISA can effectively enhance the compatibility of T2V models with real-world physical laws, achieving a considerable improvement on the VideoPhy benchmark. The visual exhibitions of WISA and WISA-32K are available in the https://360cvgroup.github.io/WISA/.

1. Introduction

WISA addresses the difficulty of generating physically consistent videos by decomposing abstract physical principles into structured information and injecting it into T2V models. WISA-32K supplies videos that clearly represent physical phenomena, and experiments report improved alignment with real-world physical laws with limited overhead.

  • Current T2V models generate realistic, text-consistent videos but struggle to understand abstract physical principles and follow real-world physical laws.
  • The gap between abstract physical laws and their visual manifestations makes it difficult to translate textual physical guidance into temporally consistent video phenomena.
  • WISA decomposes physical principles into textual descriptions, qualitative categories, and quantitative properties, using tailored conditioning methods for T2V generation.
  • WISA combines structured physical information with Mixture-of-Physical-Experts Attention and a Physical Classifier to guide physics-aware video generation.
  • WISA-32K contains 32,000 manually collected clips clearly showing 17 physical phenomena across dynamics, thermodynamics, and optics.
  • WISA improves physical-law alignment while adding 3.5% parameters and 5% inference time.

2. Related Work

Related work advances video generation and evaluates or enforces physical consistency through benchmarks, physical reasoning, and differentiable motion modeling. Existing approaches remain limited by fixed physical categories or static object-motion scenarios.

  • Recent T2V research uses Diffusion Transformers to improve realism, motion dynamics, and scalable spatiotemporal modeling.
  • VideoPhy and PhyGenBench evaluate whether generated videos follow physical laws using annotated physics classifiers or vision-language question answering.
  • MotionCraft and PhysGen estimate object motion using properties such as mass, inertia, friction, and rotation before animating images into videos.
  • Differentiable motion methods are restricted to fixed physical categories and static object-motion scenarios, limiting their generalizability.

3. WISA-32K

WISA-32K is organized around 17 physical phenomena across dynamics, thermodynamics, and optics, with videos selected to clearly exhibit these phenomena. Its pipeline filters and captions videos, then adds structured physical annotations that WISA uses to guide T2V generation.

  • Physical laws and collection: WISA-32K defines 17 phenomena across dynamics, thermodynamics, and optics as the basis for dataset collection.
  • Physical laws and collection: The dataset manually collects 32,000 samples, excludes videos with text or low-quality content, and omits some infrequent phenomena such as sublimation and condensation.
  • Pre-processing and annotation: The processing pipeline applies shot detection and aesthetic filtering, extracts captions with Qwen2-VL, and generates physical annotations with GPT-4o mini.
  • Physical information decomposition: Structured annotations decompose physical information into textual descriptions, qualitative categories, and quantitative properties.
  • Physical information decomposition: The annotation scheme includes 29 qualitative categories and quantitative labels for motion density, phenomenon duration, and temperature range.
  • Annotation evaluation: Caption-based annotation achieves 76% versus 78% for multimodal annotation while reducing token cost from approximately 10k to 2k per sample.
  • WISA integration: WISA uses structured annotations through a Physical Module and Physical Classifier, including phenomenon-specific attention experts activated by qualitative categories.

4. Method

WISA incorporates structured physical information into existing T2V models through specialized condition-injection mechanisms. Its Physical Module combines physical experts with a classifier to guide modeling of qualitative and quantitative physical properties.

  • Condition injection: WISA injects textual descriptions, qualitative categories, and quantitative properties into existing T2V models using distinct condition-injection methods.The framework is designed for CogVideoX and aims to preserve original capabilities while learning physical knowledge from limited video data.
  • Physical Module: Qualitative physical categories are encoded as a 29-dimensional binary vector, with each position indicating whether a defined phenomenon is activated.The categories may contain incorrect activations or suppressions, so training applies random perturbations to the category indicators.
  • Physical Module: MoPA assigns multi-head attention heads to physical experts that activate or suppress phenomena-specific features.The denoising feature interacts with perturbed category indicators, then returns to the original feature dimension through concatenation and a linear layer.
  • Quantitative properties: Quantitative temperature and time values are represented in scientific notation, mapped through a linear layer, concatenated with timestep embeddings, and injected using AdaLN.
  • Physical Classifier: A Physical Classifier predicts qualitative physical categories after the Physical Module, using multi-label BCE supervision because phenomena may be coupled.Its predicted probabilities are obtained by applying the classifier and sigmoid function to denoising features, while the classification loss is balanced with diffusion loss.

5. Experiments

Experiments evaluate WISA with CogVideoX-5B on physics-oriented prompts using semantic consistency and physical consistency measures. WISA achieves state-of-the-art quantitative results and produces more physically aligned qualitative examples, while MoPA focuses on phenomenon-relevant regions.

  • Setup: WISA is evaluated with CogVideoX-5B using VideoCon-Physics on 160 PhyGenBench prompts and 344 VideoPhy prompts.Semantic consistency and physical consistency values are binarized at a threshold of 0.5.
  • Quantitative comparison: WISA achieves state-of-the-art results on PhyGenBench, supporting generalizability across physics-oriented prompt sets.
  • Qualitative comparison: In qualitative comparisons, WISA correctly removes pencil marks and models an apple’s fall, splash, and post-submersion buoyancy, unlike the compared methods.The alternatives omit, distort, or mishandle parts of these processes, including darker pencil marks, omitted falling motion, chaotic movements, and duplicate apples.
  • Human evaluation: Human evaluation ranks models on semantic consistency and physical alignment, with WISA showing a significant advantage in physical alignment.Each aspect assigns 3 points for first place, 2 for second, and 0 for last among three models.
  • Expert analysis: MoPA attention maps show the rigid-body-motion expert focusing on a swing region while the non-dynamics expert attends to a static background.The visualization is presented as evidence that different experts capture distinct physical attributes.

6. Limitation

WISA improves physical-law alignment but remains limited in category coverage, physical-mechanism guidance, and performance across all scenarios.

  • WISA-32K covers 17 physical phenomena but excludes real-world cases such as corrosion and vacuum environments.The authors attribute this limitation to constraints in time and manpower.
  • WISA primarily provides high-level semantic guidance rather than detailed constraints such as energy conservation or Newton’s laws.More detailed constraints would require image- or 3D-based object-motion modeling, which has poor generalization and limited category and scenario coverage.
  • Limited data and parameters prevent WISA from generating videos that perfectly align with physical principles in all scenarios.

7. Conclusion

WISA structures physical principles into multiple information types and adds dedicated components to help T2V models learn physical aspects. WISA-32K supplies 32,000 clips covering 17 phenomena, while experiments report better physical-law alignment with under 5% computational overhead.

  • WISA decomposes physical principles into textual descriptions, qualitative categories, and quantitative physical properties.
  • WISA uses Mixture-of-Physical-Experts Attention and a Physical Classifier to help T2V models learn physical aspects.
  • WISA-32K contains 32,000 video clips covering 17 physical phenomena across three fundamental physics categories.The dataset is presented as a high-quality data foundation for physical learning.
  • WISA and WISA-32K help produce videos that better align with real-world physical laws, with additional computational overhead under 5%.

A. Training Detail

The training setup uses CogVideoX-5B with a physical-category scheme spanning five classes and 29 categories, including dynamics, thermodynamics, optics, camera motion, and object state.

  • Training Detail: CogVideoX-5B is used as the base T2V model, trained for 8,000 steps at a 2e-5 learning rate and batch size 8.Training uses 480x720 videos with 49 frames.
  • The Definition of Physical Categories: The physical-category scheme defines 29 qualitative categories organized into five major classes.
  • The Definition of Physical Categories: The dynamics categories include collision, rigid body motion, elastic motion, liquid motion, gas motion, deformation, and no obvious dynamic phenomenon.
  • The Definition of Physical Categories: The thermodynamics categories include melting, solidification, vaporization, liquefaction, explosion, combustion, and no obvious thermodynamic phenomenon.
  • The Definition of Physical Categories: The optics categories include reflection, refraction, scattering, interference and diffraction, unnatural light sources, and no obvious optical phenomenon.
  • The Definition of Physical Categories: Additional categories encode camera motion and object-state changes such as appearance, decomposition, splitting, mixing, disappearance, and no change.Object-state categories distinguish liquids, solids, and gases.

C. Dataset Property Analysis

WISA-32K is constructed through physical-information annotation and contains videos distributed across physics categories, with reported annotation accuracy varying by domain.

  • Dynamics accounts for 47% of WISA-32K, while optics and thermodynamics account for 29% and 24%, respectively.The authors associate the distribution with the frequency and environmental requirements of these phenomena.
  • Qualitative physical-category annotation accuracy is 84% for dynamics, 71% for optics, 64% for thermodynamics, and 75% overall.
  • The dataset annotation pipeline is designed to provide accurate and detailed labels for each video’s physical properties and phenomena.Example videos and corresponding annotations are shown in Figure 10.
  • Figures 11, 12, and 13 illustrate the detailed prompts used for physical-information annotation.

F. Word Cloud

The dataset’s textual physical descriptions emphasize physically relevant concepts, with terms such as motion, phenomenon, and light appearing frequently. Example annotations connect scene captions with physical principles and phenomena.

  • Physical terms such as “motion,” “phenomenon,” and “light” occur frequently in WISA-32K’s textual physical descriptions.The word cloud retains frequent nouns to highlight the dataset’s physical relevance.
  • A mechanical-device example links a caption to force and pressure during compression.The annotation describes a descending component exerting pressure on a lower component.
  • An explosion example invokes conservation of momentum and energy together with gas and smoke behavior.The scene includes a flash, smoke, debris, and camera vibration following the explosion.
  • Dynamics categories include collision, gas motion, and deformation.

G. Discussion of Quantitative Evaluation

The quantitative evaluation discussion identifies misjudgments by VideoCon-Physics and Qwen2.5-VL when assessing the ordering of physical events. These findings motivate more reliable video-based assessments of physical properties.

  • VideoCon-Physics assigns a low score of 0.08 to a physically plausible sequence in which an object enters water before the splash.The authors report that the generated sequence aligns with real-world physical laws despite the low score.
  • Qwen2.5-VL also struggles to distinguish correct from incorrect sequences of physical events.
  • Existing video-based physics evaluation metrics have limitations in assessing physical event sequences.The discussion calls for future research into more reliable physical-property assessments for videos.
Loading 2503.08153v1…