Source-linked AI summary
DreamWorld: Unified World Modeling in Video Generation
Boming Tan, Xiangdong Zhang, Ning Liao, Yuqing Zhang, Shaofeng Zhang, Xue Yang, Qi Fan, Yanyong Zhang
TL;DR
Video generators often lack structured, unified world understanding, while multi-source knowledge alignment can create conflicting optimization objectives. DreamWorld jointly models pixels and heterogeneous world features, using annealed training constraints and inference guidance; evaluations report improved performance over baselines and VideoJAM.
Problem
Existing video generators provide limited structured world knowledge, and aligning multiple heterogeneous experts can cause conflicting optimization objectives.
Method
DreamWorld jointly predicts video pixels and features encoding temporal dynamics, spatial geometry, and semantic understanding, with CCA during training and Multi-Source Inner-Guidance at inference.
Results
DreamWorld significantly outperforms baselines and VideoJAM in extensive evaluations.
Takeaways & Limitations
DreamWorld integrates complementary world knowledge while targeting coherent, artifact-free video generation.
Takeaways & Limitations
The approach is constrained by computational resources and the diversity of its training datasets.
Abstract
from arXiv · showhide
Despite impressive progress in video generation, existing models remain limited to surface-level plausibility, lacking a coherent and unified understanding of the world. Prior approaches typically incorporate only a single form of world-related knowledge or rely on rigid alignment strategies to introduce additional knowledge. However, aligning the single world knowledge is insufficient to constitute a world model that requires jointly modeling multiple heterogeneous dimensions (e.g., physical commonsense, 3D and temporal consistency). To address this limitation, we introduce \textbf{DreamWorld}, a unified framework that integrates complementary world knowledge into video generators via a \textbf{Joint World Modeling Paradigm}, jointly predicting video pixels and features from foundation models to capture temporal dynamics, spatial geometry, and semantic consistency. However, naively optimizing these heterogeneous objectives can lead to visual instability and temporal flickering. To mitigate this issue, we propose \textit{Consistent Constraint Annealing (CCA)} to progressively regulate world-level constraints during training, and \textit{Multi-Source Inner-Guidance} to enforce learned world priors at inference. Extensive evaluations show that DreamWorld improves world consistency, outperforming Wan2.1 by 2.26 points on VBench. Code will be made publicly available at \href{https://github.com/ABU121111/DreamWorld}{\textcolor{mypink}{\textbf{Github}}}.
1. Introduction
DreamWorld addresses the gap between visually plausible video generation and unified world understanding by jointly modeling heterogeneous world knowledge. It combines multi-source feature prediction with training and inference mechanisms intended to preserve visual stability and consistency.
- Existing video generators often match pixel distributions without internalizing structured, comprehensive world knowledge.
- Naively aligning semantic, spatial, and dynamic experts creates conflicting gradients and optimization instability.These conflicts show that relational alignment alone is insufficient for multi-source unified world modeling.
- DreamWorld jointly predicts video pixels and world features to integrate temporal dynamics, spatial geometry, and semantic understanding.Its feature sources include Optical Flow, VGGT, and DINOv2.
- Consistent Constraint Annealing progressively regulates heterogeneous objectives to harmonize knowledge injection with coherent, artifact-free generation.
- Multi-Source Inner-Guidance uses predicted knowledge features during inference to steer generation toward trajectories adhering to real-world laws.
- DreamWorld evaluations significantly outperform baselines and VideoJAM, establishing a new standard for world models.
2. Related Work
Related work spans scalable video diffusion, representation alignment, and interactive or predictive world models. DreamWorld targets the unresolved challenge of synergizing multiple heterogeneous expert models.
- Video Diffusion Models: Diffusion Transformers and Flow Matching have supported photorealistic video generation and improved training stability or inference efficiency.
- Representation Alignment: Representation Alignment injects high-level semantic priors from pretrained foundation models into pixel-space diffusion models.
- World Modeling: World models aim to understand environmental laws for prediction and simulation, including interactive video rollouts with latent action spaces.
- World Modeling: Existing paradigms lack the capability to synergize knowledge from multiple heterogeneous expert models.DreamWorld fuses complementary priors to target long-horizon consistency and realism.
3. Method
DreamWorld jointly models video appearance with temporal, semantic, and spatial world features, using aligned expert priors, annealed training constraints, and multi-source inference guidance.
- World Knowledge Priors: DreamWorld constructs a composite world-feature space combining optical-flow dynamics, DINOv2 semantics, and VGGT spatial geometry.These priors represent temporal trajectories, prompt-consistent objects, and 2D geometric relationships.
- Unified Preprocessing and Alignment: Heterogeneous priors are converted, resampled, standardized, and compressed so they can condition the diffusion backbone.Optical flow is mapped to RGB and encoded by a pretrained 3D causal VAE; expert outputs undergo spatial and temporal alignment, standardization, and PCA compression.
- Joint Feature Integration: The model concatenates video latents with world features and expands its projections to jointly predict appearance and world knowledge.World-feature input weights are initialized to zero so the initial behavior matches pretrained Wan2.1, while the output predicts a joint velocity field.
- Training Objective: The training objective is a weighted sum of video, temporal, semantic, and spatial flow-matching losses.Each modality-specific term measures error between a predicted velocity component and its target conditional flow.
- Consistent Constraint Annealing: Consistent Constraint Annealing gradually relaxes world-knowledge weights toward zero to prioritize high-fidelity, artifact-free reconstruction late in training.The method addresses tension between visual fidelity and learning from heterogeneous feature constraints.
- Multi-Source Inner-Guidance: Multi-source inner-guidance combines fully conditioned and feature-specific predictions to regulate text, temporal, semantic, and spatial conditions during inference.The guidance formulation masks each corresponding world-knowledge feature channel independently.
4. Experiments
DreamWorld is evaluated across video quality, physical commonsense, world simulation, qualitative consistency, and ablation settings. It consistently improves benchmark performance and world consistency, while CCA and multi-source guidance address instability and demonstrate the value of comprehensive world knowledge.
- Benchmark Protocol: DreamWorld evaluates generated videos using VBench, VBench 2.0, VideoPhy, and WorldScore.These benchmarks cover video quality, semantic consistency, physical commonsense, and static and dynamic world-generation capabilities.
- Quantitative Results: 80.97 on VBench, surpassing the fine-tuned Wan2.1 baseline and VideoJAM.The evaluation reports consistent improvements across the benchmark’s video-quality and semantic-consistency dimensions.
- Quantitative Results: 52.97 on VBench 2.0, leading both large-scale Wan2.1 baselines and VideoJAM.DreamWorld maintains competitive dynamics while achieving strong high-fidelity synthesis, balancing generative freedom and controllability.
- Quantitative Results: DreamWorld achieves 52.9% Semantic Adherence and 26.2% Physical Commonsense on VideoPhy.It outperforms the standard Baseline and motion-prior-based VideoJAM across aggregated metrics.
- Quantitative Results: 51.48 Total Score on WorldScore, outperforming fine-tuned Wan2.1 and VideoJAM.The result reflects a reported balance between static fidelity and dynamic evolution in world simulation.
- Qualitative Results: Qualitative comparisons show stronger semantic realism, 3D occlusion, smooth motion, and facial-identity stability than competing models.DreamWorld follows actions such as tilting a cup, avoids physically impossible penetration, and reduces temporal deformations.
- Ablation Studies: The full spatial-semantic-temporal WorldKnowledge stack performs best, while removing guidance components degrades performance.Temporal priors most affect Quality and Overall scores, whereas Text Guidance and Semantic Priors strongly affect Semantic scores.
- Ablation Studies: CCA reduces artifacts and flickering caused by static objective weighting, while λ = 0.2 provides the best quality-semantic trade-off.Static weighting produces abnormal highlighting, exposure anomalies, and high-frequency flickering; both VBench metrics peak at λ = 0.2.
5. Conclusion
DreamWorld is presented as a unified framework for integrating multi-source expert knowledge into video generation while preserving generative freedom. The authors report state-of-the-art benchmark performance, but identify computational resources and training-data diversity as current constraints.
- Conclusion: DreamWorld integrates multi-source expert knowledge through a softer Joint World Modeling Paradigm.The framework combines structural logic with generative freedom rather than directly enforcing simultaneous alignment objectives.
- Conclusion: CCA and Multi-Source Inner-Guidance harmonize world-knowledge constraints during training and inference.CCA regulates training constraints, while Multi-Source Inner-Guidance steers generation using learned world priors.
- Limitations: The approach is constrained by computational resources and the diversity of the training datasets.The authors suggest improving multi-source integration efficiency and incorporating more diverse data curation.
Impact Statement
The paper frames its impact as methodological and theoretical work on world models for video generation. It states that the work introduces no new specific ethical risks beyond those inherent in existing video synthesis techniques.
- Impact Statement: The paper’s contributions remain within methodology and fundamental theory for world models in video generation.The authors state that the work does not introduce new specific ethical risks or loopholes beyond existing video synthesis techniques.
A. Motivation Experiments
The motivation experiments test whether a video generator can assimilate multiple heterogeneous knowledge sources through independent expert projections and a unified soft-alignment objective.
- A. Motivation Experiments: Extend-VideoREPA maps VAE latents into DINOv2, VGGT, and VideoMAEv2 embedding spaces using three independent projection layers.The projection dimensions are 768, 2048, and 768, respectively.
- A. Motivation Experiments: The total objective combines standard diffusion loss with independent TRD terms from each expert branch.This formulation trains the backbone to reconstruct relational topology across the diverse feature spaces.
B. Implementation Details
The implementation uses offline foundation-model features, a Wan2.1-initialized training setup, and guided inference with fixed denoising and auxiliary guidance scales.
- B. Implementation Details: Training initializes the backbone from Wan2.1 and uses AdamW with β1 = 0.9, β2 = 0.99, and weight decay λ = 0.2.The learning rate is 1e-5, with a 400-step linear warmup followed by consistency-constrained annealing.
- B. Implementation Details: WorldKnowledge priors are extracted offline, including dense optical flow computed with RAFT and encoded through the Wan2.1 VAE.The passage describes converting displacement fields into RGB visualizations before encoding.
- B. Implementation Details: Inference uses Flow Matching Euler Discrete Scheduling with 50 denoising steps and text guidance scale wtext = 5.Motion, semantic, and geometry guidance scales are each set to 1.
B.1. Results of WorldScore
This section presents a quantitative WorldScore comparison, with detailed multidimensional metrics reported in Table B.1.
- The WorldScore evaluation includes a detailed breakdown across multiple metrics.
- Table B.1 reports a quantitative comparison on WorldScore.The table highlights the best and second-best results.
- Best and second-best WorldScore results are visually distinguished in the table.
B.2. VBench and VBench2.0 Results
DreamWorld is evaluated on VBench and VBench-2.0 using detailed consistency, motion, object-level, appearance, style, and overall metrics. It achieves an Overall Score of 80.97 on VBench and a Total Score of 52.97 on VBench-2.0, with strong performance in selected complex categories.
- VBench Results: VBench reports consistency, motion, object-level, appearance, style, and overall metrics for the evaluated models.The detailed results are distributed across Tables B.2 and B.3.
- VBench Results: 80.97 is DreamWorld’s leading Overall Score on VBench, alongside significant margins in Spatial Relationship and Dynamic Degree.
- VBench-2.0 Results: 52.97 is DreamWorld’s highest Total Score on VBench-2.0, with strong results in Human Fidelity and Motion Order Understanding.VBench-2.0 results are detailed across Tables B.4, B.5, and B.6.
C. More qualitative results
Additional qualitative comparisons contrast Wan2.1 with DreamWorld across semantic, spatial, and temporal dimensions. DreamWorld is reported to preserve structural integrity and dynamic coherence in complex scenarios where the baseline exhibits anomalies and conflicts.
- In complex interactions such as eating ice cream or cutting a bell pepper, Wan2.1 frequently produces semantic hallucinations and spatial conflicts.The reported conflicts include objects unnaturally fusing or distorting.
- DreamWorld maintains robustness across semantic, spatial, and temporal dimensions in the qualitative comparisons.The figure-level comparison describes superior structural integrity and dynamic coherence across diverse scenarios.
- Figure C.1 compares Wan2.1 in the top rows with DreamWorld in the bottom rows.Red rectangles mark anomalies such as object penetration, unnatural disappearance, and limb distortion.