PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling

Yang Zhou, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Haoyu Guo, Zizun Li, Kaijing Ma, Xinyue Li, Yating Wang, Haoyi Zhu, Mingyu Liu, Dingning Liu, Jiange Yang, Zhoujie Fu, Junyi Chen, Chunhua Shen, Jiangmiao Pang, Kaipeng Zhang, Tong He

arXiv:2509.12201v2cs.CV

TL;DR

4D world modeling lacks datasets that jointly provide dynamic complexity, domain diversity, and rich spatial-temporal annotations. OmniWorld addresses this gap with a large multi-domain, multimodal collection and benchmark; fine-tuning existing methods on it yields significant gains across reconstruction and video generation tasks.

  • Problem

    Existing 4D modeling datasets and benchmarks often lack dynamic complexity, multi-domain diversity, and spatial-temporal annotations needed for geometric reconstruction and camera-controlled video generation.

  • Method

    OmniWorld combines a self-collected OmniWorld-Game dataset with public datasets and supplies multimodal annotations for benchmarking and training 4D world models.

  • Results

    Fine-tuning existing state-of-the-art methods on OmniWorld significantly improves performance across 4D reconstruction and camera-controlled video generation tasks.

  • Takeaways & Limitations

    OmniWorld provides a challenging evaluation benchmark and a training resource for advancing general-purpose 4D world models.

  • Takeaways & Limitations

    Current methods still struggle with dynamic content involving complex camera control because common training datasets contain static scenes and smooth camera motions.

Abstract

from arXiv · show

The field of 4D world modeling - aiming to jointly capture spatial geometry and temporal dynamics - has witnessed remarkable progress in recent years, driven by advances in large-scale generative models and multimodal learning. However, the development of truly general 4D world models remains fundamentally constrained by the availability of high-quality data. Existing datasets and benchmarks often lack the dynamic complexity, multi-domain diversity, and spatial-temporal annotations required to support key tasks such as 4D geometric reconstruction, future prediction, and camera-control video generation. To address this gap, we introduce OmniWorld, a large-scale, multi-domain, multi-modal dataset specifically designed for 4D world modeling. OmniWorld consists of a newly collected OmniWorld-Game dataset and several curated public datasets spanning diverse domains. Compared with existing synthetic datasets, OmniWorld-Game provides richer modality coverage, larger scale, and more realistic dynamic interactions. Based on this dataset, we establish a challenging benchmark that exposes the limitations of current state-of-the-art (SOTA) approaches in modeling complex 4D environments. Moreover, fine-tuning existing SOTA methods on OmniWorld leads to significant performance gains across 4D reconstruction and video generation tasks, strongly validating OmniWorld as a powerful resource for training and evaluation. We envision OmniWorld as a catalyst for accelerating the development of general-purpose 4D world models, ultimately advancing machines' holistic understanding of the physical world.

1. Introduction

OmniWorld addresses the shortage of diverse, richly annotated data needed for 4D world modeling. It combines a large synthetic dataset with public data and establishes benchmarks for geometric modeling and camera-controlled video generation.

  • Motivation: Existing datasets inadequately evaluate complex 4D modeling because they often contain short sequences, limited motion, narrow scene types, or smooth camera trajectories.These limitations affect both 3D geometric foundation models and camera-controlled video generation models.
  • Motivation: High-quality training data with multiple geometric modalities remains scarce across domains.Image-text and video-text datasets often lack depth maps, camera poses, and optical flow.
  • OmniWorld: OmniWorld combines a self-collected OmniWorld-Game dataset with public datasets to provide large-scale, multi-domain, multimodal 4D data.OmniWorld-Game contains over 96K clips, more than 18M frames, over 214 hours, 720P RGB, depth, camera poses, captions, optical flow, and foreground masks.
  • Benchmark: The paper establishes a benchmark for 3D geometric foundation models and camera-controlled video generation using challenging, dynamic OmniWorld-Game scenarios.The benchmark is designed to expose limitations of current state-of-the-art methods.
  • Validation: Fine-tuning existing state-of-the-art models on OmniWorld produces significant performance gains on public benchmarks.The reported gains support OmniWorld’s value as a training resource.

2. OmniWorld Dataset

OmniWorld integrates simulator, robot, human, and internet data through a filtering and annotation pipeline. The resulting collection provides broad scene diversity, multiple geometric and semantic modalities, and detailed captions for 4D modeling.

  • Data Acquisition: OmniWorld combines OmniWorld-Game with robot, human, and internet datasets to capture temporally consistent and dynamically diverse scenes.The simulator domain provides controlled, high-precision data, while the other domains contribute physical interactions, human activities, and in-the-wild street scenes.
  • Data Acquisition: Video slicing removes unsuitable frames and segments long videos into shorter, high-quality clips before annotation.The filter targets motion blur, insufficient feature points, and excessively large dynamic areas while preserving temporal coherence.
  • Data Annotation: The annotation pipeline provides depth maps, camera poses, text captions, optical flow, and foreground masks for comprehensive spatio-temporal modeling.Different source-specific pipelines support these modalities, including rendered depth, tracked masks, and automated camera-pose estimation.
  • Data Statistics: OmniWorld contains over 600 thousand video sequences and more than 300 million frames across 12 datasets from four domains.More than half of the data has a resolution of 720P or higher, and the collection includes multiple annotated modalities.
  • Data Statistics: OmniWorld-Game spans diverse scene types, camera perspectives, and historical eras, with outdoor-urban scenes and first-person views predominant.The reported composition includes outdoor, indoor, mixed, first-person, third-person-following, ancient, modern, and futuristic styles.

3. OmniWorld-Game Benchmark

OmniWorld-Game provides a challenging benchmark for 3D geometric prediction and camera-controlled video generation, combining long sequences, diverse dynamics, and complex scenes. Results show that current methods remain limited in maintaining accurate, temporally consistent understanding and control.

  • Benchmark design and motivation: OmniWorld-Game evaluates 3D geometric prediction and camera-controlled video generation using long, diverse, high-resolution sequences.Sequences extend up to 16 seconds and 384 frames at 720P, with varied motion and environmental conditions.
  • 3D geometric prediction: Current geometric foundation models struggle with high-dynamic, long-sequence understanding, and no single model leads across all evaluated metrics.VGGT performs best for video depth estimation, while MoGe-2 leads monocular depth estimation; VGGT is more coherent visually but still shows artifacts.
  • 3D geometric prediction: MoGe-2 achieves superior monocular depth accuracy and visually sharp depth maps on OmniWorld-Game.The comparison is qualitative in Figure 4; the supplied passages do not provide numerical values.
  • 3D geometric prediction: VGGT generates more coherent 3D structures than competing methods in dynamic scenes, but still exhibits artifacts in complex details.This result comes from qualitative point-cloud comparisons in Figure 5.
  • Benchmark implications: The benchmark exposes insufficient robustness of current geometric methods and motivates stronger spatio-temporal consistency.The benchmark includes mainstream geometric and video-generation baselines across its two core tasks.
  • Camera-controlled video generation: Camera-controlled video generation remains difficult because models must jointly preserve video quality and accurately follow camera trajectories in dynamic scenes.AC3D shows basic control but high FVD in text-to-video, while CamCtrl performs best among image-to-video models; all evaluated methods retain substantial room for improvement.

4. Model Fine-tuning and Efficacy Validation

Fine-tuning existing geometric and camera-controlled video-generation models on OmniWorld consistently improves their performance over original versions. These gains occur across depth, camera-pose, and camera-control evaluations, supporting OmniWorld as a useful training resource.

  • Overall validation: Models fine-tuned with OmniWorld consistently outperform their original versions across geometric prediction and camera-controlled video-generation tasks.The reported improvements span monocular depth, video depth, camera pose, and camera-controlled video generation.
  • Improving 3D geometric prediction with OmniWorld: Fine-tuned DUSt3R and CUT3R improve monocular depth estimation, with DUSt3R surpassing MonST3R despite the latter using multiple dynamic datasets.The comparison is reported on the benchmarks summarized in Table 5.
  • Improving 3D geometric prediction with OmniWorld: Fine-tuning DUSt3R and CUT3R on OmniWorld improves video depth estimation, demonstrating gains in temporal consistency.The comparison is reported on the datasets summarized in Table 6.
  • Improving 3D geometric prediction with OmniWorld: CUT3R’s camera-pose estimation improves after fine-tuning on OmniWorld.The supplementary evaluation reports Absolute Trajectory Error and relative translation and rotation pose errors.
  • Enhancing camera-controlled video generation with OmniWorld: The OmniWorld-fine-tuned camera-controlled video-generation model significantly outperforms the original baseline on both RealEstate10K and OmniWorld-Game.Evaluation uses 720 × 480 videos with 25-frame sequences and includes both benchmarks.

5. Related Work

Existing world-modeling datasets provide useful geometric or semantic data but remain limited in temporal length, scene diversity, and multimodal geometric annotation. OmniWorld addresses these gaps while supporting both geometric prediction and camera-controlled video generation.

  • Existing static 3D datasets advance reconstruction but cannot adequately model motion and dynamic interactions.
  • Video-text datasets provide semantic annotations but generally lack depth, camera poses, and optical flow for precise 3D world modeling.
  • Dynamic real-world datasets cover driving and human-robot interaction but often have limited scene diversity and noisy or sparse geometric annotations.
  • Synthetic datasets provide precise annotations, yet short sequences such as Sintel’s average of fewer than 50 frames limit large-scale foundation-model training.
  • OmniWorld integrates self-collected and public datasets to provide high-precision geometric annotations and rich spatio-temporal dynamics for evaluation and enhancement.
  • Camera-controlled video methods remain constrained by training data dominated by static scenes with smooth camera motion, limiting dynamic-scene handling.

6. Conclusion

OmniWorld addresses the data bottleneck in world modeling through a large-scale, diverse dataset and benchmark. Fine-tuning established models with OmniWorld improves performance, supporting its use for training and evaluation.

  • OmniWorld combines the OmniWorld-Game dataset with public datasets to address world modeling’s critical data bottleneck.
  • OmniWorld-Game provides a challenging benchmark for 3D geometric prediction and camera-controlled video generation that exposes current SOTA limitations.
  • Fine-tuning existing models with OmniWorld significantly boosts performance, underscoring its value as a training resource.

B.1. Data Statistics

OmniWorld-Game spans diverse scene categories and contains substantial internal diversity within its dominant Nature & Outdoors category. Its data collection also includes safeguards for compliant use of game content.

  • OmniWorld-Game covers 16 Point-of-Interest categories, including nature, tourist attractions, recreation, and hotels, with Nature & Outdoors the largest share.
  • Nature & Outdoors is subdivided into 5 second-level and 40 third-level categories, enabling detailed analysis of its internal composition.
  • Natural Landforms & Ecosystems dominates Nature & Outdoors, especially through Forests & Rainforests and Cliffs & Rock Formations.
  • Outdoor Sports & Scenic Routes is the second-largest category, with Rock-Climbing Areas and Scenic Drives & Viewpoints particularly prominent.
  • The dataset removes interface elements and text and manually filters scenes to comply with game-content terms, including non-commercial-use requirements.

C.1. 3D Geometric Prediction

The 3D geometric prediction benchmark evaluates monocular and video depth under standardized settings and reports camera-pose results across established datasets. It also excludes π3 because the benchmark data appears in that model’s training set.

  • The benchmark evaluates monocular depth on the first 200 frames of 18 test sequences using Abs Rel and δ<1.25 after per-frame scale alignment.
  • Video depth evaluation uses the first 100 frames and caps sequence length because some models cannot process longer inputs.
  • π3 is excluded because the benchmark data is included in its training set.
  • Camera-pose evaluation reports ATE, RPE trans, and RPE rot on Sintel, TUM-dynamics, and ScanNet, with CUT3R improving after OmniWorld fine-tuning.
  • Relative camera-pose evaluation uses AUC@5/10/20 on DynPose-100K and OmniWorld-CityWalk, with Reloc3r improving after OmniWorld fine-tuning.

D.2. Implementation Details

The implementation fine-tunes four SOTA models using OmniWorld data alongside selected original training sets, with model-specific freezing and optimization strategies.

  • D.2. Implementation Details: Four SOTA models—DUSt3R, CUT3R, Reloc3r, and AC3D—are fine-tuned with OmniWorld data and selected portions of their original training sets.The experiments use pretrained weights and model-specific training configurations.
  • D.2. Implementation Details: DUSt3R is fully fine-tuned for 40 epochs using random-resolution images, with each epoch comprising 800 iterations.Training uses AdamW and combines OmniWorld-Game with ARKitScenes, MegaDepth, and Waymo.
  • D.2. Implementation Details: CUT3R freezes its encoder while training the decoder and heads on 4–64-view sequences for 2,000 iterations.The setup uses higher-resolution images with varied aspect ratios and a total batch size of 96.
  • D.2. Implementation Details: Reloc3r freezes its ViT encoder and updates only the decoder and pose-related components using OmniWorld and multiple public datasets.Its fine-tuning mixture includes OmniWorld-Game, CityWalk, HoloAssist, and EpicKitchens.
  • D.2. Implementation Details: AC3D fine-tunes only its ControlNet while freezing the remaining network structures, using OmniWorld datasets together with RealEstate10K.The initialized ControlNet is based on CogVideoX-5B.

D.3. Visual Results.

Qualitative evaluations show that OmniWorld fine-tuning improves geometric detail and depth accuracy, while also improving camera-trajectory adherence and temporal consistency in generated videos.

  • D.3. Visual Results.: Fine-tuned models recover finer geometric details and produce more accurate depth maps on the Sintel video-depth benchmark.The comparison covers DUSt3R and CUT3R before and after fine-tuning on OmniWorld.
  • D.3. Visual Results.: The visual results indicate that OmniWorld strengthens geometric prediction and dynamic modeling across the evaluated tasks.These conclusions synthesize the reported depth-estimation and camera-controlled-generation comparisons.
  • D.3. Visual Results.: Fine-tuned AC3D videos follow desired camera trajectories more closely and maintain higher temporal consistency for moving objects.The comparison is performed on the OmniWorld-Game camera-controlled video-generation benchmark.
Loading 2509.12201v2…