PaperlayerView exact revision on arXiv

Source-linked AI summary

HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

Team HY-World, Chenjie Cao, Xuhui Zuo, Zhenwei Wang, Yisu Zhang, Junta Wu, Zhenyang Liu, Yuning Gong, Yang Liu, Bo Yuan, Chao Zhang, Coopers Li, Dongyuan Guo, Fan Yang, Haiyu Zhang, Hang Cao, Jianchen Zhu, Jiaxin Lin, Jie Xiao, Jihong Zhang, Junlin Yu, Lei Wang, Lifu Wang, Lilin Wang, Linus, Minghui Chen, Peng He, Penghao Zhao, Qi Chen, Rui Chen, Rui Shao, Sicong Liu, Wangchen Qin, Xiaochuan Niu, Xiang Yuan, Yi Sun, Yifei Tang, Yifu Sun, Yihang Lian, Yonghao Tan, Yuhong Liu, Yuyang Yin, Zhiyuan Min, Tengfei Wang, Chunchao Guo

arXiv:2604.14268v1cs.CV

TL;DR

HY-World 2.0 addresses the separation between 3D world generation and reconstruction by unifying both capabilities across sparse and dense multimodal inputs. Its four-stage framework combines panorama generation, trajectory planning, world expansion, and world composition, and experiments show state-of-the-art performance among open-source approaches comparable with Marble. The system also provides released models, code, and technical details for reproducibility.

  • Problem

    Existing 3D world methods generally specialize in generation from sparse inputs or reconstruction from dense observations, leaving open-source systems without a unified approach.

  • Method

    HY-World 2.0 adapts to text, single-view image, multi-view image, and video inputs through four stages: panorama generation, trajectory planning, world expansion, and world composition.

  • Results

    HY-World 2.0 achieves state-of-the-art performance among open-source 3D world modeling approaches and is competitive with the closed-source product Marble.

  • Takeaways & Limitations

    The released models, code, and technical details provide an open-source foundation for reproducible research on multimodal offline 3D world modeling.

  • Takeaways & Limitations

    WorldMirror 2.0 still struggles in highly challenging outdoor scenes, despite outperforming other feed-forward reconstruction methods under camera conditions.

Abstract

from arXiv · show

We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.

1 Introduction

HY-World 2.0 unifies 3D world generation and reconstruction across sparse and rich visual inputs in an open-source offline framework. Its component upgrades support navigable 3DGS generation, geometrically consistent reconstruction, and strong benchmark performance among open-source approaches.

  • Existing 3D world methods typically specialize in either generation from sparse inputs or reconstruction from dense observations, while open-source systems lack a unified solution.
  • HY-World 2.0 unifies generation and reconstruction for text, single-view image, multi-view image, and video inputs within an offline 3D world model.
  • For text or single-view inputs, the four-stage pipeline synthesizes high-fidelity, navigable 3DGS worlds through panorama generation, trajectory planning, world expansion, and world composition.
  • For multi-view images or videos, the framework reconstructs geometrically consistent 3D structures, with reconstruction also serving as a foundation for world generation.
  • HY-World 2.0 upgrades panorama generation, trajectory planning, WorldStereo, and WorldMirror, including adaptive mapping, information-aware paths, consistent memory, and enhanced 3DGS training.
  • The framework achieves state-of-the-art performance among open-source approaches, remains competitive with Marble, and releases models, code, and technical details.

2 Overview

HY-World 2.0 transforms multimodal inputs into 3D worlds through four sequential stages: panorama initialization, trajectory planning, generative expansion, and 3DGS composition.

  • Panorama Generation converts arbitrary text or image inputs into a high-fidelity 360° world initialization.
  • Trajectory Planning parses the initialized world and derives optimal, information-rich observation paths for subsequent processing.
  • World Expansion follows the planned routes to generate additional world observations before World Composition constructs the final 3DGS representation.

3 World Generation Stage I: Panorama Generation

HY-Pano 2.0 generates coherent, high-fidelity panoramas from text and single-view images to initialize 3D worlds. It combines expanded training data with implicit multimodal perspective-to-ERP mapping and seamless boundary refinement.

  • 360°×180° panoramas preserve global spatial context and semantic relationships needed for coherent viewpoint synthesis and immersive exploration.
  • HY-Pano 2.0 synthesizes high-fidelity panoramas from text and single-view image conditions using expanded data curation and a dedicated 360° generative model.
  • The upgraded dataset combines high-resolution real-world panoramas with synthetic assets to increase training-data richness and diversity.
  • HY-Pano 2.0 replaces metadata-dependent geometric warping with adaptive implicit perspective-to-ERP mapping, addressing unavailable or inaccurate camera metadata.
  • Circular padding in latent space and linear pixel blending at ERP edges smooth the 360° wrap-around transition and remove boundary artifacts.

4 World Generation Stage II: Trajectory Planning

WorldNav plans diverse, collision-free camera trajectories from parsed panoramic scenes to expand viewpoint coverage and target under-observed regions. Its five trajectory modes combine general exploration, object-focused observation, reconstruction-aware coverage, boundary reaching, and aerial views.

  • Trajectory Planning: WorldNav derives exploration trajectories from panoramas and pairs diverse camera paths with textual instructions for downstream world expansion.
  • Geometric and Semantic Scene Parsing: Scene parsing produces panoramic point clouds, meshes, semantic masks, and NavMesh representations for trajectory planning.
  • Trajectory Planning: WorldNav uses five heuristic trajectory modes from the panorama center to cover diverse viewpoints while maintaining collision-free movement.
  • Trajectory Modes: Regular trajectories expand coverage beyond the panorama origin, while surrounding trajectories orbit significant objects to improve foreground observations.
  • World Expansion: WorldStereo 2.0 is trained in three stages that progressively enable camera control, memory-based consistency, and efficient inference.
  • Trajectory Modes: Reconstruct-aware trajectories target under-observed mesh regions, while wandering and aerial trajectories extend visibility toward scene boundaries and remaining blind viewpoints.

5 World Generation Stage III: World Expansion

WorldStereo 2.0 expands worlds through camera-guided keyframe video generation with a high-fidelity latent space and complementary memories for consistent reconstruction.

  • Overview of WorldStereo 2.0: WorldStereo 2.0 bridges camera-conditioned video diffusion and 3D reconstruction through consistent multi-trajectory video generation with geometry-aware memories.The method is designed for camera control, memory-based consistency, and efficient inference.
  • Memory and Camera Control: WorldStereo 2.0 combines SSM++ in the main DiT branch with GGM point-cloud guidance in the camera-control branch.SSM++ supports fine-grained consistency, while GGM supports precise trajectory following and geometry-aware consistency.
  • Keyframe-Based Spatial VAE: Keyframe-VAE uses spatial-only compression to preserve high-frequency details and reduce motion-blur and geometric-distortion artifacts from spatio-temporal Video-VAE compression.It independently encodes a sparse keyframe set, with Tkf ≪ Tvid.
  • Camera-Guided Generation: Explicit camera control uses camera Plücker rays and point clouds as complementary guidance for precise target-view generation.The domain-adaptation stage uses reference-view point clouds and fine-tunes part of the DiT backbone for the keyframe latent space.
  • Memory and Camera Control: GGM renders extended point clouds as global 3D priors, while SSM++ retrieves keyframes to reinforce local correspondence and fine-grained details.Additional point clouds are sampled from novel views, and retrieval is tailored to dataset characteristics.

6 World Reconstruction: WorldMirror 2.0

WorldMirror 2.0 upgrades feed-forward 3D prediction for flexible, robust reconstruction by improving positional encoding, geometric supervision, and invalid-pixel handling.

  • WorldMirror Architecture: WorldMirror 2.0 is a unified feed-forward model that predicts point maps, depths, normals, cameras, and 3DGS attributes from tokenized multi-modal inputs.Its two-phase curriculum separates geometry learning from appearance modeling.
  • Model Improvements: Three model-level improvements are normalized position encoding, depth-to-normal supervision, and a dedicated depth-mask prediction head.The improvements target flexible resolution inference, geometric coupling, and robust invalid-pixel handling.
  • Normalized Position Encoding: Normalized RoPE maps positions from every resolution into [−1, 1], converting resolution extrapolation into interpolation and maintaining high cross-resolution consistency.The accompanying analysis reports cross-resolution cosine similarity above 0.95 for normalized RoPE.
  • Explicit Normal Supervision: The depth-to-normal loss derives surface normals from predicted depth and supervises them against normal targets using angular error.Normal targets come from ground-truth depth for synthetic data and monocular normal teachers for real-world data.
  • Depth Mask Prediction: The depth-mask head predicts per-pixel validity logits, allowing downstream systems to filter invalid pixels during point-cloud fusion and 3D reconstruction.Real-world pseudo-labels use extreme depths, discontinuities, and sky regions.

6.3 Data Improvements

WorldMirror 2.0 expands training data with accurate synthetic geometry and normal-only pseudo-label enhancement for real-world datasets.

  • Synthetic Data: The training set adds Unreal Engine renderings with pixel-accurate geometry across diverse indoor and outdoor environments.These synthetic data complement real-world datasets.
  • Real-World Pseudo-Labels: For real-world data, WorldMirror 2.0 uses normal-only pseudo-label enhancement instead of independently predicted per-view depth labels.The passage motivates this choice because independently predicted depths introduce multi-view inconsistencies.
  • Inference Efficiency: WorldMirror 2.0 introduces sequence parallelism and other acceleration strategies for scalable multi-GPU deployment.The approach addresses WorldMirror 1.0’s single-GPU FP32 inference limitations.

6.5 Training Strategy Improvements

WorldMirror 2.0 improves training efficiency and robustness through token-budget-first batching, bounded per-GPU workloads, and a three-stage geometry curriculum.

  • Token-Based Dynamic Batch Sizing: Token-budget-first batching fixes a maximum per-GPU token budget before deriving the allowable view count.The example budget is 25,000 tokens per GPU, with image resolution sampled first.
  • Token-Based Dynamic Batch Sizing: Packing multiple smaller samples fills the token budget, producing near-full GPU utilization across resolutions and avoiding out-of-memory errors.The design also exposes training to more diverse resolution–view-count combinations.
  • Token Budget: The total token count per GPU is bounded by Ttotal = N × H_p × W_p ≤ T_max.N counts all images on one GPU, including images from multiple packed samples.
  • Multi-Stage Curriculum Learning: WorldMirror 2.0 decomposes geometry training into two sub-stages before Gaussian-head training, yielding a three-stage curriculum.The first geometry stage uses native annotations, while later stages introduce enhanced supervision and subsequent appearance learning.

7 World Generation Stage IV: World Composition

World Composition integrates panoramic and generated keyframe inputs into a globally aligned point cloud, then optimizes a 3DGS representation for navigable world rendering. WorldMirror 2.0 supplies camera-conditioned depth and normals, while alignment and filtering address scale ambiguity, outliers, sky regions, and challenging scenes.

  • World Composition: World Composition expands the panoramic point cloud with generated keyframes, then trains a 3DGS initialized from the expanded cloud.These sequential steps produce the unified 3D representation used for complete high-fidelity world synthesis.
  • Reconstruction via WorldMirror 2.0: WorldMirror 2.0 estimates per-frame depth and normal maps for selected generated frames, conditioned on their camera poses as geometric priors.The resulting reconstructions are used for point cloud expansion.
  • Reconstruction via WorldMirror 2.0: Camera conditioning enables WorldMirror 2.0 to outperform Mapanything and DepthAnything3 under identical conditions, although challenging outdoor scenes remain difficult.Figure 15 compares the reconstructed point clouds and includes a depth-aligned WorldMirror 2.0 result.
  • Depth Alignment: RANSAC-based linear alignment estimates per-frame scale and shift over reliable overlaps, using the panoramic point cloud as geometric guidance.The method uses valid overlapping regions and omits sky regions before alignment; disparity-space alignment is applied in practice for better foreground alignment.
  • Depth Alignment: Global outlier detection replaces anomalous alignment coefficients with nearby inliers and discards sequences identified entirely as outliers before back-projecting aligned depths.The expanded point cloud is formed by uniting the back-projected extension with the panorama and voxel-downsampling the result.
  • 3D Gaussian Splatting: The 3DGS uses learnable opacity, position, covariance, and view-independent RGB attributes, while selective densification separates sky from scene regions.Scene-only growth supports detail reconstruction while preventing sky floaters; view-independent colors reduce redundancy and complexity.

8.1 World Generation from Text or Single Image

HY-World 2.0 evaluates its text- and single-image-driven world generation pipeline across panorama synthesis, trajectory planning, reconstruction, composition, and interactive rendering. The results show strong panorama quality, progressively improved scene completeness, consistent single-view reconstruction, efficient 3DGS construction, and competitive comparisons with Marble.

  • Panorama Generation: HY-Pano 2.0 achieves the best scores on most text-to-panorama and all five image-to-panorama metrics.It improves adherence to input signals, fine-grained detail quality, and aesthetics over prior methods.
  • Panorama Generation: HY-Pano 2.0 produces structurally coherent 360° panoramas with plausible spatial arrangements and consistent geometric structures.Qualitative comparisons report stronger layout coherence, details, and aesthetics than existing methods.
  • Trajectory Planning: Sequential trajectory integration progressively improves scene completeness by eliminating blind spots, completing occluded structures, enhancing distant textures, and adding bird’s-eye views.Regular, surrounding/reconstruction-aware, wandering, and aerial trajectories address different coverage and detail limitations.
  • World Expansion: WorldStereo 2.0 achieves the highest point-cloud F1 and AUC scores on the single-view reconstruction benchmark despite inherent generative uncertainty.The evaluation uses challenging closed-loop and explorable trajectories to test multi-view consistency.
  • World Expansion: Memory training and distillation improve photometric quality, multi-trajectory consistency, camera control, robustness, and training stability.Replacing spatial-stereo stitching with temporal concatenation severely degrades performance, while DMD retains comparable control and slightly improves other metrics.
  • World Composition: The full 3DGS configuration reduces Gaussian count by 77% relative to baseline while retaining comparable visual quality.MaskGaussian reduces 5.254M Gaussians to 1.383M with only −0.14 dB PSNR degradation before the full configuration’s additional reduction.
  • Interactive Worlds: HY-World 2.0 provides panoramas, aligned point clouds, 3DGS renderings, and meshes for explorable and interactive worlds.Meshes extracted from 3DGS serve as collision proxies for interactive environments.
  • Full Results: Compared with Marble, HY-World 2.0 better preserves input fidelity, detail, and geometric consistency across viewpoint changes.The complete-world generation pipeline requires 10 minutes on NVIDIA H20 GPUs.

8.2 World Reconstruction from Multi-View Images or Video

WorldMirror 2.0 is evaluated as a multi-resolution reconstruction foundation model across point maps, camera poses, depth, novel views, surface normals, geometric priors, and efficiency. It maintains or improves reconstruction quality at high resolution, where WorldMirror 1.0 often degrades substantially.

  • Multi-Resolution Reconstruction: WorldMirror 2.0 maintains or improves performance from medium to high resolution across every evaluated benchmark.WorldMirror 1.0 suffers severe high-resolution degradation from position extrapolation.
  • Point Map Reconstruction: WorldMirror 2.0 reduces 7-Scenes point-map accuracy error from 0.043 to 0.033 at medium resolution and from 0.079 to 0.037 at high resolution.With all geometric priors, it achieves the best overall results on 7-Scenes and DTU.
  • Camera Pose, Depth, and Novel View Synthesis: WorldMirror 2.0 improves camera-pose AUC@30 at every resolution, including 66.29→86.89 at high resolution.It also reduces depth AbsRel and achieves δ<1.25 accuracy of 0.815 at high resolution.
  • Camera Pose, Depth, and Novel View Synthesis: WorldMirror 2.0 maintains stable novel-view synthesis across resolutions, with PSNR 20.14/20.07/19.98 and best high-resolution SSIM of 0.726.WorldMirror 1.0’s high-resolution PSNR falls from 21.34 to 17.78.
  • Surface Normal Estimation: WorldMirror 2.0 achieves the best surface-normal results across ScanNet, NYUv2, and iBims-1 at medium resolution.At high resolution, ScanNet mean error remains close to the medium-resolution optimum: 12.5 versus 12.3.
  • Qualitative Results: WorldMirror 2.0 produces sharper normals and coherent point clouds, remaining stable under dense 32-view and sparse 8-view inputs across resolutions.WorldMirror 1.0 can collapse geometrically at high resolution, particularly with dense inputs.
  • Geometric Prior Injection: Flexible geometric-prior integration lets WorldMirror 2.0 outperform alternatives and produce more globally consistent point clouds under shared camera-pose conditioning.The model incorporates camera poses, intrinsics, and depth through Any-Modal Tokenization.
  • Inference Efficiency: BF16 mixed precision reduces per-GPU memory by approximately 40% and enables 256-view inference that is out of memory in the FP32 baseline.The benchmark uses NVIDIA H20 GPUs at 518×378 resolution.

9 Conclusion

HY-World 2.0 unifies 3D world generation and reconstruction across sparse and dense input modalities through a four-stage pipeline and interactive rendering platform. Extensive evaluations report state-of-the-art open-source performance and competitiveness with Marble.

  • Conclusion: HY-World 2.0 bridges 3D world generation and reconstruction for inputs ranging from text and single images to multi-view videos.Its pipeline combines HY-Pano 2.0, WorldNav, WorldStereo 2.0, and WorldMirror 2.0 to produce navigable 3DGS assets.
  • Conclusion: WorldLens enables interactive exploration of generated worlds with character support and lighting control.The framework produces panoramas, point clouds, 3DGS renderings, and meshes.
  • Conclusion: Extensive evaluations demonstrate state-of-the-art performance among open-source approaches and competitiveness with Marble.The authors release models, code, and technical details to support reproducibility and further research.

Contribution*

The contribution section lists project leads, core contributors, engineering and infrastructure contributors, data and art-design contributors, and an alphabetical-order convention.

  • Project Leads: The report identifies Chunchao Guo and Tengfei Wang as project leads.
  • Core Contributors: The report separately lists its core contributors in alphabetical order.
  • Contributor Groups: Engineering, infrastructure, data, and art-design contributors are listed in dedicated groups.
  • Ordering: All contributors are ordered alphabetically by first name.
Loading 2604.14268v1…