PaperlayerView exact revision on arXiv

Source-linked AI summary

World2Minecraft: Occupancy-Driven Simulated Scenes Construction

Lechao Zhang, Haoran Xu, Jingyu Gong, Xuhong Wang, Yuan Xie, Xin Tan

arXiv:2604.27578v1cs.CV

TL;DR

Embodied-intelligence research needs high-fidelity, flexible simulation environments and better occupancy data than existing scan-based resources provide. World2Minecraft reconstructs real scenes in editable Minecraft using 3D semantic occupancy prediction and introduces an automated pipeline for occupancy datasets. Its MinecraftOcc dataset challenges current occupancy models while improving performance when used with NYUv2, and the reconstructed environments support VLN.

  • Problem

    Existing simulation platforms and occupancy datasets are limited by inflexible or artifact-prone representations, scarce annotations, sensor noise, and constrained object diversity.

  • Method

    World2Minecraft predicts and integrates 3D semantic occupancy from multi-view images to generate executable Minecraft reconstruction instructions, alongside an automated occupancy-data pipeline.

  • Results

    MinecraftOcc contains 100,165 images from 156 indoor scenes, challenges current occupancy models, and improves NYUv2 performance when used as auxiliary training data.

  • Takeaways & Limitations

    The framework provides customizable, editable reconstructed environments for VLN and MinecraftOcc serves as both a challenging benchmark and a resource for robustness-oriented training.

Abstract

from arXiv · show

Embodied intelligence requires high-fidelity simulation environments to support perception and decision-making, yet existing platforms often suffer from data contamination and limited flexibility. To mitigate this, we propose World2Minecraft to convert real-world scenes into structured Minecraft environments based on 3D semantic occupancy prediction. In the reconstructed scenes, we can effortlessly perform downstream tasks such as Vision-Language Navigation(VLN). However, we observe that reconstruction quality heavily depends on accurate occupancy prediction, which remains limited by data scarcity and poor generalization in existing models. We introduce a low-cost, automated, and scalable data acquisition pipeline for creating customized occupancy datasets, and demonstrate its effectiveness through MinecraftOcc, a large-scale dataset featuring 100,165 images from 156 richly detailed indoor scenes. Extensive experiments show that our dataset provides a critical complement to existing datasets and poses a significant challenge to current SOTA methods. These findings contribute to improving occupancy prediction and highlight the value of World2Minecraft in providing a customizable and editable platform for personalized embodied AI research. Project page:https://world2minecraft.github.io/.

1 INTRODUCTION

World2Minecraft reconstructs real-world scenes as editable Minecraft environments through 3D semantic occupancy prediction, then supports downstream VLN evaluation. The paper also introduces automated occupancy-data generation to address reconstruction and generalization challenges.

  • World2Minecraft: World2Minecraft converts real-world scenes into high-quality, editable Minecraft environments using 3D semantic occupancy prediction.The framework is designed for direct use in downstream tasks such as Vision-Language Navigation.
  • World2Minecraft: The pipeline predicts single-frame semantic occupancy, integrates multi-frame observations using camera parameters, and generates Minecraft construction instructions.A visual tool can refine the unified semantic occupancy field before reconstruction.
  • Downstream VLN: MinecraftVLN contains 1,059 samples from reconstructed scenes and 2,483 additional samples from community-created scenes for Next-View and Next-Action tasks.Qwen2.5-VL-3B and Qwen2.5-VL-7B achieved notable gains, and Gemini-2.5-Pro controlled real-time navigation in reconstructed environments.
  • Motivation: Reconstruction quality remained sub-optimal for practical use, motivating improved occupancy prediction and scalable occupancy-data construction.The paper identifies accurate occupancy prediction as a key requirement for reconstruction quality.
  • MinecraftOcc: MinecraftOcc provides 100,165 high-resolution images from 156 detailed indoor scenes generated through a low-cost, automated pipeline.The pipeline uses physically based rendering and precise layout control to create realistic environments with complex structures, diverse objects, and dynamic lighting.

2 RELATED WORK

Prior work spans generative and scan-based scene construction, but existing approaches do not jointly provide faithful reconstruction, editability, and scalable occupancy data. World2Minecraft addresses these gaps with semantic occupancy-driven Minecraft reconstruction and automated dataset generation.

  • Data-Driven 3D Scene Generation: Generative methods create novel 3D content but are not designed to faithfully reconstruct specific existing real-world locations.The related work contrasts this limitation with World2Minecraft’s real-to-sim reconstruction goal.
  • Indoor 3D Occupancy Prediction Dataset: Existing occupancy datasets derived from real-world scans suffer from sensor noise, sparse annotations, constrained object diversity, and costly production.These limitations motivate a more efficient paradigm for scalable and diverse occupancy data generation.
  • Minecraft for Embodied Intelligence: Minecraft provides a customizable embodied-intelligence and reinforcement-learning environment supported by frameworks such as MineStudio.The related work situates World2Minecraft within broader Minecraft-based research on open-world interaction and vision-language agents.

3 METHOD

World2Minecraft converts real-world scenes into editable Minecraft environments through multi-view 3D semantic occupancy prediction, refinement, object-center processing, and command-based rendering. The paper also constructs MinecraftVLN to evaluate navigation in reconstructed and community-created scenes, while MinecraftOcc supports occupancy-data generation and comparison.

  • World2Minecraft: World2Minecraft predicts semantic occupancy from individual RGB views and merges the predictions with camera extrinsics into a unified scene representation.The monocular predictor uses RGB images and camera intrinsics, while multi-view fusion uses extrinsic parameters.
  • World2Minecraft: The pipeline converts fused occupancy into object centers by thresholding a binary density map and clustering centers independently within each semantic class.A 3D convolution produces local density values, and DBSCAN with a distance threshold η represents each cluster by its centroid.
  • World2Minecraft: Retrieved geometric representations are translated into Minecraft building commands and executed to render a semantically decomposed, editable virtual scene.Instance occupancy grids are matched with a furniture library while testing discrete rotations to maximize spatial overlap.
  • Enabling VLN in Minecraft: MinecraftVLN processes roomtour trajectories into navigation segments and generates question-answer items for Next-View and Next-Action prediction.The dataset combines 1,059 samples from reconstructed scenes with 2,483 samples from community-created scenes, totaling 3,801 items.
  • Enabling VLN in Minecraft: VLN experiments fine-tune Qwen2.5-VL models and deploy Gemini-2.5-Pro for direct embodied navigation control in Minecraft.The reconstructed validation scenes were manually refined in 15 cases, and five community-created scenes were added to increase navigation complexity.
  • MinecraftOcc dataset construction: MinecraftOcc acquires paired screenshots and camera poses automatically, defines viewpoint-dependent occupancy volumes, and extracts semantic labels from Minecraft map data.Axis-aligned and diagonal viewpoints use different volume placement rules; a viewpoint-aware offset addresses peripheral voxel loss in diagonal views.

4 EXPERIMENTS

Experiments evaluate World2Minecraft’s reconstruction quality, downstream VLN utility, occupancy-data challenges, layout-based generation, and construction efficiency. Results show effective scene alignment, improved navigation after fine-tuning, strong dataset complementarity, competitive reconstruction quality, and substantially faster refinement.

  • Reconstruction Quality: World2Minecraft reconstructs scenes with predicted occupancy views and reconstructed Minecraft scenes that align well across multiple viewpoints.The prediction and construction columns collectively demonstrate the pipeline’s effectiveness.
  • VLN Evaluation: Fine-tuning improves Qwen2.5-VL performance, but the best strategy varies by model size, task, and dataset.SFT is stronger for 3B Next-View, while RFT performs better on Extend and Combined Next-Action settings.
  • VLN Evaluation: Gemini-2.5-Pro successfully navigates to a piano in a reconstructed scene by following the instruction “Go to the piano.”This demonstrates real-time VLN in the reconstructed Minecraft environment.
  • MinecraftOcc Evaluation: MinecraftOcc outperforms NYUv2 and OccScanNet in image quantity and NIQE, PIQE, and LV metrics, while challenging mainstream occupancy methods.Joint training with NYUv2 improves Symphonies by IoU +0.43 and mIoU +0.21, indicating complementarity with existing data.
  • Layout-Based Comparison: World2Minecraft achieves 0.913 Semantic Integrity, 6.145 Visual Realism, 0.024 OOB Rate, and 0.2 Collision Count, outperforming layout-based baselines across most metrics.The results are attributed to semantic occupancy prediction combined with shape-aware template matching.
  • Efficiency: 70.38s per scene reduces construction time sevenfold versus 482.00s from scratch, using 24.5 refinement operations instead of 340.Refinement consists of deleting floating artifacts, filling minor surface holes, and adjusting object orientations.

5 CONCLUSION

The paper introduces World2Minecraft and MinecraftOcc as a framework and dataset for structured simulated scenes and occupancy prediction. Experiments establish downstream VLN utility and MinecraftOcc’s roles as a challenging benchmark and complementary training resource.

  • Conclusion: World2Minecraft converts real-world scenes into structured Minecraft environments through 3D semantic occupancy prediction.The framework targets editable simulated scenes for embodied AI research.
  • Conclusion: MinecraftOcc provides diverse indoor scenes with voxel-wise semantic annotations for occupancy prediction.The dataset is presented as a scalable resource for evaluation and training.
  • Conclusion: Reconstructed environments support downstream Vision-Language Navigation tasks.The experiments demonstrate their utility for navigation research.
  • Conclusion: MinecraftOcc exposes limitations in state-of-the-art models and augments existing real-world datasets.The paper describes both benchmark and training-resource value.

ETHICS STATEMENT

The work uses simulated Minecraft scenes to support safety, transparency, and repeatability while avoiding privacy issues associated with real-world data collection.

  • Ethics Statement: MinecraftOcc and MinecraftVLN are built entirely from simulated scenes and contain no real human data, private details, or identifiable personal spaces.The controlled virtual-world setup avoids privacy issues from collecting real-world data.
  • Ethics Statement: The simulated-data design supports a controlled and consistent experimental setup for embodied AI research.The stated ethical focus includes safety, transparency, and repeatability.

REPRODUCIBILITY STATEMENT

The reproducibility workflow uses supplied code, datasets, models, camera-aware occupancy processing, and Minecraft mods to reconstruct editable, high-fidelity indoor scenes.

  • Reproduction workflow: Reproduction requires downloading the World2Minecraft code, MinecraftOcc, and MinecraftVLN datasets before training EmbodiedOcc and generating Minecraft commands.The commands are executed in a TMEO-enabled Minecraft environment for scene reconstruction and subsequent VLN evaluation.
  • Environment setup: The extended environment addresses vanilla Minecraft’s low-resolution visuals, limited semantic diversity, and simplistic indoor layouts.These limitations create a domain gap and restrict perception-dependent embodied tasks.
  • Environment setup: WorldEdit, Screen with Coordinates, and TMEO provide procedural construction, pose-linked image capture, and fine-grained textured object models.TMEO adds over 1,400 semantically labeled object models and supports physically based rendering.
  • Environment setup: The resulting system contains 156 detailed scenes and approximately 1,600 rooms with multistory structures, complex layouts, and dense furnishings.High-definition textures, physically based rendering, and dynamic lighting provide realistic shadows, reflections, and global illumination.
  • Reconstruction pipeline: World2Minecraft processes multiview images and camera parameters through occupancy prediction, volumetric fusion, density filtering, clustering, and Minecraft command execution.The final commands place specified block types at 3D coordinates, producing a semantically decomposed and editable virtual world.
  • Reconstruction pipeline: Per-view monocular occupancy volumes are fused into a global scene representation, filtered by density, clustered to reduce redundancy, and translated into building commands.Camera parameters help merge information across views and resolve inconsistencies.
  • Camera data: Screen with Coordinates records screenshots together with camera position and orientation, enabling computation of intrinsic and extrinsic matrices for each image.The intrinsic matrix uses field of view and image dimensions, while the extrinsic matrix uses position and rotation.

F DATASET ANALYSIS

The dataset analysis summarizes navigation-instruction lengths across tasks and datasets, showing that Extend uses shorter and more uniform instructions.

  • Instruction statistics: Table 8 reports instruction count, mean, standard deviation, and quartiles for navigation instructions across tasks and datasets.The analysis covers the Next-View and Next-Action settings.
  • Instruction statistics: Extend contains shorter and more uniform instructions than the other compared datasets.This comparison is shown for the Next-View and Next-Action tasks.

G SEMANTIC CLASS MAPPING FOR CROSS-DATASET EXPERIMENTS

Cross-dataset experiments map MinecraftOcc’s fine-grained classes to 12 common indoor categories, creating a standardized basis for evaluation.

  • Class taxonomy: MinecraftOcc’s 1,000+ fine-grained classes are mapped many-to-one to 12 standardized indoor semantic categories.The mapping includes 200 distinct lighting fixtures and establishes a shared semantic ground with benchmarks such as NYUv2.
  • Class taxonomy: The taxonomy groups Minecraft blocks and items by functional and structural roles, with representative examples summarized in Table 10.The complete class mapping is supplied as a JSON file for reproducibility.

H IMAGE QUALITY METRICS

Image quality is evaluated with complementary no-reference metrics covering naturalness, local distortions, and sharpness.

  • Metric definitions: NIQE measures similarity between image statistics and natural images, with lower values indicating more natural quality.It evaluates global image naturalness without reference images.
  • Metric definitions: PIQE assesses local distortions and artifacts using human visual perception, with lower values indicating fewer distortions.It captures perceptual artifact levels from a local perspective.
  • Metric definitions: Laplacian Variance measures sharpness through the variance of Laplacian-filtered responses, with higher values indicating more detail.Together, NIQE, PIQE, and LV provide complementary visual-quality perspectives.

I.1 EXPERIMENT 1: REINFORCEMENT FINE-TUNING

The experiments used distinct SFT and RFT configurations, with RFT jointly training the vision and language components under GRPO, while SFT froze visual modules and trained only the language model.

  • RFT configuration: RFT used EasyR1, unfroze the vision tower, and trained it alongside the language components with GRPO.A KL divergence penalty with coefficient 1.0×10−2 regularized the policy.
  • RFT configuration: RFT prompts and responses came from Parquet files, formatted with a custom Jinja template and capped at 4096 and 1024 tokens, respectively.
  • RFT configuration: RFT ran on one 8-GPU node using fully sharded data parallelism with CPU offloading, while rollout acceleration used tensor parallel size 2.
  • SFT configuration: SFT used LLaMA Factory, froze the vision tower and multi-modal projector, and fine-tuned only the language model with full-parameter training.
  • SFT configuration: SFT used up to 1000 samples with the qwen2 vl conversation template, an 8192-token sequence limit, five epochs, ZeRO Stage 3, bfloat16, cosine scheduling, and 10% warmup.No validation was performed during SFT training.

J INTERACTIVE VISUALIZATION TOOL : SCENEFORGE

SceneForge is a standalone web-based tool for inspecting and editing semantic occupancy predictions, supporting visualization, qualitative analysis, and manual refinement for World2Minecraft environments.

  • Purpose: SceneForge provides Open3D-like 3D exploration to inspect and manually correct semantic occupancy predictions within the World2Minecraft pipeline.
  • Implementation: The standalone web application uses D3.js for 3D rendering and interaction.
  • Visualization: It loads voxel-wise occupancy data and pre-computed center points, and supports rotation, zooming, panning, and category-aware visualization.Twelve semantic categories are color-coded and individually toggleable.
  • Editing: Editing functions include repositioning center points, deleting points, splitting points for fine-grained separation, and bulk category operations.
  • Research uses: Researchers used SceneForge to inspect clustering results, identify failure cases, and understand limitations of automatic center prediction.
  • Research uses: For MinecraftVLN dataset creation, manual editing corrected inaccurate object centers to produce higher-quality navigation environments.
Loading 2604.27578v1…