PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldGen: From Text to Traversable and Interactive 3D Worlds

Dilin Wang, Hyunyoung Jung, Tom Monnier, Kihyuk Sohn, Chuhang Zou, Xiaoyu Xiang, Yu-Ying Yeh, Di Liu, Zixuan Huang, Thu Nguyen-Phuoc, Yuchen Fan, Sergiu Oprea, Ziyan Wang, Roman Shapovalov, Nikolaos Sarafianos, Thibault Groueix, Antoine Toisoul, Prithviraj Dhar, Xiao Chu, Minghao Chen, Geon Yeong Park, Mahima Gupta, Yassir Azziz, Rakesh Ranjan, Andrea Vedaldi

arXiv:2511.16825v1cs.CVcs.AI

TL;DR

Direct text-to-3D scene generation lacks sufficiently large training sets, while creating 3D content is complex and expertise-intensive. WorldGen combines procedural layouts with navmesh-conditioned scene reconstruction, and its model achieves 40–50% lower Chamfer distances than baselines on a curated scene benchmark.

  • Problem

    Direct text-to-3D scene generation lacks a sufficiently large training set, while creating 3D content is complex, time-consuming, and expertise-intensive.

    Source: §1 · p. 2

  • Method

    WorldGen guides image generation with procedurally generated scene layouts and performs holistic 3D reconstruction conditioned on a reference image and navmesh.

    Source: §1 · p. 3, §2 · p. 4

  • Results

    40–50% lower Chamfer distances than baselines were achieved by the navmesh-conditioned model on a curated scene benchmark.

    Source: §4.4 · p. 12

  • Takeaways & Limitations

    WorldGen illustrates the potential of language-guided procedural reasoning and 3D diffusion for accessible, scalable interactive world generation for games and social experiences.

    Source: §9 · p. 29

    Reliance on a single reference view restricts scene scale; kilometer-scale open worlds are not natively supported.

    Source: §9 · p. 29

Abstract

from arXiv · show

We introduce WorldGen, a system that enables the automatic creation of large-scale, interactive 3D worlds directly from text prompts. Our approach transforms natural language descriptions into traversable, fully textured environments that can be immediately explored or edited within standard game engines. By combining LLM-driven scene layout reasoning, procedural generation, diffusion-based 3D generation, and object-aware scene decomposition, WorldGen bridges the gap between creative intent and functional virtual spaces, allowing creators to design coherent, navigable worlds without manual modeling or specialized 3D expertise. The system is fully modular and supports fine-grained control over layout, scale, and style, producing worlds that are geometrically consistent, visually rich, and efficient to render in real time. This work represents a step towards accessible, generative world-building at scale, advancing the frontier of 3D generative AI for applications in gaming, simulation, and immersive social environments.

1 Introduction

WorldGen addresses the challenge of creating coherent, functional 3D worlds from text prompts in a form compatible with existing game engines. It combines language-guided scene planning, procedural layouts, image generation, 3D reconstruction, and object-level refinement to produce detailed, navigable scenes.

Introduction

  • Automating 3D creation could reduce the time, expertise, and resources needed to make interactive experiences and enable personalized content creation.Existing 3D generators can create objects from text, but full experiences also require scenes, interactions, gameplay, and other components; pixel-generating interactive video systems are not yet mature replacements for traditional world creation.Source: §1 · p. 2
  • WorldGen generates 3D worlds from a single text prompt, aiming for scenes that are coherent, contextually appropriate, and traversable.World objects must fit thematically and stylistically, make sense in context, and support interactive use without navigation-blocking obstructions.Source: §1 · p. 2
  • Because large 3D-scene training sets are unavailable, WorldGen first generates a scene image and then reconstructs it in 3D as conditional-generation tasks.This lets the system leverage text-to-image models trained on billions of images to interpret prompts and produce a scene plan specifying objects, appearance, and relationships.Source: §1 · pp. 2–3
  • An LLM maps the prompt to procedural-generation parameters for a basic layout and navmesh, while image generation supplies semantic objects and stylistic detail.The procedural layout supports viable, traversable scenes; leaving object meaning to the image generator enables stylistic and thematic variety.Source: §1 · p. 3
  • WorldGen reconstructs the scene holistically with AssetGen2, fine-tuned to account for the navmesh and preserve navigability despite occlusions or hallucinated objects.A holistic mesh alone lacks sufficient detail and is difficult to edit or make interactive, motivating subsequent object-level processing.Source: §1 · p. 3
  • A variant of AutoPartGen decomposes the holistic mesh into meaningful objects, which are individually regenerated from high-resolution views while conditioned on their original geometry.This adds detail while controlling deviations so refreshed parts continue to fit together; decomposition operates on the holistic mesh rather than assuming blockout elements correspond one-to-one with objects.Source: §1 · p. 3
  • WorldGen training draws on curated artist-created scenes and synthetic scenes composed from internal 3D objects to fine-tune generative models for world construction.The synthetic-scene method lacks WorldGen's control, diversity, and quality but provides useful training data for AssetGen2 and AutoPartGen.Source: §1 · p. 3

2 WorldGen Overview

WorldGen turns a single text prompt into a functional, compositional 3D scene through four stages, moving from high-level scene planning to detailed components.

Stage I: Scene Planning

  • Stage I creates a scene plan with a 3D blockout, a style reference image, and a navmesh of traversable regions.The blockout captures functional structures such as open spaces, loops, and chokepoints; its depth map conditions generation of the reference image.Source: §2 · p. 4

Stage II: Scene Reconstruction

  • Stage II reconstructs the planned scene as one textured mesh, using the reference image and navmesh to preserve spatial coherence and traversability.The holistic reconstruction helps resolve object relationships and occluded areas, while navmesh constraints reduce non-reachable regions.Source: §2 · p. 4, Figure 2 · p. 6

Stage III: Scene Decomposition

  • Stage III decomposes the scene mesh into individual assets, enabling piece-by-piece enhancement and local editing without regenerating the world.AutoPartGen identifies the parts and their number from the mesh, with upgrades for scene-like objects and efficient handling of many parts.Source: §2 · p. 4

Stage IV: Scene Enhancement

  • Stage IV enhances each object by generating a detailed image from a representative view, refining its mesh, and applying aligned high-resolution textures.The refinement uses both the generated image and the coarse object shape, preserving the object's overall structure while adding detail.Source: §2 · pp. 4, 6

WorldGen Overview

  • WorldGen outputs individually editable, fully textured meshes that support collision and navigation and can be deployed in game engines.The explicit geometry allows characters to climb, jump, and interact.Source: Figure 1 · p. 5

3 Stage I: Scene Planning

Stage I turns a text prompt into a blockout, reference image, and navmesh, using procedural generation to create a structural, traversable layout. The blockout also conditions the derived navigation mesh and reference image.

3.1 Procedural Blockout Generation

  • An LLM parses prompts into JSON parameters that configure procedural terrain generation, spatial partitioning, and hierarchical asset placement to build an intent-aligned blockout.The editable scaffold is a mesh of simple primitives; large landmarks are placed before smaller elements, and terrain smoothing helps prevent collisions.Source: §3.1 · pp. 6–7

3.2 Navmesh Extraction and Reference Image Generation

  • The navmesh is extracted from the blockout to identify exterior traversable surfaces while excluding indoor areas, constraining downstream synthesis.The reference image is generated from an isometric depth map rendered from the blockout, with an approximately 45° camera elevation and depth-scaled Gaussian perturbation to non-terrain values.Source: §3.2 · p. 7

3.3 Planning Stage Results

  • Figure 4 illustrates layouts spanning terrain types, verticality, and object density; increasing density yields more complex environments, while the procedural process guarantees navigability.Low-density layouts are open and easily traversable; medium density combines open grounds with denser activity areas.Source: Figure 4 · p. 8, §3.3 · p. 8

4 Stage II: Scene Reconstruction

Stage II generates scene meshes conditioned on a reference image and navmesh, then textures them. The reported results show improved navmesh alignment and coherent scene generation under layout edits.

4.1 Image-to-3D Base Model

  • AssetGen2 generates 3D shape latents from an image using diffusion over VecSet representations, which decode to SDFs and watertight meshes.The VecSet representation is a compact, permutation-invariant set of latent vectors (Zhang et al., 2023a); meshes are extracted from the SDF using Marching Cubes.Source: §4.1 · p. 9

4.2 Navmesh-Based Scene Generation

  • Conditioning AssetGen2 on both the reference image and navmesh produces meshes that preserve traversability and spatial coherence.The model adds navmesh tokens through cross-attention; its encoder samples and downsamples navmesh points, and end-to-end fine-tuning yields lower validation loss than training only the added cross-attention layers. Training jointly normalizes the navmesh and scene mesh for alignment.Source: §4.2 · pp. 9–10

4.3 Scene Texture Generation

  • TRELLIS supplies whole-scene textures that avoid many self-occlusion gaps, and is retrained on object- and scene-level assets.Although lower-resolution than multi-view texture generation, these textures guide later object-by-object scene enhancement.Source: §4.3 · p. 10

4.4 Scene Reconstruction Results

  • The navmesh-conditioned model produces smoother terrain that follows navigation constraints and better aligns structural regions such as buildings with the reference image.The comparison is against AssetGen2 conditioned only on the reference image, particularly in complex scenes.Source: §4.4 · p. 10
  • Navmesh-conditioned generation achieves Chamfer distances 40–50% lower than the baselines on the curated scene benchmark.The benchmark contains 50 scenes with moderately vertical terrain and 10–30 densely sampled objects; geometries are normalized, navmeshes aligned with ICP, and Chamfer distance is computed between them.Source: Table 1 · p. 12, §4.4 · p. 12
  • Navmesh edits let the model preserve structural and stylistic coherence when layout and image conditions diverge.Examples include removing a structure to add walkable space, lowering a building, and creating a shallow terrain dip.Source: Figure 7 · p. 11, §4.4 · p. 12

5 Stage III: Scene Decomposition

Stage III decomposes the monolithic scene mesh into semantically meaningful objects and parts so individual assets can be edited or refined, then enhances them in a subsequent stage. The section presents faster scene decomposition, a scene-level training dataset, and quantitative and qualitative evaluations.

5 Stage III: Scene Decomposition

  • Because the Stage II mesh fuses all scene objects into one geometry, individual assets are difficult to edit or refine; Stage III decomposes the mesh into meaningful objects and parts.The resulting parts are enhanced individually in the subsequent stage.Source: §5 · p. 12

5.1 Accelerating AutoPartGen for Scenes

  • Connectivity-ordered pivot extraction, followed by a remainder-part forward pass and connected-component analysis, uses five steps and cuts generation time from ten to about one minute.The model generates four pivot parts first, then the remainder part; residual geometry is further decomposed by connected-component analysis.Source: §5.1 · pp. 12–13

5.3 Decomposition Results

  • Our method outperforms other state-of-the-art methods across all metrics while maintaining the top inference speed.The evaluation compares predicted parts with ground-truth annotations using Chamfer Distance and F-score at various thresholds.Source: Table 2 · p. 13, §5.3 · p. 14

5.2 Scene Decomposition Data

  • To support scene-level finetuning, the authors create a compositional 3D scene dataset from assets selected as whole scenes and converted into meaningful object and part decompositions.A vision-language model screens rendered assets for object diversity, plausible layout, and visible ground context; processing detects connected components, separates ground and thin overlays, merges small parts into neighbors, and filters by part count, imbalance, and ground confidence.Source: §5.2 · p. 13

5.3 Decomposition Results

  • The method robustly separates terrain from objects across flat and mildly varying terrain, producing semantically meaningful components for subsequent enhancement and texturing.Figure 8 illustrates scene decomposition starting from the ground.Source: Figure 8 · p. 13, §5.3 · p. 14
  • Existing methods struggle with complex scenes: Top PartGen Model A is unstable, Model B fragments or merges geometry, and original AutoPartGen (Chen et al., 2025a) can miss major objects and is slow.These comparison failures are illustrated in Figure 9.Source: §5.3 · p. 14, Figure 9 · p. 14

6 Stage IV: Scene Enhancement

Stage IV enhances each Stage III object to improve visual and geometric quality while preserving consistency and composability across the scene. It generates enhanced images, refines geometry, and produces high-quality textures for the objects.

6.1 Per-Object Image Enhancement

  • A top-down scene view with the target object highlighted gives the LLM-VLM location and surrounding context to generate spatially aligned, style-consistent enhanced images.Without this view, the model struggles to produce style-consistent or reference-faithful images; an IoU verification step rejects drifting results and supports iterative refinement.Source: §6.1 · pp. 15–16, Figure 11 · p. 16

6.2 Per-Object Mesh Enhancement

  • Refined meshes are sharper, more detailed, more plausible, and contain fewer floaters and surface discontinuities than coarse inputs.The refinement model uses the enhanced image and coarse mesh as conditioning while preserving the coarse mesh’s orientation and overall shape; objects are rescaled and repositioned to restore the original scene layout.Source: §6.2 · pp. 17–18, Figure 14 · p. 19

6.3 Per-Object Texture Enhancement

  • A delighting model removes baked-in lighting cues from enhanced images, which condition sequential generation of ten views for 3D-consistent object textures.Disentangled attention supports local detail, reference-image adherence, and cross-view consistency; the generated views are back-projected to UV space and inpainted to fill gaps.Source: §6.3 · pp. 18–19

7 Results

The results show that WorldGen generates navigable, fully textured scenes from single prompts and qualitatively compare its outputs with image-to-3D methods and Marble. The comparisons identify differences in detail, scale, navigability, and game-engine compatibility.

7.2 Qualitative Comparison with Prior Work

  • Compared with single-shot image-to-3D methods, WorldGen generates more detailed scenes, while those methods lack the detail for immersive, navigable large-scale environments.The comparison notes that prior methods are non-compositional and not designed for large-scale or navigable environments.Source: Figure 18 · p. 23, §7.2 · p. 23

7.1 Examples of Generated Scenes

  • WorldGen generates semantically consistent, fully textured scenes with valid navmeshes that characters can freely explore in game engines.The full pipeline takes approximately five minutes when sufficient GPUs are available, with parallel execution of submodules enabling rapid prototyping with minimal user intervention.Source: §7.1 · p. 23

7.2 Qualitative Comparison with Prior Work

  • Marble’s fidelity degrades 3–5 m from its conditioned view, while WorldGen scenes span approximately 50 × 50 m with consistent geometry and style for free navigation and interaction.The comparison is qualitative because the systems differ in input and scope: Marble grows a scene from a specified viewpoint, while WorldGen conditions on a global reference image or full layout.Source: §7.2 · p. 23, Figure 19 · p. 24
  • WorldGen’s textured-mesh scenes integrate with standard game engines and artist toolsets, are easier to edit, and are more efficient and portable than Gaussian splats.Gaussian splats require specialized rendering pipelines and are orders of magnitude slower than reasonably optimized meshes, limiting use on mobile and low-end hardware.Source: §7.2 · pp. 23–24

8 Related Work

The related work surveys 3D scene reconstruction and scene generation, focusing mainly on static scenes while occasionally considering dynamic ones. It distinguishes monolithic, compositional, and procedural generation approaches.

8.1 Image-based Scene Reconstruction

  • Unlike prior methods that reconstruct only visible regions and need views covering the scene, WorldGen extracts a complete, structured scene representation from one image.The section notes that radiance-field methods often require hundreds of views, which are rarely available for text-prompted scenes.Source: §8.1 · pp. 24–25

8.2 Monolithic 3D Scene Generation

  • View-based monolithic generators often produce only a small walkable bubble, with stitching drift and distortions, and may prioritize views over recovering a robust 3D model.Latent-space methods instead generate scene geometry directly, but can lack diversity because varied 3D training data is difficult to collect.Source: §8.2 · p. 27

8.3 Compositional 3D Scene Generation

  • Compositional systems arrange retrieved 3D assets using learned layouts, scene graphs, or LLM-generated bounding boxes.Other methods generate objects and arrangements together, using techniques including text-to-3D initialization, diffusion, and physical constraints.Source: §8.3 · p. 28
  • Image-to-scene compositional methods reconstruct or decompose scenes from images, with some predicting object layouts, shapes, and textures together.Approaches also include segmenting background and objects before reconstruction, and jointly generating multiple objects from a single view.Source: §8.3 · pp. 28–29

8.4 Procedural 3D Scene Generation

  • Procedural scene generation uses hand-crafted procedures and rules, while newer systems use LLMs to interpret prompts, plan scenes, or write generation procedures.WorldGen uses procedural generation for an initial scene layout, then adds details automatically with an image generator.Source: §8.4 · p. 29

9 Conclusions and Limitations

WorldGen turns text prompts into traversable, interactive 3D worlds through an end-to-end pipeline, while its design limits supported scale and multi-layered environments.

9 Conclusions and Limitations

  • WorldGen generates traversable, interactive 3D worlds from text prompts by unifying layout generation, navmesh-guided synthesis, object decomposition, and geometry and texture enhancement.The pipeline produces coherent, detailed, explorable environments ready for game engines.Source: §9 · p. 29
  • WorldGen’s limitations include restricted scene scale, difficulty modeling multi-layered environments, and possible rendering inefficiency in very large scenes.Its single reference view restricts scene scale; kilometer-scale worlds would require stitching local regions, risking boundary artifacts. Single-view conditioning also limits multi-floor environments and seamless interior-exterior transitions, while independent object representations lack geometry or texture reuse.Source: §9 · p. 29
  • Future work should explore texture tiling, reuse, and shared materials to improve scalability and runtime performance.Source: §9 · p. 29
  • WorldGen illustrates how language-guided procedural reasoning and 3D diffusion models could support accessible, scalable interactive world generation for games and social experiences.Source: §9 · p. 29

Appendix · A Scenes generated by WorldGen

The appendix presents a gallery of additional scenes generated by WorldGen, spanning varied settings and styles.

A Scenes generated by WorldGen

Related papers

Loading 2511.16825v1…