Source-linked AI summary
WorldGen: From Text to Traversable and Interactive 3D Worlds
Dilin Wang, Hyunyoung Jung, Tom Monnier, Kihyuk Sohn, Chuhang Zou, Xiaoyu Xiang, Yu-Ying Yeh, Di Liu, Zixuan Huang, Thu Nguyen-Phuoc, Yuchen Fan, Sergiu Oprea, Ziyan Wang, Roman Shapovalov, Nikolaos Sarafianos, Thibault Groueix, Antoine Toisoul, Prithviraj Dhar, Xiao Chu, Minghao Chen, Geon Yeong Park, Mahima Gupta, Yassir Azziz, Rakesh Ranjan, Andrea Vedaldi
TL;DR
Creating interactive 3D worlds is complex, time-consuming, and expertise-intensive, while direct text-to-scene learning is constrained by limited large-scale 3D scene data. WorldGen combines language-guided procedural reasoning with navmesh-conditioned 3D diffusion and object-level processing to generate traversable, coherent environments; its navmesh-conditioned reconstruction achieves 40–50% lower Chamfer distances than baselines, while large open worlds remain unsupported natively.
Problem
Creating 3D content is complex, time-consuming, and resource-intensive, and no sufficiently large 3D-scene training set supports direct text-to-scene mapping.
Method
WorldGen unifies text-conditioned procedural layout generation, navmesh-guided holistic scene reconstruction, object-level decomposition, and geometry and texture enhancement into an end-to-end pipeline.
Results
WorldGen generates traversable, interactive 3D worlds from text, with navmesh-conditioned reconstruction achieving 40–50% lower Chamfer distances than baselines.
Takeaways & Limitations
WorldGen illustrates how language-guided procedural reasoning and 3D diffusion can support accessible and scalable interactive world generation for games and social experiences.
Takeaways & Limitations
WorldGen does not natively support kilometer-scale open worlds, and single-view conditioning limits multi-layered environments while independent object representation may reduce rendering efficiency in very large scenes.
Abstract
from arXiv · showhide
We introduce WorldGen, a system that enables the automatic creation of large-scale, interactive 3D worlds directly from text prompts. Our approach transforms natural language descriptions into traversable, fully textured environments that can be immediately explored or edited within standard game engines. By combining LLM-driven scene layout reasoning, procedural generation, diffusion-based 3D generation, and object-aware scene decomposition, WorldGen bridges the gap between creative intent and functional virtual spaces, allowing creators to design coherent, navigable worlds without manual modeling or specialized 3D expertise. The system is fully modular and supports fine-grained control over layout, scale, and style, producing worlds that are geometrically consistent, visually rich, and efficient to render in real time. This work represents a step towards accessible, generative world-building at scale, advancing the frontier of 3D generative AI for applications in gaming, simulation, and immersive social environments.
1 Introduction
WorldGen addresses the difficulty of generating coherent, functional 3D worlds from text by combining language-guided planning with procedural layout and staged 3D reconstruction. Its object-aware pipeline targets navigable, editable scenes while improving detail and consistency beyond holistic reconstruction alone.
- 1 Introduction: WorldGen targets the harder problem of composing coherent, functional 3D scenes from a single text prompt, not merely generating isolated objects.Objects must fit thematically, stylistically, contextually, and structurally within a navigable world.
- 1 Introduction: The system uses text-to-image and image-to-3D generation because sufficiently large datasets for direct text-to-3D-scene learning are unavailable.A generated scene plan specifies the objects, appearance, and relationships that guide reconstruction.
- 1 Introduction: Procedural generation supplies constraint-satisfying layouts, while an LLM maps natural-language prompts to its configuration parameters.The procedural generator is limited to coarse blockouts, preserving stylistic and thematic diversity for later image-based generation.
- 1 Introduction: WorldGen preserves global coherence through holistic reconstruction, then decomposes the result into constituent objects for local editing and enhancement.The decomposition addresses the mismatch between initial blockout components and objects hallucinated by the image generator.
- 1 Introduction: Per-object regeneration adds high-resolution shape and appearance details while conditioning on coarse geometry so refreshed parts continue to fit together.This stage uses new high-resolution object views and specialized geometry-and-texture reconstruction.
2 WorldGen Overview
WorldGen converts a text prompt into a structured scene plan, reconstructs it holistically under navigation constraints, decomposes it into editable objects, and enhances those objects individually. The four-stage design combines global composition and traversability with local detail and game-engine usability.
- Stage I: Scene Planning: Scene planning produces a blockout, reference image, and navmesh that jointly encode spatial layout, style, composition, and walkable structure.The blockout represents structures such as open spaces, loops, and chokepoints; depth conditioning generates the reference image.
- Stage II: Scene Reconstruction: Holistic reconstruction generates a single textured mesh conditioned on the reference image and navmesh, resolving global relationships while preserving traversable connectivity.Navmesh constraints reduce non-reachable regions and other spatial artifacts.
- Stage III: Scene Decomposition: Scene decomposition separates the holistic mesh into low-resolution assets such as terrain, buildings, trees, and props, enabling piece-by-piece enhancement and local editing.Local edits can target individual objects without regenerating the entire world.
- Stage IV: Scene Enhancement: Object enhancement generates detailed views, refines each object’s geometry, and applies aligned high-resolution textures while retaining coarse geometric alignment.The refined parts remain consistent with the initial reconstruction so they continue to fit together.
- System Output: WorldGen outputs individually editable, fully textured meshes with explicit geometry that supports collision, navigation, climbing, jumping, and interaction.The resulting assets are intended for immediate deployment in game engines.
- 2 WorldGen Overview: WorldGen’s four stages progress from scene planning to holistic reconstruction, object decomposition, and high-resolution scene enhancement.The pipeline begins with high-level structure and ends with scene components and low-level details.
3 Stage I: Scene Planning
Stage I converts a text prompt into a rough but functionally correct scene plan consisting of a procedural blockout, reference image, and navmesh. The blockout uses language-controlled procedural generation to establish coherent, navigable structure while leaving semantic details to image generation.
- Scene Planning: The planning output is L = (B, R, S): a blockout layout, depth-conditioned reference image, and navigational mesh.The navmesh is extracted from the blockout, while the reference image is generated by conditioning an image generator on the blockout’s rendered depth map.
- Scene Planning: Text-conditioned procedural generation produces a blockout that encodes global spatial coherence, scene structure, and navigability.An LLM maps the prompt to procedural-generation parameters, while the resulting blockout provides the structural scaffold for downstream synthesis.
- Procedural Blockout Generation: The procedural pipeline builds terrain, partitions space into functional regions, and hierarchically places hero, medium-scale, and decorative assets.Terrain controls elevation and slopes; partitioning creates open, clustered, or transitional regions; hierarchical placement fills the layout while preserving spatial organization.
- Procedural Blockout Generation: The blockout uses simple primitives as an editable scaffold, while the image generator determines the semantic identities of its placeholder volumes.This separates structural layout and navigability from the later visual interpretation of scene elements.
- Planning Stage Results: Increasing density produces progressively richer and more complex layouts, while the procedural process guarantees navigability across the generated area.Figure 4 varies density and verticality, with low-density layouts remaining open and easily traversable.
4 Stage II: Scene Reconstruction
Stage II reconstructs a scene mesh from the planned reference image and navmesh, using navmesh-conditioned image-to-3D generation to preserve global structure and traversability. Quantitative and qualitative results show improved alignment with navigation constraints, including under editable layout changes.
- Scene Reconstruction: Stage II generates a holistic textured mesh conditioned jointly on the reference image and navmesh to align appearance with navigable structure.The holistic reconstruction resolves global spatial relationships, while the navmesh provides structural guidance for regions that images may occlude.
- Scene Reconstruction: The reconstruction architecture extends AssetGen2 by encoding navmesh points and combining navmesh and image features through cross-attention.The navmesh encoder samples and downsamples surface points before embedding them for cross-attention with the image-conditioned diffusion model.
- Scene Reconstruction Results: The navmesh-conditioned model produces smoother terrain that adheres more closely to navigation constraints and improves structural alignment in complex scenes.Compared with AssetGen2 conditioned only on the reference image, the model better preserves terrain traversability and aligns buildings with the reference composition.
- Scene Reconstruction Results: Navmesh-conditioned reconstruction achieves Chamfer distances 40–50% lower than the baselines on a 50-scene benchmark.The benchmark contains procedurally generated scenes with moderate verticality and 10–30 densely sampled objects; lower Chamfer distance indicates stronger navmesh alignment.
- Scene Reconstruction Results: Layout edits remain effective when navmesh and reference-image conditions are slightly misaligned, preserving spatial and stylistic coherence.Examples include removing a structure, lowering a building, and forming a shallow terrain dip while retaining the intended navigability and scene appearance.
5 Stage III: Scene Decomposition
Stage III decomposes the coarse monolithic scene mesh into semantically meaningful parts so individual objects can be enhanced, textured, edited, and composed. The method accelerates scene-scale decomposition while preserving robust separation of terrain and objects.
- Scene decomposition: The scene mesh is decomposed into semantically meaningful objects and parts because blockout components may not correspond one-to-one with hallucinated scene geometry.The decomposition operates on the holistic mesh and extracts meaningful constituents for subsequent refinement.
- Decomposition evaluation: The model achieves significantly better results than previous state-of-the-art methods across all scene-decomposition metrics.Table 2 reports the quantitative comparison, while Figure 9 contrasts decomposition quality and inference speed against competing methods.
- Accelerating AutoPartGen for Scenes: Generation time falls from ten minutes to about one minute through connectivity-prioritized pivot extraction and remainder-geometry decomposition.The schedule generates four pivot parts followed by a remainder part, which is further split by connected-component analysis.
- Training data: Scene-level fine-tuning requires annotated data, so the authors create a compositional 3D-scene dataset from mined and processed scene assets.The processing combines connectivity-based splitting with ground-aware reasoning and other heuristics.
- Decomposition results: Across flat and mildly elevated environments, the method cleanly separates ground terrain and segments objects into meaningful components for later enhancement and texturing.Existing methods often produce unstable decompositions, fragment buildings, merge ground with objects, or fail to decompose major objects.
6 Stage IV: Scene Enhancement
Stage IV enhances decomposed objects independently while preserving their alignment and consistency with the global scene. It combines context-aware image enhancement, mesh refinement, texture synthesis, and verification to produce higher-quality editable assets.
- Stage IV pipeline: Each object is enhanced by generating a high-resolution image, refining geometry from that image and coarse shape, and synthesizing a high-quality texture.The mesh enhancer adds detail while constraining deviation from the initial geometry so refreshed parts still fit together.
- Per-object image enhancement: The LLM-VLM uses the global reference, a top-down scene view with the target highlighted, and the object render to produce spatially aligned, style-consistent detail.The top-down view supplies object location, semantics, surrounding context, material attributes, and color information.
- Per-object image enhancement: Removing the top-down view makes the LLM-VLM struggle to generate style-consistent and reference-faithful object images.The ablation identifies scene location and surrounding context as essential conditioning information.
- Verification: An IoU-based verification module rejects enhancements with insufficient foreground overlap and supports iterative refinement against geometric or stylistic drift.The verifier compares the original coarse render with the enhanced render and accepts results only above a threshold.
- Mesh refinement: The mesh refinement model preserves coarse-object orientation while adding fine geometric detail, producing sharper meshes with fewer floaters and surface discontinuities.The refined meshes are reported as suitable for subsequent high-quality texture synthesis.
7 Results
WorldGen produces diverse, textured, navigable scenes from single prompts and supports exploration in game-engine settings. Compared with image-to-3D and view-based alternatives, it emphasizes large-scale compositional consistency and editable textured meshes.
- Comparison with image-to-3D: Compared with single-shot image-to-3D methods, WorldGen generates substantially more detailed scenes than reconstructions designed for single objects or small-scale environments.The cited comparison highlights the difference in scene detail and scale rather than reporting a numeric benchmark.
- Generated worlds: WorldGen generates diverse, fully textured scenes with semantically consistent objects, coherent textures, valid navmeshes, and free character navigation from single prompts.The generated worlds are described as suitable for real-time exploration and use in game engines with minimal user intervention.
- Comparison scope: Direct comparisons with prior immersive-environment systems are difficult because those systems differ substantially in assumptions and generation scope.The paper presents qualitative comparisons rather than a common quantitative evaluation across all systems.
- Comparison with Marble: WorldGen scenes span approximately 50 × 50 m and maintain geometric and stylistic consistency while users freely navigate and interact within the environment.The comparison reports that Marble fidelity degrades after the camera moves roughly 3–5 m from its conditioned view.
- Comparison with Marble: WorldGen produces textured-mesh compositions compatible with Unreal and Unity, whereas Gaussian splats require specialized rendering pipelines and are not natively supported by those engines.Object composition also supports editing, moving, and replacing scene elements.
8 Related Work
The related work is organized around reconstruction, monolithic scene generation, compositional generation, and procedural generation. The paper positions WorldGen as a single-image-to-structured-scene system that combines large-scale layout, object decomposition, and navigable mesh output.
- Scene reconstruction: Radiance-field methods can represent complex scenes with high fidelity, but scene components are often represented indistinctly as one whole.The paper reviews NeRF- and 3DGS-based reconstruction families, including methods using multiple views or estimated camera poses.
- Image-based scene reconstruction: Single- or few-view reconstruction methods often recover only visible scene content, whereas WorldGen extracts a complete structured scene representation from one input image.The paper identifies complete coverage of all scene aspects as a requirement for complete reconstruction.
- Monolithic 3D scene generation: View-based scene generators commonly produce walkable bubbles only a few meters across, and incremental stitching can accumulate inconsistencies and distortions.Some methods prioritize novel views rather than recovering an underlying 3D model.
9 Conclusions and Limitations
WorldGen generates traversable, interactive 3D worlds from text by combining procedural layout, navmesh-guided synthesis, object decomposition, and geometry and texture enhancement. The system demonstrates accessible world generation but remains limited in scale by single-view conditioning and independent object representation.
- WorldGen unifies text-conditioned procedural layout, navmesh-guided synthesis, object-level decomposition, and geometry and texture enhancement into game-engine-ready environments.
- WorldGen’s results demonstrate the potential of text-driven world generation while leaving several limitations unresolved.
- Single-reference-view conditioning restricts scene scale, excludes large kilometer-scale open worlds natively, and complicates multi-layered environments.Larger worlds would require stitching multiple local regions, risking non-smooth transitions or boundary artifacts.
- Independent object representations may reduce rendering efficiency in very large scenes because geometry and textures are not reused.The paper suggests texture tiling, reuse, and shared materials as future scalability strategies.
- The system generates coherent, detailed, explorable worlds that support interactive use in games and social experiences.
A Scenes generated by WorldGen
The appendix presents a gallery of WorldGen scenes spanning many settings, including space ports, villages, temples, industrial areas, towns, and outposts. These examples illustrate the breadth of generated environments across distinct themes and styles.
- The gallery includes diverse settings such as a space port, fruit-themed village, sci-fi colony, industrial dockyard, and steampunk miniature city.
- Additional generated environments include ancient temples, military and forest outposts, medieval towns, mushroom villages, and cargo yards.
- The gallery also covers desert, futuristic industrial, suburban, seaside, and Halloween-themed environments.