Source-linked AI summary
UrbanWorld: An Urban World Model for 3D City Generation
Yu Shang, Yuming Lin, Yu Zheng, Hangyu Fan, Jingtao Ding, Jie Feng, Jiansheng Chen, Li Tian, Yong Li
TL;DR
Creating realistic, interactive urban environments is difficult and labor-intensive, while existing approaches do not fully provide the controllable embodied environments needed for agent research. UrbanWorld combines layout generation, MLLM scene design, diffusion-based asset rendering, and MLLM-assisted refinement; it is reported to outperform existing city-generation methods, with a 5.4% improvement on FID over the most competitive baseline.
Problem
Creating detailed 3D urban environments requires substantial designer labor, and existing approaches do not fully meet the requirements for realistic, interactive, controllable environments that support embodied agent learning.
Source: §1 · pp. 1–2Method
UrbanWorld builds urban layouts from map or semantic and depth-map inputs, plans scenes with an urban MLLM, renders controllable asset textures with diffusion, and refines scenes with MLLM suggestions.
Source: §1 · p. 2Results
UrbanWorld outperforms existing 3D city-generation methods; against the most competitive baseline, it improves FID by 5.4%.
Source: §4.2 · p. 8, §5 · p. 10Takeaways & Limitations
Generated environments are demonstrated for agent perception and navigation and are offered as an open-source platform for research including embodied intelligence and AI agents.
Source: §1 · pp. 2–3
Abstract
from arXiv · showhide
Cities, as the essential environment of human life, encompass diverse physical elements such as buildings, roads and vegetation, which continuously interact with dynamic entities like people and vehicles. Crafting realistic, interactive 3D urban environments is essential for nurturing AGI systems and constructing AI agents capable of perceiving, decision-making, and acting like humans in real-world environments. However, creating high-fidelity 3D urban environments usually entails extensive manual labor from designers, involving intricate detailing and representation of complex urban elements. Therefore, accomplishing this automatically remains a longstanding challenge. Toward this problem, we propose UrbanWorld, the first generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions. UrbanWorld incorporates four key stages in the generation pipeline: flexible 3D layout generation from OSM data or urban layout with semantic and height maps, urban scene design with Urban MLLM, controllable urban asset rendering via progressive 3D diffusion, and MLLM-assisted scene refinement. We conduct extensive quantitative analysis on five visual metrics, demonstrating that UrbanWorld achieves SOTA generation realism. Next, we provide qualitative results about the controllable generation capabilities of UrbanWorld using both textual and image-based prompts. Lastly, we verify the interactive nature of these environments by showcasing the agent perception and navigation within the created environments. We contribute UrbanWorld as an open-source tool available at https://github.com/Urban-World/UrbanWorld.
1 Introduction
UrbanWorld is proposed to address the gap between labor-intensive or video-only city generation and realistic, controllable, interactive urban environments that support embodied agents. It combines layout generation, MLLM scene design, diffusion-based asset rendering, and MLLM-assisted refinement, with evaluations and demonstrations of realism, controllability, and interaction.
1 Introduction
- Existing approaches leave a gap: generated 3D scenes may be video-only, while urban environments still fall short of realistic, interactive, controllable settings for embodied learning.The paper also identifies substantial designer labor for detailed asset modeling, texture mapping, and scene composition.Source: §1 · pp. 1–2
- UrbanWorld generates 3D layouts from urban data, uses Urban MLLM for scene planning, renders assets with controllable texture diffusion, and refines scenes with MLLM suggestions.Inputs include OpenStreetMap data and layouts with semantic and depth maps; text and visual conditions control rendering.Source: §1 · p. 2
- The paper reports evaluation on five visual metrics, diverse results from text and image prompts, and agent perception and navigation demonstrations in generated environments.UrbanWorld is presented as an open platform for research including embodied intelligence and AI agents.Source: §1 · pp. 2–3
- UrbanWorld is reported to produce high-fidelity urban environments that enhance the authenticity of interactions.The authors describe the platform as supporting realistic, customized, interactive 3D environments with flexible controls.Source: §1 · p. 3
2 Related Works
Prior urban-scene generation methods trade off visual quality, geometric fidelity, interactivity, and the ability to create new assets. UrbanWorld is positioned as a controllable approach to diverse embodied urban environments.
2.1 3D Urban Scene Generation
- Neural-rendering methods can achieve high visual quality but may lose geometric fidelity, while diffusion approaches create scene images or videos rather than embodied 3D environments.Software-script methods such as CityCraft use existing asset libraries and cannot flexibly create new assets when needed.Source: §2.1 · p. 3
2.2 3D World Simulator
- Prior urban simulators include video-based autonomous-driving models and platforms with limited implementation or fixed styles; UrbanWorld targets diverse embodied scenes with controllable, refined appearance.The framework figure depicts layout generation, MLLM-empowered scene design, diffusion-based texture rendering, and MLLM-assisted refinement.Source: §2.2 · pp. 3–4, Figure 1 · p. 4
3 Methodology
UrbanWorld uses a map-design-render-refine pipeline to generate urban environments from layout data, combining an urban MLLM with an asset-rendering module. Its stages cover layout construction, scene design, controllable texture rendering, and iterative refinement.
3 Methodology
- UrbanWorld coordinates four stages—layout generation, MLLM-empowered scene design, controllable diffusion-based texture rendering, and MLLM-assisted refinement—to address construction, planning, and texture-quality challenges.The stages form a “map-design-render-refine” pipeline.Source: §3 · p. 4
3.1 Flexible 3D Urban Layout Generation
- OSM or other 2.5D layout metadata produces an untextured 3D scene, with urban assets separated and their locations recorded for layout reconstruction.Supported inputs include layout data describing roads, buildings, vegetation, forests, and water.Source: §3.1 · pp. 4–5
3.2 Urban MLLM-empowered Scene Design
- Urban MLLM, fine-tuned on curated urban street-view imagery, converts user instructions into detailed asset descriptions that guide later texture rendering.The training set comprises approximately 300K urban street-view images with GPT-4-generated descriptions that were manually reviewed and filtered.Source: §3.2 · p. 5
3.3 Controllable Diffusion-based Urban Asset Texture Rendering
- For each asset, depth-aware diffusion generates multi-view appearances from text or image prompts, which are back-projected and merged into a UV texture map.The renderer uses an element-wise strategy and groups assets into buildings, roads and paths, forest and vegetation, and water.Source: §3.3 · pp. 5–6
- UV position-map-guided inpainting fills texture gaps caused by discrete view sampling, while ControlNet-tile upscaling sharpens texture structure and realism.The position map encodes adjacency relations among UV texture fragments.Source: §3.3 · p. 6
3.4 MLLM-assisted Urban Scene Refinement
- After rendering, Urban MLLM identifies inconsistencies with prompts and proposes revised design prompts, enabling iterative refinement of low-quality textures.The refined assets are rendered again and updated in the reconstructed 3D environment; Figure 3 depicts untextured, initial textured, and refined scenes.Source: Figure 3 · p. 6, §3.4 · p. 7
4 Experiments
The experiments evaluate UrbanWorld’s implementation, quantitative performance, controllable generation, and agent interaction. Results include improvements over baselines across quantitative metrics, ablation findings, and demonstrations of text- and image-conditioned generation and navigation.
4.1 Implementation Details
- UrbanWorld uses Blender and Blosm for OSM transformation, Stable Diffusion 1.5 with depth ControlNet for multi-view assets, and IP-Adapter for reference-image conditioning.The implementation uses four camera views and 30 inference steps by default, with experiments run on one NVIDIA Tesla A100 GPU.Source: §4.1 · pp. 7–8
4.2 Quantitative Evaluations
- UrbanWorld improves on every reported quantitative metric: 5.4% on FID, 26.7% on KID, 44.2% on depth error, and 10.4% on homogeneity versus the most competitive baseline; GPT-4 preference also exceeds all compared methods.The authors report stronger results with real street-view image conditioning than with text-only conditioning.Source: Table 2 · p. 8, §4.2 · p. 8
- Evaluation compares text- and image-conditioned UrbanWorld with SGAM, PersistentNature, SceneDreamer, and Citydreamer across five metrics; several other methods are omitted because their code is unavailable.The five metrics are FID, KID, depth error, homogeneity index, and preference score.Source: §4.2 · p. 8
- Urban MLLM scene design contributes most to quality, texture enhancement most notably affects depth error, and final scene refinement improves all evaluation metrics.The ablations cover both text-prompt and real-street-view image conditions.Source: §4.2 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 9
4.3 Qualitative Results
- UrbanWorld generates diverse, instruction-controlled urban elements and image-guided buildings with high fidelity, alongside visually consistent, contextually appropriate functional scenes.The examples include text-guided residential areas, commercial blocks, and parks, plus buildings generated from reference images.Source: §4.3 · pp. 9–10, Figure 5 · p. 9, Figure 6 · p. 9
4.4 Interactivity Demonstration
- Generated environments provide RGB, depth, and semantic observations for agents, and an RRT planner demonstrates navigation to target coordinates around 3D obstacles.The authors record the agent’s visual observation trajectory during navigation.Source: §4.4 · p. 10, Figure 7 · p. 10
5 Conclusion
UrbanWorld is presented as an automatic generative model for realistic, customized, interactive 3D urban environments with flexible controls. The authors report that it outperforms existing 3D city-generation methods and offer it as an open-source tool for research communities.
5 Conclusion
- UrbanWorld combines urban MLLM understanding with controllable diffusion generation to create high-fidelity, customized urban environments that outperform existing 3D city-generation methods.The model is described as automatic, interactive, and controllable through flexible conditions.Source: §5 · p. 10
- The authors position the generated environments as high-fidelity data and interactive settings for developing embodied intelligence and AI agents.Source: §5 · pp. 10–11
- UrbanWorld is contributed as an open-source tool for research communities.Source: §5 · p. 11
A.1 Details of the metrics for quantitative evaluation
The appendix defines five evaluation metrics for generated urban scenes, covering image realism, geometry, diversity, and preference.
A.1 Details of the metrics for quantitative evaluation
- FID and KID compare generated-frame image distributions with 1,000 real street-view images; lower values indicate better image quality.The reference images are randomly sampled from Google Maps.Source: §A.1 · p. 14
- Depth error is the L2 distance between normalized predicted depth maps and density-accumulation-derived ground-truth maps, averaged over 100 captured frames.The predicted depths come from a pretrained depth-estimation model.Source: §A.1 · p. 14
- Homogeneity index is the average pairwise cosine similarity of ResNet scene features; smaller values indicate more diverse generated environments.Source: §A.1 · p. 14
- Preference score uses GPT-4 to rate snapshots from 1 to 10 based mainly on texture sophistication and geometric completeness; higher scores indicate stronger preference.Source: §A.1 · p. 14
A.2 Designed Prompts for urban scene design using Urban MLLM
The appendix includes a designed prompt template for Urban MLLM-based urban scene design.
A.2 Designed Prompts for urban scene design using Urban MLLM
- Figure 8 presents a designed prompt template for urban scene design using Urban MLLM.Source: Figure 8 · p. 14