UrbanWorld2.0: A Multimodal Agentic Framework for Reality-Aligned 3D World Generation at City-Scale

1 Introduction

• Automated 3D world generation matters for immersive media, simulation, embodied intelligence, and world models, but complex scene creation remains labor-intensive, costly, and time-consuming.Reality-aligned urban worlds are relevant to urban simulation, autonomous driving, and embodied AI, and can reduce sim-to-real gaps and domain-adaptation costs.Source: §1 · p. 2
• Existing approaches face computational costs at urban scale, limited or text-only inputs, imperfect real-world data, and difficulty aligning diverse elements to real-city layouts [8] [10] [36].Cities combine buildings, roads, and green spaces, while many methods lack real-world layout retrieval and object alignment.Source: §1 · p. 2
• UrbanWorld2.0 is a training-free, modular multimodal agentic framework that curates real-world data, uses geospatial information to guide element placement, and iteratively reflects on and refines generated objects.The authors report that modularity improves controllability and parallelism, and that the mesh-based representation is compatible with established computer-graphics pipelines.Source: §1 · p. 2
• The framework also models environmental details, including urban elements and traffic patterns, to increase the realism, fidelity, and utility of generated worlds.Its code and generated 3D assets are open-sourced for community use.Source: §1 · p. 2
• UrbanWorld2.0 achieves reported state-of-the-art performance in urban 3D world generation across shape precision, texture fidelity, reality alignment, consistency, and compatibility with standard CG pipelines.The framework and generated assets are open-sourced for continued optimization and broader application.Source: §1 · p. 2

2.1 Overview

• The pipeline proceeds through Planning, Perception, Imagination, Reflection, 3D Gen, and Scene Design.Planning decomposes tasks; Perception processes visual and geographic data; Imagination refines intermediate representations; Reflection performs iterative critique and refinement; 3D Gen calls asset-generation tools; Scene Design assembles the world.Source: §2.1 · p. 3
• The Perception and Imagination stages establish and refine intermediate representations before asset generation and scene assembly.This sequence is described as the framework's organization of the full pipeline.Source: §2.1 · p. 3
• Scene Design aligns all 3D elements with reality and simulates object placement, road networks, and traffic.A suite of multimodal tools is used throughout the pipeline.Source: §2.1 · p. 3

2.2 Task Planning

• Task Planning divides generation from geographic coordinates into Perception, Imagination, Reflection, 3D Gen, and Scene Design.The modular architecture disentangles heterogeneous subtasks and is described as enhancing flexibility and controllability.Source: §2.2 · p. 3
• During Perception and Reflection, the planner generates control signals in response to integrated-tool feedback.For example, object-detection confidence scores inform image decisions, while a quality evaluator determines whether an imagined image proceeds.Source: §2.2 · p. 4
• Perception-stage object candidates and confidence scores provide a quantitative basis for decisions about input images.Reflection similarly uses quality evaluation to decide whether an imagined image is qualified for later procedures.Source: §2.2 · p. 4

2.3 Elements Perception

• The system obtains building and other urban-element details from OpenStreetMap and corresponding street-view panoramas through an Online Map API.These are the two stated sources of real-world information.Source: §2.3 · p. 4
• Because panoramas may contain irrelevant vegetation, construction, and vehicles, the agent removes obstacles and segments building regions using an object-detection model.Street-view selection follows image processing.Source: §2.3 · p. 4
• The agent uses a model's judgment to select the most suitable processed image for subsequent steps.Source: §2.3 · p. 4

2.4 Imagination and Reflection

• Street-view viewpoints omit full 3D volume, while transient occluders hide building features, constraining accurate reconstruction.Source: §2.4.1 · p. 4
• OSM-derived spatial spans and rough geometry join street-view images as inputs to the agent’s holistic building imagination.Source: §2.4.1 · p. 4
• A VLM critic scores initial building representations and iteratively directs regeneration until quality meets a threshold or attempts run out.Deeper checks assess semantic plausibility, structural integrity, and aesthetics; the agent supplies weakness reports and improvement guidance.Source: §2.4.2 · p. 4

2.5 3D Building Assets Generation

• Hunyuan-3D-Dit-v2.1 generates an untextured 3D mesh from the refined 2D building image.Source: §2.5 · p. 4
• Hunyuan3D-Paint-v2.1 synthesizes a texture map for the generated base shape.Source: §2.5 · p. 4
• Post-processing removes artifacts such as unwanted ground planes and outlying fragments before the asset enters scene design.Source: §2.5 · p. 4

2.6 3D Scene Design

• Map data systematically positions high-quality, pre-generated building models to address coherent spatial arrangement at city scale.Source: §2.6.1 · p. 4

2.6.1 Real-world Aligning.

• OSM metadata supplies a schematic scaffold; buildings inherit reference positions, scale by volume ratio, and rotate for maximum footprint overlap.Roads, vegetation, and water bodies are also placed from metadata locations.Source: §2.6.1 · p. 5

2.6.2 Fine-Grained Object Placement.

• Retrieval-based assets populate repetitive street objects, placed using OSM road anchors and VLM interpretation of geotagged imagery and tags.The VLM infers contextual categories and likely locations, such as traffic lights near intersections or crossings.Source: §2.6.2 · p. 5

2.6.3 Road Network and Traffic Modeling.

• MOSS refines OSM road data into lane-level topology and connectivity, supporting realistic road layouts populated with city-scale human and vehicle traffic.Source: §2.6.3 · p. 5

3 Experiments

• The evaluation combines quantitative metrics, qualitative demonstrations, and agent-design comparisons, applying identical regions, settings, and protocols across methods.Layout and street-level quality are evaluated as complementary perspectives.Source: §3 · p. 5, §3.1 · p. 5

3.1 Quantitative 3D World Evaluation

• UrbanWorld2.0 significantly surpasses all competitors in street-level assessments and meets or exceeds most OSM-geometry-constrained baselines in layout alignment.Table 2 reports the layout and street-level comparisons.Source: §3.1 · p. 5, Table 2 · p. 6

3.2 Qualitative 3D World Evaluation

• Across methods, UrbanWorld2.0 improves building precision, texture fidelity, and layout reasonableness, while the compared baselines show limitations in realism, geometry, or spatial relationships.SGAM has poor shape fidelity and low-resolution textures; Syncity has inconsistent realism and boundary discontinuities; CityCraft neglects roads and spatial relationships, while Urbanworld uses coarse cuboids and low-quality textures.Source: §3.2 · pp. 6–7

3.3 Reflection Mechanism Evaluation

• Approximately 6% of samples score below 3 in at least one quality dimension, triggering regeneration; reflection then mitigates hallucinations, structural anomalies, and implausible textures.The refinement restores clean building envelopes, architecturally sound layouts, and more realistic facades.Source: §3.3 · p. 7

3.4 Autonomous Agent’s Decision Evaluation

• Autonomously agent-selected street-view inputs achieve the best 2D FID and KID and 3D texture FID; multi-source methods have the two highest SSIM scores but underperform street-view-only methods on perceptual metrics.Source: §3.4 · p. 8

3.5 Downstream Application

• The framework supports building outdoor spatial reasoning and navigation tasks, as well as city-scale traffic simulation for any region.Source: §3.5 · p. 8

• Procedural methods offer scalability and control but require extensive human intervention; neural 3D methods face limited training data and scalability, while visual methods often lack geometric fidelity, view consistency, and CG-pipeline compatibility.Source: §4 · p. 8
• UrbanWorld2.0 is based on real-world geospatial data and is presented as enabling scalable urban generation with improved reality alignment, photorealism, and view consistency.Source: §4 · p. 8
• Agent-based 3D synthesis spans object-level asset creation and editing and scene-level spatial layout and world construction, using LLM planning, tool calls, and outcome checks.Source: §4 · p. 9
• Unifying object- and scene-level reasoning within one agentic framework remains an important open direction.Source: §4 · p. 9

• An intelligent agent plans data and control flow, uses multimodal tools, and applies self-reflection for iterative refinement in city-scale 3D world generation.Source: §5 · p. 9
• UrbanWorld2.0 outperforms existing 3D urban scene generation methods across reality alignment, scene quality, view consistency, scalability, and CG-pipeline compatibility.Source: §5 · p. 9
• The framework has potential applications in embodied intelligence and world models.Source: §5 · p. 9