Source-linked AI summary
CityCraft: A Real Crafter for 3D City Generation
Jie Deng, Wenhao Chai, Junsheng Huang, Zhonghan Zhao, Qixuan Huang, Mingyan Gao, Jianshu Guo, Shengyu Hao, Wenhao Hu, Jenq-Neng Hwang, Xi Li, Gaoang Wang
TL;DR
Existing city-generation approaches are described as lacking diversity, flexibility, and planning, limiting the range of realistic and controllable scenes. CityCraft combines controllable layout generation, LLM-based planning, and 3D scene construction; it reports improvements over traditional methods, including a lower FID than CityGen.
Problem
Prior city-generation methods often lack diversity and flexibility, and the paper asks how to generate infinitely scalable, detailed, controllable 3D city scenes efficiently.
Source: §1 · pp. 1–2Method
CityCraft combines DiT-generated city layouts with LLM-based urban planning and asset retrieval for 3D scene construction.
Source: §1 · p. 2, §3.2 · pp. 4–5Results
CityCraft reports improvements over traditional methods in diversity, controllability, and visual appeal; its FID is 27.60 versus CityGen's 88.38.
Source: §5.2 · p. 6, §6 · p. 9Takeaways & Limitations
CityCraft's reported control and diversity support applications requiring high detail and customization.
Source: §1 · p. 3
Abstract
from arXiv · showhide
City scene generation has gained significant attention in autonomous driving, smart city development, and traffic simulation. It helps enhance infrastructure planning and monitoring solutions. Existing methods have employed a two-stage process involving city layout generation, typically using Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers, followed by neural rendering. These techniques often exhibit limited diversity and noticeable artifacts in the rendered city scenes. The rendered scenes lack variety, resembling the training images, resulting in monotonous styles. Additionally, these methods lack planning capabilities, leading to less realistic generated scenes. In this paper, we introduce CityCraft, an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Subsequently, a Large Language Model(LLM) is utilized to strategically make land-use plans within these layouts based on user prompts and language guidelines. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction. Furthermore, we contribute two new datasets to the field: 1)CityCraft-OSM dataset including 2D semantic layouts of urban areas, corresponding satellite images, and detailed annotations. 2) CityCraft-Buildings dataset, featuring thousands of diverse, high-quality 3D building assets. CityCraft achieves state-of-the-art performance in generating realistic 3D cities.
1 Introduction
CityCraft addresses limited diversity and control in 3D city generation by combining controllable 2D layout generation, LLM-based urban planning, and 3D scene construction. It also contributes datasets for layout generation and city planning.
1 Introduction
- Prior city-generation methods lack diversity and flexibility, while realistic, controllable urban scenes remain a central challenge.The paper asks how to generate infinitely scalable, detailed, and controllable 3D city scenes efficiently.Source: §1 · pp. 1–2
- Urban-scale text-to-3D generation faces architectural and topology complexity, and prior LLM-assisted tools have limits in fine-grained editing or procedural-resource use [69].The paper identifies SceneCraft's difficulty with fine-grained 3D-asset editing and 3D-GPT's limits in using procedural-generation resources for large-scale urban scenes.Source: §1 · p. 2
- CityCraft combines DiT layout generation, LLM urban planning, and asset integration with Blender to construct 3D city scenes.The layout generator accepts class-ratio and text controls; the LLM develops plans from text prompts.Source: §1 · p. 2
- CityCraft is presented as providing increased control and diversity for applications requiring high detail and customization.Source: §1 · p. 3
- The paper contributes CityCraft-OSM and CityCraft-Buildings datasets to support more realistic and varied city-environment creation.Source: §1 · p. 3
2 Related Works
Related work covers diffusion-model development, LLM-based scene planning, and methods for generating city scenes. It describes limitations in dataset variety, physical plausibility, layout complexity, and computational cost.
2.1 Diffusion Models
- Diffusion models have succeeded across images, 3D objects, and videos, with transformer networks proposed as denoising backbones [32] [53].Source: §2.1 · p. 3
2.2 Scene Planning with Large Language Models
- LLMs can generate scene layouts, but direct numerical outputs may overlap assets; CityCraft uses semantic-layout constraints and a solver for physically plausible arrangements.Source: §2.2 · p. 3
2.3 City Scene Generation
- City-scene methods range from rule-based and procedural tools to learned approaches, while diffusion can simplify layouts and NeRF rendering is computationally expensive.The passage also notes that modularity remains a challenge for integrations of vision-language models and depth prediction.Source: §2.3 · p. 3
3 Methods
CityCraft generates controllable 2D semantic layouts, plans urban-object uses with an LLM, and matches plans to assets for 3D scene construction. Its layout generation supports class-ratio and text controls, while planning iteratively uses instance and neighborhood information.
3.1 City Layout Generation
- Class-ratio and text controls guide DiT layout generation, while BlendedDiffusion outpainting iteratively expands layouts into larger city scenes.Text captions are generated from corresponding satellite images with GPT-4V; the controls are incorporated into DiT blocks through adaptive layer normalization.Source: §3.1 · p. 4
3.2 City Planning
- For each layout instance, the LLM plans urban-object allocation from information such as size, location, neighbors, and distance to roads.Plans can specify a building’s primary and secondary functions, size, architectural style, and the planner’s reasoning.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
- CityCraft-OSM is described as having more classes and greater total area than CityDreamer-OSM, and includes satellite images and text annotations.The caption presents these additions as supporting more detailed urban planning and analysis.Source: Table 1 · p. 5
- Asset retrieval first narrows candidates by requirements such as function and scale, then selects the highest-scoring match to each plan.Similarity combines weighted pairwise image- or text-embedding comparisons; plans are revisited over several rounds as local and global information changes.Source: §3.2 · p. 5
- Blender places selected assets on the 2D layout using optimized scale and rotation, while planned textures and added features complete the 3D scene.The Powell algorithm is used to find placement scale and rotation factors; added features include trees and streetlights.Source: §3.2 · p. 5
4 Datasets
The paper introduces CityCraft-OSM for layout-model training and CityCraft-Buildings for planning and 3D city crafting. The datasets provide real-world semantic layouts with satellite imagery and annotations, alongside varied 3D building assets.
CityCraft-OSM
- CityCraft-OSM contains over 100,000 768 × 768 patches at 0.5-meter pixel distance from North American, European, and Asian cities.Its seven semantic classes are terrain, vegetation, water, building, traffic road, rail, and footpath.Source: §4 · p. 5
- CityCraft-OSM provides class ratios and corresponding satellite images, with a subset of images annotated using GPT4-V and GeoChat with human corrections.It is built upon OpenStreetMap and is used to train the city-layout generation model.Source: §4 · p. 5
CityCraft-Buildings
- CityCraft-Buildings comprises 2,000 online-sourced 3D building assets spanning diverse functions, scales, and styles.Each asset has renders from 12 angles and annotations describing multiple features; the dataset supports planning and 3D city crafting.Source: §4 · p. 5
Dataset comparison
- Table 2 compares layout and scene generation; its caption identifies CityGen-2 as substantially outperforming other methods and names FID, KID, DE, and CE as metrics.The caption specifies that lower FID and KID are better, and describes DE and CE as depth and camera errors.Source: Table 2 · p. 6
5 Experiments
CityCraft is evaluated on semantic-mask city layouts, generated 3D scenes, and controls for layout and planning. The reported comparisons favor CityCraft on layout metrics and scene consistency, while ablations examine user control and iterative planning.
5.1 Experiment Setups
- Experiments use DiT-B/2 for layout generation, GPT-4 Vision for annotations, and GPT-4o and GPT-4 for planning; layout comparisons focus on semantic-mask methods.The authors exclude methods using object bounding boxes because they constrain represented object types and shapes.Source: §5.1 · p. 6, §5.2 · p. 6
5.2 Results on City Layout Generation
- CityCraft achieves FID 27.60 and KID 0.022, versus CityGen’s FID 88.38 and KID 0.089, and its layouts show more varied styles and complex roads.The qualitative comparison also describes CityDreamer’s repetitive features and CityGen’s limited architectural diversity.Source: §5.2 · p. 6, Figure 3 · p. 7
5.3 Results on City Scene Generation.
- CityCraft reports Depth Error (DE) 0 and Camera Error (CE) 0, while rendering-based methods show discrepancies and limited consistency across views.Its user preference score is 9.2; Infinicity and CityGen scores are omitted because they are not comparable under the stated availability and single-view constraints.Source: §5.3 · p. 7, Figure 4 · p. 8
5.4 Ablation Study
- Users preferred conditional layout models despite no significant FID/KID differences; ratio guidance was favored over text and effectively controlled class ratios.Average Class Error compares generated sample class ratios with those of 10,000 input conditions.Source: §5.4 · p. 8, Table 4 · p. 9
- Planning is less stable in early rounds, supporting multi-round refinement; prompts also produce distinct residential and commercial land-use mixes.Commercial zones comprise 47% public services and 31% commercial establishments, while residential zones are 60% residential.Source: §5.4 · p. 9, Figure 5 · p. 9
6 Conclusion and Broader Impacts
CityCraft creates detailed 3D city scenes from text and ratio specifications, combining layout generation, city planning, and scene construction. The conclusion reports improvements in diversity, controllability, and visual appeal, and identifies dynamic traffic and real-time feedback as planned extensions.
6 Conclusion and Broader Impacts
- CityCraft combines layout generation, city planning, and scene construction to create detailed 3D cities aligned with user requirements.The authors report improved diversity, controllability, and visual appeal, and introduce the CityGen-OSM and CityGen-Buildings datasets.Source: §6 · p. 9
Supplementary Material
The supplementary material outlines CityCraft’s algorithm and points to supporting sections on performance, datasets, user-study settings, and demonstrations. The algorithm turns text and ratio inputs into a 3D urban scene through layout generation, planning, and asset adaptation.
Supplementary Material
- The supplement directs readers to details on the algorithm, LLM performance, dataset assets, user-study scoring, and generated-scene demonstrations.It identifies these as separate supplementary sections.Source: Supplementary Material · p. 15
7 CityCraft Algorithm
- The layout generator uses noise and an encoded condition, while the planning module refines the layout and the adaptation system selects and assigns assets.The supplementary discussion also reviews diffusion models, which add noise during training and learn to reverse that process.Source: §7 · p. 15
- CityCraft’s algorithm encodes user text and class ratios, generates a semantic layout, refines it into a plan, and adapts assets to produce a 3D scene.The pseudocode describes the process from user input through scene output.Source: §7 · pp. 15–16
8 Detailed Performance
The LLM’s output quality rises across iterations while its rate of change declines or stabilizes. The figure describes these trends as iteration-dependent output dynamics.
8 Detailed Performance
- As LLM iterations increase, output quality continues to rise while the rate of change declines or stabilizes.The passage characterizes the pattern as improving output despite slower changes.Source: §8 · p. 16
- The figure tracks both the rate of change and output quality as functions of the LLM iteration index.The two curves represent different aspects of output dynamics.Source: §8 · p. 16
- The stabilizing rate of change suggests diminishing returns in learning new patterns or making substantial output adjustments.This interpretation is stated as a suggestion in the passage.Source: §8 · p. 16
9 Dataset Example
The dataset examples compare CityCraft-OSM with CityDreamer-OSM and include satellite imagery for comparison with real-world geography. The accompanying text highlights map clarity, organization, and practical alignment.
9 Dataset Example
- CityCraft-OSM-Satellite includes real-world satellite imagery for comparison with urban geography.The text presents this imagery as a ground-truth comparison of dataset accuracy.Source: §9 · p. 16, Figure I2 · p. 17
- CityCraft-OSM is presented as clearer and more organized than the CityDreamer-OSM visualizations.The comparison describes the CityDreamer-OSM examples as showing dense urban details.Source: §9 · p. 16, Figure I2 · p. 17
- The authors describe the dataset developments as useful to urban planners, architects, and developers creating realistic, functional simulations.Source: §9 · p. 17
10 User Study Settings
The user study uses 22 participants to rate generated images across seven quality dimensions. Model scores average the ratings after excluding the highest and lowest scores.
10 User Study Settings
- The study invited 22 participants: 12 undergraduates, 8 graduate students, and 2 faculty members.Source: §10 · p. 17
- Participants rated fidelity, style, consistency, controllability, clarity, sharpness, and overall quality on a 0–10 scale.Scores from 8–9 were labeled excellent, 6–7 good, 4–5 average, 2–3 poor, and 0–1 terrible.Source: §10 · p. 17
- Each model’s final score is the average of user ratings after excluding the highest and lowest ratings.Source: §10 · p. 17
11 Demo Visualization
CityCraft demonstrations present complex urban scenes at citywide, environmental-detail, and street levels, illustrating the framework’s rendering capabilities and potential applications in urban planning and architecture.
11.1 Comprehensive Cityscape Rendering
- The panoramic cityscape depicts diverse building architectures, planned layouts, complex spatial relationships, and urban density at large scale.Figure K3’s top view shows the overall cityscape and varied architectural styles.Source: §11.1 · p. 18, Figure K3 · p. 19
11.2 Detailed Environmental Interactions
- Close views of squares, parks, buildings, roads, pathways, and facades highlight detailed textures and material qualities.Figure K3’s bottom-left view focuses on public spaces and individual buildings.Source: §11.2 · p. 18, Figure K3 · p. 19
11.3 Street-Level Perspectives
- Street-level views let planners assess human-scale design, including walkability, greenery, and interactions among urban elements.Figure K3 also shows trees and parks; Figure K4 presents multi-view building assets with function, style, object-count, and floor metadata, including 12-floor commercial buildings and a 2-floor residential estate.Source: §11.3 · p. 18, Figure K3 · p. 19, Figure K4 · p. 20
11 Demo Visualization
- The demonstrations collectively present CityCraft as a tool for visualizing and planning future urban developments with detail and realism.The section frames the visualizations as relevant to stakeholders in urban planning and architecture.Source: §11.3 · p. 18, §11 · p. 18