Source-linked AI summary
MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layouts
Zilong Huang, Jun He, Xiaobin Huang, Ziyi Xiong, Yang Luo, Junyan Ye, Weijia Li, Yiping Chen, Ting Han
TL;DR
Existing approaches struggle to balance text-based creative flexibility with the object-level editability of explicit structures in realistic 3D cities. MajutsuCity combines language-guided layout, asset, and scene generation with an editing agent and multimodal dataset. It outperforms the compared methods across reported metrics, including layout FID and AQS/RDR rankings.
Problem
Text-based city generation and explicit object-level representations have not yet been balanced, while city-scale scenes demand geometric validity, stylistic diversity, and editability.
Source: §1 · p. 2Method
Results
MajutsuCity outperforms CityDreamer, GaussianCity, UrbanWorld, and CityCraft across all metrics, with layout FID reductions of 83.7% over CityDreamer and 20.1% over CityCraft, and ranks first in all eight AQS and RDR dimensions.
Source: §1 · p. 2, §5.2 · p. 7Takeaways & Limitations
The framework provides a direction for scalable, controllable, style-adaptive urban generation and a metric suite for benchmarking city-scene quality.
Source: §6 · p. 9The layout module is sensitive to prompt logical coherence; contradictory or infeasible initial instructions can propagate and degrade assembled scenes.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Generating realistic 3D cities is fundamental to world models, virtual reality, and game development, where an ideal urban scene must satisfy both stylistic diversity, fine-grained, and controllability. However, existing methods struggle to balance the creative flexibility offered by text-based generation with the object-level editability enabled by explicit structural representations. We introduce MajutsuCity, a natural language-driven and aesthetically adaptive framework for synthesizing structurally consistent and stylistically diverse 3D urban scenes. MajutsuCity represents a city as a composition of controllable layouts, assets, and materials, and operates through a four-stage pipeline. To extend controllability beyond initial generation, we further integrate MajutsuAgent, an interactive language-grounded editing agent} that supports five object-level operations. To support photorealistic and customizable scene synthesis, we also construct MajutsuDataset, a high-quality multimodal dataset} containing 2D semantic layouts and height maps, diverse 3D building assets, and curated PBR materials and skyboxes, each accompanied by detailed annotations. Meanwhile, we develop a practical set of evaluation metrics, covering key dimensions such as structural consistency, scene complexity, material fidelity, and lighting atmosphere. Extensive experiments demonstrate MajutsuCity reduces layout FID by 83.7% compared with CityDreamer and by 20.1% over CityCraft. Our method ranks first across all AQS and RDR scores, outperforming existing methods by a clear margin. These results confirm MajutsuCity as a new state-of-the-art in geometric fidelity, stylistic adaptability, and semantic controllability for 3D city generation. We expect our framework can inspire new avenues of research in 3D city generation. Our project page: https://longhz140516.github.io/MajutsuCity/.
1. Introduction
City-scale generation must reconcile text-based creative control with geometric validity and object-level editability. MajutsuCity combines a language-to-city pipeline with interactive editing, a multimodal dataset, and evaluation metrics; it reports broad benchmark gains.
1. Introduction
- Text-driven methods offer expressive control but struggle with urban-scale geometry, while layout-guided implicit or neural methods face multi-view inconsistency and simulation incompatibility.Explicit-mesh methods improve structural reliability, but their diversity is limited by the coverage and style of predefined asset libraries.Source: §1 · p. 2
- MajutsuCity parses natural-language geometric and aesthetic intent, generates layouts and height maps, synthesizes stylized assets and materials, then assembles an editable 3D city.The framework treats language as conveying both macroscopic geometric logic and fine-grained aesthetic intent.Source: §1 · p. 2
- The framework adds MajutsuAgent for five object-level editing operations, MajutsuDataset for multimodal scene inputs, and VLM-based AQS and RDR metrics across four quality dimensions.The metrics cover structural and view consistency, scene richness and complexity, material and texture fidelity, and lighting and atmosphere.Source: §1 · p. 2
- MajutsuCity outperforms CityDreamer, GaussianCity, UrbanWorld, and CityCraft across all metrics, reducing layout FID by 83.7% over CityDreamer and 20.1% over CityCraft.It ranks first in all eight AQS and RDR dimensions.Source: §1 · p. 2
2. Related Work
Prior work offers pieces of urban generation—structured layouts, advanced 3D assets, or large scenes—but leaves gaps in language control, unified city-scale integration, and editable object-level structure. MajutsuCity draws on these directions to connect textual intent with controllable urban synthesis.
2.1. Layout Generation
- Vector-based layout methods provide structured formulations but struggle with complex fine-grained patterns, while scalable mask-based methods capture boundaries yet often lack natural-language control.MajutsuCity addresses the control gap by aligning fine-grained spatial mask synthesis with textual descriptions.Source: §2.1 · pp. 2–3
2.2. Image-to-3D Generation
- Recent image-to-3D systems use 2D visual priors and models such as Trellis, Step1X-3D, and Hunyuan3D to produce increasingly photorealistic, detailed assets.MajutsuCity explores integrating advanced 3D asset-generation models into a unified, reproducible city-scale pipeline.Source: §2.2 · p. 3
2.3. City Scene Generation
- Implicit or neural city representations can show geometric artifacts and multi-view inconsistency and lack editable object structures, while retrieve-and-place methods are limited by their asset libraries.Inspired by object-centric indoor generation, MajutsuCity scales the combination of 2D vision priors and on-demand 3D objects to urban scenes and diverse styles.Source: §2.3 · p. 3
3. MajutsuCity
MajutsuCity turns natural-language intent into a structured, object-level urban scene through design specification, layout generation, asset and material synthesis, and scene assembly. Its pipeline separates city-scale layout from building-level modeling and supports subsequent language-guided edits.
3. MajutsuCity
- MajutsuCity comprises Scene Design, Layout Generation, Assets & Materials Generation, and Scene Generation, with MajutsuAgent providing interactive, fine-grained scene editing.The stages bridge textual intent and structured 3D scene composition.Source: §3 · p. 3, Figure 2 · p. 4
3.1. Overview of Scene Design
- An LLM decomposes ambiguous free-form prompts into a structured urban design template covering layout, assets, materials, and skymap.Standardized parameters such as land use, spatial distribution, architectural style, and facade material form a blueprint for layout and asset synthesis.Source: §3.1 · p. 3
3.2. Layout Generation
- Layout Generation produces paired semantic layout and building-height maps through a two-stage cascade, using long-text conditioning for layouts and layout-derived spatial control for heights.Both stages are trained on MajutsuDataset with a latent diffusion objective; the height stage uses the generated layout mask to maintain spatial alignment.Source: §3.2 · pp. 3–4
3.3. Assets and Materials Generation
- Asset generation creates a dedicated building asset per layout instance, using image-based refinement and optional point-cloud constraints to preserve geometric alignment while adding appearance detail.Materials and sky spheres are generated with a model fine-tuned on domain-specific tilable-texture and skybox datasets.Source: §3.3 · pp. 4–5
3.4. Scene Generation
- Scene Generation composes ground, road, water, and vegetation as material-bound planar layers, then places trees, streetlights, and buildings in layout-aligned positions.A panoramic sky sphere initializes ambient lighting and global illumination in the resulting renderable scene.Source: §3.4 · p. 5
3.5. Scene Agent
- MajutsuAgent translates language commands into Add, Delete, Edit, Move, and Replace operations for object-level city-scene modification.GPT-5 decomposes commands into atomic operations; Move applies rigid transformations, while Replace substitutes surface materials.Source: §3.5 · p. 5
4. MajutsuDataset
MajutsuDataset brings together text-aligned urban layouts and heights, diverse building models, and rendering assets. Its components supply varied geographic layouts, architectural styles, tilable PBR textures, and skyboxes for text-guided scene synthesis.
4. MajutsuDataset
- The OSM-based Layout/Elevation subset contains 13,300 samples from diverse regions, each with paired 512×512 semantic and height maps and a context-aware textual description.Semantic maps cover vegetation, roads, water bodies, buildings, and ground; GPT-5-mini generates the descriptions.Source: §4 · p. 5, Figure 3 · p. 6
- The 3D Building Models subset contains 1,000 assets across ten architectural styles and twenty building types per style.The assets are generated using five commercial-grade 3D generation systems.Source: §4 · p. 6, Figure 3 · p. 6
- Material Assets include 2,300 seamlessly tilable PBR materials with complete texture maps and 1,000 HDR skybox maps covering varied lighting and atmospheric conditions.GPT-5-mini generates textual metadata for each material and skybox asset.Source: §4 · p. 6, Figure 3 · p. 6
5. Experiments
MajutsuCity’s experiments compare generated layouts and city scenes with prior methods, using quantitative layout metrics and a GPT/human evaluation framework for scenes. Results report stronger layout and scene quality, while ablations examine spatial text and LongCLIP and style-conditioned generations show fidelity and consistency.
5.1. Implementation Details and Metrics
- Layout evaluation uses FID, KID, and IS; scene evaluation uses AQS and RDR across four dimensions, with GPT-5 and 20 human users.AQS uses GPT-5 scores from 1–10, while RDR aggregates GPT-5 and human pairwise comparisons using TrueSkill; layout networks train at 512×512 pixels for 100 epochs.Source: §5.1 · p. 6
5.2. Comparison of City Layout Generation
- 83.7% and 20.1% FID reductions over CityDreamer and CityCraft, respectively, accompany more structurally plausible layouts and higher IS scores.The comparison is reported in Table 1; the higher IS is described as indicating clearer, more diverse urban structures.Source: Table 1 · p. 7, §5.2 · p. 7
- Qualitative layout comparisons show complete, sharp building footprints aligned with roads, forming coherent city blocks unlike the fragmented or misaligned prior outputs.The authors describe these layouts as a reliable structural foundation for downstream high-fidelity scene generation.Source: §5.2 · pp. 6–7, Figure 4 · p. 7
5.3. Comparison of City Scene Generation
- Across two representative scenes, MajutsuCity shows stable geometry, multi-view consistency, stylistic diversity, and high-fidelity geometry and textures versus four baselines.CityDreamer and GaussianCity show geometric artifacts and view inconsistency; UrbanWorld has coarse primitives and low texture fidelity, while CityCraft’s fixed asset library limits stylistic diversity.Source: §5.3 · pp. 7–8, Figure 5 · p. 8
- AQS and RDR results show MajutsuCity surpasses all baselines, while GaussianCity’s SVC ordering against mesh methods reverses between protocols.The authors interpret the reversal as evidence that RDR mitigates bias in absolute scoring; GPT and human evaluations align strongly.Source: Table 2 · p. 8, §5.3 · pp. 8–9
5.4. Ablation Study
- Removing LongCLIP raises FID from 22.7 to 28.0, and replacing fine-grained spatial text with short prompts raises it to 35.7.The KID and IS trends are reported as consistent; separate style-conditioned examples capture four recognized styles with high fidelity and strong intra-style consistency.Source: §5.4 · p. 9, Table 3 · p. 9
6. Conclusion
The conclusion presents MajutsuCity as a controllable, natural-language-driven and aesthetic-adaptive framework for 3D urban generation. It also highlights the dataset, editing agent, and evaluation metrics as components of a broader research direction.
6. Conclusion
- MajutsuCity combines a four-stage framework for controllable urban scene generation with MajutsuDataset, MajutsuAgent, and a metric suite.The dataset integrates layout and elevation priors, 3D assets, and PBR materials; the agent adds postgeneration language-guided editing.Source: §6 · p. 9
- Four style-conditioned city scenes show high fidelity and strong consistency within each style.Source: Figure 6 · p. 9
- AQS and RDR assess structural consistency, scene richness, material fidelity, and lighting realism as a holistic benchmarking system for urban scene generation.The conclusion presents this system as the first holistic metric suite for that purpose.Source: §6 · p. 9
7. Limitations and Futurk Work
The current framework has limitations involving prompt consistency, complex building geometry, and visual scale after assembly. Future work will explore a hybrid paradigm to improve scene rationality and harmony while retaining controllability.
7. Limitations and Futurk Work
- Contradictory or infeasible initial layouts can cascade through the hierarchical pipeline, producing implausible roads or unreasonable building configurations.Source: §7 · p. 9
- High visual fidelity and geometric validity remain challenging for buildings with highly complex or irregular architectural topologies, despite image- and point-cloud constraints.Source: §7 · p. 9
- Independent asset synthesis in local coordinates lacks global scale and context, so adjacent buildings can have inconsistent story heights after assembly.Source: §7 · p. 10
- WorldGen [44] uses holistic planning followed by local asset decomposition and refinement, avoiding instance-stitching scale fragmentation while maintaining terrain topology and stylistic unity.Source: §7 · p. 10
- Future work will explore a hybrid paradigm to improve physical rationality and visual harmony while maintaining the current method’s controllability.Source: §7 · p. 10