CityLift: Compositional 3D Urban Generation from a Single Image

1. Introduction

• Existing approaches can lack complete building meshes, require decomposition of unified city meshes, constrain diversity to a retrieval library, or incur iterative planning costs.Source: §1 · p. 1
• Urban generation must resolve ambiguous building positions, orientations, scales, and heights while preserving detail across buildings and supplying paired scene and building supervision.Occlusion and height variation complicate spatial inference, while a fixed-resolution block-wide grid limits individual building detail.Source: §1 · p. 2
• Urban Scaffold Generation jointly produces coarse building geometry and layout through coupled scene and instance streams, encoding placement in scene-space geometry without a separate pose-regression stage.A shared coordinate mapping preserves correspondence between the scaffold and local building generation.Source: §1 · p. 2
• CityLift-5K contains 4,997 synthetic multi-building scenes pairing oblique images and instance masks with complete meshes and placement annotations for scene- and building-level supervision.Source: §1 · p. 2
• CityLift reduces building- and scene-level Chamfer distances by 60.3% and 58.4%, respectively, on CityLift-5K versus the strongest baseline at each level, and achieves the best scene-level geometric results on UrbanScene3D and InstanceBuilding.The results are reported for the held-out CityLift-5K test split and external synthetic and real-world datasets.Source: §1 · p. 2
• CityLift generates complete, independently editable textured building meshes from an oblique urban image and instance masks, with relative positions, orientations, scales, and heights consistent with the input.Source: §1 · p. 2
• Generative Spatial Inheritance maps scaffold occupancy and scene-conditioned features into aligned building volumes, where Conditioned Building Regeneration synthesizes finer sparse structures.Source: §1 · p. 2

2.1. 3D urban generation

• CityLift introduces single-image compositional urban generation that jointly produces complete textured building meshes and their spatial arrangement, with assets that can be replaced, rearranged, and reused.Source: §2.1 · p. 3

2.2. Single-image 3D object generation

• Object-generation approaches use image-diffusion optimization, view synthesis, feedforward reconstruction, or native 3D priors; CityLift draws on SAM 3D Objects for textured building synthesis.Source: §2.2 · p. 3

2.3. 3D scene generation

• Scene-generation work ranges from predicted layouts and coupled denoising to separate asset placement and structure-conditioned refinement; CityLift jointly learns urban placement and height, then transports occupancy and hidden features into building-specific volumes.Source: §2.3 · p. 3

3.1. Problem Definition

• Each input image and instance mask defines a task of generating editable textured building meshes with plausible appearance and geometry, while inferring placement, orientation, and relative dimensions.The generated meshes share a normalized scene domain and should remain consistent with the input image.Source: §3.1 · p. 3

3.2. Overview of CityLift Pipeline

• A building-specific map transports the scene scaffold into a local volume, where a conditional flow generates refined structure from the scaffold and local image tokens.The same map constructs the local training target and returns generated meshes to the scene.Source: §3.2 · p. 3

3.3. Urban Scaffold Generation

• Coupled scene and instance streams use image-conditioned attention to model coarse occupancy in scene coordinates, encoding building shape, placement, and relative scale in occupied cells.The resulting scaffold retains occupancy support and scene-conditioned 3D features for subsequent spatial inheritance.Source: §3.3 · p. 4

3.4. Generative Spatial Inheritance

• A deterministic map transfers each building’s scaffold occupancy and contextual features into an axis-preserving local volume with a dedicated voxel budget.Local tokens receive geometric support and scene-conditioned features; the inherited frame also defines mesh supervision and maps generated meshes back into the scene.Source: §3.4 · pp. 4–5

3.5. Conditioned Building Regeneration

• Conditioned on inherited scaffold features and a tight building-image crop, a flow model regenerates finer sparse structure, with inherited conditions injected after transformer blocks.Training combines flow matching with geometric losses for occupancy support, centroid position, and axial geometry distribution.Source: §3.5 · p. 5
• The regenerated sparse structure supports image-conditioned SLAT generation and mesh decoding, after which each mesh is transformed into the scene while retaining its building instance.The shared inherited coordinate reference connects local generation to scene assembly.Source: §3.5 · p. 5

Dataset overview

• 4,997 synthetic urban blocks and 36,338 building instances pair rendered images and instance masks with individually accessible building meshes in a shared coordinate frame.The paired data supports supervision of both building geometry and spatial arrangement.Source: §4 · p. 5

Asset preparation

• 42,339 building meshes were curated through multiview screening and manual filtering to remove non-building content, disconnected groups, and incomplete geometry.Retained assets receive function, architectural-style, floor-count, and dimension annotations used for selection and scale priors.Source: §4 · pp. 5–6

Scene construction and annotations

• Procedural parcel placement and layout planning assemble scenes spanning five settings, with 2–12 buildings per scene.Buildings are selected by annotated function and size, then scaled and positioned within the planned layout.Source: §4 · p. 6
• Each scene has a 1024 × 1024 rendered image, pixel-aligned instance segmentation, individual meshes, placement transforms, camera parameters, and instance identifiers.The oblique orthographic camera uses off-nadir angles sampled from 30° to 40°; the dataset splits into 4,497 training, 250 validation, and 250 test scenes.Source: §4 · p. 6

5.1. Experimental Setup

• The evaluation covers 250 held-out CityLift-5K scenes with unseen buildings, plus synthetic UrbanScene3D and real reconstructed InstanceBuilding blocks.The study reports building- and scene-level geometric metrics and a multi-view visual evaluation.Source: §5.1 · p. 6, §5 · p. 6

5.2. Comparison on CityLift-5K

• Building CD fell 60.3% to 1.011 cells and scene CD fell 58.4% to 1.079 cells; scene F@2 rose from 0.597 to 0.855.CityLift also requires approximately 115 seconds per scene, faster than SceneGen, 3D-Fixer, and Pixal3D.Source: §5.2 · p. 6, Table 1 · p. 7

5.3. Evaluation on External Scene Sources

• On external building geometry, Pixal3D retains the best accuracy and F-scores, while CityLift improves completeness and symmetric CD across both datasets.On UrbanScene3D, CityLift's building CD is 1.421 versus Pixal3D's 1.490 cells; the stated comparison attributes the difference to lower completeness error despite higher accuracy error.Source: §5.3 · pp. 7–8
• CityLift leads all five scene-level metrics on UrbanScene3D and InstanceBuilding, including scene CD reductions of 23.7% and 23.9%, respectively.On UrbanScene3D, scene F@2 improves by 5.7 percentage points; on InstanceBuilding it improves by 2.3 percentage points.Source: §5.3 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8

5.4. Multi-view Visual Evaluation

• CityLift scores 82.8 overall versus 77.6 for TRELLIS.2 and leads the visual assessor's layout, shape, appearance, and multi-view dimensions.The assessor scores the input image and four rendered views for each output on shared UrbanScene3D blocks.Source: Table 4 · p. 8, §5.4 · p. 8

5.5. Ablation Studies

• Removing scene guidance raises scene CD from 1.079 to 2.590, while local regeneration reduces building CD from coarse decoding's 1.150 to 1.011 and restores finer facade details.Direct coarse decoding produces smoother, simplified surfaces with less distinct windows and other facade details.Source: §5.5 · pp. 8–9
• Removing both inherited-conditioning pathways raises building CD to 2.510, while removing occupancy injection alone raises scene CD to 1.260.Removing or pooling hidden features yields similar scene CDs of 2.170 and 2.200; the reported visual results show fragmented surfaces in these conditions.Source: §5.5 · p. 9

• Experiments on CityLift-5K, UrbanScene3D, and InstanceBuilding demonstrate improved scene-level geometric fidelity.Source: §6 · p. 10
• Urban scaffold generation captures layout and coarse building geometry, while spatial inheritance transfers geometric support and hidden features into local building frames.Source: §6 · p. 10
• Conditioned building regeneration refines individual structures into explicit, individually editable meshes assembled with inherited spatial transforms.Source: §6 · p. 10
• CityLift-5K was introduced to support training and evaluation of compositional urban generation.Source: §6 · p. 10