PaperlayerView exact revision on arXiv

Source-linked AI summary

CityDreamer: Compositional Generative Model of Unbounded 3D Cities

Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu

arXiv:2309.00610v3cs.CV

TL;DR

Generating 3D cities is challenging because buildings sharing a semantic class have diverse appearances, and structural distortions are particularly salient in urban environments. CityDreamer composes separate neural fields for building instances and background stuff, and the paper reports state-of-the-art city generation and localized editing. The authors also identify geometric and inference-cost limitations.

  • Problem

    3D city generation is challenging because urban structural distortions are salient and buildings of the same class vary widely in appearance.

    Source: §1 · p. 1

  • Method

    CityDreamer composes separate building-instance and background generators, using bird’s-eye-view representation and volumetric rendering for both.

    Source: §1 · p. 2

  • Results

    The authors report state-of-the-art performance in generating realistic 3D cities and localized editing within them.

    Source: §6 · p. 8

  • Takeaways & Limitations

    CityDreamer is presented as supporting realistic 3D city generation and localized editing, with CityGen datasets providing city layouts and appearances.

    Source: §1 · p. 2, §6 · p. 8

    The layout-generation process raises voxels to a specified height, so it cannot model concave geometries such as caves and tunnels.

    Source: §5.5 · p. 8

Abstract

from arXiv · show

3D city generation is a desirable yet challenging task, since humans are more sensitive to structural distortions in urban environments. Additionally, generating 3D cities is more complex than 3D natural scenes since buildings, as objects of the same class, exhibit a wider range of appearances compared to the relatively consistent appearance of objects like trees in natural scenes. To address these challenges, we propose \textbf{CityDreamer}, a compositional generative model designed specifically for unbounded 3D cities. Our key insight is that 3D city generation should be a composition of different types of neural fields: 1) various building instances, and 2) background stuff, such as roads and green lands. Specifically, we adopt the bird's eye view scene representation and employ a volumetric render for both instance-oriented and stuff-oriented neural fields. The generative hash grid and periodic positional embedding are tailored as scene parameterization to suit the distinct characteristics of building instances and background stuff. Furthermore, we contribute a suite of CityGen Datasets, including OSM and GoogleEarth, which comprises a vast amount of real-world city imagery to enhance the realism of the generated 3D cities both in their layouts and appearances. CityDreamer achieves state-of-the-art performance not only in generating realistic 3D cities but also in localized editing within the generated cities.

1. Introduction

CityDreamer addresses the challenge of generating diverse buildings in unbounded 3D cities by separating building instances from background stuff. The paper also introduces CityGen datasets and reports capabilities in large-scale generation and localized editing.

1. Introduction

  • Buildings sharing a semantic label can vary more in appearance than natural-scene objects such as trees, which may reduce generated building quality.The introduction contrasts this diversity with the relatively consistent appearance of objects in natural scenes.Source: §1 · pp. 1–2
  • CityDreamer separates building instances from background stuff in a compositional model for unbounded 3D cities.The model uses bird’s-eye-view representation and volumetric rendering; a compositor combines the rendered buildings and background.Source: §1 · p. 2
  • CityDreamer tailors its scene parameterization to background textures and building facades, using a generative hash grid and periodic positional encoding, respectively.The hash grid targets naturalness and 3D consistency in irregular background textures; periodic encoding addresses regular facade patterns.Source: §1 · p. 2
  • CityGen includes OSM layout and height data for 80 cities and GoogleEarth’s 24,000 city images from 400 New York orbit trajectories.GoogleEarth images include semantic and building-instance segmentation, and the dataset can be expanded to cities worldwide.Source: §1 · p. 2
  • The paper reports large-scale, diverse city generation and localized editing as CityDreamer capabilities.Its contributions also describe CityGen as providing realistic layouts and appearances, with GoogleEarth offering multi-view consistency and building-instance segmentation.Source: §1 · p. 2

2. Related Work

Prior work extends 3D-aware generation from GANs and semantic scene synthesis to unbounded scenes, while indoor synthesis often relies on costly 3D datasets or CAD retrieval.

3D-Aware GANs

  • Voxel-based 3D GANs face high computational and memory costs, while volume rendering has become an inductive bias for learning 3D representations from 2D images.Most cited 3D-aware GAN methods are trained on curated bounded-scene datasets, including faces, bodies, and objects.Source: §2 · p. 2

Scene-Level Content Generation

  • Semantic image synthesis generates scene-level content using dense correspondence such as semantic segmentation or depth maps, and some approaches achieve 3D-aware synthesis.The passage notes that such approaches may lack full 3D consistency or feed-forward generation for novel worlds.Source: §2 · p. 2
  • Recent works achieve 3D-consistent scenes at infinity scale through unbounded layout extrapolation, whereas indoor synthesis often uses expensive 3D datasets or CAD retrieval.Source: §2 · p. 2

3. Our Approach

CityDreamer creates an unbounded city layout, renders background and building instances with separate generators, then composites their images into a unified result.

3.1. Unbounded City Layout Generator

  • CityDreamer extrudes semantic-map pixels according to building heights to form a 3D layout of city objects and empty space.The layout represents roads, buildings, green lands, construction sites, water areas, and others.Source: §3.1 · pp. 2–3
  • MaskGIT and VQVAE generate extendable semantic maps and height fields, with sliding-window extrapolation producing arbitrary-sized layouts.The sliding windows overlap by 25%; the VQVAE uses L1, cross-entropy, and smoothness losses.Source: §3.1 · p. 3

3.2. City Background Generator

  • The background generator uses a BEV representation and generative neural hash grid, then ray-based volumetric rendering to produce background pixels.Its training combines reconstruction and adversarial losses, including L1 and perceptual losses.Source: §3.2 · pp. 3–4

3.3. Building Instance Generator

  • Each building instance is rendered from a local layout window; connected components identify buildings, with distinct labels for roofs and facades.The generator uses local pixel-level features, positional encoding, a style code, and GAN loss.Source: §3.3 · p. 4

3.4. Compositor

  • A compositor combines the generated background and building images using their corresponding masks to produce the unified city image.The compositing equation sums the masked building images with the masked background.Source: §3.4 · p. 5

4. CityGen Datasets

CityGen combines OpenStreetMap-derived city layouts with Google Earth imagery and annotations. The datasets cover many cities and diverse camera viewpoints.

GoogleEarth Dataset

  • Among datasets offering 3D models, GoogleEarth covers the largest area of real-world images and provides both semantic and instance segmentation annotations.Source: §4 · p. 5

OSM Dataset

  • OSM contains rasterized semantic maps and height fields for 80 cities worldwide, spanning more than 6,000 km2 at approximately 0.597 meters per pixel.Source: §4 · p. 5

GoogleEarth Dataset

  • GoogleEarth includes 400 orbit trajectories in Manhattan and Brooklyn, with 60 images per trajectory and camera parameters used to generate segmentation annotations.Source: §4 · p. 5
  • GoogleEarth viewpoints vary in altitude and elevation, and the underlying Google Earth and OpenStreetMap sources allow expansion to more cities worldwide.Source: §4 · p. 6

5. Experiments

CityDreamer is evaluated against prior city and scene-generation methods using image-quality, geometry, view-consistency, and user-study measures. Ablations examine its layout, building-instance, and scene-parameterization components; the discussion notes geometry and inference-cost limitations.

5.1. Evaluation Protocols

  • Evaluation samples 15,000 frames from generated videos and compares them with 15,000 GoogleEarth images using FID, KID, depth error, and camera error.Depth error is assessed on 100 frames per method; camera error measures the difference between generated and COLMAP-estimated trajectories.Source: §5.1 · p. 6

5.2. Implementation Details

  • Both city generators train for 298,500 iterations with batch size 8; the layout VQVAE and autoregressive transformer train for 1,250,000 and 250,000 iterations, respectively.The generators use randomly cropped 192×192 images.Source: §5.2 · p. 6

5.3. Main Results

  • CityDreamer improves FID and KID, achieves the lowest DE and CE errors among baselines, and produces more realistic and diverse results.Table 2 excludes InfiniCity because its code is unavailable.Source: Table 2 · p. 6, §5.3 · p. 7, Figure 4 · p. 7
  • In a user study, 20 volunteers rated generated camera trajectories for perceptual imagery quality, 3D realism, and 3D view consistency; the proposed method significantly outperformed baselines by a large margin.Ratings used a 1–5 scale, with 5 representing the best rating.Source: §5.3 · p. 7

5.4. Ablation Study

  • The Unbounded Layout Generator outperforms IPSM and InfinityGAN on all reported layout metrics.The comparison uses FID and KID.Source: §5.4 · p. 7, Table 3 · p. 8
  • Ablations support the building instance generator and instance labels; removing the generator or omitting labels degrades the reported results.The no-generator variant falls back to SceneDreamer.Source: §5.4 · pp. 7–8, Table 4 · p. 8
  • HashGrid suits background generation and improves its multi-view consistency, while periodic SinCos encoding better captures building-facade periodicity and supports more structurally reasonable buildings.Using local encoders for backgrounds or global encoders for buildings considerably degrades image quality.Source: §5.4 · p. 8, Table 5 · p. 8

5.5. Further Discussions

  • The layout representation cannot model concave forms such as caves and tunnels, and generating buildings individually slightly increases inference cost.The paper identifies reducing inference cost as future work and cites entertainment content creation, including games and movies, as an application area.Source: §5.5 · p. 8

6. Conclusion

CityDreamer separates building-instance generation from background generation and introduces the OSM and GoogleEarth CityGen datasets. The paper reports state-of-the-art 3D city generation and localized editing.

6. Conclusion

  • CityDreamer separates building instances from background stuff to better handle buildings’ diverse appearances, and reports state-of-the-art city generation and localized editing.The authors also introduce OSM and GoogleEarth CityGen datasets for more realistic city layouts and appearances, designed to scale to other cities worldwide.Source: §6 · p. 8

A.1. Qulitative Results for Ablation Studies

The qualitative ablations examine the Unbounded Layout Generator and the Building Instance Generator. The reported comparisons show that removing the latter or its instance labels degrades generated-image quality.

Effectiveness of Unbounded Layout Generator

  • Figure 6 qualitatively compares city layout generation methods, with height maps normalized to [0, 1] by each map’s maximum value.The supplementary comparison is described as demonstrating the Unbounded Layout Generator’s effectiveness.Source: §A.1 · p. 11, Figure 6 · p. 11
  • For the InfinityGAN comparison, semantic-map classes are assigned colors and converted back by choosing the nearest color.This conversion procedure follows the approach used in InfiniCity.Source: §A.1 · p. 11

Effectiveness of Building Instance Generator

  • Removing the Building Instance Generator or its instance labels significantly degrades generated-image quality.Figure 7 presents this qualitative comparison.Source: §A.1 · p. 11, Figure 7 · p. 11

A.2. More Discussions on Scene Parameterization

The scene-parameterization comparisons test encoder and positional-encoding combinations. Qualitative results describe trade-offs between city-background naturalness, building quality, periodic patterns, and multi-view consistency.

A.2. More Discussions on Scene Parameterization

  • CityDreamer’s scene parameterization is reported superior across the additional combinations in Table 6.Table 6 covers effectiveness of different generative scene parameterizations; its caption identifies encoder and positional-encoding abbreviations.Source: §A.2 · p. 12, Table 6 · p. 12
  • Global Encoder with Hash Grid yields more natural backgrounds but severely reduces building quality; Local Encoder weakens this irregularity.The qualitative comparison spans sixteen parameterization settings.Source: §A.2 · p. 12
  • Global Encoder with SinCos introduces periodic patterns; Local Encoder disrupts them but slightly reduces multi-view consistency.Figure 8 labels Global and Local as the background and building encoders, and Hash and SinCos as the respective positional encodings.Source: §A.2 · p. 12, Figure 8 · p. 12

B. Additional Experimental Results

Additional experiments report view-consistency comparisons, building-style interpolation, localized instance editing, and relighting demonstrations. These results show supported capabilities and describe the role of the lighting effects.

B.1. View Consistency Comparison

  • Only SceneDreamer and CityDreamer achieve dense reconstruction among evaluated methods; CityDreamer shows superior view consistency to SceneDreamer.The comparison uses COLMAP on 600 generated frames, and the authors attribute CityDreamer’s advantage to images more conducive to feature matching.Source: §B.1 · p. 13, Figure 9 · p. 13

B.2. Building Interpolation

  • Building style can be interpolated using variable z while the background remains unchanged.Figure 10 depicts gradual style changes from left to right.Source: §B.2 · p. 13, Figure 10 · p. 13

B.3. Localized Editing

  • Building-instance style and height can be modified independently, and styles can be exchanged or changed with a new style vector.Figure 11 illustrates localized edits to instances marked by bounding boxes.Source: §B.3 · p. 14, Figure 11 · p. 14

B.4. Relighting

  • Relighting computes Lambertian shading from light direction and surface normals, while shadow mapping adds light visibility, shadows, and occlusion.CityDreamer generates albedo, normals, and depth for subsequent lighting and shading under provided lighting conditions.Source: §B.4 · p. 14, Figure 12 · p. 14

B.5. Additional Dataset Examples

The appendix expands dataset examples, qualitative comparisons, and an interactive city-creation demo.

B.5. Additional Dataset Examples

  • Figure 13 pairs OSM height fields and semantic maps with generated layouts, and shows GoogleEarth images with segmentation maps.The GoogleEarth examples include six rows from New York City and two showing Singapore and San Francisco.Source: Figure 13 · p. 15, §B.5 · p. 15

B.6. Additional Qualitative Comparison

  • Figure 14 qualitatively compares CityDreamer with state-of-the-art methods, and its caption reports more realistic and diverse results than all baselines.The text also points readers to additional video results.Source: Figure 14 · p. 16, §B.6 · p. 16

C. Interactive Demo

  • The interactive demo lets users generate or load layouts, draw masks, customize camera trajectories, and render and stylize city scenes.Users can preview orbit, point-to-point, or multiple-keypoint trajectories before rendering.Source: C. Interactive Demo · p. 17, Figure 15 · p. 17

Related papers

Loading 2309.00610v3…