PaperlayerView exact revision on arXiv

Source-linked AI summary

GridFlow: Structured Latent Flow for Seamless City-Scale 3D Point Cloud Generation

Xinyu Wang, Muhammad Ibrahim, Atif Mansoor, Ajmal Mian

arXiv:2608.29793v1cs.CV

TL;DR

City-scale point cloud generation from remote sensing remains difficult because prior methods do not jointly handle scale, seamless tiling, and partial observability. GridFlow uses structured latent grids, conditional rectified flow, and orientation-aware colorization, together with the City3D-MultiGen benchmark. It outperforms existing point cloud generation baselines across geometry metrics and produces coherent colored outputs with seamless boundaries over arbitrarily large urban extents.

  • Problem

    Existing point cloud generation methods are limited to single objects or bounded scenes and do not handle city-scale remote-sensing generation with seamless tiling and partial observability.

    Source: §1 · pp. 1–2

  • Method

    GridFlow combines a topology-preserving Grid-Aligned VAE, conditional rectified flow, orientation-aware diffusion colorization, and latent edge consistency for multimodal city-tile generation.

    Source: §1 · pp. 2–3, §4 · p. 6, Latent-Space Edge Consistency · p. 9

  • Results

    GridFlow outperforms existing point cloud generation baselines across all geometry metrics and produces visually coherent colored outputs with seamless boundaries over arbitrarily large urban extents.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The framework supports dense, colored city-scale point cloud generation from satellite imagery with optional semantic and elevation conditions.

    Source: §4 · p. 6, §6 · p. 13

    Cross-city generalization degrades moderately, with Chamfer Distance increasing from 0.00272 to 0.00394 on London.

    Source: Limitations and Future Work · p. 14

Abstract

from arXiv · show

Generating realistic 3D city environments from remote sensing data is important for simulation, urban planning, and mixed reality, yet existing point cloud generation methods are limited to single objects or bounded indoor scenes and cannot handle the scale, seamless tiling, and partial observability challenges of city-scale generation. We present \ours{}, a multi-stage framework that generates dense, colored point clouds ($10^5$ points per $150\text{m}{\times}150\text{m}$ tile) at city scale, conditioned on satellite imagery, semantic segmentation maps, and digital surface models (DSM). A \emph{Grid-Aligned VAE} encodes each tile into a topology-preserving latent grid where tokens correspond to fixed spatial regions, enabling spatially coherent multi-modal conditioning and compact latent-space edge consistency that implicitly aligns thousands of boundary points for seamless cross-tile generation. A conditional rectified flow model synthesizes geometry latents from the fused multi-modal conditions, and an orientation-aware diffusion colorizer separately handles satellite-visible horizontal surfaces and occluded vertical façades. To support standardized evaluation, we build on public 3D data sources to introduce \emph{City3D-MultiGen}, a benchmark of $163$K densely annotated tiles from Melbourne and London with aligned point clouds, satellite images, semantic maps, and elevation data. Experiments show that \ours{} outperforms adapted point cloud generation baselines across all geometry metrics and produces visually coherent colored point clouds with seamless boundaries over arbitrarily large urban extents. Our benchmark details are available at https://huggingface.co/datasets/e32/City3D-MultiGen

1 Introduction

GridFlow targets city-scale point cloud generation from remote sensing data, addressing the scale, multimodal conditioning, and seamless-tiling limitations of prior bounded-scene methods. Its structured latent representation supports coherent conditioning and efficient cross-tile consistency.

  • Remote-sensing generation could expand realistic 3D city creation beyond photogrammetry’s extensive aerial-capture requirements.Source: §1 · p. 1
  • Prior point cloud methods largely target bounded objects or scenes and do not address city-scale tiling, remote-sensing conditioning, or seamless generation.Source: §1 · p. 2
  • A Grid-Aligned VAE maps each tile to a topology-preserving latent grid whose tokens correspond to fixed 6m×6m physical regions.Source: §1 · p. 2
  • The latent grid reduces generation from a 10^5-point output space to 625 tokens while aligning latent structure with conditioning modalities.Source: §1 · p. 3
  • Latent boundary consistency aligns thousands of decoded boundary points, while rectified flow and orientation-aware colorization complete the multimodal generation pipeline.Source: §1 · p. 3

2 Related Work

Prior 3D generation research progresses from object and indoor synthesis toward outdoor and unbounded city scenes, but existing representations and tiling strategies remain mismatched with explicit, seamless city-scale point clouds. GridFlow instead combines remote-sensing conditioning, explicit colored point clouds, and learned latent edge consistency.

Object-Level Point Cloud Generation

Indoor and Outdoor Scene Generation

City-Scale 3D Generation

  • Unbounded scene generators use overlapping regions, blending, or joint denoising, which couples inference cost to overlap and can leave residual seams requiring registration.Source: City-Scale 3D Generation · p. 4
  • GridFlow imposes latent-space edge consistency during training, allowing tiles to be generated independently at inference without overlap or blending.Source: City-Scale 3D Generation · pp. 4–5

GridFlow’s Distinction

  • The method uses satellite and semantic inputs, produces explicit colored point clouds, and enforces latent edge consistency for seamless tiling.Source: City-Scale 3D Generation · p. 5

3 City3D-MultiGen Benchmark

City3D-MultiGen addresses the lack of large-scale benchmarks with aligned multimodal conditions by pairing public 3D city data with satellite imagery, semantic maps, and elevation data. Its automated pipeline produces spatially aligned tiles for training and evaluation.

  • City3D-MultiGen contains 163K densely annotated tiles from Melbourne and London with aligned point clouds, satellite images, semantic segmentation, and elevation data.Source: §3 · p. 5
  • The benchmark uses complementary public 3D sources: aerial-photogrammetry point clouds covering Melbourne and high-fidelity FBX models covering central London.Source: Source Data · p. 5
  • An automated pipeline partitions both datasets into 150m×150m regular-grid tiles and pairs each with aligned satellite, semantic, and DSM modalities.Source: Automated Multi-Modal Alignment Pipeline · p. 5, Figure 2 · p. 6
  • The processing pipeline provides paired 3D geometry and multimodal conditions for training and evaluation through reproducible data retrieval scripts.Source: §3 · p. 5, Figure 2 · p. 6
  • Raw tiles contain approximately 150,000 points and are split by contiguous spatial regions with at least 150m separation between training and test data.Source: Automated Multi-Modal Alignment Pipeline · p. 6

4 Method

GridFlow uses a three-stage pipeline that maps city tiles to a structured latent grid, generates geometry from aligned remote-sensing conditions, and colorizes surfaces with orientation-aware conditioning. Its design supports spatial coherence, latent-space boundary consistency, and separate treatment of visible horizontal surfaces and occluded façades.

Pipeline Overview

  • GridFlow takes a 150m×150m satellite image as its primary input and can additionally use semantic maps and DSM to generate a colored 100,000-point cloud.Structured condition dropout trains the model to operate with any subset of available modalities.Source: §4 · p. 6
  • The three-stage pipeline uses a Grid-Aligned VAE, a conditional latent generator, and an orientation-aware colorization model.Separating stages avoids joint geometry-color optimization and lets each stage use a suitable generative framework.Source: §4 · pp. 6–7, Figure 3 · p. 7

Grid-Aligned VAE

  • The Grid-Aligned VAE represents each tile as a 64×25×25 latent grid whose tokens correspond one-to-one with 6m×6m physical regions.The 25×25 grid exploits local spatial regularity while preserving spatial correspondence for conditioning.Source: §4.1 · p. 7
  • The VAE objective combines bidirectional Chamfer distance, height-histogram regularization, boundary Chamfer distance, and low-weight KL regularization.The KL term uses λKL=10^-5, prioritizing reconstruction.Source: Decoding · p. 8

Conditional Latent Generation

  • A conditional rectified-flow model generates geometry latents from fused satellite, semantic, and DSM features aligned to the latent grid.The model transports Gaussian noise toward normalized VAE latents using a Transformer velocity network and global and local condition paths.Source: Multi-Modal Condition Fusion · p. 8, Rectified Flow · pp. 8–9
  • Latent-space edge consistency matches boundary columns or rows of adjacent tile latents instead of decoded point clouds.With a two-column boundary margin, matching 64-dimensional tokens implicitly aligns approximately 6,400 boundary points.Source: Latent-Space Edge Consistency · p. 9

Orientation-Aware Colorization

5 Experiments

GridFlow achieves strong geometry quality, conditioning fidelity, seamless tiling, and cross-city generalization in city-scale point-cloud generation experiments.

Geometry Comparison

  • GridFlow achieves the best performance across all five geometry metrics, reducing Chamfer Distance by 29% relative to PVD and reaching an F-Score of 0.884.Source: Geometry Comparison · p. 10

Visual Comparison

Geometry Comparison

  • GridFlow’s JSD is 0.010 versus PVD’s 0.037, while COV-CD reaches 51.8% versus 24.4%, indicating stronger distributional coverage.Source: Geometry Comparison · p. 11

Visual Comparison

Large-Tile Visualization

  • Latent-space edge consistency enables independently generated tiles to continue buildings, roads, and vegetation across boundaries without visible seams.Source: Large-Tile Visualization · p. 12

Cross-Dataset Generalization

  • Without retraining, Melbourne-trained GridFlow achieves CD 0.00394 and F-Score 0.783 on London data, showing cross-city generalization under domain shift.Source: Cross-Dataset Generalization · p. 12

Ablation: Conditioning Modalities

6 Conclusion

GridFlow combines structured latent representations, latent-space edge consistency, and staged generation to produce dense colored city-scale point clouds. It also introduces a two-city benchmark and reports strong geometry and seamless-tiling results, while identifying cross-city generalization and training-stage accumulation as limitations.

Method

  • GridFlow generates dense colored city-scale point clouds using a Grid-Aligned VAE, conditional rectified flow, and orientation-aware diffusion colorization.The framework separates geometry synthesis from colorization and uses structured latent grids for spatial conditioning.Source: §6 · p. 13

Benchmark

  • City3D-MultiGen contains 163K tiles from Melbourne and London with aligned point clouds, satellite images, semantic maps, and elevation data.The benchmark supports standardized evaluation across multiple remote-sensing modalities and cities.Source: §6 · p. 13

Seamless generation

  • GridFlow produces continuous geometry across independently generated Melbourne tiles, with buildings, roads, and vegetation continuing naturally across boundaries.The visualization demonstrates practical city-scale deployment of latent-space edge consistency.Source: Figure 6 · p. 14

Results

  • GridFlow outperforms existing point cloud generation baselines across all geometry metrics and produces visually coherent colored outputs over arbitrarily large urban extents.The reported outcome combines quantitative geometry gains with qualitative color coherence and seamless boundaries.Source: §6 · p. 14

Limitations

  • Cross-city generalization degrades moderately, while sequential training may accumulate errors and fixed tile size or cell resolution may not suit every urban morphology.The reported London Chamfer Distance changes from 0.00272 to 0.00394, and façade colorization lacks direct vertical supervision from satellite imagery.Source: Limitations and Future Work · p. 14

Related papers

Loading 2608.29793v1…