Source-linked AI summary
GridFlow: Structured Latent Flow for Seamless City-Scale 3D Point Cloud Generation
Xinyu Wang, Muhammad Ibrahim, Atif Mansoor, Ajmal Mian
TL;DR
City-scale point cloud generation from remote sensing remains difficult because prior methods do not jointly handle scale, seamless tiling, and partial observability. GridFlow uses structured latent grids, conditional rectified flow, and orientation-aware colorization, together with the City3D-MultiGen benchmark. It outperforms existing point cloud generation baselines across geometry metrics and produces coherent colored outputs with seamless boundaries over arbitrarily large urban extents.
Problem
Existing point cloud generation methods are limited to single objects or bounded scenes and do not handle city-scale remote-sensing generation with seamless tiling and partial observability.
Method
GridFlow combines a topology-preserving Grid-Aligned VAE, conditional rectified flow, orientation-aware diffusion colorization, and latent edge consistency for multimodal city-tile generation.
Results
GridFlow outperforms existing point cloud generation baselines across all geometry metrics and produces visually coherent colored outputs with seamless boundaries over arbitrarily large urban extents.
Takeaways & Limitations
The framework supports dense, colored city-scale point cloud generation from satellite imagery with optional semantic and elevation conditions.
Takeaways & Limitations
Cross-city generalization degrades moderately, with Chamfer Distance increasing from 0.00272 to 0.00394 on London.
Abstract
from arXiv · showhide
Generating realistic 3D city environments from remote sensing data is important for simulation, urban planning, and mixed reality, yet existing point cloud generation methods are limited to single objects or bounded indoor scenes and cannot handle the scale, seamless tiling, and partial observability challenges of city-scale generation. We present \ours{}, a multi-stage framework that generates dense, colored point clouds ($10^5$ points per $150\text{m}{\times}150\text{m}$ tile) at city scale, conditioned on satellite imagery, semantic segmentation maps, and digital surface models (DSM). A \emph{Grid-Aligned VAE} encodes each tile into a topology-preserving latent grid where tokens correspond to fixed spatial regions, enabling spatially coherent multi-modal conditioning and compact latent-space edge consistency that implicitly aligns thousands of boundary points for seamless cross-tile generation. A conditional rectified flow model synthesizes geometry latents from the fused multi-modal conditions, and an orientation-aware diffusion colorizer separately handles satellite-visible horizontal surfaces and occluded vertical façades. To support standardized evaluation, we build on public 3D data sources to introduce \emph{City3D-MultiGen}, a benchmark of $163$K densely annotated tiles from Melbourne and London with aligned point clouds, satellite images, semantic maps, and elevation data. Experiments show that \ours{} outperforms adapted point cloud generation baselines across all geometry metrics and produces visually coherent colored point clouds with seamless boundaries over arbitrarily large urban extents. Our benchmark details are available at https://huggingface.co/datasets/e32/City3D-MultiGen
1 Introduction
GridFlow targets city-scale point cloud generation from remote sensing data, addressing the scale, multimodal conditioning, and seamless-tiling limitations of prior bounded-scene methods. Its structured latent representation supports coherent conditioning and efficient cross-tile consistency.
- Remote-sensing generation could expand realistic 3D city creation beyond photogrammetry’s extensive aerial-capture requirements.
- Prior point cloud methods largely target bounded objects or scenes and do not address city-scale tiling, remote-sensing conditioning, or seamless generation.
- A Grid-Aligned VAE maps each tile to a topology-preserving latent grid whose tokens correspond to fixed 6m×6m physical regions.
- The latent grid reduces generation from a 10^5-point output space to 625 tokens while aligning latent structure with conditioning modalities.
- Latent boundary consistency aligns thousands of decoded boundary points, while rectified flow and orientation-aware colorization complete the multimodal generation pipeline.
2 Related Work
Prior 3D generation research progresses from object and indoor synthesis toward outdoor and unbounded city scenes, but existing representations and tiling strategies remain mismatched with explicit, seamless city-scale point clouds. GridFlow instead combines remote-sensing conditioning, explicit colored point clouds, and learned latent edge consistency.
- Object-Level Point Cloud Generation: Object-level methods generate high-quality 2–4K-point shapes but assume centered single-object inputs without spatial conditioning or cross-tile consistency.
- Indoor and Outdoor Scene Generation: Indoor and outdoor scene methods expand scale and conditioning, yet use bounded layouts, voxels, range views, or implicit representations rather than seamless city-scale point clouds.
- City-Scale 3D Generation: Unbounded scene generators use overlapping regions, blending, or joint denoising, which couples inference cost to overlap and can leave residual seams requiring registration.
- City-Scale 3D Generation: GridFlow imposes latent-space edge consistency during training, allowing tiles to be generated independently at inference without overlap or blending.
- GridFlow’s Distinction: The method uses satellite and semantic inputs, produces explicit colored point clouds, and enforces latent edge consistency for seamless tiling.
3 City3D-MultiGen Benchmark
City3D-MultiGen addresses the lack of large-scale benchmarks with aligned multimodal conditions by pairing public 3D city data with satellite imagery, semantic maps, and elevation data. Its automated pipeline produces spatially aligned tiles for training and evaluation.
- City3D-MultiGen contains 163K densely annotated tiles from Melbourne and London with aligned point clouds, satellite images, semantic segmentation, and elevation data.
- The benchmark uses complementary public 3D sources: aerial-photogrammetry point clouds covering Melbourne and high-fidelity FBX models covering central London.
- An automated pipeline partitions both datasets into 150m×150m regular-grid tiles and pairs each with aligned satellite, semantic, and DSM modalities.
- The processing pipeline provides paired 3D geometry and multimodal conditions for training and evaluation through reproducible data retrieval scripts.
- Raw tiles contain approximately 150,000 points and are split by contiguous spatial regions with at least 150m separation between training and test data.
4 Method
GridFlow uses a three-stage pipeline that maps city tiles to a structured latent grid, generates geometry from aligned remote-sensing conditions, and colorizes surfaces with orientation-aware conditioning. Its design supports spatial coherence, latent-space boundary consistency, and separate treatment of visible horizontal surfaces and occluded façades.
- Pipeline Overview: GridFlow takes a 150m×150m satellite image as its primary input and can additionally use semantic maps and DSM to generate a colored 100,000-point cloud.Structured condition dropout trains the model to operate with any subset of available modalities.
- Pipeline Overview: The three-stage pipeline uses a Grid-Aligned VAE, a conditional latent generator, and an orientation-aware colorization model.Separating stages avoids joint geometry-color optimization and lets each stage use a suitable generative framework.
- Grid-Aligned VAE: The Grid-Aligned VAE represents each tile as a 64×25×25 latent grid whose tokens correspond one-to-one with 6m×6m physical regions.The 25×25 grid exploits local spatial regularity while preserving spatial correspondence for conditioning.
- Grid-Aligned VAE: The VAE objective combines bidirectional Chamfer distance, height-histogram regularization, boundary Chamfer distance, and low-weight KL regularization.The KL term uses λKL=10^-5, prioritizing reconstruction.
- Conditional Latent Generation: A conditional rectified-flow model generates geometry latents from fused satellite, semantic, and DSM features aligned to the latent grid.The model transports Gaussian noise toward normalized VAE latents using a Transformer velocity network and global and local condition paths.
- Conditional Latent Generation: Latent-space edge consistency matches boundary columns or rows of adjacent tile latents instead of decoded point clouds.With a two-column boundary margin, matching 64-dimensional tokens implicitly aligns approximately 6,400 boundary points.
- Orientation-Aware Colorization: Orientation-aware colorization uses satellite and geometric conditioning for horizontal surfaces and learned texture priors for occluded vertical façades.An orientation-aware blending network combines the two paths, while diffusion training adds local smoothness and vertical texture-gradient objectives.
5 Experiments
GridFlow achieves strong geometry quality, conditioning fidelity, seamless tiling, and cross-city generalization in city-scale point-cloud generation experiments.
- Geometry Comparison: GridFlow achieves the best performance across all five geometry metrics, reducing Chamfer Distance by 29% relative to PVD and reaching an F-Score of 0.884.
- Visual Comparison: GridFlow produces detailed building structures and sharp boundaries, while CityDreamer is location-agnostic and Skyfall-GS produces blurred reconstructions.
- Geometry Comparison: GridFlow’s JSD is 0.010 versus PVD’s 0.037, while COV-CD reaches 51.8% versus 24.4%, indicating stronger distributional coverage.
- Visual Comparison: Rendered comparisons report GridFlow at SSIM 0.70 and FID 58.52 against the conditioning satellite imagery, which provides a structural advantage.
- Large-Tile Visualization: Latent-space edge consistency enables independently generated tiles to continue buildings, roads, and vegetation across boundaries without visible seams.
- Cross-Dataset Generalization: Without retraining, Melbourne-trained GridFlow achieves CD 0.00394 and F-Score 0.783 on London data, showing cross-city generalization under domain shift.
- Ablation: Conditioning Modalities: Removing any single conditioning modality degrades CD by 25–43%, while removing DSM lowers COV-CD to 23.0%.
6 Conclusion
GridFlow combines structured latent representations, latent-space edge consistency, and staged generation to produce dense colored city-scale point clouds. It also introduces a two-city benchmark and reports strong geometry and seamless-tiling results, while identifying cross-city generalization and training-stage accumulation as limitations.
- Method: GridFlow generates dense colored city-scale point clouds using a Grid-Aligned VAE, conditional rectified flow, and orientation-aware diffusion colorization.The framework separates geometry synthesis from colorization and uses structured latent grids for spatial conditioning.
- Benchmark: City3D-MultiGen contains 163K tiles from Melbourne and London with aligned point clouds, satellite images, semantic maps, and elevation data.The benchmark supports standardized evaluation across multiple remote-sensing modalities and cities.
- Seamless generation: GridFlow produces continuous geometry across independently generated Melbourne tiles, with buildings, roads, and vegetation continuing naturally across boundaries.The visualization demonstrates practical city-scale deployment of latent-space edge consistency.
- Results: GridFlow outperforms existing point cloud generation baselines across all geometry metrics and produces visually coherent colored outputs over arbitrarily large urban extents.The reported outcome combines quantitative geometry gains with qualitative color coherence and seamless boundaries.
- Limitations: Cross-city generalization degrades moderately, while sequential training may accumulate errors and fixed tile size or cell resolution may not suit every urban morphology.The reported London Chamfer Distance changes from 0.00272 to 0.00394, and façade colorization lacks direct vertical supervision from satellite imagery.