PaperlayerView exact revision on arXiv

Source-linked AI summary

Sat3DGen: Comprehensive Street-Level 3D Scene Generation from Single Satellite Image

Ming Qian, Zimin Xia, Changkun Liu, Shuailei Ma, Wen Wang, Zeran Ke, Bin Tan, Hang Zhang, Gui-Song Xia

arXiv:2605.14984v1cs.CVcs.AI

TL;DR

Generating street-level 3D scenes from single satellite images requires both rich semantics and reliable geometry, but existing approaches trade one for the other. Sat3DGen uses a geometry-first feed-forward framework with geometric constraints and perspective-view training, reducing geometric RMSE from 6.76m to 5.20m and FID from approximately 40 to 19 against Sat2Density++.

  • Problem

    Existing satellite-to-street methods either provide geometrically limited scenes with artifacts or lack the semantic richness needed for holistic street-level 3D generation.

  • Method

    Sat3DGen enhances feed-forward image-to-3D generation with geometry-focused constraints and perspective-view training to address structural errors and sparse supervision.

  • Results

    Geometric RMSE decreases from 6.76m to 5.20m, while FID decreases from approximately 40 to 19 against Sat2Density++ on the constructed VIGOR-OOD and DSM benchmark.

  • Takeaways & Limitations

    The generated 3D assets support semantic-map-to-3D synthesis, surround-view video generation, large-area meshing, and single-image DSM generation.

  • Takeaways & Limitations

    Comparisons with Sat2Scene and Sat2City rely on results reported in their papers and project pages because official implementations were not fully released.

Abstract

from arXiv · show

Generating a street-level 3D scene from a single satellite image is a crucial yet challenging task. Current methods present a stark trade-off: geometry-colorization models achieve high geometric fidelity but are typically building-focused and lack semantic diversity. In contrast, proxy-based models use feed-forward image-to-3D frameworks to generate holistic scenes by jointly learning geometry and texture, a process that yields rich content but coarse and unstable geometry. We attribute these geometric failures to the extreme viewpoint gap and sparse, inconsistent supervision inherent in satellite-to-street data. We introduce Sat3DGen to address these fundamental challenges, which embodies a geometry-first methodology. This methodology enhances the feed-forward paradigm by integrating novel geometric constraints with a perspective-view training strategy, explicitly countering the primary sources of geometric error. This geometry-centric strategy yields a dramatic leap in both 3D accuracy and photorealism. For validation, we first constructed a new benchmark by pairing the VIGOR-OOD test set with high-resolution DSM data. On this benchmark, our method improves geometric RMSE from 6.76m to 5.20m. Crucially, this geometric leap also boosts photorealism, reducing the Fréchet Inception Distance (FID) from $\sim$40 to 19 against the leading method, Sat2Density++, despite using no extra tailored image-quality modules. We demonstrate the versatility of our high-quality 3D assets through diverse downstream applications, including semantic-map-to-3D synthesis, multi-camera video generation, large-scale meshing, and unsupervised single-image Digital Surface Model (DSM) estimation. The code has been released on https://github.com/qianmingduowan/Sat3DGen.

1 INTRODUCTION

Sat3DGen targets the challenge of generating semantically rich, street-level 3D scenes from widely available satellite imagery while overcoming the geometric weaknesses of existing proxy-based methods. Its geometry-first approach improves both 3D accuracy and photorealism and enables diverse downstream applications.

  • Motivation: Satellite imagery motivates street-level 3D generation because it offers wide coverage, low cost, and frequent updates, unlike costly and uneven ground-level capture.Street-level 3D scenes support mapping, robotics, simulation, and media creation.
  • Limitations of Existing Methods: Existing approaches trade off semantic coverage and geometry: geometry-colorization methods produce clean buildings but miss non-building elements, while proxy methods preserve semantics but generate poor geometry.Proxy outputs exhibit degraded boundaries, unrealistic roofs, and floating artifacts.
  • Method: Sat3DGen introduces a holistic, geometry-first methodology that strengthens a general feed-forward image-to-3D framework with geometric constraints targeting vertical structures, floating artifacts, and footprint-mismatch boundary errors.The proposed losses include Gravity-based Density Variation Loss and a boundary-focused component.
  • Evaluation: 5.20m geometric RMSE, down from Sat2Density++’s 6.76m, demonstrates Sat3DGen’s improvement on a benchmark pairing VIGOR-OOD with 1-meter resolution DSM data.The comparison is against the leading method, Sat2Density++.
  • Results and Applications: 19 FID, reduced from Sat2Density++’s ∼40 on the VIGOR-OOD unseen-city split, shows that improved geometry also boosts photorealism.The resulting assets support semantic-map-to-3D synthesis, surround-view video, large-area mesh generation, and single-image DSM generation without ground-truth depth supervision.

2 RELATED WORKS

Related work spans feed-forward image-to-3D generation and satellite-to-street-view synthesis. Existing methods either primarily target object-level 3D assets or generate street-view images without usable 3D and multi-view consistency.

  • Feed-Forward Image to 3D Works: Feed-forward image-to-3D research has gained popularity for producing high-quality 3D assets, but existing works primarily focus on object-level generation.Large reconstruction models improve generated-asset quality through larger datasets, refined annotations, and larger models.
  • Single Satellite to Street-view Synthesis: Early satellite-to-street-view studies generate individual street-view images from a single satellite patch, without producing usable 3D or multi-view consistency.The cited early studies include Regmi and Borji, Toker et al., Shi et al., Lu et al., and Tang et al.
  • Single Satellite to Street-view Synthesis: Later works synthesize street-view videos by learning a colored 3D asset from the satellite input.The passage attributes this direction to Li et al. (2021; 2024b) and Qian et al. (2026).

3 METHOD

Sat3DGen uses a feed-forward tri-plane NeRF pipeline to convert one satellite image into a renderable, lighting-controllable 3D scene. Its geometry-first design adds gravity, satellite-depth, and panoramic-to-perspective constraints to improve structural fidelity and view coverage.

  • Rendering: The renderer supports satellite, arbitrary-pose perspective, and panoramic street views under controllable illumination, with sky appearance represented consistently on a spherical feature map.During training, illumination features come from ground-truth street-view panoramas; at test time, they control rendered lighting.
  • Rendering and mesh export: The learned field can be exported as a watertight mesh by evaluating density on a dense grid and applying Marching Cubes while excluding the sky branch.The same volumetric renderer handles perspective and spherical cameras, with the latter producing full panoramas.
  • Geometry-oriented components: Three geometry-oriented components target gravity-inconsistent structures, rooftop ambiguity, and sparse viewpoint supervision: gravity density regularization, satellite relative-depth regularization, and panoramic-to-perspective supervision.The gravity loss permits hollow or overhanging structures through a slack variable ϵ set to 1, while the depth prior uses pseudo labels from Depth Anything v2 without metric depth.
  • Backbone: Spatial-token padding expands the effective scene extent beyond the satellite crop, accommodating peripheral buildings and roads while stabilizing interior geometry.With a 16×16 token grid, padding produces a 20×20×1024 representation and can expand a 50 m scene cube to 62.5 m.

4 EXPERIMENTS

Experiments on VIGOR-OOD show that Sat3DGen improves street-level geometry, image realism, semantic faithfulness, and video quality through geometry-focused design choices. Ablations and downstream demonstrations further validate its components and versatility across DSM estimation, 3D reconstruction, meshing, and multi-camera video synthesis.

  • Datasets and Splits: Training uses 78,188 GPS-matched satellite–ground image pairs from Chicago, New York, and San Francisco, with Seattle held out for out-of-domain testing.VIGOR provides multiple street-view panoramas per satellite tile, with satellite zoom fixed at 20.
  • 3D Comparison: Sat3DGen qualitatively preserves road markings, crosswalks, medians, tree belts, and sidewalks better than Sat2Scene and Sat2City across diverse urban layouts.Compared with Sat2Density++, its design strengthens boundary layout, gravity alignment, rooftop depth cues, and viewpoint coverage through panorama-to-perspective supervision.
  • Image and Video Comparison: Sat3DGen leads in FID, KID, and DINO, with lower FID and KID than diffusion baselines because its view-consistent 3D representation improves realism and semantic correctness.The evaluation measures distributional similarity, semantic alignment, pixel-level fidelity, and structural similarity on VIGOR-OOD.
  • Geometric Evaluation: 5.20m RMSE and 3.47m MAE establish the full model as a new state of the art, improving over Sat2Density++ at 6.76m RMSE and Canonical Image-to-3D at 6.21m RMSE.The model also reconstructs 62.69% of the surface with an error of less than the threshold continued in the supplied passage.
  • Ablation Study: Adding Ldep, Lgrav, and Spatial Tokens raises FID performance from the Canonical Image-to-3D baseline’s 35.6 to 21.6, with Lgrav most critical for photorealism.Spatial Tokens regularize boundaries, Lgrav produces straighter facades and fewer floaters, and Ldep corrects rooftop geometry.
  • Applications: Applications include satellite-to-DSM conversion without ground-truth depth supervision, semantic-map-to-3D reconstruction, large-area meshing, and surround-view multi-camera video synthesis.Generated videos also reduce artifacts and smooth building and scene-boundary edges through higher-quality 3D representations.

5 CONCLUSION

Sat3DGen generates outdoor 3D scenes from satellite images using GPS-aligned satellite and panorama street-view data. Its gravity-based Density Variation Loss and spatial tokens improve generated 3D assets and videos.

  • 5 CONCLUSION: Sat3DGen generates outdoor 3D scenes from satellite images using exclusively GPS-aligned satellite and panorama street-view image data.The method is presented as a novel algorithm for satellite-to-3D scene generation.
  • 5 CONCLUSION: Gravity-based Density Variation Loss and spatial tokens improve the quality of generated 3D assets and videos.The conclusion attributes these improvements to the novel approaches proposed in the paper.

ETHICS STATEMENT

The work uses publicly available, licensed satellite and street-view data without collecting or releasing personally identifiable information, while acknowledging potential misuse for large-scale surveillance and limiting its stated intent to academic and positive applications.

  • Data and privacy: The study uses the publicly available VIGOR dataset and does not gather or release personally identifiable information.The dataset contains satellite images and street-view panoramas collected from widely accessible map platforms.
  • Data and privacy: All data sources follow the original dataset licenses and terms of use, without attempting to identify individuals or private properties beyond released benchmarks.
  • Potential misuse: The authors acknowledge that irresponsible deployment could enable large-scale surveillance despite beneficial applications in simulation, planning, and geographic visualization.
  • Risk mitigation: The framework is intended solely for academic research and positive social applications.

REPRODUCIBILITY STATEMENT

The authors emphasize reproducibility through standard frameworks, detailed methodological documentation, explicit dataset and evaluation specifications, and widely used benchmarks for fair comparison.

  • Implementation and documentation: The implementation uses standard deep learning frameworks, specifically PyTorch.The model architecture, training strategies, and evaluation protocols are described in Section 4 and Appendix F.
  • Experimental specification: Dataset splits, preprocessing steps, and evaluation metrics are specified in detail.
  • Benchmarking: The study adopts the widely used VIGOR benchmark to facilitate fair comparisons.

LLM USAGE … B.4 SEMANTIC MAPS TO 3D ASSETS.

The supplementary results clarify that LLMs were used only for language polishing, while Sat3DGen demonstrates improved geometry, depth reconstruction, video generation, scalable meshing, and semantic-map-driven 3D asset creation.

  • LLM USAGE: LLMs were used exclusively for grammar, style, and readability polishing, without contributing to research design, analysis, or conclusions.
  • A.1 MESH RESULTS COMPARED TO SAT2DENSITY++: Compared with Sat2Density++, Sat3DGen produces smoother, sharper, and more coherent 3D assets, including cleaner ground surfaces and fewer discontinuities across large meshes.The supplementary mesh results use VIGOR-OOD satellite images, while sliding-window inference supports approximately 150 m × 150 m areas and larger images are theoretically possible.
  • A.2 MESH RESULTS COMPARED TO CANONICAL IMAGE-TO-3D: Against Canonical Image-to-3D, Sat3DGen avoids noisy ground, indistinct object shapes, flat-rooftop failures, and spiky mesh boundaries, yielding more plausible geometry and more watertight meshes.
  • A.3 PANORAMA VIDEO RESULTS.: The supplementary video results contain 16 VIGOR-OOD test sets and compare Sat3DGen with Sat2Density++ using satellite-image and camera-trajectory overlays.
  • A.4 SATELLITE-VIEW DEPTH COMPARISON: Despite temporal misalignment and DSM noise from transient objects, Sat3DGen reconstructs sharper, more geometrically plausible depth than Sat2Density++, with flat rooftops and clean ground surfaces.The comparison notes that the DSM was collected over a year, making perfect reconstruction unattainable.
  • B.1 SINGLE SATELLITE IMAGE TO DSM (METRIC DEPTH): Sat3DGen renders satellite-view metric depth from its learned NeRF-based representation without using metric-depth annotations during training.
  • B.2 SURROUND-VIEW MULTI-CAMERA VIDEO GENERATION FROM A SINGLE SATELLITE IMAGE.: Sat3DGen generates diverse multi-view perspective videos from one satellite image without video or 3D-geometry training inputs, with variable FOV and image sizes and quality nearly on par with panoramas.The method uses NeRF-based 3D representations and four fixed 120-degree FOV perspectives are shown in the supplementary videos.
  • B.4 SEMANTIC MAPS TO 3D ASSETS.: From colored semantic maps, a diffusion model first generates satellite imagery, after which Sat3DGen produces spatially consistent 3D assets for applications including spatial planning and game modeling.Training pairs were collected from OpenStreetMap and Google Satellite Maps.

C IMPLEMENTATION DETAILS · D GROUND TRUTH DSM PREPARATION FOR VIGOR-OOD · D.1 DATA SOURCE AND JUSTIFICATION

The implementation uses controlled perspective sampling, volume-rendering settings, and loss-specific weighting to prioritize realism in perspective-view generation. Ground-truth DSMs for VIGOR-OOD are prepared from Seattle LiDAR data selected for temporal alignment, geographic coverage, and documented positional accuracy.

  • C IMPLEMENTATION DETAILS: Perspective images are randomly sampled with pitch [-30, 30], roll 0, yaw [-79, 179], FOV [90, 105, 120], and render size 256 × 256.These settings define the perspective-view training samples.
  • C IMPLEMENTATION DETAILS: The volume renderer uses a triplane dimension of 32 and 96 samples per ray.These are the stated volume-rendering configuration values.
  • C IMPLEMENTATION DETAILS: The Gravity-based Density Variation Loss has weight 3.5, while Satellite View Depth Regularization based on Midas loss has weight 0.1.Other reconstruction and adversarial losses retain Sat2Density++ weight settings.
  • C IMPLEMENTATION DETAILS: Perspective-view reconstruction loss is weighted at half the panorama reconstruction loss, while perspective-view GAN loss matches the panorama GAN loss.The weighting reflects perspective images’ detail and their large viewpoint difference from satellite inputs.
  • C IMPLEMENTATION DETAILS: The method prioritizes perspective-image realism and image quality through GAN constraints over visual consistency with ground-truth images targeted by reconstruction loss.The full code is stated to be released after acceptance.
  • D GROUND TRUTH DSM PREPARATION FOR VIGOR-OOD: Accurate 3D-geometry evaluation requires high-quality DSMs, which are scarce publicly and often limited to a few cities; Seattle VIGOR-OOD enabled corresponding high-precision DSM acquisition.The section presents a processing and alignment pipeline for creating ground truth, though the supplied passage is truncated.
  • D.1 DATA SOURCE AND JUSTIFICATION: Ground-truth DSMs come from the publicly available King County West 2021 dataset of Quality Level 1 LiDAR collected in spring and summer 2021.Its acquisition period closely matches the creation period of the VIGOR dataset.
  • D.1 DATA SOURCE AND JUSTIFICATION: Six large GeoTIFF tiles collectively cover the geographic extent of the VIGOR-OOD Seattle test images.The supplied passage states that this coverage is visualized in Figure 12.

D.2 METHODOLOGY FOR SATELLITE-DSM ALIGNMENT · E MORE ABLATIONS · E.1 VARIATION REGULARIZATION ABLATION.

Sat3DGen constructs aligned DSM ground truth by reconciling disparate spatial references and reprojection grids, enabling pixel-wise geometric evaluation. Ablations show that variation regularization, especially Lgrav with ϵ = 1.0, improves FID and KID over alternatives.

  • D.2 METHODOLOGY FOR SATELLITE-DSM ALIGNMENT: The DSM alignment challenge arises because raw GeoTIFFs and VIGOR-OOD images differ in CRS, resolution, and spatial alignment.A dedicated processing pipeline addresses these incompatibilities.
  • D.2 METHODOLOGY FOR SATELLITE-DSM ALIGNMENT: VIGOR metadata supplies WGS84 coordinates and Google Maps zoom levels, enabling geographic bounding-box estimation for each satellite image.The corresponding high-resolution DSM region is then reprojected onto the image’s exact pixel grid.
  • D.2 METHODOLOGY FOR SATELLITE-DSM ALIGNMENT: Algorithm 1 processes each VIGOR-OOD image by parsing filename metadata, estimating its WGS84 footprint, and producing a ground-truth DSM array from raw GeoTIFF tiles.The pipeline initializes an in-memory spatial index for fast tile lookup.
  • D.2 METHODOLOGY FOR SATELLITE-DSM ALIGNMENT: The automated pipeline yields aligned, unit-corrected, quality-controlled DSMs for direct pixel-wise comparison in geometric accuracy evaluation.Figure 8 visualizes paired satellite images and corresponding DSM data.
  • E.1 VARIATION REGULARIZATION ABLATION.: Removing variation regularization produces the worst results, with FID 25.90 and KID 19.0.These results use the base model without perspective training and remove Lgrav.
  • E.1 VARIATION REGULARIZATION ABLATION.: ϵ = 1.0 with Lgrav performs best overall, achieving FID 21.60 and KID 16.2.The passage attributes degradation at ϵ = 0 to over-penalization of genuine voids such as gaps beneath tree canopies.

F DATASET PREPARATION AND DETAILS

The study uses VIGOR for training and evaluation, with Seattle reserved as an unseen-city OOD test set. CVACT and CVUSA are excluded because their supervision is geometrically insufficient and their acquisition differences confound fair OOD evaluation.

  • VIGOR: VIGOR covers four cities with multiple relative-pose street-view panoramas per satellite image at zoom level 20.Training uses Chicago, New York, and San Francisco, while Seattle is reserved for unseen-city OOD testing.
  • VIGOR: 78,188 satellite–panorama pairs support training, and 11,875 pairs support metric evaluation.
  • Other Datasets: CVACT and CVUSA provide only one centered ground-level image per satellite tile, lacking the multi-view coverage needed for consistent street-level 3D geometry.Their sparse supervision makes the method’s geometry-learning optimization components ineffective.
  • Other Datasets: CVACT and CVUSA are unsuitable for fair OOD evaluation because different satellites and zoom levels introduce appearance and pixel-resolution shifts beyond geometric generalization.
  • Generalization Evaluation: The VIGOR-OOD split trains on three cities and tests on unseen Seattle, creating domain gaps in urban layouts and architectural styles to assess robustness.

G LIMITATIONS

Sat3DGen is limited by inaccurate camera poses, assumptions about architecture and terrain, and evaluation constraints imposed by VIGOR’s sparse still-image collection. These limitations reflect challenges in both data availability and model assumptions.

  • Pose Inaccuracy: The method lacks precise pose data, treating satellite images as ideal orthogonal projections and panoramas as perpendicular to the ground despite missing intrinsic/extrinsic parameters and roll angles.Only GPS data are available for panoramas, so terrain- or road-banking-induced roll is ignored.
  • Geometric and Terrain Assumptions: The generative model struggles with atypical architectures because rare structures lack explicit 3D ground-truth shapes for supervision.This limitation is tied to the assumptions underlying the model and the distribution of its training data.
  • Geometric and Terrain Assumptions: The framework assumes locally flat ground and does not model significant terrain variations such as hills, which are difficult to infer from sparse imagery alone.The passage identifies terrain modeling as a direction for future work.
  • Evaluation Metrics: VIGOR’s sparse, non-sequential still images prevent evaluating temporal flicker and dense multi-view photometric consistency.These metrics are powerful but inapplicable to the VIGOR dataset because it lacks sequential observations and dense view coverage.
Loading 2605.14984v1…