PaperlayerView exact revision on arXiv

Source-linked AI summary

Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery

Jie-Ying Lee, Yi-Ruei Liu, Shr-Ruei Tsai, Wei-Cheng Chang, Chung-Ho Wu, Jiewen Chan, Zhenjun Zhao, Chieh Hubert Lin, Yu-Lun Liu

arXiv:2510.15869v4cs.CV

TL;DR

Realistic, large-scale, explorable 3D urban scenes are difficult to build because high-quality real-world 3D training data are costly and limited. Skyfall-GS uses satellite reconstruction plus open-domain diffusion refinement with curriculum-based iterative updates, and reports more consistent geometry, realistic textures, real-time exploration, and stronger generalization than state-of-the-art methods.

  • Problem

    Building generalizable models for large-scale, geometrically accurate, explorable 3D cities is difficult because detailed 3D and textured reconstructions are limited, expensive, and labor-intensive.

  • Method

    Skyfall-GS combines satellite-specific 3D Gaussian Splatting with open-domain diffusion priors in a curriculum-based iterative refinement framework.

  • Results

    Skyfall-GS reports improved cross-view-consistent geometry, more realistic textures, better generalization and robustness than state-of-the-art methods, with real-time interactive rendering.

  • Takeaways & Limitations

    The framework supports scalable, automated creation of immersive urban virtual scenes for virtual entertainment, simulation, and robotics.

  • Takeaways & Limitations

    The method may leave blind spots at extreme street-level views, does not cover strict pedestrian-level synthesis, and remains challenged by purely nadir input.

Abstract

from arXiv · show

Synthesizing large-scale, explorable, and geometrically accurate 3D urban scenes is a challenging yet valuable task for immersive and embodied applications. The challenge lies in the lack of large-scale and high-quality real-world 3D scans for training generalizable generative models. In this paper, we take an alternative route to create large-scale 3D scenes by leveraging readily available satellite imagery for realistic coarse geometry and open-domain diffusion models for high-quality close-up appearance synthesis. We propose Skyfall-GS, a novel hybrid framework that synthesizes immersive city-block scale 3D urban scenes by combining satellite reconstruction with diffusion refinement, eliminating the need for costly 3D annotations, and also featuring real-time, immersive 3D exploration. We tailor a curriculum-driven iterative refinement strategy to progressively enhance geometric completeness and photorealistic texture. Extensive experiments demonstrate that Skyfall-GS provides improved cross-view consistent geometry and more realistic textures compared to state-of-the-art approaches. Project page: https://skyfall-gs.jayinnn.dev/

1 Introduction

Skyfall-GS addresses the difficulty of creating immersive, navigable 3D urban scenes from limited 3D data by using satellite imagery and diffusion priors. Its satellite-specific reconstruction and curriculum-based iterative refinement improve geometric completeness, texture realism, and cross-view consistency without domain-specific 3D training.

  • Motivation: Limited large-scale 3D reconstructions, camera-registration challenges, and noisy Internet imagery make realistic, navigable 3D city generation difficult.Satellite imagery offers broad coverage and high resolution, but limited parallax and invisible building facades cause incorrect geometry and artifacts.
  • Skyfall-GS: Skyfall-GS combines satellite reconstruction with diffusion refinement to synthesize immersive, real-time, freely navigable urban scenes from satellite imagery alone.The framework avoids fixed-domain 3D training and supports interactive rendering.
  • Results: Experiments report better generalization and robustness than state-of-the-art methods, while ablations show that each component improves perceptual plausibility and semantic consistency.The paper positions the method as a route toward scalable urban virtual scene creation for entertainment, simulation, and robotics.
  • Reconstruction: Satellite-specific 3DGS pairs entropy-based opacity regularization with pseudo-camera depth supervision to address limited parallax and multi-date imagery.These designs are tailored to the satellite-to-ground setting.
  • Synthesis: A curriculum-based iterative dataset update progressively distills open-domain diffusion priors into photorealistic, cross-view-consistent facades in occluded regions.Refined renderings provide pseudo ground truth for iterative supervision, improving degraded ground-level views caused by insufficient parallax.

2 Related Work

Prior work spans satellite reconstruction, urban scene synthesis, and diffusion-based 3D reconstruction, but the cited approaches address different parts of the satellite-to-ground problem. Skyfall-GS combines these directions in a two-stage pipeline for immersive city-block-scale scenes.

  • Motivation: Existing satellite novel-view methods produce blurred or distorted facades, while city-generation methods yield oversimplified geometries and unrealistic appearances under strong data assumptions.The comparison motivates combining satellite reconstruction with open-domain diffusion information.
  • Satellite and aerial 3D reconstruction: Satellite reconstruction methods use SfM-MVS, neural reconstruction, NeRF, RPC warping, multispectral data, or 3DGS, but cited methods do not recover occluded facades.The related methods target geometric reconstruction from satellite or aerial imagery, with varying strategies for sparse and multi-date views.
  • Urban scene synthesis: Urban scene synthesis methods generate ground-level imagery or use point clouds, density fields, voxels, diffusion, lifting, or BEV-conditioned representations as intermediate structures.The cited literature includes both methods without explicit 3D representations and approaches toward explicit 3D generation.
  • Diffusion models for 3D reconstruction and editing: Diffusion-based 3D work includes score distillation, Gaussian Splatting extensions, sparse-view consistency methods, inversion-based editing, and occlusion-aware inpainting.These approaches provide generative priors and editing mechanisms for reconstructing or modifying 3D scenes.
  • Skyfall-GS: Skyfall-GS uses a two-stage pipeline that fits an appearance-aware, regularized 3DGS model before recovering occluded regions through curriculum-based iterative diffusion refinement.The loop repeatedly refines renders while preserving satellite-faithful textures and geometry for navigable scenes.

3 Method

Skyfall-GS reconstructs urban scenes from satellite imagery and progressively refines them with diffusion-guided, curriculum-based iterative updates. The pipeline addresses sparse-view artifacts, occlusions, appearance variation, and cross-view consistency through specialized reconstruction losses and staged synthesis.

  • Skyfall-GS uses a two-stage pipeline: satellite-view 3D Gaussian Splatting reconstruction followed by curriculum-based iterative diffusion refinement of occluded ground-view regions.The reconstruction stage models illumination and regularizes sparse multi-date views; the synthesis stage repeatedly refines rendered views while preserving satellite-ground consistency.
  • Initial 3DGS reconstruction: The satellite reconstruction combines appearance embeddings, opacity regularization, pseudo-camera depth supervision, and a weighted loss for color, opacity, and depth.Appearance modeling handles illumination and localized changes, while entropy-based opacity regularization sharpens geometry and improves the synthesis foundation.
  • Curriculum learning strategy: Curriculum refinement starts from higher elevations and progressively lowers viewpoints, revealing occluded regions while improving geometric detail and texture realism.Initial satellite-trained 3DGS renders are higher quality at high elevations but degrade at lower elevations, motivating the descending viewpoint schedule.
  • Render refinement by text-to-image diffusion model: Diffusion refinement edits blurry, artifact-laden 3DGS renders to synthesize disocclusions, remove artifacts, and enhance geometry while preserving structural content.The method uses prompt-to-prompt editing with a pretrained text-to-image diffusion model.
  • Multiple diffusion samples: Independent diffusion edits can produce inconsistent views, so Skyfall-GS samples multiple refined outputs per viewpoint and optimizes a consensus representation across them.Photometric optimization implicitly averages the samples, balancing individual-view fidelity with geometric coherence.

4 Experiments

Skyfall-GS is evaluated on DFC2019 and GoogleEarth against reconstruction and city-generation baselines using distribution, pixel-level, qualitative, and user-study measures. It outperforms baselines on generative quality, while ablations identify the contributions of reconstruction supervision, multi-sample refinement, curriculum scheduling, and iterative updates.

  • Comparisons with Baselines: Skyfall-GS substantially outperforms all baselines on FIDCLIP and CMMD across the evaluated datasets, while remaining competitive on pixel-level metrics.Mip-Splatting achieves a slightly better average LPIPS on GoogleEarth, and CoR-GS sometimes achieves higher SSIM on DFC2019.
  • Comparisons with Baselines: Qualitative comparisons show sharper building contours, higher texture fidelity, fewer artifacts, and recovery of challenging facade details than competing methods.The comparisons include satellite-reconstruction and city-generation baselines on DFC2019 and GoogleEarth.
  • Comparisons with Baselines: User studies with 44 participants per study find Skyfall-GS strongest in geometric accuracy, spatial alignment, and overall perceptual quality across both datasets.Participants compared the method with satellite-reconstruction baselines in one study and city-generation baselines in the other.
  • Ablation Studies: Opacity regularization and pseudo-camera depth supervision improve geometry, while their combination achieves the lowest MAE/RMSE in LiDAR-based DSM evaluation.Appearance modeling is reported as crucial for multi-date convergence, and the full reconstruction-stage combination yields the lowest FIDCLIP and CMMD.
  • Ablation Studies: Multi-sample diffusion and high-to-low curriculum refinement improve synthesis, with Ns = 2 selected for visual quality and random or reversed schedules performing worse.Replacing refinement with SDEdit causes significant quality degradation, while iterative updating and multi-sample consensus improve consistency.
  • Performance and Scalability: The full pipeline takes approximately 6h 45min per scene offline, while the resulting representation supports real-time rendering and combined multi-block models reduce FIDCLIP by approximately 36–38%.The offline cost is reported as a one-time expense that amortizes across unlimited renders.

5 Conclusion

The paper concludes that Skyfall-GS can synthesize real-time, immersive, freely navigable urban scenes from multi-view satellite imagery without human intervention or domain-specific 3D training data. It reports consistent superiority over state-of-the-art baselines, while identifying low-elevation close-up views, blind spots, and purely nadir imagery as limitations.

  • Conclusion: Skyfall-GS synthesizes real-time, immersive, freely navigable 3D urban scenes solely from multi-view satellite imagery without human intervention or domain-specific 3D training data.The framework combines 3D Gaussian Splatting with open-domain diffusion priors in curriculum-based iterative refinement.
  • Conclusion: Experiments show consistent improvements over state-of-the-art reconstruction and generation baselines while addressing limited parallax, illumination variation, and large-scale occlusion.The supplementary free-flight renders use withheld smooth camera paths to assess globally coherent exploration.
  • Limitations: Strict pedestrian-level facade-close synthesis is beyond scope, and purely nadir input remains an open challenge because low-elevation captures and facade reconstruction are constrained.Fixed heuristic camera trajectories may also leave blind spots in complex geometries, causing minor artifacts at extreme street-level views.

B.1 Method Components

The method components combine satellite-view 3DGS reconstruction with pseudo-camera depth supervision and FlowEdit-based diffusion refinement. Curriculum-based iterative updates progressively expose degraded views while using multi-view consistency and appearance modeling to improve geometry and texture.

  • Pseudo-Camera Depth Supervision: Pseudo-camera depth supervision compares MoGe scale-invariant depth from rendered RGB images with rasterized 3DGS depth to constrain novel views.Cameras vary in azimuth, elevation, and radius, with 24 views sampled every 10 iterations.
  • 3DGS Reconstruction: Satellite-view reconstruction modifies 3DGS to address overhead imagery, including reduced scaling learning rates and pruning of unsuitable Gaussians.The reconstruction stage runs for 30,000 iterations with densification enabled between iterations 1,000 and 21,000.
  • FlowEdit-Based Refinement: FlowEdit refinement transforms degraded rendered views into sharper, more realistic images using source and target prompts while balancing artifact removal against geometry preservation.Lower noise preserves more original structure, whereas higher noise creates larger changes and can alter underlying geometry.
  • Curriculum-Based Refinement: The curriculum-based IDU process uses multiple refinement episodes and fixed appearance embeddings, while multi-view consistency helps mitigate floating artifacts.Opacity regularization is disabled during IDU so Gaussians can retain variable opacities for semi-transparent structures.

B.2 Resource and Approximation Analysis

Skyfall-GS combines practical rendering efficiency with substantial synthesis-time resource demands and uses a weak-perspective approximation for satellite imaging. The approximation is motivated by satellite altitude being much larger than scene depth variation.

  • Rendering Efficiency: The method achieves 11 FPS on an NVIDIA T4 and 40 FPS on a MacBook Air M2, while CityDreamer reaches 0.18 FPS on an NVIDIA A100.GaussianCity reaches 10.72 FPS but requires the higher-end A100.
  • Memory Consumption: Peak memory reaches 46 GB during synthesis, while final training memory is 28.04 GB after pruning redundant and low-opacity points.The peak is driven by loading FLUX.1 and temporary Gaussian densification.
  • RPC-to-Perspective Approximation: The RPC-to-perspective approximation relies on a weak-perspective assumption that is valid when satellite altitude Z is much larger than scene depth variation ΔZ.The paper adopts the SatelliteSfM methodology for this approximation.

C.1 Dataset Details

The evaluation uses DFC2019 and GoogleEarth datasets, supplemented by Jacksonville scenes with varied urban structures to test robustness across settings.

  • DFC2019 provides standard evaluation areas of interest, while four additional Jacksonville areas broaden scene diversity.
  • The additional Jacksonville areas include a city hall, an American football stadium, and other notable urban structures.
  • GoogleEarth supplies semantic maps, height fields, and Google Earth Studio renders across four areas containing complex architecture, squares, residential areas, and riverside scenes.

C.2 Evaluation Protocol

Evaluation combines user judgments, dataset-specific video comparisons, and controlled camera settings for quantitative comparisons against reconstruction and city-generation baselines.

  • Participants compare methods on geometric accuracy, spatial alignment, and overall perceptual quality against reference footage and satellite imagery.
  • The user studies compare Skyfall-GS with dataset-specific baselines, including Sat-NeRF and EOGS on DFC2019 and CityDreamer and GaussianCity on GoogleEarth.
  • Quantitative comparisons use fixed camera parameters within each dataset, with different elevation angles and radii for reconstruction and city-generation comparisons.

C.3 Results

Skyfall-GS achieves strong quantitative and qualitative results across datasets while preserving consistency with satellite-view geometry and handling varied urban scenes.

  • Skyfall-GS achieves the best average performance across both datasets, despite one flat GoogleEarth scene where Mip-Splatting performs better.
  • On DFC2019, Skyfall-GS obtains the best FIDCLIP and CMMD across all four areas, reducing average FIDCLIP from approximately 84 to approximately 27.
  • Skyfall-GS matches or exceeds Mip-Splatting on PSNR, SSIM, and LPIPS when reprojected into 62 held-out satellite views across four areas.
  • Additional scenes show coherent renderings that preserve satellite-derived spatial relationships and architectural features across diverse urban typologies.
  • Qualitative comparisons report superior geometry and texture, including sharper building details and fewer floating artifacts than competing methods.

D Additional Experiments

Additional experiments examine robustness to complex structures, appearance variation, prompt wording, scene scale, and iterative refinement, supporting the framework’s broader operating range.

  • Complex structures: Skyfall-GS synthesizes plausible views of irregular historical buildings and bridges, preserving complex geometry and reducing melting artifacts in thin structures.
  • Transient appearance: Per-image appearance embeddings make transient objects vary across renderings while keeping static building geometry consistent.
  • Prompt sensitivity: Refined render quality remains largely consistent across six prompt strategies, indicating tolerance to moderate prompt-specificity changes.
  • Multi-block scalability: The framework reconstructs a seamless approximately 1km × 512m urban region from combined adjacent areas without boundary-stitching artifacts.
  • Curriculum refinement: Iterative refinement improves coverage over episodes and substantially outperforms single-pass refinement because each episode updates the 3DGS inputs for later edits.
  • Consistency and diversity: Cross-view consistency improves as refinement episodes increase, while varying diffusion seeds produces diverse plausible textures over the same geometry.
Loading 2510.15869v4…