PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing through Satellite Images at Street Views

Ming Qian, Bin Tan, Qiuyu Wang, Xianwei Zheng, Hanjiang Xiong, Gui-Song Xia, Yujun Shen, Nan Xue

arXiv:2505.17001v2cs.CV

TL;DR

SatStreet-view synthesis asks how to render photorealistic, view-consistent street panoramas and videos from sparse satellite–street pairs despite extreme viewpoint changes and street-only sky and illumination cues. Sat2Density++ learns an illumination-adaptive radiance field while separately generating sky and ground components, and experiments on urban and suburban datasets report improved image, geometry, and video quality without 3D annotations. The method remains bounded by difficult image alignment and its use of simple sky-histogram illumination cues.

  • Problem

    SatStreet-view synthesis must infer 3D scenes and render street views from sparse satellite–street pairs with extreme viewpoint changes and street-specific elements absent from satellite images.

    Source: §1 · pp. 1–2

  • Method

    Sat2Density++ learns an illumination-adaptive neural radiance field for ground content and a separate illumination-sensitive 2D sky generator from paired satellite and street-view images.

    Source: §1 · p. 3, §3.1.2 · p. 5

  • Results

    Sat2Density++ renders photorealistic, multi-view consistent street-view videos and improves image quality, geometric accuracy, faithfulness, consistency, and quality over prior methods and Sat2Density without 3D annotations.

    Source: §1 · p. 3, §6 · p. 16

  • Takeaways & Limitations

    The results support satellite-conditioned, feedforward 3D-aware generation of street-view panorama videos across suburban and urban scenes.

    Source: §6 · p. 16

    The method depends on challenging alignment and pose calibration, including non-nadir satellite views and variable ground-image altitude, and uses simple sky-region histograms rather than richer illumination cues.

    Source: §5 · p. 16

Abstract

from arXiv · show

This paper studies the task of SatStreet-view synthesis, which aims to render photorealistic street-view panorama images and videos given any satellite image and specified camera positions or trajectories. We formulate to learn neural radiance field from paired images captured from satellite and street viewpoints, which comes to be a challenging learning problem due to the sparse-view natural and the extremely-large viewpoint changes between satellite and street-view images. We tackle the challenges based on a task-specific observation that street-view specific elements, including the sky and illumination effects are only visible in street-view panoramas, and present a novel approach Sat2Density++ to accomplish the goal of photo-realistic street-view panoramas rendering by modeling these street-view specific in neural networks. In the experiments, our method is testified on both urban and suburban scene datasets, demonstrating that Sat2Density++ is capable of rendering photorealistic street-view panoramas that are consistent across multiple views and faithful to the satellite image.

1 INTRODUCTION

SatStreet-view synthesis seeks photorealistic, view-consistent street panoramas and videos from sparse satellite–street image pairs despite extreme viewpoint changes and street-level elements absent from satellite views. Sat2Density++ addresses these challenges by learning illumination-adaptive radiance fields and separately modeling ground and sky, improving geometric and video fidelity over Sat2Density.

Task and challenges

  • SatStreet-view synthesis renders photorealistic street-view panoramas and videos from satellite images and specified camera positions or trajectories despite extreme satellite-to-ground viewpoint changes.The paired setting uses one satellite image and one street-view panorama to learn faithful 3D geometry and queried BEV trajectories.Source: §1 · p. 1
  • The task must learn neural fields from sparse two-view pairs while handling sky, illumination, and detailed structures visible only from street level.These elements are not directly observable in satellite imagery, complicating both field learning and rendering.Source: §1 · pp. 1–2

Sat2Density++

  • Sat2Density++ replaces Sat2Density’s density-only representation with an illumination-adaptive neural radiance field that jointly models scene geometry and appearance.The learned density is intended to remain invariant across lighting scenarios while color adapts to explicit illumination.Source: §1 · p. 3
  • The method separates street-view generation into neural-field ground rendering and illumination-sensitive 2D sky generation before combining them into panoramas.This design directly models street-view-specific sky and illumination effects.Source: §1 · p. 3

Results

  • Sat2Density++ outperforms existing single-image generation models in image quality and improves geometry, faithfulness, consistency, and quality over Sat2Density without 3D annotations.The reported validation includes suburban and urban datasets, predicted satellite-view depth, interpolated illumination, and out-of-distribution data.Source: §1 · p. 3

2 RELATED WORKS

Prior work addresses satellite–ground viewpoint changes through image transformations, auxiliary supervision, and conditional 3D generation, but often lacks intrinsic 3D modeling or requires costly annotations. Sat2Density++ uses paired aerial and ground images to learn 3D geometry and appearance for reliable view-consistent video generation.

Cross-view synthesis

  • Previous approaches model drastic viewpoint changes with homographies, depth or semantic supervision, transformation matrices, and geospatial attention, often focusing on image-level correlations.These methods can be effective but may overlook intrinsic correspondences in 3D space.Source: §2.1 · p. 3

Sat2Density++

  • The framework aims to learn intricate 3D geometries and appearances from latent correspondences in cross-view pairs, enabling reliable view-consistent video generation.Source: §2.1 · p. 3

3D scene generation

  • Semantic voxel conditioning represents 3D occupancy more completely than 2D semantic BEV maps, but refined voxel annotation requires extensive manual labor.The annotation burden increases with the number of semantic categories.Source: §2.2 · p. 3

Sat2Density++

  • Sat2Density++ exploits satellite imagery as environmental context and a condition for precise 3D geometry estimation in street-view synthesis.The framework is trained on paired aerial and ground images and targets correspondence-consistent generation.Source: §2.1 · p. 3, §2.2 · p. 4

3 THE PROPOSED SAT2DENSITY++

Sat2Density++ decouples satellite-conditioned ground-scene representation from sky-background generation while allowing illumination to influence both components. This supports rendering ground and sky consistently under varying daylight conditions.

Scene decomposition

  • The satellite image generates a learnable implicit 3D representation of outdoor ground scenes, while the sky is modeled as a 2D panoramic background from illumination input.Camera parameters determine the rendered ground content for satellite or street views.Source: §3 · p. 4

Illumination adaptation

  • Illumination input controls sky appearance and also adapts the rendered ground appearance to varying daylight conditions.The model therefore decouples sky generation from 3D scene generation while retaining shared lighting effects.Source: §3 · p. 4

3.1 The 3D Representation

The 3D representation uses satellite-derived tri-plane features queried along camera rays and decoded into density and color, while sky histograms provide compact illumination conditioning. An illumination-adaptive decoder modulates appearance without changing the density branch across lighting conditions.

Tri-plane representation

  • Satellite imagery is transformed into tri-plane feature maps that represent the outdoor scene within a tri-plane radiance field.The feature maps are split into three spatial planes and queried to obtain point features.Source: §3.1.1 · p. 4

Radiance-field rendering

  • For each sampled 3D location, queried tri-plane features are decoded into density and color values used to volume-render street-view ground content.Source: §3.1.1 · pp. 4–5

Inference process

  • Different sky illumination inputs can generate street-view images or videos under varied daylight conditions from one satellite image.The paper uses random training-set illumination or corresponding ground-truth illumination for quantitative rendering tests.Source: §3.1.2 · p. 5

Illumination-adaptive decoder

  • The illumination feature is mapped to a style vector that conditions an adaptive decoder producing density and appearance features from each tri-plane point feature.The style vector has dimension 512 and is produced by an eight-layer fully connected mapping network.Source: §3.1.3 · p. 5
  • Unlike the vanilla decoder, the adaptive decoder uses separate density and color branches, injecting illumination only into the color branch.This design addresses varying lighting conditions while preserving the distinction between geometry and appearance.Source: §3.1.3 · p. 5

3.2 Street-View Image Generation

Street-view generation combines volumetric ground rendering with a learned sky component, alpha blending, and super-resolution to produce high-resolution panoramas from camera parameters.

Ground Part Generation

  • Volumetric rendering produces ground features, color, opacity, and depth from sampled points along camera rays.The rendered output uses the camera intrinsics to determine image size, with practical ground renderings at 64 × 256.Source: §3.2.1 · pp. 5–6

Sky Part Generation and Alpha Blending

  • A 2D sky generator produces sky features from an illumination vector, while the ground and sky are combined through opacity-weighted alpha blending.The blended feature is computed as O_grd × F_grd + (1 − O_grd) × F_sky.Source: §3.2.2 · p. 6

Street-View Super Resolution

  • The blended feature’s first three layers provide the raw street-view color, which is then passed to a lightweight super-resolution module.The super-resolution stage reduces the computational cost of volumetric rendering before producing the final high-resolution image.Source: §3.2.2 · p. 6

3.3 Training-Time Satellite-View Generation

Training-time satellite-view generation adds satellite-view supervision by rendering the learned 3D representation from an approximated orthographic satellite camera without street-view-specific sky or super-resolution modules.

Training-Time Satellite-View Generation

  • The model renders the satellite view during training to align the learned 3D representation with the input satellite image.Because datasets lack satellite camera intrinsics and extrinsics, the satellite camera is approximated as vertically downward-facing and orthographic.Source: §3.3 · p. 6
  • Satellite rendering omits the sky and super-resolution modules because they are unnecessary for satellite-view images.A null illumination style vector disables street-view-specific style modulation when generating satellite images.Source: §3.3 · p. 6

3.4 Learning Objectives

The training objective combines reconstruction, non-sky opacity, and adversarial losses to supervise satellite views, street views, sky regions, and realistic novel-view renderings.

Loss Design

  • Paired satellite and street-view images, together with pseudo sky masks, supervise reconstruction, opacity, and discriminator losses.The pseudo masks come from an off-the-shelf sky segmentation model.Source: §3.4 · p. 6

Non-sky Opacity Loss

  • Binary cross-entropy on pseudo sky masks regularizes the density field by assigning opacity to ground regions and suppressing it in sky regions.The mask convention uses 1 for sky and 0 for ground, so the target opacity is 1 − M_sky.Source: §3.4.1 · p. 6

Reconstruction Loss

  • The reconstruction losses supervise satellite images, sky regions, and complete high-resolution street-view panoramas.Sky reconstruction uses an L1 loss only at pixels selected by the sky mask, while street-view reconstruction combines LPIPS and L1 losses.Source: §3.4.2 · pp. 6–7

Loss Design

  • The final reconstruction objective weights satellite, street-view, and sky losses, while satellite images are rendered at one-quarter resolution with random crops during training.The loss weights λ_sat, λ_str, and λ_sky balance the three reconstruction terms.Source: §3.4.2 · p. 7

Discriminator Loss

  • Two discriminators enforce realistic renderings for street and satellite views because reconstruction loss alone does not produce detailed novel-view results.The street-view discriminator operates on a six-channel concatenation of the final image and raw neural rendering to maintain high-resolution and view-consistent outputs.Source: §3.4.3 · p. 7

4 EXPERIMENTS

Sat2Density++ is evaluated on suburban and urban datasets using image-level metrics, qualitative video comparisons, and user studies. It consistently improves perceptual quality, realism, and video quality, consistency, and faithfulness, with larger gains in complex urban scenes.

Comparison on Video Results

  • 79%/82%/79% approval on CVACT and 95%/95%/92% on VIGOR for Quality, Consistency, and Faithfulness favor Sat2Density++ over Sat2Density.The user study covers suburban CVACT scenes and more complex urban VIGOR scenes.Source: §4.4.1 · p. 8
  • Sat2Density++ maintains more stable and coherent videos than Sat2Density, especially for trees, crosswalks, buildings, and other fine-grained structures.The qualitative comparisons cover CVACT and VIGOR trajectories with multiple sampled camera positions.Source: Figure 3 · p. 8, §4.4.1 · p. 9, Figure 4 · p. 9
  • Sat2Density++ generates diverse scene elements such as trees, green belts, and landmarks without requiring height maps or metric-depth supervision.Prior video methods could not be directly compared because their full datasets and code were unavailable.Source: §4.4.1 · p. 9

Comparison on Image Results

  • Across CVUSA, CVACT, VIGOR, and VIGOR-OOD, Sat2Density++ outperforms existing methods on perceptual, semantic, and realism metrics, with larger gains on complex urban test sets.The comparison includes randomly sampled illumination inputs, marked by ‘*’ in Tab. 1.Source: §4.4.2 · p. 9
  • Real illumination improves perceptual, PSNR, and SSIM scores while FID and KID remain stable, showing that low-level metrics are sensitive to appearance changes.The result supports using distributional realism metrics alongside low-level similarity metrics when evaluating illumination-controllable synthesis.Source: §4.4.2 · pp. 9–10

4.5 Generalizability of sky generation

Sky-region histograms support plausible sky synthesis in both in-domain and out-of-domain settings, while illumination-aware components improve geometry, visual quality, and controllability. Ablations show that separating sky from ground and preserving satellite-view supervision are important for faithful, consistent generation.

Sky generation

  • Out-of-domain sky inputs still produce plausible tone, lighting, and cloud distribution, indicating robust generalization from sky-region illumination histograms.In-domain resemblance can be higher when training and test captures share similar lighting conditions.Source: Figure 6 · p. 11, §4.5 · p. 11

Sky illumination modeling

  • Removing illumination modeling produces larger boundary deviations and angular depth artifacts, degrading 3D representations and video generation.Using random noise for both views performs worst, while a null-style satellite vector provides only a slight improvement when street-view compensation remains absent.Source: §4.6.1 · p. 11
  • The full illumination configuration achieves the best performance by combining a null-style satellite input with a valid street-view sky histogram.This configuration leverages illumination cues from both views for more accurate geometry, better visual quality, and controllable synthesis.Source: §4.6.1 · pp. 11–12

Sky branch

  • Removing the sky branch forces the tri-plane NeRF to represent finite ground and infinitely distant sky together, reducing visual realism as camera motion changes the rendered sky.The limited spatial range makes the sky acquire incorrect depth and increases the model’s learning burden.Source: Table 3 · p. 11, §4.6.3 · pp. 12–13, Figure 10 · p. 14

Loss design

  • The non-sky opacity and sky reconstruction losses separate 3D ground learning from 2D sky generation, whereas removing them causes region-generation errors and inaccurate depths.Without these losses, the sky generator can render ground features, and generated videos can exhibit static roads or sky misalignment.Source: §4.6.4 · pp. 13–14

Satellite and adversarial supervision

  • Satellite-view losses improve faithfulness to input imagery, while GAN loss can improve visual quality even when standard metrics do not fully reflect it.Without satellite supervision, crosswalks, trees, buildings, and zebra crossings diverge from the satellite image; without GAN loss, metric scores may rise despite visibly poorer results.Source: Figure 8 · p. 13, Table 5 · p. 13, Table 4 · p. 13

4.7 Applications

The model supports illumination-controllable street-view synthesis, satellite-depth prediction, and synthesized video generation. It also maintains multi-view consistency and city-layout fidelity in zero-shot evaluation on Seattle imagery.

Illumination-controllable street-view image synthesis

  • Manipulating the illumination feature enables controllable street-view image synthesis under varying lighting conditions.The application is illustrated through illumination interpolation and image-quality comparisons.Source: §4.7 · p. 14, Figure 12 · p. 15

Applications

  • The model supports downstream satellite-depth prediction, with gold indicating higher elevations and purple indicating lower heights.The predicted depth visualization is presented as an application of the learned representation.Source: Figure 13 · p. 15

Out-domain generalization in Seattle City

  • Zero-shot Seattle evaluation preserves multi-view consistency and reproduces city-layout details such as parking spaces and building architecture.Seattle was not included in training, providing an out-of-domain test of generalization.Source: §4.7 · p. 15

5 DISCUSSION & LIMITATIONS

The discussion identifies limitations in data alignment, illumination representation, and the assumed extent of the generative 3D space. These constraints motivate richer inputs, improved calibration, and more precise modeling of elements beyond satellite coverage.

Discussion

  • The paper frames these data-processing and algorithmic constraints as directions for future research rather than as resolved aspects of the current system.The limitations discussion explicitly covers challenges in the synthesis approach and data pipeline.Source: §5 · pp. 15–16

Data challenge

  • Satellite-to-street training requires well-aligned imagery, but non-nadir satellite captures, uncertain street-view pose calibration, and varying camera altitude complicate coordinate placement.Accurate placement is critical for defining the generative space.Source: §5 · p. 16

Illumination representation

  • Representing illumination with a sky-region color histogram omits potentially useful cues such as sun position, cloud distribution, and atmospheric radiation.The authors defer richer illumination inputs because daylight estimation is ill-posed and added models could introduce noise and complexity.Source: §5 · p. 16

3D space for generation

  • Approximating the sky as the only region beyond satellite coverage can misrepresent distant mountains, buildings, and other elements.More precise identification and modeling of these regions is left for future work.Source: §5 · p. 16

6 CONCLUSION

Sat2Density++ models sky regions and street-view illumination within a feedforward 3D-aware framework for photorealistic panorama synthesis. Experiments on suburban and urban datasets show multi-view-consistent videos without 3D annotations.

Conclusion

  • Sat2Density++ synthesizes street-view panorama videos from satellite images by modeling sky regions and street-view illuminations.The method addresses the two-view neural-field learning problem with a feedforward 3D-aware generation framework.Source: §6 · p. 16
  • Experiments on suburban and VIGOR urban datasets demonstrate the method’s effectiveness for photorealistic street-view panorama synthesis.The reported system is trained without relying on 3D annotations.Source: §6 · p. 16
  • The authors identify the model as the first, to their knowledge, to synthesize multi-view-consistent street-view videos from satellite images without 3D training annotations.They position the approach as facilitating 3D representation learning and outdoor-scene video generation.Source: §6 · p. 16

Related papers

Loading 2505.17001v2…