PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Guided Street-View Panorama Synthesis from Satellite Imagery

Yujiao Shi, Dylan Campbell, Xin Yu, Hongdong Li

arXiv:2103.01623v4cs.CV

TL;DR

Satellite-to-street panorama synthesis is challenging because the views have little field-of-view overlap and markedly different appearances. The paper uses an explicit geometric projection module before generation, and experiments report geometrically consistent panoramas.

  • Problem

    Satellite-to-street synthesis is challenging because the views have minimal field-of-view overlap and radically different appearances.

    Source: §1 · p. 1

  • Method

    The differentiable S2SP module estimates satellite-image heights and projects pixels into the street view to establish geometric correspondences for panorama generation.

    Source: §1 · pp. 1–2

  • Results

    Quantitative and qualitative experiments demonstrate that the method generates geometrically consistent street-view panoramas from satellite images.

    Source: §9 · p. 13

  • Takeaways & Limitations

    The paper concludes that explicitly exploiting two-view geometry supports accurate, occlusion-aware image synthesis.

    Source: §9 · p. 13

    Satellite images show object tops rather than façades, so the method hallucinates façade appearance and can fail when object heights are misestimated.

    Source: §8 · p. 13

Abstract

from arXiv · show

This paper presents a new approach for synthesizing a novel street-view panorama given an overhead satellite image. Taking a small satellite image patch as input, our method generates a Google's omnidirectional street-view type panorama, as if it is captured from the same geographical location as the center of the satellite patch. Existing works tackle this task as an image generation problem which adopts generative adversarial networks to implicitly learn the cross-view transformations, while ignoring the domain relevance. In this paper, we propose to explicitly establish the geometric correspondences between the two-view images so as to facilitate the cross-view transformation learning. Specifically, we observe that when a 3D point in the real world is visible in both views, there is a deterministic mapping between the projected points in the two-view images given the height information of this 3D point. Motivated by this, we develop a novel Satellite to Street-view image Projection (S2SP) module which explicitly establishes such geometric correspondences and projects the satellite images to the street viewpoint. With these projected satellite images as network input, we next employ a generator to synthesize realistic street-view panoramas that are geometrically consistent with the satellite images. Our S2SP module is differentiable and the whole framework is trained in an end-to-end manner. Extensive experimental results on two cross-view benchmark datasets demonstrate that our method generates images that better respect the scene geometry than existing approaches.

1 INTRODUCTION

Satellite-to-street-view synthesis aims to generate a panorama from the location at the center of a satellite image while preserving geometric consistency and visual similarity. The paper introduces a differentiable projection-and-generation framework that uses estimated scene geometry and also addresses misalignment in image pairs.

1 INTRODUCTION

  • The framework aims to produce a panorama at the satellite patch center whose scene structures are geometrically consistent with the satellite image and visually similar to the ground-truth panorama.Structure-preserving synthesis is also described as useful for enriching media content where street-view images are expensive or unavailable.Source: §1 · p. 1
  • The task is challenging because overhead and ground views have little field-of-view overlap and can differ in appearance due to imaging modality, time, season, and weather.Prior approaches often treat synthesis as pure image generation, while the transformations also depend on scene geometry.Source: §1 · p. 1
  • S2SP estimates satellite-image heights and uses them to establish geometric correspondences, recover street-view spatial layout, and handle occlusions.The module estimates height maps and projects satellite pixels toward the street viewpoint; a generator then synthesizes the panorama from the projection.Source: §1 · pp. 1–2
  • S2SP also helps align collected satellite and street-view pairs, providing cleaner training signals and more accurate evaluation measurements.The module is presented as a way to alleviate pair misalignment and identify the location shift between the street-view camera and satellite-image center.Source: §1 · p. 2

2 RELATED WORK

The related work situates the method among novel-view synthesis, cross-view synthesis, and geo-localization approaches. It contrasts the proposed use of two-view geometry with prior representations and supervision requirements.

2.1 Novel view synthesis

  • Because the viewpoint change is large, the method uses an overhead MPI as an intermediate representation and conditions a subsequent generator on projected street-view images to fill missing textures.Traditional novel-view synthesis methods model smaller camera movements, including plane- and MPI-based approaches,,,.Source: §2.1 · pp. 2–3

2.2 Satellite-view and street-view synthesis

  • Prior satellite-to-street-view methods use semantic transformations, conditional GANs, homographies, or target-view semantics; this method models correspondences for all pixels without requiring target-view semantic segmentation.Unlike a single ground-plane homography, its correspondences cover all pixels between views.Source: §2.2 · p. 3
  • Unlike Lu et al., whose approach required ground-truth height and semantic supervision and could not train end-to-end due to discretized projection operations, our method trains end-to-end without satellite-map height supervision.Source: §2.2 · p. 3

2.3 Cross-view image geo-localization

  • Geo-localization matches ground-camera queries to geo-tagged satellite databases, while this synthesis work replaces the earlier polar-transform approximation with two-view geometry that does not distort scene geometry.Prior geo-localization work used polar transforms as approximate correspondences,, [22].Source: §2.3 · p. 3

3 SATELLITE AND STREET-VIEW GEOMETRY

The geometry section describes the distinct camera projections and the mapping between views. It motivates a dense, inverse-mapped MPI representation to address limitations of direct forward projection.

3.1 Parallel projection of a satellite camera

  • The satellite camera is approximated by a parallel projection from camera coordinates to image coordinates, with the satellite image center and a scale factor defining the mapping.The image center is (u0, v0), and s is the scale factor between image and world coordinates.Source: §3.1 · p. 3

3.2 Perspective projection of an omnidirectional street-view camera

  • The omnidirectional street-view camera maps camera coordinates to panorama coordinates using an equirectangular projection.The passage parameterizes panorama pixels by spherical coordinates.Source: §3.2 · p. 3

3.3 Satellite-view and street-view geometry

  • For a satellite pixel visible from the street view, its panorama location is deterministic given its height; direct forward mapping is rejected because of weak occlusion modeling, missing pixels, and nondifferentiable hard visibility selection.The method instead uses a dense MPI representation and inverse mapping; MPI color and alpha channels represent texture and transparency, including occluded elements.Source: §3.3 · p. 4

4 STREET-VIEW SYNTHESIS GUIDED BY TWO-VIEW GEOMETRY

The framework uses differentiable two-view geometry to project satellite imagery into a street-view representation, then conditions a generator to synthesize realistic, geometrically consistent panoramas. Height probabilities and the rendered-image error provide dense, end-to-end learning signals.

4 STREET-VIEW SYNTHESIS GUIDED BY TWO-VIEW GEOMETRY

  • S2SP establishes geometric correspondences before a projection-conditioned generator synthesizes realistic panoramas, and differentiable operations enable end-to-end training.The generator is trained to produce panoramas geometrically consistent with the satellite image.Source: §4 · p. 4, §4.1 · p. 5

4.1 Satellite to street-view image projection

  • A per-pixel probability distribution over discretized heights builds a multi-plane image (MPI), with cumulative height probabilities modeling lower points as opaque.The MPI uses satellite-image colors, with its final plane representing the ground.Source: §4.1 · p. 4
  • The overhead MPI is converted into street-view planes by projecting concentric cylinders, then rendered far-to-near with alpha compositing.All satellite pixels contribute to the rendered image, providing dense gradients during training.Source: §4.1 · p. 5, Figure 5 · p. 6

4.2 Network architecture

  • A Pix2Pix-based generator receives implicit height supervision through geometric constraints and error propagated from the generated image, without ground-truth height maps.Incorrect estimated heights can reorder projected points and increase the final-image error.Source: §4.2 · p. 5

4.3 Adversarial learning from corresponding satellite and street-view image pairs

  • Adversarial training is combined with perceptual and L1 reconstruction losses to handle feature and pixel-color similarity despite seasonal mismatches in paired images.The generator maps satellite images to street-view counterparts while competing with a discriminator.Source: §4.3 · pp. 5–6

5 ALIGNING SATELLITE AND STREET-VIEW IMAGE PAIRS

GPS positioning errors leave some nominally matching satellite and street-view pairs misaligned. The authors search for a small shift that maximizes similarity after projecting the satellite image, and report that the corrected projection aligns with the street-view image.

5 ALIGNING SATELLITE AND STREET-VIEW IMAGE PAIRS

  • GPS positioning error means dataset pairs tagged as matching may not be strictly location-aligned.The street-view camera location may differ from the satellite image center.Source: §5 · p. 6
  • The correction searches 40 × 40 candidate points across a central 11.25 × 11.25-meter region and adopts the shift whose projection has maximum SSIM against the street-view image.The candidate satellite projections are compared with the original street-view panorama.Source: §5 · p. 6
  • The corrected projected image aligns with its corresponding street-view image in the illustrated example.The alignment is used to create clean training pairs through preprocessing under a ground-plane assumption, with no trainable parameters.Source: §5 · p. 6

6 BENCHMARKS FOR SATELLITE TO STREET-VIEW IMAGE SYNTHESIS

The paper uses CVACT and CVUSA for satellite-to-street-view synthesis, where exact location alignment is needed beyond the matching allowed in their geolocalization setup. It introduces an offset-annotated CVACT split and specifies dataset-specific alignment and field-of-view constraints.

6 BENCHMARKS FOR SATELLITE TO STREET-VIEW IMAGE SYNTHESIS

  • CVACT and CVUSA each provide 35,532 training pairs and 8,884 test pairs, but synthesis evaluation requires exact alignment to the satellite-image center.The datasets originally allow some location shift between matching pairs.Source: §6 · p. 6
  • The proposed CVACT split has 26,519 training pairs and 6,288 test pairs, augmented with ground-truth translation offsets.Pairs with unestimable offsets, including large shifts or severe cross-class vertical occlusion, are filtered.Source: §6 · pp. 6–7
  • CVUSA panoramas are approximated as having a 90-degree vertical field of view because their top-and-bottom cropping is not known to be uniform; CVACT panoramas have a 180-degree vertical field of view.Both datasets have a 360-degree horizontal field of view.Source: §6 · p. 7
  • CVACT and CVUSA are selected for evaluation because the alternative OP dataset has significant uncorrelated position and orientation misalignments and is an order of magnitude smaller.A dataset with ground-truth semantics and height maps was not released.Source: §6 · p. 7

7 EXPERIMENTS

Experiments compare the method with existing cross-view generators, test alignment and component choices, and evaluate alternative projection strategies. The full method performs best across reported metrics and produces panoramas that better preserve scene geometry.

7.1 Implementation details

  • Experiments use 256 × 256 satellite inputs, 128 × 512 panoramas, 64 MPI planes, and an assumed 2-meter camera height; synthesis averages 0.2 seconds per image.Source: §7.1 · p. 7

7.2 Evaluation metrics

  • Evaluation combines pixel-level RMSE, SSIM, PSNR, and sharpness difference with perceptual similarity and semantic accuracy and mean intersection over union.Perceptual similarity uses pretrained AlexNet and Squeeze backbones; the semantic classifier is trained on CityScapes and fine-tuned on CVUSA.Source: §7.2 · p. 7

7.3 Comparison with existing methods

  • The method consistently outperforms Pix2Pix and XFork on all quantitative metrics and produces more geometrically consistent panoramas.Examples show recovery of roundabouts, lane lines, and complex road and building structures where Pix2Pix and XFork confuse regions or hallucinate.Source: Table 1 · p. 8, §7.3 · p. 9

7.4 Comparison on aligned or unaligned datasets

  • On aligned CVACT tests, unaligned training hurts all methods least for ours; on unaligned tests, ours instead fares worse after aligned training.The authors attribute the latter effect to the projection module's explicit alignment rule reducing the network's capacity to learn misalignment bias.Source: §7.4 · p. 9, Table 2 · p. 10

7.5 Ablation study

  • The full pipeline beats handcrafted projections and deep-learning ablations across metrics; height estimation recovers elevated objects, while perceptual and L1 losses provide complementary guidance.Without height estimation, trees may be rendered as grass; removing perceptual loss reduces image quality, while removing L1 makes CVUSA outputs blurry.Source: §7.5 · p. 11
  • Increasing satellite-view MPI planes above one improves performance, but further increases bring no significant gains without explicit height supervision.The authors expect larger N to be more useful when height supervision is available.Source: §7.5 · p. 12, Table 4 · p. 12

7.6 Other satellite to street-view image projection methods

  • Projection variants perform similarly and outperform existing methods; volumetric generator inputs cost slightly more parameters and training time, so the authors choose depth-wise rendered input for interpretability.Source: Table 5 · p. 12, §7.6 · p. 13

8 DISCUSSION AND LIMITATIONS

Skip connections made little difference to performance, while the method’s speed and its dependence on satellite-image evidence constrain its use.

8 DISCUSSION AND LIMITATIONS

  • Skip connections made no performance difference for the proposed method or the Pix2Pix and XFork baselines.The authors say skip connections contribute only marginally to the final performance of their approach.Source: §8 · p. 13
  • Skip connections can help recover fine detail in image-generation tasks, but contribute little to the final performance here.The authors suggest the satellite-to-street-view modality and resolution differences may explain the baselines’ results.Source: §8 · p. 13
  • The method hallucinates object facades from satellite views that show only top surfaces, and can fail when heights are misestimated or objects are indistinguishable.The authors identify facades, an unexpectedly tall hill, and a small fence as examples.Source: §8 · p. 13
  • Rendering takes 0.2 s per image, compared with 0.02 s for Pix2Pix and XFork.The authors suggest the baselines for time-critical applications when scene-structure variation is small, while their method is needed when that variation is large.Source: §8 · p. 13

9 CONCLUSION

The paper proposes a differentiable projection module that explicitly establishes satellite-to-street-view geometric correspondences. Experiments show geometrically consistent panorama synthesis, and the authors expect the approach’s key idea to be useful in related problems.

9 CONCLUSION

  • The proposed method explicitly establishes geometric correspondences between satellite and street-view images using a novel, differentiable projection module.The module exploits the two-view geometry for occlusion-aware image synthesis.Source: §9 · p. 13
  • Quantitative and qualitative experiments demonstrate that the method generates geometrically consistent street-view panoramas from satellite images.Source: §9 · p. 13
  • The authors expect explicit correspondences and implicit height-map estimation to be useful for satellite/street-view height or depth estimation and two-view novel view synthesis.Source: §9 · p. 13

Supplementary Material: Geometry-Guided Street-View Panorama

The supplementary material includes additional qualitative comparisons of synthesized images on CVACT and CVUSA.

Supplementary Material: Geometry-Guided Street-View Panorama

  • The supplementary material is titled “Synthesis from Satellite Imagery” and lists Yujiao Shi, Dylan Campbell, Xin Yu, and Hongdong Li as authors.Source: Supplementary Material:… · p. 15
  • An additional qualitative comparison of synthesized images is provided for CVACT.Source: Figure 1 · p. 16
  • An additional qualitative comparison of synthesized images is provided for CVUSA.Source: Figure 2 · p. 17

Related papers

Loading 2103.01623v4…