Source-linked AI summary
Cross-View Image Synthesis using Conditional GANs
Krishna Regmi, Ali Borji
TL;DR
Cross-view synthesis between aerial and street-level imagery is difficult because the views differ greatly in coverage, detail, and appearance. The paper proposes X-Fork and X-Seq cGAN architectures that jointly generate target images and semantic segmentations. Qualitative and quantitative evaluations report that the proposed methods outperform baselines, although evaluation scores do not consistently identify which proposed architecture is better.
Problem
Cross-view image synthesis must transform appearance and semantic information between aerial and street-level views with little overlap, occlusion, clutter, and view-dependent object differences.
Method
The paper proposes X-Fork, which forks one generator into image and segmentation outputs, and X-Seq, which connects two end-to-end cGANs for sequential image and segmentation generation.
Results
The proposed methods are consistently superior to the baselines across the reported evaluations; on CVUSA, X-Fork improves over Zhai et al. by 5.03% in SSIM, 8.93% in PSNR, and 12.35% in Sharpness difference.
Takeaways & Limitations
Jointly generating semantic segmentations with target-view images helps the networks generate better cross-view images, with higher resolution adding visual quality and detail.
Takeaways & Limitations
The Dayton segmentation maps are generated by RefineNet and are imperfect, introducing label noise and model errors that can create artifacts.
Abstract
from arXiv · showhide
Learning to generate natural scenes has always been a challenging task in computer vision. It is even more painstaking when the generation is conditioned on images with drastically different views. This is mainly because understanding, corresponding, and transforming appearance and semantic information across the views is not trivial. In this paper, we attempt to solve the novel problem of cross-view image synthesis, aerial to street-view and vice versa, using conditional generative adversarial networks (cGAN). Two new architectures called Crossview Fork (X-Fork) and Crossview Sequential (X-Seq) are proposed to generate scenes with resolutions of 64x64 and 256x256 pixels. X-Fork architecture has a single discriminator and a single generator. The generator hallucinates both the image and its semantic segmentation in the target view. X-Seq architecture utilizes two cGANs. The first one generates the target image which is subsequently fed to the second cGAN for generating its corresponding semantic segmentation map. The feedback from the second cGAN helps the first cGAN generate sharper images. Both of our proposed architectures learn to generate natural images as well as their semantic segmentation maps. The proposed methods show that they are able to capture and maintain the true semantics of objects in source and target views better than the traditional image-to-image translation method which considers only the visual appearance of the scene. Extensive qualitative and quantitative evaluations support the effectiveness of our frameworks, compared to two state of the art methods, for natural scene generation across drastically different views.
1. Introduction
The paper frames cross-view synthesis between aerial and ground-level imagery as a challenging image-translation problem and proposes cGAN architectures that jointly generate target images and semantic segmentations.
- Approach: The paper formulates cross-view synthesis as image-to-image translation using conditional generative adversarial networks.The approach addresses both aerial-to-ground and ground-to-aerial generation.
- Challenges: Natural scenes contain clutter and detailed objects whose positions, orientations, and inclinations must be learned across views.Examples include trees, cars, roads, and buildings with substantial variation between corresponding image pairs.
- Challenges: Cross-view synthesis is difficult because aerial and street-view images differ in coverage, detail, viewpoint, occlusion, and object appearance.Aerial images provide wider but noisier and less informative regions for street-view synthesis, while street-view images contain finer object details.
- Proposed architectures: X-Fork modifies an image-translation baseline by forking the generator to produce both a target-view image and semantic segmentation map.The architecture uses one discriminator and shared decoder features before producing the two outputs.
- Proposed architectures: X-Seq connects two cGANs so the first generates the target image and the second generates its corresponding segmentation map.The two networks are trained end-to-end, and their input-output dependency constrains both generated products.
- Proposed architectures: Generating segmentation maps alongside images improves image generation compared with a baseline that generates images alone.The paper presents this joint generation as a response to the semantic and appearance challenges of cross-view synthesis.
2. Related Works
Related work spans novel-view synthesis, aerial–ground applications, conditional image generation, and cross-domain mappings, but prior approaches often address narrower view or object settings.
- Aerial–ground relations: Aerial–ground research has addressed cross-view colocalization, geo-localization, geo-tagging, and ground-level panorama synthesis from aerial imagery.Zhai et al. predict a ground-image semantic layout from aerial imagery before synthesizing a ground-level panorama.
- Viewpoint transformation: Prior viewpoint-transformation methods synthesize novel views of the same objects, including cars and chairs, while preserving object identity and structure.Some methods copy pixel information; others learn 3D representations for unseen object images.
- Conditional generation: Conditional GANs generate images conditioned on labels, text embeddings, image representations, or aerial images of the same location.Related applications include digit generation, image translation, panoramic scene synthesis, and image inpainting.
- Cross-domain transformations: Cross-domain GAN methods learn mappings between image domains using either large unpaired datasets or cycle-consistency.Cycle-consistency assumes that mapping to another domain and back reconstructs the original image.
3. Background on GANs
The paper introduces GAN and cGAN objectives for conditional image generation, combining adversarial training with an L1 reconstruction term for sharper outputs and cross-view synthesis.
- GANs: A GAN trains a generator and discriminator simultaneously: the generator synthesizes images, while the discriminator distinguishes generated from real samples.The networks are optimized through a min-max objective.
- Conditional GANs: In a cGAN, both generator and discriminator receive a conditioning variable, enabling synthesis and discrimination based on the condition–image pair.The condition may be a label, text embedding, or image representation.
- Conditional GANs: The generated image is represented as x′ = G(z, c), where z is noise and c is the conditioning variable.The cited formulation defines x′ as the generator’s output.
- Reconstruction loss: L1 distance is used because it produces less blurred and sharper generated images than L2 distance.The paper therefore includes L1 reconstruction in its method.
- Cross-view baseline: The baseline objective combines conditional GAN loss with L1 loss.For ground-from-aerial synthesis, the conditional GAN and L1 terms are given separately and then summed; the input roles reverse for ground-to-aerial synthesis.
4. Proposed cGAN-based Approaches
The paper proposes X-Fork and X-Seq cGAN architectures for cross-view synthesis that generate target-view images together with semantic segmentation maps. X-Fork shares decoder features before branching, while X-Seq trains an image-generating cGAN and a segmentation-generating cGAN in tandem.
- The proposed architectures generate both target-view images and semantic segmentation maps for cross-view synthesis.
- Crossview Fork (X-Fork): X-Fork forks a single generator to produce the target-view image and segmentation map, while retaining a baseline-style discriminator.
- Crossview Fork (X-Fork): X-Fork shares decoder weights across the first six blocks because image and segmentation outputs contain shared features.
- Crossview Fork (X-Fork): The X-Fork loss adds an L1 distance for the generated segmentation map alongside the image L1 distance.
- Crossview Sequential (X-Seq): X-Seq chains two cGANs: the first generates the target-view image, and the second uses it to synthesize the corresponding segmentation map.
- Crossview Sequential (X-Seq): X-Seq trains both cGANs end-to-end, using their input-output dependency and feedback to improve generated outputs compared with the baseline.
5. Experimental Setting
Experiments use the Dayton and CVUSA cross-view datasets with 256×256 resized images and semantic segmentation supervision. Dayton segmentation labels are generated by pre-trained RefineNet, while Pix2pix provides the baseline.
- Dataset: The Dayton Dataset contains 76,048 selected image pairs from Dayton, split into 55,000 training and 21,048 test pairs.
- Dataset: Dayton and the original dataset images are resized to 256×256 for the experiments.
- Dataset: CVUSA uses a 35,532/8,884 train/test split, with aerial images center-cropped and resized to 256×256.
- Dataset: The experiments generate a single camera-angle ground image by resizing the first quarter of each CVUSA ground image and segmentation map.
- Dataset: Because Dayton lacks annotated segmentation maps, pre-trained RefineNet supplies 20-class maps used as ground-truth supervision.
- Implementation Details: Pix2pix is used as the baseline conditional GAN architecture, with an encoder-decoder generator and an encoder-like discriminator.
- Implementation Details: The balancing factor λ between GAN and L1 losses is fixed at 100.
6. Results
Experiments evaluate cross-view synthesis qualitatively and with several quantitative measures across Dayton and CVUSA, at 64×64 and 256×256 resolutions. The proposed methods generally outperform baselines, although different metrics do not consistently identify X-Fork or X-Seq as superior.
- Qualitative Evaluation: 64×64 experiments transfer image representations across views, rendering details such as roads, trees, sky, clouds, pedestrian lanes, grass, and roofs.These proof-of-concept experiments were conducted in both aerial-to-ground and ground-to-aerial directions on Dayton.
- Qualitative Evaluation: 256×256 Dayton images contain more object detail and appear less granulated and more natural than lower-resolution results.Houses, trees, pedestrian lanes, and roads are specifically reported as more natural.
- Qualitative Evaluation: On CVUSA, the proposed X-Fork and X-Seq images are visually better than those produced by Zhai et al. and Pix2pix.Figure 7 compares true image pairs with outputs from Zhai et al., Pix2pix, X-Fork, and X-Seq.
- Quantitative Evaluation: X-Fork produces images closest to the real distribution in most evaluated settings, while X-Seq is slightly better than X-Fork for higher-resolution a2g KL divergence.The reported exception occurs in the higher-resolution aerial-to-ground experiment.
- Quantitative Evaluation: 5.03% SSIM, 8.93% PSNR, and 12.35% Sharpness difference improvements are reported for X-Fork over Zhai et al. on CVUSA.The paper also reports that X-Seq is best for Dayton a2g, whereas X-Fork leads in Dayton g2a for these measures.
- Quantitative Evaluation: GAN evaluation metrics disagree, making it difficult to determine whether X-Fork or X-Seq is better, although both are consistently superior to the baselines.The paper therefore uses several scores, including Inception Score, Top-k accuracy, KL divergence, SSIM, PSNR, and Sharpness Difference.
7. Discussion and Conclusion
The paper explores cross-view image generation between drastically different views using conditional GANs, jointly generating target images and semantic segmentations. Higher-resolution inputs improve visual quality and detail, while extensive evaluations support the methods' effectiveness; the challenging task remains open to further improvement.
- Conditional GANs are used to generate images between drastically different views.
- Jointly generating target-view semantic segmentations helps the networks produce better images than baselines.
- Higher-resolution images significantly improve visual quality and add more detail to synthesized images.
- Extensive qualitative and quantitative evaluations support the effectiveness of the proposed methods.
- The challenging nature of cross-view synthesis leaves room for further improvements.