Source-linked AI summary
Multi-Channel Attention Selection GAN with Cascaded Semantic Guidance for Cross-View Image Translation
Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J. Corso, Yan Yan
TL;DR
Cross-view translation is difficult when views differ substantially and overlap little, making natural-scene synthesis from one view to another challenging. SelectionGAN uses semantic-guided coarse generation followed by multi-channel attention refinement, and reports significantly better results than state-of-the-art methods on Dayton, CVUSA, and Ego2Top.
Problem
Cross-view image translation must synthesize natural scenes across drastically different views with severe deformation and limited overlap.
Method
SelectionGAN cascades a cycled semantic-guided generation stage with multi-channel attention selection for coarse-to-fine synthesis, using uncertainty maps to guide pixel loss.
Results
SelectionGAN reports significantly better results than state-of-the-art methods on Dayton, CVUSA, and Ego2Top.
Takeaways & Limitations
The method produces more detailed cross-view results through cascaded semantic guidance and attention-based selection across multiple intermediate generations.
Abstract
from arXiv · showhide
Cross-view image translation is challenging because it involves images with drastically different views and severe deformation. In this paper, we propose a novel approach named Multi-Channel Attention SelectionGAN (SelectionGAN) that makes it possible to generate images of natural scenes in arbitrary viewpoints, based on an image of the scene and a novel semantic map. The proposed SelectionGAN explicitly utilizes the semantic information and consists of two stages. In the first stage, the condition image and the target semantic map are fed into a cycled semantic-guided generation network to produce initial coarse results. In the second stage, we refine the initial results by using a multi-channel attention selection mechanism. Moreover, uncertainty maps automatically learned from attentions are used to guide the pixel loss for better network optimization. Extensive experiments on Dayton, CVUSA and Ego2Top datasets show that our model is able to generate significantly better results than the state-of-the-art methods. The source code, data and trained models are available at https://github.com/Ha0Tang/SelectionGAN.
1. Introduction
Cross-view translation becomes especially difficult when source and target views have little or no overlap, producing substantially different scene structures and appearances. SelectionGAN addresses this setting with cascaded semantic guidance and multi-channel attention selection for coarse-to-fine synthesis.
- Challenge: Cross-view translation synthesizes a scene from one viewpoint into a new viewpoint, but little or no view overlap creates severe structural and appearance differences.The paper targets this more challenging setting than prior scenarios with substantial overlap.
- Method: SelectionGAN uses two generation stages: cycled semantic-guided generation first produces coarse results, followed by attention-based fine-grained refinement.Stage I reconstructs semantic information through a generation cycle; Stage II refines coarse outputs using learned deep features.
- Method: The multi-channel attention module spatially selects among several intermediate generations to synthesize a final output.The module expands beyond a three-channel RGB generation space and learns attention maps matching the intermediate generations.
- Method: Uncertainty maps learned from attention maps guide the pixel loss, addressing inaccurate semantic labels during optimization.This mechanism is intended to make network optimization more robust.
- Results: Extensive experiments on Dayton, CVUSA, and Ego2Top report significantly better results than state-of-the-art methods.The paper also presents arbitrary-view synthesis results on outdoor and indoor Ego2Top scenes.
2. Related Work
Prior work applied GANs and image-to-image translation to conditioned synthesis, unpaired translation, attention-based translation, and viewpoint transformation. SelectionGAN is presented as the first GAN-based image-to-image translation model to incorporate a multi-channel attention selection module.
- GAN and Image-to-Image Translation: Conditional GANs combine adversarial generation with external information such as labels, text, human pose, or reference images.Pix2pix uses a conditional GAN to learn a supervised mapping from input to output image domains.
- GAN and Image-to-Image Translation: Image-to-image translation methods learn mappings between input and output domains, with CycleGAN addressing unpaired translation through cycle-consistency loss.Attention mechanisms have also been explored to improve image-translation performance.
- Learning Viewpoint Transformations: Viewpoint-transformation research has synthesized novel views of objects such as cars, chairs, and tables, while other work studies cross-view scene synthesis.The paper places its natural-scene task within this broader viewpoint-transformation literature.
- Positioning: The authors identify SelectionGAN as the first image-to-image translation GAN incorporating a multi-channel attention selection module.The module is designed to select among multiple generation channels for translation.
3. Multi-Channel Attention Selection GAN
SelectionGAN uses cascaded semantic guidance to generate a coarse cross-view image and refine it through multi-channel attention selection. The second stage combines diverse intermediate generations and learned uncertainty maps to produce finer synthesis and guide pixel-loss optimization.
- Cascade Semantic-guided Generation: Stage I concatenates a source-view image with a target-view semantic map, generates the other-view image, and reconstructs the semantic map through a cycled semantic-guided network.The semantic map is used both as network input and reconstruction supervision, providing constraints on semantic structure consistency.
- Cascade Semantic-guided Generation: Because Stage I produces coarse, blurred outputs, the cascade passes its synthesis and deep features to Stage II for fine-grained refinement.The second stage is designed to address high pixel-level dissimilarity and blurred scene details in the first-stage output.
- Multi-Channel Attention Selection: Multi-scale spatial pooling uses different kernel sizes, strides, and receptive fields to capture spatial information under large source-target deformation.Pooled features are rescaled and combined with input features through element-wise multiplication before producing features for attention selection.
- Multi-Channel Attention Selection: The attention module constructs multiple intermediate image generations and corresponding attention maps, then applies channel-wise selection to synthesize a final output.Channel-wise Softmax normalizes the attention maps, which spatially select among diverse intermediate generations.
- Uncertainty-guided Pixel Loss: The learned attention maps also generate uncertainty maps that weight pixel-level losses to reduce the effect of inaccurate pretrained semantic labels.The uncertainty maps are produced from concatenated attention maps and act as an automatically learned weighting scheme for optimization.
4. Experiments
Experiments evaluate SelectionGAN on Dayton, CVUSA, and Ego2Top using quantitative metrics, ablations, and qualitative comparisons. Results examine semantic guidance, attention selection, and coarse-to-fine generation.
- Experimental Setting: Experiments use Dayton, CVUSA, and Ego2Top datasets with images standardized primarily to 256×256 resolution.Dayton uses 55,000/21,048 train/test pairs, CVUSA uses 35,532/8,884 pairs, and Ego2Top contains more than 230,000 frames.
- Experimental Setting: Evaluation uses Inception Score, top-k prediction accuracy, and KL score for quantitative analysis.The table captions specify that accuracy and Inception Score are higher-is-better metrics.
- Ablation Analysis: The attention selection module produces multiple intermediate generations and selects spatially important regions for the final scene image.The associated refinement uses learned uncertainty maps to guide pixel loss.
- Ablation Analysis: Baseline B outperforms Baseline A because the target semantic map contains more structural information.The ablation study compares image-only generation with generation conditioned on the corresponding semantic map.
- Ablation Analysis: Semantic-guided generation improves SSIM, PSNR, and SD over the image-only baseline by 8.19, 3.1771, and 0.3205, respectively.This comparison supports the importance of conditioning generation on target semantic information.
- Qualitative Results: Qualitative results show clearer details in roads, trees, clouds, cars, grass, and house roofs than competing methods across generated viewpoints.The reported comparisons cover CVUSA, Dayton, and Ego2Top qualitative results.
5. Conclusion
The conclusion presents SelectionGAN as a two-stage semantic-conditioned approach for cross-view image synthesis. It combines coarse semantic structure generation, appearance refinement, and uncertainty-guided optimization, with stronger results reported across three datasets.
- SelectionGAN conditions cross-view image synthesis on a reference image and a target semantic map.
- Its cascade separates semantic-structure capture in Stage I from appearance-detail refinement in Stage II.Stage II uses the multi-channel attention selection module.
- Uncertainty map-guided pixel loss addresses inaccurate semantic labels during optimization.
- Experiments on three public datasets report much better results than state-of-the-art methods.
6. Influence of the Number of Attention Channels N
The study evaluates how the number of attention channels affects performance and selects N = 10 as a practical setting. Performance tends to stabilize beyond this value.
- Performance tends to be stable after N = 10 attention channels.
- The experiments set N = 10 by balancing performance and training speed.
7. Coarse-to-Fine Generation
Additional comparisons evaluate coarse-to-fine generation against a one-stage model. The reported two-stage method produces visually better results.
- The two-stage coarse-to-fine method generates visually better results than the one-stage model.The comparison is reported in Table 7 and Figures 8, 9, and 10.
8. Visualization of Uncertainty Map
SelectionGAN’s uncertainty maps learn target-image layout and structure, highlighting junctions and edges where different regions meet.
- Uncertainty maps learn the layout and structure of target images.
- Textured regions are mostly similar in generated images, while junctions and edges between regions remain uncertain.
9. Arbitrary Cross-View Image Translation
On Ego2Top, SelectionGAN generates the same scene from different viewpoints using an image and novel semantic maps, across outdoor and indoor environments.
- SelectionGAN generates the same scene with different viewpoints from an image and novel semantic maps.
- The arbitrary cross-view translation works in both outdoor and indoor environments on Ego2Top.
- The experiments evaluate arbitrary cross-view image translation on the Ego2Top dataset.
10. Generated Segmentation Maps
SelectionGAN’s generated segmentation maps are evaluated on common Dayton classes using per-class accuracy and mean IOU, outperforming X-Fork and X-Seq on both metrics.
- Generated segmentation maps are evaluated on vegetation, road, building, and sky using per-class accuracy and mean IOU.For both metrics, higher values are better.
- SelectionGAN achieves better segmentation results than X-Fork and X-Seq on both evaluation metrics.
- The segmentation-map comparison is conducted on the Dayton dataset.
11. State-of-the-art Comparisons
SelectionGAN is compared visually with Pix2pix, X-Fork, and X-Seq across Dayton, CVUSA, and Ego2Top, with the paper reporting significantly better results.
- SelectionGAN achieves significantly visually better results than Pix2pix, X-Fork, and X-Seq across the presented datasets.The comparisons cover Dayton, CVUSA, and Ego2Top.
- Additional figures show SelectionGAN outputs for Dayton in both a2g and g2a directions and for CVUSA in the a2g direction.
- Ego2Top visualizations present arbitrary cross-view image translation results.
- Dayton examples compare methods in a2g and g2a directions at 64×64 and 256×256 resolutions.
- CVUSA examples show a2g translation at 256×256 resolution, while Ego2Top examples show results at 256×256 resolution.