Source-linked AI summary
Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, Hongsheng Li
TL;DR
Semantic image synthesis must generate photorealistic complex scenes from semantic layouts, but existing methods provide limited spatially distinct control. CC-FPSE predicts layout-conditioned convolutional kernels and uses a feature pyramid semantics-embedding discriminator, achieving state-of-the-art performance on Cityscapes, COCO-Stuff, and ADE20K.
Problem
Generating photorealistic images of complex scenes with diverse objects and stuff from semantic layouts remains challenging.
Method
CC-FPSE predicts spatially varying convolutional kernels from semantic layouts and uses a feature pyramid semantics-embedding discriminator to promote details and semantic alignment.
Results
CC-FPSE achieves state-of-the-art image-synthesis performance on Cityscapes, COCO-Stuff, and ADE20K.
Takeaways & Limitations
The approach enables semantic layouts to adaptively control generation while producing photorealistic images with high-fidelity details and aligned semantic meanings.
Takeaways & Limitations
Multi-scale PatchGAN comparison involves independently processing images at different scales, increasing network parameters, memory footprint, and computational cost.
Abstract
from arXiv · showhide
Semantic image synthesis aims at generating photorealistic images from semantic layouts. Previous approaches with conditional generative adversarial networks (GAN) show state-of-the-art performance on this task, which either feed the semantic label maps as inputs to the generator, or use them to modulate the activations in normalization layers via affine transformations. We argue that convolutional kernels in the generator should be aware of the distinct semantic labels at different locations when generating images. In order to better exploit the semantic layout for the image generator, we propose to predict convolutional kernels conditioned on the semantic label map to generate the intermediate feature maps from the noise maps and eventually generate the images. Moreover, we propose a feature pyramid semantics-embedding discriminator, which is more effective in enhancing fine details and semantic alignments between the generated images and the input semantic layouts than previous multi-scale discriminators. We achieve state-of-the-art results on both quantitative metrics and subjective evaluation on various semantic segmentation datasets, demonstrating the effectiveness of our approach.
1 Introduction
Semantic image synthesis seeks photorealistic complex scenes from semantic layouts, but existing generators inadequately preserve or exploit spatially varying label information. CC-FPSE addresses this with layout-conditioned convolution kernels and a feature pyramid semantics-embedding discriminator.
- Semantic image synthesis generates photorealistic images conditioned on semantic layouts for controllable synthesis and interactive image manipulation.
- Prior generators either provide label maps only at the input or use affine normalization modulation, limiting how layout information is preserved or represented.
- Different spatial locations may require different convolutional kernels because semantic labels and layouts vary across objects, stuff, and samples.
- CC-FPSE predicts spatially varying kernels from the semantic layout using depthwise separable convolutions and a global-context-aware weight prediction network.The factorization uses conditional depthwise and conventional pointwise convolutions, reducing the parameter and computational burden of naive kernel prediction.
- Its feature pyramid semantics-embedding discriminator promotes high-fidelity details and spatial semantic alignment between generated images and input layouts.
- CC-FPSE achieves state-of-the-art results on CityScapes, COCO-Stuff, and ADE20K for complex-scene image generation.
2 Related Work
The related work spans conditional and image-to-image generation, dynamic or hypernetwork-based filter prediction, and semantic image synthesis methods using label inputs or normalization modulation.
- Conditional GANs synthesize images from conditions such as labels, sentence descriptions, or semantic layouts, while image-to-image translation maps one image representation to another.
- Semantic synthesis methods include encoder-decoder, coarse-to-fine, multi-scale discriminator, and affine normalization modulation approaches.
- Dynamic filter networks and HyperNetworks generate filters or weights for another network, with applications including style transfer, super-resolution, segmentation, motion prediction, and tracking.
3 Method
CC-FPSE uses semantic-layout-conditioned convolutions and a feature-pyramid semantics-embedding discriminator to control image generation, preserve contextual layout information, and improve detail and alignment supervision.
- Conditional Convolution Generator: CC-FPSE predicts conditional convolution kernels from semantic layouts so distinct labels and spatial locations can adaptively control generation.The generator uses conditional convolution blocks and upsampling layers to refine feature maps into output images.
- Conditional Convolution Generator: Naively predicting spatially varying full kernels requires D × C × k × k × H × W parameters, making repeated generator layers computationally and memory intensive.The approach also identifies overfitting as a consequence of this parameterization.
- Conditional Convolution Generator: Depthwise separable convolution reduces prediction cost by dynamically predicting spatial depthwise kernels and applying a conventional 1 × 1 pointwise convolution.Depthwise filtering operates independently on each input channel before channels are mapped to D output channels.
- Conditional Convolution Generator: A feature-pyramid weight-prediction network combines multilevel features with the original semantic map, capturing local neighborhoods, long-range context, and relative locations.Its encoder-decoder structure avoids relying only on small local receptive fields or heavily downsampled label maps.
- Feature Pyramid Semantics-Embedding Discriminator: The discriminator combines multi-scale feature pyramids for textures and edges with patch-based semantic embeddings to encourage image–layout spatial alignment.It evaluates whether patch features are real or fake and whether they match semantic labels in the corresponding patch.
4 Experiments
Experiments on Cityscapes, COCO-Stuff, and ADE20K compare CC-FPSE with prior methods using visual, human perceptual, segmentation, and FID evaluations. Results support improved image quality, semantic alignment, and component effectiveness, while ablations show benefits from conditional convolutions, feature-pyramid weight prediction, and the FPSE discriminator.
- Datasets and Evaluation: Experiments cover Cityscapes, COCO-Stuff, and ADE20K, with annotated semantic segmentation masks and dataset-specific image resolutions.COCO-Stuff and ADE20K use 256 × 256 images, while Cityscapes uses 256 × 512 images.
- Qualitative Results: CC-FPSE produces finer edges and textures, fewer artifacts, and better matches to input semantic layouts than pix2pixHD and SPADE in qualitative comparisons.The comparisons are presented in Figures 3 and 4.
- Human Perceptual Evaluation: 55%, 76%, and 61% of human evaluators preferred the proposed images over SPADE on Cityscapes, COCO-Stuff, and ADE20K, respectively.Workers selected the image with higher quality and better semantic-layout matching from paired outputs.
- Quantitative Results: Table 1 reports segmentation performance and FID scores, and the proposed approach performs consistently better than previous approaches.Higher mIOU or accuracy and lower FID indicate better performance; SIMS has better FID but poorer segmentation because retrieved patches can be semantically mismatched.
- Ablation Studies: Replacing SPADE with conditional convolutions improves ablation performance across discriminator settings, supporting layout-conditioned generation.The comparisons include SPADE and conditional-convolution generators with MsPatch, FPSE, and MsPatch+SE discriminators.
- Ablation Studies: Removing the feature pyramid from weight prediction reduces performance, indicating that global and long-range information are necessary for predicting convolutional weights.The feature-pyramid structure is compared with a stack of two convolutional layers.
- Ablation Studies: The semantics-embedding constraint improves alignment, while the feature-pyramid discriminator yields further gains by combining low-level and semantic features across scales.The discriminator compares image patches with semantic layouts in a joint correspondence objective.
5 Conclusion
The paper proposes CC-FPSE for semantic-layout-conditioned image synthesis. Its conditional-convolution generator and feature-pyramid semantics-embedding discriminator produce high-quality details, semantic alignment, and state-of-the-art performance across three datasets.
- Conclusion: CC-FPSE predicts spatially varying conditional-convolution weights so semantic layouts can control image generation based on distinct labels at different locations.The generator uses conditional convolution layers to exploit layout information during generation.
- Conclusion: The feature-pyramid semantics-embedding discriminator guides generation toward high-fidelity details and alignment with the conditional semantic layout.The discriminator complements the layout-conditioned generator by supervising detail and semantic correspondence.
- Conclusion: The approach achieves state-of-the-art performance and photorealistic image generation on Cityscapes, COCO-Stuff, and ADE20K.
6 Appendix
The proposed approach produces diverse semantic image synthesis results and, compared with pix2pixHD and SPADE, generates finer details including small objects specified by the semantic layout.
- The proposed approach synthesizes diverse scenes with high-quality images and fine details.
- The proposed approach generates small labeled objects that previous approaches often ignore, including a driver inside a bus.
- Figure 6 compares semantic layouts with outputs from pix2pixHD, SPADE, and the proposed approach.