Source-linked AI summary
Semantic Image Synthesis with Spatially-Adaptive Normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu
TL;DR
Semantic image synthesis methods can wash away information from input segmentation masks during normalization. SPADE modulates normalized activations with the semantic layout, producing significantly better results across challenging datasets while supporting controllable semantic and style variation.
Problem
Conventional convolutional-normalization networks for semantic image synthesis can wash away information contained in input semantic masks.
Method
SPADE uses the input semantic layout to spatially adapt learned scale and bias transformations in normalization layers.
Results
Across COCO-Stuff, ADE20K, and Cityscapes, SPADE significantly outperforms state-of-the-art methods; on COCO-Stuff, it achieves 35.2 mIoU, about 1.5 times the previous leader.
Takeaways & Limitations
The method enables photorealistic synthesis across diverse scenes with multimodal and style-guided control for diverse outputs.
Takeaways & Limitations
The information-loss analysis is illustrated for convolution followed by normalization with a single-label segmentation mask.
Abstract
from arXiv · showhide
We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to ``wash away'' semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style. Code is available at https://github.com/NVlabs/SPADE .
1. Introduction
The paper targets semantic image synthesis from segmentation masks and identifies conventional normalization as a source of lost semantic information. It introduces spatially-adaptive normalization to propagate layout information through the network and reports stronger results on challenging datasets.
- Semantic image synthesis converts a semantic segmentation mask into a photorealistic image for applications including content generation and image editing.
- Conventional stacks of convolution, normalization, and nonlinearity layers are suboptimal because normalization can wash away information in semantic masks.
- Spatially-adaptive normalization modulates activations using semantic layouts through a learned, spatially-adaptive transformation that propagates semantic information throughout the network.
- Experiments on COCO-Stuff, ADE20K, and Cityscapes show that a compact network with the proposed layer produces significantly better results than several state-of-the-art methods.An extensive ablation study also evaluates the normalization layer against several variants for semantic image synthesis.
2. Related Work
Related work spans deep generative models for image synthesis, conditional synthesis under varied input modalities, and normalization layers that may be unconditional or conditioned on external data. The paper builds on GANs for conditional image synthesis and situates its approach among conditional normalization methods.
- Deep Generative Models: Deep generative image-synthesis methods include generative adversarial networks (GANs) and variational autoencoders (VAEs), while this work builds on GANs for conditional synthesis.GANs use a generator and discriminator, with the generator aiming to produce images indistinguishable from real ones to the discriminator.
- Conditional Image Synthesis: Conditional image synthesis varies by input type, including category labels, text, and image-to-image translation with conditional GANs.These forms differ in whether the conditioning input is a label, textual description, or another image.
- Normalization Layers: Unconditional normalization layers include Local Response Normalization, BatchNorm, InstanceNorm, Layer Normalization, Group Normalization, and Weight Normalization.They are used in modern deep networks and classifiers such as AlexNet and Inception-v2.
- Conditional Normalization: Conditional normalization layers include Conditional BatchNorm and AdaIN, which require external data and were first used in style transfer before adoption in other vision tasks.Their operation begins by normalizing layer activations to zero mean and unit deviation.
3. Semantic Image Synthesis
This section introduces SPADE, which synthesizes photorealistic images from semantic segmentation masks by using the mask to spatially modulate normalized activations. The design preserves semantic information that conventional normalization can erase while supporting lightweight and multimodal generation.
- Semantic Image Synthesis: The task is to learn a mapping from a semantic segmentation mask m to a photorealistic image.Each mask entry specifies the semantic label of a pixel.
- Spatially-adaptive denormalization: SPADE normalizes activations channelwise and modulates them with learned, spatially varying scaling and bias parameters derived from the segmentation mask.The mask is projected into an embedding space and convolved to produce spatial tensors γ and β, which act element-wise on normalized activations.
- Spatially-adaptive denormalization: SPADE generalizes Conditional BatchNorm and AdaIN when the conditioning data and modulation parameters are made spatially invariant, with AdaIN additionally using N = 1.Its spatial adaptation makes SPADE better suited for semantic image synthesis.
- SPADE generator: The SPADE generator uses SPADE residual blocks with upsampling, removes the encoder because modulation parameters encode the label layout, and can accept a random vector for multimodal synthesis.An optional encoder can map a real image to a random vector capturing style, which the generator combines with the segmentation mask through SPADE.
- Why does the SPADE work better?: Conventional normalization can erase semantic information from uniform masks, because InstanceNorm maps uniform convolution outputs to all-zero activations regardless of the input label.SPADE instead normalizes only activations from the previous layer and feeds the mask through spatially adaptive modulation without normalization.
4. Experiments
Experiments across challenging semantic image synthesis benchmarks show that SPADE improves segmentation alignment and image realism over leading baselines. Ablations and user studies further support SPADE’s effectiveness, while additional results demonstrate multimodal and style-controlled synthesis.
- Datasets and metrics: Experiments cover COCO-Stuff, ADE20K, and Cityscapes, using mIoU, pixel accuracy, and FID to evaluate semantic alignment and image distribution quality.COCO-Stuff contains 118,000 training and 5,000 validation images with 182 classes; ADE20K contains 20,210 training and 2,000 validation images with 150 classes.
- Quantitative comparisons: SPADE outperforms pix2pixHD, CRN, and SIMS across benchmark datasets in semantic segmentation and FID.The comparison uses leading semantic image synthesis models, with higher mIoU and accuracy and lower FID indicating better performance.
- Quantitative comparisons: 35.2 mIoU on COCO-Stuff is about 1.5 times better than the previous leading method, while SPADE’s FID is 2.2 times better.These results are reported as large-margin improvements over prior state-of-the-art methods.
- Human evaluation: Users strongly favor SPADE outputs on all datasets, especially the challenging COCO-Stuff and ADE20K benchmarks.On Cityscapes, users still prefer SPADE even though competing methods already achieve high image fidelity.
- Effectiveness of the SPADE: SPADE consistently outperforms corresponding architectures, whereas concatenating segmentation masks at every intermediate layer does not match its performance.The result holds for both decoder-style and encoder-decoder architectures, and the compact SPADE model also outperforms all baselines.
- Variations and controllability: Random-noise and downsampled-segmentation inputs produce similar performance, while SPADE enables diverse appearances for the same layout and external style-image control.Style control replaces input noise with an image-encoder embedding computed from the chosen style image.
5. Conclusion
The paper proposes spatially-adaptive normalization, which uses the input semantic layout during affine transformations in normalization layers. This enables photorealistic synthesis across diverse scenes and supports multi-modal and guided image synthesis.
- 5. Conclusion: Spatially-adaptive normalization uses the input semantic layout while performing affine transformations in normalization layers.The method integrates semantic-layout information into normalization rather than discarding it during processing.
- 5. Conclusion: The resulting model produces photorealistic outputs for diverse indoor, outdoor, landscape, and street scenes.The conclusion characterizes it as the first semantic image synthesis model with this capability across these scene categories.
- 5. Conclusion: The approach also supports multi-modal synthesis and guided image synthesis.These are demonstrated applications of the proposed normalization method.
A. Additional Implementation Details
The implementation uses SPADE ResBlks with nearest-neighbor upsampling, synchronized BatchNorm, and Spectral Norm in the generator, while the image encoder supports multimodal and style-guided synthesis. Training follows pix2pixHD with Hinge loss, an added KL Divergence term for encoder-based synthesis, and dataset-specific schedules.
- Generator: The generator stacks SPADE ResBlks with nearest-neighbor upsampling, synchronized BatchNorm across 8 GPUs, and Spectral Norm on every convolutional layer.The SPADE, SPADE ResBlk, and generator designs are shown in Figures 10–12.
- Image Encoder: The image encoder uses 6 stride-2 convolutional layers followed by two linear layers that produce the output distribution’s mean and variance.These components produce the latent representation used for the output distribution.
- Learning objective: The learning objective matches pix2pixHD’s loss weighting but replaces its LS-GAN term with Hinge loss.The modified adversarial loss is used while retaining the pix2pixHD weighting among loss terms.
- Learning objective: For multimodal and style-guided synthesis with the image encoder, training adds KL Divergence loss weighted by 0.05 and uses the reparameterization trick for back-propagation.The prior p(z) is standard Gaussian, while q is determined by a mean vector and variance vector.
- Training details: Training lasts 200 epochs on Cityscapes and ADE20K, 100 on COCO-Stuff, and 50 on Flickr Landscapes, with batch size 32.Images are 256 × 256 except Cityscapes at 512 × 256; learning rates linearly decay to 0 from epochs 100 to 200 for Cityscapes and ADE20K.
B. Additional Ablation Study
The ablation study finds that perceptual and GAN feature matching losses are important, while increasing discriminator depth does not improve performance. Spectral Norm, synchronized BatchNorm, TTUR, hinge loss, and SPADE each contribute to improvements over pix2pixHD++.
- Loss and discriminator ablations: Removing either the perceptual loss or GAN feature matching loss causes a performance drop under the mIoU metric.Both losses are inherited from pix2pixHD’s learning objective.
- Loss and discriminator ablations: Adding one convolutional layer to deepen the pix2pixHD discriminator does not improve performance.The additional layer is inserted at the top of the discriminator.
- pix2pixHD++ component ablations: Spectral Norm, synchronized BatchNorm, TTUR, and the hinge loss objective each contribute to pix2pixHD++’s performance boost.pix2pixHD++ is the strong baseline derived from pix2pixHD.
- pix2pixHD++ component ablations: Adding SPADE to pix2pixHD++ further improves performance.The ablation compares SPADE against the strong pix2pixHD++ baseline.
C. Additional Results
Additional experiments show SPADE’s synthesis results across several datasets and baselines, while multi-modal sampling and an interactive interface enable diverse and user-controlled image generation.
- Dataset comparisons: Additional results compare the proposed method with CRN and pix2pixHD on COCO-Stuff and ADE20K.Figures 16–18 present these comparisons across the two datasets.
- Dataset comparisons: Additional results compare the proposed method with CRN, SIMS, and pix2pixHD on ADE20K-outdoor and Cityscapes.Figures 19–20 show these dataset-specific comparisons.
- Multi-modal synthesis: Sampling different z vectors from a standard multivariate Gaussian produces images with diverse appearances.Figure 21 presents additional multi-modal synthesis results on the Flickr Landscapes Dataset.
- User control: The semantic image synthesis interface lets users paint labels to create photorealistic landscapes, vary appearance for one mask, and transfer a style image’s appearance.These capabilities are demonstrated in the accompanying video.