Semantic Image Synthesis with Spatially-Adaptive Normalization

1. Introduction

• Semantic image synthesis converts a semantic segmentation mask into a photorealistic image and supports applications such as content generation and image editing [6] [22].Neural-network mapping methods are faster than earlier image-stitching or image-collection approaches and require no external image database [3] [6] [22].Source: §1 · p. 1
• Spatially-adaptive normalization modulates activations using semantic layouts through a learned spatial transformation, helping propagate semantic information throughout the network.The paper identifies conventional convolution, normalization, and nonlinearity stacks as suboptimal because normalization tends to wash away information in the input masks.Source: §1 · pp. 1–2
• Across COCO-Stuff [32], ADE20K, and Cityscapes [9], a compact network with the proposed layer produces significantly better results than several state-of-the-art methods.An extensive ablation study also demonstrates the proposed normalization layer's effectiveness against variants for semantic image synthesis.Source: §1 · p. 2
• The method supports multimodal and style-guided image synthesis, enabling controllable, diverse outputs.The paper points to Figure 1 as an illustration.Source: §1 · p. 2

Generative models

• The work uses GANs for conditional synthesis; a generator produces realistic images that a discriminator cannot distinguish from real ones.The passage also identifies variational autoencoders (VAEs) [28] as another deep generative model.Source: §2 · p. 2

Conditional image synthesis

• Conditional synthesis includes category-label models [3] [37], text-to-image methods [18], and image-to-image translation [20] [22] [24] [25] [33].Learning-based translation methods typically run faster at test time and produce more realistic results than earlier non-parametric methods. This work converts segmentation masks to photorealistic images using registered mask-image training pairs, and reports that its compact network achieves better results than leading methods.Source: §2 · p. 2

Normalization layers

• Unconditional normalization layers do not depend on external data and include Local Response Normalization, BatchNorm [21], InstanceNorm [46], Layer Normalization [2], Group Normalization, and Weight Normalization.Source: §2 · p. 2
• Conditional BatchNorm [11] and AdaIN [19] normalize activations, then use learned affine parameters inferred from external data; for style transfer, those parameters control global style uniformly across spatial coordinates.These layers were first used for style transfer and later adopted in vision tasks [3] [8] [10] [20] [26].Source: §2 · p. 2
• Unlike style-transfer normalization, the proposed layer uses spatially varying affine modulation for semantic-mask synthesis; Wang et al. use related spatial modulation for super-resolution.The paper focuses on modulating normalized activations with semantic maps at different scales for coarse-to-fine generation and aims to disentangle style and semantics.Source: §2 · p. 2

Spatially-adaptive denormalization

• SPADE normalizes activations channelwise, then applies learned, spatially varying scale and bias parameters derived from the semantic mask.A two-layer convolutional network computes the modulation parameters; the mask is embedded and convolved to produce γ and β tensors.Source: §3 · pp. 2–3, Figure 2 · p. 3
• With spatially-invariant modulation, SPADE reduces to Conditional BatchNorm [11] or AdaIN [19] under the corresponding input substitutions.The paper argues SPADE is better suited to semantic image synthesis because its modulation adapts to the segmentation mask.Source: §3 · p. 3

SPADE generator

• SPADE removes the generator encoder because mask-derived modulation encodes the layout, and uses ResNet blocks with upsampling and resolution-matched masks.A random-vector input also enables multimodal synthesis.Source: §3 · p. 3

Why does the SPADE work better?

• InstanceNorm can erase a uniform semantic mask: convolution produces uniform activations that normalize to zero, while SPADE modulates with the mask without normalizing it.The information loss can affect generator architectures that apply convolution followed by normalization to the semantic mask; SPADE instead normalizes only activations from the previous layer.Source: §3 · pp. 3–4

Multi-modal synthesis

• An image encoder and SPADE generator form a VAE that combines encoded style with segmentation-mask information; a KL-Divergence loss is added during training.At test time, the encoder can guide the generator using the style of a target image.Source: §3 · p. 4

Datasets and metrics

• Evaluation covers COCO-Stuff, ADE20K, ADE20K-outdoor, Cityscapes, and Flickr Landscapes, using segmentation accuracy and FID to assess layout alignment and image-distribution similarity.For Flickr Landscapes, the input masks are computed with a pretrained DeepLabV2 model; competing methods are trained on the same data and evaluated on the same validation set.Source: §4 · pp. 4–5

Quantitative comparisons

• SPADE outperforms leading methods across the benchmark datasets; on COCO-Stuff it achieves mIoU 35.2, about 1.5 times the previous leading method, and its FID is 2.2 times better.SIMS has lower FID on Cityscapes but poor segmentation performance there, reflecting that copied training-image patches can mismatch the input segments.Source: Table 1 · p. 5, §4 · p. 6

Qualitative results

• Across the evaluated datasets, SPADE produces visually realistic images with fewer visible artifacts, while SIMS can generate visually good images whose content deviates from the input layout.The reported qualitative comparisons cover COCO-Stuff, ADE20K, and Cityscapes.Source: Figure 5 · p. 5, Figure 6 · p. 5, §4 · p. 6

Human evaluation

• Users strongly favor SPADE’s results on every dataset, especially challenging COCO-Stuff and ADE20K; they also prefer it on Cityscapes despite high fidelity from all methods.The AMT study used 500 randomly generated questions per comparison and five workers per question.Source: §4 · pp. 6–7, Table 2 · p. 6

Effectiveness of SPADE

• Adding SPADE improves mIoU over corresponding baselines in both decoder-style and encoder-decoder architectures, while concatenating masks at intermediate layers does not match SPADE.The decoder-style SPADE generator also outperforms the strong baselines with fewer parameters.Source: §4 · p. 7, Table 3 · p. 7

Variations of SPADE generator

• SPADE’s performance is similar with random-noise or downsampled-segmentation generator input and remains reliable across parameter-free normalization choices; a 1x1 label-map kernel hurts performance.The authors suggest the 1x1 kernel limits use of label context.Source: Table 4 · p. 7, §4 · p. 8

Multi-modal synthesis

• For the same input segmentation mask, different noise inputs produce outputs with diverse appearances, demonstrating multimodal synthesis.The examples are shown on Flickr Landscapes.Source: §4 · p. 8, Figure 9 · p. 8

Semantic manipulation and guided image synthesis

• Users can draw segmentation masks to guide landscape synthesis and select an external style image to control the output’s global appearance.The style image’s embedding, computed by the image encoder, replaces the input noise.Source: §4 · p. 8

• The spatially-adaptive normalization uses the input semantic layout while performing affine transformations in normalization layers.Source: §5 · p. 8
• The method enables the first semantic image synthesis model to produce photorealistic outputs for indoor, outdoor, landscape, and street scenes.Source: §5 · p. 8
• The paper also demonstrates the method for multi-modal synthesis and guided image synthesis.Source: §5 · p. 8

A. Additional Implementation Details

• The generator uses nearest-neighbor upsampling and SPADE ResBlks, with synchronized BatchNorm across 8 GPUs and Spectral Norm on every convolution.The segmentation mask enters through the SPADE ResBlks; when a residual block changes channel count, its skip connection is learned.Source: A. Additional Implementation Details · p. 11, Figure 12 · p. 11, Figure 11 · p. 11
• The discriminator follows pix2pixHD with a multi-scale InstanceNorm design and Spectral Norm on all convolutional layers.It takes the concatenated segmentation mask and generated image as input and uses a Patch-GAN design.Source: A. Additional Implementation Details · p. 11, Figure 13 · p. 12
• The image encoder uses 6 stride-2 convolutions and 2 linear layers to produce mean and variance; the objective replaces pix2pixHD’s LS-GAN [34] term with hinge loss [31].Other loss-term weights match pix2pixHD.Source: A. Additional Implementation Details · p. 11, Figure 14 · p. 12
• For multi-modal and style-guided synthesis, the image encoder’s mean and variance vectors determine q, whose reparameterized noise inputs the generator; the KL Divergence loss weight is 0.05 [28].The prior distribution p(z) is standard Gaussian.Source: A. Additional Implementation Details · p. 11
• Training runs for 200 epochs on Cityscapes and ADE20K, 100 on COCO-Stuff, and 50 on Flickr Landscapes, with batch size 32 and Glorot initialization.Images are 256 × 256 except Cityscapes at 512 × 256; learning rate decays linearly from epoch 100 to 200 on Cityscapes and ADE20K.Source: A. Additional Implementation Details · p. 12

B. Additional Ablation Study

• Removing either the perceptual loss or GAN feature matching loss lowers performance, while adding a discriminator convolutional layer does not improve results.These losses are inherited from pix2pixHD.Source: B. Additional Ablation Study · p. 13, Table 5 · p. 13
• Spectral Norm, synchronized BatchNorm, TTUR [17], and the hinge loss objective each boost pix2pixHD++, and adding SPADE further improves it.Table 5 reports these ablations using mIoU.Source: B. Additional Ablation Study · p. 13, Table 5 · p. 13
• The pix2pixHD++ variants without Sync BatchNorm or Spectral Norm still differ from pix2pixHD by using hinge loss, TTUR, a large batch size, and Glorot initialization.Source: B. Additional Ablation Study · p. 13

• Figures 16–18 show additional COCO-Stuff and ADE20K synthesis results compared with CRN [6] and pix2pixHD.Source: C. Additional Results · p. 13, Figure 16 · p. 14, Figure 17 · p. 15
• Figures 19–20 show additional ADE20K-outdoor and Cityscapes synthesis results compared with CRN [6], SIMS, and pix2pixHD.Source: C. Additional Results · p. 13, Figure 19 · p. 17, Figure 20 · p. 18
• Sampling latent vectors from a standard Gaussian produces diverse-appearance multi-modal results on the Flickr Landscapes Dataset in Figure 21.Source: C. Additional Results · p. 13, Figure 21 · p. 19
• The accompanying video demonstrates painting semantic labels to create photorealistic landscapes, synthesizing diverse appearances from one mask, and transferring style from a provided image.Source: C. Additional Results · p. 13