PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Semantic Image Synthesis via Class-Adaptive Normalization

Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Gang Hua, Nenghai Yu

arXiv:2012.04644v2cs.CVcs.GR

TL;DR

SPADE succeeds at semantic image synthesis but incurs substantial parameter and computation overhead, and the source of its advantage is not fully understood. This paper analyzes SPADE and proposes class-adaptive CLADE and spatially adaptive CLADE-ICPE. CLADE achieves comparable synthesis results with reduced overhead, and the authors report using it in SPADE-based methods with comparable or better results.

  • Problem

    SPADE improves semantic image synthesis, but its substantial parameter and computation overhead and the source of its advantage remain insufficiently understood.

    Source: §1 · p. 1

  • Method

    The paper analyzes SPADE and proposes class-adaptive CLADE, plus CLADE-ICPE, which adds intra-class positional encoding to modulation parameters.

    Source: §1 · pp. 1–2

  • Results

    CLADE achieves comparable synthesis results to SPADE with reduced parameter and computation overhead, and its use in SPADE-based methods yields comparable or even better results.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    The reported findings support CLADE as a more efficient alternative to SPADE that can also be adopted in other SPADE-based methods.

    Source: §1 · p. 2, §5 · p. 12

Abstract

from arXiv · show

Spatially-adaptive normalization (SPADE) is remarkably successful recently in conditional semantic image synthesis \cite{park2019semantic}, which modulates the normalized activation with spatially-varying transformations learned from semantic layouts, to prevent the semantic information from being washed away. Despite its impressive performance, a more thorough understanding of the advantages inside the box is still highly demanded to help reduce the significant computation and parameter overhead introduced by this novel structure. In this paper, from a return-on-investment point of view, we conduct an in-depth analysis of the effectiveness of this spatially-adaptive normalization and observe that its modulation parameters benefit more from semantic-awareness rather than spatial-adaptiveness, especially for high-resolution input masks. Inspired by this observation, we propose class-adaptive normalization (CLADE), a lightweight but equally-effective variant that is only adaptive to semantic class. In order to further improve spatial-adaptiveness, we introduce intra-class positional map encoding calculated from semantic layouts to modulate the normalization parameters of CLADE and propose a truly spatially-adaptive variant of CLADE, namely CLADE-ICPE.Through extensive experiments on multiple challenging datasets, we demonstrate that the proposed CLADE can be generalized to different SPADE-based methods while achieving comparable generation quality compared to SPADE, but it is much more efficient with fewer extra parameters and lower computational cost. The code and pretrained models are available at \url{https://github.com/tzt101/CLADE.git}.

1 INTRODUCTION

The introduction examines why SPADE improves semantic image synthesis yet adds substantial computational and memory overhead. It argues that semantic-awareness may matter more than spatial adaptiveness, motivating CLADE and its spatially adaptive extension, CLADE-ICPE.

1 INTRODUCTION

  • SPADE addresses semantic information washed away by conventional normalization by conditioning spatially varying modulation on the segmentation mask.Earlier methods fed masks directly into networks built from convolution, normalization, and nonlinearity layers.Source: §1 · p. 1
  • SPADE's modulation parameters are nearly spatial-invariant within regions of one semantic class, especially with high-resolution masks, making repeated spatial modulation potentially redundant.The paper attributes this pattern to the shallow two-layer modulation network and notes that SPADE blocks recur before almost every convolution.Source: §1 · p. 1
  • CLADE modulates normalized activations by semantic class rather than spatial position, semantic shape, or mask layout, without an extra modulation network.For the ADE20k generator, CLADE adds 4.57% parameters and 0.07% computation, versus SPADE's 39.21% and 234.73%, respectively.Source: §1 · pp. 1–2
  • CLADE-ICPE adds an intra-class positional encoding map to modulation parameters, making them spatially adaptive within regions of the same semantic class.The map represents normalized relative distance from each pixel to its semantic object center.Source: §1 · p. 2
  • Across multiple challenging datasets, replacing SPADE layers with CLADE achieves comparable performance with much smaller models and lower computation cost.The introduction names Cityscapes, COCO-Stuff, and ADE20k, including ADE20k-outdoor.Source: §1 · p. 2

2 RELATED WORKS

The related work places semantic image synthesis among conditional generation tasks and reviews GAN and normalization advances. It highlights SPADE's influence and frames its efficiency as an understudied concern.

2.1 Generative Adversarial Networks

  • GAN research spans training stability, architectural constraints, higher-resolution synthesis, and improved realism and control through latent codes.The authors distinguish their focus on intuitive control via additional conditional inputs.Source: §2.1 · p. 2

2.2 Conditional Image Synthesis

  • Conditional image synthesis generates photo-realistic images from inputs such as text or images; semantic image synthesis conditions generation on segmentation masks.The review mentions pix2pix, pix2pixHD, SIMS, and SPADE, and notes SPADE's use as a backbone in multiple downstream tasks.Source: §2.2 · pp. 2–3

2.3 Normalization Layers

  • Normalization layers normalize features before applying learned scale and shift; conditional variants make these parameters depend on external inputs.SPADE restores mask semantics through spatially adaptive modulation, while CLADE is proposed as a comparable-performing alternative with negligible cost.Source: §2.3 · p. 3

3 SEMANTIC IMAGE SYNTHESIS

SPADE’s modulation parameters become increasingly concentrated within semantic classes at higher mask resolutions, suggesting semantic awareness matters more than spatial adaptiveness. CLADE retains class awareness with much lower overhead, while CLADE-ICPE adds positional encoding for spatial adaptation.

3.1 Revisit Spatially-Adaptive Normalization

  • SPADE’s γ and β distributions within semantic classes become more concentrated as input-mask resolution increases, supporting semantic awareness as a key contributor to its performance.The histograms cover building, sky, tree, human, and car classes in ADE20k validation data.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Class-Adaptive Normalization

  • CLADE assigns modulation parameters by semantic class, preserving semantic awareness while remaining independent of mask position, shape, and layout.It maintains class-wise parameter banks and uses guided sampling to fill each semantic region with its class parameters.Source: §3.2 · p. 4

3.3.1 Analysis of SPADE

  • For ADE20k SPADE, average parameter overhead is 39.21% and average extra FLOPs are 234.73% relative to convolutional layers.Some SPADE layer parameter ratios exceed 600%.Source: §3.3.1 · pp. 5–6

3.3.2 Analysis of CLADE

  • CLADE achieves comparable performance to SPADE with average parameter and FLOPs ratios of 4.57% and 0.07%, respectively.The ratios are relative to following convolutional layers in the ADE20k generator.Source: §3.3.2 · p. 6

3.4 Spatially-Adaptive CLADE

  • CLADE-ICPE derives normalized relative-distance maps from semantic object centers and uses them to modulate CLADE’s γ and β spatially.Object centers are found from the largest connected component of each semantic category; a 1 × 1 convolution maps positional encoding to modulation parameters.Source: §3.4 · p. 6
  • CLADE-ICPE’s FLOPs ratio is nearly twice CLADE’s, but remains low compared with SPADE: 0.14% versus 234.73%.The parameter ratio is almost the same as CLADE’s.Source: §3.4 · p. 7

3.5 CLADE Generator

  • The CLADE generator follows SPADE’s architecture with residual blocks and upsampling, progressively increasing output resolution; multimodal synthesis adds an encoder whose sampled style vector guides global diversity.CLADE replaces SPADE blocks, and the encoder produces a mean and variance vector for sampling.Source: §3.5 · p. 7

4 EXPERIMENTS

Experiments use four main datasets and two additional datasets to assess generalization to SPADE-dependent methods. The generator follows SPADE’s training setting, with specified losses and comparisons against several semantic image synthesis methods.

4.1 Datasets

  • The main experiments use ADE20k, ADE20k-outdoor, COCO-Stuff, and Cityscapes, spanning 150, 182, and 35 categories for ADE20k, COCO-Stuff, and Cityscapes, respectively.ADE20k has 25,210 images; COCO-Stuff has 118,000 training and 5,000 validation images; Cityscapes has 2,975 training and 500 validation images.Source: §4.1 · p. 7
  • CelebAMask-HQ and DeepFashion provide additional evaluations of CLADE generalization to recent SPADE-dependent methods.CelebAMask-HQ has 30,000 masks; the processed DeepFashion split has 30,000 training and 2,247 validation images.Source: §4.1 · p. 7

4.2 Implementation Details

  • Training follows SPADE’s setting with a multi-scale discriminator, hinge GAN, feature-matching and perceptual losses, and 200 epochs.For multi-modal synthesis, a KL-divergence term is added; image resolution is 256 × 256 except for Cityscapes at 256 × 512.Source: §4.2 · p. 7
  • Evaluation compares CLADE with SPADE, pix2pixHD, CC-FPSE, and DAGAN, using author-provided pretrained models for SPADE, CC-FPSE, and DAGAN.The authors train pix2pixHD using the code and settings provided by its authors.Source: §4.2 · p. 7

4.3 Evaluation Metrics

CLADE achieves performance comparable to SPADE while reducing model complexity across datasets. The evaluation also finds improved FID with CLADE-ICPE and worse performance from a similarly low-FLOP SPADE variant.

4.3 Evaluation Metrics

  • Evaluation measures semantic-mask prediction quality with mIoU and pixel accuracy, and image-distribution distance with FID.FID compares generated validation images with real training images; segmentation models vary by dataset.Source: §4.3 · p. 7

4.4 Quantitative Results

  • CLADE matches SPADE’s performance while reducing complexity across datasets; on COCO-Stuff, it scores 36.77 mIoU and 68.08 pixel accuracy, slightly better than SPADE.On Cityscapes, CLADE has about 74% of SPADE’s parameters and about 4× fewer FLOPs.Source: §4.4 · pp. 7–8, Table 1 · p. 8
  • When backbone costs are excluded, CLADE’s parameter and runtime advantages over SPADE become more apparent.Table 3 separates backbone costs from those of the normalization layers.Source: Table 3 · p. 9, §4.4 · p. 9
  • At similar FLOPs, SPADE-light performs much worse than CLADE on every dataset.SPADE-light reduces channels in its convolution layers to match CLADE’s FLOPs.Source: Table 4 · p. 9, §4.4 · p. 9
  • CLADE uses less training memory than other methods, and its runtime is about 2× faster than SPADE on GPU, with a more significant speedup on a single-threaded CPU.The authors note that the runtime advantage is more useful on low-end devices.Source: Table 2 · p. 8, §4.4 · p. 9
  • CLADE-ICPE significantly improves FID on all datasets, with negligible additional parameters and FLOPs and only small increases in runtime and training memory.Its FID advantage over SPADE is especially pronounced on Cityscapes.Source: §4.4 · p. 9

4.5 User Study

In a pairwise user study, participants showed no clear preference between CLADE and SPADE, while preferring CLADE to pix2pixHD. They also generally preferred CLADE-ICPE to CLADE, particularly on Cityscapes.

4.5 User Study

  • The user study reports the percentage of users favoring CLADE over each competing method.The authors use pairwise comparisons of synthesis images made from the same semantic mask.Source: Table 5 · p. 9, §4.5 · p. 9
  • Users showed no obvious preference between CLADE and SPADE, but clearly preferred CLADE over pix2pixHD across all four datasets.The study compared image pairs generated from the same semantic mask; 20 users assessed 40 results per method for each pairwise comparison.Source: §4.5 · p. 9, Table 5 · p. 9
  • Users preferred CLADE-ICPE over CLADE, especially on Cityscapes, while showing little preference between them on COCO-Stuff.This pattern agrees with the reported FID comparisons.Source: §4.5 · p. 9

4.6 Qualitative Results

Qualitative comparisons show CLADE producing high-fidelity images comparable to SPADE, including on challenging COCO-Stuff and higher-resolution Cityscapes. CLADE-ICPE adds richer within-class textures where SPADE and CLADE can produce blurry or repeated patterns.

4.6 Qualitative Results

  • CLADE’s visual results are comparable to SPADE and are sometimes slightly better, while both are more realistic overall than Pix2pixHD.The comparison includes indoor ADE20k and outdoor ADE20k-outdoor examples.Source: §4.6 · p. 9, Figure 8 · p. 10
  • The method can still work very well and generate high-fidelity results, and Figure 10 shows a similar conclusion for higher-resolution semantic image synthesis on Cityscapes (256 × 512).Source: §4.6 · p. 10
  • On large semantic regions, CLADE-ICPE can render vivid textures where SPADE and CLADE show blurry views or repeated patterns.The examples contrast a textured view out of a window with the blurred output of SPADE and CLADE.Source: §4.6 · p. 10

4.7 Multi-Modal and Style-Guided Synthesis

The generator supports multimodal and style-guided synthesis by changing its input or adding style guidance. Results show diverse outputs from noise vectors and global style control from reference images.

4.7 Multi-Modal and Style-Guided Synthesis

  • For style-guided synthesis, a style encoder uses reference images to guide the generated image’s global style.The examples include styles such as sunny days, dusk, and night.Source: §4.7 · pp. 10–11
  • Different noise vectors or reference style images yield diverse realistic outputs while retaining the semantic layout as guidance.The figure shows noise-guided examples in its top row and reference-style examples in its bottom row.Source: Figure 11 · p. 11, §4.7 · p. 11
  • For multimodal synthesis, the generator input is replaced with random noise instead of the default downsampled semantic mask.Different noise vectors can produce diverse images from the same semantic layout.Source: §4.7 · p. 11, Figure 11 · p. 11

4.8 Ablation Analysis of Combining CLADE and SPADE

Using SPADE on low-resolution layers and CLADE on higher-resolution layers can balance quality and efficiency; selected mixtures even slightly exceed all-SPADE mIoU while running faster.

4.8 Ablation Analysis of Combining CLADE and SPADE

  • The seven SPADE ResBlks are numbered by increasing resolution, with the second and third at the same resolution for 256 × 256 synthesis.The ablation tests different SPADE-to-CLADE transition points on ADE20k-outdoor and Cityscapes.Source: §4.8 · p. 11
  • Selected mixtures using SPADE on low-resolution layers and CLADE from middle ResBlks achieved slightly higher mIoU than all-SPADE while running more efficiently.The reported transitions include ResBlks 6 and 7 on ADE20k-outdoor and ResBlks 3, 5, and 6 on Cityscapes.Source: §4.8 · p. 11
  • Running time decreases as more layers use CLADE, while SPADE on low-resolution layers and CLADE on high-resolution layers is proposed as a quality-efficiency balance.The motivation is that SPADE parameters are less centralized and can supply spatial variance at low resolutions, whereas higher-resolution layers have more centralized parameters and greater computation cost.Source: §4.8 · p. 11

4.9 Ablation Analysis of Intra-Class Positional Encoding

The positional map encodes relative position within semantic objects, but its integration matters: modulation of CLADE parameters performs best among the tested approaches. The reported gain is more pronounced on Cityscapes, which the authors associate with its large categories’ clearer internal structure.

4.9 Ablation Analysis of Intra-Class Positional Encoding

  • The positional map encodes each pixel’s normalized relative distance to its semantic object center, adding intra-class spatial information that may enrich details but can also be harmful when used inappropriately.The tested integrations are direct concatenation with the downsampled mask (+disti), concatenation with normalized features after a 1 × 1 convolution (+distf), and modulation of CLADE’s semantic-adaptive parameters (+distp).Source: §4.9 · p. 11
  • +distp achieves the best FID on ADE20k, ADE20k-outdoor, and Cityscapes, while +disti performs worst.Comparisons indicate that adding spatial information at normalized-feature level is beneficial, and element-wise multiplication in +distp is more effective than channel concatenation in +distf.Source: §4.9 · p. 11
  • The performance gain is reported as much more significant on Cityscapes than ADE20k; the authors attribute this to Cityscapes’ large categories having clearer internal structure.They note that ADE20k-outdoor also has large categories such as sky and sea, but these have relatively less complex internal structure and benefit less from spatial adaptiveness.Source: §4.9 · p. 11

4.10 Generalization Ability to SPADE-Based Methods

Replacing SPADE with CLADE in representative SPADE-based methods largely preserves performance while reducing parameters and computation. The comparisons cover GroupDNet and SEAN on their respective datasets, with Cityscapes added as a supplement.

4.10 Generalization Ability to SPADE-Based Methods

  • The comparison replaces only the normalization layer in GroupDNet and SEAN, using DeepFashion, CelebAMask-HQ, and Cityscapes.GroupDNet is a semantic-level multimodal image synthesis method and SEAN focuses on face image synthesis; models use the official-code settings.Source: §4.10 · p. 12
  • About 20M fewer parameters and 7%–40% lower running time accompany CLADE replacements, while original performance is almost unaffected and most reported metrics improve slightly.For SEAN on CelebAMask-HQ, FID improves significantly; reported metrics include mIoU, accuracy, and FID.Source: §4.10 · p. 12
  • CLADE can be applied to SPADE-based or other semantic-mask-input methods, including where GroupDNet’s Cityscapes efficiency remains unsatisfactory in FLOPs and running time.The authors describe CLADE as an efficient counterpart in that setting.Source: §4.10 · p. 12

5 CONCLUSION

The paper argues that semantic-awareness, rather than spatial-adaptiveness, is SPADE’s essential advantage, motivating CLADE’s lower-overhead design. It also proposes CLADE-ICPE for spatial adaptiveness and reports comparable or better results when applying CLADE to SPADE-based methods.

5 CONCLUSION

  • The authors identify semantic-awareness, rather than spatial-adaptiveness, as SPADE’s most essential advantage.This observation motivates CLADE as a more efficient conditional normalization structure.Source: §5 · p. 12
  • CLADE achieves comparable synthesis results to SPADE while greatly reducing parameter and computation overhead.Source: §5 · p. 12
  • CLADE-ICPE introduces true spatial adaptiveness by using intra-class positional encoding to modulate CLADE parameters.Source: §5 · p. 12
  • Applying CLADE to recent SPADE-based methods yields comparable or even better results with substantially reduced parameters and computation costs.Source: §5 · p. 12
Loading 2012.04644v2…