SEAN: Image Synthesis with Semantic Region-Adaptive Normalization

• SPADE’s single style code cannot provide detailed per-region control, and style is undefined for labeled output regions absent from the input style image.SEAN addresses this limitation by accepting one style image per region or region instance; SPADE was the starting point for the work.Source: §1 · p. 2
• SEAN creates spatially varying normalization parameters per semantic region, conditioned on both the segmentation mask and style input images.The proposed building block injects style through normalization parameters at multiple network layers.Source: §1 · p. 2
• SEAN improves synthesized-image quality over SPADE and Pix2PixHD on quantitative metrics including FID and visual inspection.The evaluation spans CelebAMask-HQ, CityScapes, ADE20K, and the authors’ Fac¸ades dataset, using metrics including FID, PSNR, RMSE, and segmentation performance.Source: §1 · p. 2
• SEAN improves per-region style encoding, making reconstructed images more similar to input style images by PSNR and visual inspection.Source: §1 · p. 2
• Users can select a separate style image for each semantic region, enabling interactive region-by-region style transfer and per-region style interpolation.The introduction describes these editing capabilities as providing higher quality and better control than current state-of-the-art methods.Source: §1 · p. 2

Generative Adversarial Networks

• GANs have advanced through architecture, loss, and regularization improvements, while conditional GANs [36] add user control through conditioning inputs.The paper builds on image-input conditional GANs for image-to-image translation.Source: §2 · p. 3

Image-to-Image Translation

• Image-to-image translation expanded to unsupervised, few-shot, high-resolution, multimodal, and multi-domain settings, while semantic synthesis enables control through layout edits.SPADE [39], also called GauGAN, is described as producing the highest-quality results to date; SEAN aims to improve it with per-region style encoding.Source: §2 · p. 3

Style Encoding

• Style can be encoded in feature statistics or network weights, but statistics matching is slow and weight-based methods require a separate network for each style image.Feature-statistics methods optimize to match image-classification-network features [13], while weight-based style transfer runs in real time.Source: §2 · p. 3
• Normalization-layer encoding supports arbitrary real-time style transfer [21] and is used by StyleGAN, FUNIT, and SPADE [39]; SEAN builds on this approach.The paper says SEAN produces higher-quality results and enables more-detailed user control.Source: §2 · p. 3

3.1. How to Encode Style?

• The per-region style encoder produces a 512 × s style matrix, using region-wise average pooling to distill a 512-dimensional code for each semantic label.Its bottleneck structure removes information considered irrelevant to style; absent regions receive zero codes, and instance-level encoding is also possible for datasets with instance labels.Source: §3.1 · p. 3

3.2. How to Control Style?

• SEAN modulates generator scales and biases using parameters that depend on both per-region style codes and the segmentation mask.Style codes are broadcast to their corresponding regions, then style- and mask-derived parameters are combined in a learnable weighted sum.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

4.1. Network Architecture

• The generator builds on SPADE [39], interleaving SEAN ResNet blocks with upsampling layers; its style encoder maps an input image to a style matrix.Source: §4.1 · p. 4, Figure 4 · p. 5
• Each SEAN ResBlk has three convolutional layers modulated by SEAN blocks that take per-region style codes and a semantic mask.Masks are downsampled to feature-map size, and 1 × 1 convolutions transform style codes per region. Learned per-channel noise scaling can improve synthesis quality.Source: §4.1 · p. 4

4.2. Training and Inference

• Training reconstructs input images from extracted per-region style codes and segmentation masks, using conditional adversarial, feature matching [47], and perceptual losses.Source: §4.2 · p. 4
• At inference, users can supply an arbitrary segmentation mask and select a separate 512-dimensional style code for each semantic region.Source: §4.2 · pp. 4–5
• SEAN outperforms current leading methods on all datasets for SSIM and PSNR (higher is better) and RMSE (lower is better).Source: Table 1 · p. 5

Datasets

• Across CelebAMask-HQ, ADE20K, Cityscapes, and Facades, all images shown are generated from the test set only; neither segmentation masks nor style images were seen during training.CelebAMask-HQ has 19 region categories, ADE20K 150 region labels, and Cityscapes 35; Facades masks are automatically calculated by a pre-trained DeepLabV3+ network. The recommended split is followed for each dataset.Source: §5 · p. 5

Quantitative comparisons

• SEAN clearly beats the state of the art on all four datasets, and Table 2 reports better FID than leading methods on every dataset.The authors identify FID as the metric most indicative of visual quality, while noting that lower FID does not always correspond to better visual results.Source: §5 · p. 6, Table 2 · p. 6

Qualitative results

• Visual comparisons show substantial quality differences; SPADE and Pix2Pix struggle with extreme poses and older people.The authors conjecture that SEAN’s improved style encoding helps it learn more data variability.Source: §5 · p. 6, Figure 6 · p. 7
• Per-region style encoding enables iterative image editing, style interpolation, and style crossover across ResBlocks.Crossover results indicate that earlier layers control larger features while later layers mainly determine color scheme.Source: Figure 5 · p. 6, §5 · p. 8, Figure 7 · p. 8

Variations of SEAN generator (Ablation studies)

• Combining Pix2PixHD’s style encoder with SPADE’s generator yields SPADE++, which modestly improves SPADE, while SEAN variants outperform it in FID and PSNR.The authors attribute Pix2PixHD’s better PSNR reconstruction to style extraction and SPADE’s better FID to its generator.Source: §5 · p. 8
• Among style encoders, the unified encoder has better FID and visual quality despite weaker reconstruction; a bottleneck helps, and learnable noise improves PSNR and FID.The compared encoder designs are SEAN-level, ResBlk-level, and unified; the first two perform style extraction per layer, unlike the unified encoder’s shared computation plus per-layer computation.Source: §5 · p. 8

6. Conclusion

• SEAN extends SPADE with a normalization block that extracts style from a reference image for each semantic region and converts it into spatially varying normalization parameters.This enables a separate style reference image to be specified for each region.Source: §6 · p. 8
• Across multiple datasets, SEAN reports better quantitative metrics than the current state of the art and improves reconstruction quality, variability, and visual quality over the best previous method.The conclusion also reports improvements in the basic setting where one style image is provided for all regions.Source: §6 · p. 8
• SEAN supports interactive image editing by changing segmentation masks or the style assigned to a region, and by interpolating between two reference-image styles per region.These capabilities are also described in the paper context.Source: §6 · p. 8
• Future work will extend SEAN to processing meshes, point clouds, and textures on surfaces.Source: §6 · p. 8

A. Additional Implementation Details

• The generator uses SEAN ResBlks followed by nearest-neighbor upsampling, injecting style codes only into its first six blocks and other inputs into every block.Figure 8 depicts the generator and its style-code and segmentation-mask inputs.Source: A. Additional Implementation Details · p. 11, Figure 8 · p. 11
• The discriminator uses two scales, instance normalization, Leaky ReLU, and spectral normalization; it receives concatenated segmentation masks and style images and uses PatchGAN loss calculation.The stated design follows SPADE and Pix2PixHD.Source: A. Additional Implementation Details · p. 11, Figure 9 · p. 11
• The style encoder processes a style image with its segmentation mask through a bottleneck CNN and region-wise average pooling to produce style codes.Figure 11 shows the style image and mask as encoder inputs and the style codes as output.Source: A. Additional Implementation Details · p. 11, Figure 11 · p. 12
• The final loss combines adversarial, feature-matching, and perceptual terms, with the adversarial loss built using the Hinge loss.The experiments set λ1 = λ2 = 10, following SPADE and Pix2PixHD.Source: A. Additional Implementation Details · pp. 11–12
• Training runs for 50 epochs; input images are resized to 256 × 256, except CityScapes images at 512 × 256, and weights use Glorot initialization.The stated schedule applies to all datasets mentioned in the main paper.Source: A. Additional Implementation Details · p. 12

B. Additional Experimental Details

• Figure 10 supplements rows 5 and 6 of Table 3 with two style-encoder variants used in a SEAN ResBlk.Source: Table 3 · p. 12
• The SEAN-level encoder extracts different style codes for each SEAN block within a SEAN ResBlk.Source: Table 3 · p. 12
• The ResBlk-level encoder supplies the same extracted style codes to all SEAN blocks within a SEAN ResBlk.Source: Table 3 · p. 12

C. Justification of Encoder Choice

• The unified encoder generates better-quality images than the SEAN-level encoder, especially for extreme poses and unlabeled regions.This visual comparison justifies the choice of unified encoder.Source: C. Justification of Encoder Choice · p. 12
• The visual-quality advantage of the unified encoder is particularly evident on challenging inputs.The cited examples are extreme poses and unlabeled regions.Source: C. Justification of Encoder Choice · p. 12

ST-branch vs. Mask-branch

• ST-branch and mask-branch both contribute, with mixing parameters typically 0.35–0.7; removing the mask-branch worsens results, and their interaction is more complex than mixing alone.Figure 13 illustrates the branch contributions across SEAN normalization blocks.Source: D. Additional Analysis · p. 12, Figure 13 · p. 13

Encoder choice

• The unified encoder produces more robust style transfer than the SEAN-level encoder, which is more sensitive to poses and unlabeled style-image regions due to overfitting.Figure 12 compares the two encoder choices.Source: Figure 12 · p. 13

Extreme Cases

• SEAN transfers highly complex textures from an artistic image onto a human face.Figure 14 demonstrates complex texture transfer.Source: D. Additional Analysis · p. 13, Figure 14 · p. 13
• The method is highly flexible, allowing users to paint a semantic region at a spatially unreasonable location arbitrarily; users can also put eyes anywhere on a face.Source: Figure 14 · p. 13, Figure 15 · p. 13

E. User Study

• 575 AMT workers answered 600 questions, each showing ground truth, its segmentation mask, and reconstructions from SEAN, Pix2PixHD [47], and SPADE [39].Workers selected the reconstruction closest to ground truth with the fewest artifacts.Source: E. User Study · p. 13
• The 600 questions enumerated all six orders of the three reconstructions across 100 randomly selected image sets to reduce image-order effects.Source: E. User Study · p. 13
• SEAN significantly outperformed Pix2PixHD [47] and SPADE [39] in the CelebAMask-HQ user preference study.The study asked workers to choose the reconstruction closest to ground truth and with the fewest artifacts.Source: Table 4 · p. 13, E. User Study · p. 13

F. Additional Results

• The interactive demo transfers styles from various images to arbitrary segmentation masks and creates new styles by interpolating existing ones.The UI is presented as a demonstration of flexible image editing; recorded videos appear in the supplementary material.Source: F. Additional Results · p. 13
• Additional style-transfer results are shown on CelebAMask-HQ, with style-interpolation results on CelebAMask-HQ and ADE20K.Figures 16–18 show these examples.Source: F. Additional Results · p. 13, Figure 16 · p. 14, Figure 17 · p. 15
• SEAN produces higher-quality reconstructions than Pix2PixHD and SPADE on CelebAMask-HQ, ADE20K, CityScapes, and Fac¸ades.The paper presents visual comparisons in Figures 19–22.Source: F. Additional Results · p. 13, Figure 19 · p. 16, Figure 20 · p. 17