PaperlayerView exact revision on arXiv

Source-linked AI summary

SEAN: Image Synthesis with Semantic Region-Adaptive Normalization

Peihao Zhu, Rameen Abdal, Yipeng Qin, Peter Wonka

arXiv:1911.12861v2cs.CVcs.GReess.IV

TL;DR

Conditional GANs need to generate images with segmentation-controlled layouts while offering more precise style control than global conditioning provides. SEAN introduces normalization driven by per-region style codes and masks, and reports improved synthesis quality, reconstruction similarity, and interactive editing capabilities over prior methods. The method is evaluated across multiple datasets with quantitative and visual comparisons.

  • Problem

    Conditional GANs must control image layout through segmentation masks, but SPADE’s single global style code limits detailed regional control and leaves unmatched regions undefined.

  • Method

    SEAN extracts style codes per semantic region and uses them with segmentation masks to produce spatially varying normalization parameters throughout the generator.

  • Results

    SEAN reports clear improvements over SPADE and Pix2PixHD in synthesized-image quality and improves reconstruction similarity to input styles as measured by PSNR and visual inspection.

  • Takeaways & Limitations

    SEAN enables selecting, transferring, and interpolating styles independently for semantic regions during interactive image editing.

  • Takeaways & Limitations

    Evaluation uses recommended dataset splits with test images, segmentation masks, and style images unseen during training.

Abstract

from arXiv · show

We propose semantic region-adaptive normalization (SEAN), a simple but effective building block for Generative Adversarial Networks conditioned on segmentation masks that describe the semantic regions in the desired output image. Using SEAN normalization, we can build a network architecture that can control the style of each semantic region individually, e.g., we can specify one style reference image per region. SEAN is better suited to encode, transfer, and synthesize style than the best previous method in terms of reconstruction quality, variability, and visual quality. We evaluate SEAN on multiple datasets and report better quantitative metrics (e.g. FID, PSNR) than the current state of the art. SEAN also pushes the frontier of interactive image editing. We can interactively edit images by changing segmentation masks or the style for any given region. We can also interpolate styles from two reference images per region.

1. Introduction

The paper targets conditional image generation with segmentation-controlled layout and addresses SPADE’s limited global style control by introducing region-specific style conditioning and normalization. SEAN improves synthesis quality, reconstruction fidelity, and interactive region-level editing.

  • Motivation: Conditional GANs generate images whose layout is specified by semantic segmentation masks.The motivation includes adding realistic styles to labeled regions such as facial features or landscape components.
  • Limitations of SPADE: SPADE’s single style code cannot define styles for regions absent from the input style image or support distinct style images for different regions.The paper identifies this as limiting high-quality synthesis and detailed control.
  • SEAN: SEAN injects style through spatially varying normalization parameters that depend on both per-region style inputs and the segmentation mask.This extends SPADE by applying style information as normalization parameters across network layers.
  • Results: SEAN achieves clear quantitative and visual improvements over SPADE and Pix2PixHD for synthesized-image quality.The evaluation spans multiple datasets and metrics including FID, PSNR, RMSE, and segmentation performance.
  • Results: SEAN improves reconstructed-image similarity to input styles, measured by PSNR and visual inspection.Its per-region style encoding supports more faithful reconstruction.
  • Interactive Editing: Users can select a style image for each semantic region, enabling interactive region-wise style transfer and per-region style interpolation.These editing capabilities are illustrated in the cited figures, including the ADE20K editing sequence.

2. Related Work

Related work establishes semantic image synthesis and style encoding as complementary foundations for controllable image generation. SEAN builds on normalization-based style encoding to provide more detailed per-region control than prior approaches.

  • Image-to-Image Translation: Image-to-image translation includes semantic image synthesis, where users control outputs by modifying semantic layout images.SPADE is described as the highest-quality existing model and the starting point for SEAN.
  • Style Encoding: Existing style-encoding methods represent style through image-feature statistics, neural-network weights, or normalization-layer parameters.These approaches differ in optimization cost, flexibility, and whether separate networks are needed for each reference style.
  • Style Encoding: Normalization parameters are the most practical prior style representation because they support arbitrary real-time style transfer and are used in high-quality networks.SEAN extends this approach with per-region style encoding and more detailed user control.

3. Per Region Style Encoding and Control

SEAN encodes a separate style code for each semantic region and combines these codes with segmentation masks to produce spatially varying normalization parameters for controlled synthesis.

  • 3.1. How to Encode Style?: The per-region style encoder outputs a 512 × s style matrix, with each column representing one semantic region’s style code.Region-wise average pooling removes shape information and produces 512-dimensional vectors; absent regions are set to 0.
  • 3. Per Region Style Encoding and Control: SEAN uses a style matrix and segmentation mask to synthesize photo-realistic images with detailed per-region style control.The style matrix and mask are the two inputs to the conditional normalization technique.
  • 3.2. How to Control Style?: Style codes are convolved per region and broadcast according to the mask to form a style map that produces per-pixel γs and βs values.The style map and segmentation mask are processed to learn spatially varying modulation parameters.
  • 3.2. How to Control Style?: The final normalization parameters γ and β are weighted sums of style-derived and mask-derived parameters.The mask-only pathway provides region information similar to SPADE, while the style pathway incorporates per-region style codes.
  • 3.2. How to Control Style?: SEAN modulates normalized generator activations using channel-wise mean and standard deviation computed from the activation map.The supplied definition identifies μc and σc as the mean and standard deviation of channel c.

4. Experimental Setup

The generator builds on SPADE with interleaved SEAN residual blocks and upsampling, while training reconstructs input images from extracted regional styles and masks and inference accepts arbitrary masks with per-region codes.

  • 4.1. Network Architecture: The generator consists of interleaved SEAN ResBlocks and upsampling layers, following the SPADE generator design.Figure 4 presents the style encoder, generator overview, and detailed SEAN ResBlock structure.
  • 4.1. Network Architecture: Each SEAN ResBlk contains three convolutional layers whose scales and biases are modulated by three SEAN blocks.Masks are downsampled to feature-map resolution, and style codes receive layer-specific 1 × 1 transformations.
  • 4.2. Training and Inference: Training reconstructs input images from per-region style codes extracted by the style encoder and their corresponding segmentation masks.The reconstruction difference is measured using losses following SPADE and Pix2PixHD.
  • 4.2. Training and Inference: During inference, an arbitrary segmentation mask can be paired with a separate 512-dimensional style code for each semantic region.This setup supports high-quality image synthesis applications with independent regional style control.
  • 4.2. Training and Inference: SSIM and PSNR increase while RMSE decreases for the proposed method across all datasets in the reconstruction comparison.The table caption specifies that higher SSIM and PSNR are better, whereas lower RMSE is better.

5. Results

SEAN is evaluated across four datasets using quantitative metrics, visual comparisons, and ablations. It outperforms leading methods in generation quality, reconstruction, and style-control evaluations, while supporting interpolation and crossover editing.

  • Quantitative comparisons: Our method outperforms current leading methods in FID on all the datasets.Table 2 also reports semantic segmentation performance using mIoU and pixel accuracy.
  • Quantitative comparisons: SEAN clearly beats SPADE and Pix2PixHD on all four datasets in quantitative comparisons using a single style image.The evaluation covers CelebAMask-HQ, ADE20K, Cityscapes, and Facades.
  • Qualitative results: Visual comparisons show significant quality differences, with SEAN handling extreme poses and older people better than SPADE and Pix2Pix.The authors conjecture that improved style encoding helps capture more variability in the data.
  • Interactive editing: SEAN supports per-region style interpolation and style crossover, with earlier layers controlling larger features and later layers mainly determining color schemes.Crossover selects different styles per ResBlk, while interpolation combines per-region style codes.
  • Ablation studies: All SEAN architecture variations achieve better FID and PSNR scores than SPADE++, while SPADE++ improves SPADE by a bit.The comparison combines Pix2PixHD’s style encoder with SPADE’s generator in SPADE++.
  • Ablation studies: The unified encoder yields lower FID scores but better visual quality than alternative encoders with better reconstruction performance.The unified encoder’s advantage is especially noticeable for difficult inputs.
  • Ablation studies: A bottleneck in the style encoder is confirmed as beneficial, and learnable noise improves both PSNR similarity and FID performance.The ablations evaluate downsampling and learnable noise as design choices.

6. Conclusion

SEAN is a semantic region-adaptive normalization block for segmentation-conditioned GANs. It improves style encoding, transfer, synthesis, and interactive editing relative to previous methods.

  • Conclusion: SEAN controls the style of each semantic region individually by extracting region-specific style information into spatially varying normalization parameters.The method can specify one style reference image per semantic region.
  • Conclusion: SEAN reports better quantitative metrics, including FID and PSNR, than the current state of the art across multiple datasets.The paper also reports improvements when one style image is provided for all regions.
  • Conclusion: SEAN enables interactive region-by-region style transfer, per-region style interpolation, style crossover, and editing through segmentation-mask changes.These operations extend the paper’s style-control capabilities beyond standard synthesis.

A. Additional Implementation Details

The implementation uses SEAN ResBlks to pass style codes and segmentation masks through the generator, with a style encoder producing region-specific codes and adversarial, feature-matching, and perceptual losses guiding training.

  • Generator: The generator consists of SEAN ResBlks followed by nearest-neighbor upsampling, with style codes injected only into the first six blocks.Other inputs are injected into all SEAN ResBlks.
  • Style Encoder: The style encoder combines a bottleneck convolutional network with region-wise average pooling to process style images and masks.Its inputs are the style image and segmentation mask.
  • Discriminator: The discriminator follows SPADE and Pix2PixHD by using the concatenated segmentation mask and style image with a PatchGAN-style loss.The implementation uses two multi-scale discriminators with instance normalization, Leaky ReLU, and spectral normalization.
  • Loss function: Training combines adversarial, feature-matching, and perceptual loss terms, with λ1 = λ2 = 10 in the experiments.The adversarial term uses Hinge loss, while the other terms follow the SPADE and Pix2PixHD-inspired design.
  • Training details: All datasets are trained for 50 epochs after resizing inputs to 256 × 256, except CityScapes images resized to 512 × 256.Glorot initialization is used for network weights.

B. Additional Experimental Details

Additional experiments clarify how encoder variants are assigned within SEAN ResBlks and support the unified encoder because it produces better visual quality on challenging inputs.

  • Encoder variants: SEAN-level encoders extract different style codes for each SEAN block, whereas a ResBlk-level encoder supplies the same codes to all blocks within one SEAN ResBlk.Figure 10 illustrates these two encoder variants.
  • Comparison setup: The Pix2pixHD comparison encodes the style image into one style vector, broadcasts it, and concatenates it with the generator’s mask input.The comparison uses the ground-truth image as the style input for all methods.
  • Encoder choice: Unified-encoder outputs have better visual quality than SEAN-level-encoder outputs, especially for extreme poses and unlabeled regions.This result motivates choosing the unified encoder.

D. Additional Analysis

The additional analysis examines branch contributions and demonstrates SEAN’s flexibility in transferring complex textures and placing semantic regions at spatially unusual locations.

  • ST-branch vs. Mask-branch: ST-branch and mask-branch contributions typically use mixing parameters between 0.35 and 0.7, indicating that both branches actively contribute.Dropping the mask branch worsens results, while the interaction between branches is more complicated than a simple rough-structure/detail split.
  • Encoder choice: The unified encoder is more robust than the SEAN-level encoder for style transfer because the latter is more sensitive to poses and unlabeled style-image regions.The figure attributes this sensitivity to overfitting.
  • Extreme cases: Complex textures from an artistic image can be transferred to a human face.This example is presented as evidence of SEAN’s texture-transfer capability.
  • Extreme cases: Users can paint semantic regions at spatially unreasonable locations, including placing eyes anywhere on a face.This demonstrates spatially flexible editing.

E. User Study

A user preference study compares SEAN with Pix2PixHD and SPADE using reconstruction images judged by Amazon Mechanical Turk workers, with SEAN reported as the preferred method.

  • Study design: The AMT study created 600 questions and received answers from 575 workers.Each question showed a ground-truth image, its segmentation mask, and reconstructions from SEAN, Pix2PixHD, and SPADE.
  • Results: SEAN significantly outperforms Pix2PixHD and SPADE in the CelebAMask-HQ user preference study.The comparison is summarized in Table 4.

F. Additional Results

Additional results show SEAN’s per-region style transfer, interpolation, reconstruction, and interactive editing across multiple datasets, with reconstructions reported as higher quality than Pix2PixHD and SPADE.

  • The interactive UI transfers per-region styles to arbitrary segmentation masks and creates new styles by interpolating existing ones.
  • Additional figures demonstrate style transfer on CelebAMask-HQ and style interpolation on CelebAMask-HQ and ADE20K.
  • Additional reconstruction comparisons cover CelebAMask-HQ, ADE20K, CityScapes, and Façades against Pix2PixHD and SPADE.
  • The authors report that SEAN reconstructions are of much higher quality than those of Pix2PixHD and SPADE.
Loading 1911.12861v2…