PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Image Synthesis via Diffusion Models

Wengang Zhou, Weilun Wang, Jianmin Bao, Dongdong Chen, Dong Chen, Lu Yuan, Houqiang Li

arXiv:2207.00050v4cs.CV

TL;DR

Semantic image synthesis needs realistic and diverse images from discrete semantic layouts, while existing GAN-based approaches can be limited on complex scenes. The paper introduces Semantic Diffusion Model, separating noisy-image and layout processing in a DDPM and adding classifier-free guidance. Experiments on four benchmark datasets achieve state-of-the-art performance in FID and LPIPS.

  • Problem

    Existing GAN-based semantic image synthesis approaches can produce limited fidelity and diversity on complex scenes, while diffusion image-to-image methods are not suited to discrete label-map inputs.

  • Method

    Semantic Diffusion Model processes noisy images in a U-Net encoder, injects semantic layouts into its decoder through spatially-adaptive normalization, and uses classifier-free guidance during sampling.

  • Results

    Experiments on four benchmark datasets achieve state-of-the-art performance in generation fidelity measured by FID and diversity measured by LPIPS.

  • Takeaways & Limitations

    The framework generates high-fidelity, diverse, and semantically correlated images from semantic layouts.

  • Takeaways & Limitations

    The pixel-level diffusion model is trained from scratch on each dataset, giving good dataset performance but limited generality to natural images.

Abstract

from arXiv · show

Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks compared with Generative Adversarial Nets (GANs). Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches, which may lead to unsatisfactory quality or diversity of generated images. In this paper, we propose a novel framework based on DDPM for semantic image synthesis. Unlike previous conditional diffusion model directly feeds the semantic layout and noisy image as input to a U-Net structure, which may not fully leverage the information in the input semantic mask, our framework processes semantic layout and noisy image differently. It feeds noisy image to the encoder of the U-Net structure while the semantic layout to the decoder by multi-layer spatially-adaptive normalization operators. To further improve the generation quality and semantic interpretability in semantic image synthesis, we introduce the classifier-free guidance sampling strategy, which acknowledge the scores of an unconditional model for sampling process. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our proposed method, achieving state-of-the-art performance in terms of fidelity (FID) and diversity (LPIPS). Our code and pretrained models are available at https://github.com/WeilunWang/semantic-diffusion-model.

I. INTRODUCTION

Semantic image synthesis converts semantic layouts into photo-realistic images, but GAN-based methods can struggle with fidelity and diversity in complex scenes. The paper proposes Semantic Diffusion Model, which separates noisy-image and layout processing and uses classifier-free guidance, achieving state-of-the-art FID and LPIPS results on four benchmarks.

  • Motivation: Semantic image synthesis generates photo-realistic images from semantic layouts, serving as the reverse problem of semantic segmentation.Applications include image editing, interactive painting, and content generation.
  • Motivation: GAN-based approaches have limitations on complex scenes, particularly for generating high-fidelity and diverse results.
  • Background: DDPMs generate samples from Gaussian noise through iterative denoising and have achieved state-of-the-art image-generation quality.
  • Proposed framework: Semantic Diffusion Model processes noisy images through the U-Net encoder while injecting semantic layouts into the decoder with multi-layer spatially-adaptive normalization.
  • Sampling: Classifier-free guidance combines conditional and unconditional diffusion predictions to improve fidelity and correspondence with semantic label maps.
  • Results: Experiments on Cityscapes, ADE20K, CelebAMask-HQ, and COCO-Stuff report superior generation fidelity and diversity, with state-of-the-art FID and LPIPS performance.

II. RELATED WORK

Diffusion-based image-to-image methods have expanded beyond low-level tasks, but semantic image synthesis remains challenging because its input is a discrete label map and GAN methods struggle to balance fidelity and diversity.

  • Conditional DDPMs: Conditional DDPM research has addressed super-resolution, colorization, inpainting, uncropping, JPEG decompression, linear inverse problems, and text-to-image generation.These methods primarily focus on low-level vision tasks or single-dimensional conditions rather than dense semantic conditions.
  • Diffusion image-to-image translation: Earlier diffusion image-to-image methods target RGB inputs such as stroke paintings or natural images, making them unsuitable for discrete semantic label maps.The paper therefore proposes SDM for semantic image synthesis.
  • Semantic image synthesis: Semantic image synthesis transforms semantic layouts into diverse realistic images and is framed as image-to-image translation.It supports applications including image editing, interactive painting, and content generation.
  • GAN-based methods: GAN-based semantic synthesis methods include multi-scale generation, spatially-adaptive normalization, class-adaptive normalization, dynamic weighting, and multimodal latent or instance-level sampling.Representative methods include Pix2PixHD, SPADE, CLADE, SCGAN, BicycleGAN, DSCGAN, and INADE.
  • GAN-based methods: Existing multimodal GAN methods still fail to achieve satisfactory generation quality and learned correspondence while maintaining diversity.The paper identifies simultaneous high fidelity and diversity as non-trivial for these approaches.

III. METHODOLOGY

The proposed Semantic Diffusion Model applies conditional DDPMs to semantic image synthesis, using iterative denoising from Gaussian noise and a U-Net-based architecture with specialized encoder and decoder resblocks.

  • Semantic Diffusion Model: SDM transforms semantic layouts into realistic images through an iterative denoising process modeled as a parameterized Markov chain.At each step, a denoising network estimates noise from the noisy image conditioned on the semantic label map.
  • Conditional DDPM preliminaries: Conditional DDPMs define forward and reverse processes, with the reverse process learning Gaussian transitions from standard Gaussian noise.The forward process progressively adds Gaussian noise according to a variance schedule.
  • Conditional DDPM preliminaries: The conditional DDPM objective is derived from an upper variational bound on negative log likelihood and becomes a denoising objective under a fixed variance assumption.The loss includes a timestep-specific component Lt−1 and a constant γt associated with timestep t.
  • Network architecture: The network architecture is organized around semantic diffusion encoder resblocks, semantic diffusion decoder resblocks, and attention blocks.The detailed resblock structures are presented as complementary components of the denoising network.

B. Semantic Diffusion Model.

SDM separates noisy-image processing from semantic conditioning in a U-Net: the encoder processes the noisy image, while the decoder receives the semantic layout through spatially-adaptive normalization.

  • Training: During finetuning, the semantic mask is randomly removed in some training examples so the model learns predictions with and without conditioning.The procedure uses noisy inputs formed by combining data with Gaussian noise and optimizes denoising losses.
  • Sampling: Classifier-free guidance combines conditional and unconditional noise predictions during the T-step sampling loop before returning the denoised image.The procedure starts from Gaussian noise and updates the sample iteratively using the guided prediction.
  • Decoder: This separation addresses the limitation that direct condition concatenation and U-Net normalization can fail to preserve semantic information.The noisy image enters the encoder, and semantic information is injected throughout the decoder.
  • Decoder: The decoder injects the semantic label map through multi-layer SDDResblocks using SPADE instead of group normalization.SPADE applies a spatially-adaptive, learnable transformation to regulate denoising features.

C. Classifier-free guidance.

The paper uses classifier-free guidance to improve the fidelity and semantic correspondence of conditional diffusion samples while balancing quality and diversity through a guidance scale.

  • Conditional diffusion samples can be diverse but insufficiently photorealistic and weakly correlated with semantic label maps.
  • The guidance scale s trades off sample quality and diversity.
  • Classifier-free guidance replaces an extra trained classifier by comparing predictions conditioned on the semantic mask with unconditional predictions using a null label.The null label is defined as ∅, and the method obtains guidance from the generative model itself.
  • The disentangled conditional–unconditional component is increased during sampling to improve samples from conditional diffusion models.

A. Experimental Setup

Experiments use four benchmark datasets and report quantitative comparisons, user preferences, and qualitative Cityscapes results. Semantic interpretability is evaluated with off-the-shelf segmentation networks, whose capability affects mIoU.

  • Datasets: Experiments cover Cityscapes, ADE20K, CelebAMask-HQ, and COCO-Stuff.Cityscapes uses 35 semantic classes, CelebAMask-HQ uses 19, and both additionally use instance edge maps; ADE20K uses 151 classes including an unknown object.
  • Implementation: The framework uses dataset-specific guidance scales of 1.5, 2.0, 1.5, and 1.5 for CelebAMask-HQ, Cityscapes, ADE20K, and COCO-Stuff, respectively.Training uses AdamW and an exponential moving average.
  • Metrics and comparisons: Table I compares semantic image synthesis methods using FID and LPIPS, with arrows indicating whether higher or lower values are better.
  • User study: Table II reports the percentage of user preferences favoring the proposed approach in paired comparisons across four benchmark datasets.The study compares the method with SPADE, INADE, and OASIS.
  • Qualitative evaluation: Cityscapes qualitative results use zoomed-in comparisons and emphasize fine-grained objects such as distant cars and traffic lights.The compared methods include Pix2PixHD, SPADE, CC-FPSE, INADE, OASIS, and ControlNet.
  • Semantic interpretability: mIoU is computed from generated images and semantic layouts using off-the-shelf segmentation networks as an evaluation of semantic interpretability.The reported mIoU depends strongly on the capability of the selected segmentation network.

B. Comparison with previous methods

Across four datasets, the proposed method is compared with GAN-based and diffusion-based approaches using quantitative, qualitative, diversity, semantic-correspondence, and user-study evaluations.

  • Quantitative comparison: +2.2, +0.8, +2.0, +1.1 FID score improvements are reported on the four datasets compared with the most recent methods.
  • Qualitative comparison: Qualitative comparisons report more reasonable structures, content, and fine-grained details than previous methods, including fences, side-view faces, motorcycles, distant cars, and traffic lights.
  • Diffusion-based comparison: Against ControlNet, SDM achieves better FID but worse LPIPS, while qualitative results report more realistic images.ControlNet inherits generation capability from Stable Diffusion trained on LAION.
  • User study: The user study reports that the proposed method is clearly preferred over SPADE, INADE, and OASIS across four benchmark datasets.Twenty volunteers evaluated paired generated results.
  • Diversity: +0.035, +0.065, +0.007, +0.190 LPIPS improvements are reported over the most diverse methods on the four datasets.Multimodal examples are presented as diverse results with high quality.
  • Semantic correspondence: SDM reaches 77.0 and 77.5 mIoU on CelebAMask-HQ and Cityscapes, surpassing previous state of the art by +0.4 and +12.3.On ADE20K and COCO-Stuff, the method has weaker mIoU than some existing methods.

C. Ablation Studies

Ablations examine condition embedding, embedding position, and classifier-free guidance. Results favor independent multi-layer semantic conditioning in the decoder and show guidance improves fidelity and semantic structure with a small diversity cost.

  • Diffusion Procedure: Training and diffusion visualizations show noise prediction during training and iterative refinement toward realistic images during diffusion.
  • Condition Embedding: Independent semantic conditioning highly outperforms direct concatenation with the noisy image on all reported metrics.The proposed multi-layer spatially adaptive embedding also improves visual fidelity and correspondence with the semantic label map.
  • Embedding Position: Decoder-only condition embedding achieves superior performance compared with embedding the condition at the encoder or at both encoder and decoder.
  • Classifier-free Guidance: Classifier-free guidance highly improves mIoU and FID at the expense of little LPIPS.Qualitative results show better semantic information and more structured content than sampling without guidance.
  • Generalization: SDM-LoRA adapts the design to finetune a pretrained Stable Diffusion text-to-image model by adding LoRA and spatially adaptive semantic-layout injection.

E. Visualization of SDM

SDM visualizations show iterative denoising at different noise levels, faster sampling with DPM-Solver, and semantic editing that preserves both edited labels and surrounding context.

  • At small timesteps, SDM recovers image details, while at large timesteps it restores coarse-grained shapes and outlines.
  • 100-step DPM-Solver sampling produces realistic SDM images on ADE20K and COCO-Stuff.
  • 17.5s/image on a single V100 GPU is achieved with DPM-Solver, compared with standard DDPM sampling at T = 1000.
  • Semantic editing modifies label maps to add or remove objects, then inpaints the affected region while retaining harmony with the original image.
  • Edited semantic maps yield realistic completions matching both the semantic labels and surrounding context.

V. LIMITATION AND FUTURE WORK

The paper identifies limited generalization from pixel-level models trained from scratch and proposes leveraging large-scale pretrained models and improved spatial injection methods in future work.

  • Pixel-level SDM models trained from scratch perform well on their datasets but are not general on natural images.
  • Large-scale pretrained text-to-image models are proposed as a future direction because of their inherent generation capability and semantic understanding.
  • Future SDM research may combine improved SPADE variants, including INADE, CLADE, and CC-FPSE, to improve performance.
Loading 2207.00050v4…