PaperlayerView exact revision on arXiv

Source-linked AI summary

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, Sungroh Yoon

arXiv:2108.02938v2cs.CV

TL;DR

DDPMs generate high-quality unconditional images but are difficult to control because stochastic transitions can yield inconsistent semantics. ILVR guides those transitions by matching latent variables to a reference image, enabling controllable high-quality generation and adapting one unconditional model to multiple tasks without additional learning.

  • Problem

    DDPM stochasticity makes it challenging to generate images with desired semantics consistently.

  • Method

    ILVR is a learning-free method that refines each unconditional DDPM transition by matching latent variables with a given reference image.

  • Results

    ILVR enables one unconditional DDPM to support controllable generation across downsampling factors, multi-domain translation, paint-to-image, and scribble editing without additional learning or models.

  • Takeaways & Limitations

    Downsampling factors and conditioning range provide user control over semantic similarity and diversity while preserving the quality of unconditional DDPM samples.

Abstract

from arXiv · show

Denoising diffusion probabilistic models (DDPM) have shown remarkable performance in unconditional image generation. However, due to the stochasticity of the generative process in DDPM, it is challenging to generate images with the desired semantics. In this work, we propose Iterative Latent Variable Refinement (ILVR), a method to guide the generative process in DDPM to generate high-quality images based on a given reference image. Here, the refinement of the generative process in DDPM enables a single DDPM to sample images from various sets directed by the reference image. The proposed ILVR method generates high-quality images while controlling the generation. The controllability of our method allows adaptation of a single DDPM without any additional learning in various image generation tasks, such as generation from various downsampling factors, multi-domain image translation, paint-to-image, and editing with scribbles.

1. Introduction

The paper addresses the difficulty of controlling stochastic DDPM generation by introducing ILVR, a learning-free method that refines each transition using a reference image. ILVR enables one unconditional DDPM to support controllable generation across several image-generation tasks without additional learning or models.

  • DDPM stochastic transitions can produce inconsistent high-level semantics, making desired image generation difficult even from the same initial state.
  • ILVR conditions an unconditional DDPM by refining each generative transition through latent-variable matching with a given reference image.
  • Generation from various downsampling factors provides diversity control while maintaining visual quality, evaluated using FID and LPIPS.
  • ILVR provides user control over semantic similarity between generated images and the reference.
  • A single unconditional DDPM can support multi-domain image translation, paint-to-image, and scribble-based editing without additional learning or models.

2. Background

DDPMs learn a reverse Markov process that transforms noise into data through sequential Gaussian transitions. Because each reverse-generation step is stochastic, controlling the resulting image semantics is difficult.

  • DDPMs learn a reverse Markov chain that converts a simple distribution, such as isotropic Gaussian noise, into the data distribution.
  • The forward transition uses a fixed variance schedule β1, ..., βT rather than learned parameters.
  • The forward process gradually adds Gaussian noise to latent variables, while the learned reverse process removes it through parameterized Gaussian transitions.
  • A noisy latent xt can be expressed as a linear combination of clean data x0 and Gaussian noise ϵ.
  • The reverse sampling step combines the current latent, a neural noise prediction, and stochastic Gaussian noise z.
  • ILVR's graphical model first samples an unconditional proposal xt−1 and then matches it with an encoded condition yt−1.

3. Method

ILVR conditions an unconditional DDPM without additional learning by refining each stochastic transition with a reference image’s low-frequency contents. Downsampling and conditioning-range controls direct sampling toward reference-defined subsets while preserving the unconditional model’s sample-quality scope.

  • 3.1. Iterative Latent Variable Refinement: ILVR conditions an unconditional DDPM by refining each transition with a downsampled reference image, without additional learning or models.The method first obtains an unconditional proposal, then modifies it to impose the reference-based condition.
  • 3.1. Iterative Latent Variable Refinement: The condition requires the generated image’s low-pass representation ϕN(x0) to equal the reference representation ϕN(y).ϕN is a dimensionality-preserving sequence of downsampling and upsampling by factor N.
  • 3.1. Iterative Latent Variable Refinement: At each step, ILVR samples an unconditional proposal, encodes a corrupted reference, and replaces the proposal’s low-frequency component with the reference component.This latent-variable matching enforces the local condition during the reverse process.
  • 3.1. Iterative Latent Variable Refinement: The resulting samples are diverse, high-quality images that share semantics with the reference while remaining within the unconditional DDPM’s image set.The method samples from a reference-directed subset of the unconditional model’s generated image set.
  • 3.2. Reference selection and user controllability: The directed subset RN(y) contains images x whose low-pass representations match the reference, and reference images may come from unseen data domains.This enables reference-directed sampling and multi-domain image translation using a single unconditional DDPM.
  • 3.2. Reference selection and user controllability: Higher downsampling factors produce broader, more diverse subsets with lower semantic similarity, whereas lower factors preserve finer reference features.Because RN is a subset of μ, the sampling method maintains the unconditional DDPM’s sample quality.
  • 3.2. Reference selection and user controllability: Limiting conditioning to selected reverse-process steps broadens the sampled subset while retaining the learned image distribution, with finer-grained effects on diversity.Changing conditioning range affects sample diversity more finely than changing downsampling factors.

4. Experiments and Applications

Experiments show that ILVR controls reference similarity and diversity through downsampling factors and conditioning ranges while preserving unconditional DDPM image quality. A single unconditional model supports translation from unseen domains, paint-to-image, and scribble editing without additional learning.

  • User controllability: Downsampling factor N controls semantic similarity: higher N preserves coarse features and increases diversity, whereas lower N preserves finer reference details.N=64 retains features such as color scheme; N=8 retains fine details such as hair curls, eye color, and earrings.
  • User controllability: Narrower conditioning ranges provide finer control over diversity, with samples beginning to deviate from references when conditioning is limited below steps 1000–500.The conditioning range affects diversity more finely than changing the downsampling factor.
  • Multi-domain image translation: ILVR supports diverse image translation from source domains unseen during training using a single model trained on the target domain.Experiments include portrait-to-face, face-to-portrait, and dog generation from dog, cat, and wildlife references.
  • Paint-to-image: ILVR translates oil paintings, watercolors, and clip art into photo-realistic images by preserving coarse reference information with N=64.The method uses coarse shared features such as color scheme while changing detailed texture.
  • Editing with scribbles: ILVR edits images from scribble references, generating objects or markings such as clouds, watermarks, article numbers, and earrings.For scribble editing, the experiments use factor N=8 over steps 1000 to 200 to retain original details while harmonizing scribbles.
  • Quantitative evaluation: FID scores remain mostly comparable to unconditional baselines, while higher downsampling factors produce greater LPIPS diversity.Lower factors align more closely with references and therefore yield lower diversity.

5. Related Work

Related work includes conditional generative models and methods that manipulate pretrained unconditional generators. ILVR instead adapts one unconditional DDPM across applications without additional models or retraining.

  • Conditional generative models: Conditional generative models use inputs such as class labels, segmentation masks, classifier features, or images for task-specific generation.These approaches include image-to-image translation, style transfer, inpainting, and super-resolution.
  • ILVR’s distinction: Unlike models trained for desired settings or learned source domains, ILVR adapts a single unconditional model to various applications and unseen source domains.The paper positions this as a contrast with conditional models that operate only in their trained settings.
  • Pretrained unconditional generators: Pretrained unconditional generators have been leveraged for editing, style transfer, and super-resolution by projecting images into latent vectors and manipulating them.Prior work largely uses GANs as the underlying unconditional generators.
  • Condition representations: Semantic conditions such as CNN features, segmentation masks, and low-resolution images can guide generation, but feature- and segmentation-based conditions require additional models.ILVR uses low-resolution images to control DDPM without additional models.

6. Conclusion

The conclusion presents ILVR as a learning-free way to condition unconditional DDPM transitions and control generated-image similarity through sampling choices. It reports adaptation of one unconditional model across multiple applications without additional learning or models.

  • Conclusion: ILVR conditions unconditional DDPM generation by refining each transition with a given reference image.The method enables sampling from a space of plausible images.
  • Conclusion: Downsampling factors and conditioning ranges provide user control over the generated images.These controls regulate semantic similarity and diversity relative to the reference.
  • Conclusion: A single unconditional DDPM can support multiple applications without additional learning or models.The conclusion summarizes the method’s use across the demonstrated generation tasks.

A. Derivation of approximation

The derivation approximates each conditioned DDPM transition by refining an unconditional proposal to match reference-image features. This yields the ILVR transition used in the main method.

  • A. Derivation of approximation: ILVR replaces each transition’s condition with a local condition based on an approximation.The approximation is introduced as the basis for matching each generative transition with a reference image.
  • A. Derivation of approximation: ϵθ(xt, t) predicts the noise added at timestep t, enabling a closed-form approximation of the denoised data.The approximation is fθ(xt, t)=(xt−√(1−ᾱt)ϵθ(xt,t))/√ᾱt.
  • A. Derivation of approximation: The conditioned transition is approximated by matching transformed denoised predictions of the generated and reference images.The derivation uses ϕ(f(xt−1))=ϕ(f(yt−1)) as the local matching condition.
  • A. Derivation of approximation: ILVR first computes an unconditional proposal x′t−1, then refines it so that ϕ(xt−1)=ϕ(yt−1).This refinement produces the conditional transition while retaining the unconditional model proposal.
  • A. Derivation of approximation: NIQE evaluates generation quality for reference images downscaled by factors of 16 and 64, with lower scores indicating better quality.The accompanying table reports ILVR as having the highest perceptual quality.
  • A. Derivation of approximation: FID evaluates image translation from the AFHQ-dog test set, where ILVR is described as comparable to a state-of-the-art model.The passage identifies FID as the comparison metric for translation quality.

B. Additional evaluations

Additional evaluations examine ILVR’s generation quality and translation performance against established baselines. The results report high perceptual quality and competitive image-translation performance.

  • B.1. Generation quality: ILVR generates high-quality images from reference images downsampled by factors of 16 and 64.Qualitative comparisons include bicubic interpolation and PULSE, which uses a pre-trained StyleGAN model.
  • B.1. Generation quality: PULSE optimizes a StyleGAN2 latent vector so its generated image matches the reference image in low-resolution space.Its objective combines low-resolution MSE, geodesic cross loss, and noise regularization.
  • B.2. Image translation: ILVR has FID comparable to CUT for cat-to-dog translation on the AFHQ test set.ILVR uses a model trained only on dog images, whereas the compared models were trained on both cat and dog images.

B.3. Additional samples

Additional samples evaluate ILVR with guided-diffusion models and examine robustness to resizing-kernel choices. The same-noise comparison produces nearly identical images across kernels.

  • B.3. Additional samples: Additional editing with scribbles is demonstrated using guided-diffusion models trained on LSUN datasets.The passage identifies Fig. 9 for samples and Fig. C for scribble editing.
  • B.3. Additional samples: Using the same noises through the generative process, images are almost identical across resizing kernels.The comparison changes only the resizing kernels.
  • B.3. Additional samples: The resizing-kernel comparison suggests that ILVR is robust to kernel choice.This conclusion follows from the near-identical generated images under the same noise inputs.

C.2. Datasets and training

The experiments use 256×256 training across face, animal, scene, and church datasets, with task-specific training sets and reference images. Training details vary from large-scale datasets to fine-tuning and single-category models.

  • C.2. Datasets and training: All datasets use 256×256 training resolution and batch size 8.These settings are stated as common training details across the datasets.
  • C.2. Datasets and training: FFHQ contains 70,000 high-resolution face images, and its model was trained for 1.2M steps.FFHQ provides the face-image training data.
  • C.2. Datasets and training: The METFACES model was fine-tuned from FFHQ for 20k steps to reduce overfitting on 1,000 portrait images.The dataset consists of high-resolution portrait images.
  • C.2. Datasets and training: For AFHQ, the model was trained on dog images and tested with reference images from dog, cat, and wild categories.AFHQ contains 15,000 high-resolution animal-face images equally split across the three categories.
  • C.2. Datasets and training: A waterfall-category model from Places365 was used for the paint-to-image task.The waterfall category contains 5,000 images.
  • C.2. Datasets and training: The LSUN Church model was trained for 1M steps on 126,227 church images.The training data consists of church images from LSUN Church.

C.4. Evaluation

The evaluation reports FID computation details and examines additional scribble editing and low-pass-filter ablations. The filter comparison finds only minor visual differences across interpolation methods.

  • FID was calculated using 50k real and 50k generated images with PyTorch code3.
  • Additional scribble-editing examples include faces from FFHQ and bedrooms from LSUN Bedroom models.
  • Low-pass-filter ablations compare bicubic, lanczos2, lanczos3, and bilinear interpolation at downsampling factors N=4 and N=32.
  • The interpolation filters produce only minor differences, such as the exact position of teeth and hair.
Loading 2108.02938v2…