PaperlayerView exact revision on arXiv

Source-linked AI summary

Pyramidal Denoising Diffusion Probabilistic Models

Dohoon Ryu, Jong Chul Ye

arXiv:2208.01864v3cs.CVcs.LGstat.ML

TL;DR

Diffusion models achieve strong image-generation performance, but their training and evaluation consume substantial time and computational resources. The paper proposes pyramidal diffusion using one positional-embedding-conditioned score function to generate high-resolution images from coarser ones. It reports faster reverse diffusion and performance on image generation and multi-scale super-resolution.

  • Problem

    Diffusion models deliver strong image-generation performance, but training and evaluation consume substantial time and computational resources, and slow generation remains a critical drawback.

    Source: §1 · p. 1

  • Method

    Pyramidal DDPM conditions one score function on positional information and progressively refines images from coarser to higher resolutions.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    The authors report improved reverse-diffusion speed and performance on super-resolution and image generation, while the conclusion notes that removing positional encoding prevents proper image prediction across resolutions.

    Source: §4.4 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    The approach supports multi-scale super-resolution and very high-resolution image generation with a single score model, including patch-wise training without full-resolution images.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Recently, diffusion model have demonstrated impressive image generation performances, and have been extensively studied in various computer vision tasks. Unfortunately, training and evaluating diffusion models consume a lot of time and computational resources. To address this problem, here we present a novel pyramidal diffusion model that can generate high resolution images starting from much coarser resolution images using a {\em single} score function trained with a positional embedding. This enables a neural network to be much lighter and also enables time-efficient image generation without compromising its performances. Furthermore, we show that the proposed approach can be also efficiently used for multi-scale super-resolution problem using a single score function.

1 INTRODUCTION

Diffusion models generate high-quality images but are slow to sample. This paper proposes a positional-information-conditioned pyramidal DDPM that uses one score function for multi-resolution generation and super-resolution.

1 INTRODUCTION

  • Diffusion models offer strong image-generation performance and have been extended to many computer-vision tasks, but slow generation remains a critical drawback.Prior approaches address speed through lower-dimensional representations, alternative sampling schemes, or larger denoising steps.Source: §1 · pp. 1–2
  • Unlike cascaded approaches that train multiple diffusion models, this method uses one lighter architecture for refinement from low to high resolution.The paper positions this design as improving training and inference speed without compromising generation quality.Source: §1 · p. 2
  • A single position-conditioned score function generates multiple resolutions and supports multi-scale super-resolution.The method replaces separate resolution-specific models with conditional training using positional information.Source: §1 · p. 2
  • Pyramidal DDPM uses coarse-to-fine refinement to reduce reverse-diffusion computation and sampling time while preserving generation quality.The approach starts from coarser images and progressively refines them to higher resolutions.Source: §1 · p. 2

2 BACKGROUND

DDPMs define a forward process that progressively adds Gaussian noise and learn reverse transitions to generate samples. Positional encoding supplies coordinate information that can support image generation across scales and arbitrary image sizes.

2.1 DENOISING DIFFUSION PROBABILISTIC MODELS

  • The forward diffusion process is a Markov chain that gradually adds Gaussian noise to data at each time step.The process uses a variance schedule and converts the data distribution toward an isotropic Gaussian as t approaches T.Source: §2.1 · p. 3
  • Because the reverse of a forward transition is intractable, DDPMs learn parameterized Gaussian reverse transitions.The reverse chain starts from a standard isotropic Gaussian and uses learned means with fixed variance.Source: §2.1 · p. 3

2.2 POSITIONAL EMBEDDING

  • Sinusoidal positional encoding expands coordinate values into a higher-dimensional representation of different frequencies.Periodic encoding provides relative positional information through distances between encoded vectors.Source: §2.2 · pp. 3–4
  • Pyramidal reverse diffusion starts with Gaussian noise at the coarsest resolution, then upsamples and refines the image toward high resolution.At higher resolutions, the algorithm adds noise through forward diffusion and performs scaled reverse diffusion before repeating the process.Source: §2.2 · p. 4
  • Coordinate-conditioned image generation can use image patches and their positions to produce full images or images at arbitrary sizes.Prior work cited in the passage trains models on patches and coordinates, including methods that generate arbitrary-size images.Source: §2.2 · p. 4

3 PYRAMIDAL DENOISING DIFFUSION PROBABILISTIC MODELS

PDDPM trains one position-conditioned score network across scales, then samples from coarse to fine to reduce reverse-diffusion work. The pyramid is also adapted to super-resolution with constraints tied to the low-resolution measurement.

3.1 MULTI-SCALE SCORE FUNCTION TRAINING

  • Random resizing to 64/128/256 resolutions and sinusoidally encoded normalized pixel coordinates condition a score function trained across scales, with a cost function invariant to resolution.The invariant cost function allows optimization with a single network and can alleviate slow-training and low-batch-size problems when resources are limited.Source: §3.1 · pp. 4–5

3.2 FAST REVERSE DIFFUSION THROUGH PYRAMIDAL REVERSE SAMPLING

  • Full reverse diffusion runs at the coarsest scale; each upscaled reconstruction is noised and reverse-diffused for fewer steps to produce the next finer image.The procedure uses separate full and scaled time steps; the authors attribute the reduced reverse-step requirement to better initialization at each finer scale.Source: §3.2 · p. 5

3.3 PYRAMIDAL SUPER RESOLUTION WITH STABLE GRADIENT GUIDANCE

  • For super-resolution, gradient-guided reverse updates enforce constraints at every step, and the upscaled low-resolution input can initialize the coarsest level without full reverse sampling.The method uses a single model and a down-sampling operator to relate the reconstruction to the low-resolution measurement.Source: §3.3 · pp. 5–6, Figure 3 · p. 6

4 EXPERIMENTS

The experiments report faster image generation with smaller models, evaluate multi-scale super-resolution, and test the effects of resolution steps and positional encoding. They also demonstrate 1024×1024 generation from patch training without full-resolution examples.

4.1 DATASETS AND IMPLEMENTATION

  • The experiments used an improved DDPM with 64 channels and one residual block, trained on one GeForce 1080 Ti in a limited-resource setting.Training used 1000 diffusion steps; inference’s base setting used Tf = 1000 and Ts = 100.Source: §4.1 · p. 6

4.2 IMAGE GENERATION

  • The method outperformed the image-generation baselines despite faster sampling and a much smaller architecture, based on 10k generated images evaluated with FID.The authors also report that their LSUN-Church training produced qualitative results while baseline training was incomplete.Source: Table 1 · p. 7, §4.2 · p. 8
  • With Tf = 100 and △ts = 0.1, sampling was 82.4 times faster than the full diffusion process while generating convincing images.Larger high-resolution sampling steps increased FID, while sampling speed decreased proportionally to △ts; results were comparable to the Table 1 baselines.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 SUPER RESOLUTION

  • The proposed method achieved the best FID and LPIPS in most super-resolution cases, but not the best PSNR or SSIM.SRGAN had the best PSNR and SSIM, while the authors describe the proposed method’s results as more realistic despite SRGAN’s image artifacts.Source: Figure 4 · p. 7, §4.3 · p. 8, Table 3 · p. 8

4.4 ABLATION STUDY

  • Using every resolution step in the 32→64→128→256 pyramid achieved the best super-resolution result among the tested upsampling procedures.The tested alternatives omitted one or more intermediate resolutions; applying CCDF acceleration at each resolution also improved reconstructed-image quality.Source: §4.4 · p. 9, Table 4 · p. 9
  • Without positional encoding, the model generated faces with odd appearances, crushed faces, and dislocated features, supporting the encoding’s importance.This ablation used the same experimental setting except for positional encoding and ran for 800k iterations.Source: §4.4 · p. 9, Figure 6 · p. 9
  • Patch-trained models generated 1024×1024 images despite training only on 256×256 and 512×512 image patches.The model had not seen maximum-resolution inputs during training.Source: §4.4 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion presents Pyramidal DDPM, conditioned on positional information, as improving reverse-diffusion speed and image-generation and super-resolution performance. It also reports that positional encoding supports proper generation across resolutions, while patch-wise training can generate very high-resolution images without full-resolution training images.

5 CONCLUSION

  • Pyramidal DDPM conditions training on positional information.Source: §5 · p. 10
  • The authors report improved reverse-diffusion speed and performance on super-resolution and image generation.Source: §5 · p. 10
  • Without positional encoding, the model lost the ability to predict proper images at different resolutions.Patch-wise training also generated very high-resolution images without using full-resolution images.Source: §5 · p. 10

A.1 ADDITIONAL SAMPLES

The appendix provides additional samples at different resolutions generated by a single score model on FFHQ and LSUN-Church.

A.1 ADDITIONAL SAMPLES

  • Additional samples show different-resolution images generated by one score model on FFHQ and LSUN-Church.Source: §A.1 · p. 13

A.2 SUPER-RESOLUTION: INTERMEDIATE RESULTS

The appendix presents intermediate pyramidal super-resolution results on FFHQ and AFHQ-dog, starting from 32×32 images.

A.2 SUPER-RESOLUTION: INTERMEDIATE RESULTS

  • Intermediate results show images upscaled from 32×32 across resolutions on FFHQ and AFHQ-dog.Source: §A.2 · p. 13

A.3 SUPER-RESOLUTION: SELF-COMPARISON RESULTS

The ×8 super-resolution comparisons indicate that a direct path from low to maximum resolution produces more image details, suggesting iterative refinement has a nontrivial effect.

A.3 SUPER-RESOLUTION: SELF-COMPARISON RESULTS

  • Direct paths from low to maximum resolution produce more image details in both ×8 comparisons, indicating iterative refinement has a nontrivial effect.The comparison is visualized for FFHQ and AFHQ-dog.Source: §A.3 · p. 13

B.1 COMPARISON METHODS

This section describes the implementation and comparison settings for image generation and super-resolution methods. The accompanying figures show generated images at increasing resolutions and super-resolution results.

B.1.1 IMAGE GENERATION

  • Pyramidal DDPM used 64 convolution channels and one residual block per connection to generate 64 × 64 images, and was trained on a GeForce 1080 Ti.The implementation was based on improved diffusion model code.Source: §B.1.1 · p. 13
  • FON, DDIM, S-PNDM, and F-PNDM used the PNDM score function baseline, trained for 1.2M iterations on LSUN-Church and FFHQ.The setup used 128 channels, two residual blocks, attention resolution 16, Adam at 0.0002, and a Quadro RTX 6000.Source: §B.1.1 · p. 13

B.1.2 SUPER RESOLUTION

  • SRGAN was trained from scratch on FFHQ and AFHQ-dog for 500k iterations per dataset, using batch size 8 and a GeForce 1080 Ti.Training used Adam with learning rate 0.001, β1 0.9, and β2 0.999.Source: §B.1.2 · p. 13
  • SR3 used a single upsampler for 32 →256 and 64 →256, trained for 500k iterations per dataset with batch size 2.The model had 64 inner channels and two residual blocks, and used Adam at learning rate 0.0001.Source: §B.1.2 · p. 13
  • ILVR used pretrained networks trained for 1M iterations on FFHQ and 500k on AFHQ-dog, with the DDIM/DDPM sampling model architecture and settings.Source: §B.1.2 · p. 14

B.1.1 IMAGE GENERATION

B.1.2 SUPER RESOLUTION

Related papers

Loading 2208.01864v3…