PaperlayerView exact revision on arXiv

Source-linked AI summary

SinDDM: A Single Image Denoising Diffusion Model

Vladimir Kulikov, Shahar Yadin, Matan Kleiner, Tomer Michaeli

arXiv:2211.16582v3cs.CVcs.LGeess.IV

TL;DR

Existing diffusion models generally require large datasets, motivating a single-image alternative. SinDDM learns multi-scale internal statistics with a lightweight fully convolutional denoiser and generates arbitrary-size samples coarse-to-fine. It produces diverse, high-quality samples and supports text-guided control of content and style through CLIP.

  • Problem

    Existing denoising diffusion models rely on large training datasets, while relevant examples can be scarce.

    Source: §1 · p. 1

  • Method

    SinDDM trains a multi-scale diffusion model on one image using a fully convolutional denoiser conditioned on noise level and scale.

    Source: §1 · p. 1

  • Results

    SinDDM generates diverse, high-quality arbitrary-size samples and supports text-guided control of image content and style.

    Source: §1 · pp. 1–2, §4 · p. 6, SinGAN · p. 8

  • Takeaways & Limitations

    Single-image diffusion can support synthesis, manipulation, and externally guided generation from the internal statistics of one training image.

    Source: §1 · pp. 1–2, §5 · p. 9

    SinDDM is often less confined to the training image’s internal statistics, which can over- or under-represent objects in unconditional generation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Denoising diffusion models (DDMs) have led to staggering performance leaps in image generation, editing and restoration. However, existing DDMs use very large datasets for training. Here, we introduce a framework for training a DDM on a single image. Our method, which we coin SinDDM, learns the internal statistics of the training image by using a multi-scale diffusion process. To drive the reverse diffusion process, we use a fully-convolutional light-weight denoiser, which is conditioned on both the noise level and the scale. This architecture allows generating samples of arbitrary dimensions, in a coarse-to-fine manner. As we illustrate, SinDDM generates diverse high-quality samples, and is applicable in a wide array of tasks, including style transfer and harmonization. Furthermore, it can be easily guided by external supervision. Particularly, we demonstrate text-guided generation from a single image using a pre-trained CLIP model.

1. Introduction

SinDDM addresses scarce training data by learning a hierarchical diffusion model from one image, generating diverse arbitrary-size variants and supporting externally guided manipulation.

1. Introduction

  • Unlike the field’s increasingly data-intensive generative models, SinDDM targets settings where relevant training examples are scarce.Existing image-generation and restoration systems have achieved strong capabilities while often relying on large amounts of training data.Source: §1 · p. 1
  • SinDDM combines SinGAN’s multi-scale structure with denoising diffusion models to learn a generative model from a single image.Its fully convolutional denoiser is trained across image scales and noise levels, then used coarse-to-fine for generation.Source: §1 · p. 1
  • A small receptive field captures fine-detail statistics at each scale, enabling diverse samples of arbitrary dimensions.The generated images can contain realistic configurations absent from the training image, such as new skyline structures or mountain angles.Source: §1 · p. 1, Figure 1 · p. 2
  • SinDDM supports image-manipulation tasks and can be guided by external supervision, particularly through pretrained CLIP text guidance.The paper demonstrates control over generated content and style, including text-guided generation from a single image.Source: §1 · pp. 1–2

2. Related Work

Related work spans single-image generative models, diffusion-based synthesis, and text-guided manipulation. SinDDM follows hierarchical single-image modeling while combining diffusion with text guidance to broaden controllability.

Single image generative models

  • Single-image models learn internal patch or deep-feature distributions, with SinGAN establishing hierarchical unconditional synthesis from one image.Later work replaces SinGAN’s GAN framework with energy-based models, nearest-neighbor search, or test-time feature-statistics optimization.Source: §2 · p. 2
  • SinDDM retains SinGAN’s hierarchical approach but uses denoising diffusion probabilistic models to support high-quality and guided image generation.The paper positions this combination as distinct from prior single-image methods and related concurrent diffusion approaches.Source: §2 · p. 2

Diffusion models

  • Diffusion models generate samples by reversing a gradual noising process and have achieved strong results in image generation, restoration, translation, and editing.The cited literature presents diffusion models as state-of-the-art for image generation and manipulation.Source: §2 · p. 2

Text-guided image manipulation and generation

  • Text-guided methods use language to control image generation or editing, often employing pretrained CLIP models to connect text and image representations.SinDDM combines CLIP’s external knowledge with its internal single-image model for text-guided generation.Source: §2 · p. 2, Figure 2 · p. 3
  • Compared with Text2Live, SinDDM can change entire scenes and generate dimensions differing from the original image, rather than only adding details on top.Text2Live edits an image or video through a separately generated edit layer combined with the original.Source: §2 · p. 3

3. Method

SinDDM trains a single fully convolutional denoiser on multi-scale, progressively noisier and blurrier versions of one image, then samples coarse to fine. The framework also supports image-region and text guidance during reverse diffusion.

3. Method

  • SinDDM learns a single image’s internal multi-scale statistics with a hierarchical diffusion process combining blur and noise.The forward process constructs image and blurry pyramids, while the denoiser is trained across scales and noise levels.Source: §3 · p. 3, §3.1 · p. 3

3.2. Reverse Multi-Scale Diffusion

  • The reverse process starts from noise at the coarsest scale, removes noise, upsamples the clean result, and repeats through the finest scale.At each higher scale, the upsampled image is combined with noise before reverse diffusion continues.Source: §3.2 · pp. 3–4, Figure 3 · p. 4
  • Blur is added during training because upsampling introduces blur during sampling, requiring the model to remove both blur and noise.Removing blur from training produces smoother samples lacking fine details such as pyramid texture and stars.Source: §3.2 · p. 4, Figure 5 · p. 5
  • A shared fully convolutional denoiser is conditioned on timestep and scale, uses a 35×35 receptive field, and avoids memorizing the whole image.The small receptive field targets local structure statistics; the model has four convolutional blocks and about 1.1 × 10^6 parameters.Source: §3.2 · pp. 4–5, Figure 4 · p. 4

3.3. Guided Generation

  • Image-region guidance constrains selected content with an L2 loss, while text guidance uses CLIP gradients to control content, style, strength, and spatial extent.The text-guided procedure uses fill factor f to restrict affected regions and strength η to control guidance intensity.Source: §3.3 · p. 5
  • Text-based style guidance acts at the finest scale, preserving random global structure while modifying fine textures according to the prompt.For content guidance, CLIP is applied at all scales except the coarsest, with gradients masked spatially.Source: §3.3 · pp. 5–6, Figure 6 · p. 6

4. Experiments

SinDDM is evaluated across unconditional generation, text-guided generation, ROI control, style transfer, and harmonization. Across these tasks, it produces diverse samples while preserving or modifying image structure and style as required.

4. Experiments

  • SinDDM generates diverse, high-quality samples of arbitrary dimensions, with higher diversity than competing single-image models and comparable no-reference quality.Its SIFID is higher, indicating generalization beyond the training image’s structures.Source: §4 · p. 6
  • Text guidance controls image contents and local ROIs while preserving internal training-image statistics and supporting diverse output dimensions.Compared with Text2Live and Stable Diffusion, SinDDM is not constrained to the training image’s aspect ratio or scene arrangement and uses one prompt for the desired result.Source: §4 · p. 6, SinGAN · p. 8

SinGAN

  • Text-guided style generation adds characteristic textures, brush strokes, and fine semantic details associated with the requested style.The method also supports text-guided style transfer of the training image itself.Source: SinGAN · p. 8, Figure 9 · p. 8
  • ROI guidance fixes user-selected image content while generating coherent surrounding regions, including outpainting when the ROI is the entire training image.With a wider target aspect ratio, SinDDM generates diverse content outside the constrained region that stitches coherently to it.Source: SinGAN · p. 8, Figure 10 · p. 9
  • Style transfer preserves the content image’s global structure while applying the training image’s textures and style.The content image is injected during the multi-scale diffusion process, with histogram matching at the injection scale.Source: SinGAN · p. 8, Figure 11 · p. 9
  • Harmonization blends a naively pasted object into a background by adapting its texture to the background image, producing a less blurry result than SinIR.The composite is injected at an intermediate scale and timestep before reverse diffusion.Source: SinGAN · p. 9

5. Conclusion

SinDDM combines single-image diffusion with multi-scale structure, supporting externally guided image generation and applications including text-guided content, style, style transfer, and harmonization. Its weaker confinement to the training image can benefit style transfer but cause object over- or under-representation in unconditional generation.

5. Conclusion

  • SinDDM combines a single-image generative model with multi-scale diffusion and supports externally guided text-based control of sample contents and style.The method is presented as combining DDM flexibility with SinGAN-like multi-scale structure.Source: §5 · p. 9
  • SinDDM is often less confined to the training image’s internal statistics, which can aid style transfer but over- or under-represent objects during unconditional generation.The conclusion identifies this as a limitation and points to colors in Fig. 11 and examples in Appendix H.Source: §5 · p. 9

A. Additional Examples

  • The method demonstrates text-guided content generation, text-guided style generation, text-guided style transfer, ROI-guided generation, and harmonization.Additional examples cover unconditional sampling, content and style guidance, ROI guidance, style transfer, and harmonization.Source: Figure 12 · p. 9, A. Additional Examples · p. 12, Figure S9 · p. 20
  • SinDDM can modify the training image’s fine textures according to a text prompt by injecting that image at the finest scale.This produces text-guided style transfer rather than style transfer from an example image.Source: Figure S9 · p. 20

B. SinDDM Architecture

SinDDM uses a fully convolutional denoiser conditioned on timestep and scale, with a restricted receptive field intended to reduce memorization when training on one image.

B. SinDDM Architecture

  • The denoiser receives a noisy, blurry image plus timestep t and scale s, then predicts noise through a fully convolutional pipeline.The timestep and scale are embedded and supplied to each convolutional block.Source: B. SinDDM Architecture · p. 22, Figure S11 · p. 22
  • Four SinDDM convolutional blocks produce a total 35 × 35 receptive field, limiting access to the training image’s broader context.Each block has a 9 × 9 receptive field, and the restricted field is used to avoid memorization.Source: B. SinDDM Architecture · p. 22
  • SinDDM embeds timestep t and scale s with sinusoidal positional embeddings and fully connected GeLU layers before conditioning each residual convolutional block.The image pipeline uses 2D convolutions, residual connections, and GeLU activations.Source: B. SinDDM Architecture · p. 22

C. Training Details

Training uses Adam with exponential moving averaging, a scheduled learning rate, and batch-size adjustments for different GPUs. A 200 × 250 image takes around 7 hours on an A6000 and 20 hours on an RTX2080.

C. Training Details

  • Training on an A6000 uses Adam, EMA decay 0.995, 120 × 10^3 steps, initial learning rate 0.001, scheduled halvings, and batch size 32.The learning rate is halved at [20, 40, 70, 80, 90, 110] × 10^3 steps.Source: C. Training Details · p. 22
  • A 200 × 250 image takes around 7 hours to train on an A6000 and around 20 hours on an RTX2080.The RTX2080 setting uses batch size 16 and four times as many steps, with corresponding scheduler changes.Source: C. Training Details · p. 22

D. Derivation of the Forward and Reverse Diffusion Processes

SinDDM trains each scale on mixtures of clean, noisy, and upsampled blurry images, so reverse diffusion performs both denoising and deblurring across scales.

D. Derivation of the Forward and Reverse Diffusion Processes

  • At scales s > 0, training combines the clean image, Gaussian noise, and an upsampled blurry image from the previous scale.The resulting model learns the deblurring and denoising required when reverse diffusion begins from an upsampled prior-scale image.Source: D. Derivation of the Forward and… · p. 23
  • At each non-coarsest scale, reverse diffusion starts from a noisy version of the upsampled output generated at the previous scale.The coarse-to-fine process therefore propagates previously generated structure while refining the current scale.Source: D. Derivation of the Forward and… · p. 23, §D.1 · p. 23
  • The denoiser predicts noise from the mixed image at every timestep, after which the sampling equations estimate the clean-scale image and the mixed state.The construction follows the DDPM noise-prediction procedure while incorporating the blurry previous-scale conditioning.Source: D. Derivation of the Forward and… · p. 23

D.1. The Maximal Noise Level in Each Scale

  • The maximal sampling noise at each scale is selected so its noise-to-signal ratio matches the L2 distance between the original and blurry scale images.This adds enough noise to generate missing details without masking details already produced at the previous scale.Source: §D.1 · p. 23
  • Training on timesteps beyond those used during sampling produced better results than training only on the sampled timesteps.Each scale is trained for T = 100 timesteps, although sampling starts from a scale-specific maximal timestep.Source: §D.1 · p. 23

D.2. The γst Schedule

SinDDM schedules blur and noise jointly so each reverse step can remove both, while its text-guidance algorithm modifies denoised images through CLIP gradients and momentum.

D.2. The γst Schedule

  • For s = 0, γs_t = 0 because the coarsest scale performs denoising only, whereas finer scales choose γs_t so noise and blur levels are similar.At finer scales, each reverse step therefore performs denoising together with deblurring.Source: §D.2 · p. 23
  • Removing blur from training produces smoother samples lacking fine details, demonstrating the role of blur in the diffusion schedule.The comparison is shown in Figure 5.Source: §D.2 · p. 23

D.3. The Sampling Process

  • The sampling formulation uses DDPM sampling for s = 0 and a deterministic process for s > 0 through different σs_t choices.The cited formulation identifies the first choice with DDPM sampling and the second with deterministic sampling.Source: §D.3 · p. 24

E.1. Algorithm

  • Text guidance updates the denoised image with an adaptive CLIP-gradient step whose strength η controls modification intensity.A spatial mask restricts the update to the selected region, while the update combines the CLIP gradient with the current denoised image.Source: §E.1 · p. 24
  • Momentum is added because the denoiser’s strong training-image prior can otherwise undo each preceding CLIP update.The momentum parameter λ is set to 0.05 in the reported experiments.Source: §E.1 · p. 24
  • In the text-guidance variant, γs_t is set to zero for every scale and timestep, so the reverse process explicitly denoises without removing blur.This differs from the blur-matched schedule used in the derivation.Source: §E.1 · p. 24

E.2. Data Augmentation for Text Guided Generation

Text-guided generation uses augmented image-text inputs and exposes controls over guidance scale, affected area, strength, and object size; comparisons cover sampling and guided generation tasks.

E.2. Data Augmentation for Text Guided Generation

  • For ROI text guidance, SinDDM feeds the ROI and augmentations to CLIP after upsampling the ROI to 224 × 224, using additional low-quality and blurry text templates.The templates address the blur introduced by resizing the ROI for CLIP’s image encoder.Source: §E.2 · p. 24

E.3. Effect of Initial Scale in Generation with Text Guided Contents

  • Starting CLIP guidance at finer scales preserves global structure, whereas starting at coarse scales allows guidance to modify global structure as well.The main-text results begin guidance from the second-coarsest scale, s = 1; guidance from s = 0 begins after T/2 unguided steps.Source: Figure S12 · p. 25, §E.3 · p. 25

E.4. Effect of the Fill-Factor and Strength Parameters

  • Increasing the fill factor f affects a larger image region, while increasing strength η makes modifications within that region more prominent.Both parameters are user-selectable controls for text-guided generation.Source: Figure S13 · p. 26, §E.4 · p. 26

F. Controlling Object Sizes

  • Starting from a larger coarsest-scale noise map makes generated objects appear smaller relative to the image.This follows because the model conditions on the smallest training-image scale while the generated image has larger dimensions.Source: F. Controlling Object Sizes · p. 26, Figure S14 · p. 27

G. Comparisons

  • SinDDM is compared with competing methods on unconditional sampling, text-guided content and style generation, ROI generation, style transfer, and harmonization.The unconditional-sampling comparison includes SinGAN, ConSinGAN, GPNN, and SinDDM.Source: G. Comparisons · p. 27, Figure S15 · p. 28

G.2. Generation with Text-Guided Content

SinDDM is evaluated against established methods across text-guided content, ROI editing, style generation, style transfer, and harmonization, with different trade-offs across tasks.

G.2. Generation with Text-Guided Content

  • SinDDM is not constrained to the input image’s aspect ratio or precise object configurations in text-guided content generation.The comparison presents multiple SinDDM samples at different aspect ratios against Text2Live and Stable Diffusion.Source: §G.2 · p. 29, Figure S16 · p. 29, Figure S17 · p. 30
  • Stable Diffusion image-to-image editing trades fidelity for text influence as strength increases, while its official implementation produced poor non-photo-realistic results on these images.The reported comparisons use a modified resizing strategy to obtain natural-looking Stable Diffusion results.Source: §G.2 · p. 29, Figure S19 · p. 31

G.3. Generation with Text-Guided Content in ROI

  • In ROI generation, Stable Diffusion inpainting produced results that were not loyal to the text and did not blend well, while DALL-E 2 outputs were sometimes unrealistic or unrelated to the prompt.The comparison is shown in Figure S20.Source: §G.3 · p. 32, Figure S20 · p. 32

G.4. Generation with Text-Guided Style

  • Text-guided style generation produces desired-style textures, brush strokes, and fine semantic details such as characteristic scenery or sunflowers.Comparisons are provided with Text2Live and Stable Diffusion.Source: §G.4 · p. 33, Figure S21 · p. 33, Figure S22 · p. 34

G.5. Style Transfer

  • SinDDM generates style-transfer results that are simultaneously more loyal to the content image and the style image than SinIR.This comparison is reported in Figure S24.Source: §G.5 · p. 35, Figure S24 · p. 35

G.6. Harmonization

  • For harmonization, SinDDM produces stronger blending effects than SinIR and ConSinGAN, but with some blur in the pasted object.The method trains on the background and injects a downsampled pasted composite at an intermediate scale and timestep.Source: §G.6 · p. 36, Figure S25 · p. 36

H. Limitations and Directions for Future Work

SinDDM has limitations in faithfully representing object distributions and in generating text-guided content beyond the training image’s internal textures. Padding choices also alter where sample diversity appears, especially near image corners.

H.1. Misrepresentation of the Distribution of Certain Objects

  • Object distributions can be misrepresented, with manual timestep tuning helping but no automatic choice working reliably across all images.The sampling scheme can overproduce elephants, omit trees, or severely underrepresent birds; removing cross-scale blending instead introduces blur.Source: §H.1 · p. 37, Figure S26 · p. 38

H.2. Inner Distribution Preservation Under Text Guided Content Generation

  • Text-guided generation remains constrained to textures present in the training image’s internal distribution.For example, fire in a forest appears realistic because the generated fire uses the image’s orange sky texture.Source: §H.2 · p. 37

H.3. Boundary conditions and the effect of padding

  • Padding changes sample diversity spatially: layer padding reduces border variability while initial padding increases it at the borders and reduces it in the interior.The effect is illustrated through location-wise standard deviations among generated samples.Source: §H.3 · p. 37, Figure S27 · p. 39

Related papers

Loading 2211.16582v3…