Source-linked AI summary
D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
Abhishek Sinha, Jiaming Song, Chenlin Meng, Stefano Ermon
TL;DR
Paired condition-image supervision can be costly, limiting how easily generative models can support new conditions. D2C combines contrastive representations with latent diffusion in an unconditional VAE, and reports strong few-shot conditional generation results, including faster image manipulation than StyleGAN2.
Problem
Method
Results
With 100 labeled examples, D2C significantly outperforms state-of-the-art VAEs and diffusion models on conditional generation; its manipulation generations are around two orders of magnitude faster than StyleGAN2.
Source: §1 · p. 2, §7.3 · p. 9Takeaways & Limitations
The reported demonstrations cover few-shot generation from labels and image manipulation constraints, using an unconditional image-trained model.
Source: §1 · p. 2, §5 · p. 6D2C is trained on images only, without other modalities such as image-text pairs or supervision techniques such as meta-learning.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% - 60% of the human evaluators in a double-blind study.
1 Introduction
D2C addresses the cost of acquiring paired condition-image data by training an unconditional VAE for few-shot conditional generation. Its contrastive representations and latent diffusion support conditional generation, with reported gains over state-of-the-art models on several tasks.
1 Introduction
- Learning conditional generators directly often requires large amounts of costly paired data, motivating approaches that use unlabeled data and little paired supervision.Generative models trained on unlabeled data have succeeded in images, text, audio, and graphs, while downstream applications often require conditions such as labels, text, rewards, or similarity.Source: §1 · p. 1
- Contrastive self-supervised learning can improve few-shot representations, but GANs and diffusion models lack explicit tractable input-to-latent mappings, while VAEs support encoder-based SSL but typically have worse sample quality.This contrast motivates combining self-supervised representations with a generative model designed to improve sample quality.Source: §1 · pp. 1–2
- D2C combines contrastive self-supervised learning with a diffusion model over latent representations in a VAE suited to few-shot conditional generation.The authors argue that using the same latent distribution for training and generation may improve sample quality over hierarchical VAEs.Source: §1 · p. 2
- With 100 labeled examples, D2C significantly outperforms state-of-the-art VAE and diffusion models on conditional generation; for image manipulation, it is two orders of magnitude faster than StyleGAN2 and preferred by 50%–60% of evaluators.The authors evaluate D2C against state-of-the-art generative models over six datasets.Source: §1 · p. 2
2 Background
The background introduces latent-variable generative models, variational autoencoders, self-supervised representations, and diffusion models. It describes their latent mappings and the denoising process used to generate samples.
Latent variable generative models
- Latent-variable generative models map latent variables to generated samples, with common paradigms including VAEs, normalizing flows, GANs, and diffusion or score-based models.Sampling draws z from p(z) and maps it through pθ(x|z).Source: §2 · p. 2
- VAEs use an inference model qφ(z|x) and train by minimizing an upper bound on negative log-likelihood, combining reconstruction and KL-divergence terms.The objective averages over the data distribution and the inference distribution over z.Source: §2 · p. 2
Diffusion models
- Diffusion models learn to predict added Gaussian noise at different noise levels, using a mean-squared-error loss between true and predicted noise.The diffusion objective can weight losses across noise levels; setting w(α)=1 recovers denoising score matching.Source: §2 · p. 3
- Sampling begins with Gaussian noise and gradually denoises toward a clean sample; DDIM uses an Euler discretization of a neural ODE.The notation for diffusion sampling abstracts the procedure, which depends on choices of noise levels.Source: §2 · p. 3
Self-supervised learning of representations
- Self-supervised learning trains representations through pretext tasks without manual labels; contrastive learning brings positive pairs closer and pushes negative pairs farther apart.Non-contrastive methods such as BYOL and SimSiam also exist, but typically require additional care to prevent representation collapse.Source: §2 · p. 3
3 Problem Statement
The problem is to learn unconditional image generators that can support conditioning from limited supervision, including conditions not known during training. Such models need both high-quality samples and informative latent representations.
Few-shot conditional generation
- Conditional generation seeks samples x satisfying an event f(x)=c, but when f is unknown, learning it accurately may require labeled image-condition pairs.The paper frames its goal as learning an unconditional generator that can later be used for conditional generation.Source: §3 · p. 3
Desiderata
- Many existing methods target known conditions or rely on abundant paired image-condition data, whereas this work does not assume training on paired data.Examples include conditional GANs trained for known labels and image-text methods pretrained on paired data.Source: §3 · p. 3
- Informative latent representations can help specify conditions with limited supervision, while VAEs support rich representations through encoder-based SSL but generally trail GANs and diffusion models in sample quality.The paper points to image manipulation for detecting and removing dataset biases as a use case for adapting to hard-to-anticipate needs.Source: §3 · pp. 3–4
- The comparison criteria for generative paradigms include trainable x-to-z mappings for SSL, matched training and generation latent distributions, and non-adversarial training.The table associates the latter two criteria with improved generation and training stability, respectively.Source: Table 1 · p. 4
4 Diffusion-Decoding Generative Models with Contrastive Learning
D2C combines a VAE decoder with latent diffusion and contrastive self-supervision. The paper connects its objective to a variational lower bound and argues that diffusion addresses latent prior holes that can persist despite small distribution divergences.
4 Diffusion-Decoding Generative Models with Contrastive Learning
- D2C combines latent diffusion with a decoder and an inference model that can incorporate contrastive self-supervised learning.Its objective adds a weighted contrastive loss to reconstruction and diffusion losses.Source: §4 · p. 4
4.1 Relationship to maximum likelihood
- The D2 objective is described as deeply connected to the variational lower bound of log-likelihood.The paper says the full statement and proof appear in Appendix B.1.Source: §4.1 · p. 4
4.2 D2C models address latent posterior mismatch in VAEs
- Low divergences between a VAE’s aggregate posterior and prior do not necessarily eliminate large prior holes, especially when the posterior is flexible.The paper gives a construction with a prior hole despite small KL and Wasserstein distances.Source: §4.2 · p. 5
- Diffusion is argued to eliminate prior holes by construction, keeping training latents arbitrarily close to those used in generation.The paper presents this as an explanation for better sample quality from GANs and diffusion models than from VAEs and normalizing flows.Source: §4.2 · pp. 5–6
5 Few-shot Conditional Generation with D2C
D2C adapts an unconditional image model to few-shot conditions by learning a latent-space condition model and sampling latents with the diffusion prior. The paper describes approaches for both labels and image-manipulation constraints, using images rather than other modalities or meta-learning.
5 Few-shot Conditional Generation with D2C
- The described conditional-generation setting trains D2C on images, without image-text pairs or meta-learning..Source: §5 · p. 6
- The general procedure encodes few labeled examples, trains a latent condition model, samples condition-matched latents with the diffusion prior, and decodes them into images.The condition model and diffusion prior form an unnormalized sampling distribution.Source: §5 · p. 6
- For labeled examples, a latent classifier guides generation; when examples are unlabeled, a positive-unlabeled classifier can mark them positive and training data unlabeled.The classifier can guide rejection sampling from the diffusion model.Source: §5 · p. 6
- For manipulation, D2C combines label probability with closeness to a reference latent and corrects classifier-guided gradient steps with the diffusion model.The authors say this avoids test-time inversion optimization used by many GAN methods and helps retain fine-grained features through reconstruction loss.Source: §5 · p. 6
6 Related Work
Related work contrasts the sample quality, invertibility, and training properties of generative-model families. It also reviews approaches to posterior mismatch and conditioning unconditional models with discriminative guidance.
6 Related Work
- GANs can achieve better sample quality than VAEs and normalizing flows, but may be difficult to invert and can have unstable training; diffusion models offer high sample quality without adversarial training.The paper notes that diffusion models’ latent dimension must equal the image dimension.Source: §6 · p. 6
- Prior-mismatch methods improve inference models, priors, or objectives by bringing posterior and prior closer, while two-stage approaches fit a generative model over autoencoder latents.The paper argues that divergence optimization does not necessarily remove large prior holes.Source: §6 · pp. 6–7
- Conditional generation with unconditional models often uses a discriminative model to guide image- or latent-space changes, but imperfect reconstruction can hinder manipulation of real images.The paper contrasts this with D2C’s reconstruction objective.Source: §6 · p. 7
7 Experiments
D2C is evaluated on unconditional generation, few-shot label-conditioned generation, and image manipulation, with conditional tests using small labeled sets. Across these settings, the reported results favor D2C against the compared models, with especially large speed gains for manipulation.
7 Experiments
- Experiments cover six image datasets and use an NVAE-based autoencoder, U-Net diffusion model, and MoCo-v2 contrastive learning; DDIM and D2C generally use 100 diffusion steps.The authors report that results are relatively insensitive to the contrastive weight λ.Source: §7 · p. 7
7.1 Unconditional generation
- For unconditional generation, D2C beats NVAE on FID across all datasets and DDIM on CelebA-64 and CelebA-HQ-256, while its linear classification accuracy is much better.Reconstruction MSE is low and comparable across D2C, NVAE, and DDIM; the accuracy advantage is attributed to contrastive SSL.Source: §7.1 · p. 7, Table 2 · p. 7, Table 3 · p. 8
7.2 Few-shot conditional generation from examples
- Across eight CelebA-64 label tasks, D2C outperforms the other approaches and beats the unlabeled-data “naive” approach in most cases.The tasks use binary classifiers trained on 100 examples and positive-only classifiers trained on 100 positive and 10k unlabeled examples.Source: §7.2 · p. 8, Table 5 · p. 9
7.3 Few-shot conditional generation from manipulation constraints
- D2C preserves original-image details better than StyleGAN2 in blond and red-lipstick edits; AMT evaluators slightly prefer it for blond and significantly for red lipstick.The qualitative comparison highlights details such as eyes, earrings, and background.Source: Figure 4 · p. 9
- For manipulation, D2C is preferred over StyleGAN2 by 51.5% for blond and 60.8% for red lipstick, and generations take around two orders of magnitude less time.Latent-code acquisition takes 0.013 seconds for D2C versus 8 seconds for StyleGAN2 on an Nvidia 1080Ti GPU.Source: §7.3 · p. 9, Figure 5 · p. 9
8 Discussions and Limitations
The discussion frames D2C as a VAE-based model for few-shot conditional generation, while noting open directions and a potential misuse concern.
8 Discussions and Limitations
- The authors identify potential negative societal impacts, including deepfakes, and say D2C should be used properly to mitigate them.This is a stated concern, not a reported evaluation of misuse.Source: §8 · p. 10
- The authors describe their model as the first unconditional VAE to demonstrate superior image-manipulation performance to StyleGAN2, despite using a regular NVAE architecture.They suggest that architectures such as StyleGAN2 designs or Transformers could improve performance further.Source: §8 · p. 10
- Future directions include integrating other latent-space conditions and training D2C with other domains or modalities, such as text.The discussion compares this prospective training direction to semi-supervised learning.Source: §8 · p. 10
A.1 Training diffusion models
The appendix outlines diffusion training and sampling, contrastive representation learning, D2C training, and few-shot conditional sampling procedures.
A.1 Training diffusion models
- The diffusion training objective can be viewed as weighted noise-prediction mean-square errors; unit weights are reported to work decently for image generation.The appendix also notes that alternative weights can be derived for non-Markovian forward processes.Source: §A.1 · p. 16
A.2 DDIM sampling procedure
- DDIM sampling deterministically maps noise to a clean sample, supports selectable diffusion steps, and can be run in reverse to obtain a latent code from an observation.The passage reports better sample quality when few sampling steps are allowed and a near-invertible mapping.Source: §A.2 · p. 17
A.3 Contrastive representation learning
- Contrastive learning distinguishes two augmented views of the same image from views of different images; CPC frames this as classification of the positive pair among negatives.MoCo-v2 is the paper’s implementation, though other CPC implementations can be integrated.Source: §A.3 · p. 17
A.4 Training D2C
- D2C training integrates autoencoder reconstruction, latent diffusion, and latent contrastive objectives, with global mean-and-standard-deviation normalization before diffusion training.The components may be optimized jointly or separately.Source: §A.4 · p. 18
A.5 Few-shot conditional generation
- For labels, the procedure rejects diffusion-prior latent samples with low classifier scores; for manipulation, it steps from the original latent and regularizes with the diffusion prior.The authors note that gradients over the full energy would be more principled, while their current version is computationally efficient enough to perform well.Source: §A.5 · pp. 18–19
B Formal Statements and Proofs
The formal appendix relates the diffusion objective to a likelihood bound and analyzes how latent prior holes can coexist with small divergence measures. It gives a construction in which increasing the number of rings reduces the Wasserstein upper bound while the prior hole and KL upper bound remain constant.
B.1 Relationship to maximum likelihood
- The appendix states that the diffusion objective −LD2 is a variational lower bound on the model’s log-likelihood.It defines pθ(x) as the marginal probability under the D2C model.Source: §B.1 · p. 20
- For t ≥ 2, matching the standard deviations of the Gaussian reverse distributions makes the KL term a weighted Euclidean distance between their means.The derivation also gives weights for α2:T and separately specifies the weight for α1.Source: §B.1 · pp. 20–21
B.2 D2 models address latent posterior mismatch in VAEs
- A latent distribution can have an (ϵ, δ)-prior hole while its KL divergence is at most log 2 and its 2-Wasserstein distance is below any γ > 0.The formal statement specifies pθ(z) = N(0, I), δ < 0.5, and any ϵ > 0.Source: §B.2 · p. 21
- The construction partitions Gaussian probability mass across 2n circles in an arithmetic progression from 0 to 2δ.The radii are selected so successive rings have equal increments in probability mass.Source: §B.2 · p. 21
- Using more rings keeps the prior hole and KL-divergence upper bound constant while decreasing the Wasserstein-distance upper bound.Figure 6 illustrates this construction in two dimensions.Source: Figure 6 · p. 22
- The construction moves probability mass from odd-numbered rings to even-numbered rings within the region bounded by f⁻¹(2δ).Outside that region, qφ(z) is set equal to pθ(z).Source: §B.2 · pp. 21–22
- For every z, qφ(z) ≤ 2pθ(z), which bounds DKL(qφ∥pθ) by log 2.The passage states this inequality as the KL-divergence argument for the construction.Source: §B.2 · p. 22
C Experimental details
The experiments use a modified NVAE architecture, MoCo-v2 contrastive learning, and a linear latent-space classifier for conditional tasks. The appendix also specifies manipulation step sizes and the human-evaluation procedure.
C.1 Architecture details and hyperparameters used for training
- The encoder and decoder remove NVAE’s Combiner Cells, while contrastive learning uses MoCo-v2 with standard image augmentations.The diffusion model uses the same architecture with channel multiplications as in [40] [81]; hyperparameters are listed in Table 6.Source: §C.1 · p. 23
C.2 Additional details for conditional generation
- Conditional tasks use a linear latent-space classifier; manipulation step sizes are η = 10 for red lipstick and η = 15 for blond.Label-conditioned samples are rejected below classifier threshold 0.5, and the authors note larger η can increase change intensity.Source: §C.2 · p. 23
C.3 Amazon Mechanical Turk procedure
- Evaluators compare image pairs for identity retention and attribute presence across 15 tasks of 10 comparisons per attribute.Each task takes about 2.5 minutes and pays $0.25; Figure 7 shows the instructions and comparison interface.Source: §C.3 · p. 23, Figure 7 · p. 24
D Additional Results
Additional results report unconditional CIFAR-10 performance with a compute and architecture comparability caveat, and point to further unconditional, manipulation, and label-conditioned examples. The broader-impact discussion notes potential data-bias and misuse risks alongside a possible latent-variable screening defense.
D Additional Results
- D2C’s unconditional CIFAR-10 results are slightly worse than state-of-the-art diffusion models, but the comparisons differ in resources, architecture, and compute.D2C training used 8 GPUs for 24 hours, compared with NVAE’s 8 GPUs for 100 hours and DDPM’s v3-8 TPUs for 24 hours; Table 7 covers CIFAR-10 image generation.Source: D Additional Results · p. 24, Table 7 · p. 24
- The appendix provides additional unconditional FFHQ-256 samples and image-manipulation examples for blond hair, red lipstick, beard, and gender.Figures 8–12 show those examples.Source: D Additional Results · p. 24, Figure 8 · p. 25, Figure 9 · p. 26
E Broader Impact
- The paper frames its broader aim as reducing the need for paired data by focusing on unsupervised data rather than resource-intensive paired image-text training.The passage cites large vision-and-language models as requiring substantial data, compute, and energy.Source: E Broader Impact · p. 24
- The authors identify possible inherited or exacerbated data-collection bias and unwanted deepfake use, and propose rejecting problematic latent variables before decoding as a possible defense.They frame this defense conditionally on being able to monitor and control downstream latent-variable use.Source: E Broader Impact · p. 24
D Additional Results
- Figures 13–16 show conditional label-generation examples from D2C and DDIM, each learned from 100 labeled examples.The captions identify D2C examples in Figures 13 and 15, and DDIM examples in Figures 14 and 16.Source: Figure 13 · p. 30, Figure 14 · p. 30, Figure 15 · p. 31