PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Diffusion Latent Optimization Improves Classifier Guidance

Bram Wallace, Akash Gokul, Stefano Ermon, Nikhil Naik

arXiv:2303.13703v2cs.CVcs.AIcs.LG

TL;DR

Classifier guidance is limited by the need to train noise-aware classifiers or use one-step approximate images that can misalign gradients with final generations. DOODL optimizes diffusion latents using losses on fully generated pixels and an invertible process for memory-efficient backpropagation. Across several guidance tasks, the paper reports improved generations and classifier-guided vocabulary expansion.

  • Problem

    Classifier guidance either requires training noise-aware classifiers or uses gradients from a one-step approximation that may be misaligned with final generations.

    Source: §1 · p. 2, §3.2.1 · p. 4

  • Method

    DOODL iteratively optimizes diffusion latents against differentiable losses on fully generated pixels, using invertible diffusion for constant-memory backpropagation.

    Source: §1 · p. 2

  • Results

    DOODL demonstrates improvements across complex-prompt guidance, fine-grained vocabulary expansion, personalized entity generation, and aesthetic-quality guidance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DOODL offers a way to incorporate pretrained recognition-network knowledge into diffusion generation, with future work aimed at broader guidance and more efficient optimization.

    Source: §6 · p. 9

Abstract

from arXiv · show

Classifier guidance -- using the gradients of an image classifier to steer the generations of a diffusion model -- has the potential to dramatically expand the creative control over image generation and editing. However, currently classifier guidance requires either training new noise-aware models to obtain accurate gradients or using a one-step denoising approximation of the final generation, which leads to misaligned gradients and sub-optimal control. We highlight this approximation's shortcomings and propose a novel guidance method: Direct Optimization of Diffusion Latents (DOODL), which enables plug-and-play guidance by optimizing diffusion latents w.r.t. the gradients of a pre-trained classifier on the true generated pixels, using an invertible diffusion process to achieve memory-efficient backpropagation. Showcasing the potential of more precise guidance, DOODL outperforms one-step classifier guidance on computational and human evaluation metrics across different forms of guidance: using CLIP guidance to improve generations of complex prompts from DrawBench, using fine-grained visual classifiers to expand the vocabulary of Stable Diffusion, enabling image-conditioned generation with a CLIP visual encoder, and improving image aesthetics using an aesthetic scoring network. Code at https://github.com/salesforce/DOODL.

1. Introduction

Pretrained diffusion models are difficult to guide with arbitrary external signals, while existing classifier-guidance approaches require noise-aware retraining or rely on a potentially misaligned one-step approximation. DOODL optimizes diffusion latents against losses on fully generated pixels, and the paper demonstrates it across several guidance tasks.

1. Introduction

  • Text-conditioned diffusion models are difficult to steer with external signals such as depth maps, image classifiers, or audio without a plug-and-play guidance approach.Training a model conditioned on multiple modalities is another approach, but the paper identifies plug-and-play guidance as desirable.Source: §1 · p. 2
  • One-step classifier guidance can use gradients misaligned with final generations because it evaluates an approximate image, while noise-aware classifiers require retraining.The one-step approximation is inaccurate at high noise levels.Source: §1 · p. 2, Figure 2 · p. 2
  • DOODL optimizes initial diffusion noise against a differentiable loss on fully generated pixels, using invertible diffusion for constant-memory backpropagation.This avoids noise-aware classifiers and one-step approximations while enabling iterative optimization of diffusion latents.Source: §1 · p. 2
  • DOODL is evaluated across CLIP guidance for complex prompts, fine-grained classifier vocabulary expansion, personalized entity generation, and aesthetic-quality improvement.The paper reports quantitative and human evaluation studies across these guidance signals.Source: §1 · p. 2

2. Related Work

Related work contrasts learned conditioning methods with guidance that uses pretrained recognition networks. It also situates DOODL's use of EDICT within invertible diffusion and memory-saving reversible networks.

2.1. Text-to-Image Diffusion Models

  • Prior latent optimization with pixel-based losses had been applied to GANs and VAEs, but the authors report that it had not been applied to diffusion models before DOODL.The authors describe this as their knowledge-bound claim.Source: §2.1 · p. 2
  • Learning-based methods require data and training, whereas guidance-based methods use pretrained recognition networks trained on non-noisy data in this setting.The paper frames guidance as avoiding the data and training requirements of learned methods.Source: §2.1 · p. 2, Table 1 · p. 3

2.2. Invertible Neural Networks (INNs)

  • EDICT reformulates denoising diffusion to be invertible, while prior work discussed it for image editing rather than invertible-network properties.DDIM is theoretically invertible in the discretization limit, but that limit cannot be achieved in practice.Source: §2.1 · p. 3, §2.2 · p. 3

3. Background

The background explains how invertible networks reduce activation-storage costs, how diffusion maps noise to images, and how classifier guidance modifies generation. It then describes EDICT's invertible paired-latent process, which DOODL uses for image encoding.

3.1. Invertible Neural Networks w.r.t Memory

  • Invertible networks can reconstruct intermediate states during backpropagation, avoiding activation caching and reducing memory costs.The general non-invertible approach requires storing intermediate activations; checkpointing lowers memory at a runtime cost.Source: §3.1 · p. 3

3.2. Denoising Diffusion Models (DDMs)

  • A diffusion model iteratively denoises a sampled noise vector through timestep-conditioned functions to produce the final image.The model is trained to predict added noise at sampled timesteps, with conditioning information such as an image caption.Source: §3.2 · p. 3

3.2.1 Classifier Guidance

  • Classifier guidance incorporates gradients of a classifier loss on estimated pixels into the noise prediction, either using a noise-aware model or a one-step image estimate.The one-step estimate can yield gradients misaligned with the true derivative, while noise-aware classifiers require training for intermediate noisy inputs.Source: §3.2.1 · pp. 3–4

3.2.2 Exact Inversion of the Diffusion Process

  • EDICT denoises a pair of latent sequences through an invertible process; its mixing parameter p mitigates latent drift.DOODL uses the construction to encode images into latents for optimization.Source: §3.2.2 · p. 4

4. Direct Optimization of Diffusion Latents

DOODL optimizes the initial diffusion noise using a model-based loss on the full-chain generated image, with EDICT enabling backpropagation at constant memory cost. Its latent updates are designed to meet guidance criteria while preserving realistic outputs.

4. Direct Optimization of Diffusion Latents

  • The method targets guidance without classifier retraining, using gradients on the true output and seeking semantically meaningful rather than adversarial-style changes.The stated criteria also include avoiding one-step approximations.Source: §4 · p. 4
  • Naive backpropagation stores activations across diffusion steps, approaching a terabyte for 50 steps; checkpointing trades memory savings for slower backward passes.EDICT reconstructs intermediate states during backpropagation with constant memory cost and without sacrificing runtime.Source: §4 · p. 4
  • DOODL uses EDICT to backpropagate a loss on the final generated image and optimize the fully noised latent xT directly.Figure 3 contrasts this with standard guidance, which uses a one-step denoising approximation at each timestep.Source: §4 · p. 4, Figure 3 · p. 5
  • After each optimization step, DOODL averages the fully noised latent pair and renormalizes it to the initial draw’s norm.Averaging prevents latent drift that degrades quality, while normalization keeps the latent on the Gaussian shell and in-distribution.Source: §4 · p. 4
  • DOODL uses 16 sampled crops per image and momentum with η = 0.9; it clips gradients at 10^-3 and adds N(0, 10^-4 · I) noise to xT at each update.The authors report that Nesterov momentum was not useful.Source: §4 · pp. 4–5

5. Applications and Results

This section evaluates DOODL across CLIP text guidance, fine-grained vocabulary expansion, aesthetic editing, and visual personalization. It reports stronger prompt alignment, reduced rare-class FID, more satisfactory aesthetic edits, and improved personalized dog generation.

5.1. Reinforcing Text Guidance

  • DOODL raises baseline classifier guidance’s CLIP score by 3.1% on DALLE and 2.6% on Reddit.On Reddit, prompt alignment success rate is 11.3% higher than vanilla Stable Diffusion and 6% higher than one-step guidance.Source: §5.1 · p. 6

5.2. Vocabulary Expansion

  • Across CUB, Dogs, and FGVC-A, DOODL reduces FID versus original Stable Diffusion by about 3% on average, while classifier guidance improves none.The reported qualitative examples show DOODL generating fine-grained features that identify the target category.Source: Table 2 · p. 6, §5.2 · p. 7, Figure 6 · p. 7

5.3. Aesthetic Improvement

  • For edits of existing images, DOODL produces satisfactory aesthetic improvements significantly more often than baseline classifier guidance in human evaluation.The method inverts an image with EDICT and optimizes the latents against an aesthetic cost; the baseline’s full-step inversion largely decorrelates generated content from the original.Source: §5.3 · pp. 7–8, Figure 8 · p. 8

5.4. Visual Personalization Guidance

  • DOODL maintains faithfulness to text and visual conditioning; human-rated same-dog/context success is 14%, versus 2% for original generation and 1% for baseline guidance.The study uses four target dogs, four prompt templates, and six random seeds.Source: §5.4 · p. 8

6. Conclusion & Future Work

The conclusion presents DOODL as a way to incorporate pretrained recognition networks into diffusion generation and calls for broader guidance types and more efficient optimization.

6. Conclusion & Future Work

  • The authors present DOODL as a new way to incorporate pretrained recognition networks into diffusion generation, while identifying broader guidance types and faster optimization as future work.They point to applications requiring higher compute efficiency as a motivation for accelerating optimization.Source: §6 · p. 9

A. Text-to-Image Generative Aesthetics Results

The aesthetic-guidance experiments compare DOODL against original generations and baseline classifier guidance, with human ratings favoring DOODL over the originals but showing a less consistent advantage over the baseline.

A. Text-to-Image Generative Aesthetics Results

  • The aesthetic-scoring experiment used the standard algorithm with λDOODL = 0.1 and showed human comparisons over eight prompts and 16 seeds.The prompts asked raters to select the image they thought most people would prefer visually.Source: A. Text-to-Image Generative Aesthetics… · p. 11
  • 0.58−0.11−0.31 was DOODL’s win-draw-loss rate against original generations, while its rate against baseline classifier guidance was 0.48−0.07−0.45.The study used 128 comparisons against baseline classifier guidance.Source: A. Text-to-Image Generative Aesthetics… · p. 11
  • DOODL optimization was more prone than baseline guidance to warp or deform image content, which the authors describe as visually unappealing.They suggest further stabilizing DOODL could broaden its performance gap.Source: A. Text-to-Image Generative Aesthetics… · p. 11

B. Further Analysis of Personalization Results

The personalization analysis reports higher success rates for DOODL than for original generations or the baseline, including under unanimous labeling, while also examining a more permissive similarity threshold.

B. Further Analysis of Personalization Results

  • 19.4% of DOODL generations were labeled as appearing to be the original dog, versus 4.5% for original generations and 5.6% for the baseline.Label (4) was the response that the dog in the second image appeared to be the original dog.Source: B. Further Analysis of Personalization… · p. 11
  • 3.1% (3/96) of DOODL images received unanimous label (4), compared with 0 for both other methods.The authors note that few images were unanimously classified this way given the challenging task and noisy labeling.Source: B. Further Analysis of Personalization… · p. 11
  • The analysis also visualizes success rates under a more permissive criterion that accepts either label (3) or label (4).Label (3) allows significant differences while still judging the dog similar to the original.Source: B. Further Analysis of Personalization… · p. 11

C. Details on Multicrop in DOODL Optimization

The multicrop augmentation samples a square crop size between the model input size and original image size, then selects a crop uniformly from the image.

C. Details on Multicrop in DOODL Optimization

  • The multicrop procedure samples r uniformly from 0 to 1 to set the square crop size, then selects the crop uniformly from the image.The crop-size expression uses ModelInputSize + (OriginalImageSize − ModelInputSize)r^CutPower, with cut power 0.3.Source: C. Details on Multicrop in DOODL… · p. 11

D. Additional Qualitative Results

The supplementary section provides additional qualitative examples for aesthetic results, DrawBench, fine-grained visual classification, and personalization, alongside a human aesthetic evaluation and a personalization success-rate table.

D. Additional Qualitative Results

Loading 2303.13703v2…