PaperlayerView exact revision on arXiv

Source-linked AI summary

CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models

Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, Jong Chul Ye

arXiv:2406.08070v2cs.CVcs.AIcs.LG

TL;DR

High CFG scales can compromise sample diversity and DDIM inversion, while classifier-free guidance remains a key tool for text-guided generation. CFG++ reformulates text guidance as an inverse problem with a text-conditioned score-matching loss. The authors report improved text-to-image quality and DDIM inversion compared with CFG.

  • Problem

    High guidance scales can improve image quality and condition alignment but are prone to mode collapse and reduced diversity, while DDIM inversion with ω > 1.0 compromises reconstruction.

    Source: §1 · p. 2

  • Method

    CFG++ reformulates text guidance as an inverse problem with a text-conditioned score-matching loss and derives a sampling strategy using decomposed diffusion sampling.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    CFG++ consistently improves text-to-image sample quality and DDIM inversion, reconstruction, and editing compared with CFG.

    Source: §1 · p. 3

  • Takeaways & Limitations

    CFG++ treats guidance as interpolation between unconditional and conditional sampling trajectories rather than CFG’s heuristic extrapolation.

    Source: §6 · p. 10

    Most experiments use latent diffusion models, although the framework is stated to hold for pixel- and latent-diffusion models.

    Source: §2 · p. 3

Abstract

from arXiv · show

Classifier-free guidance (CFG) is a fundamental tool in modern diffusion models for text-guided generation. Although effective, CFG has notable drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image editing; furthermore, high guidance scales, essential for high-quality outputs, frequently result in issues like mode collapse. Contrary to the widespread belief that these are inherent limitations of diffusion models, this paper reveals that the problems actually stem from the off-manifold phenomenon associated with CFG, rather than the diffusion models themselves. More specifically, inspired by the recent advancements of diffusion model-based inverse problem solvers (DIS), we reformulate text-guidance as an inverse problem with a text-conditioned score matching loss and develop CFG++, a novel approach that tackles the off-manifold challenges inherent in traditional CFG. CFG++ features a surprisingly simple fix to CFG, yet it offers significant improvements, including better sample quality for text-to-image generation, invertibility, smaller guidance scales, reduced mode collapse, etc. Furthermore, CFG++ enables seamless interpolation between unconditional and conditional sampling at lower guidance scales, consistently outperforming traditional CFG at all scales. Moreover, CFG++ can be easily integrated into high-order diffusion solvers and naturally extends to distilled diffusion models. Experimental results confirm that our method significantly enhances performance in text-to-image generation, DDIM inversion, editing, and solving inverse problems, suggesting a wide-ranging impact and potential applications in various fields that utilize text guidance. Project Page: https://cfgpp-diffusion.github.io/.

1 INTRODUCTION

The introduction frames classifier-free guidance as central to text-guided diffusion but notes that high guidance scales trade better alignment and image quality against diversity and inversion performance. It presents CFG++ as an inverse-problem-based correction and reports improvements across text-to-image generation and DDIM inversion.

1 INTRODUCTION

  • Classifier-free guidance uses a scale ω > 1 to emphasize conditioning by moving from the unconditional score toward the conditional score.Figure 2 describes CFG++’s fix as using ˆϵ∅(xt) instead of ˆϵω.Source: §1 · p. 2, Figure 2 · p. 3
  • High guidance scales can improve image quality and condition alignment but are prone to mode collapse, reduced diversity, and weaker DDIM inversion reconstruction.The introduction notes that DDIM inversion with ω > 1.0 significantly compromises reconstruction performance.Source: §1 · p. 2
  • CFG++ reformulates text guidance as an inverse problem with a text-conditioned score-matching loss and uses a small guidance scale, typically λ ∈[0.0, 1.0].The introduction describes the change as a simple fix without computational overhead.Source: §1 · p. 2
  • CFG++ improves text-to-image sample quality and DDIM inversion, reconstruction, and editing compared with CFG.The introduction also reports that it works with smaller guidance scales and enables interpolation between unconditional and conditional sampling.Source: §1 · p. 3

2 BACKGROUND

The background introduces diffusion sampling through reversal of a forward noising process and describes how conditional guidance and inverse-problem solvers relate to that process. It also establishes the paper’s notation and the latent- and pixel-diffusion scope.

2 BACKGROUND

  • Diffusion models learn to reverse a forward noising process, and sampling can solve either a reverse SDE or the equivalent probability-flow ODE.The forward process moves from an initial distribution toward a standard Gaussian distribution.Source: §2 · p. 3
  • The paper denotes the model’s noise prediction under condition c by ˆϵc and its prediction under null conditioning by ˆϵ∅.Conditional models allow the condition to modulate the network output through cross-attention.Source: §2 · p. 3
  • Although most experiments use latent diffusion models, the framework is stated to apply to both pixel- and latent-diffusion models.The paper uses x as notation for either representation.Source: §2 · p. 3
  • CFG sharpens the conditional posterior by scaling the difference between conditional and unconditional scores by ω.The displayed score expression adds the scaled score difference to the unconditional score.Source: §2 · p. 3
  • Figure 3 attributes CFG’s off-manifold behavior to guidance-scale extrapolation and a renoising offset, which CFG++ mitigates.The caption identifies the scale effect as deviation from a piecewise linear data manifold.Source: Figure 3 · p. 4
  • Diffusion inverse-problem solvers seek to minimize a loss over the clean data manifold, and DPS uses a denoised estimate to compute a manifold-constrained gradient.Under a linear-manifold assumption, the update can transition precisely to the preceding noisy manifold; DDS offers an equivalent representation under further assumptions.Source: §2 · p. 4

3 CFG++ : MANIFOLD-CONSTRAINED CFG

CFG++ recasts text guidance as a score-matching inverse problem and changes the sampling trajectory to address off-manifold behavior. The section derives this approach and explains its implications for text alignment and DDIM inversion.

3.1 DERIVATION OF CFG++

  • The method formulates text guidance as an inverse problem that minimizes a text-conditioned score-matching loss, using DDS to avoid Jacobian computation.Its target is a solution on the clean manifold that aligns with the text condition.Source: §3.1 · pp. 4–5
  • CFG++ differs from CFG in the renoising process: it uses unconditional noise there, producing a smoother generation trajectory and superior quality.The conditional denoised estimate is used while the renoising components remain equivalent to unconditional sampling.Source: §3.1 · p. 5
  • CFG++ yields smoother generation and superior quality, improves DDIM inversion, and has a smoother early text-conditioned loss trajectory than CFG.The reported loss comparison is for SDXL and averages 55 prompts.Source: §3.1 · p. 5, §3.2 · pp. 6–7, Figure 4 · p. 6
  • The approach extends to higher-order solvers and distilled models by conditioning the denoised estimate while retaining unconditional estimates for other noise components.The paper names Karras Euler and DPM-solver among widely used alternatives to DDIM.Source: §3.1 · p. 5

3.2 GEOMETRY OF CFG++

  • CFG++ uses λ in [0, 1] to interpolate between unconditional and conditional estimates, whereas CFG scales above 1 extrapolate and can move estimates off the clean manifold.The geometric argument assumes the clean manifold is piecewise linear.Source: §3.2 · p. 6
  • CFG++ directly minimizes text-conditioned score-matching loss; its loss trajectory is smoother early in reverse sampling, while CFG fluctuates and retains a noticeable final gap.The paper relates CFG’s fluctuations to the off-manifold issue.Source: §3.2 · p. 6, Figure 4 · p. 6
  • CFG inversion errors grow with guidance scale, while CFG++ has a smaller approximation error under the usual DDIM assumption, supporting improved inversion.The paper attributes the smaller CFG++ error to λ < ω.Source: §3.2 · p. 7

4 EXPERIMENTAL RESULTS

Across text-to-image generation, inversion and editing, and text-conditioned inverse problems, CFG++ improves on CFG in the reported evaluations. The experiments also examine accelerated and higher-order sampling settings.

4.1 TEXT-TO-IMAGE GENERATION

  • FID improves across guidance scales for SD v1.5 and SDXL, with approximately the same or better CLIP similarity.The comparison uses 10k images generated from COCO captions.Source: §4.1 · p. 7
  • CFG++ yields larger quality gains than CFG on distilled models, including SDXL-turbo and SDXL-Lightning at 6 NFE.The comparison uses the same 5k prompts; the reported gains are also reflected in Table 2.Source: Table 2 · p. 7, §4.1 · p. 8, Figure 5 · p. 8
  • With DPM++2M at 20 NFE, CFG++ results are reported as favorable to CFG, with CFG scale 5.0 corresponding to CFG++ scale 1.0.The experiments slightly extrapolate λ ≥ 1.0 to obtain stronger guidance results in this low-NFE regime.Source: §4.1 · p. 8

4.2 DIFFUSION IMAGE INVERSION AND EDITING

  • Across guidance scales, CFG++ consistently improves image reconstruction after DDIM inversion, while CFG fails to reconstruct the source image in the reported comparison.The evaluation includes PSNR and RMSE on 5k COCO-2014 validation images; the figure shows examples and quantitative comparisons.Source: §4.2 · p. 8, Figure 6 · p. 9
  • In SDXL editing at 5 NFEs, CFG++ changes the target concept while preserving scene elements, whereas CFG can lose background content and produce less faithful edits.The example retains background water splashing; CFG++ edits are described as precise and high-fidelity.Source: §4.2 · pp. 8–9, Figure 6 · p. 9

4.3 TEXT-CONDITIONED INVERSE PROBLEMS

  • On the evaluated FFHQ linear inverse problems, CFG++ mostly outperforms both vanilla PSLD and PSLD with CFG.The comparison uses 512×512 FFHQ images and the prompt “a high-quality photo of a face”; reconstructions are reported as higher quality across tasks.Source: §4.3 · p. 9

5 RELATED WORKS AND DISCUSSIONS

Prior work explored when to apply guidance, while CFG++ changes the sampling trajectory; the paper also describes an equivalent time-varying CFG scale and compares inverse-problem results.

5 RELATED WORKS AND DISCUSSIONS

  • The cited inverse-problem comparison evaluates PSLD under CFG and CFG++ qualitatively and reports a quantitative comparison across FID, LPIPS, and PSNR.The quantitative comparison is scoped to PSLD, PSLD with CFG, and PSLD with CFG++ on a latent diffusion inverse solver.Source: Table 3 · p. 10, Figure 7 · p. 10
  • Guidance only near the middle performed best in prior observations, while CFG++ instead designs a different sampling trajectory.Earlier CFG guidance was reported to reduce diversity; later-stage guidance had minimal effects.Source: §5 · pp. 9–10
  • CFG++ can be reproduced by a specific time-varying CFG scale that removes undesirable oscillations in the posterior mean.The schedule is relatively complex to draw heuristically.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents CFG++ as a geometric correction that can work at smaller guidance scales. The appendices describe adapting it to other solvers and explain how its posterior-mean shifts differ from CFG.

6 CONCLUSION

  • CFG++ uses a geometric correction so guidance with λ ∈[0.0, 1.0] might suffice, shifting from CFG’s extrapolation toward interpolation between unconditional and conditional trajectories.The conclusion presents this as more interpretable than conventional heuristic ω scaling.Source: §6 · p. 10

A EXTENSION OF CFG++ TO OTHER SOLVERS

  • The CFG++ transition rule extends beyond DDIM to Euler, Euler Ancestral, and DPM-solver++ variants.The extension section gives solver-specific update constructions, including DPM-solver++ 2M and 2S.Source: A EXTENSION OF CFG++ TO OTHER SOLVERS · pp. 14–15
  • For these extensions, CFG++ keeps the Tweedie estimates unconditional except for the conditional first term of the update.The stated transition rule changes the first term while retaining unconditional estimates for the other components.Source: A EXTENSION OF CFG++ TO OTHER SOLVERS · pp. 14–15

B EVOLUTION OF THE POSTERIOR MEAN: CFG VS. CFG++

  • CFG++ posterior-mean evolution combines an unconditional shift with a λ-scaled conditional shift, producing a small nudge toward the condition.CFG instead includes an additional negative conditional shift from the previous timestep; its large ω pushes the trajectory off-manifold before partially cancelling that effect.Source: B EVOLUTION OF THE POSTERIOR MEAN: CFG… · p. 16

C.1 TEXT-CONDITIONED INVERSE PROBLEMS

The text-conditioned inverse-problem evaluation specifies four degradation settings and uses FFHQ images with a pretrained latent diffusion model; the accompanying table lists PSLD hyperparameters for CFG++ and corresponding CFG scales.

C.1 TEXT-CONDITIONED INVERSE PROBLEMS

  • The inverse-problem settings include x8 super-resolution, motion deblurring with a 61 x 61 kernel and intensity 0.32, Gaussian deblurring with a 61 x 61 kernel and intensity 0.5, and inpainting with 10-20% free-form masks.For inpainting, regions outside the mask are overlaid with the ground-truth image.Source: §C.1 · p. 16
  • Evaluation uses the first 1,000 FFHQ 512x512 images and SD v1.5 trained on LAION, with PSLD as the latent inverse-solver baseline.The PSLD baseline combines projection onto the subspace of A with the DPS loss term.Source: §C.1 · p. 16
  • Table 4 lists hyperparameters for PSLD with CFG++ and the corresponding CFG scale ω.Source: Table 4 · p. 16

D.1 T2I

The appendix reports that CFG++ removes several visible artifacts in SDXL examples and addresses early-time off-manifold error observed under CFG; guidance scales were matched using LPIPS and the same seed.

D.1 T2I

  • CFG and CFG++ guidance values were matched by computing LPIPS distance with the same seed.Source: §D.1 · p. 16
  • CFG produces significant error in clean estimates, particularly early in sampling, while CFG++ addresses the off-manifold issue by adjusting DDIM re-noising and guidance scale.Source: §D.1 · pp. 16–17
  • CFG++ examples omit four artifacts seen under CFG: a missing lion cub, a two-tailed dog, unusually shaped goggles, and a folded tree trunk.Source: Figure 9 · p. 17

D.2 REAL IMAGE EDITING

The real-image editing comparisons report that CFG++ edits images successfully where CFG did not. Additional examples show improved T2I depictions and better reconstruction across several text-conditioned inverse-problem tasks.

D.2 REAL IMAGE EDITING

  • CFG++ successfully edits real images via DDIM inversion where CFG did not.The comparisons use “a photography of [source concept]” for inversion, then replace the source concept with a target concept; the reported scales are ω = 9.0 and λ = 0.8.Source: §D.2 · p. 17, Figure 15 · p. 22, Figure 16 · p. 22
  • CFG++ corrects unnatural hand depictions and text renderings in SD v1.5 T2I examples, and improves overall image quality with SDXL-Lightning at 6 NFE.Source: Figure 10 · p. 18, Figure 11 · p. 18

D.3 TEXT-CONDITIONED INVERSE PROBLEMS

  • On the FFHQ 512x512 validation set, CFG++ consistently leads to better reconstruction of the true solution across several text-conditioned inverse-problem tasks.The accompanying examples cover x8 super-resolution, motion deblurring, Gaussian deblurring, and inpainting.Source: §D.3 · p. 19, Figure 18 · p. 23, Figure 19 · p. 24
Loading 2406.08070v2…