PaperlayerView exact revision on arXiv

Source-linked AI summary

Stage-wise Dynamics of Classifier-Free Guidance in Diffusion Models

Cheng Jin, Qitan Shi, Yuantao Gu

arXiv:2509.22007v2cs.LG

TL;DR

CFG’s effect on sampling dynamics is poorly understood because prior analyses often assume unimodal conditionals or impose weak conditions. This paper analyzes CFG with multimodal Gaussian-mixture conditionals, identifies three sampling stages, and finds that stage-specific guidance explains diversity loss while motivating schedules that improve the quality–diversity trade-off.

  • Problem

    Prior CFG analyses provide only partial understanding because they often assume unimodal conditionals or weak conditions, leaving guidance-induced diversity loss unexplained.

  • Method

    The paper models conditional distributions as Gaussian mixtures and analyzes CFG through three stages: Direction Shift, Mode Separation, and Concentration.

  • Results

    The theory and experiments show that strong early guidance suppresses weaker modes, while strong late guidance intensifies within-mode contraction and reduces fine-grained variability.

  • Takeaways & Limitations

    A time-varying guidance schedule that reduces guidance early and late while emphasizing the middle stage can improve the quality–diversity trade-off.

  • Takeaways & Limitations

    The analysis uses a simplified yet representative setting grounded in stated distributional and noise-schedule assumptions.

Abstract

from arXiv · show

Classifier-Free Guidance (CFG) is widely used to improve conditional fidelity in diffusion models, but its impact on sampling dynamics remains poorly understood. Prior studies, often restricted to unimodal conditional distributions or simplified cases, provide only a partial picture. We analyze CFG under multimodal conditionals and show that the sampling process unfolds in three successive stages. In the Direction Shift stage, guidance accelerates movement toward the weighted mean, introducing initialization bias and norm growth. In the Mode Separation stage, local dynamics remain largely neutral, but the inherited bias suppresses weaker modes, reducing global diversity. In the Concentration stage, guidance amplifies within-mode contraction, diminishing fine-grained variability. This unified view explains a widely observed phenomenon: stronger guidance improves semantic alignment but inevitably reduces diversity. Experiments support these predictions, showing that early strong guidance erodes global diversity, while late strong guidance suppresses fine-grained variation. Moreover, our theory naturally suggests a time-varying guidance schedule, and empirical results confirm that it consistently improves both quality and diversity.

1 INTRODUCTION

The paper addresses why CFG’s strong conditional alignment often comes with diversity loss, extending analysis beyond unimodal assumptions to multimodal conditionals. It identifies three sampling stages and validates that early and late guidance affect global and fine-grained diversity differently.

  • Motivation: CFG theory has largely relied on unimodal conditionals or weak assumptions, leaving diversity collapse under large guidance weights theoretically unexplained.The paper frames this gap as important for improving conditional generation.
  • Framework: Modeling conditional distributions as Gaussian mixtures reveals a three-stage structure in guided sampling.The stages are Direction Shift, Mode Separation, and Concentration.
  • Three-stage dynamics: In Direction Shift, strong guidance attracts trajectories toward the class-weighted mean, causing initialization bias and norm inflation.
  • Three-stage dynamics: In Mode Separation, local attraction geometry remains largely unchanged, but inherited early bias suppresses weaker modes and reduces global diversity.
  • Three-stage dynamics: In Concentration, strong guidance intensifies within-mode contraction, suppressing local variability and fine-grained diversity.
  • Empirical validation: Experiments on state-of-the-art diffusion models confirm that strong early guidance reduces global diversity, while strong late guidance diminishes fine-grained variation.A time-varying guidance schedule further improves the quality–diversity trade-off.

2 PRELIMINARY

The preliminary section introduces diffusion sampling through forward and reverse stochastic dynamics, then adopts the deterministic probability flow ODE for analysis. It defines CFG as extrapolation between unconditional and conditional scores to strengthen alignment with conditioning information.

  • 2.1 DIFFUSION MODEL: The forward diffusion process progressively transforms data into an isotropic Gaussian, allowing sampling to begin from a tractable prior.The schedule functions control drift and diffusion, while the process converges toward Gaussian noise.
  • 2.1 DIFFUSION MODEL: Time-reversal theory expresses sampling through a reverse SDE driven by the score function, the gradient of the log-density.
  • 2.1 DIFFUSION MODEL: The probability flow ODE removes sampling randomness while preserving the same marginal distributions, enabling efficient high-order solvers and tractable analysis.
  • 2.1 DIFFUSION MODEL: Conditional generation replaces the unconditional score with a score conditioned on external information such as a class label or text prompt.This retains the computational benefits of the ODE formulation but may suffer from weak semantic alignment when the learned conditional score is under-confident.
  • 2.2 CLASSIFIER-FREE GUIDANCE: CFG extrapolates between unconditional and conditional scores using guidance scale ω, with larger values enforcing stronger alignment with the conditioning signal.At ω = 1, CFG reduces to the plain conditional model, while larger values no longer correspond to a valid probabilistic model.
  • 2.2 CLASSIFIER-FREE GUIDANCE: Substituting the guided score into probability flow dynamics yields the CFG probability flow ODE used for subsequent analysis.

3 STAGE-WISE BEHAVIOR OF CLASSIFIER FREE GUIDANCE IN SAMPLING

Under multimodal conditional distributions, CFG sampling unfolds through three stages: early guidance shifts trajectories toward a weighted mean, inherited bias suppresses weaker modes, and late guidance contracts samples within modes. This explains how CFG can improve alignment and sharpness while reducing both global coverage and fine-grained diversity.

  • 3.1 FIRST STAGE: ACCELERATION AND DIRECTION SHIFT: CFG accelerates early trajectories toward the class-weighted mean, inflates their norms, and seeds initialization bias that favors dominant modes later.In the high-noise regime, guidance amplifies global statistics while fine-grained multimodal structure remains suppressed.
  • 3.2 SECOND STAGE: INTRA-CLASS MODE SEPARATION: During mode separation, CFG preserves attraction-basin geometry and weaker modes theoretically, but inherited early bias makes weaker basins rarely reached in practice.The effective loss of global diversity therefore results from reduced basin occupancy rather than intrinsic second-stage destruction of weaker modes.
  • 3.3 THIRD STAGE: CONCENTRATION: Late-stage CFG strengthens contraction toward local mode centers, reducing within-mode dispersion and suppressing fine-grained variability.This contraction can make samples appear sharper and more semantically aligned while removing pose, texture, and stylistic variation.
  • 3.4 SUMMARY ACROSS STAGES: The three stages leave sequential effects on the final sample distribution rather than producing an isolated change at one sampling time.Early displacement shapes mode access, while late local contraction shapes variation within the selected mode.
  • 3.4 SUMMARY ACROSS STAGES: Together, CFG improves conditional fidelity and visual sharpness but diminishes both global coverage through weaker-mode loss and local diversity through within-mode contraction.The framework connects these two diversity losses to distinct early and late sampling dynamics.

4 EXPERIMENTS

Experiments validate that guidance timing controls distinct forms of diversity loss and show that low-high-low schedules improve the quality–diversity trade-off. Across visual comparisons and metrics, weakening guidance early and late while strengthening it mid-sampling preserves variability and reduces over-saturation.

  • 4.1 VALIDATION OF THEORY.: Strong early guidance erodes global diversity by inducing mean-shift bias that suppresses weaker modes and reduces multimodal coverage.
  • 4.1 VALIDATION OF THEORY.: Strong late guidance amplifies within-mode contraction, making local structures nearly identical and suppressing fine-grained diversity.
  • 4.2 METHODOLOGICAL IMPLICATIONS.: Low-high-low schedules strengthen guidance in the middle while weakening it early and late, translating the stage-wise theory into practical sampling designs.
  • 4.2 METHODOLOGICAL IMPLICATIONS.: The proposed schedule preserves diversity and reduces over-saturation, while all three methods following the design principle improve diversity without sacrificing good generation quality.
  • 4.2 METHODOLOGICAL IMPLICATIONS.: At low NFE budgets, vanilla CFG nearly collapses with poor IR and FID, whereas theory-guided schedules are especially effective in low- and medium-NFE regimes.

5 CONCLUSION

The conclusion frames the paper as a multimodal analysis of CFG that explains diversity loss through three sampling stages and motivates stage-aware guidance schedules. Related work is largely limited by unimodal or simplified assumptions, while the proof appendix derives the posterior expressions supporting the analysis.

  • 5 CONCLUSION: The paper characterizes CFG under multimodal conditional distributions through three stages: early Direction Shift, mid Mode Separation, and late Concentration.
  • 5 CONCLUSION: The stage-wise theory explains diversity loss through early mean-shift bias that suppresses weaker modes and late contraction that reduces intra-class variability.
  • 5 CONCLUSION: Reducing guidance early and late while emphasizing the middle offers a simple way to mitigate diversity loss.
  • B RELATED WORK: Prior work reports effects including edge concentration, deviations from gamma-weighted outputs, norm inflation, anomalous diffusion, and non-uniform classification-confidence improvements.
  • B RELATED WORK: Existing CFG analyses primarily study unimodal or simplified distributions, limiting their ability to explain diversity loss under large guidance weights.
  • C PROOF OF THEOREM AND PROPOSITION: The appendix establishes posterior-mean identities for isotropic Gaussian and Gaussian-mixture settings using Tweedie’s formula and posterior computations.

C.1 PROOF OF THEOREM3.2

The proof analyzes early-time CFG dynamics for Gaussian-mixture conditionals by comparing guided and unguided probability-flow trajectories from shared noise. Early approximations show that CFG changes the drift toward the scaled class-weighted mean and produces a strict trajectory-gap inequality.

  • Setup: The reverse probability-flow ODE and its CFG variant are driven by estimators obtained for a K-component Gaussian mixture with shared covariance.
  • Early-time expansions: As t approaches 1, the conditional estimator converges uniformly on compact sets to the class-weighted mean ¯µ, while the unconditional estimator converges to zero.The approximations have O(1 −t) remainder control on bounded sets.
  • Early-time comparison: For any guidance weight ω > 1, an early interval exists in which the CFG trajectory is, in expectation, closer to ω ¯µ than the unguided trajectory.The comparison uses shared random initialization and the Lyapunov functional V(x) = ∥x −ω ¯µ∥2.
  • Derivative argument: Using bounded second moments and continuity, the derivative comparison is extended from the limiting time t = 1 to a full interval [te1, 1).The proof justifies differentiation under the expectation through dominated convergence.
  • Conclusion: Because the trajectories share the same initial condition, the gap vanishes at t = 1, and integrating the strict derivative inequality backward yields the desired expected-distance comparison.
  • Scope: The argument extends beyond the stated conditional model when the conditional and unconditional estimators have the same early-time limits and bounded second moments hold.

C.2 PROOF OF THEOREM3.3

The proof establishes persistence of the weaker mode under CFG in a two-component Gaussian mixture. A separating hyperplane has a CFG field pointing toward the weaker mode while moving toward the stronger mode in backward time, yielding a backward-invariant weaker-mode region.

  • Theorem statement: Under mild assumptions, an ω-independent region exists such that trajectories starting inside it remain aligned with the weaker mode for every guidance weight ω ≥ 1.This is the theorem’s basin-of-attraction conclusion.
  • Geometric construction: The separating hyperplane divides the two modes, and its weaker-mode side has higher posterior responsibility for the weaker component.
  • Invariance: Combining the field direction and hyperplane motion, trajectories initially on the weaker-mode side cannot cross into the stronger-mode side.The proof invokes backward invariance through a moving-set viability criterion.
  • CFG direction: On the separating hyperplane, the CFG vector field points strictly toward the weaker mode in backward time.
  • Hyperplane motion: As time decreases, the separating hyperplane moves toward the stronger mode, preserving the weaker-mode side under the relevant dynamics.
  • Assumption: The construction requires a mild condition ensuring that the separating set is nonempty, which holds in a specified small-noise, small-time regime.

C.3 PROOF OF PROPOSITION3.4

The proposition characterizes a forward-invariant stronger-mode region around a scaled class-weighted mean. Its radius increases with the scaling factor, quantifying how larger early displacements favor the stronger mode.

  • Invariant region: For any k > 1, a radius r(k) exists such that trajectories initialized in B_r(k)(k ¯µ) remain in the stronger-mode half-space under CFG.
  • Scaling effect: The radius r(k) grows monotonically with k, so larger scaling produces a larger certified region dominated by the stronger component.The proof states that r(k) grows linearly in k.
  • Proof mechanism: A separating hyperplane orthogonal to ∆µ has a vector field pointing toward the stronger mode and divides space according to posterior dominance.

C.4 PROOF OF THEOREM 3.5

The proof compares CFG and conditional flows within a well-separated Gaussian-mixture mode. Under small component variance, the CFG flow contracts within-mode pairwise distances more strongly than the conditional flow.

  • Theorem statement: For any guidance weight ω > 1, sufficiently separated small-variance mixture modes admit a region where CFG produces stronger within-mode contraction than conditional sampling.The comparison applies to two solutions initialized together inside a mode-specific ball.
  • Technical conditions: The proof bounds posterior-gradient contributions uniformly by taking the mode separation sufficiently large and the local region and time interval sufficiently small.
  • Single-mode reduction: When trajectories are governed by the same mode, the conditional and unconditional estimators have affine forms that enable a direct comparison of their pairwise dynamics.
  • Contraction comparison: The stronger conditional prior leaves a residual negative term in the CFG difference dynamics, yielding a strictly larger contraction gap under the stated small-variance conditions.
  • Conclusion: Integrating the positive distance-gap derivative backward from the shared initialization establishes the pointwise distance inequality between the two flows.

D DETAILMENT OF EXPERIMENT

Experiments compare constant, interval, β-based, and time-varying guidance schedules under standardized diffusion-model evaluation settings.

  • Schedule settings: Early-high and late-high guidance experiments use N = 50 NFEs with low and high weights of 3 and 9, while the high-guidance phase begins after or before 20% of iterations, respectively.
  • Schedule settings: Late-stage tests use identical noise initialization plus a perturbation N(0, 0.042) at 20% of iterations, with late-high guidance set to 3, then 1, then 5 across successive intervals.
  • Evaluation setup: Experiments use Stable Diffusion v3.5 on 5,000 COCO validation captions, generating 5,000 images per configuration at 1024×1024 and evaluating FID, CLIP, ImageReward, and saturation.All runs use NVIDIA A100-SXM4-40GB GPUs with PyTorch and Hugging Face Diffusers.
  • Guidance schedules: The comparison includes vanilla-CFG with constant guidance, TVCFG with weaker early and late guidance, β-CFG with a Beta-distribution schedule, and interval-CFG applied only within a chosen interval.
  • Diversity metric: Diversity is measured from 16 images per prompt using mean squared pairwise distances between LPIPS features, where larger values indicate greater diversity.

E ABLATION STUDY ON PEAK TIMING

The peak-timing ablation finds that low→high→low schedules are robust when their peak lies in the intermediate regime, while extreme timing degrades selected outcomes.

  • Ablation design: The ablation varies peak positions at 20%, 40%, 60%, and 80% with NFE = 10 and ω = 7, evaluating CLIP, ImageReward, FID, saturation, and Diversity.
  • Peak-timing results: CLIP saturates at its highest value of 0.319, and ImageReward peaks at 40% when the guidance peak lies within the intermediate 40%–60% regime.Performance remains stable and optimal across most metrics in this interval.
  • Extreme timings: An 80% peak produces the lowest Diversity, 1.1622, and the lowest Saturation, consistent with delayed guidance restricting the available generation space.
  • Extreme timings: A 20% peak lowers ImageReward and raises FID, indicating that applying strong guidance too soon harms sample quality.
  • Interpretation: The successful schedule depends primarily on the low→high→low shape, remaining robust to exact peak location when the peak is within the mode-separation stage.

F ADDITIONAL VISULIZATION RESULTS

Additional visualizations illustrate that strong early or constant guidance collapses global diversity, whereas time-varying guidance is presented as an alternative schedule for generated samples.

  • Generated samples: Figures 8–11 present generated samples from CFG, TV-CFG, APG, and TV-APG under the reported visualization settings.The accompanying text specifies NFE = 20 and ω = 9 for these schedule visualizations.
  • Guidance-schedule comparisons: Figures 6 and 7 show that constant high and early high guidance produce directional collapse or nearly identical settings across samples for different prompts.
Loading 2509.22007v2…