PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Learning of Classifier-Free Guidance Schedules

Ashwini Pokle, Alexandre Galashov, Arnaud Doucet, Mauricio Delbracio, Valentin De Bortoli

arXiv:2608.14038v1cs.LG

TL;DR

Static CFG schedules cannot adapt guidance to different timesteps, prompts, or noisy states. The paper learns dynamic schedules adversarially and outperforms constant, heuristic, and prior learned guidance on text-alignment and human-preference metrics.

  • Problem

    Static guidance weights and manually designed schedules do not adapt to timestep, conditioning, or noisy-sample differences and can introduce artifacts or tuning difficulties.

  • Method

    An adversarial density-ratio framework learns guidance scales conditioned on diffusion time, text, and the current noisy sample by matching guided and true marginal distributions.

  • Results

    The method outperforms constant, heuristic, and prior learned schedules on text-alignment and human-preference metrics across text-to-image benchmarks, including CLIP 0.3068.

  • Takeaways & Limitations

    Dynamic, conditioning-dependent guidance improves the trade-off between image realism and adherence to complex text prompts on the evaluated benchmarks.

  • Takeaways & Limitations

    The learned scheduler is backbone-specific and requires retraining for a different architecture or velocity field, while adversarial training is complex.

Abstract

from arXiv · show

Modern text-to-image diffusion models rely on classifier-free guidance (CFG) to achieve high image fidelity and text alignment. However, CFG typically applies a static, global scale across all timesteps, samples, and conditions -- a choice that is generally suboptimal and can introduce artifacts, as different states may benefit from different levels of guidance. While time-varying schedules are known to improve quality, designing them by hand is non-trivial and application-dependent. In this paper, we learn the guidance schedule as a function of diffusion time, conditioning and the current noisy sample, in order to better align sampled images with the text prompt. We frame this as a density ratio estimation problem: a discriminator is trained to estimate the time-dependent log-density ratio between the true and guided marginal distributions, while a lightweight generator network predicts the optimal, state-dependent guidance scale. Empirically, our approach outperforms both heuristic CFG schedules and prior methods for learning dynamic guidance on text-to-image generation benchmarks.

1 INTRODUCTION

The paper argues that constant classifier-free guidance can cause artifacts and that manually designed schedules are difficult to tune. It therefore learns dynamic guidance scales adversarially by matching guided and true marginal distributions at every diffusion timestep.

  • Motivation: Modern text-to-image applications commonly use a constant guidance weight of ω ≈7.5, despite guidance being defined as a conditional–unconditional velocity combination.ω = 0 gives conditional sampling, while ω = −1 gives unconditional sampling.
  • Motivation: Constant guidance can produce artifacts and over-saturation, while manually designed schedules introduce difficult-to-tune hyperparameters and typically ignore conditioning-specific needs.The passage contrasts these limitations with dynamic schedules intended to mitigate such issues.
  • Approach: The method targets marginal consistency by matching guided trajectories’ marginal distributions to the true data distribution at every timestep.This shifts focus from strict self-consistency to a weaker marginal consistency condition.
  • Approach: An adversarial framework estimates the log-density ratio between real and classifier-free guided samples while learning guidance scales that maximize this ratio.The discriminator estimates the ratio, and the generator predicts guidance scales.
  • Contributions: The learned schedules depend on diffusion time, conditioning, and the current noisy sample, and are designed to match true and guided sample distributions along the sampling trajectory.These dependencies are presented as the paper’s central schedule-learning contribution.

2 BACKGROUND

The background introduces flow matching as a continuous-time path from Gaussian noise to conditional data, then describes CFG and the motivation for state-dependent guidance schedules. It also reviews learnable schedules through marginal consistency and limitations of prior MMD-based approaches.

  • Flow matching: Flow matching connects p0 = N(0, Id) to pdata at t = 1 through the interpolation xt = (1 − t)ε + t x1.Here, ε ∼ N(0, Id) and x1 ∼ p1|c.
  • Classifier-Free Guidance (CFG): CFG replaces the conditional velocity with a weighted combination of conditional and unconditional velocities, where ω > 0 amplifies conditioning.The standard T2I guidance weight is typically ω ≈ 7.5, while constant guidance can cause over-saturation artifacts.
  • Dynamic guidance schedules: Dynamic guidance lets ω vary with time, but heuristic schedules still apply the same trajectory-wide schedule across all samples and conditionings.Examples include LIG, CLG, and time-dependent rescaling.
  • Learnable guidance schedules and marginal consistency: Learnable schedules parameterize ω as a function of conditioning and source-target timesteps, while marginal consistency requires guided marginals to approximate true marginals.A prior MMD-based self-consistency method improved FID but did not consistently outperform constant guidance on text-alignment metrics and suffered high-variance gradients.

3 LEARNING TO GUIDE WITH DENSITY RATIO ESTIMATION

The method learns state-dependent CFG weights with an adversarial density-ratio objective that matches guided and true marginal distributions. A discriminator estimates the log-density ratio while a lightweight guidance network predicts non-negative weights from diffusion state, time, conditioning, and trajectory information.

  • State-dependent guidance: Guidance weights depend on times (s, t), conditioning c, and current noisy sample x_s rather than using one static scalar across inputs and timesteps.This parametrization is intended to adapt guidance to specific noise trajectories.
  • Adversarial objective: The adversarial objective trains guided particles to match the true data marginal by minimizing their KL divergence through discriminator-based density-ratio estimation.The discriminator distinguishes true marginal samples from guided samples, and its optimal logit recovers the log-density ratio.
  • Network parametrization: The guidance network is a lightweight MLP that predicts non-negative weights, while summary statistics replace raw high-dimensional latents to reduce computational inefficiency.The discriminator is a convolutional network; the guidance MLP uses features summarizing state, conditioning, and sampling trajectory relationships.
  • Training strategy: Regularization stabilizes learning by penalizing excessive guidance and adding practitioner-defined rewards, while alternating discriminator and generator updates uses TTUR for convergence.The generator combines adversarial, time-dependent L2, and reward losses; the reward can be a CLIP score for text-to-image models.
  • Training strategy: Independent image-conditioning pairs define real and fake marginals, preventing the discriminator from identifying a single trajectory and enforcing global marginal consistency at each timestep.One pair generates the real target and another defines the fake guided sample.

4 RELATED WORK

The paper situates its method among CFG analyses, heuristic dynamic schedules, learned guidance methods, discriminator guidance, and adversarial approaches in generative modeling. Its key distinctions are learning a dedicated lightweight scheduler, using the discriminator only during training, and enforcing weaker marginal consistency than MMD guidance.

  • Classifier-free guidance and Variants: CFG is standard for conditional sampling, but analyses suggest it may drive trajectories toward class archetypes rather than the intended tilted distribution.Chidambaram et al. (2024) analyze this potential failure mode.
  • Classifier-free guidance and Variants: Heuristic dynamic schedules vary guidance over diffusion time, including limited-interval guidance, clamp-linear scheduling, beta-density modulation, rectified guidance, and geo-guide.The paper adapts the clamp-linear schedule as a baseline.
  • Learned guidance schedules: Unlike Felix et al. (2025), the method learns a dedicated guidance scheduler rather than using internal model predictions as inference-time feedback.It is closely related to MMD guidance but imposes weaker marginal consistency instead of strict self-consistency in an adversarial framework.
  • Discriminator guidance: Unlike discriminator guidance, which supplies auxiliary score gradients during inference, this method uses the discriminator only as a training supervisor for a lightweight MLP scheduler.The stated design aims to improve prompt alignment while maintaining sampling efficiency.
  • Adversarial training in Generative Modeling: The method also draws on adversarial training principles from GAN-based generative modeling and their integration with diffusion models.The supplied passage identifies Diffusion GAN and Denoising Diffusion GAN as examples.

5 EXPERIMENTS

Experiments on MS-COCO 512×512 evaluate the learned guidance schedule across two MMDiT model sizes, multiple guidance baselines, and fidelity, alignment, aesthetic, and human-preference metrics. The adversarial marginal-consistency method generally improves alignment and preference scores, with prompt-dependent guidance and robustness explored through ablations.

  • Datasets and Models: Experiments use frozen 740M MMDiT-XS and 1.05B MMDiT-S flow-matching models on MS-COCO 2014 at 512×512 resolution.The guidance network is trained while the underlying model remains frozen.
  • Baselines: Comparisons include unguided sampling, constant CFG at ω = 7.5, manually designed schedules, and learned self-consistency and marginal-consistency methods.The evaluation covers fixed, heuristic, and learned guidance strategies.
  • Evaluation Metrics: The evaluation reports FID, CLIP score, LAION Aesthetics, and Human Preference Score to capture fidelity, text alignment, visual quality, and human preference.FID measures distributional distance, while CLIP measures alignment with input prompts.
  • Results: GAN + MC consistently improves alignment and human-preference metrics across both model sizes, outperforming constant and manually tuned guidance in most such metrics.The reported gains include superior CLIP, Aesthetic, and HPSv2 scores compared with the cited baselines.
  • Results: 0.3068 CLIP, 5.34 Aesthetic, and 0.2856 HPSv2 are achieved by GAN + MC on MMDiT-S, while MMDiT-XS reaches 0.305 CLIP and 5.32 Aesthetic.LIG attains the lowest MMDiT-XS FID at 25.64 but with lower alignment; the proposed method incurs a marginal FID increase.
  • Qualitative Results and Ablations: Qualitative results show more realistic, better text-aligned images, while learned guidance weights vary substantially with the prompt.Ablations find that adding x_t improves performance but worsens FID, and performance is robust to δ choices.

6 DISCUSSION AND LIMITATIONS … B.1 EXTENSIVE COMPARISONS TO BASELINES

The paper presents an adversarial method for learning dynamic guidance schedules, achieving stronger human-preference and text-alignment results than fixed, heuristic, and prior learned schedules. It also discusses training complexity, backbone specificity, related work, and supplementary baseline comparisons and ablations.

  • 6 DISCUSSION AND LIMITATIONS: The method learns dynamic guidance by enforcing marginal consistency between guided and true data distributions along the diffusion sampling trajectory.A lightweight MLP is trained with an objective matching marginal densities throughout the trajectory.
  • 6 DISCUSSION AND LIMITATIONS: Empirically, the method outperforms constant CFG, heuristic dynamic schedules, and prior learned guidance approaches on human preference and text alignment.The paper reports negligible computational overhead during inference.
  • 6 DISCUSSION AND LIMITATIONS: Training is complex because adversarial optimization over a diffusion trajectory requires carefully chosen objectives and strategies such as Two-Time-Scale Update Rule (TTUR).The limitation concerns training procedure complexity rather than inference cost.
  • 6 DISCUSSION AND LIMITATIONS: The learned guidance scheduler is backbone-specific because it relies on local geometric statistics from a particular pretrained flow-matching model, such as MMDiT.The supplied passage indicates that the scheduler cannot be transferred across backbones.
  • ORGANIZATION OF THE APPENDIX: The appendix’s additional-results section provides further baseline comparisons and hyperparameter ablations, including analyses of the time distribution p(s, t) and related settings.These materials are organized as supplementary results beyond the main paper.
  • A EXTENDED RELATED WORK: Related work situates the approach among general guidance methods that use external differentiable models or constraints without retraining the diffusion model.Examples include CLIP feature extractors, segmentation networks, and object-detection networks.
  • A EXTENDED RELATED WORK: The approach is conceptually related to adversarial diffusion distillation methods that use discriminators to match generated and real marginal distributions.The cited examples include DMD, DMDv2, and LADD, contrasted with trajectory-based distillation methods.

B.2 TEXT-TO-IMAGE GENERATION BASELINE ABLATIONS

This section evaluates baseline guidance schedules for text-to-image generation on the MMDiT-XS model. It reports limited interval guidance across a broad hyperparameter sweep and clap-linear guidance results.

  • Limited interval guidance: The limited interval guidance baseline is tested over a large sweep of guidance weights and intervals.These experiments examine how the baseline performs across different guidance-weight and interval settings.
  • Clap-linear guidance: Table 8 reports results for the clap-linear guidance schedule.The schedule is attributed to Wang et al. (2024).
  • Limited interval guidance: Table 7 evaluates limited interval guidance on MMDiT-XS using 128 sampling steps.The evaluation uses MS COCO at 512 × 512 resolution.

B.3 ABLATION ON TIME DISTRIBUTION p(s, t) FROM SECTION 3.2 · B.4 IMPACT OF NUMBER OF SAMPLING STEPS

The method is robust to the hyperparameters of the time distribution p(s, t), with CLIP score largely unchanged. Increasing sampling steps improves both perceptual quality and prompt alignment.

  • B.3 ABLATION ON TIME DISTRIBUTION p(s, t) FROM SECTION 3.2: The ablation varies hyperparameters of the time distribution p(s, t) to assess their effect on performance.The experiments are reported in Table 9.
  • B.3 ABLATION ON TIME DISTRIBUTION p(s, t) FROM SECTION 3.2: Overall performance remains quite robust to the tested p(s, t) hyperparameters.This robustness is assessed using the MMDiT-XS model and the metrics reported in Table 9.
  • B.3 ABLATION ON TIME DISTRIBUTION p(s, t) FROM SECTION 3.2: The CLIP score does not seem to be affected much by the tested time-distribution parameters.The ablation also varies the CLIP reward scale η.
  • B.3 ABLATION ON TIME DISTRIBUTION p(s, t) FROM SECTION 3.2: Table 9 reports performance metrics for different δ values in p(s, t) and different CLIP reward scales η.These settings constitute the reported time-distribution ablation.
  • B.4 IMPACT OF NUMBER OF SAMPLING STEPS: The number of sampling steps is evaluated for the method using an MMDiT-S backbone.The results are presented in Table 10.
  • B.4 IMPACT OF NUMBER OF SAMPLING STEPS: More sampling steps lead overall to better perceptual metrics.This trend is reported for the GAN-based guidance scheduler.
  • B.4 IMPACT OF NUMBER OF SAMPLING STEPS: More sampling steps also lead overall to better prompt alignment.Prompt alignment is assessed alongside perceptual metrics in the sampling-step study.

B.5 IMPACT OF GENERATOR ARCHITECTURE · B.6 IMPACT OF HOW THE NOISE IS SAMPLED DURING TRAINING

The method favors a lightweight statistics-based MLP generator over a more complex convolutional generator, while independent target and proposal noise gives minor quality gains. These results also indicate robustness to the noise-coupling choice.

  • B.5 IMPACT OF GENERATOR ARCHITECTURE: A lightweight MLP generator receiving statistics of x_s performs better overall than a convolutional network receiving raw latent x_s.This comparison is reported for the MMDiT-XS backbone.
  • B.5 IMPACT OF GENERATOR ARCHITECTURE: The architecture ablation compares a lightweight statistics-based MLP against a more complex convolutional network using raw latents.The experiment varies both generator architecture and input representation.
  • B.5 IMPACT OF GENERATOR ARCHITECTURE: The reported architecture results support using compact summary statistics rather than raw latent inputs for the generator.The paragraph attributes the overall advantage to the lightweight MLP receiving x_s statistics.
  • B.6 IMPACT OF HOW THE NOISE IS SAMPLED DURING TRAINING: Algorithm 1 constructs the real target x_real and proposal base x_s from Gaussian noise that may be coupled or independent.The two choices are ϵ = z and ϵ ≠ z, respectively.
  • B.6 IMPACT OF HOW THE NOISE IS SAMPLED DURING TRAINING: Independent target and proposal noise produces minor improvements in downstream generation quality.The evaluation uses an MMDiT-XS backbone and reports examples including lower FID and improved HPSv2.
  • B.6 IMPACT OF HOW THE NOISE IS SAMPLED DURING TRAINING: The noise-sampling ablation indicates that the marginal consistency objective remains robust to the underlying noise coupling.The conclusion follows the comparison of coupled and independent Gaussian noise vectors.

B.7 IMPACT OF HOW IMAGE-CONDITIONING PAIRS ARE SAMPLED · B.8 IMPACT OF CLIP REWARD LOSS · B.9 ALTERNATIVE CHOICES FOR THE DISTRIBUTION p(s, t)

The method is robust to image-conditioning pair sampling, while CLIP reward loss provides marginal gains in CLIP and human-preference metrics. Alternative time-pair distributions are mathematically characterized and show that α = 1, β = 2 performs best, comparably to the original distribution.

  • B.7 IMPACT OF HOW IMAGE-CONDITIONING PAIRS ARE SAMPLED: Independent and correlated image-conditioning pair sampling yield nearly identical performance across most metrics.Independent sampling is theoretically more aligned with marginal consistency, but its observed advantage is limited.
  • B.7 IMPACT OF HOW IMAGE-CONDITIONING PAIRS ARE SAMPLED: 0.2927 vs. 0.2829: independent sampling marginally improves HPSv2 over correlated sampling.The comparison is reported for the HPSv2 metric in the image-conditioning pair ablation.
  • B.8 IMPACT OF CLIP REWARD LOSS: Removing the CLIP reward loss causes marginal drops in CLIP score and human-preference metrics for the MMDiT-XS model.The ablation evaluates the method without CLIP reward loss Equation (19).
  • B.9 ALTERNATIVE CHOICES FOR THE DISTRIBUTION p(s, t): The original p(s, t) samples t uniformly and then samples step size ∆s uniformly conditional on t, but this does not produce a uniform joint distribution over the valid region T.Valid pairs satisfy ζ + δ ≤ t ≤ 1 − ζ and ζ ≤ s ≤ t − δ.
  • B.9 ALTERNATIVE CHOICES FOR THE DISTRIBUTION p(s, t): The uniform-joint sampler draws u ∼ U[0, 1], sets t = ζ + δ + (1 − 2ζ − δ)√u, and then samples s uniformly conditional on t.The resulting joint density is constant over T.
  • B.9 ALTERNATIVE CHOICES FOR THE DISTRIBUTION p(s, t): Uniform joint sampling over T requires s | t ∼ U[ζ, t − δ] and sampling t with probability proportional to the valid strip width w(t) = t − δ − ζ.This makes p(s, t) = 1/|T| constant over T.
  • B.9 ALTERNATIVE CHOICES FOR THE DISTRIBUTION p(s, t): Among Beta-trapezoid alternatives, β < α worsens results, whereas α ≥ β performs better; α = 1, β = 2 is best and comparable to the original p(s, t).When α = β = 1, the Beta construction recovers uniform sampling over T.

C EXPERIMENTAL DETAILS

The experiments use a DCGAN-inspired convolutional discriminator and an MLP guidance network, with both conditioned on diffusion-time and text embeddings. Training runs for 60000 iterations, selects checkpoints by CLIP score, and evaluates baselines under matched protocols with specified architectural variations.

  • Discriminator architecture: The discriminator uses five Conv-Residual-AdaGN-Leaky-ReLU downsampling modules with 4 × 4, stride-2 convolutions in its first three stages.It is inspired by the DCGAN architecture and progressively reduces spatial resolution while doubling feature depth.
  • Discriminator architecture: The discriminator combines separately encoded source and target times with projected CLIP text embeddings through a joint temporal representation.Each time scalar is mapped to 64 dimensions using a separate MLP-based encoder before the temporal embeddings are combined.
  • Guidance network architecture: The guidance network is an MLP that embeds source time, target time, and their gap using 256-dimensional Fourier embeddings, alongside projected CLIP text features.The embeddings are concatenated and processed with linear layers, Layer Normalization, and SiLU activation.
  • Guidance network architecture: The guidance output is a non-negative scalar produced by a softplus activation, with zero-initialized final weights and bias set to softplus−1(α).This initialization starts training from a constant schedule with the starting guidance multiplier α, such as 1.0.
  • Training details: 60000 iterations are used for all models, with CLIP score on 3000 MS-COCO 512 × 512 images selecting the best checkpoint.The same training protocol is generally used for MMD+SC and MMD+MC, except they omit the discriminator and regularization equation 17; variants also omit x_s from the guidance input.

D IMAGE SAMPLES

The section presents qualitative text-to-image samples comparing the proposed method with established guidance methods and showing its variation across random seeds. Results are shown on MS-COCO using MMDiT-XS and MMDiT-S.

  • Figure 6: Figure 6 compares images generated from the same text prompt using CFG ω = 7.5, CLG, LIG, MMD, and the proposed method.The comparison uses MS-COCO with MMDiT-XS.
  • Figure 7: Figure 7 shows the proposed method generating images from the same text prompt with seeds 0, 1, 2, and 3.The seed variation results use MS-COCO with MMDiT-S.
Loading 2608.14038v1…