PaperlayerView exact revision on arXiv

Source-linked AI summary

CFG-Ctrl: Control-Based Classifier-Free Diffusion Guidance

Hanyang Wang, Yiyang Liu, Jiawei Chi, Fangfu Liu, Ran Xue, Yueqi Duan

arXiv:2603.03281v2cs.CVcs.LG

TL;DR

Existing CFG guidance relies on linear feedback that can become unstable and distort semantic generation at high guidance scales. CFG-Ctrl reinterprets guidance as control of the flow and proposes SMC-CFG with switching feedback, a sliding manifold, and Lyapunov analysis. Across three text-to-image models, SMC-CFG improves semantic alignment and visual fidelity while remaining robust across guidance scales, though it adds tuning complexity.

  • Problem

    Linear CFG-based guidance can distort generative trajectories and become unstable or semantically inconsistent under highly nonlinear dynamics and large guidance scales.

  • Method

    CFG-Ctrl models flow-matching sampling as a controlled dynamical system, while SMC-CFG uses a sliding mode surface, switching control, and Lyapunov analysis.

  • Results

    SMC-CFG consistently improves semantic alignment and visual fidelity while maintaining robustness across diverse guidance scales and text-to-image models.

  • Takeaways & Limitations

    The control-theoretic formulation offers a framework for designing more effective and robust guidance for large-scale generative models.

  • Takeaways & Limitations

    SMC-CFG introduces two additional hyperparameters, increasing deployment complexity and potentially requiring manual tuning for different models.

Abstract

from arXiv · show

Classifier-Free Guidance (CFG) has emerged as a central approach for enhancing semantic alignment in flow-based diffusion models. In this paper, we explore a unified framework called CFG-Ctrl, which reinterprets CFG as a control applied to the first-order continuous-time generative flow, using the conditional-unconditional discrepancy as an error signal to adjust the velocity field. From this perspective, we summarize vanilla CFG as a proportional controller (P-control) with fixed gain, and typical follow-up variants develop extended control-law designs derived from it. However, existing methods mainly rely on linear control, inherently leading to instability, overshooting, and degraded semantic fidelity especially on large guidance scales. To address this, we introduce Sliding Mode Control CFG (SMC-CFG), which enforces the generative flow toward a rapidly convergent sliding manifold. Specifically, we define an exponential sliding mode surface over the semantic prediction error and introduce a switching control term to establish nonlinear feedback-guided correction. Moreover, we provide a Lyapunov stability analysis to theoretically support finite-time convergence. Experiments across text-to-image generation models including Stable Diffusion 3.5, Flux, and Qwen-Image demonstrate that SMC-CFG outperforms standard CFG in semantic alignment and enhances robustness across a wide range of guidance scales. Project Page: https://hanyang-21.github.io/CFG-Ctrl

1. Introduction

CFG improves semantic alignment but its linear guidance can destabilize generative trajectories at high guidance scales. CFG-Ctrl reframes guidance as feedback control and introduces SMC-CFG, using nonlinear switching control and Lyapunov analysis to support robust convergence.

  • CFG enhances semantic alignment, but linear extrapolation can distort the learned data manifold and produce oversaturated colors, warped structures, and guidance-scale sensitivity.
  • The conditional–unconditional velocity discrepancy diminishes during flow progress and serves as an error signal for feedback control.
  • Vanilla CFG is interpreted as proportional control with fixed gain, while subsequent variants modify the corresponding feedback control law.
  • Approximately linear feedback laws cannot ensure stable convergence in highly nonlinear dynamics, especially with larger model capacity or guidance scales.
  • SMC-CFG directs trajectories onto a rapidly converging sliding mode surface using a switching control term that provides nonlinear feedback-driven correction.
  • Lyapunov stability analysis supports finite-time convergence toward the desired semantic manifold, while experiments across three text-to-image models report improved semantic fidelity, fewer visual artifacts, and robustness.

2. Related Work

The related work situates CFG within diffusion and flow-matching generation, guidance methods, and control theory. Flow matching uses learned velocity fields, while sliding mode control motivates stability-oriented feedback for nonlinear systems.

  • Diffusion and Flow Matching: Diffusion models iteratively transform simple distributions into complex ones to generate high-quality samples.
  • Diffusion and Flow Matching: Flow matching models the transformation process with a learned velocity field, simplifying the generative formulation and improving empirical performance.
  • Guidance in Diffusion: Guidance techniques steer conditional generation toward desired semantic targets across visual tasks.
  • Control Theory: Control theory provides frameworks for regulating complex systems toward desired objectives while addressing performance, safety, and efficiency.
  • Control Theory: Sliding Mode Control uses a discontinuous law to force trajectories onto a predefined manifold and provide resilience to disturbances.

3. Method

CFG-Ctrl reframes classifier-free guidance as feedback control over a continuous-time flow, interpreting standard CFG and variants through their control laws. SMC-CFG adds a nonlinear sliding-mode controller to stabilize semantic guidance and support finite-time convergence.

  • CFG-Ctrl formulation: CFG linearly combines conditional and unconditional velocity fields, with guidance weight w controlling the strength of semantic alignment.When w=1, the model is the standard conditional predictor; larger weights improve alignment while reducing sample diversity.
  • CFG-Ctrl formulation: CFG-Ctrl models flow-matching sampling as a continuous-time controlled dynamical system with semantic error feedback.The framework uses a guidance schedule K_t and direction operator Π_t to determine correction strength and direction.
  • Control interpretation: Under CFG-Ctrl, standard CFG is a proportional state-feedback controller whose constant guidance scale w acts as the proportional gain.The framework recovers the standard CFG update when K_t is a constant scalar and Π_t is the identity operator.
  • Motivation: Existing predominantly linear feedback laws can become unstable in nonlinear flows at high model capacity or large guidance scales, producing oscillations and visual distortions.Reported effects include oversaturated textures, semantic inconsistency, color distortion, lost fine details, and inconsistent textures.
  • Sliding Mode Control CFG: SMC-CFG defines a sliding-mode surface over semantic error and applies nonlinear switching feedback to drive the trajectory toward a stable manifold.The Lyapunov-based analysis supports finite-time convergence of the sliding variable, while the error dynamics provide smooth exponential convergence toward the target.
  • Sliding Mode Control CFG: The method enforces stable semantic guidance by constraining the diffusion trajectory toward the desired manifold and eliminating oscillations during guided sampling.The complete procedure is presented in Algorithm 1.

4. Experiments

Experiments evaluate SMC-CFG on multiple flow-based text-to-image models using standard and recent CFG variants, with quantitative and qualitative assessments of image quality, semantic alignment, and human preference. SMC-CFG is tested across model backbones, guidance settings, and hyperparameter choices.

  • Experimental Setups: Experiments use 5,000 MS-COCO image-text pairs across SD3.5, Flux-dev, and Qwen-Image models, comparing SMC-CFG with CFG, CFG-zero⋆, and Rectified-CFG++.The evaluated models contain 8B, 12B, and 20B parameters, respectively.
  • Experimental Setups: Evaluation reports FID, CLIP Score, aesthetic quality, and human-preference metrics to measure realism, semantic consistency, and perceptual preference.The reported metric set includes FID, CLIP Score, Aesthetic Score, ImageReward, PickScore, and HPSv2.1.
  • Quantitative Evaluation: SMC-CFG consistently achieves lower FID and higher CLIP Scores than standard CFG and baselines across different text-to-image models.The results indicate improved visual quality, realism, and semantic alignment.
  • Quantitative Evaluation: SMC-CFG also attains superior ImageReward, HPSv2.1, and MPS scores, while achieving comparable or better results on additional metrics.These metrics are described as reflecting human aesthetic and preference judgments and overall generation quality.
  • Qualitative Evaluation: Across model backbones and challenging prompts, SMC-CFG produces sharper details, more coherent object structures, and more faithful adherence to textual descriptions.The qualitative comparisons cover positional relationships, text generation, detailed object representation, complex compositions, fine-grained semantics, and stylistic attributes.
  • Ablation on Hyperparameters: Ablations show that extreme λ values distort the sliding-mode manifold, while modest k values slow convergence and reduce CFG-induced distortions.The study examines how λ shapes the sliding-mode surface and how k controls the force toward that surface.

5. Conclusion

The paper presents CFG-Ctrl as a control-theoretic framework for flow-matching guidance and proposes SMC-CFG as a nonlinear mechanism for fast, stable convergence. Experiments report improved semantic alignment and visual fidelity across guidance scales, while ablations examine hyperparameter effects on stability and perception.

  • CFG-Ctrl reinterprets classifier-free guidance as feedback control in flow-matching models and analyzes its nonlinear behavior under high guidance scales.
  • SMC-CFG introduces a switching control term that enforces fast and stable convergence along a sliding-mode surface.
  • Experiments show that SMC-CFG consistently improves semantic alignment and visual fidelity while remaining robust across diverse guidance scales.
  • Ablation studies reveal how SMC-CFG hyperparameters affect stability and perception.

6. More Theoretical Analysis

The theoretical analysis frames SMC-CFG as a robust nonlinear controller that drives semantic error toward a sliding manifold despite drift and gain uncertainty. Under stated boundedness and dominance assumptions, Lyapunov analysis supports finite-time convergence, while discrete-time analysis yields a stability corridor for tuning.

  • CFG-Ctrl framework: CFG-Ctrl represents the semantic discrepancy as an error signal and interprets guidance variants through their schedules and correction-direction operators.The framework models guidance schedules as K_t and direction operators as Π_t, with velocity fields and semantic errors summarized in the notation table.
  • CFG variants: Rectified-CFG++ uses time-varying gain scheduling and predictive future-state error information to adjust guidance during sampling.Its control interpretation combines current-state error with predicted future-state information.
  • Robustness analysis: SMC-CFG introduces a nonlinear switching term that drives the generative trajectory toward the semantic sliding manifold S = {(x, t) | s(t) = 0}.The semantic error is e(t) = vθ(x_t, t, c) − vθ(x_t, t, ∅), and the sliding variable is s(t) = ė(t) + λe(t).
  • Assumptions and control law: The switching law Δe(t) = −k · sign(s(t)) is analyzed under bounded intrinsic drift and nominal control dominance assumptions.The effective gain is decomposed into a nominal isotropic component and an anisotropic deviation, with the guidance scale assumed to dominate that deviation.
  • Stability proof: Theorem 1 establishes robust convergence when the switching gain satisfies the stated condition with safety margin ϵ > 0.The Lyapunov function V(s) = 1/2∥s∥² leads to a differential inequality that guarantees finite-time convergence of s(t).
  • Discrete-time tuning: Discrete-time implementation requires a stability corridor: k must exceed model drift while remaining bounded to prevent numerical oscillations.The analysis attributes the upper bound to discretization and reports that moderate fixed k achieves the optimal balance in experiments.

7. Additional Implementation Details

The implementation evaluates SMC-CFG on compositional text-to-image generation using three flow-matching models and disjoint auxiliary data for hyperparameter selection. It reports the evaluation metrics, benchmark structure, and model-specific settings used in the main experiments.

  • Benchmark setup: T2I-CompBench contains 6,000 compositional text prompts spanning color, shape, texture, and spatial relationships.The benchmark targets open-world compositional text-to-image fidelity across four relevant subcategories.
  • Models: The experiments use publicly available checkpoints for Stable Diffusion 3.5, Flux-dev, and Qwen-Image.The setup is designed to evaluate SMC-CFG across three flow-matching text-to-image models.
  • Metrics: FID measures image quality and diversity, while CLIP Score measures semantic alignment between generated images and text prompts.Both metrics are used in the main-text evaluation.
  • Hyperparameter selection: Hyperparameters λ and k are selected by grid search over an auxiliary set of 200 MS-COCO cases disjoint from the evaluation set.The disjoint selection procedure is used to avoid test-set leakage.
  • Hyperparameter selection: The selected settings are λ = 6 with k = 0.1 for Stable Diffusion 3.5, λ = 6 with k = 0.7 for Flux, and λ = 6 with k = 0.1 for Qwen-Image.These configurations are used in the main experiments without further modification.

8. More Experiments

Additional experiments report stronger compositional alignment across three text-to-image models, transfer to text-to-video generation, comparable computational cost, and distinct roles for the two SMC-CFG hyperparameters.

  • Text-to-image evaluation: SMC-CFG improves Color, Shape, Texture, and Spatial compositional-generation performance on SD3.5, Flux-dev, and Qwen-Image.Gains are generally larger on spatial and attribute-related dimensions.
  • Text-to-image evaluation: SMC-CFG achieves the best Base, Advance, and Overall VQAScore results on SD3.5 among the compared methods.It outperforms standard CFG and recent variants including CFG-Zero and Rect-CFG++.
  • Text-to-video evaluation: SMC-CFG improves total VBench, Quality, and Semantic scores over CFG on Wan2.2-TI2V-5B.It also performs better on Color, Human Action, and Subject Consistency.
  • Text-to-video evaluation: In qualitative video comparisons, SMC-CFG produces smoother motion with fewer artifacts or flickering and maintains object structure and identity in compositional scenarios.The reported improvements concern temporal consistency and semantic adherence.
  • Computational cost: SMC-CFG has memory consumption, FLOPs, and average inference time comparable to standard CFG in a single inference pass.The comparison evaluates practicality at different output resolutions.
  • Hyperparameter effects: λ shapes structural-detail stability through the sliding-mode manifold, whereas k regulates semantic alignment and its trade-off with aesthetic realism.The hyperparameter comparison uses fixed initial noise.

9. More Discussion

SMC-CFG is more robust than conventional guidance as CFG scales increase, continuing to improve where baseline methods degrade. The method’s main stated limitation is the added tuning burden from two extra hyperparameters.

  • CFG scale: At CFG scale 2, standard CFG and other baselines reach their best performance on Flux-dev before degrading rapidly at larger scales.The degradation is attributed to nonlinear distortions introduced by high guidance.
  • CFG scale: SMC-CFG continues improving as CFG scale increases and shows only a slight performance drop at extremely large scales.This indicates robustness against over-guidance effects in the reported curves.
  • Discussion: SMC-CFG alleviates nonlinear effects associated with high CFG scales while improving compositional image generation.The discussion presents this as the method’s reported advantage over conventional guidance behavior.
  • Limitations and future work: SMC-CFG introduces two additional hyperparameters that increase deployment complexity and may require manual tuning for different models.Future work proposes adaptive control to reduce manual tuning as the generative process evolves.
Loading 2603.03281v2…