TFG: Unified Training-Free Guidance for Diffusion Models

1 Introduction

• Training-free guidance steers generation using an off-the-shelf differentiable target predictor, avoiding additional training for each conditioning signal.The predictor can be a classifier, loss, probability, or energy function.Source: §1 · p. 1
• Existing training-free guidance methods lack theoretical grounding and comprehensive benchmarks, and can produce unsatisfactory CIFAR10 label-guidance samples.The lack of quantitative comparisons also makes it difficult to identify an appropriate algorithm for a new application.Source: §1 · p. 2
• TFG represents existing approaches as special cases in a unified hyperparameter design space, turning their study and comparison into analysis of hyperparameter choices.The paper analyzes hyperparameter motivations and experimentally studies their influence.Source: §1 · p. 2
• 8.5% average improvement over existing methods across 16 tasks and 40 targets is reported for TFG.The paper presents the benchmark as spanning qualitatively different tasks and offers guidelines for future research.Source: §1 · p. 3

2 Background

• A diffusion model predicts noise added to data, and sampling starts from Gaussian noise before iteratively producing less noisy samples.The model estimates the score of the noisy-data distribution; under stated assumptions, sampling yields data from the target distribution.Source: §2 · p. 3
• A target predictor scores how well a sample matches a requested target, and training-free guidance seeks conditional samples without additional training.The predictor is defined on clean data, making guidance of noisy samples more difficult than training-based guidance.Source: §2 · pp. 3–4

2.1 Existing algorithms

• Prior methods vary in their guidance operations, making it difficult to determine which are essential without quantitative comparisons.DPS evaluates guidance at the predicted clean sample, while LGD estimates the predictor using Gaussian-kernel Monte Carlo sampling.Source: §2.1 · p. 4
• FreeDoM and UGD use recurrent denoising and re-noising; FreeDoM varies guidance strength over time, while UGD also optimizes and guides the predicted clean sample.FreeDoM's recurrent strategy can strengthen guidance at additional computational cost, but its schedule is not comprehensively studied.Source: §2.1 · p. 4
• MPGD differentiates with respect to the predicted clean sample to avoid backpropagating through the diffusion model, but its generality and theoretical relationship to gradients on noisy samples remain unclear.An autoencoder variant is not applied because it may be inaccessible or require training.Source: §2.1 · p. 4

3.1 Unification and extension

• TFG places existing training-free guidance algorithms in a shared hyper-parameter space, enabling unified analysis and comparison.The framework is presented as a way to choose hyper-parameters for downstream tasks.Source: §3.1 · p. 5
• LGD, DPS, FreeDoM, and MPGD correspond to specific HTFG subspaces, while TFG covers the full space.The paper states that existing algorithms cover only subsets of HTFG and that TFG is guaranteed to perform better.Source: §3.1 · pp. 5–6

3.2 Algorithm and design space analysis

• Mean Guidance iteratively adds predictor gradients at the estimated clean sample to move it out of low-probability regions where guidance can be unstable.Under perfect optimization, the estimated clean sample is the conditional expectation given the noisy sample.Source: §3.2 · p. 6
• Variance Guidance takes gradients with respect to the noisy sample and, under perfect optimization, incorporates covariance-based second-order information.Correlated components can reinforce one another, while negatively correlated components can have guidance canceled.Source: §3.2 · p. 6
• Implicit Dynamic smooths the predictor with a Gaussian kernel; performance remains nearly unchanged as Monte-Carlo samples for the expectation decrease to one.The paper attributes this to the addition of noise being more important than precise estimation of the expectation.Source: §3.2 · p. 6, Table 1 · p. 6
• Increasing recurrence first improves and then degrades generation quality, reflecting a trade-off between inherited error and error accumulated in the current step.Figure 2 also reports that the structure comparison is unchanged as other hyper-parameters vary.Source: §3.2 · pp. 6–7, Figure 2 · p. 7

Structure analysis

• Decomposing ρ and µ into scalar magnitudes and non-negative structures whose values sum to T supports preselecting structures before optimizing scalar parameters.The parameter space comprises time-dependent vectors ρ and µ, and time-independent Nrecur, Niter, and γ̄; the broader search-space analysis motivates a general search strategy.Source: §4 · p. 7
• The analysis considers three qualitatively different structures, each justified under certain conditions, while leaving broader structure exploration to future work.The accompanying benchmark table describes 14 task types and 38 targets, evaluated for guidance validity and fidelity according to task type.Source: §4 · p. 7, Table 2 · p. 8
• Across CIFAR-10 and ImageNet, validity–FID curves consistently improve when structures change from decrease to constant to increase, enabling structure selection before scalar tuning.The ordering remains stable as other parameters vary; the analysis tests label guidance on all ten CIFAR-10 labels and four ImageNet labels.Source: §4 · p. 7

Computation cost analysis

• Recurrence increases computation cost by Nrecur times with a clear performance gain, while iteration’s computational increase is smaller and its effect plateaus.An upper bound of 4 for either parameter suffices for near-optimal performance; users can choose values based on available computation.Source: §4 · p. 8, Figure 3 · p. 8

Searching strategy

• Beam search doubles the scalar parameters ρ̄, µ̄, and γ̄ from small initial values, evaluates small generation trials, and retains the top K configurations.The search stops when T stabilizes or reaches its maximum number of steps; its smaller trial sample size keeps computation controllable.Source: §4 · p. 8

5 Benchmarking

• The benchmark covers 7 datasets, 16 tasks, and 40 targets, with existing methods searched within their corresponding design subspaces.The experiments cost more than 2,000 A100 GPU hours and unify methods, tasks, search strategies, and evaluations in one codebase.Source: §5 · p. 8

5.2 Benchmarking results

• TFG outperforms existing algorithms in 13 of 14 settings, with a 7.4% average guidance-validity improvement over the best existing algorithm.Table 3 reports guidance validity and generation fidelity averaged across targets; the comparison is against the method with the highest guidance validity.Source: Table 3 · p. 9, §5.2 · p. 9

5.1 Settings

• Tasks span image restoration and label guidance, style transfer, molecule properties, audio restoration, and combinations of guidance targets.Tasks were run across multiple datasets or targets; settings use accuracy for search and report both accuracy and FID.Source: §5.1 · p. 9

5.2 Benchmarking results

• For out-of-distribution fine-grained bird-species guidance, recurrence yields at most 2.24% accurate birds, versus 0% from unconditional generation.The predictor classifies 525 bird species, and the paper describes the problem as highly difficult even for leading text-to-image models.Source: §5.2 · pp. 9–10
• On CIFAR10 label guidance, increasing recurrence reduces TFG’s accuracy gap to the training-based method from 33% to 8%.Similar improvement is observed on other datasets; Table 4 reports accuracy and FID across recurrence steps on three label-guidance datasets.Source: §5.2 · p. 10, Table 4 · p. 10
• For CelebA multi-label guidance, generated male-and-blonde accuracy reaches 46.7%, compared with a 1% share of that group in the dataset.The analysis considers gender paired with age or hair color and identifies label imbalance as a source of classifier spurious correlations.Source: §5.2 · p. 10, Table 5 · p. 10
• For molecule guidance, TFG has the highest guidance validity on 5 of 6 targets and a 5.64% MAE improvement over existing methods.Molecules with better validity usually have lower generation fidelity, consistent with previous work [3].Source: §5.2 · p. 10
• In audio guidance, TFG achieves significant relative improvements over existing methods in a domain rarely explored in training-free guidance.The paper presents its audio benchmarks as a contribution to future research in this area.Source: §5.2 · p. 10

6 Discussions and Limitations

• The framework covers representative algorithms, but not every approach, to keep the unification clear and practical.The authors describe this as a balance between breadth and practicality.Source: §6 · p. 10
• The paper does not systematically analyze which conditional-generation tasks suit training-free guidance or comprehensively analyze predictor–diffusion misalignment.It notes that the gap between training-based guidance and TFG remains high on some tasks, including molecule property guidance.Source: §6 · p. 10

A.1 Failure case of image generation with GPT4

• GPT-4’s molecule example contains invalid carbon bonding and is not the claimed benzene, illustrating difficulty with subtle, fine-grained, combinatorial, or open-ended targets.The prompted task asks for a molecule with polarizability α = 3.Source: §A.1 · p. 16, Figure 4 · p. 16
• Training-free guidance uses differentiable objectives for specific targets and plug-and-play predictors to add tasks without retraining the diffusion or prediction models.The approach is described as lightweight and applicable to varied downstream tasks.Source: §A.1 · p. 17
• GPT-4’s requested CelebA-style young man is described as outside the CelebA distribution, despite first being prompted about the dataset.Figure 5 points to Figure 12 for comparison.Source: Figure 5 · p. 17, §A.1 · p. 17
• Selecting different diffusion models lets users change the unconditional background distribution while a predictor steers generation toward selected properties.The section presents this flexibility as an opportunity to generate a wider range of user-defined targets.Source: §A.1 · p. 17

A The Motivation of Studying Training-free Guidance

• The CIFAR10 illustration reports a large performance gap across training-free settings and shows MPGD ship samples and predicted-clean-image trajectories.The figure averages accuracy and FID across ten labels and 2048 samples per label; its UGD and FreeDoM suffixes indicate recurrent-step counts, and “fake” denotes a synthetic classifier setting.Source: Figure 6 · p. 18

A.2 The fundamental challenge of training-free guidance

• A CIFAR10 label-guidance case study compares an oracle training-based time-dependent classifier, a clean-only training-free standard classifier, and a “fake” training-free classifier to illustrate the challenge and insufficiency of existing methods.The fake classifier shares parameters across time steps during training, resulting in implicit regularization unavailable to practical predictors; it helps identify the difficulty of training-free guidance.Source: §A.2 · p. 18
• Training-free approaches significantly underperform training-based guidance on CIFAR10, and generated images can be unnatural under strong guidance or label-irrelevant under weak guidance.The “fake” classifier produces less messy images than a real clean-only predictor, but is inaccessible in practice because training exposes it to noisy images.Source: §A.2 · pp. 18–19, Figure 7 · p. 18
• The central challenge is the lack of a “smoothing” classifier that gives faithful guidance in noisy image regions unseen during predictor training.The fake classifier’s implicit exposure to noisy images during training distinguishes it from practical clean-only predictors.Source: §A.2 · p. 19
• Ambiguous comparisons across methods tested on distinct, primarily qualitative applications motivate a unified framework for systematic study.The authors state that this ambiguity hinders in-depth study of the field.Source: §A.2 · p. 19

B Pseudo-code and schematics

• The appendix supplies pseudocode for TFG, DPS, MPGD, FreeDoM, UGD, and LGD, plus a schematic of existing algorithms and the reversed diffusion process.LGD’s pseudocode is presented as a modification of DPS following its paper; the authors refer to Appendix C for equivalence analysis.Source: B Pseudo-code and schematics · p. 20, Figure 8 · p. 21

C Proofs

• The appendix provides proofs for Theorem 3.2 and Lemma 3.3.Source: C Proofs · p. 22

• When γ̄ = 0, the modified function f̃ reduces to f.Source: §C.1 · p. 22
• MPGD is recovered with Nrecur = 1, Niter = 1, γ̄ = 0, ρ = 0, and μ = (c1, …, cT)⊤, using f(x0|t) = exp{−L(x0|t; y)}.With these settings, the extra TFG steps have no effect and its sampling formula matches MPGD.Source: §C.1 · p. 22
• DPS and FreeDoM are special cases with f(x0|t) = exp{−D(c, x0|t)}; FreeDoM uses Nrecur = r, while DPS is FreeDoM without recurrence.Their equivalence follows from matching the recurrent update and the shared resampling step.Source: §C.1 · p. 22
• LGD with t = 1 and sample size n maps to TFG using f(x) = exp{−ℓy(x)} and the stated recurrence and hyperparameter settings.The proof identifies the correspondence through the two algorithms’ pseudocode.Source: §C.1 · p. 22
• UGD maps to TFG with f(x) = exp{−ℓ(c, f(x))} and ρ and μ set from its sampler coefficients and schedule.The proof derives the variance-guidance update from δt, s(t), and the gradient of the loss, and notes the mean-guidance deduction is similar.Source: §C.1 · p. 22

• The derivation starts from x0|t = (xt − √(1−ᾱt) εθ(xt,t))/√ᾱt and applies the chain rule.Source: §C.2 · p. 23
• Under perfect optimization, the relationship between εθ and the score of pt(xt) supplies a step in the derivation.Source: §C.2 · p. 23
• The proof connects the conditional covariance Σ0|t to ∇²xt log pt(xt), then substitutes the resulting expression into Equation (12) to finish.Source: §C.2 · p. 23

D.1 Gaussian Deblur

• TFG produces clean, low-background-noise images with faithful features and vivid details on Gaussian deblurring in the reported comparison.The task adds Gaussian blur and noise, evaluates 256 Cat-DDPM samples with FID and LPIPS, and compares methods with Nrecur = 1.Source: §D.1 · p. 24, Figure 9 · p. 25

D.2 Super Resolution

• TFG produces clean, faithful, detailed images in the reported super-resolution comparison.The task downsamples 256 × 256 images to 64 × 64, adds noise, and evaluates 256 Cat-DDPM samples using FID and LPIPS.Source: §D.2 · p. 26, Figure 10 · p. 27

D.3 Label Guidance

• Label guidance addresses a training-free performance gap relative to training-based guidance; the experiments target CIFAR-10 and selected ImageNet classes.Validity is measured with a separate classifier, while FID assesses fidelity; Figure 11 reports the Kuvasz target comparison.Source: §D.3 · p. 28, Figure 11 · p. 29

D.4 Combined Guidance

• Combined guidance targets multiple attributes in one sample, avoiding the need to train on the potentially huge space of target combinations.Experiments cover gender with hair color or age; KID assesses fidelity and separate classifiers assess joint validity.Source: §D.4 · p. 30, Figure 12 · p. 31

D.5 Fine-grained Guidance

• Fine-grained guidance tests out-of-distribution bird-species labels using an EfficientNet classifier trained on 525 species.The evaluation uses FID and a separate classifier; Figure 13 shows sampled fairy terns with some images exhibiting the target black-headed feature.Source: §D.5 · p. 32, Figure 13 · p. 33

D.6 Style Transfer

• Style transfer guides Stable Diffusion v1.5 to match reference-image style while following text prompts, using CLIP-feature Gram matrices and their Frobenius distance.For The Starry Night target, TFG has the most faithful style in the reported comparison; MPGD’s style score is substantially lower.Source: §D.6 · pp. 33–34, Figure 15 · p. 35

D.7 Molecule Property Guidance

• Molecule guidance targets six quantum-mechanics properties, evaluating property error and chemical validity across 4096 generated molecules per property.The setup uses QM9, an unconditional EDM diffusion model, and a 100-step DDIM sampler; Figure 16 reports valid molecules with better target properties.Source: §D.7 · p. 36, Figure 16 · p. 37

D.8 Audio Declipping

• Audio declipping targets reconstruction after the highest and lowest 40 mel-spectrogram dimensions are zeroed.The task evaluates 256 Audio-diffusion samples using DTW for validity and FAD for fidelity.Source: §D.8 · p. 38

E.1 Details of Table 1

• Monte Carlo sample size has a negligible performance role when hyperparameters are optimal, but searched γ̄ is apparently nonzero for many targets.The analysis separates noise added to mean guidance from noise added to variance guidance.Source: §E.1 · p. 40

E.2 Comparison with grid search

• Beam search and exhaustive grid search have identical overall performance on CIFAR-10 label guidance, while grid search is much slower.Table 10 reports searched parameters, validity, and the gap between the two methods’ validity results.Source: §E.2 · p. 40, Table 10 · p. 40

E.3 Detailed results of each target and hyper-parameters

• 18.7% validity improvement over UGD on fine-grained classification contrasts with 0.133% on gender-age guidance, while optimal parameters vary widely across targets.These results illustrate the reported dependence of TFG’s improvement on how the optimal parameters relate to existing methods’ subspaces.Source: §E.3 · p. 40
• MPGD slightly outperforms TFG on highest MO energy guidance under the reported search budget; the authors state eight beam-search steps suffice for TFG to outperform it.TFG searches at most six steps across hyper-parameters, while other methods search seven steps in their subspaces.Source: §E.3 · p. 40

E.4 Tricks implemented in FreeDoM codebase

• FreeDoM’s codebase uses application-specific guidance-strength schedules; the authors omit those tricks and report similar performance with their unified search strategy.The cited schedules include a coefficient for face generation and a more complex schedule for style transfer.Source: §E.4 · pp. 41–42

E.5 Hardware and Software

• The experiments used NVIDIA A100 clusters, PyTorch, and HuggingFace, with an estimated total compute cost of 2,000 GPU hours.Source: §E.5 · p. 42

NeurIPS Paper Checklist

• The checklist reports experimental reproducibility information in the supplement and marks compute-resource disclosure as satisfied.The authors also state that their contributions are theoretically and empirically justified.Source: NeurIPS Paper Checklist · pp. 43, 44, 46
• The authors report no error bars, citing the scale of their experiments and the stability of unreported results in related work.Source: NeurIPS Paper Checklist · p. 45