PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-free Diffusion Model Alignment with Sampling Demons

Po-Hung Yeh, Kuang-Huei Lee, Jun-Cheng Chen

arXiv:2410.05760v2cs.CVcs.AIcs.LGmath.OCstat.ML

TL;DR

Diffusion models may not match diverse user preferences, and existing alignment approaches require retraining or differentiable rewards. Demon uses stochastic optimization to guide denoising at inference time, including with non-differentiable reward sources. It improves text-to-image aesthetics scores over Best-of-N random-sampling upper bounds.

  • Problem

    Diffusion-model alignment methods either require retraining or rely on differentiable rewards, limiting use of sources such as VLM APIs and human judgments.

    Source: §1 · p. 1

  • Method

    Demon uses inference-time stochastic optimization to evaluate and synthesize reward-guided noises for denoising without backpropagation through rewards or model retraining.

    Source: §1 · p. 1, §4 · p. 3, §6 · p. 10

  • Results

    Averages well above 8.0 on aesthetics exceed Best-of-N random-sampling upper bounds of 6.5 for SD v1.4 and 7 for SDXL across text-to-image tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Demon supports inference-time preference alignment with non-differentiable rewards and can be integrated with existing diffusion models without retraining.

    Source: §1 · p. 2

    For simplicity, the preliminary derivation sets β constant across all t.

    Source: §3 · p. 3

Abstract

from arXiv · show

Aligning diffusion models with user preferences has been a key challenge. Existing methods for aligning diffusion models either require retraining or are limited to differentiable reward functions. To address these limitations, we propose a stochastic optimization approach, dubbed Demon, to guide the denoising process at inference time without backpropagation through reward functions or model retraining. Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through stochastic optimization. We provide comprehensive theoretical and empirical evidence to support and validate our approach, including experiments that use non-differentiable sources of rewards such as Visual-Language Model (VLM) APIs and human judgements. To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models. Our method can be easily integrated with existing diffusion models without further training. Our experiments show that the proposed approach significantly improves the average aesthetics scores for text-to-image generation. Implementation is available at https://github.com/aiiu-lab/DemonSampling.

1 INTRODUCTION

User preferences can diverge from the data distribution used to train diffusion models, while existing alignment approaches either require retraining or differentiable rewards. Demon uses stochastic optimization at inference time to guide denoising, including with non-differentiable reward sources.

1 INTRODUCTION

  • Existing reward-guided denoising approaches require differentiable rewards, excluding sources such as third-party VLM APIs and human judgments.Fine-tuning with reinforcement learning instead requires retraining when preferences change, which is computationally expensive and time-consuming.Source: §1 · p. 1
  • Demon evaluates candidate noises against a reward source and synthesizes “optimal” noises to seed successive reverse-time denoising steps.The method uses PF-ODE or Consistency Model evaluations to assess possible backward steps seeded with Gaussian noises.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Demon supports reward signals regardless of differentiability, making previously inaccessible sources such as VLM APIs usable for inference-time alignment.The authors describe it as the first inference-time, backpropagation-free preference-alignment method, to their knowledge.Source: §1 · p. 2
  • The method is designed to integrate with existing diffusion models without retraining or fine-tuning.Source: §1 · p. 2
  • The authors provide a theoretical explanation for reward improvement that can be used to tune hyperparameters.Source: §1 · p. 2
  • Averages well above 8.0 on aesthetics exceed Best-of-N random-sampling upper bounds of 6.5 for SD v1.4 and 7 for SDXL across text-to-image tasks.The results use prompts from prior work and do not rely on backpropagation-based preference alignment or model retraining.Source: §1 · p. 2

2 RELATED WORK

The related-work section situates Demon among diffusion-model research and human-preference alignment methods. It also compares methods across capabilities and optimization constraints.

Diffusion Model

  • Diffusion-model work progresses from initial data-generation models through high-fidelity image generation and SDE formulations; this paper bases many derivations on Karras et al. (2022).The evaluation focuses on text-to-image generation.Source: §2 · p. 2

Human Preference Alignment

  • Preference-alignment approaches include reinforcement-learning policy optimization and reward backpropagation, while proposed metrics and benchmarks serve as optimization objectives or evaluation measures.Source: §2 · p. 2
  • Table 1 compares open-vocabulary generalization, backpropagation requirements, mode-collapse avoidance, and divergence control, and identifies Demon as having zero-shot learning capabilities.Source: Table 1 · p. 3

3 PRELIMINARY

The preliminary section bases its diffusion derivation on EDM and specifies a sampling schedule and simplifying noise-rate assumption.

3 PRELIMINARY

  • The derivation uses EDM with sampling schedule σ_t = t for reverse-time SDE sampling toward the diffusion marginal distribution.Source: §3 · p. 3
  • For simplicity, β is constant across time; at sufficiently large t, the initial noisy sample is drawn from a Gaussian with covariance t_max²I_N.β denotes the relative rate at which existing noise is injected with new noise.Source: §3 · p. 3
  • Appendix A provides the paper’s full list of notation and conventions.Source: §3 · p. 3

4 REWARD-GUIDED DENOISING WITH DEMONS

Demon scores possible reverse-time SDE steps using an ODE-based reward estimate, then synthesizes noise weights to steer denoising toward higher rewards. Its reward-estimation implementation trades fidelity against runtime.

4.1 SCORING NOISES IN REVERSE-TIME SDE

  • Demon scores noisy states by estimating the reward of their clean samples, using a PF-ODE mapping as a one-evaluation approximation to the SDE reward estimate.The approximation becomes exact as β approaches zero, and rβ ≈ r ◦c when the reward is approximately harmonic.Source: §4.1 · pp. 3–4

4.2 DEMONS FOR REWARD-GUIDED DENOISING

  • Tanh Demon gives positive weights to above-average-reward noises and negative weights to below-average ones; Boltzmann Demon uses softmax weights to favor high-reward candidates.Tanh’s improvement guarantee assumes rβ ≡ r ◦c and negligible truncation error; Boltzmann has an expected-improvement guarantee under rβ ≡ r ◦c, while temperature controls deviation from the original SDE distribution.Source: §4.2 · pp. 5–6

4.3 COMPUTATIONAL CONSIDERATIONS

  • Each trajectory requires O(K · T) reward-estimate evaluations, and the r ◦c implementation sets the speed–fidelity trade-off.The default uses Heun’s ODE solver; a consistency-model version, Tanh-C, computes c faster but may lose fidelity, while Heun-based Tanh performs better at larger T.Source: §4.3 · p. 6
  • In five minutes, Tanh-C reaches 7.27 ± 0.33 aesthetics on SD v1.4, versus 6.72 ± 0.26 for Demon; random sampling’s approximate 6.5 upper bound takes over ten minutes.The reported settings are K = 16, T = 64 for Tanh-C and K = 16, T = 16 for Demon; the random-sampling bound uses over 800 reward queries.Source: §4.3 · p. 6

5 EXPERIMENTS

Experiments report gains across reward objectives and comparisons with sampling and alignment baselines, alongside tests using non-differentiable VLM preferences and interactive human judgments. Reward-estimation quality and computational budget affect the observed trade-offs.

Baseline Comparison

  • In most cases, Tanh/Tanh-C uses fewer reward queries than other baselines, and both consistently run faster than DOODL; Tanh-C further suits limited runtime.The reported comparison includes Best-of-N and Boltzmann Demon; the authors also report greater resistance to reward hacking for their backpropagation-free method.Source: §5 · p. 7

Comparison of Reward Estimation Approaches

  • At equal reward-query counts, 20-step-ODE Tanh outperforms Best-of-N, while 1-step-CM Tanh-C outperforms Tanh under limited execution time.More ODE steps generally improve fidelity but increase computation; the 1-step CM has inferior results to ODE methods.Source: §5 · pp. 7–8

Image Generation with Various Reward Functions

  • With the Ensemble reward, Tanh surpasses the individual best-of-N samples on each objective and improves all four metrics over baseline SDv1.4.Table 3 reports results across reward functions and generation methods.Source: Table 3 · p. 8, §5 · p. 9

Alignment with preferences of VLMs (Non-differentiable)

  • With Gemini Flash v1.5 and GPT-4o preferences, Tanh Demon improves PickScore on every image in any degree.The VLMs provide non-differentiable reward preferences through binary comparisons.Source: §5 · p. 9

Manual Selection

  • Online human selections raise DINOv2 cosine similarity to the reference image from 0.594 to 0.708.Users select preferred samples from 16 candidates at each iterative step.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents Demon as inference-time sampling guidance for pretrained diffusion models, with reward estimation quality and efficiency central to its performance. The appendices provide notation and convention references.

6 CONCLUSION

  • Demon uses PF-ODE clean-sample reward estimates to guide denoising with arbitrary reward sources, including non-differentiable VLMs and interactive human judgments, without training.The conclusion identifies reward-estimation quality and efficiency as important to balancing speed and reward performance.Source: §6 · p. 10

A NOTATIONS AND CONVENTIONS

A.2 CONVENTIONS

B GUIDELINE ON PARAMETER SETTING

The appendix gives parameter-setting guidance and develops theoretical analyses of reward-estimate accuracy, martingale behavior, and the Tanh Demon’s noise selection. It also supplies pseudocode for the Demon algorithms.

B GUIDELINE ON PARAMETER SETTING

C PSEUDOCODES

  • The pseudocodes take ODE reward estimates as input and return noise weights for the Tanh and Boltzmann Demon algorithms.The Tanh version uses adaptive temperature.Source: C PSEUDOCODES · p. 16

D.1 ERROR COMPREHENSION FOR REWARD ESTIMATE APPROXIMATION

  • The theoretical analysis represents reward-estimate approximation error using an Itô integral and examines the terms that remain after taking expectation.The derivation uses Itô’s lemma on the SDE trajectory and removes the Brownian-motion term by expectation.Source: §D.1 · p. 17, §D.1.1 · p. 17

D.1.2 DISCUSSION

  • When r is harmonic, its reward differential is a martingale, making the ODE-generated image reward the mean reward of SDE-generated variants; estimates also become more accurate as β decreases.Under the stated gradient-orthogonality condition as well, the paper states that r ◦ c and rβ coincide.Source: §D.1.2 · p. 18

D.1.3 ILLUSTRATION OF MISMATCH

  • Near a sharp local maximum of r, with small β and the generated sample near that maximum, the paper illustrates that rβ(xt) − (r ◦ c)(xt) < 0.The illustration assumes ∇2r ≪ 0 near c(xt).Source: §D.1.3 · p. 18

D.2 MARTINGALE PROPERTY OF REWARD ESTIMATES

  • For a fixed SDE, the current reward estimate equals the expected reward estimate at a later reverse-time step.The stated property applies when t > t − Δ > 0.Source: §D.2 · p. 19

D.3 TANH DEMON

  • Tanh Demon reverses a candidate noise’s contribution according to its estimated reward relative to the mean; under Lemma 2’s assumptions, the ideal noise improves reward at each numerical step.The guarantee assumes negligible truncation error and rβ ≡ r ◦ c.Source: §D.3 · pp. 20–21

D.4 BOLTZMANN DEMON

The Boltzmann Demon uses reward estimates to favor high-reward noise samples, with a theoretical expected-reward improvement guarantee. The appendix also justifies spherical-noise sampling as an approximation to Gaussian noise.

D.4 BOLTZMANN DEMON

  • Under the stated Lipschitz and truncation-error conditions, Lemma 3 gives an expected-reward improvement bound based on the sample maximum and reward-estimate accuracy.The expectation uses the maximum reward estimate among i.i.d. SDE samples at each numerical step.Source: §D.4 · p. 22
  • When τ = ∞ and the time step is small enough, Boltzmann Demon sampling is identically distributed as SDE sampling.Adjusting τ transitions between prioritizing high-reward noise and standard SDE sampling.Source: §D.4 · p. 22
  • As the number of sphere-sampled vectors grows, their normalized sum converges in distribution to a Gaussian with mean 0 and covariance I_N.The paper uses this central-limit result to justify spherical sampling’s Gaussian-like statistics in the diffusion term.Source: §D.4 · p. 23

D.5 HIGH DIMENSIONAL GAUSSIAN ON SPHERE

  • The appendix analyzes high-dimensional Gaussian norm concentration, approximating the squared norm by a normal distribution when dimension is large.It gives the chi-squared norm’s mean and variance as N and 2N, respectively.Source: §D.5 · p. 23

E.1 PICKSCORE COMPARISONS.

On PickScore, Tanh Demon exceeds the reported baselines while using fewer reward queries and less computation time than resampling methods. The appendix also reports qualitative comparisons, benchmark trade-offs, and human preference and prompt-alignment survey results.

E.1 PICKSCORE COMPARISONS

  • PickScore is 0.253 for the method, versus 0.246 for Diffusion-DPO(1440x), 0.243 for SDXL(1440x), 0.226 for SDXL-DPO, and 0.222 for SDXL.The method uses 1,440 reward queries and takes 180 minutes, compared with 240 minutes for resampling methods.Source: §E.1 · p. 24

E.2 QUALITATIVE RESULTS

  • Figure 8 compares PF-ODE images with Tanh Demon images and describes improvements in visual fidelity and adherence to targeted characteristics.The comparison uses PickScore and SDXL.Source: Figure 8 · p. 25

F COMPARISON ON HPSV2

  • The method’s selected image has a higher HPSv2 score than the presented Best-of-N image, using 1,424 reward queries and 1.8k seconds versus 2,336 queries and 3.8k seconds (5,440 queries for CM).For reward-query efficiency, Tanh Demon outperforms Tanh-C, followed by Boltzmann Demon; for execution time, Tanh-C is recommended.Source: F COMPARISON ON HPSV2 · p. 24, Table 11 · p. 26, Figure 9 · p. 26

G.1 IMAGE GENERATION RESULTS WITH DIFFERENT REWARD FUNCTIONS

  • The appendix reports additional SD v1.4 and SDXL generation results using four reward functions as objectives.The cited passage identifies the additional results as Tables 12–15.Source: §G.1 · p. 26

G.2 SUBJECTIVE TEST OVERVIEW

  • The subjective survey involved 101 participants ranking images across four sections covering method comparisons and different objectives, with aesthetic preference and prompt alignment as evaluation criteria.The survey compares methods including DOODL, Baseline, and Ensemble, and objectives including IR, Pick, HPSv2, and Aes.Source: §G.2 · p. 26, §G.2.1 · p. 31

G.2.2 EVALUATION RESULT OVERVIEW

  • The survey finds that Ensemble surpasses DOODL and Baseline in aesthetic preference and prompt alignment, while all objectives outperform Baseline in prompt alignment, with HPSv2 leading.The two comparisons are shown in Figure 10a and Figure 10b, respectively.Source: §G.2.2 · p. 31, Figure 10 · p. 31

G.2.3 ANALYSIS

In the subjective comparisons, Ensemble leads the method comparisons, while HPSv2 performs best among objectives. Aes is an exception for prompt alignment, where it is weaker than the other objectives.

G.2.3 ANALYSIS

  • Ensemble consistently outperforms DOODL and Baseline in visual aesthetics and semantic alignment, with especially substantial gains in tasks requiring visual refinement.DOODL marginally improves over Baseline in both criteria.Source: §G.2.3 · p. 32
  • Almost all tested objectives surpass Baseline in preference and prompt alignment, but Aes shows weaker prompt alignment because it lacks explicit text guidance.The tested objectives are IR, Pick, HPSv2, and Aes.Source: §G.2.3 · p. 32
  • HPSv2 achieves the best performance on both criteria and is reported to outperform even the Ensemble method among the objectives considered.The paper relates this comparison to preference alignment for real humans.Source: §G.2.3 · p. 32

H.1 EXPERIMENTS SETTINGS

The appendix documents prompt materials, Stable Diffusion’s adaptation to the EDM formulation, sampling settings, simplifications, and implementation constraints.

H.1 EXPERIMENTS SETTINGS

I.1 ADAPTING STABLE DIFFUSION TO EDM FRAMEWORK

  • The implementation adapts fp16 Stable Diffusion v1.4 and SDXL v1.0 to EDM, using a reparameterized time domain for numerical stability and sample quality.The passage gives the score expression ∇x log p(x, t) = (D(x, t) − x) / t^2 and defines D using the original model F.Source: §I.1 · p. 32

I.2 NUMERICAL METHODS FOR IMAGE GENERATION

  • Sampling uses ρ = 7, T ≥ 20, ln tmax ≈ 2.7, ln tmin ≈ −6.2, and classifier-free guidance 2, with K and β held constant.For t < 0.11, SDE samples appear similar to human perception; remaining evaluations use ODE, yielding slightly fewer than K · T samples.Source: §I.2 · p. 33

I.3 SIMPLIFICATIONS IN DIFFUSION PROCESS MODIFICATION

  • For simplicity, the formulation omits the VAE, prompt c, and classifier-free-guidance η, using p(x) for the unnormalized p(x)p(c | x)^η.Source: §I.3 · p. 33

I.4 BATCH SIZE AND MEMORY CONSTRAINTS

  • On an RTX 3090, SDXL VAE evaluation is limited to batch size 1, preventing further acceleration through larger batches; DOODL ran on an RTX A6000 at 0.92x speed.ODE/SDE solving uses batch size 8 for Stable Diffusion v1.4, v1.5, and SDXL.Source: §I.4 · p. 33

I.5 EXPERIMENTAL SETUP AND HYPERPARAMETERS

The appendix specifies experimental prompts and hyperparameters, then records theoretical and implementation limitations, future-work directions, ethics approval, and misuse considerations.

I.5 EXPERIMENTAL SETUP AND HYPERPARAMETERS

  • Baseline, reward-estimation, and reward-function comparisons use distinct specified settings, including β = 0.5 for the baseline and 200 Monte Carlo samples for reward estimation.Source: §I.5 · p. 35
  • Ensemble settings scale PickScore by 98.86 and HPSv2 by 40; DOODL uses 25 Aes or 100 Pick iterations, while non-differentiable and manual selection use separate stopping rules.Source: §I.5 · p. 35

J LIMITATIONS

  • The approximation r∘c ≈ rβ relies on the reward being near-harmonic around the ODE sample.Source: J LIMITATIONS · p. 35
  • Implementing r∘c faces time-complexity and accuracy bottlenecks.Source: J LIMITATIONS · p. 35

K FUTURE WORKS

  • Tanh-C and Tanh Demon differ only in how r∘c is implemented; future work proposes higher-fidelity CM distillation or a dedicated distilled model to improve Tanh-C reward performance without compromising speed.Source: K FUTURE WORKS · p. 35

L CODE OF ETHICS

  • Human-judgment experiments received IRB approval from Academia Sinica under AS-IRB-HS 02-24031.Source: L CODE OF ETHICS · p. 35

M SOCIETAL IMPACT

  • Manual image selection may be used to generate harmful content; the authors retain Stable Diffusion safeguards and assign end-users responsibility for using them.Source: M SOCIETAL IMPACT · p. 35
Loading 2410.05760v2…