Aligning Diffusion Models by Optimizing Human Utility

1 Introduction

• Human-preference alignment for text-to-image models remains relatively under-explored compared with language-model alignment.The introduction frames adapting preference-based fine-tuning for image generation as an opportunity to tailor outputs to user preferences.Source: §1 · p. 2
• Per-image binary feedback can capture users’ desired and undesired generations and is abundant online, whereas collecting pairwise rankings is expensive and time-consuming.The method is intended to enable learning from this readily available feedback at scale.Source: §1 · p. 2
• Diffusion-KTO extends human utility maximization to diffusion models and optimizes at each denoising step rather than sampling the entire reverse process.The per-step objective addresses the computational expense of back-propagating through all sampling steps.Source: §1 · p. 2
• Diffusion-KTO generations are generally preferred over existing approaches by human evaluators and preference models.The approach is presented as a simple, robust framework for aligning T2I models with human preferences.Source: §1 · p. 2

Text-to-Image Generative Models

• Text-to-image diffusion models can produce high-quality caption-faithful images but may still show issues such as poorly rendered hands and faces.The work focuses on diffusion models because of their popularity and open-source availability.Source: §2 · p. 3

Improving Language Models using Human Feedback

• KTO [18] trains language models to maximize output utility and uses per-sample binary feedback rather than pairwise preferences.The paper generalizes this utility-maximization framework to diffusion models.Source: §2 · p. 3

Improving Diffusion Models using Human Feedback

• Reward-based diffusion alignment methods back-propagate through reverse sampling, creating memory costs that motivate low-rank updates or sampling only some steps.Diffusion-KTO’s objective extends to each denoising step, avoiding these memory issues.Source: §2 · p. 3
• Diffusion-KTO builds on similarities with Diffusion-DPO and D3PO to introduce a reward-model-free alignment objective, using only per-image binary feedback rather than pairwise preference data.Source: §2 · p. 4

3.1 Diffusion Models

• DDPM trains a reverse process with an ELBO objective, while its diffusion loss weights the difference between added and predicted noise over time.The forward process gradually adds noise until the sample reaches a standard normal distribution.Source: §3.1 · p. 4

3.2 Direct Preference Optimization

• DPO replaces explicit reward-model fitting with supervised policy training, but its preference objective uses paired winning and losing samples.Adapting DPO to diffusion is challenging because optimizing an image likelihood requires accounting for paths through a multi-step process.Source: §3.2 · p. 4, §3.3 · p. 4

3.3 Implicit Reward Model of a Diffusion Model

• D3PO treats diffusion as a Markov decision process, where each sampling step is an action and its value is represented by Q(xt−1, xt, c).The approximate objective uses the state-visitation distribution under policy π.Source: §3.3 · p. 4
• Diffusion-KTO applies utility maximization independently to each generation and diffusion step, using per-image binary feedback rather than paired preferences or reward backpropagation through the full sampling process.The diffusion process is framed in terms of local sampling steps.Source: Figure 3 · p. 5

3.4 Kahneman-Tversky Optimization

• Expected utility evaluates actions using subjective probabilities over outcomes, and prospect theory makes utility depend on a reference outcome.KTO uses a sigmoid approximation to a utility function that is concave for gains and convex for losses.Source: §3.4 · p. 5

4.1 Diffusion-KTO

• Diffusion-KTO applies a nonlinear utility to each action value relative to a reference point, using desirable or undesirable image labels to optimize the local sampling policy.The reference value is set to a KL-divergence penalty between the policy and reference policy at an action and state.Source: §4.1 · pp. 5–6

4.2 Utility Functions

• The authors note uncertainty about whether other assumptions about human behavior apply and that people may have different utility functions.They center utilities at zero without changing the objective's gradient or optimal policy, and sample desirable and undesirable examples with probabilities γ and 1 − γ.Source: §4.2 · p. 6
• The method tests loss-averse concave, risk-seeking convex, and Kahneman-Tversky sigmoid utilities; the sigmoid choice performs best empirically.The exponential is avoided for the risk-seeking case because it explodes on positive inputs and is hard to optimize.Source: §4.2 · p. 6

5 Experiments

• The appendix provides comparisons across prompt datasets, ablations, implementation and evaluation details, and synthetic experiments on tailoring models to a specific user's preferences.Source: §5 · p. 6
• Training fine-tunes SD v1-5 on Pick-a-Pic v2 with Kahneman-Tversky utility, labeling images preferred at least once desirable (237,530 samples) and the rest undesirable (690,538).Source: §5 · pp. 6–7

5.1 Quantitative Results

• 75% user-study win-rate over SD v1-5; Diffusion-KTO also won 72% versus SFT and 69% versus Diffusion-DPO, with automated win-rates up to 87.2%.Source: Table 1 · p. 7, §5.1 · pp. 7–8, Figure 4 · p. 7

5 Experiments

• Evaluation compares Diffusion-KTO with SD v1-5, SFT, CSFT, AlignProp, D3PO, and Diffusion-DPO using five automated metrics and same-prompt human choices.Source: §5 · p. 7

5.2 Qualitative Results

• Across examples, Diffusion-KTO better balances prompt fidelity and image appeal, retaining key objects omitted by baselines and improving anatomy, symmetry, and pose in a portrait prompt.Source: Figure 5 · p. 8, §5.2 · p. 8
• In MLP diffusion samples, the Kahneman-Tversky utility function best aligns with the desirable distribution while avoiding the undesirable distribution.Source: Figure 6 · p. 9

• The experiment pretrains small MLP diffusion models on a Gaussian distribution, then fine-tunes them on separate desirable and undesirable Gaussian samples.Source: §6 · p. 9
• The risk-averse function used by Diffusion-DPO avoids loss so strongly that samples deviate from the desirable distribution; risk-seeking behaves roughly like SFT and tolerates undesirable samples.Source: §6 · p. 9
• Diffusion-KTO's objective achieves a balance between matching desirable samples and avoiding undesirable ones in this experiment.Source: §6 · p. 9

• Pick-a-Pic uses online-user prompts and images from off-the-shelf T2I models, so its preference distribution may skew toward inappropriate or otherwise unwanted imagery.Source: §7 · p. 9
• Although Kahneman-Tversky performed best empirically among three examined utility models, choosing a utility function and its assumptions remains open.Source: §7 · p. 9
• Fine-tuning a pretrained T2I model inherits its weaknesses, including generating images that reflect and propagate negative stereotypes.Source: §7 · p. 9

• Diffusion-KTO adapts a human-utility maximization framework to align text-to-image diffusion models with human preferences.Source: §8 · p. 9
• The method uses simple per-image binary feedback, avoiding the need to collect paired preference data.Source: §8 · p. 9
• Human judgments and automated metrics show improvements in image preference and image-text alignment, while the best utility function remains an open question.Source: §8 · p. 9

A.1 Implementation Details

• Training used SD v1-5 and classified any image preferred at least once in Pick-a-Pic comparisons as desirable, with the rest labeled undesirable.The setup used β=5000 and sampled desirable images at γ=0.8.Source: §A.1 · p. 13

A.2 Evaluation Details

• Human evaluation used 100 HPS v2 prompts and 300 blinded, randomized side-by-side judgments, with judges choosing the image they preferred for each prompt.The 100 prompts were sampled across four styles; judges were not told which methods they were comparing.Source: §A.2 · p. 13

B.1 Performance in terms of Average Score per Preference Models.

• Diffusion-KTO achieves state-of-the-art performance on numerous automated metrics and performs comparably on the remaining metrics in the Pick-a-Pic evaluation.Average scores and win rates are reported for the test set.Source: Table 2 · p. 14, Table 3 · p. 14, §B.1 · p. 14

B.2 Performance on HPS v2 and PartiPrompts.

• Diffusion-KTO outperforms existing baselines across diverse prompts from HPS v2 and PartiPrompts.The reported automatic win rates compare Diffusion-KTO generations against other methods on each prompt set.Source: Table 4 · p. 14, §B.2 · p. 14, Table 5 · p. 15
• Across metrics, Diffusion-KTO performs best on painting and concept-art styles, which the authors attribute to stylistic biases in its training data.The authors note that Pick-a-Pic prompts reflect user biases toward artistic content.Source: §B.2 · p. 15, Table 6 · p. 15
• Different metrics rank styles differently: photos score highest on PickScore but lowest on ImageReward, and the authors suggest less biased training data could avoid such discrepancies.They describe Diffusion-KTO as agnostic to the preference distribution when feedback is per-sample and binary.Source: §B.2 · p. 15

B.3 Using Stable Diffusion v2-1.

• With SD v2-1, Diffusion-KTO outperforms the base model and Diffusion-DPO on most metrics and performs comparably on others.The authors describe the approach as architecture-agnostic for improving text-to-image diffusion-model alignment.Source: §B.3 · p. 15

C Synthetic Experiment: Aligning with a Specific User

• The experiments used Pick-a-Pic v2 images, labeling red-channel-enhanced images as desirable in the red-filter test and images scoring at least 7 as desirable in the aesthetics test.The remaining images in each experiment were treated as undesirable.Source: C Synthetic Experiment: Aligning with… · pp. 15–16

E

• Diffusion-KTO learns both tested preferences: high-aesthetics generations gain detail, colors, and textures, while red-preference generations become redder.The red preference is learned while minimally changing image content.Source: C Synthetic Experiment: Aligning with… · p. 16, Figure 7 · p. 16

D Ablations

• The best ablation combination uses winning and intermediate samples, a positive ratio of 0.8, the KTO objective, and β=5000.The ablations report mean PickScore on HPS v2, which contains 3,500 prompts.Source: Figure 8 · p. 17, D Ablations · p. 17

D.1 Data Partitioning

• Treating images preferred at least once as desirable performs better than requiring an image to win every pairwise comparison.Both strategies convert Pick-a-Pic pairwise preferences into per-image binary labels.Source: §D.1 · p. 17

D.2 Data Sampling

• Sampling 80% of each minibatch from desirable images achieves the optimal result in the authors’ experiments.The fixed ratio is intended to account for potentially imbalanced desirable and undesirable sets.Source: §D.2 · p. 17

D.3 Choice of Beta

• Performance rises as β increases from 2000 to 5000, with diminishing returns and an optimal score around 5000.β controls deviation from the policy.Source: §D.3 · p. 17

D.4 Utility function

• KTO is the best tested utility function, reducing updates at both high and low rewards; the authors say this makes it more robust to noisy, contradictory preferences.The comparison includes Loss-Aversion and Risk-Seeking utility functions.Source: §D.4 · p. 17, Figure 9 · p. 18

E Additional Qualitative Results

• Visual comparisons show improvements in photorealism, color richness, and fine detail for Diffusion-KTO-aligned SD v1-5.The authors also provide examples of failure cases.Source: E Additional Qualitative Results · p. 18

F Safety

• 5.4% vs 4.4% of generations were flagged NSFW on Pick-a-Pic prompts, and 1.3% vs 1.0% on HPSv2 prompts, for Diffusion-KTO versus SD v1-5.The authors characterize the Pick-a-Pic increase as marginal.Source: F Safety · p. 18

G Details of Qualitative Results

• The failure examples include misplaced coral, a dragon merging with stone, misspelled text, and missing requested figures; the authors note inherited text-to-image limitations.The appendix also provides side-by-side comparisons using varied prompts.Source: Figure 10 · p. 19, Figure 11 · p. 20, Figure 12 · p. 21