Diffusion Model Alignment Using Direct Preference Optimization

Bram Wallace; Meihua Dang; Rafael Rafailov; Linqi Zhou; Aaron Lou; Senthil Purushwalkam; Stefano Ermon; Caiming Xiong; Shafiq Joty; Nikhil Naik

Source-linked AI summary

Diffusion Model Alignment Using Direct Preference Optimization

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, Nikhil Naik

arXiv:2311.12908v1Nov 21, 2023cs.CVcs.AIcs.GRcs.LG

TL;DR

Text-to-image diffusion models have been less extensively aligned to human preferences than LLMs. The paper introduces Diffusion-DPO, which adapts direct preference optimization to diffusion likelihoods and fine-tunes SDXL-1.0 on preference data. The resulting model outperforms both SDXL-base and the larger base-plus-refinement pipeline in human evaluations, while AI-feedback training performs comparably to human-preference training.

Problem
Human preference learning has been much less explored for text-to-image diffusion models than for LLMs, where preference data are used for alignment.
Method
Diffusion-DPO adapts Direct Preference Optimization to diffusion models by directly optimizing on paired preference data using a diffusion-model likelihood formulation.
Results
Diffusion-DPO fine-tuned SDXL-1.0 outperforms SDXL-base and the larger SDXL base-plus-refinement pipeline in human evaluations, and AI-feedback training has comparable performance to human-preference training.
Takeaways & Limitations
Diffusion-DPO establishes a preference-alignment paradigm for open-vocabulary diffusion models and supports using AI feedback as an alternative training signal.
Takeaways & Limitations
The approach can learn and propagate labeler biases, making a diverse and representative set of labelers essential.

Abstract

from arXiv · show

Large language models (LLMs) are fine-tuned using human comparison data with Reinforcement Learning from Human Feedback (RLHF) methods to make them better aligned with users' preferences. In contrast to LLMs, human preference learning has not been widely explored in text-to-image diffusion models; the best existing approach is to fine-tune a pretrained model using carefully curated high quality images and captions to improve visual appeal and text alignment. We propose Diffusion-DPO, a method to align diffusion models to human preferences by directly optimizing on human comparison data. Diffusion-DPO is adapted from the recently developed Direct Preference Optimization (DPO), a simpler alternative to RLHF which directly optimizes a policy that best satisfies human preferences under a classification objective. We re-formulate DPO to account for a diffusion model notion of likelihood, utilizing the evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with Diffusion-DPO. Our fine-tuned base model significantly outperforms both base SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement model in human evaluation, improving visual appeal and prompt alignment. We also develop a variant that uses AI feedback and has comparable performance to training on human preferences, opening the door for scaling of diffusion model alignment methods.

1. Introduction

Diffusion-DPO addresses the limited use of human preference learning in text-to-image diffusion models by directly optimizing on paired preference data. It adapts DPO to diffusion-model likelihoods and achieves strong preference-based results after fine-tuning SDXL-1.0.

Text-to-image diffusion models generally lack a preference-alignment stage comparable to the supervised fine-tuning and RLHF stages used for LLMs.
Diffusion-DPO directly optimizes diffusion models on human preference data, extending DPO to this setting through a novel diffusion-model data-likelihood formulation.The resulting loss is described as stable and efficient, and the method connects to a multi-step reinforcement-learning approach.
Diffusion-DPO fine-tunes SDXL-1.0 and achieves a 69% human-evaluation preference rate over the complete SDXL base-plus-refinement pipeline on PartiPrompts.The method is also reported as effective with AI feedback, rather than only human preferences.
The approach targets open-vocabulary preference alignment while retaining the diffusion model’s inference setting and avoiding issues identified for prior method classes.
The method uses paired ranked samples and models preferences with a Bradley-Terry formulation whose reward can be parameterized for binary classification.

4. DPO for Diffusion Models

Diffusion-DPO adapts preference optimization to diffusion paths by replacing intractable image likelihoods with ELBO-based objectives and tractable approximations. The resulting objective rewards preferred reverse processes while regularizing them toward a reference process.

The method learns from prompt-image pairs labeled by human preference, seeking a model aligned with preferred generations over a reference model.
Because pθ(x0|c) marginalizes over intractable diffusion paths, the method introduces latent variables x1:T and defines rewards over complete chains.
The KL regularizer is upper-bounded by the joint path KL, producing an objective that maximizes reverse-process reward while matching the reference reverse-process distribution.
Direct optimization requires sampling from pθ(x1:T|x0), which is inefficient for large T and intractable because the reverse process is parameterized by trainable transitions.The passage specifically notes T = 1000 as a typical setting.
The method applies Jensen’s inequality and convexity to derive a bound, then approximates the reverse process with the forward process q(x1:T|x0) for efficient gradient-based training.
The weighting function is constant in practice, with its factor T absorbed into β; increasing β increases loss-surface curvature.

5. Experiments

Experiments show that Diffusion-DPO improves human preference, prompt alignment, visual appeal, and image-editing performance across SDXL and SD1.5 settings. The method also supports AI-feedback training, implicit reward modeling, and preference tuning, while supervised fine-tuning behaves differently across model scales.

Primary Results: Aligning Diffusion Models: DPO-SDXL is preferred 70.0% of the time over SDXL-base on PartiPrompts General Preference, with similar win rates for Visual Appeal and Prompt Alignment.On HPSv2, DPO-SDXL achieves a 64.7% General Preference win rate and an average HPSv2 reward of 28.16.
Primary Results: Aligning Diffusion Models: DPO-SDXL produces more appealing imagery with vivid colors, dramatic lighting, strong composition, and realistic people and animal anatomy.The paper notes that preferences are not universal: some users may prefer quieter or subtler scenes.
Primary Results: Aligning Diffusion Models: DPO-SDXL achieves 69% and 64% General Preference win rates against the complete SDXL pipeline on PartiPrompts and HPSv2, respectively.The comparison is against the 6.6B-parameter base-plus-refinement pipeline, whereas DPO-SDXL uses the 3.5B-parameter base architecture only.
Image-to-Image Editing: DPO-SDXL is preferred 65% of the time over SDXL for TEd-Bench image edits, versus 24% for SDXL and 11% draws.The evaluation uses 100 real image-text pairs and SDEdit with noise strength 0.6.
Learning from AI Feedback: AI-feedback training improves targeted capabilities, but optimizing Aesthetics can reduce CLIP-measured text-image alignment.Training on PickScore and HPS preference estimates increases performance, while pseudolabeled PickScore training raises General Preference win rate from 59.8% to 63.3%.
Analysis: DPO-SDXL’s implicit reward model performs well on binary preference classification and exceeds existing recognition models on the evaluated split.This supports comparable expressivity and generalization to the classical reward-modeling objective and architecture.
Analysis: Dreamlike-DPO improves its baseline on in-distribution preferences, although the gain is limited, possibly because the training subset contains only 15% of Pick-a-Pic.The reported experiment trains on preferences generated by the Dreamlike model alone.
Analysis: SFT improves vanilla SD1.5 to a 55.5% win rate over its base model, but any amount of SFT deteriorates SDXL performance.The authors attribute the contrast to SDXL-1.0 being superior to the models represented in Pick-a-Pic.

6. Conclusion

Diffusion-DPO enables diffusion models to learn directly from human feedback and establishes a state-of-the-art open-source text-to-image model. The work also identifies safety and data-quality considerations for future deployment.

Fine-tuning SDXL-1.0 with Pick-a-Pic preferences improves generic preference, visual appeal, and prompt alignment.
DPO-SDXL outperforms the complete SDXL base-plus-refinement pipeline despite using 53% of its total model parameters.
The authors observe preliminary performance improvements from dataset cleaning and identify dataset cleaning and scaling as future directions.
Web-collected data and labeler preferences create risks involving harmful content, sexualized outputs, and encoded bias.
The authors withhold the model until additional safety filtering is added to remediate toxic content.

Supplementary Material

The supplementary material contrasts Diffusion-DPO with reinforcement, reward-maximization, dataset-curation, and inference-time methods, then details its variational objective and approximations.

Comparisons to existing work: RL-based methods struggle to generalize from limited prompt sets to open-vocabulary generation.
Comparisons to existing work: Reward-maximization methods can work in open-vocabulary settings but lack Diffusion-DPO’s distributional guarantees and may suffer mode collapse when over-trained.
Comparisons to existing work: Diffusion-DPO avoids the data bottleneck of curated high-quality training sets while directly optimizing preference data.
Preference objective: The Bradley-Terry preference objective follows after reparameterizing the reward and canceling the partition function for image pairs.
Objective derivation: The derivation uses q(x1:T|x0) to approximate the intractable reverse-process posterior and applies Jensen’s inequality.
Objective derivation: The Gaussian reverse-process parameterization reduces the objective to noise-prediction losses with timestep weighting controlled by a signal-to-noise term.
Alternative approximation: An alternative approximation q(xt|x0)pθ(xt−1|xt) yields lower error than approximating the joint with q(xt−1,t|x0).

S3. Alternate Derivation: Reinforcement Learning Perspective

The reinforcement-learning derivation interprets denoising as a sequential decision process, then recovers Diffusion-DPO as an off-policy objective rather than a policy-gradient procedure.

MDP formulation: The denoising model is treated as a policy whose states are noisy images, actions are denoising steps, and final-image reward measures human-preference alignment.
Off-policy derivation: Diffusion-DPO uses off-policy methods instead of the policy-gradient approaches used by prior reinforcement-learning work.
Objective recovery: A telescoping sum through the diffusion chain leaves the terminal preference objective because intermediate rewards are zero.
Objective recovery: This derivation yields the same objective as prior methods while simplifying optimization and avoiding sampling through the entire diffusion chain.

S4. Alternative Derivation: Noise-Aware Preference Model

The noise-aware derivation extends DPO likelihoods with timestep and noising conditioning, expressing likelihood ratios through diffusion training losses. The resulting formulation retains reference-policy control through β.

Noise-aware formulation: The policy evaluates image likelihood conditioned on the prompt, timestep, and a noised observation, initialized from a reference diffusion policy.
Noise-aware formulation: The noising operator transforms an image as qt(x0) = √αt x0 + √1 −α ϵqt, adding timestep-dependent Gaussian noise.
Noise-aware formulation: The optimal policy is proportional to the reference policy times an exponentiated reward, with normalization over captions, timesteps, and noising draws.
Preference model: The Bradley-Terry preference model compares the reward difference between preferred and dispreferred images under shared conditioning.
Loss construction: The likelihood-ratio terms reduce to differences between diffusion training losses for preferred and dispreferred images.
Reference control: The β parameter controls the penalty for deviating from the reference distribution through the asymmetry of the log-sigmoid objective.
Reward estimation: Averaging over multiple timestep and noising draws improves estimation of noise-aware reward differences for image pairs.

S5. β Ablation

The β ablation examines median PickScores across β values and identifies two failure regimes: very low β collapses the model into reward scoring, while very high β restricts adaptation.

β Ablation: Median PickScores are reported for different β choices on the Pick-a-Pic v2 validation set.The ablation uses median PickScore as its evaluation measure.
β Ablation: Very low β causes the diffusion model to degenerate into a pure reward scoring model.
β Ablation: Very high β makes the KL-divergence penalty greatly restrict any appreciable adaptation.

S6. Further SFT Discussions

The discussion relates Diffusion-DPO’s limited benefit from preferred-data SFT to the absence of a pretraining-to-downstream task gap in text-to-image generation.

Further SFT Discussions: SFT effectiveness differs partly because the original DPO settings varied in their pretraining-to-downstream task gap.
Further SFT Discussions: Diffusion-DPO uses generic text-to-image pretraining, training, and evaluation across open-vocabulary prompts and styles.
Further SFT Discussions: Like DPO summarization, an already task-tuned model does not benefit from finetuning on preferred data in this setting.

S7. Additional Automated Metrics

Additional automated metrics compare DPO-SDXL with baseline SDXL using head-to-head generation comparisons for each prompt.

Additional Automated Metrics: The y-axis measures the fraction of head-to-head prompt comparisons where DPO-SDXL scores higher than baseline SDXL.

S8. PickScore Rejection Sampling

Rejection sampling requires substantially more inference computation than Diffusion-DPO to surpass its performance, and some prompts remain unsolved even after 100 draws.

PickScore Rejection Sampling: 10× higher mean inference compute is required for baseline rejection sampling to surpass the DPO-trained model for both SDXL and SD1.5.
PickScore Rejection Sampling: 7% of SDXL prompts and 16% of SD1.5 prompts remain insufficiently served even after 100 rejection-sampling draws.
PickScore Rejection Sampling: Rejection sampling draws 100 images from prompt variants, ranks them with PickScore, and compares the highest-scored image with one random draw.
PickScore Rejection Sampling: Figure S4 plots expected PickScore win rate against the number of reference-model draws, using one learned-model generation as the comparison.

S9. Pseudocode for Training Objective

The training objective pseudocode computes a Diffusion-DPO loss for preferred and non-preferred image latents using a trainable diffusion model and a frozen reference model. Supplementary figures show DPO-SDXL generations on miscellaneous, DALLE3, and other evaluation prompts.

Inputs: The loss function takes a trainable diffusion model, a frozen reference model, preferred and non-preferred latents, text conditioning, and a regularization parameter.These inputs define the single-image-pair Diffusion-DPO loss calculation.
Noising: Training samples a timestep and noise, then adds the same noise to the preferred and non-preferred latents.The pseudocode uses the resulting noisy latents for subsequent model predictions.
Predictions: The trainable and reference models each predict noise for both the preferred and non-preferred noisy latents under the text conditioning and timestep.This produces four noise predictions for the pairwise objective.
Error computation: The pseudocode computes squared norms of prediction errors relative to the sampled noise for both models and both preference labels.These errors provide the quantities used to form the preference-training loss.
Qualitative results: DPO-SDXL generations are shown for miscellaneous prompts and complex prompts from DALLE3, while Figures S7 and S8 show qualitative HPSv2 evaluation examples.The HPSv2 examples use random seeds from human evaluation for specified prompt indices.