PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, Xiu Li

arXiv:2311.13231v3cs.LGcs.AIcs.CV

TL;DR

RLHF for diffusion models commonly depends on costly reward-model training, while directly applying DPO is hindered by denoising-time GPU memory demands. D3PO reformulates denoising as a multi-step MDP for direct preference-based updates, achieving competitive or better performance than reward-model fine-tuning while reducing distortion and improving safety.

  • Problem

    Diffusion-model RLHF requires costly reward models built from extensive images and human preferences, while direct DPO application is typically unsustainable because diffusion latents consume substantial GPU memory.

  • Method

    D3PO views denoising as a multi-step MDP and directly updates diffusion-model parameters from human preferences without training a separate reward model.

  • Results

    D3PO achieves competitive or better performance than diffusion models fine-tuned with reward models trained on large image-preference datasets.

  • Takeaways & Limitations

    Human feedback can directly fine-tune diffusion models while addressing hand and full-body deformities, image safety, and prompt-image alignment.

  • Takeaways & Limitations

    Reward-based fine-tuning remains difficult when rewards must evaluate highly noisy intermediate denoising states, especially near Gaussian noise.

Abstract

from arXiv · show

Using reinforcement learning with human feedback (RLHF) has shown significant promise in fine-tuning diffusion models. Previous methods start by training a reward model that aligns with human preferences, then leverage RL techniques to fine-tune the underlying models. However, crafting an efficient reward model demands extensive datasets, optimal architecture, and manual hyperparameter tuning, making the process both time and cost-intensive. The direct preference optimization (DPO) method, effective in fine-tuning large language models, eliminates the necessity for a reward model. However, the extensive GPU memory requirement of the diffusion model's denoising process hinders the direct application of the DPO method. To address this issue, we introduce the Direct Preference for Denoising Diffusion Policy Optimization (D3PO) method to directly fine-tune diffusion models. The theoretical analysis demonstrates that although D3PO omits training a reward model, it effectively functions as the optimal reward model trained using human feedback data to guide the learning process. This approach requires no training of a reward model, proving to be more direct, cost-effective, and minimizing computational overhead. In experiments, our method uses the relative scale of objectives as a proxy for human preference, delivering comparable results to methods using ground-truth rewards. Moreover, D3PO demonstrates the ability to reduce image distortion rates and generate safer images, overcoming challenges lacking robust reward models. Our code is publicly available at https://github.com/yk7333/D3PO.

1. Introduction

Diffusion models generate compelling, semantically coherent images, but adapting them with RLHF traditionally requires costly reward-model training. D3PO applies preference optimization directly during denoising to avoid that overhead and address DPO’s memory challenge.

  • RLHF is used to adapt diffusion models for aesthetic quality, image complexity, and prompt-image alignment.
  • Reward-model approaches require substantial image datasets, human evaluations, and costly training effort.
  • DPO avoids separate reward-model training by directly fine-tuning models from human preferences, but diffusion latent representations make its direct application typically unsustainable in memory.
  • D3PO formulates denoising as a multi-step MDP and updates diffusion-model parameters at each denoising step using human feedback.
  • The method directly uses human feedback, bypasses reward-model training, and is presented as equivalent to learning an optimal reward model before policy updates.
  • Experiments report improvements in hand and full-body deformities, generated-image safety, and prompt-image alignment.

2. Related Work

Related work spans diffusion guidance, RLHF, reward-based diffusion fine-tuning, and preference optimization. These approaches motivate D3PO’s focus on reducing dependence on extensive human-evaluated datasets and separately trained reward models.

  • Diffusion models are used for realistic image synthesis from textual descriptions, while adapters and compositional methods improve control over generation.
  • RLHF has been applied across gaming, robotics, and language modeling when objectives are difficult to define explicitly.
  • Reward-based diffusion methods use reinforcement learning or reward-weighted objectives to improve image quality, complexity, and human-preference alignment.
  • Reward-model approaches demand substantial image datasets and extensive human evaluations.
  • DPO fine-tunes language models from preferences rather than training a separate reward model.

3. Preliminaries

The preliminaries define MDPs, diffusion denoising, and preference-based reward learning. Together, they establish the sequential decision framework and human-preference modeling that D3PO extends.

  • MDP: An MDP models states, actions, transitions, scalar rewards, and a policy that maximizes expected cumulative trajectory returns.
  • Diffusion models: Diffusion models learn data distributions by reversing a Markovian noise-adding process through iterative denoising.
  • Diffusion models: For conditional generation, diffusion models incorporate conditioning information such as image categories and captions into the denoising network.
  • Reward learning for preferences: Preference learning represents human judgments over trajectory segments using a reward function and a Bradley-Terry preference distribution.
  • Reward learning for preferences: A reward network is trained from binary human preferences recorded as triples containing two segments and their preference label.

4. Method

D3PO treats diffusion denoising as a multi-step MDP and extends DPO to update the model directly from human preferences without a separate reward model. It addresses memory limitations by training on sub-segments and using every state-action pair during denoising.

  • Denoising process as a multi-step MDP: D3PO reformulates diffusion denoising as a multi-step MDP, mapping diffusion states, transitions, and policies to reinforcement-learning components.This formulation enables policy optimization over the denoising trajectory.
  • Direct Preference Optimization for MDP: Directly applying DPO to diffusion models is impractical because preference segments contain 20–50 latent images and can require more than 100G GPU memory.Human preferences are available only for final images, while intermediate segment representations are large.
  • Direct Preference Optimization for MDP: The method extends DPO to MDPs, updating the diffusion policy from human preferences while avoiding the computational cost and separate reward-model training.The framework uses a reference-policy deviation constraint controlled by β.
  • Direct preference for Denoising Diffusion Policy Optimization: Figure 3 evaluates D3PO against existing methods across sample-pair updates using image-size, negative-image-size, and LAION aesthetic-score rewards over 5 seeds.The reward definition depends on whether the objective is incompressibility, compressibility, or aesthetics.
  • Direct preference for Denoising Diffusion Policy Optimization: Because humans judge final images rather than noisy intermediate states, D3PO assumes every state-action pair in a preferred segment is better than the corresponding pair in another segment.This assumption constructs T sub-segments from one preference comparison for training.
  • Direct preference for Denoising Diffusion Policy Optimization: Using all state-action pairs increases the segment’s data utilization by a factor of T compared with updating only the initial state-action pair.Equation (14) trains across sub-segments indexed by i ∈ [0, T − 1].

5. Experiment

The experiments evaluate D3PO on quantitative objectives and human-feedback tasks, including image quality, distortion reduction, safety, and prompt-image alignment. D3PO achieves comparable quantitative-objective performance while improving image normalcy, safety, and alignment.

  • 5.1. Pre-defined Quantifiable Objectives Results: D3PO is evaluated first on quantitative objectives, using relative objective values to establish preferences and metric growth to measure effectiveness.The setup benchmarks D3PO against DDPO, DPOK, and Reward Weighted, which require reward models.
  • 5.1. Pre-defined Quantifiable Objectives Results: D3PO achieves results nearly on par with methods trained using standard rewards for compressibility, incompressibility, and aesthetic quality.The experiments use image size for compressibility objectives and the LAION aesthetics predictor for aesthetic ratings.
  • 5.2.1 Reduce Image Distortion: Across five epochs with 1,000 hand images categorized per epoch, fine-tuning reduces hand deformities and increases images with the correct number of fingers.The evaluation tracks normalcy rate rather than deformity rate.
  • 5.2.1 Reduce Image Distortion: Anime-character distortion rates decrease substantially at first and stabilize during later epochs, with visual comparisons shown between pretrained and fine-tuned outputs.The comparison uses the Anything v5 model and assumes non-deformed outputs are more favorable than deformed ones.
  • 5.2.2 Enhance Image Safety: After fine-tuning on human safety annotations across 10 epochs, the model consistently produces safe images.The study generated 1,000 images per epoch using unsafe prompts and used two annotators.
  • 5.2.3 Prompt-Image Alignment: Prompt-image alignment is evaluated through human comparisons and quantitative metrics, with D3PO reported as achieving the best performance across Table 1 comparisons.Figure 7 uses 300 prompts, five raters per image, and reports favorable-evaluation percentages, including majority-vote percentages.

6. Conclusion

The conclusion presents D3PO as a direct human-feedback method for fine-tuning diffusion models without a separate reward model. It frames denoising as a multi-step MDP and reports competitive performance alongside improvements in distortion, safety, and prompt-image alignment.

  • 6. Conclusion: D3PO fine-tunes diffusion models directly from human feedback without learning a separate reward model.The method applies a DPO-style optimization formula during the denoising process.
  • 6. Conclusion: D3PO models denoising as a multi-step MDP and updates parameters at each denoising step using the reference and fine-tuned models.The formulation is intended to reduce GPU memory overhead relative to directly applying DPO.
  • 6. Conclusion: The method achieves competitive or better performance than reward-model-based fine-tuning across image compressibility, incompressibility, and aesthetic quality.The conclusion also reports benefits for reducing image distortion, improving image safety, and aligning prompts with images.

B. Proof

This proof section states the RL objective and identifies the optimal solution through KL-divergence minimization. The key condition is that the divergence reaches zero exactly when the two distributions are identical.

  • B. Proof: The RL objective is expressed in terms of the action-value function and a KL-divergence regularization term.The supplied passage preserves only the objective’s partial expression, including βQ*(s, a).
  • B. Proof: Because KL-divergence is minimized at zero if and only if the two distributions are identical, the proof derives the optimal solution from distribution matching.The optimality argument depends on equality of the compared distributions.

B.2. Proof of Proposition 2

The proof analyzes the preference-probability estimator using expectations, second moments, variance, and Chebyshev’s inequality. It derives a probability bound for the estimator’s deviation from the target preference probability.

  • B.2. Proof of Proposition 2: The expected estimated preference probability is written as a ratio involving exp(Q1) and exp(Q0).The expression is identified as E[˜p*(σ1 ≻ σ0)].
  • B.2. Proof of Proposition 2: The derivation computes the second moment using expectations of exponential Q terms and cross-products.The intermediate expressions expand terms involving Q1, Q0, and σ^2.
  • B.2. Proof of Proposition 2: The variance calculation is simplified into an expression containing exp(σ^2) − 1 and the normalization factor [exp(Q1) + exp(Q0)]^4.The proof also uses the equality of variances for complementary preference probabilities.
  • B.2. Proof of Proposition 2: The proof applies the Chebyshev inequality to bound the probability that the estimated preference probability deviates from the target by less than t.The bound depends on ξ, σ, and the exponential term exp(σ^2).

C. Prompts of Experiments

The experiments use task-specific prompts spanning animal generation, hand distortion, anime-image distortion, safety, and prompt-image alignment.

  • Anime-image distortion experiments use 442 prompts reverse-inferred from retained illustrations collected from anime games.The study initially collected 483 illustrations and filtered out non-character roles, retaining 442 images.
  • Safety experiments use ambiguous and potentially misleading terms to prompt image generation.
  • Prompt-image alignment experiments use 10,000 prompts covering arts, people, outdoor scenes, animals, and other categories.

D. More Samples

Additional samples illustrate how fine-tuning affects objective alignment and prompt-image quality across several generation settings.

  • Samples compare D3PO fine-tuned, preferred-image fine-tuned, and reward-weighted fine-tuned models in the reported experiments.
  • Fine-tuned images more closely align with specified compressibility, incompressibility, and aesthetic-quality objectives than pretrained outputs.

E. Implementation Details and Experimental Settings

The experiments use LoRA-based UNet fine-tuning, specified optimization settings, human-selected images, evaluation metrics, and qualitative samples for distortion and safety.

  • Implementation: LoRA updates UNet attention-layer key, query, and value linear parameters while keeping the text encoder and autoencoder frozen.This substantially mitigates memory consumption.
  • Experimental procedure: Distortion and safety experiments generate seven images per prompt, then select distorted or unsafe images using an open-source tagging website.Prompt-image alignment instead generates two images per prompt and selects the better image with the same website.
  • Hyperparameters: D3PO uses learning rate 3e-5, Adam optimization, gradient clipping at 1.0, 20 inference timesteps, and classifier-free guidance weight 5.0.
  • Evaluation: Prompt-image matching is evaluated with CLIP score, BLIP score, and ImageReward.
  • Qualitative results: Fine-tuning increases the occurrence of normal five-fingered hands, although some hand deformities remain.
  • Qualitative results: Samples generated from unsafe prompts contain no explicit-content images after fine-tuning.
  • Qualitative results: After alignment fine-tuning, generated images better match prompt descriptions and become more aesthetically pleasing.
  • Qualitative results: The appendix includes additional image samples and a website interface for tagging images as deformed or unsafe.
Loading 2311.13231v3…