PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, Quanquan Gu

arXiv:2402.10210v1cs.LGcs.AIcs.CLcs.CVstat.ML

TL;DR

Preference-based RL for diffusion fine-tuning can require multiple images per prompt, a barrier when datasets provide only one. SPIN-Diffusion fine-tunes a model through self-play against earlier versions, using an objective based on diffusion trajectories. It surpasses SFT from its first iteration and Diffusion-DPO by its second, despite the latter using additional loser-sample data.

  • Problem

    Preference-based RL fine-tuning often requires multiple images per prompt, limiting its feasibility on datasets that contain only one image for each prompt.

    Source: §1 · p. 2

  • Method

    SPIN-Diffusion fine-tunes diffusion models through self-play against earlier versions, using a decomposed objective over the joint distributions of diffusion trajectories that requires score-function estimation.

    Source: §1 · p. 2

  • Results

    SPIN-Diffusion surpasses SFT from its first iteration and outperforms Diffusion-DPO by its second, although Diffusion-DPO uses additional loser-sample data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The method’s dataset efficiency may benefit people with restricted access to datasets containing multiple images per prompt.

    Source: §1 · p. 3

    Computing the marginal image probability pθ(x0|c) directly requires integrating over all trajectories, which is computationally intractable.

    Source: §4 · p. 6

Abstract

from arXiv · show

Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs). While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing a certain volume of data. Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images ("winner" and "loser" images) for each text prompt. In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the diffusion model engages in competition with its earlier versions, facilitating an iterative self-improvement process. Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment. Our experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms the existing supervised fine-tuning method in aspects of human preference alignment and visual appeal right from its first iteration. By the second iteration, it exceeds the performance of RLHF-based methods across all metrics, achieving these results with less data.

1 Introduction

SPIN-Diffusion adapts self-play fine-tuning to diffusion models, addressing the difficulty of using preference-based RL when prompts have only one associated image. It improves on supervised fine-tuning from its first iteration and surpasses Diffusion-DPO by its second.

1 Introduction

  • Preference-based RL can be infeasible for community-sourced datasets that contain only one image per prompt, because it often needs multiple images per prompt.Source: §1 · p. 2
  • SPIN-Diffusion is a supervised fine-tuning method that removes the need for human preference data by having a diffusion-model main player distinguish target-distribution samples from samples generated by an opponent.The opponent can be an earlier copy of the main player, enabling self-play.Source: §1 · pp. 2–3
  • The method handles diffusion-specific challenges by using an objective over intermediate reverse-process images and a stepwise approximation of probabilities with score functions, plus DDIM Gaussian reparameterization.An approximate objective further avoids needing intermediate images for computational efficiency.Source: §1 · p. 3
  • By iteration two, SPIN-Diffusion outperforms Diffusion-DPO, which uses additional data from loser samples; by iteration three, it beats SD-1.5 on PickScore 79.8% of the times and on Aesthetic score 88.4% of the times.The introduction also reports that SPIN-Diffusion surpasses SFT from the first iteration.Source: §1 · p. 3
  • SPIN-Diffusion retains its advantage over current state-of-the-art fine-tuning algorithms even against models trained with more extensive data, supporting its dataset efficiency.The paper notes this may benefit people with restricted access to datasets containing multiple images per prompt.Source: §1 · p. 3

2 Related Work

Related work covers diffusion models and fine-tuning strategies spanning supervised image-text training, preference-based reinforcement learning, and direct preference optimization. SPIN-Diffusion is positioned as an SFT approach using high-quality image-text pairs.

Diffusion Models

  • Text-to-image diffusion models condition image synthesis on text prompts, while Stable Diffusion uses latent variables in generation; obtaining desired content from prompts remains challenging with limited high-quality data.Source: §2 · p. 4

Fine-Tuning Diffusion Models

  • Prior supervised fine-tuning uses high-quality image or text-fidelity datasets, with reported gains in performance and aesthetics across image-generation models.Examples include COCO fine-tuning and SDXL fine-tuning on a private high-quality dataset [Podell et al., 2023].Source: §2 · p. 4
  • Preference-based approaches include reward-model RL [Black et al., 2023; Fan et al., 2023] and implicit-reward methods [Wallace et al., 2023; Yang et al., 2023]; SPIN-Diffusion instead assumes high-quality image-text pairs.Such pairs can come from preference winners or high-reward examples selected with a reward model.Source: §2 · p. 4

3 Problem Setting and Preliminaries

The section introduces conditional diffusion for text-to-image generation and the denoising score-matching objective, then describes SPIN as iterative optimization against an opponent based on the previous model.

3.1 Text-to-Image Diffusion Model

  • DDIM generalizes DDPM and provides the diffusion framework used for text-conditioned image generation; its marginal at step t is Gaussian, and a special noise choice recovers DDPM.The section describes images as samples from a target distribution conditioned on prompt c.Source: §3.1 · p. 5
  • The score function is trained by minimizing the denoising score-matching objective, with noisy images, prompts, and time steps drawn from specified distributions and weights.Source: §3.1 · p. 5

3.2 Self-Play Fine-Tuning

  • SPIN iteratively trains a main player against an opponent; its closed-form opponent copies the previous model, and the objective uses prompt-response SFT data with a convex, monotonically decreasing loss.The players compare target-distribution examples with opponent-generated examples, with the opponent seeking responses indistinguishable from target data to the main player.Source: §3.2 · p. 6

4 Method

SPIN-Diffusion adapts self-play by comparing data and model diffusion trajectories rather than their marginals. It decomposes the objective into score-function terms and derives an approximation that avoids storing all reverse-process intermediates.

4.1 Differentiating Diffusion Processes

  • At iteration k+1, a test function is trained with a generalized IPM to distinguish data-generated from current-model trajectories.The model is then updated against its prior version.Source: §4.1 · pp. 6–7

4.2 Deceiving the Test Function

  • A KL-regularized opponent objective discourages excessive deviation from pθk, and its closed-form target is constrained to diffusion-process distributions.The resulting opponent corresponds to the updated diffusion parameter θk+1.Source: §4.2 · p. 7

4.3 Decomposed Training Objective

  • The full-trajectory objective is decomposed through the backward diffusion process into per-step terms, supporting σt > 0 and remaining valid as σt → 0.The latter limit covers deterministic generation.Source: §4.3 · p. 8

4.4 Approximate Training Objective

  • Jensen’s inequality and convexity yield an upper-bound approximation that moves the timestep average outside ℓ and needs only two consecutive sampling steps.This addresses the memory cost of storing all intermediate images during reverse sampling.Source: §4.4 · pp. 8–9

5 Main Theory

The analysis characterizes the approximate objective through reweighting, matching, and pushing terms, then gives conditions linking SPIN updates to the SFT objective and target-distribution alignment.

5 Main Theory

  • The gradient reweights the loss, matches target-sampled steps to the model mean, and pushes model-generated steps away from the generative process.Under Assumption 5.1, the reweighting term decreases as its loss argument increases.Source: §5 · pp. 9–10
  • If θk is not an LDSM global optimum, some appropriately chosen βt makes it nonminimal for (4.9), forcing θk+1 ≠ θk.The authors infer that SFT with (3.3) cannot improve over SPIN-Diffusion, but do not claim every LDSM global minimum meets the convergence criterion; the result assumes a decreasing, convex loss.Source: §5 · pp. 9–10
  • When pθk(·|x) = pdata(·|x), θk is a global minimum of (4.9) for any λ ≥ 0.The authors describe this as convergence when the model distribution matches the target distribution.Source: §5 · p. 10

6 Experiments

SPIN-Diffusion is evaluated against SD-1.5, SFT, and Diffusion-DPO on text-to-image benchmarks. It improves on the baselines across reported metrics, with further results on image quality and training dynamics.

6.1 Experiment Setup

  • The experiments use SD-1.5, fine-tuned on Pick-a-Pic, with SFT and Diffusion-DPO baselines; evaluation includes Pick-a-Pic, PartiPrompts, and HPSv2.The evaluation reports HPS, PickScore, ImageReward, and Aesthetic scores, selecting the best-scoring image from five generated per prompt.Source: §6.1 · pp. 10–11

6.2 Main Results

  • 91.6% winning rate over SD-1.5 at iteration 3 exceeds SFT’s 73.2% and Diffusion-DPO (ours)’s 84.8%; against Diffusion-DPO (ours), overall winning rate is 70.2%.At iteration 3, SPIN-Diffusion’s winning rate against Diffusion-DPO (ours) is 56.2% on HPS, 86.8% on Aesthetic, 62.4% on PickScore, and 55.8% on Image Reward.Source: §6.2 · pp. 12–13, Figure 4 · p. 14
  • 22.00 PickScore at iteration 3 exceeds 21.45 for both SFT and Diffusion-DPO; SPIN-Diffusion leads all baselines on four Pick-a-Pic test metrics.Figure 2 reports that SPIN-Diffusion beats SFT at iteration 1 and all baselines after iteration 2.Source: §6.2 · p. 11, Figure 2 · p. 12, Table 3 · p. 13

6.3 Qualitative Analysis

  • Across three Pick-a-Pic prompts, SPIN-Diffusion images show improvements in alignment, shading, visual appeal, and detail over the baseline methods.The qualitative comparison includes SPIN-Diffusion iterations 1–3 alongside SD-1.5, SFT, and Diffusion-DPO (ours).Source: §6.3 · p. 13

6.4 Training Dynamics of SFT and DPO

  • After about 50k data, SFT plateaus at 20.8 PickScore and 8.9 average score, versus SPIN-Diffusion’s 21.2 and 9.1; SPIN-Diffusion also beats Diffusion-DPO without loser images.The reported SFT values also include 0.270 HPS and 5.6 Aesthetic, compared with SPIN-Diffusion’s 0.272 HPS and 5.9 Aesthetic.Source: §6.4 · p. 14

7 Conclusion

SPIN-Diffusion iteratively refines diffusion models toward the target distribution, with theoretical and empirical results supporting its performance. Qualitative comparisons also report improved image quality.

7 Conclusion

  • SPIN-Diffusion iteratively refines diffusion models toward the target distribution, and its stationary point aligns with that distribution.The conclusion states that traditional supervised fine-tuning cannot surpass this stationary point.Source: §7 · p. 14
  • SPIN-Diffusion surpasses state-of-the-art fine-tuning methods without additional data, including in settings with one image per text prompt.The conclusion identifies demonstration data as sufficient for the reported evaluations.Source: §7 · p. 14
  • Figure 5’s qualitative comparison reports notable image-quality improvement for SPIN-Diffusion across the shown prompts and model iterations.The figure compares SD-1.5, SFT, Diffusion-DPO, and SPIN-Diffusion iterations 1–3.Source: Figure 5 · p. 15

A.1 Hyperparameters

The appendix reports training settings for SPIN-Diffusion, Diffusion-DPO, and SFT, alongside validation trends as training data increases.

A.1 Hyperparameters

  • SPIN-Diffusion uses 8 NVIDIA A100 GPUs, 512 × 512 images, and AdamW; its learning rate and training steps vary by iteration.The learning rate is 2.0e−5 for iterations 1–2 and 5.0e−8 for iteration 3; the respective step counts are 50, 500, and 200.Source: §A.1 · p. 15
  • SFT uses 4 NVIDIA A6000 GPUs, a global batch size of 512, and a 1e-5 learning rate, with its best checkpoint selected after 2000 steps.Diffusion-DPO uses AdamW, batch size 8, gradient accumulation 32, learning rate 2.0e−5, and 350 training steps.Source: §A.1 · p. 15
  • As training data exceeds 800k examples, SFT reaches its limit quickly while Diffusion-DPO and SPIN-Diffusion continue improving.Figure 6 plots Pick-a-Pic validation results against the number of training data.Source: Figure 6 · p. 16
  • Generation uses guidance scale 7.5 and fixed random seed 5775709.Source: §A.1 · p. 16

A.2 Additional Results

The appendix reports median Pick-a-Pic test scores and detailed winning rates for SPIN-Diffusion and baseline comparisons.

A.2 Additional Results

  • The appendix reports median Pick-a-Pic test scores in Table 4 and winning rates over SD-1.5 and Diffusion-DPO in Tables 5–6.The text says these test results are consistent with Table 3.Source: §A.2 · p. 16

A.3 Additional Ablation Study

The additional studies report benchmark results, compare continued training with updating the self-play opponent, and provide details on the diffusion process and objective derivation.

A.3 Additional Ablation Study

  • The Pick-a-Pic median-score table reports SPIN-Diffusion ahead on HPS, Aesthetic, PickScore, and average score, with the highest winning rates over SD-1.5 and Diffusion-DPO.Source: Table 4 · p. 17, Table 5 · p. 17, Table 6 · p. 17
  • Iteration 2 boosts performance, while continued iteration-1 training reaches a limit quickly and can reduce PickScore, HPS, and average score.Continued iteration-1 training does not guarantee consistent Aesthetic-score improvement; iteration 2 uses the iteration-1 checkpoint as opponent and image source.Source: §A.3 · p. 17, Figure 7 · p. 18
  • Tables 7–10 report SPIN-Diffusion outperforming baselines across PickScore, HPS, ImageReward, and Aesthetic score on PartiPrompts and HPSv2.Figure 8 provides qualitative PartiPrompts comparisons and reports notable image-quality improvement.Source: Table 7 · p. 18, Table 8 · p. 19, Table 9 · p. 19
  • 9.265 on PartiPrompts and 9.326 on HPSv2 are SPIN-Diffusion’s second-iteration average scores, surpassing all other baselines by a large margin.At iteration 1, it is comparable to Diffusion-DPO and surpasses other baselines on both benchmarks.Source: §A.3 · p. 19

B.1 Additional Details of DDIM.

  • The DDIM details define a forward diffusion distribution using a decreasing noise schedule, with α0 = 1 and αT approximately zero; its score functions approximate noise.The forward marginal is stated as q(xt|x0) = N(√αt x0, (1−αt)I).Source: §B.1 · pp. 20–21

B.2 Decoupling Technique

  • The decoupling reformulation reduces the objective to two consecutive sampling steps, t − 1 and t, rather than all intermediate steps.The derivation cites Jensen’s inequality and the dependence on only those two sampling steps.Source: §B.2 · p. 21

B.3 Objective Function of SPIN-Diffusion

  • The objective derivation approximates the backward process with the forward process and simplifies the decoupled objective by taking σt → 0.The expectation is specified over prompts, target images, and sampled noise.Source: §B.3 · pp. 22–23

C Proof of Theorems in Section 5

The proofs establish a fixed point of the objective under one condition and show non-optimality under a small-λ condition. They also use symmetry and a distributional equality in the proof of Theorem 5.3.

C Proof of Theorems in Section 5

  • For 0 < λ < λ0, g(λ) < ℓ(0), leading to the conclusion that θk is not a global optimum of (4.9).The proof states that such a λ0 exists; the condition on βt is given as βt = sγt/h2.Source: C Proof of Theorems in Section 5 · p. 24
  • The proof concludes that (xt−1, xt) and its primed counterpart are independent and identically distributed because pdata(·|c) = pθt(·|c), then invokes symmetry.Source: C Proof of Theorems in Section 5 · p. 24
  • The proof concludes that θk is a global optimum of (4.9), so θk+1 = θk.The objective is simplified by parameterizing with ϵθ, and the proof uses Jensen’s inequality under the convexity assumption on ℓ.Source: C Proof of Theorems in Section 5 · pp. 23–24
Loading 2402.10210v1…