Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization

• SFT is more stable but has weaker generalization than RL, while common RL methods may require preference data or reward models.Reward models add computational cost and risk reward hacking; comprehensive feedback on visual content remains challenging.Source: §1 · pp. 1–2
• Random Checkpoint Replay constructs paired data from historical checkpoints to address the choice of replay distribution and whether sampled examples are truly negative.The introduction reports that uniform sampling from previous checkpoints performs comparably to using the previous iteration’s checkpoint while providing more stable training.Source: §1 · p. 2
• Self-Sampling Regularization evaluates generated-sample quality and determines whether the current sample is used for DPO or SFT.Source: §1 · p. 3
• SSPO outperforms previous optimization methods on text-to-image benchmarks and is effective on text-to-video benchmarks.Source: §1 · p. 3

• Prior self-improvement methods use demonstrated responses as winners and generate losers, but may require human preference data or assume demonstrations are always preferred.The cited approaches also focus on text-modality transformer models, and their reference-model selection is fixed.Source: §2 · p. 3
• Reward-model-free DPO-style methods avoid using a reward model during training, but the reviewed approaches require human-annotated preference pairs and consider off-policy data.The passage describes Diffusion-DPO, Diffusion-RPO, MaPO, and Diffusion-NPO.Source: §2 · p. 3
• The reviewed prior diffusion self-improvement work is limited to text-to-image generation, while the passage describes SSPO’s target as text-to-visual alignment.Source: §2 · pp. 3–4

3.1 Denoising Diffusion Probabilistic Model

• DDPM’s forward process gradually adds Gaussian noise to a sample, and the model estimates that noise to optimize an ELBO-related loss.The passage describes t as uniformly sampled and notes that the noise-variance weight is usually set to a constant in practice.Source: §3.1 · p. 4

3.2 Direct Preference Optimization

• Conventional RLHF trains a reward function from human preferences and maximizes cumulative reward subject to a KL constraint; DPO converts this target into a loss minimization.The weight of the KL constraint is controlled by β.Source: §3.2 · p. 4

4.1 Random Checkpoint Replay

• In experiments, initial-checkpoint replay overfits as training steps increase, latest-checkpoint replay is unstable, and uniform replay is selected for performance and stability.The experiments use Pick-a-Pic and Stable Diffusion 1.5, saving a checkpoint every 30 updates.Source: §4.1 · p. 5, Figure 2 · p. 5
• Uniform sampling from previous checkpoints has a PAC-Bayesian upper bound on generalization loss no greater than selection of the latest checkpoint.The paper adopts uniform replay from all previously saved checkpoints as Random Checkpoint Replay.Source: §4.1 · p. 5

4.2 Self-Sampling Regularization

• Self-Sampling Regularization evaluates replayed samples relative to a winning-image anchor and uses that assessment in the loss function.The criterion addresses uncertainty about whether replayed samples are genuinely losing relative to the current policy.Source: §4.2 · p. 6
• The sign criterion determines whether SSPO performs DPO updates or updates equivalent to SFT.The passage says SSR’s effectiveness is assessed through an ablation study and an SSR Rate.Source: §4.2 · p. 6

5.1.1 Text-to-Image

• Text-to-image experiments use SD-1.5, Pick-a-Pic v2 training data, and Pick-a-Pic validation, Parti-prompt, and HPSv2 evaluation datasets.The training set contains 851,293 preference pairs and 58,960 unique prompts after removing tied pairs.Source: §5.1.1 · p. 7

5.1.2 Text-to-Video

• Text-to-video experiments train on MagicTime with AnimateDiff and evaluate on ChronoMagic-Bench-150 using 256 × 256 resolution and 16 frames per video.The setup uses LoRA, a learning rate of 5 × 10^-6, and 1,000 training steps.Source: §5.1.2 · p. 7

5.2.1 Analysis of Text-to-Image

• SSPO outperforms previous methods across most Pick-a-Pic validation metrics, including methods that use reward models.The comparison also does not require SPIN-Diffusion’s three-stage training process or complex hyperparameter selection.Source: §5.2.1 · p. 8, Table 1 · p. 9
• SSPOr, trained on randomly selected winner and loser images, still outperforms other methods on most metrics and exceeds winning-image SFT on three key metrics.SSPOr uses a lower-quality training-data distribution than training on winning images alone.Source: §5.2.1 · p. 8
• SSPO outperforms other models on HPSv2 and Parti-prompt, and on SDXL achieves higher PickScore, HPSv2, and Image Reward than baselines.SPO has a slightly higher Aesthetic score on SDXL; its results on HPSv2 and Parti-prompt were reproduced from available checkpoints.Source: §5.2.1 · p. 8, Table 2 · p. 9, Table 3 · p. 9
• A low early SSR rate supports learning from positive samples, while a rising later rate indicates the current model has learned the sample distribution.SSR rate is defined as (#sign = −1)/(#total) in a batch; higher values indicate ERD samples are more likely to be positive relative to the current model.Source: §5.2.1 · p. 8, Figure 3 · p. 9

5.2.2 Memory and Time Cost

• SSPO’s GPU usage and execution time are comparable to SPIN and DPOK, second only to DPO, and it adds only iteration count K versus Diffusion-DPO.The authors attribute reduced reward-model GPU and inference overhead to SSPO not requiring a reward model.Source: §5.2.2 · p. 10, Table 4 · p. 11

5.2.3 Analysis of Text-to-Video

• On ChronoMagic-Bench-150, SSPO improves all reported metrics over vanilla and SFT AnimateDiff, with visual results showing higher text alignment and realism.The text-to-video evaluation reports FID, PSNR, FVD, MTScore, and CHScore.Source: §5.2.3 · p. 10, Figure 6 · p. 11, Table 5 · p. 12

6.1 Limitations

• The theoretical analysis of exploration in reference-constrained diffusion policy space remains open.The paper also leaves consideration of pixel space, including images before encoding, for future work.Source: §6.1 · p. 12

6.2 Conclusion

• SSPO combines SFT and DPO, using checkpoint replay to construct paired data and sample-quality assessment to switch between DPO and SFT.The conclusion says the strategies mitigate overfitting risk and prevent learning biases from non-negative samples.Source: §6.2 · p. 12

Broader Impact Statement

• SSPO reduces dependence on expensive human annotations or external reward models, but the paper warns that aligned models could be misused to generate misleading or harmful content with higher perceptual quality.Source: Broader Impact Statement · p. 12
• Autonomous preference-aligned learning may encode and amplify biases in training data without human oversight.Source: Broader Impact Statement · pp. 12–13

• The analysis defines policy generalization and empirical losses, then invokes a PAC-Bayesian lemma bounding generalization error with confidence radius Δ.Source: §A.1 · p. 17
• The proof assumes Gaussian initial policies and simplifies convergence as linear to compare the bounds for uniform and latest-checkpoint sampling.It notes that Gaussian initialization aligns with practical implementations.Source: §A.1 · pp. 17–18
• The proof represents the latest checkpoint and uniform sampling over previous checkpoints as distinct posterior distributions for SPIN and SSPO.The latest checkpoint is the K-th checkpoint; uniform sampling averages over previous checkpoints.Source: §A.1 · p. 17
• The proof derives an upper bound on generalization error for uniform sampling and compares it with the latest-checkpoint bound.Source: §A.1 · p. 17
• For large K, the average loss can approach the final loss, so the difference between bounds is dominated by the second term ε2 := CK − CUNI.Source: §A.1 · p. 17
• Uniform sampling has a lower confidence upper bound than latest-checkpoint selection, indicating smaller loss and less overfitting risk.Source: §A.1 · p. 18

• The proof measures DDPM recovery by the KL distance between the recovered image and the original, defining better models by lower divergence.Source: §A.2 · p. 18
• The argument uses equal sample variances so Gaussian KL divergence is proportional to the Euclidean distance between means, then applies Jensen’s inequality for squared error.Source: §A.2 · p. 18
• A model prediction with smaller KL distance to the original image recovers better samples and has a higher probability of generating a good result.Source: §A.2 · p. 18

• SSPO achieves the highest PickScore and HPSv2 scores on the Parti-prompt dataset while maintaining high Aesthetic and stable IR.The authors describe its overall evaluation as balancing generation quality and alignment with human needs.Source: Table 7 · p. 19, §B.1 · p. 19

B.2 Reward Distribution

• SSPO's mean and median rewards are relatively higher than those of other methods, with the reward distribution shifted to the right.The authors say this suggests better returns in most scenarios under the same training environment and task setup.Source: Figure 7 · p. 19, §B.2 · p. 19
• SSPO's flatter reward-distribution curve has a less pronounced peak, which the authors interpret as greater training stability.They describe the distribution as more uniform, with fewer extremely high or low values.Source: Figure 7 · p. 19, §B.2 · p. 19
• SSPO's more concentrated reward distribution avoids excessive dispersion while retaining a higher average reward, which the authors associate with greater robustness.Source: Figure 7 · p. 19, §B.2 · p. 20

B.3 Human Evaluation

• Human evaluation on Pick-a-Pic reports SSPO's superiority over SPIN-diffusion, using the average percentage of images rated favorably by five raters.Figure 8 presents the reported evaluation results.Source: §B.3 · p. 20, Figure 8 · p. 20

B.4 Step-wise Performance for SFT and SSPO

• SSPO steadily improves PickScore during training, while SFT quickly converges and begins to fluctuate.The trajectories are shown in Figure 9.Source: §B.4 · p. 20, Figure 9 · p. 20

B.5 Pseudocode

• The training pseudocode specifies an SFT MSE branch and an SSPO branch using preferred and unpreferred losses, reference-policy predictions, and a log-sigmoid loss.Source: §B.5 · p. 21

B.6 Visual Generation Examples

• The appendix presents text-to-image comparisons among SD-1.5, SPO, SPIN-Diffusion, and SSPO, plus text-to-video examples comparing AnimateDiff (Raw), SFT, and SSPO.Figures 10–12 show the example generations for the listed prompts.Source: §B.6 · p. 21, Figure 10 · p. 22, Figure 11 · p. 23