PaperlayerView exact revision on arXiv

Source-linked AI summary

Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization

Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, Liang Zheng

arXiv:2406.04314v4cs.CV

TL;DR

Generic human preferences can conflict with aesthetic judgments, and two-trajectory preference methods make fine visual differences difficult to assess. The paper introduces step-by-step preference optimization (SPO), which uses a step-aware preference model to select close candidate pairs from a shared noisy image. SPO improves aesthetics over DPO methods and converges faster, with applicability limited by its need for stochastic intermediate steps.

  • Problem

    Generic preference datasets can record judgments that conflict with aesthetics, while aesthetic-only annotations are costly and two-trajectory methods make fine visual differences difficult to learn.

    Source: §6 · p. 9

  • Method

    SPO selects a win-lose pair from candidates denoised from the same noisy image at each step, using a step-aware preference model and a randomly chosen candidate to continue denoising.

    Source: §1 · p. 2

  • Results

    SPO improves aesthetics over popular DPO methods on SD-1.5 and SDXL and converges faster than Diffusion-DPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SPO distills aesthetic details from generic preference data without using a dedicated aesthetic preference dataset.

    Source: §4.6 · p. 5

    SPO requires stochastic intermediate steps, so it is not applicable to flow-matching models such as SD3 and Flux.

    Source: §4.6 · p. 5

Abstract

from arXiv · show

Generating visually appealing images is fundamental to modern text-to-image generation models. A potential solution to better aesthetics is direct preference optimization (DPO), which has been applied to diffusion models to improve general image quality including prompt alignment and aesthetics. Popular DPO methods propagate preference labels from clean image pairs to all the intermediate steps along the two generation trajectories. However, preference labels provided in existing datasets are blended with layout and aesthetic opinions, which would disagree with aesthetic preference. Even if aesthetic labels were provided (at substantial cost), it would be hard for the two-trajectory methods to capture nuanced visual differences at different steps. To improve aesthetics economically, this paper uses existing generic preference data and introduces step-by-step preference optimization (SPO) that discards the propagation strategy and allows fine-grained image details to be assessed. Specifically, at each denoising step, we 1) sample a pool of candidates by denoising from a shared noise latent, 2) use a step-aware preference model to find a suitable win-lose pair to supervise the diffusion model, and 3) randomly select one from the pool to initialize the next denoising step. This strategy ensures that diffusion models focus on the subtle, fine-grained visual differences instead of layout aspect. We find that aesthetics can be significantly enhanced by accumulating these improved minor differences. When fine-tuning Stable Diffusion v1.5 and SDXL, SPO yields significant improvements in aesthetics compared with existing DPO methods while not sacrificing image-text alignment compared with vanilla models. Moreover, SPO converges much faster than DPO methods due to the use of more correct preference labels provided by the step-aware preference model.

1. Introduction

The paper targets aesthetic alignment from human preference data, arguing that generic preferences can conflict with aesthetics and that two-trajectory methods obscure fine visual nuances. It introduces SPO to assess details step by step and reports improved aesthetics and faster convergence than DPO methods.

1. Introduction

  • On SD-1.5 and SDXL, SPO improves aesthetics over popular DPO methods and converges faster than Diffusion-DPO.The qualitative comparison describes SPO-SDXL as showing more visual details and appealing styles than Vanilla SDXL and Diffusion-DPO-SDXL.Source: Figure 2 · p. 2, §1 · p. 2
  • Existing DPO methods use clean-image preferences and propagate them to intermediate samples along both generation trajectories.They encourage predictions toward the preferred image and away from the dispreferred image.Source: §1 · p. 2
  • Generic preference labels can conflict with aesthetic judgments, while collecting aesthetic-only preferences is costly because layout and aesthetic opinions are complex.Two-trajectory methods also produce pairs with large layout discrepancies, making aesthetic nuances difficult to learn even with accurate labels.Source: §1 · p. 2
  • SPO discards preference propagation: at each step, a step-aware preference model selects the most and least preferred candidates denoised from a shared noisy image.A randomly selected candidate initializes the next step, while the selected pair’s small differences emphasize fine-grained details.Source: §1 · p. 2

2. Related Work

Prior diffusion-model post-training methods use rewards, policy gradients, or preference optimization to align generated images with human preferences. SPO differs from trajectory-level preference methods by focusing on nuanced visual differences over one or a few denoising steps.

2. Related Work

  • Post-training approaches include reward-weighted fine-tuning, policy gradients, and gradients propagated through differentiable reward models.Examples include AligningT2I [13], DPOK, DDPO [1], ReFL [31], DRaFT [3], and AlignProp [19].Source: §2 · p. 3
  • DPO-inspired methods use image preferences, including Diffusion-DPO [29] on Pick-a-Pic [11], D3PO [32] with model-labeled pairs, and DenseReward [33] with temporal discounting.These methods optimize trajectory-level preference.Source: §2 · p. 3
  • SPO’s step-by-step mechanism focuses on nuanced visual differences within a single or a few denoising steps.The paper contrasts this with trajectory-level methods, where accumulated image differences are too large for the network to focus on aesthetic subtleties.Source: §2 · p. 3

3. DPO Revisit and Diagnosis

The section reviews DPO’s preference-pair loss and its diffusion adaptation, which propagates clean-image preferences through denoising trajectories. It diagnoses a mismatch between those large trajectory differences and the fine-grained aesthetic judgments the paper seeks to learn.

3. DPO Revisit and Diagnosis

  • DPO fine-tunes a conditional generation model using preferred and dispreferred outputs labeled from pairs generated under the same condition.The reference model and sigmoid function appear in the loss, with β controlling regularization strength.Source: §3 · p. 3
  • Diffusion-DPO and D3PO assign final-image preferences across intermediate states in preferred and dispreferred denoising trajectories.Their step-wise loss encourages denoising toward the preferred image and away from the dispreferred one.Source: §3 · p. 3
  • Trajectory pairs often differ substantially in layout, while generic preferences may conflict with aesthetics and dedicated aesthetic preference data is costly to collect.Propagating noisy generic preferences through all diffusion steps can compromise fine-tuning.Source: §3 · p. 3

4. Proposed Approach

SPO replaces propagated clean-image preferences with step-local comparisons of denoised candidates, aiming to use generic preference data to guide aesthetic details. Its framework combines a timestep-aware preference model, random initialization of subsequent steps, and a multi-step extension for stronger models.

4.1. Framework Overview

  • SPO compares candidates denoised from the same intermediate sample, selects the highest- and lowest-quality images as a win-lose pair, and randomly picks a candidate to start the next step.The pair’s small differences are intended to reflect image details and aesthetics.Source: §4.1 · p. 3

4.2. Step-Aware Preference Model (SPM)

  • The step-aware preference model (SPM) scores noisy intermediate images conditioned on timestep and prompt, and is trained to predict preferences after adding the same noise to labeled image pairs.SPM is initialized from PickScore and modified for timestep-conditional prediction.Source: §4.2 · pp. 3–4

4.3. Random Selection of xt to Start Next Step

  • Random selection preserves a shared latent for each win-lose pair; candidate-pool selection is applied only when t ≤ κ, while earlier steps use standard diffusion sampling.The selected candidate initializes the next denoising step.Source: §4.3 · p. 4

4.4. Objective Function of SPO

  • At each denoising timestep, SPO uses the SPM’s most- and least-preferred candidates to form preference pairs and combines per-timestep objectives into a final objective.The preference pairs are optimized using a revised DPO loss.Source: §4.4 · p. 4

4.5. Extension to Multi-step Preference Optimization for SDXL

  • For stronger models such as SDXL, MSPO uses multiple denoising steps to increase candidate diversity when single-step differences are too small and can confuse fine-tuning.The paper reports that the extension provides more distinct samples and clearer preference signals.Source: §4.5 · p. 5, Figure 4 · p. 5

4.6. Discussions and Insights

  • Because diffusion generation has many intermediate steps, it requires DPO designs specific to iterative generation rather than language-model-style single-step prediction.The paper describes SPO as an implicit aesthetic optimizer and distillator: SPM comparisons of similar images emphasize details and distill them from generic opinions.Source: §4.6 · p. 5
  • The authors say generic and aesthetic preferences are difficult to compare quantitatively, and limit their study to crowd-sourced data without addressing subjective, political, or historical aspects of aesthetics.They note that aesthetic disagreements may be more numerous than currently identified.Source: §4.6 · p. 5

5. Experiments

Experiments compare SPO with DPO methods and vanilla models on SD-1.5 and SDXL, including aesthetic and image-text alignment evaluations. They report stronger aesthetic performance, mixed alignment comparisons with Diffusion-DPO, and analyses of SPO’s design and training cost.

5.1. Experimental Setup

  • The evaluation uses Pick-a-Pic preference data, 500 validation prompts, automatic preference and aesthetic metrics, and human comparisons across 300 image pairs.SPO trains its preference model on Pick-a-Pic V1, which has over 580k labeled pairs; fine-tuning uses 4k prompts, while GenEval evaluates 553 test prompts.Source: §5.1 · pp. 5–6

5.2. Main Evaluation on Aesthetic Alignment

  • SPO achieves the best scores across four metrics on SDXL, including Aesthetic score 6.364 and a +0.349 improvement over Diffusion-DPO.The reported metrics are PickScore 23.06, HPSV2 31.80, ImageReward 1.0803, and Aesthetics 6.364; the corresponding improvements over Diffusion-DPO are +0.42, +2.49, +0.1367, and +0.349.Source: §5.2 · p. 6, Table 1 · p. 6
  • In user studies, SPO wins 58.27% of visual-appeal comparisons and produces more visually appealing samples than SDXL and Diffusion-DPO.The studies also report 52.8% wins for general preference; qualitative examples show richer textures and vibrant colors with Glyph-ByT5-SDXL while preserving text-generation ability.Source: §5.2 · p. 7, Figure 6 · p. 7, §5.4 · p. 8

5.3. Evaluation on Image-Text Alignment

  • SPO slightly improves image-text alignment over vanilla SDXL but has mixed results against Diffusion-DPO, which performs better on GenEval.User studies show a tie or slight SPO advantage in prompt alignment; the paper attributes the smaller improvement relative to Diffusion-DPO to SPO’s limited capture of layout changes.Source: Table 3 · p. 6, §5.3 · p. 7

5.4. Further Analysis

  • Removing timestep conditioning from SPM or replacing it with PickScore lowers performance, while random initialization better preserves diverse trajectories than repeatedly choosing win samples.Repeatedly selecting win or lose samples biases training toward those intermediate samples and can deteriorate generalization.Source: §5.4 · pp. 7–8, Table 5 · p. 7, Table 4 · p. 7
  • Applying SPO to timesteps [0-750] performs better than using very noisy [750-1000] steps, with [250-500] appearing especially useful.Removing [0-250] causes a considerable performance drop; the authors select κ = 750 and apply SPO over [0-750].Source: Table 8 · p. 7, §5.4 · p. 8
  • Choosing the highest- and lowest-quality candidates generally beats random pair selection; increasing candidate count strengthens detail contrasts, but too many candidates can degrade performance.The authors choose k = 4; they also report that MSPO performance initially increases with inner steps before saturating, and select j = 4.Source: Table 9 · p. 7, Table 6 · p. 7, Table 7 · p. 7
  • SPO-SD-1.5 and SPO-SDXL use about 20.8% and 4.9% of Diffusion-DPO’s total training GPU hours, respectively.The authors suggest detail-focused learning probably contributes to the efficiency gain.Source: §5.4 · p. 8

6. Conclusion

The conclusion identifies data and method challenges in aligning diffusion models with aesthetic preference. It presents SPO as an approach that distills aesthetics from generic preference data and reports improved aesthetic alignment on SD-1.5 and SDXL.

6. Conclusion

  • SPO better aligns SD-1.5 and SDXL with human aesthetic preference than other DPO methods and is efficient to train.The conclusion frames this as an experimental finding for these two model families.Source: §6 · p. 9
  • SPO forms preference pairs from the same image after one or a few denoising steps, making their relatively small differences reflect image details or aesthetics.The preference model captures these differences to guide the model toward better image details.Source: §6 · p. 9
  • The conclusion identifies large differences in two-trajectory methods and the cost of collecting aesthetic-only preference data as challenges to aesthetic alignment.Existing generic preference datasets may record preferences that conflict with aesthetic preference.Source: §6 · p. 9

Supplementary Material

The supplementary material includes SDXL-generated examples illustrating disagreement between generic preference and aesthetic preference.

Supplementary Material

  • One illustrated prompt describes a glazed donut with sprinkles, vibrant colors, and soft lighting.Source: Supplementary Material · p. 11
  • Figure 8 presents two SDXL-generated examples in which the generic-preference win trajectories have inferior aesthetics or details.Source: Figure 8 · p. 11

1. More Examples of Disagreements Between Generic and Aesthetic Preferences

The examples show that generic preference can favor prompt correctness even when aesthetic preference favors cleaner details or more coherent color. The supplementary material also presents disagreements found in Pick-a-Pic V1.

1. More Examples of Disagreements Between Generic and Aesthetic Preferences

  • For the purse prompt, generic preference favors the image matching the prompt more closely, while aesthetic preference favors the image with cleaner details.The upper trajectory has artifacts on the purse; the lower trajectory has clean details.Source: §1 · p. 11
  • For the donut prompt, generic preference favors the correct number of donuts, but aesthetic preference favors the image with more consistent inner and background colors.Source: §1 · p. 11
  • Pick-a-Pic V1 contains examples where generic preference labels disagree with aesthetic preference, and the paper says these disagreements hinder aesthetic improvement.Figure 9 shows examples across varied prompts, including a blue fox, racing cars, and a suit of armor made from meat.Source: §1 · p. 11, Figure 9 · p. 11

2. Timestep-conditional CLIP Vision Encoder

The CLIP vision encoder is adapted with timestep-conditioned ViT blocks that receive shared timestep embeddings. Dimension-wise modulation is initialized to preserve PickScore’s output at the start of training.

2. Timestep-conditional CLIP Vision Encoder

  • Timestep-conditioned ViT blocks replace the CLIP vision encoder’s original transformer blocks, with sinusoidal timestep embeddings passed through a Linear-SiLU-Linear MLP and shared across blocks.Figure 10 distinguishes original blocks in blue from newly added components in red.Source: §2 · pp. 11–12, Figure 10 · p. 12
  • Dimension-wise γ, β, and α parameters modulate features through Scale, Shift after layer normalization and Scale after attention and feedforward blocks, before residual connections.Scale, Shift computes x = x × (1 + γ) + β; Scale computes x = x × α.Source: §2 · p. 12
  • Initializing γ = 0, β = 0, and α = 1 makes the SPM output match pretrained PickScore at the start of training.The SPM is initialized with PickScore weights.Source: §2 · p. 12

3. Detailed Prompts

Table 10 collects the detailed prompts used for the generated images in Figure 2.

3. Detailed Prompts

  • Table 10 lists the detailed prompts used for generated images in Figure 2.Source: §3 · p. 12

4. More Sample Images Generated by SPO-SDXL

The supplementary material presents additional images generated by SPO-SDXL and provides the corresponding prompts. The paper describes the samples as visually attractive and high quality.

4. More Sample Images Generated by SPO-SDXL

  • The supplementary text also identifies Table 10 as containing detailed prompts for the generated images in Figure 2.Source: Table 10 · p. 13
  • Table 11 lists the prompts corresponding to the additional Figure 11 samples.Source: §4 · p. 12, Table 11 · p. 13
  • Figure 11 presents additional SPO-SDXL sample images, described as high-quality, visually attractive, and stunning.Source: §4 · p. 12, Figure 11 · p. 14
Loading 2406.04314v4…