Source-linked AI summary
Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
Minghao Fu, Guo-Hua Wang, Tianyu Cui, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang
TL;DR
Widening the winner–loser preference margin in diffusion DPO does not necessarily improve absolute generation quality, and standard training can increase both branches’ errors. Diffusion-SDPO adaptively scales the loser gradient using its alignment with the winner gradient to safeguard the preferred output’s loss. Across multiple models and generation settings, the paper reports consistent gains on preference, aesthetic, and prompt-alignment metrics.
Problem
Diffusion preference optimization can widen the winner–loser margin without improving absolute quality, and standard Diffusion-DPO may increase both branches’ denoising losses.
Source: §1 · p. 1Method
Diffusion-SDPO uses a first-order, closed-form safeguard to adaptively scale the loser gradient according to its alignment with the winner gradient.
Source: §1 · p. 2, §4.2 · p. 4Results
Across SD 1.5, SDXL, and Ovis-U1, Diffusion-SDPO reports consistent improvements in automated preference, aesthetic, and prompt-alignment metrics with negligible computational overhead.
Source: §6 · p. 8Takeaways & Limitations
The method is model-agnostic and applicable to multiple DPO variants, preserving the preferred branch while improving preference matching.
Source: §6 · p. 8The safeguard relies on a first-order approximation that weakens under strong curvature, and noisy or biased gradient-alignment estimates can cause imperfect loser-gradient scaling.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Text-to-image diffusion models deliver high-quality images, yet aligning them with human preferences remains challenging. We revisit diffusion-based Direct Preference Optimization (DPO) for these models and identify a critical pathology: enlarging the preference margin does not necessarily improve generation quality. In particular, the standard Diffusion-DPO objective can increase the reconstruction error of both winner and loser branches. Consequently, degradation of the less-preferred outputs can become sufficiently severe that the preferred branch is also adversely affected even as the margin grows. To address this, we introduce Diffusion-SDPO, a safeguarded update rule that preserves the winner by adaptively scaling the loser gradient according to its alignment with the winner gradient. A first-order analysis yields a closed-form scaling coefficient that guarantees the error of the preferred output is non-increasing at each optimization step. Our method is simple, model-agnostic, broadly compatible with existing DPO-style alignment frameworks and adds only marginal computational overhead. Across standard text-to-image benchmarks, Diffusion-SDPO delivers consistent gains over preference-learning baselines on automated preference, aesthetic, and prompt alignment metrics. Code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO.
1. Introduction
The introduction identifies a gap between widening the winner–loser preference margin and preserving absolute generation quality. Diffusion-SDPO addresses it by adaptively scaling the loser gradient to safeguard the preferred output.
1. Introduction
- Existing diffusion DPO methods optimize pairwise human comparisons, but increasing the winner–loser margin does not necessarily improve absolute generation quality.DPO has been adapted to diffusion models to improve visual appeal and prompt alignment.Source: §1 · p. 1
- Standard Diffusion-DPO can widen the preference margin while both winner and loser denoising losses rise, so the preferred output may degrade.Figure 1 compares training without and with the safe-λ mechanism and shows generated samples at steps 0, 500, 1000, 1500, and 2000.Source: §1 · p. 1, Figure 1 · p. 2
- Diffusion-SDPO adaptively scales the loser gradient according to its geometric alignment with the winner gradient, guaranteeing to first order that the preferred output’s reconstruction loss does not increase.The closed-form safeguard controls the loser branch’s influence at each step and is designed as a plug-in for existing diffusion alignment frameworks.Source: §1 · p. 2
- “X+SDPO” names an integration of SDPO with a specific base DPO variant, such as Diffusion-DPO, DSPO, or DMPO.“Diffusion-SDPO” is used as an umbrella term for the method and its guiding principles.Source: §1 · p. 2
- The paper reports consistent preference-metric improvements while preserving or enhancing aesthetic quality across SD 1.5, SDXL, and Ovis-U1, including text-to-image, editing, and unified-generation settings.The tested backbones include UNets for SD 1.5 and SDXL and a DiT for Ovis-U1.Source: §1 · p. 2
2. Related Work
The related work situates the method among diffusion-generation approaches and preference-optimization methods. Diffusion-SDPO is distinguished by per-step, geometry-aware scaling that directly controls the winner loss.
Diffusion Models for Text-to-Image and Unified Generation
- Diffusion-SDPO applies to text-to-image and unified generators and is described as architecture-agnostic across UNet- and DiT-style backbones.The passage also reviews latent diffusion, guidance, EDM, rectified flow, and flow matching as developments in diffusion generation.Source: §2 · p. 2
Preference Optimization for Diffusion Models
- Diffusion preference methods include margin calibration, update guidance, positivity constraints, prior-informed objectives, timestep reweighting, and entropy regularization.The passage names DPOP, MaPPO, BalancedDPO, DSPO, and SEE-DPO among related approaches.Source: §2 · p. 3
- Unlike these approaches, Diffusion-SDPO uses a per-step safe scaling factor based on the inner product of winner and loser output-space gradients to control winner loss while expanding the preference margin.The passage places diffusion preference optimization in the context of adapting DPO to human comparisons without full reinforcement learning.Source: §2 · p. 3
3. Preliminaries
The preliminaries describe diffusion denoising and the pairwise preference objective used for diffusion DPO. They also explain why enlarging the preference margin alone does not safeguard the preferred sample’s absolute error.
Diffusion Models
- Diffusion models progressively corrupt clean data with noise and train a denoiser to reverse that process; the standard objective minimizes mean squared error between true and predicted noise.The denoiser receives a noised input and time index, then is applied in reverse order to synthesize samples.Source: §3 · p. 3
Diffusion Model Alignment via Preference
- Diffusion DPO compares preferred and less-preferred images for a prompt using stepwise log-likelihood ratios against a frozen reference model.The preference pair is denoted winner and loser, respectively.Source: §3 · p. 3
- Diffusion-DPO applies Bradley-Terry-style logistic regression to the winner–loser pair at each timestep and averages the objective across timesteps, or samples one timestep for an unbiased estimator.Under DDPM-style Gaussian parameterization, the log-ratio can be expressed through predicted, reference, and ground-truth noise residuals; analogous forms apply to velocity or flow-matching parameterizations.Source: §3 · p. 3
Limitations of Standard DPO
- Although Diffusion-DPO encourages lower winner error and higher loser error at the same timestep, it does not guarantee that winner loss decreases monotonically.The passage reports that the margin can become increasingly negative while both losses rise, exposing a gap between relative alignment and absolute error control.Source: §3 · pp. 3–4
4. Method: Diffusion-SDPO (Safe DPO)
Diffusion-SDPO safeguards the preferred output by adaptively scaling the loser-branch gradient. Its first-order bound is approximated using output-space gradients, avoiding explicit parameter-space gradient calculations.
4. Method: Diffusion-SDPO (Safe DPO)
- Diffusion-SDPO scales the loser-branch gradient by a time-dependent safety factor so the preferred sample’s loss does not increase after each update.It modifies the standard Diffusion-DPO pipeline using per-sample winner and loser losses at the same diffusion time.Source: §4 · p. 4
4.1. Safe Update via First-Order Approximation
- The first-order safety condition bounds the loser weight using the winner and loser loss gradients, preventing an increase in winner loss when their dot product is positive.When the dot product is nonpositive, the update is intrinsically safe for any nonnegative loser weight; otherwise, exceeding the finite threshold can increase winner loss.Source: §4.1 · p. 4
4.2. Closed-Form Safeguard in Output Space
- The output-space safeguard avoids computing full parameter-space gradients, using winner and loser output-loss gradients with a slack parameter to control conservativeness at negligible additional cost.For an appropriate slack choice, its scaling trajectories closely match those from parameter-space gradients.Source: §4.2 · pp. 4–5
- In training, the safety factor scales the backpropagated loser gradient and is clipped to [0, 1]; the same scaling can be used with other methods.When the winner and loser output gradients have a nonpositive dot product, the factor is set to 1.Source: §4 · p. 4, §4.2 · p. 5
5. Experiments
Experiments across SD 1.5, SDXL, and Ovis-U1 evaluate SDPO’s preference, alignment, and editing performance, alongside ablations of its training dynamics, computation, and hyperparameters. SDPO consistently improves reported metrics, while ablations examine its stability and efficiency across settings.
5.1. Experimental Setting
- Experiments finetune SD 1.5 and SDXL on Pick-a-Pic V2 preference pairs, evaluate on Pick V2, HPS V2, and PartiPrompts, and also test Ovis-U1 on mixed preference data and editing benchmarks.Models are finetuned for 2,000 steps; evaluations include automatic preference metrics and, for Ovis-U1, structured alignment and image-editing benchmarks.Source: §5.1 · pp. 5–6
5.3. Ablation Study
- Adaptive λsafe yields further gains over fixed scaling, while fixed λsafe already improves PickScore and HPS over MaPO and DPOP.MaPO weakens absolute-error calibration despite lower computation and memory; DPOP’s thresholded filtering does not fully match diffusion dynamics.Source: §5.3 · p. 7
- Output-space λsafe trajectories closely match parameter-space trajectories; parameter-space computation increases training time by about 72% and peak memory by about 11%.Output-space computation reuses standard backward-pass signals; tuning µ can compensate for its approximation mismatch.Source: Figure 2 · p. 6, §5.3 · p. 7
- Across Diffusion-DPO, DSPO, and DMPO, SDPO stabilizes training trajectories; DSPO’s existing branch regularization means SDPO causes no degradation and typically slightly improves reward results.Without SDPO, the winner loss drifts upward in Diffusion-DPO and DMPO; with SDPO, it drops early and remains low.Source: §5.3 · p. 7
- SDPO maintains near-optimal SDXL performance over a broad µ interval, while SD 1.5 slack choices differ by objective: larger for Diffusion-DPO and DMPO, smaller for DSPO.An early-phase heuristic raises µ if winner loss drifts upward or oscillates, and lowers it if the preference margin stalls despite stable winner loss.Source: §5.3 · p. 7
5.2. Main Results
- Across SD 1.5 and SDXL, SDPO consistently improves automatic reward metrics for Diffusion-DPO, DSPO, and DMPO; Ovis-U1 also improves in preference and editing scores.On SD 1.5, win-rate results further show that SDPO variants outperform the SD 1.5 baseline.Source: §5.2 · p. 6, Table 5 · p. 8
5.3. Ablation Study
- During extended training, SDPO sustains preference-metric gains at a higher level, while baseline generations develop saturation and texture artifacts.The baseline’s HPS V2 and PickScore rise early, then plateau or decline; SDPO trajectories continue gaining and stabilize higher.Source: §5.3 · p. 8
6. Conclusions and Limitations
Diffusion-SDPO preserves the preferred branch while improving preference matching, but its safeguard has first-order and gradient-estimation limitations.
6. Conclusions and Limitations
- Across SD 1.5, SDXL, and Ovis-U1, Diffusion-SDPO improves automated preference, aesthetic, and prompt-alignment metrics with negligible overhead.The method is described as model-agnostic, straightforward to implement, and applicable to multiple DPO variants.Source: §6 · p. 8
- The method scales the loser gradient by its alignment with the winner and guarantees, to first order, that winner reconstruction loss does not increase.Source: §6 · p. 8
- The safeguard is a first-order approximation that weakens under strong curvature, while noisy or biased ρ estimates can cause imperfect loser-gradient scaling.The authors report that a suitably chosen µ makes estimation accurate enough in practice and propose second-order or trust-region safeguards as future work.Source: §6 · p. 8
A. Second-Order Considerations of SDPO
The safeguard controls the first-order change in winner loss, while slack contraction and modest update sizes help limit curvature effects in practice.
A. Second-Order Considerations of SDPO
- The guarantee controls the linear Taylor term, but the Hessian contribution can be positive under large curvature, allowing winner loss to rise for non-infinitesimal steps.The Hessian governs how quickly the loss can bend upward along the update direction.Source: A. Second-Order Considerations of SDPO · pp. 10–11
- Slack µ contracts the loser-direction increment and bounds its curvature contribution by no more than the uncontracted update under a local spectral bound.The analysis describes this contraction as an implicit trust-region-style safeguard.Source: A. Second-Order Considerations of SDPO · p. 11
- With small learning rates and large gradient accumulation, effective updates are modest; combined with slack contraction, this explains almost-monotone or gently decreasing winner-loss trajectories.Source: A. Second-Order Considerations of SDPO · p. 11
- Broad µ plateaus keep HPS V2 and PickScore near-optimal, suggesting that winner loss follows first-order geometry in the regions visited during training.The quadratic term appears to act as a small correction rather than a dominant force.Source: A. Second-Order Considerations of SDPO · p. 11
- Parameter-space and output-space trajectories approximately match, suggesting that a scalar slack can absorb remaining second-order discrepancies; an explicit second-order constraint remains future work.Source: A. Second-Order Considerations of SDPO · p. 11
B. Extension: SDPO on FLUX.1-dev
On FLUX.1-dev, SDPO improves reported metrics at the original learning rate, whereas Diffusion-DPO alone either degrades the model or yields negligible gains when made highly conservative.
B. Extension: SDPO on FLUX.1-dev
- The FLUX.1-dev extension applies Diffusion-DPO and SDPO at the loss level to per-step velocity or denoising predictions, using an in-house dataset of 186K preference examples.The corpus spans diverse image and text domains, with short, noisy, or underspecified prompts re-captioned by Qwen-VL-Max.Source: B. Extension: SDPO on FLUX.1-dev · p. 12
- At the default learning rate, naive Diffusion-DPO collapses reward and structured-alignment scores; reducing its learning rate 100-fold restores near-base performance but yields little gain.The authors describe these outcomes as an aggressive-versus-overly-conservative tension when applying Diffusion-DPO directly to large rectified-flow models.Source: B. Extension: SDPO on FLUX.1-dev · p. 12
- Diffusion-DPO + SDPO improves all reported reward metrics and both GenEval and DPG-Bench over the base FLUX.1-dev model.The reported reward metrics are PickScore, HPS V2, Aesthetics, CLIP, and ImageReward; Table 6 covers these alongside the structured-alignment metrics.Source: B. Extension: SDPO on FLUX.1-dev · p. 12, Table 6 · p. 12
C. Other Experimental Results
Additional results show that SDPO improves preference metrics without a quality penalty, with qualitative comparisons illustrating sharper, more faithful generations and greater stability during training.
C. Other Experimental Results
- Across Pick-a-Pic V2, HPS V2, and PartiPrompts, adding SDPO to Diffusion-DPO, DSPO, or DMPO improves PickScore, HPS V2, and ImageReward while preserving or slightly enhancing aesthetic scores.Full reward-score comparisons are reported for SD 1.5 and SDXL, alongside SDXL win-rate comparisons over HPS V2.Source: C. Other Experimental Results · pp. 12–13, Table 7 · p. 14, Table 8 · p. 15
- Qualitative examples on UNet and DiT backbones show sharper details, more faithful layouts, and fewer artifacts with SDPO than without it.The comparisons include SD 1.5 and Ovis-U1 variants finetuned with DPO or SDPO.Source: C. Other Experimental Results · p. 13, Figure 7 · p. 16, Figure 8 · p. 17
- As training progresses, Diffusion-DPO drifts toward saturated or distorted images, whereas SDPO maintains coherent structure and style.Figure 9 compares fixed prompts across training steps; the text attributes SDPO's stability to its winner-preserving update and safe scaling.Source: C. Other Experimental Results · p. 13, Figure 9 · p. 18