PaperlayerView exact revision on arXiv

Source-linked AI summary

FestDPO: Few-step Generator Alignment with Direct Preference Optimization

Jaewoo Lee, Kyuil Sim, Hyeongyu Kang, Kanghoon Lee, Woocheol Shin, Jinkyoo Park

arXiv:2609.34673v1cs.LG

TL;DR

Implicit few-step generators make the likelihood evaluation required by standard DPO intractable. FestDPO approximates those likelihoods with sample-based nonparametric density estimation. Across toy, text-to-image, and protein-generation tasks, it reports reward-target matching, strong preference results, and improved protein designability and secondary-structure control.

  • Problem

    Few-step generators are often implicit, making the likelihood evaluation required by DPO intractable.

    Source: §4 · p. 4

  • Method

    FestDPO replaces DPO’s intractable likelihood terms with sample-based nonparametric density estimates, leveraging fast sampling by few-step generators.

    Source: §1 · p. 1, §5.1 · p. 5

  • Results

    FestDPO matches the reward-induced target in toy experiments, leads preference-optimization baselines in text-to-image win rates and human ratings, and improves protein designability and secondary-structure control.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The reported evaluations support FestDPO across few-step generator families and image and protein domains, with robustness to tested sample counts and feature encoders.

    Source: §1 · p. 2

    FestDPO’s density estimation depends on a pretrained encoder’s feature space, although tested encoders yield robust performance.

    Source: §7 · p. 9

Abstract

from arXiv · show

Few-step generative models can generate high-fidelity samples within a few function evaluations. Despite this efficiency, generated samples may not exhibit desirable properties. When these properties are difficult to encode as an explicit reward function, direct preference optimization (DPO) can align generative models using pairwise preference feedback without training a separate reward model. However, extending DPO to few-step generative models is challenging because few-step generative models are generally implicit, making the likelihood evaluation required by DPO intractable. To address this challenge, we introduce Few-step DPO (FestDPO), an extension of DPO for few-step generative models that leverages nonparametric likelihood estimation from empirical samples. By exploiting the fast sampling capabilities of few-step generative models, our approach makes sample-based approximation of DPO loss computationally feasible. Furthermore, the sample-based formulation makes FestDPO agnostic to the model family and sampling procedure. Our toy experiment demonstrates that FestDPO matches the reward-tilted target distribution across four few-step generators. For real-world tasks, we evaluate FestDPO in two domains: text-to-image generation and protein backbone generation. In text-to-image generation, FestDPO outperforms preference optimization baselines in both win rates against the base models and human evaluation scores. In protein backbone generation, it achieves a higher $\beta$-sheet fraction and better structural designability than the baselines.

1 INTRODUCTION

FestDPO addresses the intractable likelihoods that complicate applying DPO to implicit few-step generators. It uses sample-based density estimation and is evaluated on toy, image, and protein-generation tasks.

Motivation

  • Few-step generators can produce high-quality samples quickly, but their outputs may not satisfy user preferences, and explicit rewards can be difficult to specify.Their implicit nature also makes the likelihood evaluation required by DPO intractable.Source: §1 · p. 1

Contribution

  • FestDPO replaces DPO’s intractable likelihood terms with sample-based nonparametric density estimates, exploiting fast few-step sampling.The approach is agnostic to the sampling procedure and pretrained model family, allowing use across few-step generators without model-specific modifications.Source: §1 · pp. 1–2

Method

  • For image and protein domains where raw-space distance misses structure, FestDPO estimates densities in a pretrained encoder’s embedding space.The paper gives semantic similarity in images and rotational invariance in protein structures as examples.Source: §1 · p. 2

Evaluation

  • In a 1-D toy setting, FestDPO learns the reward-induced target density across diverse few-step generators.For text-to-image generation, it achieves the highest win rates among preference optimization baselines and the highest human ratings; for protein generation, it improves structural designability and control over secondary-structure composition.Source: §1 · p. 2

2 RELATED WORK

The related work situates FestDPO among fast but often implicit generators, preference-based alignment methods, and approaches to sampling from unnormalized densities. It distinguishes FestDPO by approximating DPO from a fixed preference dataset without explicit reward evaluations.

2.1 FEW-STEP GENERATIVE MODELS

  • Many few-step generators sample efficiently but lack tractable likelihoods, while noise-to-data generators also lack the denoising trajectories used by score-matching approaches.The section distinguishes direct noise-to-data mappings from flow-map models.Source: §2.1 · pp. 2–3

2.2 ALIGNMENT

  • Prior few-step alignment methods often rely on diffusion-specific formulations or denoising-trajectory likelihoods; DrPO uses preference-derived drift fields without an established connection to DPO.FestDPO instead develops a sample-based DPO approximation and establishes its asymptotic consistency.Source: §2.2 · p. 3

2.3 SAMPLING FROM UNNORMALIZED DENSITY

  • Unlike prior sampling-based alignment methods that use explicit reward evaluations, FestDPO learns an implicit sampler from a fixed preference dataset.The sampling perspective relates alignment to learning from a reward-tilted unnormalized density.Source: §2.3 · p. 3

3 BACKGROUND

The background explains preference optimization through latent rewards and a reference-policy penalty, then presents DPO as direct optimization from preference pairs. DPO’s objective trains a policy toward the reward-tilted distribution.

3.1 PREFERENCE OPTIMIZATION

  • Preference optimization maximizes expected latent reward while penalizing reverse KL divergence from a reference model; earlier approaches learn a reward model and optimize it with PPO.The cited background notes that online RL with a learned reward model can be unstable and hyperparameter-sensitive.Source: §3.1 · pp. 3–4

3.2 DIRECT PREFERENCE OPTIMIZATION

  • DPO avoids a separate reward model and online RL by optimizing the generator directly from preference pairs, and minimizing its objective matches the reward-tilted distribution.The derivation substitutes the closed-form reward expression into the Bradley–Terry preference model, canceling the intractable normalizer.Source: §3.2 · p. 4

4 EXTEND DPO TOWARDS FEW-STEP GENERATIVE MODELS

Standard DPO is difficult to extend to implicit few-step generators because its likelihood evaluations are intractable. FestDPO uses sample-based nonparametric estimation to construct a tractable DPO surrogate.

Challenge

  • Implicit few-step generators make the likelihood evaluation required by DPO intractable, while diffusion-specific likelihood surrogates and denoising-transition likelihoods are generally unavailable for them.The section identifies this as the challenge motivating a new formulation.Source: §4 · p. 4

FestDPO

  • FestDPO approximates DPO with sample-based nonparametric estimation, using efficient few-step sampling and remaining agnostic to pretraining objective and sampling procedure.The paper presents this as a sample-based approximation of DPO for few-step generators.Source: §4 · p. 4

5 METHODS

FestDPO estimates trainable and reference generator likelihoods from samples to construct a tractable surrogate for DPO alignment. Under stated consistency conditions, its objective and minimizer converge to those of DPO, while feature-space kernels can better reflect semantic and geometric similarity.

5.1 FESTDPO

  • FestDPO uses samples from trainable and frozen reference generators to estimate likelihoods at preferred and unpreferred outputs, then substitutes these estimates into the DPO objective.Kernel density estimation makes the surrogate tractable for few-step implicit generators.Source: §5 · p. 4, Figure 2 · p. 5, §5.1 · p. 5

5.2 THEORETICAL ANALYSIS

  • As sample count grows under KDE consistency conditions, the FestDPO objective converges to DPO, and its minimizer converges to the DPO minimizer.The corresponding target is induced by KL-regularized reward maximization.Source: §5.2 · p. 5
  • In the toy experiment, distributions from four FestDPO-fine-tuned generators closely match the reward-tilted target, with KL divergences of approximately 3 × 10^-3.The comparison includes Drifting, IMM, MeanFlow, and sCM.Source: Figure 3 · p. 6

5.3 KERNEL DENSITY ESTIMATION IN FEATURE SPACE

  • Kernel similarities are computed in a pretrained encoder's embedding space to better reflect image semantics and protein geometric similarities.Raw-space kernels may poorly capture image semantics or depend on arbitrary protein rotations and translations.Source: §5.3 · p. 6

6 EXPERIMENTS

FestDPO is evaluated on a 1D toy task, text-to-image generation, and protein backbone generation. The experiments test preference-tilted sampling across these settings.

6.1 TOY EXPERIMENTS

  • Across four few-step generators, FestDPO matches the toy reward-tilted target distribution with approximately 3 × 10^-3 KL divergence while preserving its multimodal structure.The experiment uses preference pairs labeled with a Bradley-Terry model.Source: §6.1 · p. 6

6.2.1 EXPERIMENTAL SETUP

  • Text-to-image experiments fine-tune SDXL-Turbo and SDXL-DMD2 on Pick-a-Pic v2 preferences and evaluate PickScore, Aesthetic score, and win rates across three prompt sets.The evaluation also includes a human study on 80 prompts.Source: §6.2.1 · p. 7

6.2.2 MAIN RESULTS

  • FestDPO achieves the highest win rates on all three prompt datasets for PickScore and Aesthetic score, and improves PickScore rapidly within 150 steps on SDXL-DMD2.The comparisons include SDXL-Turbo and SDXL-DMD2; FestDPO remains stable while DrPO and PSO show slower, smaller improvements.Source: §6.2.2 · p. 7, Figure 4 · p. 7

6.2.3 ABLATION STUDY

  • Increasing KDE sample count from 8 to 24 improves PickScore while Aesthetic scores remain approximately stable; performance is comparable across three feature encoders.Even a small sample count yields substantial gains over the pretrained model.Source: §6.2.3 · p. 8, Table 1 · p. 8, Table 2 · p. 8

6.2.4 HUMAN EVALUATIONS

  • Human evaluators give FestDPO the highest ratings for prompt alignment and aesthetics and consistently prefer its images over baseline outputs.The study evaluates prompt alignment and aesthetic quality.Source: §6.2.4 · p. 8, Table 3 · p. 8

6.3.1 EXPERIMENT SETUP

  • Protein experiments fine-tune single-step RMF-S against DrPO; FestDPO is assessed using β-sheet and scRMSD rewards, with separate 8,000-pair datasets.The SS-match dataset uses one-step samples and the scRMSD dataset uses ten-step samples; a frozen ProteinMPNN encoder supplies structural features for density estimation.Source: §6.3.1 · pp. 8–9

6.3.2 RESULTS

  • FestDPO achieves a higher β-strand residue fraction and lower scRMSD than the base model and DrPO, with better diversity than DrPO but less than the base model.The protein-backbone comparison also shows increased β-sheet content after optimization with the SS-match reward.Source: §6.3.2 · p. 9, Table 4 · p. 9, Table 5 · p. 9

7 DISCUSSION

FestDPO uses sample-based nonparametric estimation to approximate DPO for few-step generators, and its objective converges to DPO as sample counts grow. The paper also connects DPO’s optimum under the Bradley–Terry model to a reward-tilted distribution and notes FestDPO’s dependence on pretrained feature encoders.

7 DISCUSSION

  • FestDPO’s objective converges in probability to DPO’s as the number of samples grows, and its minimizer is consistent.In toy, text-to-image, and protein tasks, the paper reports effective preference optimization, including higher win rates and human ratings for images and improved protein metrics.Source: §7 · p. 9
  • FestDPO estimates densities in a pretrained encoder’s feature space because direct estimation in high-dimensional data space is challenging.This choice makes the method dependent on encoder selection, although the reported ablation finds robustness across different pretrained encoders.Source: §7 · p. 9

A OPTIMAL DISTRIBUTION OF THE KL-REGULARIZED OBJECTIVE

  • Under the Bradley–Terry model, DPO and KL-regularized reward maximization share the reward-tilted distribution as their optimal policy.The appendix derives the optimal policy from the latent reward and reference policy, then relates it to DPO’s preference likelihood objective.Source: A OPTIMAL DISTRIBUTION OF THE… · p. 16

B PROOF OF PROPOSITION 1

The proof establishes convergence of the FestDPO objective to the DPO objective by applying KDE consistency to estimated densities and controlling the per-example losses.

B PROOF OF PROPOSITION 1

  • As sample count N grows with bandwidth h → 0 and Nh^d → ∞, KDE estimates converge at evaluated points where the densities are continuous and positive.The proof assumes a strictly positive bounded kernel satisfying standard pointwise KDE consistency conditions.Source: B PROOF OF PROPOSITION 1 · p. 16
  • KDE consistency implies convergence in probability of the estimated log-density ratios used by the losses.The proof obtains this step by the continuous mapping theorem.Source: B PROOF OF PROPOSITION 1 · pp. 16–17
  • Uniform integrability of per-example FestDPO losses, together with convergence in probability, yields convergence in L1.One sufficient condition given is log-density ratios uniformly bounded in N, ensured when estimated densities are uniformly bounded above and away from zero at evaluated samples.Source: B PROOF OF PROPOSITION 1 · p. 17
  • The resulting L1 convergence establishes convergence of the FestDPO objective to the DPO objective.This conclusion follows from the loss convergence under the proof’s stated assumptions.Source: B PROOF OF PROPOSITION 1 · p. 17

C PROOF OF PROPOSITION 2

The supplied sections describe conditions used to prove minimizer consistency, two preference-alignment baselines, and a toy experiment comparing four few-step generators with a reward-tilted target.

C PROOF OF PROPOSITION 2

  • The minimizer-consistency proof uses uniform convergence, a unique well-separated DPO minimizer, and approximate minimizers of the FestDPO objective.The proof applies the assumed uniform convergence and well-separatedness condition to show convergence of the approximate minimizers.Source: C PROOF OF PROPOSITION 2 · p. 17

D.1 PSO

  • PSO handles intractable marginal likelihoods by comparing relative margins between target and reference samples using joint trajectory likelihoods.It makes transition densities explicit by representing few-step denoising as an MDP with Gaussian policies; in the described offline setting, reference samples are pre-collected.Source: §D.1 · p. 18

D.2 DRPO

  • DrPO constructs a preference-derived drift field and trains the generator by regressing features toward stop-gradient targets.Its reference term uses a mean-shift difference between reference and current-model features; the offline setting uses dataset preference features.Source: §D.2 · p. 18

E.1 TOY EXPERIMENTS

  • After FestDPO fine-tuning, the toy experiment compares Drifting, IMM, MeanFlow, and sCM distributions with the reward-tilted target and reports KL divergence.The reference is a two-component Gaussian mixture, and the target is its reward-tilted distribution; the caption identifies pre- and post-fine-tuning distributions but gives no KL values.Source: Figure 7 · p. 19, §E.1 · p. 19
  • The toy setup uses a shared fixed offline preference dataset and evaluates Drifting with one sampling step and IMM, MeanFlow, and sCM with four.The dataset contains 10^5 samples for pre-training and 10^5 preference pairs for fine-tuning; the target is available in closed form.Source: §E.1 · p. 19

E.2.1 DATASETS DETAILS

The supplied material specifies image evaluation datasets and implementation settings, protein optimization targets and preference-pair construction, and the scope of additional image results tables.

E.2.1 DATASETS DETAILS

  • The image evaluations use 500 prompts each from Pick-a-Pic v2, Parti-Prompts, and HPSv2, with dataset-specific sampling or test splits.Pick-a-Pic v2’s test split has 500 prompts; Parti-Prompts uses an equal-per-category sample, and HPSv2 uses 125 prompts per style.Source: §E.2.1 · p. 20

E.2.2 IMPLEMENTATION DETAILS

  • Image experiments fine-tune SDXL-Turbo and SDXL-DMD2 with LoRA and single-step generation; FestDPO estimates each prompt’s KDE from 12 generated samples.The stated setup uses rank 16, regularization coefficient β = 0.5, and a pretrained encoder feature space; baseline implementations are adapted to the offline setting.Source: §E.2.2 · pp. 20–21

E.3 PROTEIN BACKBONE GENERATION

  • Protein experiments use RMF-S and optimize either β-strand fraction or scRMSD, with lower scRMSD treated as a proxy for greater structural designability.ProteinMPNN designs sequences and ESMFold predicts structures for the scRMSD target; secondary-structure labels are assigned using DSSP.Source: §E.3 · p. 21

E.3.1 PREFERENCE DATASET CONSTRUCTION

  • Separate fixed offline preference datasets are built from 8,192 reference-generated backbones for each protein optimization target.Pairs are constrained by target-specific score gaps, each backbone appears in at most four pairs, and the procedure yields 8,000 pairs per target.Source: §E.3.1 · p. 22

E.3.2 IMPLEMENTATION DETAILS

  • Protein fine-tuning updates all 16.35 million RMF-S parameters for 1,000 updates using single-step generation and regenerated trainable/reference samples.Density estimates use 64 samples from each generator in the frozen ProteinMPNN encoder’s 128-dimensional feature space.Source: §E.3.2 · pp. 22–23

F ADDITIONAL EXPERIMENTAL RESULTS

  • Additional image results report mean and standard deviation of PickScore and Aesthetic score across 500 prompts per dataset and four random seeds per prompt.The cited tables cover main results and win-rate comparisons for SDXL-Turbo and SDXL-DMD2 on Pick-a-Pic v2, Parti-Prompts, and HPSv2; the supplied captions contain no cell values.Source: F ADDITIONAL EXPERIMENTAL RESULTS · p. 24, Table 10 · p. 24, Table 11 · p. 24

F.2 NUMBER OF KDE SAMPLES

The KDE sample-count study finds PickScore improves as sample count rises while Aesthetic scores stay approximately stable; encoder choice and two-step generation also preserve strong performance. Qualitative comparisons show FestDPO alongside PSO and DrPO across three prompt sets.

F.2 NUMBER OF KDE SAMPLES

  • Increasing KDE samples from 8 to 24 consistently improves PickScore across three benchmarks, while Aesthetic scores remain approximately stable.Even a small sample count yields substantial gains over the pretrained model.Source: §F.2 · p. 24, Table 14 · p. 25

F.3 FEATURE ENCODER FOR KDE

  • FestDPO performs comparably with Latent MAE, CLIP, and DINOv2 encoders; Latent MAE is the default for consistently strong PickScore and Aesthetic scores.The same encoder is applied to DrPO for a fair comparison.Source: §F.3 · p. 25, Table 15 · p. 25
  • With two function evaluations, FestDPO performs comparably to the 1-step setting and consistently improves PickScore across all three benchmarks.The authors suggest additional function evaluations may benefit from improved sample quality.Source: §F.3 · p. 25, Table 16 · p. 25

G QUALITATIVE RESULTS

  • The qualitative comparison uses SDXL-Turbo as the base model to compare FestDPO with PSO and DrPO across three text-to-image prompt sets.The accompanying figures show examples for Pick-a-Pic v2, Parti-Prompts, and HPSv2.Source: G QUALITATIVE RESULTS · p. 26

G.1 PICK-A-PIC V2

G.2 PARTI-PROMPTS

G.3 HPSV2

H HUMAN EVALUATION

In a human evaluation across 80 prompts, FestDPO receives the highest mean ratings for prompt alignment and aesthetics across all three prompt sets. Its pairwise win rates are higher against each baseline on both criteria.

Evaluation protocol

  • The evaluation covers 80 prompts—29 from Pick-a-Pic v2, 20 from Parti-Prompts, and 31 from HPSv2—with four anonymized, randomly ordered images rated for alignment and aesthetics.Prompt order and image arrangement were randomized independently for each annotator.Source: H HUMAN EVALUATION · p. 29, Figure 11 · p. 29

Participants

  • Fifteen annotators contributed 9,600 ratings in total, rating 320 images each on two criteria.The median completion time was about 36 minutes per annotator.Source: H HUMAN EVALUATION · p. 29

Results and reliability

  • FestDPO has the highest mean ratings across all three prompt sets, with statistically significant improvements over every baseline (p < 0.001); win rates are 76–84% for prompt alignment and 72–81% for aesthetics.All 15 annotators individually gave FestDPO the highest mean rating on both criteria.Source: H HUMAN EVALUATION · p. 29, Table 17 · p. 29, Figure 12 · p. 30

Related papers

Loading 2609.34673v1…