Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

• Existing preference datasets often retain only final images, forcing DPO-style methods to reconstruct or approximate the latent trajectories needed for step- or trajectory-level optimization.Source: §1 · pp. 1–2
• Diffusion-DPO methods approximate reverse trajectories with independent forward-noising samples, which can mismatch inference dynamics and cause instability, inefficient credit assignment, and a larger reward-allocation decision space.Source: §1 · p. 2
• Rectified-flow trajectories are near-straight, and prior noise identifies the trajectory and final image for a fixed prompt, making it relevant to preference-based trajectory adaptation.Source: §1 · p. 2
• PNAPO stores winner and loser prior-noise/image pairs plus a continuous reward gap, forming noise-augmented preference-data sextuples that retain trajectory identity.Source: §1 · p. 2
• PNAPO interpolates noise and image endpoints to estimate intermediate states, compares policy and reference models on the same states, and dynamically regularizes updates by reward-gap difficulty and training stage.The framework is positioned as an offline, RL-free alternative to on-policy rollouts; its analysis argues that stored-noise conditioning yields a tighter RF estimator.Source: §1 · p. 2
• PNAPO shows consistent gains across multiple preference and alignment benchmarks on FLUX.1-dev and SD3-M, with large compute savings compared to Diffusion-DPO.Source: §1 · p. 3

Text-to-Image Generative Models

• Rectified-flow models learn velocity fields between Gaussian priors and data, with structured trajectories that support interpolation-based reasoning.The paper focuses on post-training alignment of RF-based text-to-image models.Source: §2 · p. 3

Preference Optimization of Diffusion Models

• Diffusion preference methods span reward-based backpropagation, sampling-as-MDP, denoising-step optimization, and latent-variable approaches, but most target conventional diffusion.Existing RF methods often replace noise prediction with velocity prediction without accounting for the critical role of prior noise in post-training.Source: §2 · p. 3

Online Preference Alignment

• Online RL and direct reward optimization sample from updated policies, but require substantial on-policy sampling and careful tuning to avoid instability.PNAPO instead targets offline preference optimization, enabling stored-data, RL-free updates without continuous rollouts when compute, latency, or engineering constraints make online RL impractical.Source: §2 · p. 3

Flow Matching and Diffusion Models

• Flow matching learns a coupling between data and Gaussian noise through an ODE with a network-parameterized velocity field (Lipman et al., 2022).Conditional paths and velocity fields are marginalized over the data and noise distributions to recover the marginal path and field.Source: §3 · p. 3
• The intermediate state is defined by xt = atx0 + btxT, and rectified flow establishes a straight-line trajectory between data and Gaussian noise.Source: §3 · p. 3

DPO for Diffusion Models

• Preference datasets pair a prompt with human-ranked winning and losing images, and RLHF adapts the Bradley–Terry model through maximum likelihood (Bradley & Terry, 1952).Source: §3 · p. 3
• Wallace et al. (2024) reformulate diffusion-model preference optimization as a tractable surrogate objective.Source: §3 · p. 3

4.1. Off-Policy Data Construction

• PNAPO constructs preference data by pairing prompt-conditioned images with their retained prior noises and labeling winner–loser pairs using continuous HPSv2.1 reward differences.The reward differences provide interpretable feedback that acts as a proxy for human preferences and a data cleanser.Source: §4.1 · p. 4

4.2. RF-Consistent Optimization via Prior Noise

• By leveraging rectified flow’s straight trajectories, PNAPO estimates intermediate states through prior-noise–image interpolation, reducing the decision space and yielding a lower-error estimate than Diffusion-DPO.The method avoids resource-intensive trajectory sampling and iterative reverse processes.Source: §4.2 · pp. 4–5, Figure 3 · p. 5

4.3. Dynamic Regularization

• PNAPO makes regularization β depend on reward gap and training progress: it increases β for negative margins but decreases it for positive margins, while an annealing controller decays over training.The progress controller starts high and halves by the user-defined threshold n2; the schedule targets pair difficulty and avoids fixed regularization impeding updates later in training.Source: §4.3 · p. 5

5.1. Experimental Setup

• Experiments evaluate FLUX and SD3-M against Diffusion-DPO, SFT, IPO, and CaPO using preference, CLIP text-alignment, and GenEval measures.The experiments use 20,000 DiffusionDB prompts per model and reproduce Diffusion-DPO, SFT, and IPO with identical hyperparameters and model configurations.Source: §5.1 · pp. 5–6

5.2. Primary Results

• PNAPO required 35.2 H800 GPU hours for FLUX and 20.8 for SD3-M, versus 422.4 and 249.6 for Diffusion-DPO, respectively.The paper characterizes this as 12× less training cost while improving generation quality.Source: Table 1 · p. 6, §5.2 · p. 7
• On OPDv1, PNAPO-SD3-M and PNAPO-FLUX achieved median HPSv2.1 reward scores of 33.09 and 32.10, exceeding SD3-M and FLUX by +1.13 and +1.36; PNAPO-FLUX won 84.6% against DPO-FLUX and 81.1% against IPO-FLUX.The paper reports superior PNAPO performance across test datasets and reward scores.Source: Table 2 · p. 7, §5.2 · p. 7
• In a user evaluation, PNAPO-FLUX outperformed several strong baselines in overall preference (56%), visual appeal (72%), and text alignment (52%); each participant evaluated 20 randomly selected image pairs.Qualitative comparisons report higher textual fidelity and visual appeal than Diffusion-DPO on SD3-M and FLUX, and mitigation of background blurring in FLUX-generated samples.Source: §5.2 · pp. 6–7
• On GenEval, PNAPO raises SD3-M from 0.68 to 0.73 (+7.4%) and FLUX from 0.65 to 0.69 (+6.2%).These results are reported as 2.8% and 4.5% absolute improvements over CaPO-SD3-M (0.71) and DPO-FLUX (0.66), respectively.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Ablation Studies and Analysis

• Ablations find that prior noise and dynamic regularization each improve image quality, while combining the training-sample and process controllers yields optimal results.Dynamic training also improves performance without prior noise, and the two controllers independently contribute to gains.Source: §5.3 · p. 8, Figure 5 · p. 8, Table 4 · p. 8
• Text-aware reward models such as PickScore and HPS v2.1 improve visual appeal and text-rendering fidelity, whereas other reward models may prioritize aesthetics at text fidelity's expense.The analysis describes reward models as pseudo-labeling mechanisms for dataset refinement and reports HPSv2.1 as superior across comprehensive metrics.Source: §5.3 · p. 8, Table 6 · p. 8
• The configuration (n1, n2) = (1000, 2000) performs best; reducing regularization degrades effectiveness, while strong regularization gradually pulls the model toward the reference.Source: §5.3 · p. 8, Table 7 · p. 8

6. Conclusion

• PNAPO retains the prior noise paired with each winner and loser, enabling endpoint-conditioned trajectory estimates through noise–image interpolation.This addresses standard preference datasets’ omission of the trajectory identity tied to each final image.Source: §6 · p. 8
• PNAPO’s endpoint-conditioned estimates yield a lower-variance DPO-style objective than independent noising, leveraging rectified flow’s straight-line trajectories.Source: §6 · p. 8
• Its dynamic regularization scales updates by reward gap and training progress to improve stability and efficiency.Source: §6 · p. 8
• Across FLUX and SD3-M benchmarks, PNAPO improves alignment and fidelity while reducing training compute.The theoretical results attribute its gains to endpoint conditioning and rectified-flow straightness.Source: §6 · pp. 8–9

Impact Statement

• PNAPO can lower compute and engineering costs for offline preference tuning, while better-aligned generation may increase misuse, privacy, copyright, and bias risks.The passage recommends prompt and data curation, bias audits, multiple metrics, human review, content filtering, and provenance or watermarking; it says PNAPO does not create fundamentally new capabilities but lowers the barrier to optimizing existing models.Source: Impact Statement · p. 9

A.1. More Related Works

• PPO-based RLHF requires multiple model components and can be computationally intensive; alternatives include REINFORCE-derived methods, RAFT (Dong et al., 2023), RRHF (Yuan et al., 2023), and rejection sampling.The passage describes RLHF as first training a preference model, then optimizing a policy to maximize predicted rewards.Source: §A.1 · p. 18
• Preference optimization also includes DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), ORPO, KTO (Ethayarajh et al., 2024), and PRO (Song et al., 2024), which vary their reward, utility, or reference-model formulations.The passage characterizes DPO as using implicit Bradley–Terry reward signals, IPO as probability-based, ORPO as combining supervised fine-tuning and preference optimization without a reference model, and KTO as utility-based.Source: §A.1 · p. 18
• Preference alignment extends beyond text-to-image diffusion, while human-preference alignment in diffusion remains nascent; proposed directions include LLM-to-visual transfer and expansion to audio and haptic modalities.The passage frames these as future directions for alignment methods across generative domains.Source: §A.1 · p. 18

B. More Preliminaries

• The preliminaries define a probability path from data to a terminal normal distribution and use the conditional vector field for a computationally manageable flow-matching formulation.The passage also states that time-varying weights can alter training behavior without changing the objective’s optimal solution.Source: B. More Preliminaries · p. 19

C. Details of the Primary Derivation

• The derivation reparameterizes whole-chain rewards and cancels pairwise partition functions to obtain a tractable maximum-likelihood objective for diffusion models.The passage identifies this instance-specific formulation as Diffusion-DPO.Source: C. Details of the Primary Derivation · p. 20
• PNAPO estimates intermediate states by interpolation along rectified-flow trajectories, and its approximation is lower-error than Diffusion-DPO’s independent forward-process estimate.The comparison uses pθ(xT|x0)q(x1:T−1|x0,xT) rather than Diffusion-DPO’s forward-process estimate; the passage attributes the lower error to the conditional trajectory approximation.Source: C. Details of the Primary Derivation · pp. 20–21

D.1. Disscussion of Dataset

• Because human preferences are multifaceted and noisy and the Diffusion-DB prompt set repeats text, the study cleanses data and applies KNN-based diversity sampling to build a more balanced prompt set.The passage notes that preference diversity and uncertainty can complicate modeling and contribute to distributional shifts.Source: §D.1 · p. 22

D.2. Limitations and Future Work

• The method can enhance performance only with model-generated noise-image pairs and cannot fine-tune SD3-M on FLUX-generated data because their noise distributions differ.Source: §D.2 · p. 22
• Future work will extend the method to online learning and develop adaptive parameter optimization strategies.Source: §D.2 · p. 22
• Because Diffusion-DB prompts lack coherence, the authors propose MLLM-based alignment and refinement for the dataset or selected high-quality samples.The proposed refinement could improve overall image quality.Source: §D.2 · p. 22

E. Experiment Details

• HPDv2 collects human preferences through Stable Foundation’s Discord Dreambot and contains 25,205 prompts and 98,807 images, with paired preference labels and varying image counts per prompt.Source: E. Experiment Details · p. 22
• The experiments use HPDv2’s test set of 3,200 text prompts.Source: E. Experiment Details · p. 22
• OPDv1 is a collaborative Hugging Face, Argilla, and open-source community dataset comprising 7,459 curated text-to-image preference pairs for evaluation across semantic categories and prompt complexity.It supports comparison of model outputs against human perceptual preferences.Source: E. Experiment Details · p. 22

E.1. Additional Implementation details

• Detoxify’s NSFW threshold was 0.1, and Jaccard similarity threshold was 0.8 for textual redundancy removal.Source: §E.1 · p. 22
• ViT-H/CLIP embedding cosine similarity threshold was 0.8 for semantic deduplication; prompt sampling used KNN clustering (K=100) with 200 prompts per cluster.Source: §E.1 · p. 22
• Training used 8 NVIDIA H800 GPUs; SD3-M used 8-step gradient accumulation at batch size=1/GPU, while FLUX used single-step accumulation.Source: §E.1 · p. 22
• Evaluation held sampling settings constant across experiments: CFG scale=1, 50 sampling steps, and fixed random seeds.Source: §E.1 · p. 22
• Optimization used a learning rate of 1e−6 with 500-step linear warmup, and FLUX used LoRA rank 32.Source: §E.1 · p. 22

E.2. Off-Policy Data Construction

• FLUX samples in Figures 6 and 7 compare rectification-related quality improvements, with HPSv2.1 used to assess and discriminate between sample variations.Source: §E.2 · p. 22

F. Additional Quantitative Results

• The additional evaluation uses 3,000 prompts sampled from Diffusion-DB and reports superior aesthetics and text alignment versus various baselines.The passage identifies the prompt set as a test set.Source: F. Additional Quantitative Results · p. 23
• On FLUX, PNAPO achieves the best results in nearly all five reward evaluations on the Diffusion-DB test set.Table 8 reports median and mean values for the five evaluators.Source: Table 8 · p. 23
• On SD3-M, PNAPO achieves the best results in nearly all five reward evaluations on the Diffusion-DB test set.Table 9 reports median and mean values for the five evaluators.Source: Table 9 · p. 23

G. Additional Qualitative Results

• FLUX preference-dataset samples illustrate quality improvements through rectification and HPSv2.1’s ability to discriminate between sample variations.Figures 6 and 7 present these samples.Source: Figure 6 · p. 24, Figure 7 · p. 25

FLUX PNAPO-FLUX

• Figure 8 shows PNAPO-aligned FLUX images improving text-image alignment and aesthetic quality compared with the FLUX base model.The paper presents the comparison as an extended qualitative result.Source: G. Additional Qualitative Results · p. 23, Figure 8 · p. 26