PaperlayerView exact revision on arXiv

Source-linked AI summary

Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Yunhong Lu, Qichao Wang, Hengyuan Cao, Xiaoyin Xu, Min Zhang

arXiv:2605.09433v1cs.CV

TL;DR

Preference datasets often omit the prior noise that identifies rectified-flow trajectories, limiting trajectory-faithful alignment. PNAPO retains paired noises, interpolates RF states, and dynamically regularizes optimization, improving alignment and fidelity while reducing training compute.

  • Problem

    Standard preference datasets omit prior noise and trajectory identity, leaving rectified-flow models without the endpoint information needed for trajectory-faithful preference alignment.

  • Method

    PNAPO stores paired prior noises with winner–loser images, estimates intermediate states by noise–image interpolation, and dynamically regularizes updates using reward gaps and training progress.

  • Results

    Across FLUX, SD3-M, and multiple preference and alignment benchmarks, PNAPO improves alignment and fidelity while reducing training compute.

  • Takeaways & Limitations

    PNAPO provides an offline, RL-free approach that can reduce post-training compute and engineering costs for preference-based improvement of rectified-flow T2I models.

  • Takeaways & Limitations

    PNAPO’s offline preference data and reward-model choices can steer outputs toward biased stereotypes, while improved generation may increase misuse risks.

Abstract

from arXiv · show

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient for rectified flow (RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, prior DPO-style alignment for diffusion models commonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states via noise-image interpolation, which constrains the trajectory estimation space and yields a tighter surrogate objective for preference optimization. In addition, we introduce a dynamic regularization strategy that adapts the DPO regularization based on (i) the reward gap between winner and loser and (ii) training progress, improving stability and sample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.

1. Introduction

PNAPO addresses trajectory mismatch in preference optimization by retaining the prior noises associated with winner and loser images and exploiting rectified flow’s near-straight trajectories. It combines noise-aware data, interpolation-based trajectory estimation, and dynamic regularization in an offline, RL-free framework that improves alignment benchmarks with lower compute.

  • Motivation: High-capacity text-to-image models still exhibit imperfect text rendering, compositional errors, spatial inconsistencies, and hallucinated objects.
  • Problem: Discarding trajectory-defining information forces DPO-style methods to reconstruct or approximate latent paths for step- or trajectory-level optimization.
  • Problem: Independent forward noising can mismatch stochastic, curved diffusion reverse trajectories and the model’s inference behavior.
  • Method: PNAPO stores prior noise with generated images because rectified-flow trajectories are near-straight and prior noise identifies the trajectory and final image.
  • Contributions: PNAPO uses sextuple preference data, noise–image interpolation, and dynamic regularization based on reward-gap difficulty and training stage.The objective compares policy and reference models on the same endpoint-conditioned intermediate states, while the schedule scales updates to improve stability.
  • Results: PNAPO is an offline, RL-free alternative that yields a lower-variance surrogate and consistent gains on FLUX.1-dev and SD3-M benchmarks with large compute savings versus Diffusion-DPO.

2. Related Works

Related work spans the evolution of text-to-image generation from GANs and diffusion to flow-matching and rectified-flow formulations, alongside supervised, reward-based, and online preference-alignment methods. PNAPO addresses the complementary offline regime, enabling stable RL-free updates after one-time data generation when online rollouts are costly or constrained.

  • Text-to-Image Generative Models: Text-to-image generation has progressed from GANs and diffusion models to flow-matching and rectified-flow formulations.Rectified-flow models learn velocity fields along continuous-time trajectories between a Gaussian prior and the data distribution.
  • Preference Optimization of Diffusion Models: Supervised fine-tuning remains dominant for diffusion-model preference alignment, while other approaches use learned rewards or differentiable reward optimization.DRaFT and AlignProp backpropagate through differentiable rewards, and reward models are trained to mimic human judgment.
  • Online Preference Alignment: Online preference-alignment methods continuously sample from updated policies using online RL or direct reward optimization.The supplied passage identifies GRPO-family methods as examples, though the excerpt is truncated.
  • Online Preference Alignment: PNAPO instead performs offline preference optimization by generating and storing data once, followed by stable RL-free updates without continuous online rollouts.This regime is particularly attractive under training-compute, latency, or engineering constraints that make online RL impractical.

3. Preliminaries

This section introduces flow matching and rectified flow as ODE-based generative frameworks connecting data with Gaussian noise, then outlines DPO for diffusion models using ranked image pairs. It highlights rectified flow’s straight-line trajectory and the tractable preference-optimization surrogate used in diffusion models.

  • Flow Matching and Diffusion Models: Flow matching learns an ODE velocity field vθ that transports data x0 ∼ p0 to Gaussian noise xT ∼ pT = N(0, I).The framework uses conditional paths and fields whose marginalization recovers the overall path and vector field.
  • Flow Matching and Diffusion Models: Conditional flow matching defines intermediate states as xt = atx0 + btxT along paths between data and noise.The diffusion-model objective is expressed using this intermediate-state parameterization.
  • Flow Matching and Diffusion Models: Rectified flow establishes a straight-line forward trajectory between the data distribution and a Gaussian noise distribution.Its formulation connects the trajectory-based representation with the conditional flow-matching objective.
  • DPO for Diffusion Models: Diffusion preference datasets contain a prompt c, a human-ranked winning image xw_0, and a losing image xl_0.RLHF adapts the Bradley–Terry model by maximum likelihood, while diffusion-model DPO reformulates the problem into a tractable surrogate.
  • DPO for Diffusion Models: The diffusion DPO surrogate estimates preference-related quantities over denoising trajectories using noise variables sampled from N(0, I).The cited formulation includes a randomly sampled ϵ* during training.

4. Method

PNAPO is an off-policy rectified-flow alignment method that preserves prior noise in fine-grained preference data, estimates trajectories through noise–image interpolation, and dynamically regularizes training. Its design targets trajectory-specific refinement, lower estimation error, and more stable, efficient optimization.

  • Preference Dataset Construction: PNAPO constructs fine-grained preference data by retaining the prior noise paired with each winner and loser image.The pipeline prepares prompts, generates prior noise–image pairs with a rectified-flow model, and collects preference labels using HPSv2.1.
  • Preference Dataset Construction: The dataset uses a fine-tuned model as its generation base, providing paired images and prior noises for stable preference alignment.Unlike traditional preference datasets that discard prior noise, PNAPO preserves it as training information.
  • RF-Consistent Preference Objective: PNAPO replaces independent forward-noise trajectory estimation with prior-noise-conditioned interpolation between the endpoint noise and image states.Rectified-flow trajectory straightness enables an interpolation-based approximation that reduces the decision space and improves training efficiency.
  • RF-Consistent Preference Objective: Compared with Diffusion-DPO, PNAPO estimates trajectories using pθ(xT|x0)q(x1:T−1|x0,xT), yielding lower error than forward-process estimation.Diffusion-DPO uses q(x1:T|x0) to estimate the reverse process, whereas PNAPO conditions on the generated prior noise.
  • Dynamic Regularization: PNAPO dynamically adjusts DPO regularization according to reward gaps and training progress because fixed regularization ignores pair difficulty and increasingly impedes updates.The sample controller f(δr) increases with reward difference, while the training controller g(n) anneals over training.

5. Experiments

Across FLUX and SD3-M experiments, PNAPO improves qualitative quality, human preference, text-image alignment, and preference rewards over baselines. It also reduces training cost substantially, while ablations support prior-noise interpolation, dynamic regularization, and text-aware reward models.

  • Qualitative Results: PNAPO consistently outperforms baselines in text-image alignment, visual aesthetics, and photorealism, while mitigating background blurring in FLUX samples.The qualitative comparisons include Figures 2 and 4 and specifically contrast PNAPO with Diffusion-DPO.
  • Quantitative Results on Preference Alignment: On OPDv1, PNAPO-SD3-M and PNAPO-FLUX reached median HPSv2.1 scores of 33.09 and 32.10, surpassing their original counterparts.The passage reports similar improvements across other preference metrics and validates PNAPO against baseline methods.
  • Computational Cost: 35.2 and 20.8 H800 GPU hours were required for PNAPO on FLUX and SD3-M, versus 422.4 and 249.6 for Diffusion-DPO.PNAPO therefore achieved 12× less training cost while significantly improving generation quality.
  • Ablation Studies: Ablations show that prior noise and dynamic regularization improve performance, while text-aware reward models enhance visual appeal and text-rendering fidelity.The best parameter configuration used (n1, n2) = (1000, 2000), and reducing regularization degraded model effectiveness.

6. Conclusion

PNAPO is an offline, RL-free preference-alignment method for rectified-flow text-to-image models that uses prior-noise information to estimate trajectories through noise–image interpolation. Across FLUX, SD3-M, and multiple benchmarks, it improves alignment and fidelity while reducing training compute.

  • Contribution: PNAPO addresses preference datasets’ omission of the prior noise tying each generated image to its trajectory.Standard datasets retain only final image pairs, whereas PNAPO preserves trajectory identity through prior-noise information.
  • Contribution: Endpoint-conditioned noise–image interpolation yields a lower-variance DPO-style objective than independent noising.The method exploits rectified flow’s straightness to estimate trajectories from the paired prior noise and endpoint image.
  • Results: PNAPO improves alignment and fidelity while reducing training compute across FLUX, SD3-M, and multiple benchmarks.The reported gains are theoretically attributed to endpoint conditioning and rectified-flow straightness.

Impact Statement … D.1. Disscussion of Dataset

PNAPO is an offline, RL-free preference optimization method for rectified-flow text-to-image models that uses stored prior noise and interpolation-based trajectory estimates. Its derivation connects preference optimization, flow-matching preliminaries, and rectified-flow-specific approximations, while the discussion highlights alignment benefits and misuse risks.

  • Impact Statement: PNAPO uses stored prior noise to improve alignment and training efficiency for rectified-flow text-to-image models without reinforcement learning.The impact discussion associates this design with reduced compute and engineering costs for post-training.
  • Impact Statement: Improved alignment and generation quality can reduce post-training costs while also increasing misuse risks, including deceptive imagery.The impact statement frames accessibility and faster quality or safety tuning as benefits alongside this limitation.
  • A. Background; A.1. More Related Works: Preference optimization builds on RLHF, DPO, and IPO, with DPO using implicit Bradley-Terry rewards and IPO emphasizing pairwise preference structure.These approaches are presented as successive developments for aligning generative models and language models.
  • A.1. More Related Works; B. More Preliminaries: Diffusion and flow-matching models generate data through iterative denoising or a probability path between data and a normal prior, with conditional vector fields making optimization tractable.The preliminaries explain that conditional flow-matching objectives replace intractable marginal optimization while preserving an equivalent formulation.
  • C. Details of the Primary Derivation: The derivation defines rewards over the whole chain and obtains a tractable Diffusion-DPO objective through reward reparameterization and cancellation of pairwise partition functions.This establishes the preference-learning objective before introducing rectified-flow-specific trajectory estimation.
  • C. Details of the Primary Derivation: Rectified flow enables interpolation-based estimation of intermediate states because its sampling trajectories are inherently straight.The resulting approximation replaces a resource-intensive reverse-process calculation with an interpolation-conditioned estimate.
  • C. Details of the Primary Derivation: Compared with Diffusion-DPO’s forward-process estimate, PNAPO estimates the reverse process using the model’s prior-noise distribution and a conditional interpolation approximation.The paper states that this approximation yields lower error than the forward-process-based alternative.
  • D.1. Disscussion of Dataset: Human preference data for text-to-image models is noisy and subjective because feedback reflects image quality, photorealism, artistic style, and cultural context.The dataset discussion identifies preference diversity and uncertainty as challenges for effective learning.

D.2. Limitations and Future Work · E. Experiment Details

The method is currently limited to model-generated noise-image pairs because cross-model fine-tuning is hindered by noise-distribution discrepancies. Experiments use preference datasets including HPDv2 and OPDv1, with HPDv2 evaluated on 3,200 test prompts.

  • D.2. Limitations and Future Work: The approach is restricted to noise-image pairs generated by the model itself.
  • D.2. Limitations and Future Work: Noise-distribution discrepancies prevent fine-tuning SD3-M with data generated by FLUX.
  • D.2. Limitations and Future Work: Future work will extend the method to online learning and develop adaptive parameter optimization strategies.
  • D.2. Limitations and Future Work: Text prompts in Diffusion-DB currently lack coherence, which may limit the approach.
  • E. Experiment Details: HPDv2 contains 25,205 distinct prompts and 98,807 generated images with paired labels representing relative user preferences.
  • E. Experiment Details: The experiments use the HPDv2 test set, comprising 3,200 text prompts.
  • E. Experiment Details: OPDv1 provides 7,459 curated text-to-image preference pairs as a benchmark resource for image-generation-model evaluation.

E.1. Additional Implementation details · E.2. Off-Policy Data Construction

The implementation uses explicit filtering, redundancy controls, balanced prompt sampling, and specified H800 training configurations. Off-policy data construction includes FLUX visual samples assessed with HPSv2.1 to compare rectification-related quality differences.

  • E.1. Additional Implementation details: NSFW filtering used a Detoxify score threshold of 0.1.
  • E.1. Additional Implementation details: Textual redundancy removal used a Jaccard similarity threshold of 0.8.
  • E.1. Additional Implementation details: Semantic deduplication used a ViT-H/CLIP embedding cosine similarity threshold of 0.8.
  • E.1. Additional Implementation details: Balanced prompt sampling used KNN clustering with K=100 and 200 prompts per cluster.
  • E.1. Additional Implementation details: All models were trained on 8 NVIDIA H800 GPUs, with SD3-M using eight-step gradient accumulation and FLUX using single-step accumulation.SD3-M used batch size=1/GPU.
  • E.2. Off-Policy Data Construction: FLUX samples were presented in Figure 6 and Figure 7 for comparative assessment.
  • E.2. Off-Policy Data Construction: Comparisons used HPSv2.1 (Human Preference Score v2.1) as the evaluation metric.
  • E.2. Off-Policy Data Construction: The visualizations addressed quality improvements through rectification and HPSv2.1’s discrimination between sample variations.

F. Additional Quantitative Results

On 3,000 Diffusion-DB test prompts, PNAPO produces images with superior aesthetics and text alignment versus baselines. It achieves the best results in nearly all reward evaluations for both FLUX and SD3-M comparisons.

  • Test setup: 3,000 Diffusion-DB prompts form the test set for additional quantitative evaluation of aesthetics and text alignment against multiple baselines.The experiments use prompts sampled from Diffusion-DB.
  • FLUX baselines: PNAPO achieves the best results in nearly all reward evaluations compared with the existing alignment baseline on FLUX.The comparison reports median and mean values from five reward evaluators.
  • SD3-M baselines: PNAPO achieves the best results in nearly all reward evaluations compared with the existing alignment baseline on SD3-M.The comparison reports median and mean values from five reward evaluators.

G. Additional Qualitative Results · FLUX PNAPO-FLUX

The section presents extended qualitative comparisons for PNAPO, including FLUX preference-dataset samples and additional results. These comparisons highlight rectification-related quality improvements, HPSv2.1’s discrimination of sample variations, and improved text-image alignment and aesthetic quality over the FLUX base model.

  • G. Additional Qualitative Results: Extended qualitative comparisons in Figure 8 highlight the advantages of PNAPO.The paper presents these results to provide more comprehensive insights.
  • G. Additional Qualitative Results: FLUX preference-dataset samples compare loser and winner generations.Figures 6 and 7 label paired samples as loser and winner.
  • G. Additional Qualitative Results: Rectification produces quality improvements in the FLUX preference-dataset samples.This effect is identified in the descriptions of Figures 6 and 7.
  • FLUX PNAPO-FLUX: PNAPO-aligned images improve text-image alignment compared with the FLUX base model.This comparison is stated in the caption for the additional qualitative results.
  • FLUX PNAPO-FLUX: PNAPO-aligned images improve aesthetic quality compared with the FLUX base model.The Figure 8 caption presents this as evidence for the approach’s superiority.
Loading 2605.09433v1…