PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models

Yansen Han, Shengyi Liao, Peng Sun, Deyuan Liu, Yuanxing Zhang, Pengfei Wan, Tao Lin

arXiv:2609.09905v1stat.MLcs.AIcs.CVcs.LG

TL;DR

Flow-based preference alignment lacks a clear offline forward-KL formulation that uses both preferred and dispreferred fixed samples, while exact flow likelihoods are intractable. The paper introduces FLOWCPO, a contrastive flow-matching surrogate under linear interpolation and regularity conditions. It improves in-domain mean GenEval and OCR over baselines, but out-of-domain results are mixed.

  • Problem

    Existing forward-process methods refresh samples online, while fixed-pair methods do not provide a tractable offline forward-KL objective using both preference sides.

  • Method

    FLOWCPO uses coupled preferred and dispreferred branches and bounds forward-KL terms with a contrastive flow-matching regression loss estimated from fixed data.

  • Results

    In-domain, FLOWCPO reaches 0.84 mean GenEval and 0.87 mean OCR at CFG 3.0, while out-of-domain results are mixed.

  • Takeaways & Limitations

    FLOWCPO provides a nonnegative offline forward-KL surrogate without online rollouts and outperforms evaluated baselines on key in-domain means.

  • Takeaways & Limitations

    The bound assumes linear interpolation and uniform regularity, and evaluation on SD3.5-M leaves performance on other backbones unestablished.

Abstract

from arXiv · show

Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.

1 Introduction

The paper frames flow-preference alignment by sampling regime and KL direction, then introduces FLOWCPO as an offline forward-KL method using fixed preferred and dispreferred pairs. Its contrastive surrogate is theoretically motivated and performs strongly in-domain but yields mixed out-of-domain results.

  • Motivation: Existing methods span online and offline sampling regimes and reverse- versus forward-KL objectives, leaving offline forward-KL alignment with both preference sides unresolved.The forward-KL likelihood is intractable for flow matching, motivating a bridge to velocity-field regression.
  • Method: FLOWCPO derives a contrastive regression surrogate from forward-KL terms using a flow-matching bound for linear interpolation.The surrogate is estimable from fixed preference data and requires no model sampling during training.
  • Method: FLOWCPO’s nonnegative regression loss contrasts with simplified FlowDPO’s signed dispreferred-sample term, which can be unbounded below.With equal branch weights, the decomposition also includes corrections based on the difference between the current model and its EMA.
  • Method: FLOWCPO uses a unified divergence framework and coupled preferred/dispreferred branches to organize and implement offline forward-KL alignment.The framework separates sampling regime from divergence direction, while the two branches provide the contrastive structure.

2 Related Work

Related work includes online alignment with policy-generated samples, offline methods using fixed preference pairs, and divergence-based formulations that unify preference objectives. FLOWCPO differs by combining both sides of fixed pairs in a coupled forward-KL objective.

  • Online alignment: Online methods improve generators with samples drawn from current or recent policies, including reward-feedback, trajectory-gradient, sampler-differentiation, and flow-trajectory approaches.Examples include ReFL, DDPO, DPOK, DRaFT, AlignProp, and FlowGRPO.
  • Offline alignment: Offline methods use fixed preference data through diffusion trajectory losses, reference-relative denoising errors, reverse-KL objectives, preference margins, or preferred-only fine-tuning.FlowDPO extends DPO-style denoising-error optimization to flow matching, while RFT fits only preferred samples.
  • FLOWCPO: FLOWCPO uses both preferred and dispreferred fixed samples in a coupled two-branch forward-KL objective with a flow-matching upper-bound surrogate.The surrogate is justified under stated conditions rather than introduced as a likelihood-ratio approximation.
  • Divergence-based alignment: Divergence-based language-model alignment treats divergence and utility as design choices and relates multiple preference objectives through generalized formulations.The cited frameworks connect methods including DPO, IPO, SLiC, and EXO.

3 Preliminaries

The preliminaries introduce flow matching, RLHF/DPO, continuous-time preference optimization, and forward/reverse KL divergences. They motivate flow-based surrogates because exact likelihood computation requires costly probability-flow ODE integration.

  • Diffusion Models and Flow Matching: Flow and diffusion models connect noise to data through intermediate states, with flow matching learning a velocity field for a prescribed probability path.The paper uses linear interpolation xt = (1 − t) · x0 + t · ϵ, where x0 is clean data and ϵ is Gaussian noise.
  • Diffusion Models and Flow Matching: Flow matching trains vθ(xt, c, t) to regress toward the conditional target velocity along the chosen path.The linear path has x0 at t = 0 and noise ϵ at t = 1.
  • RLHF and DPO: RLHF aligns a generative policy by maximizing reward while regularizing deviation from a reference policy, whereas DPO directly optimizes preferred versus dispreferred pairs.DPO uses a closed-form KL-constrained reward formulation with a frozen reference policy and temperature parameter β.
  • Continuous-Time Preference Optimization: Applying DPO directly to flow or diffusion models is difficult because exact log-likelihood requires computationally prohibitive probability-flow ODE integration.DiffusionDPO uses an ELBO proxy, and FlowDPO replaces denoising-error terms with flow-matching loss for velocity-field optimization.
  • Forward and Reverse KL Divergences: Forward KL weights errors by the target distribution, while reverse KL weights them by the current model distribution.Forward-KL finiteness requires the target support to be contained in the model support.

4 Methodology

The paper casts flow-based preference alignment in a divergence framework and derives FLOWCPO as an offline forward-KL surrogate using preferred and dispreferred data. It parameterizes branch-specific velocity fields and connects the resulting nonnegative regression loss to existing alignment objectives.

  • Probabilistic formulation: Preferred and dispreferred target distributions arise by conditioning a reference policy on preferred and dispreferred optimality events.The paper denotes these distributions π+ and π− and notes that practical offline datasets may only approximate them.
  • Branch parameterization: The model avoids direct flow likelihood evaluation by using symmetric mixed velocity fields built from a trainable field and its EMA copy.The two branches are μθ = (1 − β)vold + βvθ and νθ = (1 + β)vold − βvθ, with β controlling displacement from the reference field.
  • Theoretical bridge: Under stated regularity conditions, conditional flow matching bounds each forward-KL term up to a θ-independent constant.Applying the bound separately to both branches justifies the contrastive regression surrogate on empirical preferred and dispreferred data.
  • FLOWCPO derivation: FLOWCPO derives an offline forward-KL objective that uses fixed preferred and dispreferred datasets instead of model-generated samples.The derivation applies a flow-matching upper bound to obtain a tractable contrastive regression surrogate.
  • Loss properties: For λ ≥ 0, FLOWCPO’s loss is bounded below by zero, whereas simplified FlowDPO’s signed regression objective can be unbounded below.With equal branch weights, the loss decomposition also contains corrections based on the difference between the current model and its EMA.
  • Unified divergence view: The framework separates sampling regime from divergence direction, organizing online and offline flow-alignment methods under forward- and reverse-KL branches.FlowGRPO and FlowDPO occupy online and offline reverse-KL branches, while DiffusionNFT uses a forward-process objective with online sample refresh.

5 Experiments

Experiments evaluate FLOWCPO under strictly offline in-domain and out-of-domain protocols on SD3.5-M. It performs best on several in-domain capability and preference metrics, while out-of-domain results and regularization effects depend on the metric and hyperparameter setting.

  • Experimental setup: The experiments use SD3.5-M with fixed preference data from reference-model pairs in-domain and Open Image Preferences v1 out-of-domain.The in-domain protocol trains separate fine-tunes by target domain, whereas the out-of-domain protocol trains one model on a public dataset generated by other open models.
  • Main results: In-domain FLOWCPO attains the best mean GenEval score of 0.84 and OCR score of 0.87 at CFG 3.0.These exceed FlowDPO by 0.03 and 0.12, respectively; FLOWCPO is also best or tied-best on several general-preference metrics.
  • Main results: Out-of-domain performance is mixed: FLOWCPO has the best GenEval score, remains competitive on OCR and UnifiedReward, and trails RFT on several reward-model metrics.The two regimes represent different evaluation protocols and should be interpreted primarily within regime.
  • Negative regularization: Moderate attractive matching performs best in the regularization study, with β = 0.5 and λ = 1 achieving the best GenEval score of 0.84.Repulsion with λ = −1 sharply degrades all metrics, and both signs diverge at |λ| = 10.
  • Hyperparameter sensitivity: The default sensitivity setting uses β = 0.5, λ = 1, and η = 0.99 because moderate interpolation, balanced negative weight, and slow EMA were favored.Extreme weights destabilize training, while small weights weaken the negative contribution.

6 Conclusion and Limitations

FLOWCPO provides an offline forward-KL alignment method using fixed preferred and dispreferred pairs, with a nonnegative loss and connections to simplified FlowDPO. Theoretical guarantees rely on stated regularity assumptions, while empirical gains are stronger in-domain than out-of-domain.

  • Method: FLOWCPO enables regression on fixed preference pairs without online rollouts through coupled preferred and dispreferred branches.Its loss is derived as an offline forward-KL instance within a divergence-based framework.
  • Empirical findings: Higher mean in-domain GenEval and OCR scores are reported, while out-of-domain gains are mixed.The experiments also report that moderate attractive matching of dispreferred samples is favored, whereas excessively large weights can destabilize training.
  • Limitations: The forward-KL bound assumes linear interpolation and uniform regularity.Loss nonnegativity does not by itself guarantee stable training, and the evaluation uses SD3.5-M, leaving other backbones and paths for future study.
  • Theory: In the unrestricted, well-specified population setting, the marginal target field minimizes flow matching loss and induces the data endpoint distribution.This population statement does not imply equality of the objectives or identical minimizers in a restricted parameter class.
  • Theory: The flow-matching upper-bound surrogate requires uniform regularity and finite second moments, with the population minimizer minimizing both sides under the well-specified interpretation.The bound is established under assumptions including finite data second moment and uniformly controlled quantities independent of θ.

B.4 Proof of Theorem 4.2

The proof relates forward-KL terms to a tractable contrastive flow-matching surrogate and characterizes how FLOWCPO differs from simplified FlowDPO. The resulting decomposition identifies model–EMA corrections and explains the lower-bounded structure of FLOWCPO.

  • Theorem 4.2: Expanding the forward-KL terms into cross-entropy and entropy, then applying the flow-matching bound, establishes an upper-bound surrogate for equation 11.The entropy contribution is independent of θ, and the surrogate matches branch endpoint distributions when both population targets are simultaneously realizable.
  • Assumptions: The decomposition requires β > 0, λ = 1, finite displayed expectations, and a detached EMA field during each gradient step.Under these conditions, the constant term is independent of θ for the current EMA copy.
  • Relation to simplified FlowDPO: FLOWCPO equals β times simplified FlowDPO plus corrections determined by squared differences between the current field and its EMA field.At β = 1, only the dispreferred quadratic constraint remains with coefficient 2; for 0 < β < 1, the preferred correction can have a negative coefficient.
  • EMA reference: EMA interpolation keeps the reference aligned with the trained model, whereas a frozen reference remains at initialization during fine-tuning.Because the corrections depend on squared prediction differences, a lagging reference can substantially change the objective.
  • Stability: FLOWCPO is bounded below by zero because it adds two nonnegative regression errors, unlike simplified FlowDPO’s potentially unbounded negative term.The lower bound removes one potential instability source but does not guarantee stable training.

C Connections to Existing Methods Under the Unified Divergence Framework

The unified framework maps existing methods by sampling regime and KL direction, showing how online and offline objectives relate. It places DiffusionNFT and SFT in the forward-KL branch, FlowGRPO in the reverse-KL branch, and characterizes FlowDPO as a heuristic connection.

  • FlowGRPO: FlowGRPO belongs to the reverse-KL branch of the unified objective and represents an online alignment method.
  • DiffusionNFT: DiffusionNFT corresponds to the forward-KL branch and optimizes a supervised objective using its preference-weighted samples.
  • AWM and RAM: AWM is optimized over the forward process, while RAM can be viewed as the AWM objective plus a reference term.
  • SFT and RFT: SFT corresponds to the forward-KL branch, whereas RFT uses only preferred samples and is also mapped to that branch.
  • FlowDPO: The FlowDPO connection is heuristic rather than formal, and its objective plus a residual term upper-bounds the DPO objective.
  • FlowCPO algorithm: The paper’s algorithm initializes the model and EMA velocity from the reference, mixes them during training, updates the loss, and refreshes the EMA.

D.1 Experimental Details of Sec. 5.2

The experiments use both in-domain and out-of-domain fixed preference data to evaluate offline fine-tuning of SD3.5-M. Training uses LoRA with fixed optimization settings, validation-based checkpoint selection, and no overlap between training and evaluation prompts.

  • Data construction: In-domain data are built from frozen SD3.5-M generations across GenEval, OCR, and general-preference prompt pools.The pools contain 50,000 GenEval, 19,653 OCR, and 25,432 general-preference training prompts.
  • Data construction: All images use 512×512 resolution, 40 ODE-sampler inference steps, blank negative prompts, and guidance scale 4.5 unless otherwise noted.
  • Data construction: Out-of-domain training uses roughly 10K community-annotated preference pairs generated by models other than the SD3.5-M reference policy.
  • Training configuration: SD3.5-M is fine-tuned with LoRA on 8×A100 GPUs using fp16, AdamW, learning rate 3e-4, global batch size 16, and EMA decay η = 0.99.
  • Checkpoint selection: Runs are monitored for 2K steps, with checkpoints selected by task-aligned validation scores for in-domain specialists and PickScore for the shared out-of-domain model.

D.2 Experimental Results on Optimization Targets

The evaluation covers semantic alignment, typographic generation, and general preference alignment using separate in-domain specialists and a single out-of-domain model. Fine-tuned results generally use five independent runs, while pretrained baselines are single evaluations.

  • Evaluation design: The experiments measure semantic alignment, typographic generation, and general preference alignment across separate in-domain specialists and one out-of-domain model.
  • Reporting protocol: Fine-tuned SD3.5-M entries are means and standard deviations over five independent runs, whereas pretrained baselines are single evaluations.
  • Evaluation design: GenEval, OCR, and general-preference results are reported in Tables 5, 6, and 7, while Table 8 reports the out-of-domain setting.
  • Optimization targets: The GenEval table uses GenEval-filtered data, the OCR table uses OCR-filtered data, and the general-preference table highlights metrics coupled to its filtering pipeline.

D.3.1 Understanding the Role of Negative Regularization

The negative-regularization ablation shows that how dispreferred samples enter the objective matters as much as whether they enter. Moderate attractive regularization offers the strongest reported trade-off, while strong repulsion destabilizes training.

  • Ablation pattern: Positive-only regularization remains stable and competitive, whereas repulsive negative regularization becomes less reliable as its magnitude increases.
  • Repulsive regularization: λ = −1 degrades most metrics, and λ = −10 causes numerical instability and collapsed training.
  • Attractive regularization: With β = 0.5 and λ = 1, the model reaches the highest GenEval score of 0.84 in the ablation.
  • Attractive regularization: With β = 1 and λ = 0.1, the model achieves the best OCR of 0.59, ImgReward of 1.14, and UniReward of 3.23.
  • Qualitative results: The qualitative comparisons visualize in-domain samples from models trained on GenEval and OCR preference data.

D.3.2 Hyperparameter Sensitivity and Ablation

Ablations identify moderate β, balanced λ, and high-EMA η as favorable settings in the GenEval-only sweep, while diversity experiments show a quality–diversity trade-off controlled by CFG.

  • Hyperparameter Sensitivity: β ∈[0.5, 1.0] performs best in the GenEval sweep, with β = 0.5 achieving the highest peak score.β controls how far target flows move from the reference prior.
  • Hyperparameter Sensitivity: λ = 1.0 yields the most robust long-term GenEval performance among the tested negative-regularization weights.The sweep balances positive alignment against negative regularization.
  • Hyperparameter Sensitivity: η = 0.99 produces the most stable behavior among tested reference-prior update rates.The reference model is updated with a high-EMA decay rate inside the flow-matching objective.
  • Per-Prompt Diversity Evaluation: Increasing CFG from 1.0 to 4.5 raises target quality while lowering per-prompt Vendi Score across all comparisons.Vendi Score uses eight samples per prompt; higher values indicate greater diversity.
  • Per-Prompt Diversity Evaluation: At CFG 1.0, FLOWCPO achieves 0.76 GenEval, 0.83 OCR, and 22.48 PickScore, the highest quality among compared methods for each target.Lowering CFG from 4.5 to 1.0 increases FLOWCPO’s Vendi Score by 0.2612, 0.5623, and 0.3685 on the three targets, respectively.
Loading 2609.09905v1…