PaperlayerView exact revision on arXiv

Source-linked AI summary

TempFlow-GRPO: When Timing Matters for GRPO in Flow Models

Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, Bo Zhang

arXiv:2508.04324v4cs.CV

TL;DR

Flow-based GRPO methods use sparse terminal rewards and uniform timestep optimization, failing to reflect the temporal dynamics of generation. TempFlow-GRPO uses trajectory branching, noise-aware weighting, and seed grouping to provide temporally aware optimization, achieving state-of-the-art alignment performance on text-to-image benchmarks. Its current experiments focus on algorithmic innovations rather than reward-model enhancements.

  • Problem

    Existing flow-based GRPO methods use sparse terminal rewards and uniform optimization despite different timestep noise conditions and decision importance.

  • Method

    TempFlow-GRPO combines trajectory branching for process rewards, noise-aware policy weighting, and seed grouping to isolate branching exploration from initialization effects.

  • Results

    TempFlow-GRPO achieves state-of-the-art performance in human preference alignment, sample quality, and compositional image generation on text-to-image benchmarks.

  • Takeaways & Limitations

    Temporally structured optimization improves flow-based reinforcement learning while integrating with existing flow-matching architectures.

  • Takeaways & Limitations

    Current experiments focus primarily on algorithmic innovations rather than reward-model enhancements, leaving multimodal rewards from more powerful foundation models for future work.

Abstract

from arXiv · show

Recent flow matching models for text-to-image generation have achieved remarkable quality, yet their integration with reinforcement learning for human preference alignment remains suboptimal, hindering fine-grained reward-based optimization. We observe that the key impediment to effective GRPO training of flow models is the temporal uniformity assumption in existing approaches: sparse terminal rewards with uniform credit assignment fail to capture the varying criticality of decisions across generation timesteps, resulting in inefficient exploration and suboptimal convergence. To remedy this shortcoming, we introduce \textbf{TempFlow-GRPO} (Temporal Flow GRPO), a principled GRPO framework that captures and exploits the temporal structure inherent in flow-based generation. TempFlow-GRPO introduces three key innovations: (i) a trajectory branching mechanism that provides process rewards by concentrating stochasticity at designated branching points, enabling precise credit assignment without requiring specialized intermediate reward models; (ii) a noise-aware weighting scheme that modulates policy optimization according to the intrinsic exploration potential of each timestep, prioritizing learning during high-impact early stages while ensuring stable refinement in later phases; and (iii) a seed group strategy that controls for initialization effects to isolate exploration contributions. These innovations endow the model with temporally-aware optimization that respects the underlying generative dynamics, leading to state-of-the-art performance in human preference alignment and text-to-image benchmarks.

1 INTRODUCTION

Existing flow-based GRPO methods treat all generation timesteps uniformly and provide only terminal rewards, overlooking temporal differences in decision importance. TempFlow-GRPO addresses this with trajectory branching, noise-aware optimization, and seed grouping, achieving state-of-the-art text-to-image alignment results.

  • Uniform optimization and sparse terminal rewards overlook the varying importance of decisions across generation timesteps.
  • Trajectory branching introduces stochasticity at designated timesteps while preserving deterministic evolution elsewhere, enabling precise credit assignment without modifying reward models.
  • Noise-aware weighting gives larger updates to high-noise early stages and gentler updates to low-noise later stages.
  • Seed grouping controls for initial noise so reward differences reflect branching exploration rather than initialization effects.
  • TempFlow-GRPO achieves state-of-the-art performance in sample quality, human preference alignment, and compositional image generation versus existing flow-based RL methods.

2 RELATED WORK

Prior diffusion-alignment work includes policy optimization, direct preference optimization, training-free alignment, and process-reward approaches. Figure 3 compares TempFlow-GRPO with flow-based baselines and broader text-to-image systems on efficiency and Geneval performance.

  • Diffusion alignment methods span direct policy optimization, preference optimization, training-free inference alignment, and flow-based online reinforcement learning.
  • Process-reward methods add step-level feedback, but process reward models can require expensive step-level supervision.
  • TempFlow-GRPO outperforms Flow-GRPO variants in PickScore sample and computational efficiency and achieves the highest Geneval performance among compared models.

3 PRELIMINARY: FLOW-GRPO

Flow-GRPO applies GRPO to flow models by grouping trajectories sampled from a prompt, normalizing group-level rewards into advantages, and converting deterministic ODE sampling into an equivalent SDE.

  • GRPO groups individual images and their reverse-time trajectories sampled by a flow model for a shared prompt.
  • The advantage for each image is computed by normalizing its reward against the group’s mean and standard deviation.
  • Flow-GRPO converts deterministic ODE sampling into an equivalent SDE while matching the original model’s marginal probability density at every timestep.
  • The framework’s overview depicts trajectory sampling and temporally varying optimization within the broader TempFlow-GRPO design.

4 METHODS

TempFlow-GRPO addresses temporal uniformity in flow-based GRPO with trajectory branching, noise-aware policy weighting, and seed-level grouping for isolating exploration effects.

  • Motivation: TempFlow-GRPO identifies sparse terminal rewards and uniform optimization as limitations because timesteps differ in criticality, noise, and exploration potential.High-noise early stages support broader exploration, whereas low-noise late stages support refinement.
  • Trajectory Branching: Existing outcome-based reward models can score exploratory branches, avoiding specialized reward models for noisy intermediate states.The method uses existing high-quality outcome-based reward models to assign scores to intermediate exploratory actions.
  • Trajectory Branching: Trajectory branching switches from deterministic ODE evolution to stochastic SDE sampling at a designated timestep, then resumes deterministic generation to attribute terminal rewards locally.Theorem 1 states that reward variance and parameter-dependent improvements are attributable to noise injected at the branching point.
  • Noise-Aware Policy Weighting: Noise-aware reweighting makes policy-loss weights proportional to timestep noise, increasing optimization pressure during high-noise exploration and reducing it during later refinement.The strategy aligns learning intensity with exploration capacity while preserving stability in high-fidelity later states.
  • Policy-Gradient Justification: Without reweighting, late refinement steps contribute more than early structural exploration; reweighting simplifies scale terms toward step size and can equalize contributions when flow shift equals 1.
  • Seed Group: Seed-level grouping holds initial noise fixed within prompt groups so reward differences reflect branching exploration rather than initialization effects.The approach requires K distinct explorations at each timestep.

5 EXPERIMENT

TempFlow-GRPO is evaluated on compositional generation, human preference alignment, and cross-model reward settings, with ablations and qualitative comparisons examining its components and image quality. Across these evaluations, it improves performance and convergence relative to Flow-GRPO.

  • Compositional Image Generation: 0.97 overall Geneval score improves on the base model’s 0.63 and exceeds Flow-GRPO’s 0.88 at 3,800 steps.TempFlow-GRPO reaches 0.95 in about 2,000 steps, compared with approximately 5,600 steps for Flow-GRPO.
  • Human Preference Alignment: TempFlow-GRPO surpasses Flow-GRPO by approximately 1.7% and Flow-GRPO (Prompt) by about 1.0% on PickScore.It matches Flow-GRPO in 100–200 steps and reaches Flow-GRPO (Prompt)’s level in 300–400 steps.
  • HPSv3 on FLUX.1-dev: On HPSv3 with FLUX.1-dev, TempFlow-GRPO matches Flow-GPO’s performance in 80 steps versus 300 steps while maintaining lower and more stable KL loss.The evaluation uses FLUX.1-dev at 1024 resolution with HPSv3 as the reward model.
  • Ablation Studies: Ablations evaluate trajectory branching, noise-aware policy weighting, and seed grouping on both Geneval and PickScore.The supplied experimental description identifies these components as the focus of the ablation studies.
  • Qualitative Results: Qualitative comparisons report finer details, fewer visual artifacts or mistakes, and better preservation of complex structures and realistic textures than Flow-GRPO (Prompt).The comparison includes FLUX.1-dev, Flow-GRPO (Prompt), and TempFlow-GRPO.

6 CONCLUSION

The conclusion presents TempFlow-GRPO as a temporally aware framework for flow-based GRPO, combining trajectory branching, noise-aware weighting, and seed grouping. The reported experiments show state-of-the-art human-preference performance, while the current study emphasizes algorithmic innovations over reward-model enhancements.

  • Conclusion: Trajectory branching assigns credit to intermediate actions without requiring specialized process reward models.The method introduces stochasticity at selected timesteps while retaining deterministic evolution elsewhere.
  • Conclusion: Noise-aware weighting matches optimization intensity to each timestep’s exploration potential.The conclusion frames this as part of the framework’s temporal awareness.
  • Conclusion: Seed grouping controls for initialization effects in the optimization procedure.This strategy is presented alongside trajectory branching and noise-aware weighting as a core component.
  • Conclusion: The experiments demonstrate state-of-the-art performance on human preference alignment.The conclusion reports this as an outcome of the extensive experiments.
  • Limitations: Current experiments focus primarily on algorithmic innovations rather than reward-model enhancements.Future work targets multimodal rewards from more powerful foundation models and comprehensive reward frameworks.

A.1 POLICY GRADIENT-BASED THEORETICAL FRAMEWORK

The theoretical analysis examines how timestep-dependent noise affects the policy-gradient formulation and reweighting. It concludes that the norm of the normalized reward-noise term is invariant across timesteps, motivating attention to the remaining scale terms.

  • Policy Gradient Derivation: The policy-gradient derivation starts from the multi-step formulation and substitutes the transition x_k−1 into the log-probability.The subsequent steps take the gradient with respect to θ and identify a scale term.
  • Noise-Aware Reweighting: The reweighting derivation identifies Δ_k as capturing the intrinsic exploration potential at timestep k.The resulting formulation describes how timestep-specific scale factors modulate gradient contributions.
  • Noise Perturbation Analysis: A small zero-mean Gaussian noise perturbation permits a first-order Taylor approximation of the final reward.The analysis considers the expectation of the noise vector multiplied by the normalized advantage.
  • Timestep Invariance: Equation 28 indicates that the norm of E_ϵ[ϵ Â_k] is invariant among timesteps.The derivation relates this invariant norm to the normalized gradient direction g_k.

A.2 EXPERIMENTAL SETTING DETAILS

The experiments evaluate TempFlow-GRPO across compositional generation and human preference alignment using multiple reward models, datasets, and controlled training settings.

  • Datasets and tasks: Evaluation covers Geneval, PickScore, and HPDv2 for compositional image generation and human preference alignment.Geneval assesses object co-occurrence, spatial positioning, object count, and color attributes.
  • Reward models: The reward models comprise PickScore, HPSv2, HPSv3, and Geneval.These include CLIP-based, VLM-based, and framework-based rewards.
  • Training settings: PickScore alignment follows Flow-GRPO with KL coefficient β = 0.001, while GenEval uses β = 0.004.The settings preserve comparability with Flow-GRPO for each evaluation protocol.
  • FLUX.1-dev configuration: FLUX.1-dev HPSv3 experiments use noise level 0.9, 10 sampling steps for training, 50 for evaluation, and 48 groups of size 24.Each group is arranged as 4 × 6.
  • Baselines: The study includes a direct comparison with DanceGRPO under an equitable configuration.The passage introduces the comparison but does not specify the complete training configuration.

A.3 EXPERIMENTS ON FLUX.1-DEV

On FLUX.1-dev with HPDv2 and HPSv3 rewards, TempFlow-GRPO substantially improves over Flow-GRPO and reaches comparable performance in fewer training steps.

  • Performance: Approximately 0.3 performance gain over Flow-GRPO is achieved after 300 training steps.The comparison uses FLUX.1-dev on HPDv2 with HPSv3 as the reward model.
  • Evaluation curves: Figure 10 tracks smoothed HPSv3 performance and KL loss on HPDv2 with FLUX.1-dev.The caption identifies the plotted metrics and evaluation setting.

A.4 COMPARSION WITH DANCEGRPO

Comparisons and ablations report faster convergence, stronger preference and multi-objective performance, improved visual quality, and more stable KL behavior, alongside higher sampling cost.

  • Comparison with DanceGRPO: TempFlow-GRPO reaches 38.5 at 280 iterations versus DanceGRPO’s 37.2 at 300, a 1.3% improvement.It matches DanceGRPO’s final performance in 150 iterations, yielding a 2× convergence speedup without specialized hyperparameter tuning.
  • High-resolution preference alignment: TempFlow-GRPO improves PickScore by 1.0% after 450 steps and matches Flow-GRPO’s 450-step performance in approximately 100 steps.This corresponds to a 4.5× speedup in convergence.
  • Training efficiency: At higher resolutions, TempFlow-GRPO achieves Flow-GRPO’s final performance using 33% to 50% of total training time.Wall-clock comparisons show faster convergence across all evaluated benchmarks.
  • Computational cost: For K = 10, branching increases average sampling cost to approximately 4.5 times Flow-GRPO, while training time per iteration remains identical.The method nevertheless reports faster overall convergence in wall-clock time.
  • Long-run performance: TempFlow-GRPO continues outperforming Flow-GRPO after 1200 PickScore training steps, indicating a higher final performance ceiling.The result is reported for extended training in the main-paper PickScore experiments.
  • Ablation analysis: Controlled ablations show trajectory branching and noise reweighting independently provide significant gains under a constant group strategy.The components both reduce iterations to match multi-step performance and raise the final performance ceiling.
  • Training stability: TempFlow-GRPO maintains lower KL divergence than Flow-GRPO throughout training, including when using the seed group strategy.The seed group configuration raises KL divergence marginally relative to the default but remains below Flow-GRPO.
  • Multi-reward optimization: With HPSv3 and PickScore weighted 1:0.26, TempFlow-GRPO improves HPSv3 by 0.82 while maintaining PickScore gains at 1024×1024 resolution.The multi-reward experiment evaluates simultaneous optimization after 120 training steps.
Loading 2508.04324v4…