PaperlayerView exact revision on arXiv

Source-linked AI summary

Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO

Yunze Tong, Mushui Liu, Canyu Zhao, Didi Zhu, Wanggui He, Shiyi Zhang, Hongwei Zhang, Peng Zhang, Jinlong Liu, Hao Jiang

arXiv:2602.06422v2cs.CV

TL;DR

Flow-based GRPO propagates the same outcome reward across denoising steps and overlooks delayed within-trajectory effects. TP-GRPO uses incremental step rewards and sign-based turning-point rewards, consistently outperforming Flow-GRPO across three tasks while largely preserving generalization.

  • Problem

    Flow-based GRPO uniformly assigns trajectory-level rewards to denoising steps and overlooks within-trajectory dependencies and delayed effects.

  • Method

    TP-GRPO uses reward differences for step-level incremental effects and sign-change detection to assign turning points aggregated long-term rewards.

  • Results

    TP-GRPO consistently outperforms Flow-GRPO across three tasks while largely preserving generalization performance and improving counts, text rendering, aesthetics, and alignment.

  • Takeaways & Limitations

    TP-GRPO provides a denser, step-aware signal that captures both local denoising effects and delayed cross-step interactions.

  • Takeaways & Limitations

    The step-wise reward estimator relies on ODE sampling preserving the SDE marginal distribution while removing stochasticity.

Abstract

from arXiv · show

Deploying GRPO on Flow Matching models has proven effective for text-to-image generation. However, existing paradigms typically propagate an outcome-based reward to all preceding denoising steps without distinguishing the local effect of each step. Moreover, current group-wise ranking mainly compares trajectories at matched timesteps and ignores within-trajectory dependencies, where certain early denoising actions can affect later states via delayed, implicit interactions. We propose TurningPoint-GRPO (TP-GRPO), a GRPO framework that alleviates step-wise reward sparsity and explicitly models long-term effects within the denoising trajectory. TP-GRPO makes two key innovations: (i) it replaces outcome-based rewards with step-level incremental rewards, providing a dense, step-aware learning signal that better isolates each denoising action's "pure" effect, and (ii) it identifies turning points-steps that flip the local reward trend and make subsequent reward evolution consistent with the overall trajectory trend-and assigns these actions an aggregated long-term reward to capture their delayed impact. Turning points are detected solely via sign changes in incremental rewards, making TP-GRPO efficient and hyperparameter-free. Extensive experiments also demonstrate that TP-GRPO exploits reward signals more effectively and consistently improves generation. Demo code is available at https://github.com/YunzeTong/TurningPoint-GRPO.

1. Introduction

Flow-based GRPO for text-to-image generation assigns aggregate trajectory rewards uniformly across denoising steps and overlooks delayed within-trajectory interactions. TP-GRPO addresses these issues with incremental step-wise rewards and sign-based turning-point rewards that model delayed effects.

  • Background: Flow Matching models transport simple priors to complex target distributions and have been adopted for text-to-image generation with GRPO-based methods.
  • Limitations: Outcome-based rewards are assigned identically to every denoising step, obscuring individual contributions and causing reward sparsity.
  • Limitations: Existing trajectory-level ranking overlooks implicit interactions in which earlier denoising actions alter later states, behavior, and future rewards.
  • TP-GRPO: TP-GRPO replaces sparse outcome rewards with differences between post-update and pre-update rewards to estimate each SDE step’s incremental, pure effect.
  • TP-GRPO: TP-GRPO identifies turning points through sign changes that align subsequent reward evolution with the overall trend, then assigns aggregated long-term rewards to their actions.

2. Related Work

Related work spans diffusion-based generative modeling, reinforcement-learning methods for preference alignment, and the extension of DPO, PPO-style algorithms, and group-based ranking to diffusion and flow models.

  • Diffusion-based Models: Diffusion-based models support image, video, 3D, and tabular data generation through gradual noising and denoising.The passage lists representative works across these four generative modeling tasks.
  • Common RL Techniques: RL is central to aligning large language models with human preferences and downstream objectives.A standard pipeline trains a reward model from human preference data before optimizing the policy with PPO.
  • Common RL Techniques: Group-based ranking objectives exploit multiple candidates per prompt, with GRPO and variants widely adopted for LLM post-training.The passage presents GRPO as an extension of the standard reward-model and PPO pipeline.
  • RL for Diffusion and Flow Models: Recent diffusion-model RL methods apply DPO or PPO-style algorithms to modify denoising trajectories.The passage frames these approaches as adaptations motivated by advances in LLM alignment.

3. Preliminary

Flow Matching learns a time-dependent velocity field and samples by integrating a deterministic ODE, while equivalent SDE samplers add diffusion-driven stochasticity. GRPO applies group-wise trajectory contrast to flow matching by using SDE sampling to generate diverse denoising rollouts.

  • Flow Matching: Flow Matching learns a time-dependent velocity model vθ(x_t, t) that transports a prior distribution p_0 toward a target distribution p_1.Sampling integrates the learned deterministic ODE.
  • Flow Matching: Equivalent SDE samplers convert the ODE sampler while preserving its marginal distributions at each noise level.Their stochasticity comes from both initial noise and diffusion along the trajectory, unlike ODE sampling, where randomness arises only from initialization.
  • GRPO for Flow Matching: GRPO uses contrasts among self-sampled trajectories to guide policy exploration and requires diverse rollouts under the same context or input.Diversity is therefore a key prerequisite for applying GRPO.
  • GRPO for Flow Matching: Flow-matching GRPO methods use SDE sampling to inject stochasticity into denoising, producing diverse trajectories and corresponding final images.Flow-GRPO samples a group of G images conditioned on a prompt c and computes advantages from group-level rewards.

4. Analysis on Limitations of Terminal Reward Assignment

Terminal-only reward assignment creates sparse, timestep-agnostic supervision that cannot accurately distinguish individual denoising actions. Iterative denoising also creates implicit cross-step interactions, so early actions can affect later states and the final image through delayed effects that prior methods do not explicitly model.

  • Reward sparsity: Flow-GRPO computes one reward from the final clean image and assigns it uniformly to every denoising step, causing sparse and inaccurate stepwise credit assignment.Within a fixed group, the relative ordering of advantages therefore remains constant across timesteps, even though optimization occurs at all SDE-sampled steps.
  • Reward sparsity: Terminal rewards provide cumulative trajectory feedback but are per-timestep agnostic, offering no direct supervision for each denoising action’s local effect.This limitation prevents reward assignment from distinguishing how individual steps contribute to the final outcome.
  • Implicit cross-step interaction: Iterative flow-matching denoising creates implicit interactions because each updated latent becomes the next step’s input, allowing early actions to influence later updates and the final image.These aggregated, indirect effects should inform timestep-level rewards but are not explicitly modeled by prior GRPO methods.
  • Implicit cross-step interaction: Intermediate-latent rewards commonly oscillate and are nonmonotonic, showing that local denoising effects need not follow the final trajectory’s overall reward trend.The analysis estimates intermediate-state rewards by completing each latent with ODE sampling after preceding SDE steps.
  • Turning-point motivation: A turning point is a step where the local reward trend reverses and becomes consistent with the overall trajectory trend, capturing an action’s delayed impact.Such actions can deserve stronger positive or negative preference because they flip the trend and influence subsequent reward evolution.

5. Methodology

TP-GRPO replaces sparse outcome-based rewards with step-level incremental rewards and aggregates downstream effects at selected turning points. It further extends long-term effect detection to eligible initial denoising steps and incorporates both reward-improving and reward-degrading operations into optimization.

  • Step-wise incremental rewards: TP-GRPO defines each sampling-step reward as the gain difference before and after that denoising action, isolating its local effect.Intermediate latents are cached and compared after ODE sampling to obtain step-aware rewards.
  • Step-wise incremental rewards: ODE-sampled outcomes provide a baseline-gain proxy because ODE sampling preserves the SDE marginal distribution while removing stochasticity.Replacing outcome rewards with these incremental rewards yields more accurate feedback for intermediate sampling steps.
  • Long-term turning-point rewards: At turning points, TP-GRPO replaces local rewards with aggregated multi-step rewards that capture cumulative downstream effects through the end of denoising.This models implicit interactions from key SDE-based actions and favors downstream trajectories that improve final reward while avoiding those that degrade it.
  • Long-term turning-point rewards: Consistent turning points form a stricter subset selected when aggregated effects align with local updates, filtering for purer implicit interactions.Because both rewards are incremental quantities, either can replace the other without scale mismatch; advantages include both preference- and rejection-inducing operations.
  • Initial-step long-term effects: TP-GRPO extends aggregation to eligible first denoising steps, whose early decisions can strongly steer the trajectory and receive more representative cumulative rewards.Without this constraint, initial steps receive only local rewards and are excluded from aggregation-based effect propagation.

6. Experiments

Experiments evaluate TP-GRPO on compositional image generation, human preference alignment, and visual text rendering using SD3.5-M with LoRA fine-tuning. Across these tasks and additional efficiency and robustness analyses, TP-GRPO consistently outperforms Flow-GRPO, while performance depends on retaining sufficient late-step optimization and balanced sampling stochasticity.

  • Experimental Setup: Experiments cover compositional image generation, human preference alignment, and visual text rendering with SD3.5-M and LoRA fine-tuning.The tasks use Geneval, PickScore, and rule-based OCR accuracy rewards, respectively.
  • Main Results: TP-GRPO consistently outperforms Flow-GRPO across all three tasks while largely preserving generalization performance.The comparison includes two TP-GRPO variants differing in their turning-point selection definitions.
  • Training Curves: Without the KL penalty, TP-GRPO again achieves superior performance on all three tasks, with especially notable gains on PickScore.The unconstrained experiments assess exploratory capability and the ability to exploit optimization signals.
  • Ablation Studies: Reducing the SDE-sampling window too aggressively, such as to 4 steps, causes a sharp performance drop by skipping later-step optimization and turning points.Smaller windows reduce intermediate-sampling cost and shorten training time for a fixed step budget.
  • Ablation Studies: TP-GRPO remains superior to Flow-GRPO across tested noise levels, but both insufficient and excessive stochasticity harm learning.The reference setting is α = 0.7; α = 0.4 produces insufficient stochasticity, while α = 1.0 destabilizes the optimization direction.
  • Efficiency Analysis: At equal wall-clock training time, TP-GRPO consistently outperforms Flow-GRPO on human preference alignment despite higher time cost per training step.The additional sampling overhead provides fine-grained reward signals for reinforcement-learning optimization.

7. Conclusion

The paper identifies reward sparsity and local misalignment from uniformly propagating terminal rewards, along with missed within-trajectory dependencies from matched-timestep ranking. It proposes TurningPoint-GRPO, which uses step-level incremental rewards to model each denoising step more accurately.

  • Existing Flow-based GRPO methods uniformly propagate terminal rewards to all denoising steps, causing step-wise reward sparsity and local misalignment.
  • Matched-timestep group-wise ranking ignores within-trajectory dependencies and delayed effects.
  • TurningPoint-GRPO (TP-GRPO) addresses these issues using step-level incremental rewards to model each denoising step more accurately.

A. Experimental Results on FLUX.1-dev

On FLUX.1-dev, the method is evaluated with PickScore under the Flow-GRPO configuration and is reported to consistently outperform an unspecified baseline.

  • Experimental setup: Using FLUX.1-dev with PickScore, the experiments follow Flow-GRPO settings with α = 0.8, T = 6 training steps, T = 28 inference steps, G = 24, and guidance scale 3.5.The corresponding training curves are shown in Figure 9.
  • Results: The method consistently outperforms F… according to the reported training curves.The supplied passage truncates the baseline name after “F.”

B. Details of the Experimental Configuration

The experiments use the Flow-GRPO codebase, 32 NVIDIA H20 GPUs, prompt-level advantage computation, and task-specific KL penalties. The configuration also includes reported FLUX.1-dev training curves and listed reward models.

  • Experimental setup: Experiments build on the Flow-GRPO codebase and train all models using 32 NVIDIA H20 GPUs.Advantages are computed on a per-prompt basis.
  • Experimental setup: The KL penalty coefficient β is 0.0004 for Compositional Image Generation and Visual Text Rendering, and 0.0001 for Human Preference Alignment.The coefficient balances fast convergence and prevention of reward hacking.
  • Reported configuration materials: Figure 9 reports training curves with FLUX.1-dev as the base model.
  • Reported configuration materials: Table 2 lists the reward models used and provides their links.

C. Theoretical Analysis … E. Pseudocode

TP-GRPO addresses sparse, time-invariant supervision by using step-wise and aggregated long-term rewards, whose theoretical properties support trajectory-aware optimization. Its implementation balances reward replacement and computes intermediate rewards, turning-point effects, advantages, and GRPO updates.

  • C.1. Sign Consistency of Rewards Under Definition 4.1: Under Definition 4.1, a turning point’s local reward and aggregated long-term reward have the same sign.Lemma C.1 states this sign consistency directly.
  • C.2.1. SIGN CONSISTENCY OF LOCAL AND AGGREGATED REWARDS: Under Definition 5.1, local and aggregated long-term rewards likewise share the same nonzero sign at turning points.The proof attributes this to the consistency condition and strict inequalities excluding the zero case.
  • C.2.2. COMPARISON OF ABSOLUTE VALUES OF LOCAL AND AGGREGATED REWARDS: For Definition 5.1 turning points satisfying the stated positive-product condition, the aggregated long-term reward has greater absolute magnitude than the local reward.The proof handles both positive and negative reward cases to establish the magnitude comparison.
  • C.3. Why Sparse Rewards Are Problematic for Flow-GRPO: Flow-GRPO’s constant outcome-based advantage gives identical supervision across diverse denoising actions despite time-dependent inputs.TP-GRPO instead makes the advantage step-dependent through local or aggregated rewards, restoring the mapping between timestep conditioning and optimization signal.
  • D. Balancing Operations for Improved Optimization with TP-GRPO: TP-GRPO retains equal numbers of positive and negative aggregated-reward samples for replacement, discarding the smallest magnitudes.This balancing strategy prevents either reward-sign partition from dominating optimization.
  • E. Pseudocode: The pseudocode samples noisy trajectories, computes intermediate rewards, detects turning points, selects local or aggregated rewards, and updates the policy with GRPO.Advantages are computed from step-wise rewards and applicable aggregated rewards before parameter updates.

F. More Analysis on Turning Points … F.3. Average Count of Good and Bad Turning Points

The section analyzes turning points through discussions and visualizations, contrasting TP-GRPO’s credit assignment with prior methods, examining its reward signals, and tracking turning-point frequency during training.

  • F. More Analysis on Turning Points: The section uses additional discussions and visualizations to illustrate the effects of turning points.
  • F.1. Contrast with Credit Assignment Techniques: Prior credit-assignment methods rely on algebraic constraints such as Trajectory Balance and an explicit target density.The cited prior works are Malkin et al. (2022) and Madan et al. (2023).
  • F.1. Contrast with Credit Assignment Techniques: TP-GRPO addresses continuous denoising trajectories optimized with external, non-differentiable image-scoring models, where those prior structures are unavailable.
  • F.2. Subtraction-based Rewards for Normal Points vs. Turning Points: At intermediate step t = 6, turning-point and normal-point reward distributions are distinct across training iterations.The analysis separately computes mean rewards for the two point categories.
  • F.2. Subtraction-based Rewards for Normal Points vs. Turning Points: Assigning different reward signals to turning points and normal points tailors feedback to each action’s role within the trajectory.The method integrates both reward types into the total reward.
  • F.3. Average Count of Good and Bad Turning Points: The combined count of good and bad turning points remains approximately 4–6/36, or roughly 0.11–0.16, throughout training.The count is measured within batches of 36 samples, with good points guiding trajectories toward higher rewards and bad points toward lower rewards.

G. More Qualitative Comparisons

Additional qualitative comparisons across three tasks show that TP-GRPO addresses Flow-GRPO’s failures in compositional accuracy, visual text rendering, and human-preference alignment.

  • Task Coverage: Qualitative comparisons cover Compositional Image Generation, Visual Text Rendering, and Human Preference Alignment.Results appear in Figures 12, 13, and 14, respectively.
  • Compositional Image Generation: In Compositional Image Generation, Flow-GRPO sometimes adds unnecessary objects and loses fine details, which the authors attribute to sparse outcome-based rewards.Examples include an unnecessary sandwich and hard-to-discern details.
  • Visual Text Rendering: In Visual Text Rendering, Flow-GRPO occasionally omits short words or overlaps characters, whereas TP-GRPO performs consistently well while preserving image aesthetics.One example omits “the” from a rendered sample.
  • Human Preference Alignment: In Human Preference Alignment, TP-GRPO captures prompt details, aligns styles, and produces more reasonable layouts than Flow-GRPO.Examples include rendering “webs,” matching “digital art,” and separating a city landscape from a planet.
Loading 2602.06422v2…