PaperlayerView exact revision on arXiv

Source-linked AI summary

DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment

Haoyou Deng, Keyu Yan, Chaojie Mao, Xiang Wang, Yu Liu, Changxin Gao, Nong Sang

arXiv:2601.20218v2cs.CV

TL;DR

Existing flow-matching GRPO methods use sparse terminal rewards for intermediate denoising steps, creating a credit-assignment mismatch. DenseGRPO estimates step-wise rewards and calibrates timestep-specific exploration, achieving superior performance across three text-to-image tasks.

  • Problem

    Existing flow-matching GRPO methods apply a trajectory-level terminal reward to all intermediate denoising steps, overlooking their fine-grained contributions.

  • Method

    DenseGRPO estimates step-wise rewards with an ODE-based approach and calibrates timestep-specific SDE noise injection using a reward-aware exploration scheme.

  • Results

    DenseGRPO outperforms competitors across all three tasks, surpassing them by at least 1.01 of PickScore in human preference alignment.

  • Takeaways & Limitations

    The experiments validate dense rewards as effective for fine-grained credit assignment and flow-matching model alignment.

  • Takeaways & Limitations

    DenseGRPO exhibits slight reward hacking in parts of the evaluated tasks.

Abstract

from arXiv · show

Recent GRPO-based approaches built on flow matching models have shown remarkable improvements in human preference alignment for text-to-image generation. Nevertheless, they still suffer from the sparse reward problem: the terminal reward of the entire denoising trajectory is applied to all intermediate steps, resulting in a mismatch between the global feedback signals and the exact fine-grained contributions at intermediate denoising steps. To address this issue, we introduce \textbf{DenseGRPO}, a novel framework that aligns human preference with dense rewards, which evaluates the fine-grained contribution of each denoising step. Specifically, our approach includes two key components: (1) we propose to predict the step-wise reward gain as dense reward of each denoising step, which applies a reward model on the intermediate clean images via an ODE-based approach. This manner ensures an alignment between feedback signals and the contributions of individual steps, facilitating effective training; and (2) based on the estimated dense rewards, a mismatch drawback between the uniform exploration setting and the time-varying noise intensity in existing GRPO-based methods is revealed, leading to an inappropriate exploration space. Thus, we propose a reward-aware scheme to calibrate the exploration space by adaptively adjusting a timestep-specific stochasticity injection in the SDE sampler, ensuring a suitable exploration space at all timesteps. Extensive experiments on multiple standard benchmarks demonstrate the effectiveness of the proposed DenseGRPO and highlight the critical role of the valid dense rewards in flow matching model alignment.

1 INTRODUCTION

DenseGRPO addresses sparse-reward misalignment in flow-matching model alignment by estimating step-wise rewards and calibrating timestep-specific exploration. Experiments on multiple text-to-image benchmarks demonstrate its effectiveness and the importance of dense rewards.

  • Motivation: Existing GRPO-based methods apply one terminal trajectory reward to every intermediate denoising step, mismatching global feedback with fine-grained contributions.This mismatch can mislead policy optimization.
  • Motivation: Dense rewards reveal a mismatch between uniform exploration and time-varying noise intensity in existing GRPO-based approaches.The mismatch produces an inappropriate exploration space across timesteps.
  • Method: DenseGRPO estimates reliable step-wise dense rewards for individual denoising steps using an ODE-based approach.The method aligns human-preference feedback with each step’s contribution.
  • Method: DenseGRPO calibrates the exploration space with a reward-aware scheme that balances dense reward distributions across timesteps.It adaptively adjusts stochasticity injection in the SDE sampler.
  • Results: Comprehensive experiments on multiple text-to-image benchmarks demonstrate state-of-the-art performance and highlight dense rewards’ critical role in flow-matching model alignment.The evaluation covers multiple standard benchmarks.

2 RELATED WORK

Prior text-to-image alignment methods use scalar preferences, reward-weighted regression, or offline DPO, while dense-reward methods address trajectory-level sparsity in sequential generation. Existing densification strategies include per-step KL penalties and token-entropy-based reward weighting.

  • Alignment for Text-to-Image Generation: Text-to-image alignment has used scalar preference rewards and reward-weighted regression to guide model training.The cited approaches include Prabhudesai et al. (2023), Xu et al. (2023), Lee et al. (2023), and Furuta et al. (2024).
  • Alignment for Text-to-Image Generation: Other text-to-image alignment approaches remove the need for a reward model by adopting offline Direct Preference Optimization.The passage cites Wallace et al. (2024) and Yang et al. (2024a) as examples.
  • Dense Reward: Dense reward methods address sparse trajectory-level feedback in sequential generation model alignment.The passage describes dense reward as effective for addressing the sparse reward issue inherent in trajectory-level feedback.
  • Dense Reward: Text-generation methods densify rewards through per-step KL penalties or token-level entropy-based dynamic reward weighting.Ramamurthy et al. (2022) and Castricato et al. (2022) use per-step KL penalties, while Tan & Pan (2025) dynamically weights rewards using token-level entropy.

3 PRELIMINARY

This section formulates flow-matching denoising as an MDP and reviews Flow-GRPO’s trajectory-level optimization and stochastic sampling. It highlights that terminal-only rewards provide identical feedback across timesteps, mismatching step-wise denoising contributions.

  • RL on Flow Matching Models: Flow-matching denoising is modeled as an MDP whose state contains the prompt, timestep, and latent, while the action is the predicted previous latent.The policy is π(a_t | s_t) ≜ p(x_{t−1} | x_t, c).
  • RL on Flow Matching Models: The reward is sparse: it is applied only at the terminal state and is zero at intermediate denoising steps.This assigns the entire trajectory’s reward to the final denoising step.
  • GRPO Framework: Flow-GRPO samples a group of images and estimates each image’s advantage through group normalization before optimizing the policy.The framework uses a reward model to evaluate sampled trajectories and construct the optimization signal.
  • GRPO Framework: Because the advantage depends only on the trajectory’s final reward, all timesteps receive identical feedback despite differing step-wise contributions.This creates a mismatch between trajectory-wise feedback and the contributions of individual denoising steps.
  • SDE Sampler: Flow-matching models use a deterministic ODE sampler, but Flow-GRPO converts it to an SDE sampler by injecting noise for diverse trajectory exploration.The noise level is controlled by the scalar hyper-parameter a.

4 DENSEGRPO

DenseGRPO replaces trajectory-level sparse feedback with step-wise dense rewards estimated from intermediate clean images, then calibrates timestep-specific SDE stochasticity to balance exploration and reward signals. This aligns denoising-step contributions with policy optimization and provides a suitable exploration space across timesteps.

  • Dense Rewards: DenseGRPO estimates each denoising step’s contribution as a reward gain, replacing the single trajectory reward applied uniformly to intermediate steps.This addresses the mismatch between trajectory-wise feedback and step-wise contributions, facilitating effective policy optimization.
  • Dense Rewards: ODE denoising maps each intermediate latent to a deterministically corresponding clean image whose reward can be predicted by an established reward model.The method exploits the one-to-one mapping from an intermediate latent through the ODE trajectory to the final clean image, avoiding a learned critic.
  • Dense Rewards: The dense-reward estimator performs n-step ODE denoising, uses n = t in experiments, decodes the resulting clean image, and applies reward model R.The resulting clean-image reward is assigned as the latent reward for the intermediate state.
  • Exploration Space Calibration: Uniform noise injection creates unsuitable exploration spaces because noise diversity and reward balance vary across denoising timesteps.The imbalance is especially pronounced at timesteps 3 and 2, while greater reward diversity appears at timestep 10.
  • Exploration Space Calibration: DenseGRPO adaptively increases timestep-specific noise when dense rewards are balanced and decreases it otherwise, producing a balanced exploration space across timesteps.Higher noise promotes exploration diversity, whereas lower noise improves reward balance.

5 EXPERIMENT

DenseGRPO is evaluated across three text-to-image tasks and outperforms Flow-GRPO and Flow-GRPO+CoCA across them. Ablations attribute its gains to step-wise dense rewards, time-specific exploration noise, and sufficiently accurate ODE denoising.

  • Experimental Setup: DenseGRPO is evaluated on compositional image generation, human preference alignment, and visual text rendering using GenEval, PickScore, and OCR accuracy rewards.The setup uses sampling timestep T = 10, evaluation timestep T = 40, group size G = 24, and 512-resolution images.
  • Main Results: DenseGRPO outperforms Flow-GRPO and Flow-GRPO+CoCA across all three evaluated tasks.Flow-GRPO+CoCA adapts CoCA’s latent-similarity step-wise reward idea to flow matching models.
  • Effect of Dense Reward: Step-wise dense rewards outperform trajectory-wise sparse rewards and improve policy optimization by aligning feedback with each denoising step’s contribution.Dense Reward (Baseline) performs better than Flow-GRPO, while Flow-GRPO+CoCA also surpasses vanilla Flow-GRPO.
  • Effect of Exploration Space Calibration: Time-specific noise levels advance alignment compared with the uniform a = 0.7 setting by providing a more suitable exploration space across timesteps.The ablation validates the reward-aware exploration-space calibration scheme.
  • Effect of Different ODE Denoising Steps: Increasing the ODE denoising-step count improves performance, whereas single-step ODE denoising performs worse than Flow-GRPO.The ablation evaluates n = 1, 2, and t, linking denoising accuracy to dense-reward quality.
  • Discussion of Reward Hacking: DenseGRPO shows outstanding alignment capability on DrawBench, with slight reward hacking in parts of the evaluated tasks.The discussion additionally evaluates Aesthetic Score, DeQA, ImageReward, and UnifiedReward alongside PickScore.

6 CONCLUSION

DenseGRPO addresses the mismatch between trajectory-wise rewards and step-wise denoising contributions by estimating per-timestep dense rewards. It further calibrates exploration in the SDE sampler using a reward-aware scheme based on those estimated rewards.

  • 6 CONCLUSION: DenseGRPO addresses the mismatch between trajectory-wise reward feedback and each denoising step’s contribution.The framework targets the discrepancy between global trajectory feedback and step-wise credit assignment.
  • 6 CONCLUSION: An ODE-based approach estimates per-timestep dense rewards, aligning feedback with individual denoising-step contributions.This enables fine-grained credit assignment and facilitates effective optimization.
  • 6 CONCLUSION: DenseGRPO uses estimated dense rewards to propose a reward-aware scheme for calibrating exploration in the SDE sampler.The scheme is designed to address the sampler’s current imbalance in exploration.

ETHICS STATEMENT

The work follows the ICLR Code of Ethics, uses publicly available datasets under their licenses, and reports no human-subject, sensitive-data, proprietary-content, or foreseeable misuse concerns.

  • The work adheres to the ICLR Code of Ethics.
  • All datasets are publicly available and used according to their respective licenses.
  • The research involves no human subjects, sensitive personal information, or proprietary content.
  • The proposed methods present no foreseeable risks of misuse or harm.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility by detailing DenseGRPO, implementation settings, and publicly available datasets across the main text and appendix.

  • REPRODUCIBILITY STATEMENT: DenseGRPO, experimental settings, hyperparameters, training procedures, evaluation metrics, and publicly available datasets are documented in Sections 4, 5.1, and A.Section 4 describes DenseGRPO; Sections 5.1 and A provide implementation and dataset details.

A IMPLEMENTATION DETAIL · B MORE RESULT

The experiments use the official Flow-GRPO implementation with 16 NVIDIA A100 GPUs and a fixed, precomputed exploration-space calibration. The setup otherwise follows Flow-GRPO for fair comparison.

  • A IMPLEMENTATION DETAIL: Experiments are based on the official implementation of Flow-GRPO.The cited passage identifies Liu et al., 2025 as the Flow-GRPO reference.
  • A IMPLEMENTATION DETAIL: Models are trained using 16 NVIDIA A100 GPUs.
  • A IMPLEMENTATION DETAIL: The exploration-space calibration strategy first generates the noise level ψ(t) before training.The passage refers to Algorithm 1 for this calibration.
  • A IMPLEMENTATION DETAIL: The calibration parameters ε1 and ε2 are set to 2 and 0.01, respectively.
  • A IMPLEMENTATION DETAIL: The obtained ψ(t) remains fixed throughout training.
  • A IMPLEMENTATION DETAIL: The experimental settings match Flow-GRPO to ensure a fair comparison.

B.1 TRAINING CURVE OF KL LOSS · B.2 ACCURACY OF DENSEGRPO’S REWARD

DenseGRPO exhibits a slightly higher KL loss than Flow-GRPO during training, attributed to timestep-specific noise that expands exploration diversity. Its dense reward is visualized through ODE-based latent rewards along sampled trajectories, with timestep = 0 denoting the terminal reward.

  • B.1 TRAINING CURVE OF KL LOSS: DenseGRPO’s KL loss is slightly larger than Flow-GRPO’s during training.
  • B.1 TRAINING CURVE OF KL LOSS: Timestep-specific noise encourages a more diverse exploration space in DenseGRPO.
  • B.1 TRAINING CURVE OF KL LOSS: The expanded exploration space pushes DenseGRPO farther from the original model.
  • B.1 TRAINING CURVE OF KL LOSS: Figure 7 visualizes KL-loss training curves for compositional image generation, visual text rendering, and human preference alignment.
  • B.2 ACCURACY OF DENSEGRPO’S REWARD: Figure 8 visualizes ODE-based latent rewards, denoted R_i and predicted by Eq. 9.
  • B.2 ACCURACY OF DENSEGRPO’S REWARD: Each polyline in the reward visualization represents a sampled trajectory.
  • B.2 ACCURACY OF DENSEGRPO’S REWARD: timestep = 0 represents the terminal reward of the SDE sampling trajectory.

B.3 MORE EXPERIMENT · B.4 REWARD HACKING ANALYSIS · SD 3.5-M Flow-GRPO DenseGRPO

DenseGRPO improves over Flow-GRPO across FLUX.1-Dev and 1024 × 1024 SD 3.5-M experiments, while its dense-reward approach also generalizes to diffusion models through deterministic sampling. These results support the effectiveness, scalability, and broader applicability of the estimated dense rewards.

  • B.3 MORE EXPERIMENT: On FLUX.1-Dev with PickScore, DenseGRPO achieves substantial improvements over Flow-GRPO, demonstrating the robustness of its estimated dense rewards.The experiment evaluates DenseGRPO against Flow-GRPO on FLUX.1-Dev using PickScore as the reward model.
  • B.3 MORE EXPERIMENT: At 1024 × 1024 resolution on SD 3.5-M with PickScore, DenseGRPO yields a significant gain over Flow-GRPO, indicating strong scalability.Both training and inference use the higher 1024 × 1024 resolution.
  • B.3 MORE EXPERIMENT: DenseGRPO generalizes beyond flow matching models by using a deterministic sampler to predict dense rewards for other generative models.The deterministic sampler provides a one-to-one mapping between intermediate latents and clean latents for accurate latent-reward prediction.
  • B.3 MORE EXPERIMENT: For SD 1.5, an ODE sampler predicts clean latents whose rewards are assigned to intermediate latents to calculate step-wise dense rewards.The provided implementation passages describe the reward assignment and gain calculation.
  • B.3 MORE EXPERIMENT: On the diffusion-model experiment, dense rewards improve performance within DenseGRPO, demonstrating their accuracy and effectiveness.The findings support generalization to other generative families through a deterministic denoising sampler.
Loading 2601.20218v2…