PaperlayerView exact revision on arXiv

Source-linked AI summary

DanceGRPO: Unleashing GRPO on Visual Generation

Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, Ping Luo

arXiv:2505.07818v4cs.CV

TL;DR

Prior visual-generation RL methods face optimization instability at larger prompt scales, and video-generation validation remains limited. DanceGRPO reformulates diffusion and rectified-flow sampling with SDEs and applies GRPO, achieving broad performance improvements across visual-generation settings, including gains of up to 181% over baselines.

  • Problem

    Prior policy-gradient methods such as DDPO and DPOK show instability beyond small datasets, while existing methods remain unvalidated for video generation.

  • Method

    DanceGRPO formulates diffusion-model and rectified-flow sampling as SDE-based MDPs and applies a GRPO-style objective across visual-generation tasks.

  • Results

    DanceGRPO outperforms baselines by up to 181% across HPSv2.1, CLIP score, VideoAlign, and GenEval.

  • Takeaways & Limitations

    DanceGRPO provides a unified framework for diffusion and rectified-flow models across text-to-image, text-to-video, and image-to-video generation.

  • Takeaways & Limitations

    Systematic integration of the explored extensions is deferred to future research.

Abstract

from arXiv · show

Recent advances in generative AI have revolutionized visual content creation, yet aligning model outputs with human preferences remains a critical challenge. While Reinforcement Learning (RL) has emerged as a promising approach for fine-tuning generative models, existing methods like DDPO and DPOK face fundamental limitations - particularly their inability to maintain stable optimization when scaling to large and diverse prompt sets, severely restricting their practical utility. This paper presents DanceGRPO, a framework that addresses these limitations through an innovative adaptation of Group Relative Policy Optimization (GRPO) for visual generation tasks. Our key insight is that GRPO's inherent stability mechanisms uniquely position it to overcome the optimization challenges that plague prior RL-based approaches on visual generation. DanceGRPO establishes several significant advances: First, it demonstrates consistent and stable policy optimization across multiple modern generative paradigms, including both diffusion models and rectified flows. Second, it maintains robust performance when scaling to complex, real-world scenarios encompassing three key tasks and four foundation models. Third, it shows remarkable versatility in optimizing for diverse human preferences as captured by five distinct reward models assessing image/video aesthetics, text-image alignment, video motion quality, and binary feedback. Our comprehensive experiments reveal that DanceGRPO outperforms baseline methods by up to 181\% across multiple established benchmarks, including HPS-v2.1, CLIP Score, VideoAlign, and GenEval. Our results establish DanceGRPO as a robust and versatile solution for scaling Reinforcement Learning from Human Feedback (RLHF) tasks in visual generation, offering new insights into harmonizing reinforcement learning and visual synthesis.

1 Introduction

DanceGRPO adapts GRPO for visual generation to address instability and limited scalability in prior reinforcement-learning methods. It unifies optimization across generative paradigms, tasks, models, and reward objectives while reporting substantial benchmark gains.

  • Motivation: Prior RL methods become unstable beyond small prompt datasets, and existing methods remain unvalidated for video generation.The cited challenges include instability when scaling beyond datasets with fewer than 100 prompts.
  • Method: DanceGRPO reformulates diffusion and rectified-flow sampling as SDEs and applies GRPO to stabilize visual-generation training.The framework addresses the mismatch between rectified-flow ODE sampling and Markov Decision Process formulations.
  • Evaluation Scope: DanceGRPO is evaluated across diffusion models and rectified flows for text-to-image, text-to-video, and image-to-video generation.The study uses diverse foundation models and reward metrics covering aesthetic quality, alignment, and motion dynamics.
  • Generalization and Scalability: DanceGRPO adapts across diverse generative paradigms, tasks, foundational models, and reward models, including large-scale datasets.The authors position this as a unified RL-based framework extending beyond prior validation mainly on small-scale text-to-image diffusion datasets.
  • Results: DanceGRPO outperforms baselines by up to 181% across HPSv2.1, CLIP score, VideoAlign, and GenEval.The framework also supports Best-of-N denoising-trajectory learning and initial learning from binary reward feedback.

2 Approach

DanceGRPO unifies diffusion and rectified-flow sampling within stochastic MDP-compatible processes, then applies GRPO to stabilize visual-generation optimization across tasks and models.

  • MDP formulation: Diffusion and rectified-flow sampling processes are formulated as Markov Decision Processes for policy optimization.The state includes the prompt, timestep, and noisy latent; actions represent transitions toward lower noise, with rewards assigned to generated outputs.
  • Sampling SDEs: Stochastic Differential Equations unify diffusion and rectified-flow sampling, enabling the trajectory exploration required by GRPO.Rectified flow’s deterministic reverse ODE is augmented with stochasticity, while diffusion sampling is represented through forward and reverse SDEs.
  • GRPO optimization: DanceGRPO samples groups of outputs from an old policy and updates the current policy using group-relative rewards and advantages.The method computes advantages from rewards within each output group, applies the GRPO-style objective, and generally omits KL regularization based on minimal observed differences.
  • Training design: Shared initialization noise for identical prompts prevents reward-hacking phenomena and training instability observed with different noise vectors in video generation.The framework also permits omitting subsets of denoising timesteps without compromising performance, reducing computation while maintaining output quality.
  • Reward modeling: Multiple reward models are combined through advantage aggregation to improve training stability and visual quality across distinct preference signals.For example, adding CLIP scores to HPS-v2.1 rewards helps preserve realistic image characteristics instead of producing unnatural outputs.
  • Inference scaling: Best-of-N training focuses on top-k and bottom-k candidates to emphasize high-reward and critical low-reward regions of the solution space.The paper reports that systematic integration of tree-search and greedy-search alternatives is deferred to future research.
  • Scope and evaluation: The framework evaluates diffusion and rectified-flow paradigms across text-to-image, text-to-video, and image-to-video generation using four foundation models.The selected models are Stable Diffusion, HunyuanVideo, FLUX, and SkyReels-I2V.

3 Experiments

DanceGRPO is evaluated across text-to-image, text-to-video, and image-to-video generation using diverse models, rewards, and large prompt sets. It improves reward-based and human evaluations while revealing practical effects of reward selection, timestep coverage, initialization noise, and sample scaling.

  • Experimental Setup: DanceGRPO evaluates Stable Diffusion, FLUX, HunyuanVideo-T2I, HunyuanVideo, and SkyReels-I2V across three visual-generation tasks and diverse reward metrics.The experiments use more than 10,000 optimization prompts and task-specific evaluation sets.
  • Text-to-Image Generation: DanceGRPO raises Stable Diffusion’s HPS score from 0.239 to 0.365 and CLIP Score from 0.363 to 0.395.The evaluation also includes Pick-a-Pic and GenEval metrics.
  • Reward and Scaling Analyses: Applying CLIP score can decrease HPS while producing more natural images and improving CLIP score.The framework also adapts to thresholded binary rewards and supports Best-of-N sample selection, which substantially accelerates convergence.
  • Text-to-Video Generation: 181% and 56% relative improvements are reported for HunyuanVideo visual aesthetics and motion quality using VideoAlign rewards.The text-video alignment dimension was excluded because it was unstable; sampling more frames per second improved training stability.
  • Image-to-Video Generation: 118% relative improvement is achieved in SkyReels-I2V motion quality, while optimization targets motion because input images constrain visual fidelity and text-video alignment.Classifier-free-guidance training is enabled to preserve sampling quality during RLHF.
  • Human Evaluation: Human artists consistently prefer outputs refined with RLHF across FLUX text-to-image, HunyuanVideo text-to-video, and SkyReels image-to-video.This comparison complements the reward-curve and metric-based evaluations.
  • Ablation Study: The first 30% of timesteps contribute disproportionately to foundational pattern learning, but training only on that interval performs worse than full-sequence training.The ablation attributes the degradation to insufficient exposure to late-stage refinement dynamics.

4 Related Work

Prior visual-alignment work includes DPO-style methods, direct reward backpropagation, and policy-gradient approaches. Compared with LLM alignment, alignment for diffusion models and rectified flows remains relatively underexplored.

  • LLM Alignment: LLM alignment commonly uses RLHF, where a reward function trained from output comparisons guides reinforcement-learning policy optimization.Other LLM approaches use Direct Policy Optimization.
  • LLM Alignment: DeepSeek-R1 demonstrated large-scale reinforcement learning with formatting and result-only rewards for complex reasoning in language models.The passage presents this as evidence of reinforcement learning’s potential for reasoning capabilities.
  • Visual Generation Alignment: Visual-generation alignment methods include DPO-style optimization, direct backpropagation through reward signals, and policy-gradient methods such as DPOK and DDPO.These approaches target diffusion models and rectified flows.

5 Conclusion and Future Work

DanceGRPO integrates GRPO into visual generation across diffusion models, rectified flows, and three image/video tasks, improving visual fidelity, motion quality, and text-image alignment.

  • DanceGRPO unifies GRPO-based optimization across diffusion models and rectified flows for text-to-image, text-to-video, and image-to-video generation.
  • Experiments demonstrate improvements in visual fidelity, motion quality, and text-image alignment.
  • The framework is designed for efficient human-preference alignment and robust scaling to complex, multi-task settings.

A Experimental Settings

The experimental settings specify that Table 6 applies to training without classifier-free guidance, while CFG uses one gradient update per iteration and model-specific sampling steps.

  • Table 6 reports hyperparameters for training without classifier-free guidance.
  • With classifier-free guidance enabled, training uses one gradient update per iteration.
  • Sampling uses 50 steps for Stable Diffusion and 25 steps for FLUX and HunyuanVideo.

B More Analysis

Stochastic interpolants provide a common framework for rectified flows and score-based diffusion by bridging probability densities through continuous-time processes with ODE or SDE realizations.

  • B.1 Stochastic Interpolants: The stochastic interpolant framework constructs a continuous-time stochastic process bridging arbitrary densities ρ0 and ρ1.
  • B.1 Stochastic Interpolants: Spatially linear interpolants combine endpoint samples and Gaussian noise through smooth functions α, β, and γ.
  • B.1 Stochastic Interpolants: The boundary conditions set α(0)=β(1)=1 and α(1)=β(0)=γ(0)=γ(1)=0, with γ(t)≥0 between endpoints.
  • B.1 Stochastic Interpolants: Setting γ(t)=0 with α(t)=1−t and β(t)=t recovers the linear interpolation used in Rectified Flow.
  • B.1 Stochastic Interpolants: One-sided linear interpolants connect the framework to score-based diffusion models, typically using Gaussian ρ1.
  • B.1 Stochastic Interpolants: The same interpolant density can be realized by an ODE or forward and backward SDEs whose drifts can be learned from data.

B.2 Connections between Rectified Flows and Diffusion Models

The paper derives parameter relationships that align diffusion SDEs with stochastic interpolants by matching their marginal means, variances, and noise terms, establishing equivalence under specified conditions.

  • The analysis aims to derive diffusion-model hyperparameters from a stochastic interpolant by matching the two formulations.
  • The forward diffusion process is represented as an SDE with Brownian motion and noise-schedule terms.
  • Flow matching interpolates between data x and noise ϵ through a path governed by an ODE, generalized here to a stochastic interpolant SDE.
  • The derivation matches the forward diffusion process's marginal distributions to those implied by the interpolant path and relates the generative SDE noise terms.
  • The drift term is derived by matching means, while the diffusion coefficient is derived by matching variances and noise terms.
  • The resulting relationships establish equivalence between the parameters of the two frameworks under the specified conditions.

C Classifier-Free Guidance (CFG) Training

DanceGRPO handles CFG differently across model types because CFG can destabilize optimization, while some models require it for reasonable sample quality. Its training objective also uses grouped advantages, shared prompt-level noise, and generalization across diffusion and rectified-flow video tasks.

  • CFG handling: Disabling CFG during sampling reduces gradient oscillation while preserving output quality for high-fidelity models such as HunyuanVideo and FLUX.CFG integration introduces instability during optimization in these settings.
  • CFG handling: CFG-dependent models such as SkyReels-I2V and Stable Diffusion require joint optimization of conditional and unconditional outputs to avoid divergent trajectories.This doubles VRAM consumption because both networks are computed.
  • DanceGRPO design: DanceGRPO computes advantages within prompt-level groups and maintains noise consistency across samples from each prompt.The framework extends these choices beyond diffusion models to rectified flows and video generation.

E Inserting DDPO into Rectified Flow SDEs

Applying a DDPO-style objective to rectified-flow SDEs does not provide stable optimization: the approach diverges, whereas DanceGRPO remains superior in the corresponding visualization.

  • DDPO in rectified-flow SDEs: DDPO always diverges when applied to rectified-flow SDEs.The divergence is shown in the paper’s visualization comparison.
  • Comparison: DanceGRPO outperforms the DDPO insertion in the rectified-flow SDE comparison.Figure 5 is presented as evidence of DanceGRPO’s superiority.

F More Visualization Results

The paper provides additional visualizations of RLHF outputs across FLUX, Stable Diffusion, HunyuanVideo, HunyuanVideo-T2I, and SkyReels-I2V. The figures compare original outputs with reward-driven optimizations and illustrate changes in brightness, detail, diversity, reward hacking, and naturalness.

  • Additional visualizations: Additional visualizations cover FLUX, Stable Diffusion, HunyuanVideo, HunyuanVideo-T2I, and SkyReels-I2V.The results include overall before-and-after RLHF comparisons and model-specific examples.
  • FLUX results: FLUX outputs optimized with HPS tend toward brighter tones and richer details across iterations 0, 60, 120, 180, 240, and 300.The paper states that CLIP score regularization is crucial for the accompanying examples.
  • Diversity: Different seeds tend to generate similar images after RLHF, illustrating reduced output diversity.This comparison is shown in the visualization of model diversity before and after RLHF.
  • Reward hacking: Using different initialization noise vectors can produce reward hacking and training instability in HunyuanVideo, unlike using the same initialization noise.The figure compares both settings for the prompt describing sparkling water in a clear glass.
  • CLIP regularization: HPS-only training can produce unnatural oily outputs for a cup prompt, while adding CLIP scores helps maintain more natural image characteristics.The comparison is shown for the prompt “A photo of cup.”
  • Reward comparisons: Figures compare original FLUX, HunyuanVideo-T2I, and Stable Diffusion outputs with HPS-only and HPS-plus-CLIP optimizations.The comparisons are presented across Figures 11–15.
Loading 2505.07818v4…