PaperlayerView exact revision on arXiv

Source-linked AI summary

Manifold-Aware Exploration for Reinforcement Learning in Video Generation

Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, Qifeng Chen, Harry Yang

arXiv:2603.21872v1cs.CVcs.AI

TL;DR

Video GRPO remains unreliable because ODE-to-SDE exploration can inject excess noise into a complex video solution space, harming rollout and reward reliability. SAGE-GRPO constrains exploration near the pre-trained model’s manifold through precise micro-level sampling, gradient equalization, and macro-level dual trust regions, and reports consistent gains on HunyuanVideo1.5 with the original VideoAlign evaluator.

  • Problem

    Video GRPO must explore a large structured solution space, but ODE-to-SDE conversions can inject excess noise that reduces rollout quality and makes reward-guided updates less reliable.

  • Method

    SAGE-GRPO combines a Precise Manifold-Aware SDE and Gradient Norm Equalizer with a Dual Trust Region using moving anchors and stepwise constraints.

  • Results

    SAGE-GRPO shows consistent gains over DanceGRPO, FlowGRPO, and CPS in overall reward and temporal fidelity on HunyuanVideo1.5 using the original VideoAlign evaluator.

  • Takeaways & Limitations

    The method provides a manifold-aware framework for more stable reinforcement-learning alignment of text-to-video generation models.

  • Takeaways & Limitations

    Experiments use an existing model and evaluator, while the user study includes 29 voluntary evaluators rating 32 prompts.

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.

1. Introduction

Video GRPO exploration can drift off the pre-trained model’s valid video manifold, reducing rollout and reward reliability. SAGE-GRPO addresses this with manifold-aware micro-level sampling and macro-level trust-region constraints.

  • Problem: Video GRPO converts deterministic ODE sampling into stochastic exploration, but first-order SDE approximations can inject excess noise and reduce rollout quality.The resulting exploration may make reward-guided updates less reliable.
  • Problem: The paper formulates stable video alignment as constraining exploration near the valid data manifold defined by the pre-trained generator.This formulation aims to preserve valid rollouts and reliable reward evaluation during updates.
  • Results: SAGE-GRPO shows consistent gains over DanceGRPO, FlowGRPO, and CPS in overall reward and temporal fidelity on HunyuanVideo1.5 using the original VideoAlign evaluator.The evaluation uses no reward-model fine-tuning.
  • Results: Ablations indicate that both micro-level exploration design and the macro-level Dual Trust Region are necessary to reduce the stability–plasticity gap.
  • Method: SAGE-GRPO uses a Precise Manifold-Aware SDE with logarithmic curvature correction and a Gradient Norm Equalizer for balanced, manifold-consistent exploration.The SDE computes stepwise noise variance more accurately, while gradient equalization balances optimization pressure across timesteps.
  • Method: A Dual Trust Region combines periodic moving anchors with stepwise constraints to track more manifold-consistent policies and limit long-horizon drift.The objective provides position control toward the manifold and velocity control between successive policies.

2. Related Work

Related work adapts reinforcement learning and preference alignment to diffusion, flow-matching, and video-generation models. Existing efforts include video reward models, video alignment algorithms, and GRPO-based methods such as DanceGRPO and Self-paced GRPO.

  • Reinforcement Learning for Diffusion and Flow Matching Models: Reinforcement learning has been adapted to diffusion and flow-matching models for alignment with human preferences.
  • Preference Alignment for Video Generation: Video preference-alignment research has developed video reward models and alignment algorithms on top of video diffusion models.
  • Preference Alignment for Video Generation: DanceGRPO extends image-based reinforcement learning to video, while Self-paced GRPO dynamically adjusts reward weights through curriculum learning.

3. Methodology

SAGE-GRPO stabilizes video-policy exploration by constraining noise near the data manifold, balancing timestep gradients, and combining moving-anchor with step-wise KL controls. Its framework links precise SDE sampling, temporal equalization, and dual trust regions to preserve exploration while limiting drift.

  • Overview: SAGE-GRPO formulates video alignment as reward maximization with a unified exploration strategy spanning micro-level noise injection and macro-level policy constraints.The method is designed within GRPO to stabilize exploration on the video manifold.
  • Micro-level exploration: The Gradient Norm Equalizer normalizes timestep-dependent optimization pressure, addressing vanishing high-noise gradients and exploding low-noise gradients.The method estimates per-timestep gradient scale and applies robust normalization so structural and textural updates contribute more evenly.
  • Micro-level exploration: The manifold-aware SDE keeps stochastic exploration aligned with the video manifold, using integrated diffusion variance and a logarithmic correction instead of a first-order area approximation.The resulting noise is intended to reduce off-manifold exploration and temporal artifacts while preserving Rectified Flow marginals.
  • Macro-level exploration: The framework combines moving-anchor and step-wise KL constraints into a dual trust region that provides position and velocity control during policy updates.Periodic reference updates create a resetting safe region, while step-wise constraints limit local transitions.
  • Macro-level exploration: Step-wise KL control alone permits cumulative drift from the initial parameters, whereas periodic moving anchors constrain long-horizon displacement while retaining local exploration.The combined mechanism is presented as a position-velocity controller for sustained plasticity.
  • Evaluation: Qualitative comparisons report reduced temporal jitter, improved photorealism and occlusion handling, and stronger semantic alignment across frames.These comparisons complement the method’s quantitative evaluation trends.

4. Experiments

Experiments on HunyuanVideo 1.5 compare SAGE-GRPO with baselines under averaged and alignment-focused rewards, using shared infrastructure and VideoAlign evaluation. SAGE-GRPO, especially Dual Moving KL, shows stronger reward, visual, and alignment performance, supported by ablations and qualitative comparisons.

  • Experimental Setup: Experiments use HunyuanVideo 1.5, VideoAlign, and VQ, MQ, and TA metrics, comparing SAGE-GRPO with DanceGRPO, FlowGRPO, and CPS.Training uses an effective batch size of 8, 81-frame videos, and GRPO updates every 20 diffusion sampling steps.
  • Experimental Setup: A unified training framework varies only the GRPO algorithm, reducing confounding from infrastructure differences across methods.
  • Main Results: Adding KL regularization typically improves visual performance but worsens reward behavior under averaged rewards, attributed to reward hacking in the reward model.
  • Main Results: Under alignment-focused rewards, SAGE-GRPO with Dual Moving KL achieves the best Overall, VQ, MQ, and CLIPScore while remaining close to the best TA.Table 2 indicates that emphasizing alignment provides more stable gains across reward and visual metrics.
  • Qualitative Analysis: Qualitative examples show improved coherence, photorealism, and semantic alignment, particularly for precise object interactions and long-range motion.Additional examples address alignment with emotional descriptions in text prompts.
  • User Preference Study: A user study with 29 evaluators and 32 prompts reports pairwise win rates for SAGE-GRPO against baselines across visual, motion, and semantic quality.Table 3 describes strong human preference, especially for Motion Quality.
  • KL Strategy Ablation: Dual Moving KL consistently outperforms other KL strategies in convergence speed and final reward while avoiding collapse from aggressive step-wise updates.It also maintains higher and more stable exploration than Moving KL during later training stages.
  • KL Weight Ablation: The two-stage KL schedule 10−7 →10−5 produces higher rewards and smoother trajectories across VQ, MQ, and TA than fixed or milder schedules.The result supports gradually increasing λKL to tighten the trust region.

5. Conclusion

The conclusion presents SAGE-GRPO as a manifold-aware framework for stable video-generation reinforcement learning, combining micro-level exploration controls with macro-level trust-region control. Experiments and ablations show consistent improvements over strong baselines and support the contribution of each component.

  • SAGE-GRPO constrains exploration near the video data manifold to reduce drift into high-noise regions.
  • The micro-level design combines a Precise Manifold-Aware SDE with a Gradient Norm Equalizer to keep noise near the flow trajectory and normalize optimization pressure across timesteps.
  • The macro-level Dual Trust Region combines position and velocity control to reduce off-manifold local optima while enabling sustained plasticity.
  • Experiments on Hunyuan-Video1.5 with VideoAlign show consistent improvements over strong baselines and validate each component through ablations.

Impact Statement

The impact statement describes SAGE-GRPO as a method for more stable reinforcement-learning alignment of text-to-video generation models while acknowledging deployment risks and study boundaries.

  • The method targets more stable reinforcement-learning alignment of text-to-video generation models.
  • Potential benefits include improved temporal consistency and text alignment for creative, scientific, and educational video tools under a fixed reward model.
  • The paper evaluates an existing model and evaluator in a research setting, with a user study involving 29 voluntary evaluators and 32 prompts.
  • Future deployment should address misinformation, deepfakes, bias, harmful content, and computational cost through moderation, auditing, and human oversight.

A.1. Derivation of Manifold-Aware SDE Variance

The appendix derives a marginal-preserving SDE for flow matching and computes its integrated variance for stochastic exploration. A logarithmic correction captures geometric contraction that linear approximations miss.

  • The SDE converts deterministic Rectified Flow dynamics into stochastic exploration while preserving the marginal distribution at each timestep.The Itô correction term is included to preserve the deterministic ODE's marginal distribution.
  • The derivation treats the drift and score as approximately constant over each interval and computes the integrated variance Σt for the stochastic term.
  • The diffusion coefficient is chosen with the geometric factor σt/(1−σt) and exploration scaling factor η.
  • The logarithmic correction log((1−σt+1)/(1−σt)) accounts for geometric contraction of the signal coefficient.Linear approximations fail to capture this contraction and the corresponding ideal variance scaling.
  • Euler-Maruyama discretization uses Δt = σt −σt+1, with injected stochasticity sampled as ϵ ∼ N(0, I).Σt is already the integrated variance over the interval and is used directly in the stochastic term.
  • The target variance is derived for a Rectified Flow trajectory connecting pure noise at σ = 1 to data at σ = 0.The objective is marginal preservation through second order.

A.2. Standard Deviation Comparison: Ours vs. FlowGRPO

Across three parameterization regimes, SAGE-GRPO produces smaller or more stable per-step noise standard deviations than FlowGRPO, supporting more precise exploration along the data manifold.

  • Regime (a): Near-zero standard deviation at the first step occurs when both methods use FlowGRPO’s σ schedule and the endpoints are equal.The integration interval collapses, and the logarithmic correction evaluates to zero, yielding Σt ≈ 0.
  • The integral-based formulation is sensitive to the σ schedule and requires proper boundary handling.
  • Regime (b): Under (1 − σ) ≥ 3 × 10^-3 clamping, FlowGRPO reaches standard deviation around 3.0 at the first step while SAGE-GRPO remains stable.SAGE-GRPO starts around 1.0 and decays smoothly, whereas FlowGRPO’s σ/(1 − σ) ratio explodes when the denominator is artificially small.
  • Regime (c): Under default implementations, SAGE-GRPO maintains lower standard deviation than FlowGRPO across most of the diffusion trajectory, especially later steps.Its clamping configuration reduces injected noise magnitude and supports refined manifold exploration.
  • Across all three regimes, SAGE-GRPO consistently achieves smaller or more stable standard deviation than FlowGRPO.This supports removing unnecessary high-frequency noise energy and refining exploration along the data manifold.

A.3. Theoretical Gradient Norm Analysis

The analysis links Gaussian-policy gradient norms to SDE noise magnitude and shows that low-noise timesteps can produce exploding gradients, motivating temporal gradient equalization.

  • For a Gaussian policy, the gradient of log-probability with respect to the drift parameter is analyzed through the transition’s noise variance.The derivation uses π(xt−1|xt) = N(µθ, ΣtI).
  • The step-wise standard-deviation comparison reports lower or more stable noise for the precise SDE, supporting more precise manifold exploration.
  • The expected gradient norm is proportional to the standard deviation of the sampling noise.
  • With Manifold-Aware variance Σt ≈ η^2∆σ σt/(1−σt), gradient norms inherit the timestep-dependent scaling of the noise schedule.
  • As σt → 0, the gradient norm explodes, necessitating the proposed Gradient Equalizer.

A.4. GRPO Reward and Advantage Details

SAGE-GRPO combines reward normalization, temporal gradient equalization, and dual KL regularization to stabilize GRPO optimization across timesteps and over long-horizon policy updates.

  • Reward composition: VideoAlign composes reward from visual quality, motion quality, and text alignment scores using fixed scalar weights.
  • Group-normalized advantage: GRPO samples a rollout group and uses its mean and standard deviation to define normalized advantages.A small constant ε provides numerical stability.
  • Temporal Gradient Equalizer: The Temporal Gradient Equalizer produces approximately uniform gradient scales across timesteps.The reported effect aligns with empirical observations in Figure 4 and training-curve improvement in Figure 3.
  • SAGE-GRPO objective: The complete SAGE-GRPO objective combines GRPO, the Temporal Gradient Equalizer, and an adaptively weighted Dual KL regularizer.
  • Dual KL regularization: Dual KL uses the previous policy for velocity control and a periodically refreshed anchor for position control.The two constraints separately limit instantaneous updates and cumulative displacement.
  • Adaptive KL weighting: Adaptive KL weighting adjusts the coefficient online so realized KL remains near a target scale, after warm-up and feedback control.The controller reduces the coefficient when empirical KL is too large and increases it when KL is too small.
Loading 2603.21872v1…