PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Generation Models Are Good Latent Reward Models

Xiaoyue Mi, Wenqing Yu, Jiesong Lian, Shibo Jie, Ruizhe Zhong, Zijun Liu, Guozhen Zhang, Zixiang Zhou, Zhiyong Xu, Yuan Zhou, Qinglin Lu, Fan Tang

arXiv:2511.21541v3cs.CV

TL;DR

Existing VLM-based video rewards rely on RGB inputs, limiting optimization to late denoising and incurring decoding costs. The paper introduces PAVRM and PRFL to assess and optimize rewards from noisy latents across timesteps. It reports motion-quality gains, large memory reductions, and at least 1.4× training speedup against pixel-space methods.

  • Problem

    VLM-based video rewards use RGB inputs, requiring near-complete denoising and VAE decoding, while pixel-space video optimization faces computational barriers from multi-frame synthesis.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    PAVRM evaluates noisy latent representations using query-based aggregation, and PRFL optimizes process rewards by sampling timesteps without VAE decoding.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    At least 1.4× training speedup versus pixel-space methods accompanies substantial motion-quality improvements and large memory reduction.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The method focuses on motion quality, and the authors note that multi-aspect evaluation of aesthetics and semantics may be beneficial.

    Source: §6 · p. 9

    The process-aware supervision rationale assumes rectified-flow interpolation lets final-output preference labels propagate to intermediate states.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.

1. Introduction

Video generation models can produce visually faithful content, but aligning them with human preferences remains difficult, and applying image-oriented reward feedback learning to video faces computational and optimization barriers. The paper proposes process-level latent reward modeling and optimization to address these challenges.

1. Introduction

  • Video preference alignment must address motion quality, physical plausibility, and prompt following, beyond visual fidelity alone.The introduction identifies these as persistent alignment challenges despite recent gains in video generation quality.Source: §1 · p. 1
  • RGB-space reward models require near-complete denoising and VAE decoding, delaying evaluation, creating memory bottlenecks, and leaving early structure and motion stages without direct reward guidance.Related approaches still rely on fully denoised frames and costly decoding, while first-frame-only assessment sacrifices holistic video quality.Source: §1 · p. 2, Figure 2 · p. 2
  • Although video generation models offer process-level supervision, using them as train-time video reward models remains largely unexplored.The open challenge is to compress spatiotemporal features while retaining sensitivity to noise levels.Source: §1 · p. 2
  • The paper introduces PAVRM to score noisy latents at arbitrary timesteps and PRFL to optimize generation from randomly sampled timesteps without VAE decoding.PRFL distributes a single-gradient process reward across the denoising trajectory.Source: §1 · p. 2

2. Related Work

Prior reward-feedback methods face efficiency trade-offs, and video reward models have largely assessed near-complete outputs. The paper situates timestep-aware latent rewards as a way to address these gaps.

2.1. Visual Reward Feedback Learning

  • Image-focused reward feedback faces a depth-efficiency trade-off: backpropagating through many denoising steps is costly, while optimizing only final steps misses low-level objectives such as symmetry.Video adds multi-frame synthesis and gradient-enabled VAE-decoding costs; first-frame-only evaluation cannot capture comprehensive video quality.Source: §2.1 · pp. 2–3
  • Timestep-aware fine-tuning of VGM features reaches 85.46% accuracy, with peak performance at early timesteps (t=0.8).VGM features from any DiT layer achieve 78.8% accuracy, matching the VLM baseline, while VideoAlign-MQ scores fluctuate across timesteps.Source: Figure 3 · p. 3
  • The paper proposes timestep-wise latent reward modeling for video, enabling backpropagation at arbitrary timesteps while maintaining computational tractability.This contrasts with pixel-space rewards and DOLLAR [7], which lacks timestep-wise optimization capability.Source: §2.1 · p. 3

2.2. Reward Models for Video Generation

  • Video reward models mainly use outcome-based rewards on near-complete outputs, leaving early denoising stages without direct guidance.LPO [46] explored noise-aware latent rewards for images, while training-time use of video generation models remained unexplored.Source: §2.2 · p. 3

3. Preliminaries and Feasibility Analysis

The section introduces rectified-flow generation and reward-feedback learning, then tests whether video-generation-model features can assess motion quality across denoising timesteps. Its feasibility analysis reports improved accuracy with timestep-aware fine-tuning.

3. Preliminaries and Feasibility Analysis

  • Rectified flow interpolates between clean data x0 and noise x1, while a neural network predicts the velocity field vθ(xt, t) at each timestep.The transport direction is v = x1 − x0.Source: §3 · p. 3
  • ReFL samples a late denoising timestep and backpropagates a reward from the generated output, combining reward loss with SFT regularization.The reward is evaluated on the decoded output; D denotes the VAE decoder.Source: §3 · p. 3
  • Figure 4 presents a two-stage framework: PAVRM predicts rewards from noisy latents, then PRFL optimizes the VGM at randomly sampled timesteps.The figure also depicts the VGM and PAVRM architecture.Source: Figure 4 · p. 4
  • 85.46% accuracy came from full fine-tuning with random-timestep sampling, about a 6.6% absolute gain, particularly with earlier noisy states (t=0.8).The result supports using fine-tuned VGM features as timestep-robust motion-quality signals.Source: §3 · p. 4
  • VGM features from DiT layers L8 to L40 each achieve 78.8% accuracy, matching the VideoAlign baseline; VideoAlign scores fluctuate across timesteps.The analysis uses an MLP linear probe and identifies the first eight layers as sufficient for a more efficient reward model.Source: §3 · pp. 3–4

4. Method

PRFL combines a video-generation-model reward estimator operating on noisy latents with preference optimization at intermediate denoising steps. PAVRM learns to assess quality across noise levels, while PRFL uses its rewards without VAE decoding and alternates reward optimization with supervised fine-tuning.

4.1. Overview

  • PRFL trains a Process-Aware Video Reward Model on noisy latents, then optimizes the video generation model with its process-level rewards.The framework has separate reward-model training and generation-model optimization stages.Source: §4.1 · p. 4

4.2. Process-Aware Video Reward Model

  • PAVRM uses a frozen video-generation backbone’s intermediate spatiotemporal features and query attention to produce a quality reward.Query-based aggregation compresses variable-length features into a fixed-size representation, which an MLP maps to a reward score.Source: §4.2 · pp. 4–5
  • Randomly sampled training timesteps teach PAVRM to assess quality across the denoising trajectory, with final-output preference labels propagated to intermediate states.The paper motivates this training choice by the deterministic linear interpolation between noise and data in rectified flow.Source: §4.2 · p. 5

4.3. Process Reward Feedback Learning

  • PRFL optimizes PAVRM-predicted quality at arbitrary intermediate latent timesteps, avoiding VAE decoding and spreading reward guidance across generation.It performs gradient-free denoising rollouts before one gradient-enabled step; gradients then pass through that step and PAVRM without decoding.Source: §4.3 · p. 5
  • PRFL alternates reward optimization with supervised fine-tuning to maintain generation diversity while adapting to motion-quality preferences.The supervised fine-tuning dataset is curated, and the two objectives alternate at each iteration.Source: §4.3 · p. 5

5. Experiments

Across text-to-video and image-to-video settings, PRFL improves motion-related metrics while preserving high smoothness and consistency. Experiments also assess training efficiency, human preferences, timestep sampling, and PAVRM accuracy.

5.1. Experimental Setup

  • The dataset contains 24,000 real videos and corresponding generated videos; experiments compare against SFT, RWR, RGB ReFL, and reward-model baselines using VBench-series metrics.The setup also specifies Wan2.1-I2V-14B as the primary baseline and evaluations at 720P/480P.Source: §5.1 · pp. 5–6

5.2. Video Generation

  • At 720P, T2V dynamic degree improves by +56.00 and human anatomy by +12.16; at 480P, motion smoothness is 99.05 and subject consistency is 96.34.The reported 480P decreases in MS (-0.15) and SC (-1.00) are described as marginal relative to their baselines.Source: Table 1 · p. 6, §5.2 · p. 7
  • For I2V at 480P, dynamic degree improves by +30.00 on Inner Test Set and +40.65 on VBench-I2V, while other metrics improve or remain stable.The 720P results are described as showing consistent improvements across metrics.Source: Table 2 · p. 6, §5.2 · p. 7
  • In pairwise evaluation by 30 professionals across 2,250 comparisons, PRFL consistently outperforms SFT, RWR, and RGB ReFL; qualitative examples show fewer artifacts and coherent motion.The user study sampled 25 prompts, and the qualitative comparisons include dancing and multi-person guitar scenes.Source: §5.2 · pp. 7–8, Figure 6 · p. 7
  • Full-range timestep sampling achieves Avg 89.58, DD 68.00, and HA 94.73; early and middle sampling each reach DD 51.00, while late sampling reaches DD 44.00.Middle-stage sampling has stronger VBench performance than early-stage sampling (76.39 vs. 48.61).Source: Table 3 · p. 7, §5.2 · p. 8
  • PRFL processes all 81 frames within 67 GB peak VRAM and trains 1.42× to 1.49× faster than first-frame-only RGB-ReFL.RGB-ReFL runs out of memory on full 81-frame sequences and resorts to first-frame-only training.Source: Table 4 · p. 7, §5.2 · p. 8

5.3. Reward Model

  • Attention with query achieves 84.18% average accuracy for I2V at 720P, and PAVRM maintains over 83% average accuracy across evaluated tasks, resolutions, and timestep ranges.Simpler Max Pooling degrades substantially at later stages.Source: §5.3 · p. 8, Table 5 · p. 8

6. Conclusion

The paper introduces PAVRM and PRFL for process-level reward modeling and latent-space optimization of video generation. Experiments report motion-quality improvements, lower memory use, and at least 1.4× speedup; the current focus is motion quality.

6. Conclusion

  • PAVRM evaluates motion quality from noisy latent representations at arbitrary timesteps, while PRFL fine-tunes in latent space without VAE decoding.PAVRM uses query-based aggregation for variable-length videos.Source: §6 · pp. 8–9
  • Experiments report substantial motion-quality improvements, large memory reduction, and at least 1.4× training speedup versus pixel-space methods.Source: §6 · p. 9
  • The method focuses on motion quality; the authors identify multi-aspect evaluation of aesthetics and semantics as a possible extension.Future work could also explore hybrid reward frameworks, richer preference signals, controllable generation, and video editing.Source: §6 · p. 9

A.1. Overview.

The overview points readers to separate appendix sections for experimental settings and reward-model analysis.

A.1. Overview.

  • Detailed experimental settings are provided in Section A.2.Source: §A.1 · p. 12
  • Additional experimental analysis about the reward model is provided in Section A.3.Source: §A.1 · p. 12
  • The overview directs readers to one appendix section for settings and another for reward-model analysis.Source: §A.1 · p. 12

A.2.1. Analysis Experiments

The analysis tests video-generation-model features and reward-model behavior across denoising timesteps, including comparisons of feature aggregation and fine-tuning setups.

A.2.1. Analysis Experiments

  • The analysis uses I2V at 720P and the PAVRM training dataset, with a linear probe dimension matching the 5120-token dimension of Wan2.1-I2V-14B.Source: §A.2.1 · p. 12
  • VideoAlign reward scores fluctuate between early and late denoising timesteps, indicating poor generalization to high-noise regions.The analysis decodes intermediate noisy latents to RGB before scoring them with VideoAlign-MQ.Source: §A.2.1 · p. 12
  • With timestep fixed at t = 0.2, mean-pooled VGM features were analyzed across varying DiT block counts; VideoAlign MQ reward scores yielded 78.83% test accuracy after threshold selection against ground-truth labels.The score threshold was selected to maximize test-set accuracy.Source: §A.2.1 · p. 12
  • The comparisons distinguish fixed-timestep probe training, randomly sampled timestep probe training, and random-timestep full fine-tuning of DiT blocks plus the probe.The full fine-tuning setup excludes text/image encoders and the VAE.Source: §A.2.1 · p. 12

A.2.2. Open Source Test Set

The open-source evaluation uses established benchmarks to assess motion quality and subject consistency across text-to-video and image-to-video generation.

A.2.2. Open Source Test Set

  • The T2V test sets include 72 VBench subject-consistency prompts and 120 VBench2 human-anatomy prompts, while I2V uses 246 VBench-I2V subject prompts.The evaluation also targets motion quality defined by freedom from distortions, smooth motion, and compliance with physical laws.Source: §A.2.2 · p. 12

A.2.3. Inner Data Collection and Annotations

The appendix describes reward-data construction, baselines, and evaluation, then examines training-loss and model-depth choices and cross-generator transfer.

A.2.3. Inner Data Collection and Annotations

  • The final reward dataset contains 24,000 real/generated video pairs, with generated videos for reward-model training and real videos for supervised fine-tuning.It was constructed from 31,000 portrait videos; labels use physical-plausibility and subject-deformity ratings, retaining only videos rated qualified on both or unqualified on both dimensions. Evaluation labels were determined by majority vote among at least three annotators.Source: §A.2.3 · pp. 12–13

A.2.4. Baseline Settings

  • Baselines include zero-shot VideoAlign-MQ and VideoPhy-PC rewards, SFT, RWR, and first-frame RGB ReFL; post-training methods use approximately matched sample counts and shared training settings.RGB ReFL decodes only the first frame and uses PickScore.Source: §A.2.4 · pp. 13–14

A.2.5. Evaluation

  • Evaluation combines timestep-stratified reward accuracy, video-quality metrics, and a 100-question pairwise user study of overall quality.The metrics include motion smoothness, dynamic degree, subject consistency, human anatomy accuracy, and PAVRM’s qualified-sample ratio; generation uses 40 denoising steps at 720P or 480P.Source: §A.2.5 · pp. 14–15, Figure 7 · p. 15

A.3.1. The Influence of Training Loss

  • The average accuracy gap between BCE and BT is 0.2%; BT performs better at t > 0.6, while BCE is more precise at t ≤ 0.6 and avoids pair-construction overhead.The authors use BCE as the default objective.Source: Table 7 · p. 15, §A.3.1 · p. 16

A.3.2. The Influence of the Number of DiT Blocks

  • The 16-block model peaks at 85.51% accuracy; performance declines with additional blocks, and the full 40-block model performs worse than the 8-block baseline.The authors identify the first 8 or 16 blocks as the best efficiency–accuracy trade-off.Source: Table 8 · p. 15, §A.3.2 · p. 16

A.3.3. Cross-Model Generalization

  • PAVRM trained only on Wan2.1 transfers zero-shot to HunyuanVideo and Veo3, with stronger out-of-domain performance at high-noise timesteps than near-data states.The transfer tests focus on human structural deformities; Table 9 reports average accuracy for Wan2.1-based PAVRMs on 720P I2V.Source: §A.3.3 · p. 16, Table 9 · p. 16
Loading 2511.21541v3…