PaperlayerView exact revision on arXiv

Source-linked AI summary

When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO

Xinrui Lin, Fan Li, Shumin Wang, Dehong Kong, Yanyong Zhang

arXiv:2610.20089v1cs.CV

TL;DR

Pairwise feedback gives video-generation RL relative judgments, but GRPO needs scalar rollout advantages and win-rate aggregation discards graph structure; cycles are prevalent and persist during training. RAP-Walk anchors a reliability-guided graph correction in win-rate advantages, improving over win-rate Pairwise-GRPO across matched comparisons, transfer settings, and blind human evaluations without additional evaluator queries.

Abstract

from arXiv · show

Pairwise preference feedback has become a widely adopted supervision signal for reinforcement learning in video generation models, as pairwise comparisons provide a more reliable alternative than pointwise scalar ratings for evaluating the quality of generated videos. Specifically, we define a preference graph for each GRPO group under a pairwise evaluator, where nodes represent sampled rollouts and edges encode pairwise preferences. Auditing 6,976 on-policy comparison graphs, we find that cyclic preferences are prevalent and persist throughout training. Naive pairwise reward aggregation based on average win rates discards the underlying comparison structures and treats each rollout independently, which may introduce unreliable advantage estimates for certain rollouts and consequently affect reinforcement learning optimization. Therefore, we propose the \emph{Reliability-Anchored Preference Walk} (\ours{}), a graph-aware advantage aggregation method that strengthens the global preference signal from win rates with reliable relational evidence from cyclic preference graphs through a reliability-guided preference walk. Under matched evaluator, comparison protocol, group size, prompt schedule, and query budget, \ours{} requires no additional evaluator queries beyond the observed comparisons within each rollout group and improves over Pairwise-GRPO across controlled comparisons, transfer settings, and blind human evaluations.

Introduction

Pairwise feedback preserves relative judgments, but GRPO needs one scalar advantage per rollout, and ordinary win-rate aggregation loses opponent and cycle structure. RAP-Walk anchors its graph-based correction in win-rate advantages, using reliable edges and paths to recover relational evidence.

Introduction

  • Unlike pointwise rewards, pairwise feedback avoids calibrating absolute video-quality scores, but GRPO still requires a scalar advantage for each rollout.Video quality spans prompt fidelity, visual appearance, motion quality, and temporal consistency.Source: Introduction · p. 1
  • Average win rates can assign identical scalar advantages to structurally different graphs, discarding opponent relationships and cycles.Figure 1 illustrates a three-video cycle with identical row-wise win rates.Source: Introduction · p. 1, Figure 1 · p. 2
  • The audit finds cyclic groups rise from 79.7% early to 89.0% late in training, showing cycles persist in online video preference optimization.The audit covers 6,976 complete on-policy comparison graphs.Source: Introduction · p. 1
  • RAP-Walk preserves standardized win-rate advantages as an anchor and adds a controlled graph-based correction for relational evidence.Its propagation uses reliable edges and multi-hop paths, modulated by edge reliability and local cyclic inconsistency.Source: Introduction · pp. 1–2
  • RAP-Walk treats cycles as informative relational structure, while using graph evidence only to the extent supported by edge- and group-level reliability.The paper validates the method through matched comparisons, ablations, backbone transfer, and blind human evaluations.Source: Introduction · p. 2

Related Work

Prior video-generation RL methods generally reduce evaluator feedback to scalar rewards or advantages, while this work studies the mapping from prompt-local pairwise graphs to GRPO advantages. Existing preference approaches include win-count selection and within-group win-rate rewards, whereas richer graph and game-theoretic methods model cycles differently.

Reinforcement learning for video generation

  • Video-generation RL work studies exploration, trust regions, and trajectory-level alignment, but assumes evaluator feedback is already reduced to scalar rewards or advantages.This paper instead studies mapping a prompt-local pairwise graph to GRPO advantages.Source: Related Work · p. 2

Preference optimization and aggregation

  • RewardDance selects references by pairwise win counts, while Pref-GRPO uses within-group win rates and discards opponent relations, paths, and cycles.Other preference-alignment work includes DPO variants and scorer-driven GRPO for image editing.Source: Related Work · p. 2
  • Classical aggregation uses win counts, graph decompositions, or random walks, while GPM and NLHF/SPO model cycles with richer representations or game-theoretic objectives.RAP-Walk keeps the evaluator and standard GRPO objective fixed and focuses on graph-to-advantage mapping.Source: Related Work · p. 3

Preliminaries: Pairwise-GRPO

Pairwise-GRPO compares videos sampled for a prompt with a fixed evaluator and represents the judgments as a reciprocal soft-preference graph. It reduces each rollout’s row of preferences to a standardized win-rate advantage, thereby discarding opponent identities and higher-order structure.

Preliminaries: Pairwise-GRPO

  • For each prompt, the policy samples a group of G videos, and the fixed evaluator compares every unordered pair in both presentation orders.The aligned soft preference predictions are averaged into a reciprocal matrix W.Source: Preliminaries: Pairwise-GRPO · p. 3
  • W_ij measures evaluator support for video i over j, with reciprocal entries summing to one and diagonal entries set to zero.The matrix is interpreted as a weighted directed comparison graph.Source: Preliminaries: Pairwise-GRPO · p. 3
  • Pairwise-GRPO averages each row’s off-diagonal preferences and standardizes the win rates within the group to produce rollout advantages.This reduction uses direct comparisons through row sums but discards opponent identities, paths, and cycles.Source: Preliminaries: Pairwise-GRPO · p. 3

Reliability-Anchored Preference Walk

RAP-Walk keeps standardized win-rate advantages as its optimization anchor while adding graph-derived preference credit as a reliability-controlled correction. Its walk uses opponent strength and multi-hop paths, with ambiguous or cyclically inconsistent evidence contributing less.

Reliability-Anchored Preference Walk

Reliability-Aware Graph Proposal

  • The transition moves preference mass from a video toward videos more likely to defeat it, allowing repeated walks to capture opponent strength and multi-hop paths.The destination-node prior uses standardized win-rate advantage, but does not fix the stationary ranking.Source: Reliability-Aware Graph Proposal · pp. 3–4
  • Edge clarity discounts ambiguous comparisons, while triangle-based cycle inconsistency identifies local preference structure beyond a single scalar ranking.Clarity is low near ties; local inconsistency reduces propagation confidence without removing cycles.Source: Reliability-Aware Graph Proposal · pp. 3–4

Graph Proposal from Stationary Preference Credit

  • Starting from uniform mass, the walk’s stationary credit reflects which opponents a video defeats and whether reliable multi-step paths lead to it.An entropy floor tempers an overly concentrated stationary distribution; graph credit is centered relative to uniform support and is not itself a rollout reward.Source: Graph Proposal from Stationary… · p. 4

Reliability-Anchored Advantage

  • Group-level graph reliability controls the correction weight: near-tie-heavy groups receive a small weight, while clearer comparisons can increase it up to the trust-interval bound.The corrected advantage interpolates between standardized win-rate advantage and graph proposal.Source: Reliability-Anchored Advantage · p. 4

Experiments

The experiments audit persistent cyclic preferences and evaluate RAP-Walk under matched training conditions, across evaluators and generator backbones, with ablations, blind human judgments, and qualitative comparisons. Results favor using graph evidence as a reliability-controlled correction to win-rate advantages.

Experimental Setup

  • All methods share 10,000 training prompts, 1,000 disjoint evaluation prompts, matched initialization and training budgets, and 200 training steps; evaluators sample 10 frames per video.The evaluation reports six VBench dimensions and VideoReward’s VQ, MQ, and TA scores; dynamic degree measures motion magnitude rather than perceptual motion quality.Source: Experimental Setup · p. 5

Preference Cycles Identification

  • Across 6,976 on-policy graphs, the any-cycle rate at zero margin rises from 79.7% early in training to 89.0% late, while clarity falls from 0.261 to 0.132 as near ties rise from 52.6% to 67.3%.At margin 0.05, cycle rates are 24.8/18.0/19.3% across the three stages; fixed-graph replay shifts the lowest-Li third by +0.616 and the highest by −0.544 when cycle modulation is toggled.Source: Preference Cycles Identification · p. 5, Table 1 · p. 5

Main Results of RAP-Walk

  • RAP-Walk improves eight of nine metrics over win rate with both evaluators, with only marginal smoothness regression; across two additional backbones it again improves eight of nine metrics, with one regression per backbone.On Wan2.1-14B, it gains 10.00 dynamic-degree points and 0.238 TA; on Wan2.2-TI2V-5B, it gains 1.43 aesthetic points and 0.300/0.079/0.135 VQ/MQ/TA.Source: Main Results of RAP-Walk · p. 5, Table 2 · p. 6, Table 3 · p. 6

Ablation Study

  • Graph proposal alone ranks first or second on all five non-dynamic VBench dimensions but trails the full model by 12.60 dynamic-degree points and 0.262/0.137/0.243 VQ/MQ/TA.Mean-matched fixed reliability gains 3.40 dynamic-degree points but lowers every non-dynamic VBench and VideoReward metric, supporting group-adaptive correction strength.Source: Ablation Study · pp. 5–6, Table 4 · p. 6
  • Removing the win-rate prior lowers dynamic degree by 8.60 points and all three VideoReward metrics; removing cycle modulation lowers dynamic degree by 9.20 points and all three VideoReward metrics.Removing edge clarity lowers aesthetics and imaging by 1.76 and 3.33 points; removing entropy calibration trades +5.80 dynamic degree and +0.067 TA for losses on all five non-dynamic VBench dimensions plus VQ and MQ.Source: Ablation Study · pp. 5–6, Table 4 · p. 6

User Study

  • Blind raters preferred RAP-Walk over Pairwise/Pointwise GRPO at 64.6%/78.2% for visual quality, 49.6%/59.0% for motion quality, and 53.4%/64.2% for text alignment.Motion was the closest comparison, but its win-loss margin over win-rate Pairwise-GRPO remained positive at 13.4 points.Source: User Study · p. 6, Table 5 · p. 6

Visual Comparison

  • In eight prompt comparisons, RAP-Walk preserves object identity and spatial layout across frames while maintaining coherent interactions and motion.The baselines show temporal artifacts, including inconsistent geometry and unstable object shapes.Source: Visual Comparison · p. 7, Figure 3 · p. 7

Conclusion

RAP-Walk improves over win-rate Pairwise-GRPO without additional evaluator queries, while preserving win-rate advantages and recovering relational evidence from preference graphs.

Conclusion

  • RAP-Walk improves over win-rate Pairwise-GRPO across matched comparisons, evaluators, backbones, ablations, and blind human evaluations without additional evaluator queries.It uses only comparisons already observed within each rollout group.Source: Conclusion · p. 7
  • RAP-Walk preserves win-rate advantages as anchors while recovering relational evidence through graph correction.Source: Conclusion · p. 7
  • An audit of 6,976 on-policy comparison graphs found cyclic preference structures emerging during training, indicating that opponent relationships and comparison paths contain information beyond win rates.Source: Conclusion · p. 7
  • The findings highlight the importance of preserving relational structure when transforming pairwise feedback into optimization signals.Source: Conclusion · p. 7

Related papers

Loading 2610.20089v1…