PaperlayerView exact revision on arXiv

Source-linked AI summary

KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration

Ruicheng Zhang, Kaixi Cong, Jun Zhou, Zhizhou Zhong, Zunnan Xu, Shuiyang Mao, Wei Liu, Xiu Li

arXiv:2605.14278v1cs.CV

TL;DR

Streaming autoregressive video generators remain difficult to align with human preferences, especially for long-horizon semantic coherence under deterministic ODE dynamics. KVPO addresses this with KV-cache semantic exploration and a velocity-field surrogate policy, achieving consistent gains in visual quality, motion quality, and text-video alignment across short- and long-video settings.

  • Problem

    Aligning streaming autoregressive video generators with human preferences remains challenging because preference-relevant quality includes long-horizon coherence, subject consistency, and semantic progression beyond frame fidelity.

  • Method

    KVPO combines causal-semantic exploration through historical KV-cache routing with a Trajectory Velocity Energy surrogate policy in flow-matching velocity-field space.

  • Results

    KVPO consistently improves visual quality, motion quality, and text-video alignment across single-prompt short-video and multi-prompt long-video settings, including up to 28.4% improvement on LongLive VQ.

  • Takeaways & Limitations

    The results support KVPO as an ODE-native approach for preference alignment of distilled autoregressive video generators across short- and long-horizon settings.

  • Takeaways & Limitations

    KVPO is designed for autoregressive generators with KV-cache memory, and its replay-state caching adds computational and memory costs that grow with branches and replay horizon.

Abstract

from arXiv · show

Aligning streaming autoregressive (AR) video generators with human preferences is challenging. Existing reinforcement learning methods predominantly rely on noise-based exploration and SDE-based surrogate policies that are mismatched to the deterministic ODE dynamics of distilled AR models, and tend to perturb low-level appearance rather than the high-level semantic storyline progression critical for long-horizon coherence. To address these limitations, we present KVPO, an ODE-native online Group Relative Policy Optimization (GRPO) framework for aligning streaming video generators. For diversity exploration, KVPO introduces a causal-semantic exploration paradigm that relocates the source of variation from stochastic noise to the historical KV cache. By stochastically routing historical KV entries, it constructs semantically diverse generation branches that remain strictly on the data manifold. For policy modeling, KVPO introduces a velocity-field surrogate policy based on Trajectory Velocity Energy (TVE), which quantifies branch likelihood in flow-matching velocity space and yields a reward-weighted contrastive objective fully consistent with the native ODE formulation. Experiments on multiple distilled AR video generators demonstrate consistent gains in visual quality, motion quality, and text-video alignment across both single-prompt short-video and multi-prompt long-video settings.

1 Introduction

Streaming autoregressive video generators require low-latency, long-horizon synthesis under causal dependencies, but existing alignment methods lack effective exploration or use mismatched stochastic surrogates. KVPO addresses these limitations with ODE-native causal-semantic exploration through historical KV-cache routing and a TVE-based velocity-field surrogate policy.

  • Motivation: Streaming AR video generation must combine high fidelity with low latency, causal temporal dependence, and long-horizon synthesis.Few-step autoregressive distillation enables efficient streaming inference but does not remove these alignment and deployment demands.
  • Limitations: Existing alignment approaches either lack active exploration of diverse candidate behaviors or convert deterministic ODE sampling into noise-driven SDE exploration.Reward-weighted distillation upweights high-reward trajectories without actively exploring alternatives, while SDE conversion injects noise into the process.
  • Limitations: Latent Euclidean-distance surrogate policies under pure ODE generation implicitly assume uniform geometry, limiting ODE-based policy optimization for AR video.NeighborGRPO introduced an implicit contrastive view of GRPO, and AR-CoPO extended it to AR video generation.
  • KVPO: KVPO performs ODE-native online GRPO with causal-semantic exploration by routing historical KV-cache entries instead of injecting stochastic noise.This relocates diversity generation to the historical KV cache and avoids the off-manifold distortions associated with unstructured noise injection.
  • Results: Experiments across multiple distilled AR video generators show consistent human-preference alignment gains in single-prompt short-video and multi-prompt long-video settings.The reported evaluation covers both short- and long-horizon streaming scenarios.
  • KVPO: KVPO models the surrogate policy in flow-matching velocity-field space using Trajectory Velocity Energy, producing a reward-weighted contrastive objective aligned with native ODE dynamics.The formulation avoids external geometric distances and SDE transition kernels.

2 Related Work

Related work establishes efficient streaming AR video generation and reward-based post-training alignment, while noting that existing visual and AR-specific methods leave exploration challenges unresolved. These approaches include ODE-to-SDE reformulations, reward-weighted distillation, and endpoint-based perturbation strategies.

  • Autoregressive Video Generation: AR video models generate frames causally and streamingly, with distillation compressing diffusion into few-step variants while preserving visual quality.These models exploit causal attention, dynamic KV caching, and explicit memory architectures.
  • Post-Training Alignment: Post-training alignment commonly treats sampling as policy rollout and optimizes reward-induced output distributions; VideoAlign, Flow-GRPO, and DanceGRPO apply reward or GRPO-based visual alignment.Flow-GRPO and DanceGRPO reformulate ODEs as SDEs for GRPO-style optimization.
  • AR Video Alignment: AR-specific methods use reward-weighted distillation or inference-endpoint contrastive reinforcement learning, but their exploration lacks active or structured semantic branching.Reward Forcing lacks active exploration, while Astrolabe confines exploration to noise-endpoint perturbation.

3 Methodology

KVPO aligns deterministic streaming autoregressive video generators through causal-semantic exploration in historical KV caches and replay under unperturbed deployment-time context. It models branch likelihood with Trajectory Velocity Energy, converts energies into a Gibbs-form surrogate policy, and optimizes reward-preferred ODE trajectories with PPO-style updates.

  • Streaming AR and ODE Generation: Streaming AR generation proceeds block by block, conditioning each block on the text prompt and historical video context through a learned flow-matching velocity field.The model follows a probability-flow ODE from noise to clean samples using discrete solver timesteps.
  • Causal-Semantic Exploration: KV routing creates semantically diverse candidate branches by perturbing branch-specific historical cache slots within a contiguous exploration window, while sharing preceding context and restoring standard caching afterward.Branches originate at a randomly sampled pivot block under distinct CHR refill decisions; generation before and beyond the window uses unperturbed or standard local context.
  • Rollout and Replay: During replay, cached intermediate states are evaluated under the restored unperturbed context, requiring one forward pass per step and no specialized solver.Gradient tracking is enabled only for solver steps inside the perturbed window, while replayed velocities support surrogate policy modeling.
  • Trajectory Velocity Energy: Trajectory Velocity Energy measures branch likelihood in flow-matching velocity space by comparing replayed model velocities with cached rollout velocities under deployment-time semantics.Lower TVE indicates a stronger generative tendency toward the corresponding branch.
  • Surrogate Policy and Policy Ratio: KVPO converts TVE values into a normalized Gibbs surrogate policy, assigning higher probability to branches with lower energy while remaining grounded in replay-time compatibility and native flow-matching dynamics.The policy uses relative TVE scores across branches and defines a PPO importance ratio between current and previous policies.
  • Preference Optimization: Asymmetric clipping uses ϵlow = 0.1 and ϵhigh = 0.2 to promote high-reward branches more aggressively while conservatively suppressing low-reward branches and avoiding optimization collapse.The resulting TVE policy optimization reduces to a reward-weighted contrastive flow-matching objective that aligns ODE dynamics with superior trajectories and suppresses inferior ones.

4 Experiments

KVPO is evaluated on LongLive and MemFlow across single-prompt short-video and multi-prompt long-video generation, using reward and VBench metrics. It consistently improves quantitative and perceptual performance, while ablations show that its TVE-based surrogate policy is critical.

  • Implementation Details: KVPO is evaluated on LongLive and MemFlow for single-prompt short-video and multi-prompt long-video generation, with comparisons against Astrolabe.Training prompts come from VidProM and are refined using Qwen3.
  • Evaluation Metrics: The evaluation reports three reward metrics plus VBench Quality, Semantic, Consistency Score, and CLIP Score across both generation settings.The complementary VBench metrics provide a comprehensive assessment of model performance.
  • Quantitative Results: 15.2%, 5.0%, and 200.0% improve LongLive’s VQ, MQ, and TA in short-video generation, while MemFlow improves by 9.1%, 2.7%, and 50.0%.In long-video generation, LongLive improves by 28.4%, 6.4%, and 26.3%, while MemFlow improves by 10.5%, 3.6%, and 15.0% on VQ, MQ, and TA.
  • Qualitative and Human Evaluation: KVPO improves prompt grounding, object boundaries, motion continuity, subject identity preservation, and resistance to semantic shifts and structural drift.A human study with 32 participants found KVPO secured a clear majority preference over the baseline and Astrolabe across VQ, MQ, and TA.
  • Ablation Study: Replacing TVE with a geometric latent-space ℓ2 surrogate substantially degrades performance across all metrics, highlighting velocity-field-space policy modeling’s importance.The ablation is conducted on LongLive in the multi-prompt long-video setting.

5 Conclusion · Appendix

The paper studies preference alignment for streaming autoregressive video generators under deterministic ODE dynamics. It introduces KVPO, combining causal-semantic exploration through CHR with a TVE-based velocity-field surrogate policy.

  • 5 Conclusion: KVPO targets preference alignment for streaming autoregressive video generators in the deterministic ODE regime.The work addresses alignment under ODE-based generation.
  • 5 Conclusion: KVPO addresses the mismatch between noise-driven reinforcement learning methods and ODE-based generation.This mismatch motivates the proposed framework.
  • 5 Conclusion: KVPO combines Causal History Routing with a velocity-field surrogate policy based on Trajectory Velocity Energy.These are the framework’s two stated components.
  • 5 Conclusion: Causal-semantic exploration is implemented through Causal History Routing.CHR is identified as the exploration mechanism in KVPO.
  • 5 Conclusion: Causal History Routing redirects exploration to the historical key-value cache.The passage identifies the historical key-value cache as the redirected exploration source.
  • 5 Conclusion: Redirecting exploration through the historical key-value cache induces semantically meaningful generation behavior.The supplied passage states that cache-based routing induces semantically meaningful outcomes, but its sentence is truncated.

A KVPO Training Pipeline … D Rethinking ODE-based Policy Optimization

KVPO trains autoregressive video generators through coupled semantic exploration, replay-based surrogate construction, and policy optimization. It explores by routing historical KV cache entries while preserving deterministic ODE sampling and contrasts this approach with noise-driven ODE policy optimization.

  • A KVPO Training Pipeline: Each KVPO iteration performs semantic exploration, replay-based surrogate policy construction, and policy optimization, using reward-model evaluations to compute group advantages.Branches share the initial latent while differing through local KV-cache composition.
  • A KVPO Training Pipeline: The pipeline restores the default local cache for replay, evaluates all branches with the reward model, and applies a PPO-KL update to optimize the generator.The supplied procedure describes replay states, group-normalized advantages, a Gibbs surrogate, and a KL-regularized total loss.
  • B Why KV Exploration?: Historical KV cache exploration influences future generation through the native causal pathway and produces semantically diverse branches relevant to temporal coherence, subject consistency, and progression.The intervention varies conditioning state rather than adding external stochastic perturbations to the latent trajectory.
  • C Conditional Marginal Preservation: CHR preserves each fixed-context conditional marginal because generation still solves the original deterministic ODE from an unchanged Gaussian initial latent.CHR changes conditioning context and semantic trajectory without modifying the underlying noise-to-sample transport.
  • C Conditional Marginal Preservation: Consequently, every CHR branch remains an exact sample from its corresponding conditional model distribution rather than an off-manifold perturbation.The branch variation comes from changing conditioning context while leaving the initial latent unchanged.
  • D Rethinking ODE-based Policy Optimization: Purely ODE-based methods such as NeighborGRPO explore by perturbing the initial latent and approximate surrogate policies using Euclidean distances between generated latents.KVPO’s comparison reframes exploration around the limitations of noise-driven neighborhoods for autoregressive video generation.

D.1 Diversity Exploration: Disentangling Policy from Noise Variance · D.2 Surrogate Policy Modeling: Overcoming Geometric Distortion

KVPO disentangles exploration diversity from noise variance by fixing the initial latent and varying historical KV context. It also replaces isotropic Euclidean surrogate modeling with velocity-field replay and TVE to align policy optimization with nonlinear ODE geometry.

  • D.1 Diversity Exploration: Disentangling Policy from Noise Variance: Noise-driven exploration perturbs the initial latent with Gaussian noise, making candidate variation dependent on both policy response and sampled noise magnitude.The generation process is represented as x_0 = Φ(x_T; θ, K), with perturbed initial latent x_T + σδ_i and δ_i ∼ N(0, I).
  • D.1 Diversity Exploration: Disentangling Policy from Noise Variance: Euclidean distance between generated samples cannot reliably indicate surrogate-policy likelihood because larger distances may simply reflect larger noise magnitudes.This confounding corrupts the true policy-gradient signal.
  • D.1 Diversity Exploration: Disentangling Policy from Noise Variance: KVPO shifts exploration from noise perturbation to causal-semantic routing over the historical KV cache.CHR creates branch-specific local memory while preserving the shared initial latent.
  • D.1 Diversity Exploration: Disentangling Policy from Noise Variance: With the initial latent fixed, structural and semantic trajectory differences are attributable to deterministic responses to distinct historical contexts.The resulting exploration reflects surrogate preference ordering induced by historical KV-cache changes rather than noise magnitude.
  • D.2 Surrogate Policy Modeling: Overcoming Geometric Distortion: Unweighted Euclidean surrogate modeling assumes every feature dimension contributes equally to likelihood evaluation.This imposes an isotropic assumption despite the nonlinear probability-flow ODE.
  • D.2 Surrogate Policy Modeling: Overcoming Geometric Distortion: Cascaded nonlinear transformations across AR blocks and diffusion steps produce an anisotropic true distribution governed by the flow Jacobian.The resulting covariance is represented by J_ΦJ_Φ^⊤.
  • D.2 Surrogate Policy Modeling: Overcoming Geometric Distortion: Euclidean distance ignores Riemannian curvature and can unjustly penalize valid semantic variations along the flow’s principal axes.This geometric distortion follows from the nonlinear mapping, where J_Φ ≠ I.
  • D.2 Surrogate Policy Modeling: Overcoming Geometric Distortion: KVPO constructs its surrogate policy through trajectory replay in velocity-field space, using TVE to evaluate candidates under unperturbed deployment-time context.TVE accumulates velocity-field MSE along replayed ODE trajectories, mirroring the native flow-matching objective.

E Ablation on the KL Penalty Weight

The KL penalty anchors the surrogate policy to the frozen reference and is essential for stable online preference optimization. Ablations show that no penalty causes collapse, while intermediate β values provide the strongest trade-off and β = 5 is most balanced.

  • The KL coefficient β anchors the learned surrogate policy to the frozen reference, supporting stable online preference optimization.This regularization term is described as essential for stable learning.
  • At β = 0, aggressive drift toward noisy high-reward branches causes poorly regularized updates, replay-distribution divergence, and training collapse.All metrics fall substantially below the base model in both settings, indicating degraded alignment quality and generation stability.
  • β = 1 recovers most gains, while β = 3–5 yields the strongest overall trade-off and β = 5 achieves the most balanced performance.The default β = 5 balances reward metrics with auxiliary VBench metrics; larger values produce gradually diminishing gains.

F Limitations

KVPO’s causal-semantic exploration is limited to autoregressive video generators that use KV-cache-based memory. Extending it to models without KV caches or with substantially different memory mechanisms may require additional work.

  • F Limitations: KVPO’s causal-semantic exploration is designed for autoregressive generators with KV-cache-based memory.The paper notes that this design is natural for many mainstream autoregressive video models.
  • F Limitations: Extending the exploration mechanism to models without KV caches may require additional work.The limitation concerns models whose memory mechanism differs from the KV-cache design.
  • F Limitations: Mamba-style state-space models are an example of architectures with substantially different memory mechanisms.The passage identifies such models as a representative extension case.

G Key KVPO Training Hyperparameters

This section summarizes KVPO’s training hyperparameters across model setup, optimization, replay-based reinforcement learning, and streaming rollout. It also reports LongLive training’s resource usage, including approximately 130 GB peak memory and 960 seconds per step.

  • Resource Usage: Approximately 130 GB peak GPU memory usage was recorded during LongLive training.Figure 8 reports GPU memory usage and compute utilization during training.
  • Resource Usage: Approximately 960 seconds was required per LongLive training step.The reported runtime accompanies the peak memory measurement in the resource-usage summary.
  • Training Hyperparameters: Table 4 organizes KVPO training hyperparameters into model setup, optimization, replay-based reinforcement learning, and streaming rollout configurations.The supplied passage identifies the configuration categories but does not provide their individual parameter values.

H More Qualitative Results

Additional qualitative comparisons across LongLive and MemFlow show that KVPO produces stronger semantic consistency and more stable narrative progression across diverse prompts and scene transitions.

  • Qualitative comparisons: KVPO consistently demonstrates stronger semantic consistency and more stable narrative progression than the compared methods.The comparisons evaluate long-horizon temporal coherence, subject identity preservation, motion plausibility, and text-video alignment.
Loading 2605.14278v1…