Source-linked AI summary
Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models
Songchun Zhang, Zeyue Xue, Siming Fu, Jie Huang, Xianghao Kong, Y Ma, Haoyang Huang, Nan Duan, Anyi Rao
TL;DR
Distilled autoregressive video models stream efficiently but are not directly optimized for human visual preferences, while existing RL methods impose costly re-distillation or reverse-process trajectory storage. Astrolabe uses forward-process online RL with streaming local-window training and reward-aware regularization, and experiments report consistent quality gains across distilled architectures and benchmarks while preserving inference speed.
Problem
Distilled autoregressive video models lack optimization for human visual preferences, while existing alignment methods require expensive re-distillation or solver-coupled reverse-process trajectory storage.
Method
Astrolabe combines trajectory-free forward-process RL, rolling-KV-cache streaming training with local-window updates, multi-reward optimization, and uncertainty-aware selective regularization.
Results
Astrolabe consistently improves generation quality across multiple distilled autoregressive models and benchmarks while maintaining the exact inference speed of original checkpoints.
Takeaways & Limitations
The framework provides a memory-efficient and general alignment approach for distilled streaming video architectures, including long-video training with constant peak memory.
Takeaways & Limitations
Astrolabe depends on reward-model accuracy, and current models have limited ability to assess complex physics, long-horizon causality, and multi-entity interactions in minute-scale videos.
Abstract
from arXiv · showhide
Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences. Existing reinforcement learning (RL) frameworks are not naturally suited to these architectures, typically requiring either expensive re-distillation or solver-coupled reverse-process optimization that introduces considerable memory and computational overhead. We present Astrolabe, an efficient online RL framework tailored for distilled AR models. To overcome existing bottlenecks, we introduce a forward-process RL formulation based on negative-aware fine-tuning. By contrasting positive and negative samples directly at inference endpoints, this approach establishes an implicit policy improvement direction without requiring reverse-process unrolling. To scale this alignment to long videos, we propose a streaming training scheme that generates sequences progressively via a rolling KV-cache, applying RL updates exclusively to local clip windows while conditioning on prior context to ensure long-range coherence. Finally, to mitigate reward hacking, we integrate a multi-reward objective stabilized by uncertainty-aware selective regularization and dynamic reference updates. Extensive experiments demonstrate that our method consistently enhances generation quality across multiple distilled AR video models, serving as a robust and scalable alignment solution.
1 Introduction
Distilled autoregressive video models enable efficient streaming but remain poorly aligned with human visual preferences. Astrolabe addresses the limitations of existing distillation- and reverse-process RL approaches with an efficient online RL framework for these models.
- Motivation: Distilled autoregressive models support real-time and potentially long-video generation through efficient autoregressive inference, but distillation does not optimize for human preference.They inherit efficiency from distribution matching while frequently producing artifacts.
- Limitations of Existing Alignment: Reward-guided distillation shifts outputs toward high-reward regions but lacks active exploration and does not penalize suboptimal generations.
- Limitations of Existing Alignment: Reverse-process RL requires solver-specific log-probability estimation and intermediate trajectory storage, adding memory and computational overhead.These costs erode the efficiency advantages of streaming models.
- Astrolabe: Astrolabe contrasts positive and negative generations using clean inference endpoints, avoiding solver-specific unrolling and full trajectory storage.
- Astrolabe: Streaming training applies RL updates to short segments conditioned on prior context, while multi-reward optimization and dynamic regularization mitigate reward hacking.
- Evidence: Experiments across distilled autoregressive models and benchmarks show consistent generation-quality improvements and generality across settings.
2 Related Work
Related work moves from full-sequence diffusion toward autoregressive streaming models, while addressing exposure bias and alignment costs. Existing RL methods either use alternative training paradigms or remain coupled to reverse-process trajectory estimation.
- Video Generation: Diffusion Transformers generate high-quality videos by jointly denoising frames, but simultaneous full-sequence processing causes latency and prevents real-time interaction.
- Autoregressive Video Models: Autoregressive approaches produce frames sequentially for streaming generation, although early Teacher Forcing methods suffer severe long-video error accumulation.
- Autoregressive Video Models: Diffusion Forcing and CausVid address train-test mismatch through arbitrary noise conditioning or block-causal distillation, respectively.
- Reinforcement Learning Alignment: DiffusionDPO uses off-policy pairs, whereas DanceGRPO and Flow-GRPO estimate reverse-trajectory log-probabilities with solver coupling and full trajectory storage.
3 Methodology
Astrolabe combines streaming rollout, forward-process RL, long-video local-window training, and reward-aware regularization for efficient alignment of distilled autoregressive video models. Its design shares context across candidates, avoids trajectory storage, and selectively constrains uncertain generations.
- Framework Overview: Astrolabe combines streaming rollout with forward-process RL to align distilled autoregressive video models with human preferences during post-training.
- Preliminaries: The autoregressive model factorizes video generation frame by frame and conditions velocity prediction on text and preceding-frame KV-cache context.
- Preliminaries: Self-Forcing simulates inference dynamics with autoregressive rollouts, aligning velocity predictions on self-generated trajectories with teacher scores.
- Forward-Process RL: Forward-process RL constructs noisy versions of clean samples and defines implicit positive and negative policies by interpolating current and old velocity predictors.The interpolation strength is controlled by β, and the policy loss contrasts both policies against the target forward velocity.
- Streaming Rollout: Group-wise streaming rollout shares a frozen context prefix across candidates, restricting generation overhead to local clips and reducing rollout time and memory bottlenecks.
- Clip-Level Optimization: Clip-level RL computes group-centered reward advantages, normalizes them, and optimizes the implicit policy loss using shared KV-cache context.The method discards DiffusionNFT’s adaptive loss weighting because it causes gradient explosion under large discretization gaps.
- Long-Video Training: Streaming Long Tuning detaches historical KV-cache gradients while optimizing only an active local window, simulating long-sequence inference with lower memory use.
- Reward Design: The multi-reward objective combines Visual Quality, Motion Quality, and Text-Video Alignment, with visual quality averaged over the top 30% of frames.
4 Experiments
Astrolabe is evaluated across short- and long-video settings, multiple distilled autoregressive models, and ablations of its efficiency, reward, and stabilization components. Results report consistent quality and preference improvements, while local-window streaming reduces memory use and multi-reward regularization avoids single-objective reward hacking.
- Short-Video Single-Prompt Generation: Astrolabe outperforms base models on HPSv3 and Motion Quality while maintaining the original inference speed across evaluated distilled AR architectures.Qualitative results additionally show sharper textures and superior motion coherence.
- Short-Video Single-Prompt Generation: Astrolabe improves motion quality, semantic alignment, and overall generation quality on VBench when integrated into existing distilled models.
- Long-Video Single-Prompt Generation: Astrolabe consistently improves long-video baselines across video-quality and human-preference metrics on VBench-Long.
- Long-Video Single-Prompt Generation: Astrolabe produces sharper textures, more stable sequences, and improved visual details over extended single-prompt durations.
- Long-Video Multi-Prompt Generation: Across three models, Astrolabe consistently improves both HPSv3 and MQ scores, including complex multi-prompt narrative transitions.Multi-prompt results also report improved frame-level aesthetics and temporal consistency.
- Ablation Studies: Clip-level group-wise sampling reduces memory consumption by ≈2× versus clip-level full backpropagation while improving HPSv3 and MQ.Sharing historical context across candidate clips provides the reported efficiency gain.
- Ablation Studies: Multi-reward optimization prevents single-objective overfitting, while selective KL regularization with EMA reference updates balances exploration and stable convergence.Uniform KL over-constrains learning, whereas omitting it causes instability and early MQ plateaus.
- Ablation Studies: Setting β = 1 yields higher overall visual and motion quality than β = 0.1 and is adopted as the default for generation stability.
5 Conclusion
Astrolabe presents a memory-efficient online RL framework for aligning distilled autoregressive video models with human preferences. Its forward-process optimization, streaming local-window training, and uncertainty-aware reward stabilization are validated across multiple architectures and benchmarks.
- Astrolabe combines forward-process RL, local-window streaming training with constant peak memory, and multi-reward stabilization for distilled autoregressive video alignment.
- Experiments across multiple distilled streaming architectures and benchmarks validate the effectiveness and generality of the approach.
A Hyperparameters
The appendix provides comprehensive hyperparameter configurations for Astrolabe training.
- Table S1 details configurations spanning model architecture, optimization, diffusion, reinforcement learning, and streaming rollout.
B Theoretical Proofs and Analysis
The paper extends DiffusionNFT’s theoretical foundation to autoregressive streaming generation and few-step distilled models, establishing two supporting results for Astrolabe.
- The analysis extends DiffusionNFT from global, nonautoregressive continuous-time diffusion to autoregressive streaming generation and few-step distilled models.
- It proves the optimality of local advantage guidance and establishes a reward lower bound using a selective KL penalty.
B.1 Conditional Improvement via Advantage Guidance
Theorem 1 shows that contrasting advantage-weighted positive and negative distributions yields a conditional local policy improvement under a typical parameter regime, without full-trajectory gradients.
- Positive and negative distributions are defined by reweighting the old policy with normalized advantage and its complement.
- The local policy loss combines squared deviations from positive and negative conditional flow-matching targets, with β controlling negative repulsion strength.
- The optimal update shifts toward the advantage-weighted policy when α(1 + β) < 1, aligning with v+ − v_old.
- The baseline velocity decomposes as a mixture of positive and negative velocities weighted by the posterior positive probability.
- Optimizing the local advantage strictly improves πθ(x_n | C_n) without requiring gradients across the full trajectory.
B.2 Performance Lower Bound with Selective Trust Region
Theorem 2 lower-bounds true reward using proxy reward accuracy in safe and high-uncertainty regions, while a selective KL penalty limits distribution shift only where uncertainty is high.
- The analysis partitions outcomes into a safe region and a high-uncertainty region where proxy-reward errors may differ substantially.
- The selective velocity MSE regularizes the policy toward a reference model only within the high-uncertainty region.
- The true reward is lower-bounded by expected proxy reward minus safe-region error and an uncertainty-weighted risk and KL term.
- The risk-region bound includes a total-variation distribution-shift penalty, which is connected to KL divergence through Pinsker’s inequality.
- Selective regularization permits exploration in safe regions and activates trust-region control when the policy enters the high-uncertainty region.
- The training procedure combines multi-reward evaluation, rank-disagreement uncertainty, selective KL regularization, rolling KV-cache training, and local-window updates.
D More Qualitative Results
Additional qualitative comparisons report improved quality and temporal coherence for Astrolabe outputs, including results on 30-second videos and the Krea 14B baseline.
- Additional figures compare 30-second single-prompt generations against baseline distilled models and competing methods.
- The reported qualitative comparisons describe Astrolabe outputs as higher-quality and more temporally coherent than the Krea 14B baseline.
- Figure S1 compares Krea 14B outputs in odd rows with outputs from Krea 14B plus Astrolabe in even rows.
E Discussion
Astrolabe’s discussion combines a selective-KL NFT training procedure with streaming, window-focused optimization and identifies reward-model and base-architecture limitations. Supplementary figures provide qualitative comparisons for 30-second single-prompt videos.
- Limitations: Astrolabe depends on reward-model accuracy, especially for complex physics, long-horizon causality, multi-entity interactions, and subtle temporal hallucinations in minute-scale videos.The discussion states that current open-source video evaluators are stronger for short-term aesthetics and text-video alignment than for these long-form properties.
- Limitations: Reinforcement learning cannot arbitrarily create capabilities entirely absent from the distilled base model.The paper frames this as an inherent bottleneck of applying post-training RL to the architecture.
- Training procedure: Astrolabe’s training procedure uses multi-reward evaluation, rank-disagreement uncertainty, forward-process optimization, selective KL regularization, and adaptive reference updates.The algorithm initializes policy, behavior, and KL-reference models; identifies high-uncertainty samples; optimizes policy loss; and updates the reference when KL or iteration thresholds are reached.
- Streaming training: Streaming training selects synchronized local windows, performs rolling-context rollouts, and applies policy optimization to the selected clip window.The procedure conditions generation on prior context while extracting window-specific predictions and targets for optimization.
- Objective design: The policy loss interpolates between positive and negative forward-process predictions according to clipped advantages, while KL regularization is applied to marked high-uncertainty samples.The window-focused objective combines advantage-weighted reconstruction terms with a selective KL term.