Source-linked AI summary
A Systematic Post-Train Framework for Video Generation
Zeyue Xue, Siming Fu, Jie Huang, Shuai Lu, Haoran Li, Yijun Liu, Yuming Li, Xiaoxuan He, Mengzhao Chen, Haoyang Huang, Nan Duan, Ping Luo
TL;DR
Pretrained video diffusion models remain sensitive, temporally unstable, artifact-prone, and costly for deployment. The paper proposes a unified post-training pipeline combining SFT, GRPO-based RLHF, prompt enhancement, and autoregressive distillation, achieving 31% overall GSB improvement from RLHF and an additional 20% from prompt enhancement while preserving semantic alignment.
Problem
Pretrained video generation models remain sensitive to prompt wording, unstable over long horizons, artifact-prone, and limited in controllability, instruction following, and deployment efficiency.
Method
The framework combines SFT, GRPO-based RLHF, prompt enhancement, and autoregressive distillation to align video generators with user intentions and deployment constraints.
Results
31% overall GSB improvement came from RLHF, followed by an additional 20% from prompt enhancement while preserving baseline semantic alignment.
Takeaways & Limitations
The unified pipeline provides a scalable, adaptable blueprint for deployable video generation that balances visual quality, temporal consistency, controllability, and system efficiency.
Takeaways & Limitations
Modest text-alignment improvements highlight limitations in current text-video reward models.
Abstract
from arXiv · showhide
While large-scale video diffusion models have demonstrated impressive capabilities in generating high-resolution and semantically rich content, a significant gap remains between their pretraining performance and real-world deployment requirements due to critical issues such as prompt sensitivity, temporal inconsistency, and prohibitive inference costs. To bridge this gap, we propose a comprehensive post-training framework that systematically aligns pretrained models with user intentions through four synergistic stages: we first employ Supervised Fine-Tuning (SFT) to transform the base model into a stable instruction-following policy, followed by a Reinforcement Learning from Human Feedback (RLHF) stage that utilizes a novel Group Relative Policy Optimization (GRPO) method tailored for video diffusion to enhance perceptual quality and temporal coherence; subsequently, we integrate Prompt Enhancement via a specialized language model to refine user inputs, and finally address system efficiency through Inference Optimization. Together, these components provide a systematic approach to improving visual quality, temporal coherence, and instruction following, while preserving the controllability learned during pretraining. The result is a practical blueprint for building scalable post-training pipelines that are stable, adaptable, and effective in real-world deployment. Extensive experiments demonstrate that this unified pipeline effectively mitigates common artifacts and significantly improves controllability and visual aesthetics while adhering to strict sampling cost constraints.
1 Introduction
Pretrained video generators remain unsuitable for deployment because of prompt sensitivity, instability, costly rollouts, and noisy evaluation signals. The proposed unified post-training pipeline combines SFT, GRPO-based RLHF, prompt enhancement, and autoregressive distillation to improve alignment, quality, controllability, and inference efficiency under deployment constraints.
- Motivation: Pretrained video models have advanced to longer, higher-resolution, semantically richer videos but remain sensitive to prompts and unstable in real-world deployment.These limitations contribute to a gap between pretraining performance and deployment requirements.
- Framework: The framework organizes post-training into four stages: SFT, GRPO-based RLHF, prompt enhancement, and autoregressive distillation.Together, these stages target instruction following, perceptual quality, temporal coherence, prompt alignment, and efficient causal inference.
- Motivation: Post-training addresses the gap between likelihood-based pretraining and deployment by optimizing alignment, evaluation quality, sampling cost, and system efficiency.Video generation makes these constraints especially severe because rollout generation is expensive and evaluation signals are often noisy.
- Supervised Fine-Tuning: SFT establishes a stable instruction-following reference policy, reduces generation failures, and enlarges exploration for subsequent RLHF.The method adapts the pretrained model to follow instructions and respond to controllable interfaces.
- Reinforcement Learning from Human Feedback: GRPO-based RLHF optimizes perceptual quality and temporal coherence through relative comparisons within prompt groups without unstable value-function estimation.The method is formulated with stochastic differential equations and measurable objectives.
- Prompt Enhancement and Deployment: Prompt enhancement refines user inputs while preserving alignment, whereas autoregressive distillation transfers post-trained capabilities into a causal architecture for faster inference with preserved generation quality.Prompt enhancement uses GRPO and shared rewards, while distillation uses self-forcing.
2 Related Work
Related work advances video and image generation through language-model-based prompt enhancement, preference alignment for diffusion and flow-matching models, and autoregressive approaches for streaming generation. These studies also expose persistent challenges in user-intent preservation, long-video error accumulation, and real-time distillation.
- Prompt Enhancement: Prompt enhancement increasingly uses language models to optimize text-to-image prompts while preserving user intent and improving aesthetic appeal or controllability.Promptist combines supervised fine-tuning with reinforcement learning, while NeuroPrompts uses constrained text decoding for user-controllable enhancement.
- Preference Alignment: PPO, DPO, and Flow-GRPO adapt reinforcement-learning-based preference optimization to diffusion and flow-matching models for improved task-specific visual generation.Diffusion and flow-matching models use iterative denoising processes and have achieved state-of-the-art image and video synthesis performance.
- Autoregressive Generation: Autoregressive video models support streaming by generating frames sequentially, but Teacher Forcing can cause severe error accumulation during long-video synthesis.Recent work explores alternative training paradigms to address this limitation in real-time generation.
- Autoregressive Generation: Causal Forcing identifies a frame-level injectivity violation when distilling from a bidirectional teacher and uses an autoregressive teacher for initialization to achieve superior real-time generation.The approach theoretically bridges the architectural gap in the initial ODE distillation phase.
3 Method
The method establishes a stable reference policy with SFT, then applies video-tailored GRPO optimization, balanced multi-reward learning, prompt optimization, and autoregressive distillation. These components target stable reward-driven exploration, semantic and perceptual quality, efficient prompt refinement, and scalable video generation.
- Supervised Fine-Tuning: SFT establishes a stable, well-structured reference policy rather than fully solving alignment or subjective quality.It addresses “low-hanging fruit” by transforming an erratic policy into one that is coherent and structurally sound.
- Video GRPO: The video GRPO formulation uses sparse terminal rewards and isotemporal grouping, applying one ODE-to-SDE transition per prompt group while retaining deterministic ODE updates elsewhere.The stochastic timestep enables exploration and gradient computation, whereas deterministic updates produce higher-quality generations and more reliable reward signals.
- Video GRPO: The GRPO-style objective uses group-normalized advantages and clipping to improve terminal rewards while constraining policy updates for stable optimization.The policy loss is computed at each rollout’s assigned timestep, balancing reward-driven exploration with optimization stability.
- Reward Modeling: The unified reward framework balances video aesthetics, image aesthetics, motion quality, and text-video alignment to avoid domination by any single objective.These rewards jointly cover overall visual quality, frame-level fidelity, motion realism and coherence, and semantic consistency with the prompt.
- Prompt Enhancement: RePrompt treats prompt optimization as reinforcement learning with text-video alignment, video aesthetics, and structure rewards, while freezing the generative backbone.GRPO uses group-based advantage estimation without a value network, enabling prompt-strategy learning across off-the-shelf T2I and T2V models.
- Autoregressive Distillation: The autoregressive video diffusion component factorizes frame distributions with the chain rule and trains through Distribution Matching Distillation from a pretrained bidirectional model.Its training is described as consisting of three stages.
4 Experiments
Experiments evaluate the post-training framework with internally trained reward models, curated prompts, and Good–Same–Bad human comparisons across visual quality, motion quality, and text alignment. RLHF improves overall GSB by 31%, while prompt enhancement adds a further 20% improvement without compromising semantic alignment.
- Experimental Settings: The experiments use an internal video generation model, several reward-model types, and curated prompts for RLHF, DMD, ODE regression, and self-forcing training.The SFT stage uses a separately constructed high-quality text-video dataset.
- Reward Models: The reward-model suite covers video aesthetics, text-video alignment, image aesthetics, and text-image alignment using Qwen3.5 features and an MLP.Training follows HPSv3 and uses an uncertainty-aware ranking loss.
- Evaluation Protocol: The evaluation uses Good–Same–Bad comparisons across visual quality, motion quality, and text alignment, allowing annotators to express indifference.These aspects capture appearance and artifacts, temporal coherence and motion plausibility, and prompt-semantic consistency.
- Results: 31% improvement in overall GSB is achieved by RLHF, with the largest gains in visual and motion quality and a more modest text-alignment improvement.The limited text-alignment gain is attributed to the current reward model’s limited accuracy, which constrains semantic optimization.
- Results: 20% additional improvement in overall GSB follows prompt enhancement, driven by visual and motion gains while preserving text alignment.Together, RLHF and prompt enhancement improve aesthetic appearance and temporal dynamics without compromising baseline semantic alignment.
5 Conclusion
The paper presents a unified four-stage post-training framework for deployable video generation, with human evaluations showing substantial gains in quality and coherence while revealing persistent text-alignment limitations.
- Framework: The framework integrates SFT, RLHF with novel GRPO, Prompt Enhancement, and Autoregressive Distillation to bridge pretrained video models with real-world deployment demands.These four stages are presented as synergistic components of a comprehensive post-training pipeline.
- Results: 31% overall GSB improvement was achieved by RLHF, driven by enhancements in visual quality and motion coherence.The result comes from extensive human evaluations using the Good–Same–Bad protocol.
- Results: 20% additional overall GSB improvement from the specialized prompt enhancer elevated perceptual aesthetics and temporal dynamics while preserving baseline semantic alignment.This improvement was reported alongside strict preservation of baseline semantic alignment.
- Limitations and Future Work: Modest text-alignment improvements expose limitations in current text-video reward models and motivate more robust, accurate rewards for semantic correctness.The paper frames stronger text-alignment reward models as future work to unlock greater optimization potential.
6 Broader Impact
The framework bridges foundational video diffusion models and practical deployment by enabling scalable, high-fidelity, computationally efficient synthesis. It also promotes deeper generative competence through temporal optimization, fine-grained controllability, and complex instruction alignment.
- Deployment and Applications: The framework bridges foundational video diffusion models and practical deployment through scalable, high-fidelity, computationally efficient video synthesis.It is positioned for applications across e-commerce, digital marketing, entertainment, and broader creative industries.
- Deployment and Applications: Its real-world orientation supports more reliable and adaptable video generation systems for production demands.The passage connects this reliability and adaptability to improved output quality.
- Generative Competence: Optimizing continuous temporal dynamics, fine-grained controllability, and complex instruction alignment encourages more accurate physical laws, stronger object permanence, and stable causal representations.The framework aims beyond surface-level perceptual quality by preserving structural consistency and event logic over time.