PaperlayerView exact revision on arXiv

Source-linked AI summary

SkyReels-V2: Infinite-length Film Generative Model

Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Junchen Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengcheng Ma, Weiming Xiong, Wei Wang, Nuo Pang, Kang Kang, Zhiheng Xu, Yuzhe Jin, Yupeng Liang, Yubing Song, Peng Zhao, Boyuan Xu, Di Qiu, Debang Li, Zhengcong Fei, Yang Li, Yahui Zhou

arXiv:2504.13074v3cs.CV

TL;DR

Video generation still struggles to jointly achieve prompt adherence, visual and motion quality, shot-aware control, and long duration. SkyReels-V2 combines structured multimodal captioning, staged training, motion-specific reinforcement learning, and diffusion forcing; it reports strong benchmark and human-assessment performance while acknowledging extended-generation error accumulation.

  • Problem

    Video generators remain limited by shot-language prompt alignment, motion dynamics, visual-quality trade-offs, and typical 5s-10s duration constraints.

  • Method

    SkyReels-V2 uses SkyCaptioner-V1 for structured general and shot-language captions, progressive-resolution pretraining, staged SFT and motion-specific RL, and diffusion forcing for long-video synthesis.

  • Results

    SkyReels-V2 reports the highest V-Bench score among publicly available models, outperforming several closed-source alternatives in human assessments and matching the leading model comparably.

  • Takeaways & Limitations

    The framework supports open-source ultra-long video generation alongside story, image-to-video, camera-director, and elements-to-video applications.

  • Takeaways & Limitations

    Diffusion forcing remains subject to error accumulation over extended generations, limiting the practical length of high-quality video outputs.

Abstract

from arXiv · show

Recent advances in video generation have been driven by diffusion models and autoregressive frameworks, yet critical challenges persist in harmonizing prompt adherence, visual quality, motion dynamics, and duration: compromises in motion dynamics to enhance temporal visual quality, constrained video duration (5-10 seconds) to prioritize resolution, and inadequate shot-aware generation stemming from general-purpose MLLMs' inability to interpret cinematic grammar, such as shot composition, actor expressions, and camera motions. These intertwined limitations hinder realistic long-form synthesis and professional film-style generation. To address these limitations, we propose SkyReels-V2, an Infinite-length Film Generative Model, that synergizes Multi-modal Large Language Model (MLLM), Multi-stage Pretraining, Reinforcement Learning, and Diffusion Forcing Framework. Firstly, we design a comprehensive structural representation of video that combines the general descriptions by the Multi-modal LLM and the detailed shot language by sub-expert models. Aided with human annotation, we then train a unified Video Captioner, named SkyCaptioner-V1, to efficiently label the video data. Secondly, we establish progressive-resolution pretraining for the fundamental video generation, followed by a four-stage post-training enhancement: Initial concept-balanced Supervised Fine-Tuning (SFT) improves baseline quality; Motion-specific Reinforcement Learning (RL) training with human-annotated and synthetic distortion data addresses dynamic artifacts; Our diffusion forcing framework with non-decreasing noise schedules enables long-video synthesis in an efficient search space; Final high-quality SFT refines visual fidelity. All the code and models are available at https://github.com/SkyworkAI/SkyReels-V2.

1 Introduction

SkyReels-V2 targets prompt adherence, motion quality, visual fidelity, and duration limits in video generation through structured captioning, staged training, reinforcement learning, and diffusion forcing. The model is presented as an open-source system for cinematic, ultra-long video generation and related applications.

  • Motivation: Existing video generators face weak shot-language alignment, imperfect motion dynamics, and typical 5s-10s duration limits.Diffusion and autoregressive approaches also face degraded resolution, error accumulation, or unstable combinatorial noise schedules.
  • Approach: SkyReels-V2 combines MLLMs, multi-stage pretraining, reinforcement learning, and diffusion forcing to address these limitations.The framework uses progressive-resolution pretraining followed by multiple post-training enhancement stages.
  • Approach: SkyCaptioner-V1 unifies general video descriptions with expert-generated shot language covering composition, expressions, and camera motions.The resulting structural representation is designed to improve prompt adherence while efficiently labeling video data.
  • Applications: The open-sourced model family supports story generation, image-to-video synthesis, camera direction, and elements-to-video generation.Released models include diffusion-forcing, text-to-video, image-to-video, camera-director, and elements-to-video variants in 1.3B, 5B, and 14B sizes.
  • Results: Experiments report superior performance against current state-of-the-art methods, including the highest V-Bench score among publicly available models.Human assessments on SkyReels-Bench report performance above several closed-source alternatives and comparable results with the leading video model.
  • Approach: Motion-specific preference optimization targets motion artifacts, while diffusion-forcing adaptation enables ultra-long videos and story generation.The motion stage uses a semi-automatic data collection pipeline, and the diffusion-forcing stage extends temporal coherence and narrative depth.

2 Related Work

Related work spans architectural, data, captioning, reinforcement-learning, and sequence-generation advances in video synthesis. Existing methods improve quality but retain fixed-length, inconsistency, or motion-alignment challenges.

  • Video generation architectures: Video generation has progressed from 2D spatial-plus-temporal architectures toward 3D full-attention systems.Representative developments include U-Net-to-DiT transitions and advances in VAE, text encoders, and flow-matching optimization.
  • Reinforcement learning: Visual-generation reinforcement learning includes reward-weighted regression and direct preference optimization.These approaches either weight trajectories with explicit reward models or optimize preference data without explicit reward modeling.
  • Reinforcement learning: Reward models increasingly use human-annotated data, but older generated training data can misalign learned rewards with motion quality.Earlier systems used CLIP and image-quality scores to assess visual quality and text alignment.
  • Long-video generation: Diffusion models produce strong video quality but generally remain fixed-length, while autoregressive video modeling suffers from error accumulation.This creates a trade-off between diffusion-based performance and autoregressive sequence extension.

3 Methods

SkyReels-V2’s methods build a quality-controlled, shot-aware training corpus and structured captioning system, followed by progressive data balancing and human validation. The pipeline combines automated filtering, overlay cropping, expert shot labels, and SkyCaptioner-V1 annotations.

  • Methodology Overview: The methodology covers data processing, video captioning, multi-task pretraining, and post-training optimization within one training framework.The processing pipeline includes data sources, filtering, cropping, and human-in-the-loop validation before captioning and model training.
  • Data Processing: A progressive filtering pipeline converts diverse raw videos into training data through shot segmentation, captioning, quality filters, and data croppers.Filters address basic quality, video type, and other processing issues at different training stages.
  • Data Processing: Subtitle and logo removal detects overlays in predefined frame regions, then crops to the largest interior rectangle when area and aspect-ratio constraints are satisfied.The crop is retained only if it covers over 80% of the original frame and has a similar aspect ratio.
  • Data Processing: Concept balancing reduces post-training data volume by 50% while reorganizing samples by primary subject categories and their subtypes.The resulting distributions are compared visually and summarized for the top five primary types.
  • Data Processing: Human validation samples data across production stages, including shot segmentation and pretraining, to enforce explicit error and quality thresholds.Reviewers require fewer than 1% shot-segmentation errors; a manually checked pretraining sample must keep overall bad cases below 15%.
  • Video Captioner: SkyCaptioner-V1 fuses general MLLM captions with specialized shot and structural descriptions, using dense text-to-video captions and action-focused image-to-video captions.The caption fields cover subject, appearance, expression, action, position, shot information, and camera motion, with a 10% drop rate per field.

3.4 Post Training

SkyReels-V2 uses four post-training stages to improve baseline quality, motion dynamics, long-video generation, and visual fidelity. Motion preference optimization combines human and synthetic data, while Diffusion Forcing assigns frame-specific noise levels for variable-length synthesis.

  • Post-training pipeline: Four post-training stages use 540p SFT, motion-focused reinforcement learning, Diffusion Forcing training, and final 720p SFT.The first three stages operate at 540p for efficiency; the final stage raises generation resolution to 720p.
  • Motion optimization: Motion-focused reinforcement learning targets large deformable motions and physical-law violations identified as major generator weaknesses.The approach replaces standard diffusion loss with preference optimization focused on motion quality.
  • Motion optimization: The preference pipeline preserves comparable text alignment and video quality while varying motion quality, combining automatic motion pairs with human annotation.A learned reward model captures generic motion-quality differences and guides selection of training samples.
  • Preference data: Human annotation filters mismatched or low-quality pairs and labels retained pairs as Better, Worse, or Tie; almost 80% of pairs are dropped before annotation.Filtering requires clear subjects, sufficient subject size, and simple backgrounds.
  • Preference data: Synthetic preference data deliberately adds controllable distortions to real videos to simulate motion flaws while preserving captions and first-frame references.The distortion process supplements human-labeled data because stringent annotation criteria constrain dataset scale.
  • Diffusion Forcing: Diffusion Forcing converts full-sequence diffusion into frame-specific denoising with a non-decreasing timestep schedule, enabling variable-length generation.FoPP samples a frame and timestep, then propagates valid timestep probabilities through dynamic programming before sampling surrounding frames.

4 Infrastructure

Infrastructure optimizations target memory use, training robustness, parallel execution, and inference latency. Quantization, parallelism, and distillation support faster deployment while preserving the model’s intended capabilities.

  • Training optimization: Training infrastructure combines operator fusion, gradient checkpointing, activation offloading, FSDP, and sequence parallelism to reduce memory pressure.Converting checkpointed transformer inputs from fp32 to bf16 cuts that memory use by 50% with negligible accuracy impact.
  • Training optimization: A self-healing training framework detects compromised nodes, reallocates resources, and restores tasks from checkpoints to maintain uninterrupted training.The remediation process uses three phases: detection and isolation, resource reallocation, and task migration.
  • Inference optimization: Inference optimization combines VRAM optimization, quantization, multi-GPU parallelism, and distillation to reduce latency without compromising quality.The motivation is that diffusion inference typically requires 30 to 50 sampling steps and can exceed five minutes for five seconds of video.
  • Inference optimization: FP8 quantization enables 720p generation of the 14B-parameter model on a single RTX 4090 using parameter-level offloading.FP8 dynamic quantization with GEMM acceleration targets the DiT linear layers, while sageAttn2-8bit accelerates attention.
  • Inference optimization: Content, CFG, and VAE parallelism reduce latency by 1.8× when scaling from four to eight RTX 4090 GPUs.The strategy accelerates single-video generation through three complementary parallelization modes.
  • Inference optimization: DMD distillation uses high-quality video inputs, a two-time-scale update rule, and a four-step generator to significantly reduce generation time.The authors report that a small learning rate and larger batch size are important for stable distillation training.

5 Performance

SkyReels-Bench provides human assessment across instruction adherence, motion quality, consistency, and visual quality, while VBench1.0 supplies automated comparison. SkyReels-V2 achieves the highest reported VBench total and quality scores and supports ultra-long examples.

  • Human evaluation: SkyReels-Bench evaluates T2V and I2V models using 1,020 prompts across instruction adherence, motion quality, consistency, and visual quality.Twenty professional evaluators rate each dimension on a 1–5 scale.
  • Human evaluation: The benchmark decomposes instruction adherence into motion, subject, spatial, shot, and expression adherence, alongside detailed motion, consistency, and visual criteria.Motion quality includes dynamism, fluidity and stability, and physical plausibility; visual quality includes clarity, color accuracy, and structural integrity.
  • Automated evaluation: VBench1.0 evaluation uses its long-prompt version and recommended baseline inference settings, while SkyReels-V2 generates with 50 inference steps and guidance scale 6.The evaluation compares SkyReels-V2 with leading open-source video-generation models.
  • Automated evaluation: 83.9% total and 84.7% quality are SkyReels-V2’s highest VBench scores, exceeding HunyuanVideo-13B and Wan2.1-14B.Its semantic score is slightly below Wan2.1-14B, and the authors attribute the main gap to VBench’s limited shot-scenario semantic evaluation.
  • Long-video generation: Figures 8 and 9 present ultra-long generation using single prompts and sequential prompts, respectively.The examples include a time axis extending to 30 seconds.

6 Application

SkyReels-V2 supports ultra-long video extension, narrative story generation, image-to-video variants, and camera-motion refinement, while related elements-to-video work handles multiple references and identity consistency.

  • 6.1 Story Generation: The diffusion-forcing transformer extends videos with a sliding window conditioned on previous frames and text prompts.The first iteration uses only the text prompt; later iterations use the last f_prev frames to generate f_new frames.
  • 6.1 Story Generation: Previously generated frames receive slight noise to reduce error accumulation during long rollouts.This stabilization technique supports extensions exceeding 30 seconds while preserving visual coherence.
  • 6.1 Story Generation: Sequential narrative prompts generate cohesive long shots spanning multiple actions while maintaining visual consistency.Users can manipulate actions, expressions, and object states across the generated sequence.
  • 6.2 Image-to-Video Generation: The image-to-video framework injects a reference image as a condition while retaining the underlying text-to-video architecture.The image is padded to the target length, encoded into latents, and combined with noise and binary mask channels.
  • 6.3 Camera Director: Approximately 1 million balanced camera-motion samples improved the fluidity and diversity of cinematographic effects after fine-tuning.The model was fine-tuned for 3,000 iterations using 384 GPUs.
  • 6.4 Elements-to-Video Generation: Elements-to-video generation combines multiple reference images and textual prompts to produce naturally composed videos with consistent identity.The related SkyReels-A2 framework is presented as supporting this controllable composition task.

7 Conclusion

SkyReels-V2 combines captioning, multi-stage training, reinforcement learning, and diffusion forcing to target prompt adherence, video quality, motion quality, and unconstrained duration. However, error accumulation during extended generation still limits the practical length of high-quality outputs.

  • 7 Conclusion: SkyReels-V2 targets infinite-length video generation while maintaining shot-scenario prompt adherence, visual quality, and motion quality.Its improvements address prompting, video fidelity, dynamics, and temporal extension through distinct framework components.
  • 7 Conclusion: Diffusion forcing enables seamless video extension without explicit length constraints.The paper states that extended generations remain subject to error accumulation.
  • 7 Conclusion: Error accumulation over extended generations currently limits the practical length of high-quality video outputs.Future work is directed toward improving scalability and reliability.

8 Contributors

The paper acknowledges contributors across sponsorship, infrastructure, data and captioning, and model training, with several model-training contributors marked as equal contributors.

  • 8 Contributors: Yahui Zhou is listed as the project sponsor.
  • 8 Contributors: Infrastructure contributors include Hao Zhang, Weiming Xiong, Zhiheng Xu, and Yuzhe Jin.
  • 8 Contributors: Data and captioning contributors include Mingyuan Fan, Zheng Chen, Chengcheng Ma, Peng Zhao, and Boyuan Xu.
  • 8 Contributors: Model-training contributors include Dixuan Lin, Jiangping Yang, Chunze Lin, Junchen Zhu, Sheng Chen, Wei Wang, Nuo Pang, Kang Kang, Yupeng Liang, Yubing Song, Di Qiu, Debang Li, and Zhengcong Fei.
  • 8 Contributors: A dagger symbol identifies equally contributing authors.

A SkyReels-Bench scoring guidelines

SkyReels-Bench scoring guidelines define a four-dimension human-evaluation rubric using a 1–5 scale, from complete failure to exceptional quality.

  • A SkyReels-Bench scoring guidelines: The rubric evaluates video-generation models across four dimensions.
  • A SkyReels-Bench scoring guidelines: Each dimension receives a score from 1 to 5, where 1 indicates complete failure and 5 indicates exceptional quality beyond base requirements.

B Data Processing Pipeline

The data-processing pipeline finds the largest interior rectangle in a binary matrix and returns its coordinates. It updates column heights row by row and uses stack-based boundary searches to evaluate rectangles.

  • The algorithm takes a 0-1 matrix M ∈ R^m×n and returns the coordinates of its largest interior rectangle.
  • For each matrix row, it accumulates consecutive 1s into a height value for every column and resets heights to zero at 0 entries.
  • It uses a monotonic-stack scan to compute each column’s nearest left boundary while processing the row’s histogram.
  • The procedure initializes the maximum area and rectangle coordinates before iterating through all matrix rows and columns.

C System Prompts of the SkyCaptioner-V1

The captioning system represents videos with structured subject, shot, environment, lighting, and camera-motion fields, then composes those fields into fluent captions. SkyCaptioner-V1 supplies structural captions, while Qwen2.5-32B-Instruct fuses them for T2V and I2V training.

  • Structural Caption Generation: SkyCaptioner-V1 generates structured captions containing subject categories, appearance, action, expression, position, and main-subject status.
  • Structural Caption Generation: The structural representation also records shot type, shot angle, camera motion, environment, and lighting information.
  • Caption Fusion: Qwen2.5-32B-Instruct fuses structured caption fields into dense or sparse final captions for text-to-video and image-to-video model training.
  • System Prompt Design: Caption-composition prompts preserve the original semantics, omit empty fields, and remove static or irrelevant information from action descriptions.
  • System Prompt Design: The composition order prioritizes shot information, timing-related actions, subject attributes, environment, lighting, and camera motion when those fields are present.
  • Caption Fusion: The prompts require direct output of the final composed caption without additional information.

D Video Motion Quality Scoring Criteria For Human Annotation

The video motion quality assessment uses a scoring system to identify and quantify motion-related problems in generated videos. Scores reflect both the severity and the type of motion issue.

  • The assessment criteria identify motion-related issues in generated videos.
  • The scoring system quantifies the identified motion problems using assigned point values.
  • Point values vary according to the severity and type of motion problem.
Loading 2504.13074v3…