PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Sora Plan: Open-Source Large Video Generation Model

Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, Tanghui Jia, Junwu Zhang, Zhenyu Tang, Yatian Pang, Bin She, Cen Yan, Zhiheng Hu, Xiaoyi Dong, Lin Chen, Zhang Pan, Xing Zhou, Shaoling Dong, Yonghong Tian, Li Yuan

arXiv:2412.00131v1cs.CVcs.AI

TL;DR

High-quality, long-duration video generation from diverse user conditions remains a challenge, especially for open-source systems. Open-Sora Plan addresses it with a VAE, joint image-video denoiser, condition controllers, efficient training and inference strategies, and automated data curation. The project reports impressive qualitative and quantitative results while remaining an underway open-source effort with noted physical-law understanding weaknesses.

  • Problem

    Few early video-generation efforts challenge high-quality, long-duration video generation, motivating an open-source system for this capability.

  • Method

    Open-Sora Plan combines WF-VAE, a Joint Image-Video Skiparse Denoiser, condition controllers, efficiency strategies, and a multi-dimensional data curation pipeline.

  • Results

    Open-Sora Plan achieves impressive video-generation results in qualitative and quantitative evaluations while generating videos from text, image, and structural conditions.

  • Takeaways & Limitations

    The project contributes an open-source high-quality, long-duration video-generation framework with publicly updated code and model weights.

  • Takeaways & Limitations

    The project remains underway, and the current 2B model performs poorly in understanding physical laws despite later-stage training saturation.

Abstract

from arXiv · show

We introduce Open-Sora Plan, an open-source project that aims to contribute a large generation model for generating desired high-resolution videos with long durations based on various user inputs. Our project comprises multiple components for the entire video generation process, including a Wavelet-Flow Variational Autoencoder, a Joint Image-Video Skiparse Denoiser, and various condition controllers. Moreover, many assistant strategies for efficient training and inference are designed, and a multi-dimensional data curation pipeline is proposed for obtaining desired high-quality data. Benefiting from efficient thoughts, our Open-Sora Plan achieves impressive video generation results in both qualitative and quantitative evaluations. We hope our careful design and practical experience can inspire the video generation research community. All our codes and model weights are publicly available at \url{https://github.com/PKU-YuanGroup/Open-Sora-Plan}.

1 Introduction

Open-Sora Plan targets high-quality, long-duration video generation from diverse conditions through an open-source architecture and supporting training and data strategies. Its design combines specialized model components, efficiency techniques, and automated data curation.

  • Open-Sora Plan generates high-quality, long-duration videos from text prompts, multiple images, and structural signals such as canny edges, depth, and sketches.
  • WF-VAE reduces memory usage and improves training speed by extracting multi-scale frequency-domain features and injecting them into a convolutional backbone through a pyramid structure.It also uses Causal Cache to address latent-space disruption during tiled inference.
  • The architecture comprises a VAE, a Diffusion Transformer, and conditional encoders for frame-level structural control.Conditional injection can target the first frame, selected frames, or all frames.
  • The Joint Image-Video Skiparse Denoiser uses 3D full attention for image and video generation, while Skiparse Attention reduces computation.The design is intended to improve understanding of object motion, camera movement, physics, and human actions.
  • Training and data strategies include min-max token bucketing, adaptive gradient clipping, prompt refinement, and a multi-dimensional curation pipeline with LPIPS-based jump-cut detection.The curation pipeline filters and annotates visual data through operations including motion filtering, subtitle cropping, aesthetic scoring, technical-quality assessment, and captioning.

2.1 Wavelet-Flow VAE

The Wavelet-Flow VAE decomposes videos into multi-scale frequency components, emphasizes low-frequency energy flow, and supports efficient image/video processing through causal caching.

  • Wavelet decomposition: The multi-level 3D Haar wavelet transform decomposes video signals into eight sub-band components at each layer.Each subsequent layer operates on the previous layer’s low-frequency component.
  • Wavelet decomposition: The transform separates low-frequency structure from high-frequency details and can combine 2D and 3D transforms for different downsampling rates.A 4×8×8 temporal×height×width compression rate uses two 3D layers followed by one 2D layer.
  • Training objective: The training objective combines reconstruction, perceptual, adversarial, and KL-divergence losses while enforcing symmetry between encoder and decoder.A wavelet loss additionally penalizes differences between reconstructed and target wavelet components.
  • Training objective: Dynamic adversarial-loss weighting balances the relative gradient magnitudes of adversarial and reconstruction losses.The weighting uses decoder-final-layer gradients and δ = 10^-6 for numerical stability.
  • Causal cache: Causal 3D convolutions and cached tail frames enable unified image/video processing and chunked inference for T-frame videos.The method extracts the first frame, processes remaining frames in Tchunk-sized chunks, and caches Tcache(m) tail frames between chunks.

2.2 Joint Image-Video Skiparse Denoiser

The Joint Image-Video Skiparse Denoiser performs diffusion denoising in a low-dimensional latent space and combines text conditioning, 3D positional encoding, and alternating sparse attention. Skiparse Attention reduces attention computation while retaining broader spatiotemporal interactions than factorized attention.

  • Model Overview: Input images or videos are compressed into latent space, patchified into a 1D token sequence, and conditioned on text prompts through mT5-XXL.The latent input uses a 3D convolution with kernel sizes kt = 1, kh = 2, and kw = 2, producing L = T HW/(ktkhkw) tokens.
  • Model Overview: 3D RoPE partitions token features and injects relative positional information across temporal and spatial dimensions for video sequences.With n = 3, the encoding incorporates temporal, height, and width directions.
  • Model Overview: The denoiser uses a pre-norm transformer with self-attention, cross-attention, and feedforward layers, injecting timestep conditioning through adaLN-Zero.Skiparse attention is applied in every layer except the first and last two layers.
  • Skiparse Attention: Skiparse Attention alternates Single Skip and Group Skip operations, reducing the attention sequence length to approximately 1/k of the original and lowering self-attention complexity.Single Skip bundles tokens spaced by k, while Group Skip first groups adjacent tokens and then bundles groups separated by k^2 positions.
  • Skiparse Attention: Skiparse Attention uses a sparse ratio k between Full 3D Attention and 2+1D Attention, aiming to preserve global spatiotemporal modeling with lower computation.The paper describes k as typically close to 1 but far smaller than HW; its derivation assumes k ≪ THW and THW mod k = 0.
  • Skiparse Attention: Average Attention Distance, ADavg, measures the mean number of attention calculations needed for token pairs to interact, with Full 3D Attention achieving ADavg = 1.The paper uses ADavg and related comparisons to assess how closely sparse mechanisms model full spatiotemporal interactions.

2.3 Conditional Controllers

Open-Sora Plan supports image-conditioned and structure-conditioned video generation through temporal inpainting and a dedicated structure controller. These controllers are trained with progressive masking and inject structural features into transformer blocks.

  • Image Condition Model: The image condition model treats image-conditioned generation as temporal inpainting by adding a mask and masked video alongside latent noise.The mask preserves selected frames, while the masked video provides the visible temporal content.
  • Image Condition Model: The image-conditioned model omits semantic image-feature injection because it did not noticeably improve results and limited motion range.Instead, the model alters the DiT input channels for control.
  • Image Condition Model: Progressive mask training increases task difficulty across text-to-video, continuation, random masking, image-to-video, and transition tasks.The strategy is intended to produce smoother training curves and improve motion consistency.
  • Structure Condition Controller: The Structure Condition Controller encodes a structural signal, projects it into M injection features, and adds each feature to the corresponding transformer-block tokens.The encoder and projector are designed to match the token shapes required by the base denoiser.

3 Assistant Strategies

The assistant strategies target efficient variable-token training, unstable distributed optimization, and mismatched training and user prompts. They combine bucketed token sampling, adaptive anomaly handling, and prompt refinement.

  • 3.1 Min-Max Token Strategy: Min-Max Token Training keeps token counts consistent within each global batch while allowing counts to vary across batches.This maintains nearly identical hardware compute times, reduces synchronization overhead, and remains plug-and-play for multi-resolution, multi-frame data.
  • 3.2 Adaptive Gradient Clipping Strategy: Distributed training can exhibit sporadic loss spikes that degrade output quality without producing NaN errors, and these spikes are not reproducible with a fixed seed.The reported causes include abnormal VAE outputs, multi-node desynchronization, and outlier data.
  • 3.2 Adaptive Gradient Clipping Strategy: Adaptive thresholding uses EMA statistics and a 3-sigma criterion to detect abnormal gradient norms, zeroing anomalous local gradients before all-reduce synchronization.The mechanism adapts to decreasing gradient norms as training progresses rather than relying on a fixed threshold.
  • 3.3 Prompt Refinement Strategy: A prompt refiner expands concise user inputs with actions, scene descriptions, camera language, lighting, and atmosphere to better match detailed training captions.Paired original and augmented texts are used to train the refiner model.

4 Data Curation Pipeline

Open-Sora Plan builds a multi-dimensional curation pipeline that filters video data by temporal structure, motion, subtitles, aesthetics, and low-level visual quality. The resulting image and video collections retain diverse distributions while emphasizing higher-quality material.

  • Video Curation: LPIPS comparisons between consecutive frames identify potential cuts and provide motion statistics for filtering.The reported anomaly thresholds include zthreshold = 2.0, lthreshold = 0.35, zthreshold2 = 3.2, and lthreshold2 = 0.2.
  • Video Curation: Aesthetic filtration uses five uniformly sampled frames and a threshold of 4.75 to remove blurry, low-resolution, overexposed, dark, or watermarked videos.Additional low-level filtering targets bitrate, DPI, and motion-blur artifacts that aesthetic scores may miss.
  • Dataset Statistics: The filtered image collection includes Anytext, Human-images, and SAM, with Anytext concentrated at 512 × 512 while the other datasets show more varied resolutions and scores.These distributions are summarized in Fig. 12(a).
  • Dataset Statistics: The filtered video collection combines Panda70M, VIDAL-10M, and stock-video sources, with Panda70M and VIDAL-10M showing shorter average durations and relatively lower aesthetic scores.The distributions are summarized across duration, aesthetic score, and resolution in Fig. 12(b).

5 Results

Open-Sora Plan is evaluated for VAE efficiency, video-generation quality, and conditional generation. Results report substantial WF-VAE efficiency gains, strong text-to-video performance, and control over image-to-video and structure-to-video outputs.

  • 5.1 Wavelet-Flow VAE: WF-VAE-S encodes 33-frame 512×512 videos at 11.11 videos per second, exceeding CV-VAE and OD-VAE by approximately 6× and 4×.Its memory cost is nearly 5× and 7× lower than those baselines while maintaining superior reconstruction quality.
  • 5.1 Wavelet-Flow VAE: WF-VAE-L exceeds Allegro’s encoding throughput by 7.8× with approximately 8× lower memory usage while maintaining better evaluation metrics.The authors characterize WF-VAE as maintaining state-of-the-art reconstruction performance with lower computational costs.
  • 5.1 Wavelet-Flow VAE: Block-wise inference causes substantial reconstruction degradation for OD-VAE, whereas CogVideoX shows only minor degradation under the comparison conditions.The analysis evaluates PSNR and LPIPS on Panda70M under block-wise and direct inference.
  • 5.2 Text-to-Video: Open-Sora Plan performs exceptionally well against other state-of-the-art models, with advantages in aesthetic quality, smoothness, and scene restoration fidelity.Prompt refinement can be used to further improve generation quality in the reported comparisons.
  • 5.3 Condition Controllers: The structure controller generates high-quality videos whose first, selected, or all frames can be controlled by canny, depth, or sketch signals.Image-to-video showcases also demonstrate realistic motion dynamics, though fully objective evaluation is challenging because results depend on the base model and initial frame.

6 Limitation and Future Work

The authors identify limitations in the model, training efficiency, evaluation, and data, and propose scaling, algorithmic, annotation, and dataset improvements.

  • Data: The decoder contains more parameters than necessary, motivating future streamlining of the architecture.The authors consider the additional decoder parameters redundant while stating that computational cost remains manageable.
  • Transformer Denoiser: The 2B model saturates late in training and performs poorly when understanding physical laws.The authors hypothesize that joint image-video training and larger model scale may address these issues.
  • Transformer Denoiser: Flow Matching validation loss is used to assess convergence, but supervised loss remains an area for further consideration.The discussion relates this issue to stability and convergence assessment in Flow Matching training.
  • Transformer Denoiser: Skiparse Attention reduces FLOPs, but its acceleration is limited at the lower resolutions used for most pre-training.The authors plan to explore more efficient training strategies and additional parallelization, including sequence parallelization.
  • Transformer Denoiser: Current automated evaluation metrics cannot fully replace labor-intensive and costly human review.The authors identify more accurate and reliable automated metrics as a future research priority.
  • Data: The dataset remains limited in diversity, dynamic annotations, resolution and quality, and cross-modal information.Future directions include broader dynamic scenes, camera and motion annotations, higher-resolution video, and audio or other modalities.

7 Conclusion

Open-Sora Plan is presented as an open-source high-quality, long-duration video generation project. Its framework combines specialized generation components with efficiency strategies and automated data curation.

  • Conclusion: Open-Sora Plan decomposes video generation into a Wavelet-Flow Variational Autoencoder, Joint Image-Video Skiparse Denoiser, and condition controllers.These components form the project’s framework-level design.
  • Conclusion: The project introduces min-max tokens, adaptive gradient clipping, and prompt refinement strategies for training and inference.The strategies target efficient training, prevention of outlier gradients, and improved prompt handling.
  • Conclusion: A multi-dimensional data curation pipeline is proposed for automatic high-quality data exploitation.The conclusion presents this pipeline as part of the project’s contribution to open-source video generation research.

Appendix

The appendix presents qualitative comparisons and showcases for video autoencoders, text-to-video generation, and image-to-video generation.

  • VAE Comparison: Figure 15 compares state-of-the-art VAEs on static high-detail reconstruction and dynamic reconstruction under motion blur.The two settings are shown in the figure’s top and bottom panels, respectively.
  • Showcases: The appendix includes a cinematic portrait prompt describing a man illuminated by television light.This is presented as a generated visual-content example.
  • Text-to-Video: Figure 16 compares several state-of-the-art methods in the text-to-video task.The caption identifies the comparison task but does not state an outcome.
  • Text-to-Video: Figure 17 provides text-to-video showcases.The caption labels these as showcases without specifying a quantitative or comparative result.
  • Showcases: Another showcase depicts an armored monkey-like character in a misty mountainous landscape with a pagoda.The description associates the character’s appearance with Sun Wukong based on the stated visual features.
  • Image-to-Video: Figure 18 compares several state-of-the-art methods in the image-to-video task.The caption identifies the comparison task but does not state an outcome.
  • Image-to-Video: Figure 19 provides image-to-video showcases.The caption labels these as showcases without specifying a quantitative or comparative result.
Loading 2412.00131v1…