Source-linked AI summary
Open-Sora: Democratizing Efficient Video Production for All
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, Yang You
TL;DR
Video generation and world simulation remain less developed than language capabilities, motivating accessible visual-intelligence systems. Open-Sora addresses this gap with an open-source video-generation framework using STDiT, a compressive 3D autoencoder, and structured training strategies; it supports high-quality generation across multiple settings while releasing the code, data resources, and weights.
Problem
Artificial visual intelligence, particularly video generation and world simulation, lags behind advances in AI language ability.
Method
Open-Sora combines the STDiT video diffusion architecture, a compressive 3D autoencoder, open training data, and data-processing and training pipelines.
Results
Open-Sora generates videos up to 16 seconds long at multiple resolutions up to 720p, with controllable motion for text-to-video and image-to-video tasks.
Takeaways & Limitations
Open-Sora makes video-generation training resources, code, and model weights available to the community.
Takeaways & Limitations
Applying mixed-timestep conditioning directly to a pretrained diffusion model often yields suboptimal results.
Abstract
from arXiv · showhide
Vision and language are the two foundational senses for humans, and they build up our cognitive ability and intelligence. While significant breakthroughs have been made in AI language ability, artificial visual intelligence, especially the ability to generate and simulate the world we see, is far lagging behind. To facilitate the development and accessibility of artificial visual intelligence, we created Open-Sora, an open-source video generation model designed to produce high-fidelity video content. Open-Sora supports a wide spectrum of visual generation tasks, including text-to-image generation, text-to-video generation, and image-to-video generation. The model leverages advanced deep learning architectures and training/inference techniques to enable flexible video synthesis, which could generate video content of up to 15 seconds, up to 720p resolution, and arbitrary aspect ratios. Specifically, we introduce Spatial-Temporal Diffusion Transformer (STDiT), an efficient diffusion framework for videos that decouples spatial and temporal attention. We also introduce a highly compressive 3D autoencoder to make representations compact and further accelerate training with an ad hoc training strategy. Through this initiative, we aim to foster innovation, creativity, and inclusivity within the community of AI content creation. By embracing the open-source principle, Open-Sora democratizes full access to all the training/inference/data preparation codes as well as model weights. All resources are publicly available at: https://github.com/hpcaitech/Open-Sora.
HPC-AI Tech
The paper presents Open-Sora as a high-fidelity video generation system and makes its resources publicly available.
- Open-Sora generates high-fidelity videos.
- The project is documented as a technical report with equal contribution noted.
1 Introduction
Open-Sora is introduced as an early open-source effort to reproduce Sora-style video generation. It provides comprehensive training support and generates controllable videos across durations, resolutions, and task types.
- Open-Sora is described as one of the earliest projects to reproduce Sora and as achieving strong results.
- Open-Sora provides training data processing, training code, and model weights for video generation.
- The project reproduces nearly all techniques described in the Sora report.
- Open-Sora generates videos up to 16 seconds long and 720p across text-to-video and image-to-video tasks.
- Versions 1.0, 1.1, and 1.2 correspond to releases in March, April, and June 2024.
2 Data
Open-Sora uses open datasets and a data-processing pipeline to construct video-text training data. The pipeline filters visual quality and motion, generates captions, and reports statistics for final-stage training.
- 2.1 Data Source: 30M video clips spanning 2 to 16 seconds provide 80k hours of open-sourced training data.
- 2.1 Data Source: The image dataset contains around 3M images used alongside videos for training.
- 2.2 Data Pre-processing: The processing pipeline detects scenes, cuts videos into clips, and converts raw videos into video-text pairs.
- 2.2 Data Pre-processing: Filtering uses aesthetic, optical-flow, and OCR signals to remove undesirable or low-quality video content.
- 2.2 Data Pre-processing: GPT-4V and PLLaVA generate captions, while optical flow adds camera-motion information that captioning models often miss.
- 2.2 Data Pre-processing: Final-stage data statistics cover duration, resolution, aesthetic score, optical-flow score, and object and action tags.
3 Model Architecture
Open-Sora combines a stacked 3D video autoencoder with STDiT, which separates spatial and temporal attention for efficient video generation. The architecture also supports image- and video-conditioned generation through masking and conditioning strategies.
- 3.1 3D Autoencoder: The video compression network combines spatial 8x8 compression with temporal 4x compression, avoiding frame extraction and enabling generation at the original FPS.It builds on a 2D VAE and adds temporal compression to improve temporal fluency.
- 3.1 3D Autoencoder: The 3D VAE uses a Magvit-v2-inspired architecture with 300M parameters, while the complete video compression network has 384M parameters.It was trained for 1.2M steps on 17-frame, 256×256 clips from Pexels and Pixabay.
- 3.1 3D Autoencoder: The stacked VAE requires minimal inference memory and achieves comparable performance to another open-source 3D VAE at significantly lower computational cost.Inputs are already compressed, and inference processes videos in 17-frame clips.
- 3.2 Architecture: STDiT applies spatial self-attention within each frame and temporal attention across frames at matching spatial locations instead of full attention over all tokens.The model builds on PixArt, uses T5 text encoding, and applies crossattention between video and text latents.
- 3.2 Architecture: Open-Sora supports image-to-image and video-to-video conditioning by assigning timestep 0 to unmasked conditioning frames while other frames retain diffusion timesteps.Random masking during training teaches patterns including first-frame, last-frame, prefix, suffix, combined, and random-frame conditioning.
4 Training Strategy
Open-Sora combines multi-stage adaptation, bucketed data organization, and validation across video lengths and resolutions to improve training efficiency while preserving flexible, high-quality generation. The strategy also supports scalable model training and reports strong performance among open-source video-generation models.
- Training Strategy: 35,000 H100 GPU hours were required for 68k training steps using a pre-trained image model, flow matching, and multi-stage training.The approach is described as reducing training cost while achieving high-quality video generation performance.
- Multi-resolution and Multi-aspect-ratio: Bucketed training assigns videos to fixed resolution, frame-length, and aspect-ratio triplets, selecting the largest bucket that fits each sample.Per-bucket retention probabilities and batch sizes downsample some high-resolution videos and balance GPU load.
- Multi-resolution and Multi-aspect-ratio: The bucket strategy trades implementation simplicity for computational efficiency while supporting diverse resolutions and aspect ratios.The authors characterize this as enabling flexible training across common video formats.
- Model Adaptation: Rectified flow reduces video sampling steps from 100 to 30, while QK-normalization enables more aggressive optimization and improves training efficiency.These changes are part of the sequential adaptation from an image-generation checkpoint to video generation.
- Model Adaptation: The video compression network temporally compresses representations by 4x, reducing computational cost, while supporting generation from 144p to 2K.After adaptation, the model retains high-quality image generation and gains multiple video-generation benefits.
- Validation and Results: Validation loss and VBench scores improved consistently during training, and Open-Sora achieved state-of-the-art video-generation performance among open-source models.Validation used 1k Pixabay videos across 2s–16s lengths and 144p–720p resolutions; VBench used 240p 2-second videos.
5 Conclusion
Open-Sora provides an open-source framework for video generation, including data processing, training code, and model weights. It enables high-quality videos up to 16 seconds at resolutions up to 720p with controllable motion dynamics.
- Conclusion: Open-Sora democratizes access to video-generation technology by releasing data processing, training code, and model weights.The framework is presented as a foundation for community collaboration and future advances.
- Conclusion: The released system generates high-quality videos up to 16 seconds long and 720p resolution with controllable motion dynamics.The conclusion describes this capability as applying to the open-source video-generation framework.