Source-linked AI summary
Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k
Zangwei Zheng, Xiangyu Peng, Yuxuan Lou, Chenhui Shen, Tom Young, Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, Yuhui Wang, Anbang Ye, Gang Ren, Qianran Ma, Wanying Liang, Xiang Lian, Xiwen Wu, Yuting Zhong, Zhuangyan Li, Chaoyu Gong, Guojun Lei, Leijun Cheng, Limin Zhang, Minghao Li, Ruijie Zhang, Silan Hu, Shijie Huang, Xiaokang Wang, Yuanheng Zhao, Yuqi Wang, Ziang Wei, Yang You
TL;DR
Video generation quality is improving while model size, data, and compute requirements grow. The paper presents Open-Sora 2.0, combining curated data, efficient architecture, training strategy, and system optimization. It reports a $200k training cost, 5–10× lower than comparable models, with performance comparable to leading systems.
Problem
Video generation models increasingly require larger models, more data, and more computing resources as quality improves.
Method
Open-Sora 2.0 jointly optimizes data curation, model architecture, training strategy, and AI infrastructure for efficient video generation.
Results
$200k trains Open-Sora 2.0 at 5–10× lower cost than comparable MovieGen and Step-Video-T2V models, with comparable performance to leading models.
Takeaways & Limitations
Open-Sora 2.0 demonstrates that high-quality video generation models can be developed with highly controlled costs through careful optimization.
Takeaways & Limitations
Deep-compression video VAEs remain underexplored because aggressive compression can reduce reconstruction quality and create adaptation difficulties; generated videos can also contain artifacts and limited detail control.
Abstract
from arXiv · showhide
Video generation models have achieved remarkable progress in the past year. The quality of AI video continues to improve, but at the cost of larger model size, increased data quantity, and greater demand for training compute. In this report, we present Open-Sora 2.0, a commercial-level video generation model trained for only $200k. With this model, we demonstrate that the cost of training a top-performing video generation model is highly controllable. We detail all techniques that contribute to this efficiency breakthrough, including data curation, model architecture, training strategy, and system optimization. According to human evaluation results and VBench scores, Open-Sora 2.0 is comparable to global leading video generation models including the open-source HunyuanVideo and the closed-source Runway Gen-3 Alpha. By making Open-Sora 2.0 fully open-source, we aim to democratize access to advanced video generation technology, fostering broader innovation and creativity in content creation. All resources are publicly available at: https://github.com/hpcaitech/Open-Sora.
1. Introduction
Video generation quality is improving alongside rapidly growing model, data, and compute requirements. Open-Sora 2.0 argues that a top-performing model can instead be trained for $200k while remaining competitive with leading systems.
- Motivation and contribution: Video generation research is increasingly converging on larger models, more data, and greater computing resources.The paper situates Open-Sora 2.0 as an alternative focused on cost-effective training and efficiency optimization.
- Evaluation: The evaluation uses 100 prompts designed to assess visual quality, prompt adherence, and motion quality.Win rate is the percentage of comparisons in which Open-Sora 2.0 is preferred over the competing model.
- Motivation and contribution: $200k trains Open-Sora 2.0, with a reported 5–10× lower cost than comparable MovieGen and Step-Video-T2V models.The authors attribute this efficiency to joint optimization of data curation, training strategy, and AI infrastructure.
- Evaluation: Open-Sora 2.0 outperforms evaluated leading models in at least two of three aspects: visual quality, prompt adherence, and motion quality.Comparisons include Runway Gen-3 Alpha, Luma Ray2, HunyuanVideo, and Step-Video-T2V.
2. Data
Open-Sora 2.0 constructs a hierarchical data pyramid by converting raw videos into training clips and progressively filtering them for higher purity and quality. Its curation combines preprocessing, complementary quality filters, and detailed captions for different resolutions.
- Data filtering: A hierarchical data pyramid supplies progressively smaller, higher-quality subsets for the progressive training process.The filtering strength increases across stages to match training requirements.
- Preprocessing: Raw videos are converted into short continuous clips after removing broken files and clips with duration, bpp, frame-rate, aspect-ratio, or profile violations.Shots longer than 8 seconds are divided into 8-second clips, while shots shorter than 2 seconds are discarded.
- Data filtering: Aesthetic, motion, blur, OCR, and camera-jitter filters target distinct defects through score thresholds.Together, these filters form a comprehensive data-purification system.
- Captioning: LLaVA-Video captions 256px videos, while Qwen 2.5 Max generates more accurate captions for 768px training data.Caption prompts cover subjects, actions, environment, lighting, camera movement, and video style.
3. Model Architecture
The model architecture combines a deeply compressed video autoencoder with a diffusion transformer. Video DC-AE sharply reduces latent tokens and improves speed while retaining competitive reconstruction quality, enabling efficient high-resolution generation.
- Architecture overview: The model initially uses HunyuanVideo VAE before adapting to Video DC-AE to reduce training and inference costs while maintaining reconstruction fidelity.The architecture comprises an autoencoder and a diffusion transformer.
- 3D Autoencoder: Video DC-AE uses 4 × 32 × 32 compression and patch size 1, reducing a 5-second 768px video's token count from 76K to 19K.The configuration yields 5.2× higher training throughput and over 10× faster inference than HunyuanVideo VAE.
- 3D Autoencoder: Video DC-AE achieves competitive reconstruction, with only minor LPIPS degradation versus HunyuanVideo VAE while maintaining strong PSNR and SSIM.The 128-channel variant is selected for generative-model adaptation because its reduced channel size enables faster adaptation.
- DiT Architecture: The diffusion transformer uses full attention, latent patchification, and a hybrid dualstream/single-stream design for multimodal processing.Dualstream blocks separately process text and video, while single-stream blocks integrate them; 3D RoPE captures spatial and temporal information.
4. Model Training
Open-Sora 2.0 uses a staged, budget-conscious training pipeline that concentrates computation in efficient low-resolution training and brief high-resolution adaptation. Joint optimization of data, architecture, training strategy, and infrastructure constrains one-time training cost to $200K while targeting high-quality video generation.
- Training Strategy: A three-stage pipeline trains T2V at low resolution, I2V at low resolution, then fine-tunes I2V on high-resolution videos.This progressive setup supports efficient motion learning before expensive high-resolution adaptation.
- Model Initialization: Flux initializes the T2V model, avoiding training an 11-billion-parameter image model from scratch.The paper reports that image pretraining can significantly accelerate video model training.
- Resolution Adaptation: Training a 129-frame video at 768px is 40 times slower than at 256px because self-attention scales quadratically with token count.The strategy therefore allocates most computation to low-resolution training and limits high-resolution computation.
- Resolution Adaptation: I2V adapts from 256px to 768px more efficiently than T2V by conditioning on a static image and focusing the model on motion generation.Low-resolution T/I2V training followed by brief high-resolution fine-tuning produces high-quality results with minimal additional training.
- Training Cost: $200K constrains the one-time training cost, yielding 5–10× lower costs than comparable Movie Gen and Step-Video-T2V models.The comparison uses estimated costs for the reference models based on publicly available information.
- Training Objective: Flow matching trains the model to predict the velocity X0 − X1 from an interpolated latent Xt conditioned on text and/or image inputs.The timestep is sampled from a logit-normal distribution and scaled according to video shape.
- System Optimization: Multi-bucket training handles varying frame counts, resolutions, and aspect ratios while dynamically assigning batch sizes to improve GPU utilization.Batch sizes are searched on H200 GPUs for different training configurations.
5. Conditioning
Open-Sora 2.0 supports multiple conditioning modes through channel-based image and video conditioning, while inference strategies separately control image adherence, text alignment, motion, and temporal coherence.
- Conditioning framework: Condition information is encoded and concatenated through additional channels, with a task-type mask supporting I2V and V2V tasks.The framework preserves the velocity prediction task while distinguishing input types.
- Conditioning framework: Image-condition dropout converts some training examples into T2V settings, improving robustness across T2V and I2V tasks.The dropout ratio for T/I2V training is 12.5%.
- Image-to-video inference: Image and text guidance are decoupled because strong image guidance can make videos static, whereas stronger text guidance improves semantic alignment.The model uses different guidance behavior for image fidelity and prompt adherence.
- Image-to-video inference: Guidance oscillation alternates image-guidance strength during sampling to balance stability and motion consistency and reduce flickering.In the 50-step example, alternation begins after the first 10 steps and reduces even-step image guidance to 1.
- Image-to-video inference: Linear dynamic image-guidance scaling across frame index and denoising step provides the best performance, using g_img = 3 and g_txt = 7.5 by default.The strategy strengthens guidance for later frames while reducing its importance at later denoising steps.
- Motion control: Motion score is appended to captions as a conditioning signal, enabling independent inference-time control over generated dynamics.Higher motion scores correspond to more pronounced camera and scene movement.
6. System Optimization
The training system combines GPU-aware parallelization with compiler and kernel optimizations to reduce memory bottlenecks and improve high-resolution training efficiency.
- Training infrastructure: ColossalAI training on 141GB H200 GPUs enables stronger data parallelism and more selective activation checkpointing.PyTorch compile and Triton kernels further accelerate training.
- Parallelization: Tensor parallelism partitions video-autoencoder convolution weights, while ZeroDP and context parallelism partition MMDiT sequences across GPUs.Sequence partitioning lets each GPU compute attention independently for high-resolution videos.
- Measured efficiency: 38.19% MFU is achieved in Stages 1 and 2 with DP and ZeRO-2, while Stage 3 reaches 35.75% MFU with ZeRO-2 and CP=4.Empirical evaluation identifies CP alone as the best memory-efficiency and computational-performance trade-off.
- Auxiliary optimizations: The system also uses selective activation checkpointing, automatic failure recovery, pinned-memory dataloader buffers, and checkpoint management.These auxiliary optimizations are described as additional training-system components.
7. Performance
Open-Sora 2.0 is evaluated against open-source and proprietary video models using standardized prompts, blinded human judgments, and VBench measurements.
- Evaluation setup: The model supports T2V and I2V generation at 256px and 768px resolutions for videos up to 128 frames, or 5 seconds at 24 FPS.Its default T2I2V pipeline uses FLUX to generate the starting image for video generation.
- Evaluation setup: Benchmarking uses 100 text prompts, one inference per model, default settings, and model-dependent video lengths and resolutions.The comparison includes closed-source APIs and open-source models.
- Human evaluation: Ten professional evaluators assess visual quality, prompt adherence, and motion quality in a blinded evaluation.The criteria cover fidelity and aesthetics, text alignment, and motion consistency and physical plausibility.
- VBench evaluation: The VBench gap between Open-Sora and OpenAI’s Sora decreases from 4.52% to 0.69%, while Open-Sora 2.0 scores higher than CogVideoX1.5-5B and HunyuanVideo.Figure 10 also reports significant improvement from Open-Sora 1.2 to 2.0.
8. Conclusion
Open-Sora 2.0 demonstrates commercial-level video generation at controlled cost and supports both T2V and I2V generation, while remaining challenges include reconstruction loss, artifacts, and limited content control.
- Conclusion: Open-Sora 2.0 is trained for $200k, reported as 5-10 times more cost-efficient than MovieGen and Step-Video-T2V.The paper attributes this efficiency to joint optimization of data curation, architecture, training strategy, and system optimization.
- Conclusion: Despite its lower training cost, Open-Sora 2.0 performs comparably to leading models including HunyuanVideo and Runway Gen-3 Alpha.The model supports T2V and I2V generation up to 768×768 pixels for videos up to 5 seconds.
- Remaining challenges: Higher video-autoencoder compression reduces latent tokens but can introduce reconstruction-quality loss and adaptation difficulties.The paper identifies deep-compression VAE technology as an underexplored challenge.
- Remaining challenges: Diffusion models can produce object distortion and unnatural physics, while users have limited control over these details.The paper calls for further research on artifact prevention and enhanced control over generated content.
A. Data Statistics
The dataset statistics characterize visual quality, temporal duration, aspect ratios, and caption length, while caption vocabulary spans subjects, actions, settings, and lighting.
- Most videos have aesthetic scores between 4.5 and 5.5, indicating moderate visual appeal.
- Video durations range from 2 to 8 seconds, with nearly half consisting of 6–8-second clips.The longer clips provide richer temporal information for learning dynamic patterns.
- Most aspect ratios fall between 0.5 and 0.75, corresponding to the 16:9 format.This supports adaptability across formats and model generalization.
- Over 70% of captions exceed 75 words, providing detailed descriptions for training.
- Caption vocabulary includes subjects, actions, background elements, settings, atmosphere, and lighting conditions.Frequent occurrences of “person” and “individual” indicate that many captions concern people.
C.1. Architecture Hyperparameters
Table 3 presents the architecture hyperparameters for the Open-Sora 2.0 11B-parameter video generation model.
- Table 3 lists architecture hyperparameters for the Open-Sora 2.0 11B-parameter video generation model.
D. Multi-Bucket Training
Multi-bucket training combines configuration-specific batch-size selection, highly compressed video representations, quality-aware inference scaling, and system optimizations to improve training and generation efficiency.
- Multi-Bucket Training: Batch sizes are selected per training configuration to maximize feasible throughput without exceeding memory or reference execution-time constraints.The search uses H200 GPUs and accounts for parallelism, resolution, frame counts, token count, and autoencoder or forward-backward execution time.
- Architecture and Token Reduction: A 4 × 32 × 32 autoencoder compression with 1 × 1 × 1 patches yields Dtoken = 4096 and reduces a 5-second video from 76K to 19K tokens.The comparison uses a 4 × 8 × 8 VAE with 1 × 2 × 2 patches as the 76K-token configuration.
- Inference-Time Scaling: Quality-aware inference injects controlled noise, evaluates candidates with six VBench metrics, and continues with the highest-scoring candidate.Users can prioritize metrics such as motion smoothness or aesthetic quality; full scaling produces stable, natural motion under challenging prompts.
- System Optimization: Selective activation checkpointing, offloading, asynchronous data movement, recovery, and checkpoint I/O reduce memory use, stalls, and recovery overhead.Checkpoint saving overhead is reduced to the order of seconds through pinned memory and asynchronous disk writing.
I. Generation Samples
The generation-sample section presents high-quality videos generated by Open-Sora 2.0, including a coastal-view scene with people enjoying the setting.
- One described sample features dramatic coastal views with waves crashing below and people enjoying the scene.
- The sample description includes a warm afternoon-light setting alongside the coastal activity.
- Figure 16 presents high-quality videos generated by Open-Sora 2.0.