Open-Sora: Democratizing Efficient Video Production for All

• Video generation has advanced from strong UNet-based models [2] [11] [27] to Transformer-based diffusion models such as DiT, following OpenAI’s Sora.Open-Sora is presented as one of the earliest projects to reproduce Sora.Source: §1 · p. 2
• Open-Sora provides code for data processing, filtering, and training, along with model weights, and reproduces nearly all techniques described in the Sora report.It generates videos up to 16 seconds and 720p, with controllable motion for text-to-video and image-to-video tasks.Source: §1 · p. 2
• Open-Sora 1.2 is the default reference in this paper, which reviews techniques across versions 1.0, 1.1, and 1.2 and highlights version 1.2’s performance.The paper covers data composition and preprocessing, model architecture, and training.Source: §1 · p. 2

2.1 Data Source

• The open video corpus contains 30M clips of 2–16 seconds, totaling 80k hours.Sources include Webvid-10M, a 20M high-quality subset of Panda-70M, MiraData, Vript [36], and Inter4K; HD-VG-130M captions were judged relatively poor in scene and text quality. Free-licensed videos also come from Pexels, Pixabay, and Mixkit.Source: §2.1 · p. 2
• The accompanying image dataset contains around 3M images, including LAION images with aesthetic scores above 6.5 and 25k nature-themed Unsplash-lite photos.The paper describes Unsplash-lite as covering a wide range of uses and contexts.Source: §2.1 · p. 2

2.2 Data Pre-processing

• The preprocessing pipeline cuts videos into scenes, filters visual quality, motion, and dense-text content, then creates video-text pairs.PySceneCut detects scenes; filtering uses average aesthetic scores from three sampled frames, UniMatch optical-flow scores, and DBNet++ via MMOCR to remove videos with too much text.Source: §2.2 · p. 3
• Captions are generated with pretrained PLLaVA 13B using four sampled frames, with optical-flow camera-motion descriptions appended.The paper also reports using GPT-4V and PLLaVA [35]; despite some hallucinations, captions were considered sufficient for training.Source: §2.2 · p. 3
• Last-stage video data statistics cover duration, resolution, aesthetic and optical-flow score distributions, and object and action tag frequencies.These statistics are presented in Figure 4.Source: Figure 4 · p. 3, §2.2 · p. 4

3.1 3D Autoencoder

• Open-Sora 1.2 compresses video spatially by 8x8 and temporally by 4x, eliminating frame extraction and enabling generation at the original FPS.Earlier versions used Stability-AI’s 2D VAE [25] and extracted one frame every three frames, which reduced generated FPS and temporal fluency. The 3D compression design exploits correlated features after 2D compression and is illustrated in Figure 5.Source: §3.1 · p. 4
• Mixed-length training in Stage 3 resolved blurriness in videos of non-standard lengths caused by earlier fixed-length video clips and zero-padded images.Stage 3 replaced reconstruction of 2D VAE features with direct reconstruction of original videos and randomized video lengths up to 34 frames with zero-padding; it used 24 GPUs. The earlier stages used 17-frame clips and an 80% video, 20% image dataset.Source: §3.1 · pp. 4–5
• The stacked VAE uses compressed inputs and 17-frame inference clips, achieving performance comparable to another open-source 3D VAE [17] at significantly reduced computational cost.The 2D VAE was initialized from SDXL’s pre-trained VAE [23], while the 300M-parameter 3D VAE followed Magvit-v2’s architecture [37]; together the compression network has 384M parameters.Source: §3.1 · pp. 4–5

3.2 Architecture

• STDiT applies spatial self-attention within each frame and temporal attention across frames at each spatial location, replacing full attention over all tokens.The model builds on PixArt [4] and uses T5 text encoding with cross-attention between video and text latents; its spatial-temporal design is inspired by Latte [20].Source: §3.2 · p. 5
• The transformer starts from PixArt-α, with zero-initialized temporal-attention projections preserving image-generation capabilities; adding temporal attention increases parameters from 580M to 1.1B.Rotary positional embeddings support temporal attention, while QK-normalization inspired by SD3 improves stability, particularly in half-precision training; epsilon 1 × 10−15 avoids training spikes [21].Source: §3.2 · p. 5

3.3 Conditioning

• Randomly unmasking frames during training enables image and video conditioning, while applying masking to 50% of samples yields these capabilities after 10k steps with minimal impact on text-to-video performance.Conditioning frames receive timestep 0 while other frames retain diffusion timesteps; direct use with a pretrained model can be suboptimal because it has not learned mixed timesteps. A 30% masking probability reduced conditioning effectiveness, so the model was pretrained from scratch using the strategy, inspired by UL2 [30].Source: §3.3 · p. 6
• Appending aesthetic scores, motion scores, and camera-motion descriptors to captions provides adjustable conditioning inputs for controlling video generation.For camera-motion conditioning, the authors manually labeled 13,000 high-confidence clips; inference-time scores can be adjusted to influence generation.Source: §3.3 · p. 6

4 Training Strategy

• The training strategy adapts a pre-trained image model to video with flow matching over 68k steps, using approximately 35,000 H100 GPU-hours.The paper describes this approach as reducing training cost while achieving high-quality video generation.Source: §4 · pp. 6–7

4.1 Multi-resolution and Multi-aspect-ratio

• Bucket-based training groups samples by resolution, frame count, and aspect ratio, balancing implementation simplicity with computational efficiency and format flexibility.Bucket-specific keep probabilities downsample some high-resolution videos, while batch sizes are adjusted to balance GPU load.Source: §4.1 · p. 7

4.2 Model Adaptation

• Adaptation added QK-norm, rectified-flow training, Open-Sora’s VAE, FPS conditioning, and zero-initialized temporal attention blocks, initially trained on images before masking focused them exclusively on videos.The adapted model retains high-quality image generation and supports video resolutions from 144p to 2K; rectified flow reduces video sampling steps from 100 to 30, while the new VAE compresses the temporal dimension by a factor of 4.Source: §4.2 · pp. 7–8

4.3 Multi-stage Training

• Training proceeds through three data stages, moving from lower-resolution videos to curated, higher-resolution clips.The first stage uses Webvid-10M videos at 240p and 360p; the second uses an aesthetic-filtered Panda-70M subset, mainly at 360p and 480p; the final stage uses about 2M curated clips at 720p and 1080p. Evaluation loss and VBench scores show consistent improvement during training.Source: §4.3 · pp. 8–9
• Open-Sora achieves state-of-the-art video-generation performance among open-source models on VBench.Table 2 compares VBench scores across open-source models.Source: Table 2 · p. 9, §4.3 · p. 9

5 Conclusion

• Open-Sora combines data processing, training code, and model weights in a comprehensive open-source video-generation framework.The paper describes the framework as a step forward in open-source video generation.Source: §5 · p. 9
• Open-Sora reproduces key techniques from the Sora report and generates high-quality videos up to 16 seconds long, at resolutions up to 720p, with controllable motion dynamics.Source: §5 · p. 9
• The initiative fosters community collaboration and provides a foundation for future advances in video generation.Source: §5 · p. 9