Open-Sora Plan: Open-Source Large Video Generation Model

• WF-VAE extracts multi-scale frequency-domain features with multi-level wavelets, injects them into a pyramid convolutional backbone, and uses Causal Cache to mitigate tiling-induced latent disruption.The VAE is designed to reduce memory usage and improve training speed.Source: §1 · pp. 1–2
• The Joint Image-Video Skiparse Denoiser replaces 2+1D denoising with 3D full attention for image and video generation, then uses Skiparse Attention to reduce computation.The authors say the full-attention structure improves understanding of object and camera motion, physics, and human actions.Source: §1 · p. 2
• Frame-level image controllers support Image-to-Video, Video Transition, and Video Continuation, while a novel network introduces structural controls such as images, canny edges, depth maps, and sketches.Conditional encoders can manipulate the first frame, a subset of frames, or all frames.Source: §1 · p. 2, Figure 1 · p. 2
• Training and inference strategies include min-max resolution-duration buckets, adaptive gradient clipping against outlier data, and prompt refinement that preserves semantics while improving motion stability and detail.Min-max token bucketing aggregates varied resolutions and durations; prompt refinement addresses inconsistent prompt length and descriptive granularity.Source: §1 · p. 2
• The data pipeline detects jump cuts, clips videos, filters motion speed, edge subtitles, and aesthetic and technical quality, and annotates captions; LPIPS-based cut detection avoids false splits in fast-motion shots (Zhang et al., 2018).Source: §1 · p. 2
• The authors describe Open-Sora Plan as an ongoing open-source project and state that its latest news, code, and model weights will be updated publicly.Source: §1 · p. 2

2.1 Wavelet-Flow VAE

• At each layer, a 3D Haar transform recursively decomposes the prior low-frequency component into eight sub-bands.The transform applies filters along temporal and spatial dimensions; the low-frequency component feeds the next layer, while the all-dimension high-frequency component captures details.Source: §2.1 · p. 3
• Two 3D wavelet layers followed by one 2D layer provide a 4×8×8 temporal-height-width compression rate.Source: §2.1 · p. 3
• The training objective combines L1 and perceptual reconstruction losses, adversarial loss, KL regularization, and WL loss to preserve expected wavelet energy flow.WL loss penalizes deviations in the second- and third-layer wavelet components; adversarial loss is dynamically weighted against reconstruction loss using decoder-gradient magnitudes.Source: §2.1 · pp. 3–4
• Causal 3D convolutions use kt −1 initial temporal padding, and inference processes the first frame separately before dividing the remaining video into chunks.The method caches tail frames between chunks and requires (T − kt) to be divisible by st and (T − 1) mod st = 0.Source: §2.1 · p. 4, Figure 3 · p. 4
• Training uses AdamW at a fixed 1 × 10−5 learning rate, with three stages that increase video length from 25 to 49 frames and set λlpips to 0.1.The stages run for 800,000, 200,000, and 200,000 steps, respectively, on 8 NPUs/GPUs; training uses a 3D discriminator and GAN training from the start.Source: §2.1 · p. 4

2.2.1 Model Overview

• The denoiser patchifies latent inputs with a 3D convolution, maps text prompts through mT5-XXL, and uses 3D RoPE for relative temporal and spatial positions.The convolution uses a 1×2×2 kernel and matching strides; 3D RoPE incorporates relative positions across temporal and spatial dimensions.Source: §2.2.1 · p. 4
• The pre-norm transformer block uses self-attention, cross-attention, and an FFN; version 1.2 adopts Full 3D Attention but retains it only in the first and last two layers.Timesteps modulate self-attention and the FFN through adaLN-Zero, while Skiparse attention is used in other layers.Source: §2.2.1 · p. 5, Figure 4 · p. 5

2.2.2 Skiparse Attention

• Skiparse Attention alternates sparse operations that reduce attention sequence length to 1/k and increase batch size k-fold, lowering theoretical self-attention complexity while retaining spatiotemporal modeling.The method chooses k close to 1 but far below HW to approach Full 3D Attention at lower computational cost.Source: §2.2.2 · pp. 5–7
• Group Skip groups adjacent tokens and connects groups separated by k −1 groups, unlike window attention, which only groups adjacent tokens.Together with Single Skip, it forms the alternating Skiparse operations.Source: §2.2.2 · p. 6
• Average Attention Distance (ADavg) measures mean attention-path distance between token pairs, and the implementation uses Skiparse Attention with k = 4 to balance computation and distance.The derivation assumes k ≪ THW and THW mod k = 0 as an approximation for practical use.Source: §2.2.2 · pp. 7–8

2.2.3 Training Details

• Training progresses from text-to-image pretraining through joint image-video pretraining to text-to-video fine-tuning, transferring image-model weights into the video denoiser.The stages use approximately 150k dense-image steps, 100k sparse-attention tuning steps, 200k joint steps, and 30k fine-tuning steps; the fine-tuning model nearly converges around 100k steps.Source: §2.2.3 · pp. 8–9

2.3.1 Image Condition Controller

• The image controller frames conditional generation as temporal inpainting and feeds a mask and masked video alongside latent noise to the denoiser.The mask is reshaped rather than VAE-encoded to avoid temporal downsampling damage to control accuracy; the masked video is VAE-encoded.Source: §2.3.1 · p. 9
• The image controller omits semantic-feature injection because the tested methods did not noticeably improve results and instead limited the range of motion.The authors contrast this input-channel approach with prior methods that inject image features into the UNet or DiT for cross-frame stability (Blattmann et al., 2023; Xu et al., 2024a).Source: §2.3.1 · p. 9
• Progressive training moves from simple tasks at low resolution to higher-resolution image-to-video and transition tasks, producing smoother training curves and improved motion consistency.The mask types include clear, T2V, I2V, transition, continuation, and random-frame retention.Source: §2.3.1 · p. 10, Figure 9 · p. 10

2.3.2 Structure Condition Controller

• The Structure Condition Controller encodes structural signals and projects features that are added directly to the corresponding transformer-block outputs.Its encoder downsamples and tokenizes the signal, then processes it with shape-preserving transformer blocks; token-wise projector transformations preserve the shape needed for addition.Source: §2.3.2 · p. 11

3.1 Min-Max Token Strategy

• Min-Max Token Training pairs a custom bucket sampler with bounded token counts, keeping each global batch’s token count consistent while enabling near-identical device compute times.Token counts may vary across global batches, and sampling is decoupled from model code for plug-and-play use with multi-resolution, multi-frame data.Source: §3.1 · p. 12

3.2 Adaptive Gradient Clipping Strategy

• EMA-based adaptive gradient clipping detects anomalous gradients with a 3-sigma threshold, suppresses affected local-batch gradients, and keeps training loss stable without discarding entire iterations.The method updates its threshold from moving averages of maximum gradient norms and their variance; the illustrated anomaly near step 75.6k is discarded.Source: Figure 11 · p. 13, §3.2 · p. 14

3.3 Prompt Refiner

• To bridge detailed VLM-generated training captions and short user prompts, the prompt refiner expands concise inputs with actions, scene descriptions, and optional cinematic, lighting, and atmosphere details.GPT-4o generated paired original and augmented texts for refiner training, using COCO manually annotated data and JourneyDB VLM-generated labels.Source: §3.3 · p. 14
• The refiner is fine-tuned with LoRA on LLaMA 3.1 8B for one epoch, taking under one hour on a single NPU/GPU.Training uses batch size 32, LoRA rank 64, and AdamW with β1 = 0.9, β2 = 0.999, and learning rate 1.5e-4.Source: §3.3 · p. 15

4.1 Training Data

• Training data combines image-text and OCR datasets with filtered person images, horizontal and vertical videos, and watermark-free stock footage.The sources include 11 million Pixart-Alpha image-text pairs, English-filtered Anytext-3M, 160k selected Laion-5B images, approximately 21M Panda70M horizontal videos, and around 3M VIDAL vertical videos.Source: §4.1 · p. 15

4.2 Data Filtering Strategy

• Video filtering splits long inputs into 16-second clips and uses consecutive-frame LPIPS similarities to detect anomalous cuts and estimate motion.Anomaly indices are further filtered using zero-score and LPIPS thresholds; a manual assessment of 2,000 videos met the authors’ predetermined accuracy criteria.Source: §4.2 · pp. 15–16
• OCR filtering crops subtitle text near video edges but leaves central text unprocessed and does not discard videos.The maximum crop is 20% from each side, yielding a 0.6H × 0.6W video in the extreme case; the authors note text may be meaningful in context.Source: §4.2 · p. 16
• Aesthetic filtration averages five sampled-frame scores and uses a 4.75 threshold, eliminating approximately 40% of videos.The predictor targets blur, low resolution, exposure extremes, darkness, and prominent watermarks or logos.Source: §4.2 · p. 16
• Low-level quality filtering keeps videos with DOVER technical prediction scores > 0, while post-checking motion removes cases with average frame similarity outside 0.001–0.3.These steps filter out 5% and 2% of videos, respectively; the motion recheck addresses inaccuracies caused by subtitle changes after motion detection.Source: §4.2 · pp. 16–17

4.3 Data Annotation

• Dense image and video captions provide semantic supervision for learning correspondences between text and visual features, with large image-language and video-language models used instead of costly manual annotation.Image captions are generated for SAM and Anytext; video captions describe content and chronological changes, and irrelevant prefixes such as “This image” or “The video” are removed.Source: §4.3 · p. 17

4.4 Data Statistics

• Filtered images differ in consistency and diversity, while stock videos tend to be longer and higher-aesthetic than Panda70M and VIDAL-10M.Anytext uses a unified 512 × 512 resolution, while Human-images and SAM span more aesthetic scores and resolutions; most videos are 1280×720, and VIDAL-10M is vertical while the other datasets are predominantly landscape.Source: Figure 12 · p. 17, §4.4 · p. 18
• Filtered images and videos show distinct dataset distributions in aesthetic scores, resolution, and duration.Anytext is consistently 512 × 512, Human-images and SAM are more varied, Panda70M and VIDAL-10M have shorter average durations and lower aesthetic scores than stock footage, and most videos are 1280×720.Source: Figure 12 · p. 17, §4.4 · p. 18

5.1 Wavelet-Flow VAE

• WF-VAE-S encodes 11.11 videos per second at 512×512, about 6× faster than CV-VAE and 4× faster than OD-VAE, with nearly 5× and 7× lower memory cost, respectively.WF-VAE-L exceeds Allegro’s encoding throughput by 7.8× and uses approximately 8× less memory while maintaining better evaluation metrics.Source: §5.1 · p. 18, Table 6 · p. 19
• Causal Cache yields reconstruction numerically identical to direct inference, while OD-VAE degrades substantially and CogVideoX degrades only slightly under block-wise inference.The comparison evaluates PSNR and LPIPS on Panda70M; Table 7 summarizes the results.Source: §5.1 · p. 18, Table 7 · p. 19

5.2 Text-to-Video

• Open-Sora Plan performs strongly in video-generation quality, with advantages in aesthetic quality, smoothness, and scene-restoration fidelity; prompt refinement further improves generation quality.The comparison is reported in Table 8.Source: §5.2 · p. 19, Table 8 · p. 20

5.3 Condition Controllers

• Image-to-video results show realistic motion and balance initial-frame control with text while maintaining semantic consistency and visual quality; structure conditioning controls arbitrary frames using signals such as canny, depth, or sketch.The reported image-to-video comparisons and showcases appear in Figures 18 and 19; the structure-controller results are shown in Figure 13.Source: §5.3 · p. 19

5.4 Prompt Refiner

• Because training uses substantial synthetic labels for dense captions but evaluation prompts are often brief, the reported evaluation uses the prompt refiner to rewrite inputs.The prompt-refiner ablation evaluates videos generated in 480p.Source: §5.4 · p. 20, Figure 14 · p. 20

6.2 Transformer Denoiser

• The 2B model saturates late in training and performs poorly on physical-law understanding, motivating three hypotheses for future work.Examples include overflowing milk, a car moving forward, and a person walking.Source: §6.2 · p. 21
• Two image-video training paths are proposed: joint training from scratch with more images, or image pretraining followed by joint training with more videos.The authors say the second path may offer more decoupling and the first may better align with scaling laws; they note their image training uses only 10M-level data.Source: §6.2 · p. 21
• The authors plan to scale beyond 2B, citing CogVideoX-5B’s stronger physical-law understanding than CogVideoX-2B (Yang et al., 2024b).They describe Deepspeed/FSDP with Zero3 for 10–15B models and MindSpeed/Megatron-LM for scaling to 30B (Zhao et al., 2023; Shoeybi et al., 2019).Source: §6.2 · p. 21
• Further ablations are needed to determine whether flow-based models are more suitable than v-prediction models.The authors note that Flow Matching addresses DDPM stability issues near timestep 0 and that validation loss can indicate whether training is converging in the right direction (Lipman et al., 2022; Ho et al., 2020; Lin et al., 2024b; Polyak et al., 2024; Esser et al., 2024).Source: §6.2 · p. 21
• Skiparse Attention’s FLOP savings are most apparent above 480P, while pretraining near 320 pixels limits acceleration; the model also needs more parallelization than data parallelism.The authors propose exploring sequence parallelism because each token attends to at most 2^k tokens, but efficient implementation remains unresolved.Source: §6.2 · pp. 21–22
• Automated metrics do not fully replace costly human review, so the authors prioritize developing more accurate and reliable video evaluation metrics (Polyak et al., 2024).They cite Vbench and Chronomagic Bench as existing automated evaluation efforts (Huang et al., 2024; Yuan et al., 2024).Source: §6.2 · p. 22

6.3 Data

• The dataset’s narrow domain limits complex, dynamic video generation, motivating broader coverage of realistic environments and physical effects.In a manual sample of 2,000 Panda70M videos, less than 1% showed cars in motion and fewer than 10 showed people walking; about 80% showed half-body conversations.Source: §6.3 · p. 22
• The dataset also lacks camera, style, and motion-speed annotations, sufficient resolution, and cross-modal data, motivating richer annotations, higher-resolution video, and multimodal inputs.The authors propose adding 1080P or 2K video and modalities such as audio to support detailed, contextually richer generation.Source: §6.3 · p. 22

7 Conclusion

• The framework comprises a Wavelet-Flow Variational Autoencoder, a Joint Image-Video Skiparse Denoiser, and condition controllers.The project decomposes the entire video-generation model into these components.Source: §7 · p. 23
• Training and inference strategies include min-max token selection for efficient training, adaptive gradient clipping to prevent outflow gradients, and a prompt refiner for more appreciative results.Source: §7 · p. 23
• A multidimensional data-curation pipeline is proposed to automatically exploit high-quality data.Source: §7 · p. 23
• The authors aim to further advance high-quality video-generation research and the open-source community.They describe Open-Sora Plan as having achieved a remarkable milestone.Source: §7 · p. 23

Appendix

• Figure 15 compares state-of-the-art VAEs on high-detail static-scene reconstruction and dynamic-scene reconstruction under motion blur.The caption identifies the two reconstruction conditions but does not report comparative outcomes.Source: Figure 15 · p. 29
• Figure 16 compares several state-of-the-art methods on the text-to-video task.The caption does not specify which method performs best.Source: Figure 16 · p. 30
• Figure 17 presents text-to-video showcases.The caption does not describe the showcased videos.Source: Figure 17 · p. 31
• Figure 18 compares several state-of-the-art methods on the image-to-video task.The caption does not specify which method performs best.Source: Figure 18 · p. 32
• Figure 19 presents image-to-video showcases.The caption does not describe the showcased videos.Source: Figure 19 · p. 33