Source-linked AI summary
Pyramidal Flow Matching for Efficient Video Generative Modeling
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, Zhouchen Lin
TL;DR
Video generation is expensive because it models a vast spatiotemporal space, while cascaded approaches limit knowledge sharing through separate sub-model optimization. The paper introduces unified pyramidal flow matching with spatial and temporal pyramids, and reports efficient, competitive high-quality video generation.
Problem
Video generative modeling is computationally and data intensive because it must model a large spatiotemporal space, while cascaded sub-models hinder knowledge sharing.
Method
The method uses linked spatial and temporal pyramid stages with piecewise flows, jointly optimized end-to-end in a single Diffusion Transformer.
Results
The model reduces training tokens to ≤15,360 versus 119,040 for a 10-second, 241-frame video and generates high-quality 10-second videos at 768p and 24 fps.
Takeaways & Limitations
Unified pyramidal flow matching supports efficient video generation while maintaining competitive performance on VBench and EvalCrafter.
Takeaways & Limitations
The method only supports autoregressive generation, and temporal compression can sometimes cause subtle long-term subject inconsistency.
Abstract
from arXiv · showhide
Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution latent. Despite reducing computational demands, the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models are open-sourced at https://pyramid-flow.github.io.
1 INTRODUCTION
Video generation is computationally and data intensive because it models a large spatiotemporal space. Pyramidal flow matching unifies spatial and temporal pyramid stages in one end-to-end framework, reducing redundant computation while maintaining competitive video quality.
- Modeling high-dimensional video spatiotemporal space makes video generative-model training computationally and data intensive.
- Cascaded architectures reduce high-resolution generation costs but separately optimized sub-models hinder knowledge sharing.
- Pyramidal flow matching reinterprets generation as linked stages over compressed representations, with only the final stage operating at full resolution.
- The method uses piecewise flows that interpolate between compressed noisier latents and decompressed cleaner latents across pyramid resolutions.
- Spatial and temporal pyramids reduce training tokens to ≤15,360 versus 119,040 for a 10-second, 241-frame video.The framework generates high-quality 10-second videos at 768p resolution and 24 fps using open-source training data.
- A unified flow-matching objective jointly trains pyramid stages in a single DiT with end-to-end optimization and achieves competitive performance on VBench and EvalCrafter.
2 RELATED WORK
Video generative modeling has advanced through autoregressive and diffusion models, but scaling these approaches remains costly for long videos. Image-pyramid methods motivate a compression-based approach spanning spatial representations and temporally compressed history.
- Autoregressive and diffusion video models have made rapid progress, while high-fidelity diffusion models incur significant training costs for long videos.
- Diffusion-autoregressive hybrids support native long-video generation but remain limited in context length and training efficiency.
- This work addresses efficiency from a compression perspective using spatially compressed pyramidal flow and temporally compressed pyramidal history.
- Image pyramids have been studied in visual representation learning and extended to generative models through cascaded diffusion and video generation.
3 METHOD
Pyramidal flow matching reduces video-generation cost by using lower-resolution spatial and temporal stages before the final full-resolution stage. A unified flow objective connects these stages for joint training, while compressed history conditions reduce autoregressive training overhead.
- Spatial pyramid: Pyramidal flow matching interpolates between data and compressed low-resolution noise using a piecewise flow across successive resolutions.Each time window handles one resolution transition, with downsampling and upsampling functions connecting stages.
- Unified training: A unified objective jointly models generation and decompression in one flow model, avoiding separate optimization that hinders knowledge sharing.The conditional path couples noisy, pixelated low-resolution latents with cleaner, higher-resolution results.
- Inference with renoising: Inference preserves continuity between pyramid stages by upsampling prior endpoints and applying rescaling with corrective renoising at jump points.The rescaling matches means, while corrective noise matches covariance; nearest-neighbor upsampling yields blockwise covariance structure.
- Temporal pyramid: Compressed temporal history reduces autoregressive training tokens by up to 1/4^K and improves training efficiency by up to 16^K/T.Most history frames are computed at the lowest resolution, while blockwise causal attention prevents tokens from attending to subsequent frames.
4 EXPERIMENTS
Experiments show that pyramidal flow matching achieves efficient, high-quality video generation across quantitative, human, and qualitative evaluations, while spatial and temporal pyramids accelerate convergence and improve temporal consistency.
- Efficiency: 20.7k A100 GPU hours train a 10-second, 241-frame video model, while inference takes 56 seconds for a 5-second, 384p clip.The framework uses substantially fewer tokens and computations than full-sequence diffusion during training.
- Main results: Our method surpasses all compared open-source baselines on VBench and EvalCrafter, with VBench quality score 84.74 versus 84.11 for Gen-3.It achieves comparable performance to commercial competitors despite training only on publicly accessible video data.
- Main results: A user study compares the method with six baseline models using 50 VBench prompts and ratings from more than 20 participants.Participants rank aesthetic quality, motion smoothness, and semantic alignment.
- Ablation study: The spatial-pyramid variant shows superior visual quality and prompt following, with almost three times the convergence speed in the FID ablation.The comparison uses matched training data, token budget, hyperparameters, and model architecture.
- Ablation study: Under identical training steps, the temporal-pyramid variant produces better visual quality and temporal consistency, while full-sequence diffusion remains far from convergence with fragmented details and severe artifacts.This ablation evaluates the temporal pyramid against a full-sequence diffusion baseline.
5 CONCLUSION
The framework uses unified pyramidal flow matching and temporal compression to improve efficient video generation. It is evaluated on VBench and EvalCrafter, with code and models released publicly.
- The method unifies generation and decompression across pyramid stages with one flow-matching objective and model, facilitating knowledge sharing.
- A temporal pyramid reduces computational redundancy in the full-resolution history of autoregressive video generation.
- The framework demonstrates advantageous performance on the VBench and EvalCrafter benchmarks.
- The authors open-source the code and models at pyramid-flow.github.io.
A DERIVATION
The derivation constructs corrective rescaling and renoising transformations so probability distributions remain continuous across spatial-pyramid stages. It matches means and covariances while controlling the added noise.
- Continuity across spatial-pyramid stages requires matching the endpoint probability distributions after upsampling.
- A linear transformation with corrective Gaussian noise matches the distributions at each pyramid jump point.
- The coefficient s_k/e_k+1 matches distribution means, while noise weight α matches covariance matrices.
- Under nearest-neighbor upsampling, covariance and corrective-noise matrices have blockwise structure, with γ constrained to [−1/3, 0].The lower bound −1/3 ensures the covariance matrix is semidefinite.
- The renoising rule rolls the timestep back slightly when corrective noise is added at each jump point.
B EXPERIMENTAL SETTINGS
The experiments use a large-scale image pretraining stage, specified hyperparameters, multiple baseline models, and user studies alongside detailed benchmark tables. Evaluation materials cover VBench and EvalCrafter settings.
- Model Implementation Details: The implementation uses a 24-layer, 2B-parameter MM-DiT initialized from SD3 Medium, with T5 and CLIP prompt encoders.
- Training Procedure: Image training draws from LAION-5B, CC-12M, SA-1B, and JourneyDB while preserving original image aspect ratios through bucketed training.
- Hyperparameters Setting: Detailed hyperparameters are reported in Table 4 for each optimization stage.
- Baseline Methods: VBench comparisons include eight open-source and commercial baseline methods, including Open-Sora, CogVideoX, Kling, and Gen-3 Alpha.
- User Study: The user study samples 50 VBench prompts, pairs generated videos with six baselines, and collects preferences across aesthetic quality, motion, and other dimensions.
C.1 QUANTITATIVE RESULTS
The model achieves competitive benchmark performance, particularly on motion quality and semantic alignment, while showing weaknesses in appearance-related metrics and face consistency. Ablations examine corrective noise and causal attention.
- VBench: 64.63 dynamic degree ranks the model 2nd among all VBench models, highlighting strong temporal-dynamics performance.
- VBench: On VBench, the model generally exceeds Open-Sora baselines and reaches higher scores than CogVideoX-5B on half of the reported results.
- VBench: VBench performance is relatively weaker for color and appearance style, which the authors associate with coarse-grained video captions.
- EvalCrafter: Motion AC ranks 2nd on EvalCrafter, while BLIP-BLUE and CLIP score place in the top two among compared models.
- EvalCrafter: Face consistency is the only reported EvalCrafter metric where the model performs poorly, attributed to temporal compression of history conditions.
- Ablation Studies: Ablations evaluate corrective Gaussian noise during spatial-pyramid inference and blockwise causal attention for autoregressive video generation.
C.3 VISUALIZATION
The visualization results show that the model produces cinematic-quality, prompt-aligned videos at 5 seconds, 768p, and 24 fps. It outperforms CogVideoX-2B and is comparable to CogVideoX-5B in some aspects, although prompt faithfulness remains imperfect.
- At 5s, 768p, and 24fps, the model generates videos with cinematic quality and reasonable content from textual prompts.The comparisons include Gen-3 Alpha, Kling, and CogVideoX.
- The 2B model outperforms CogVideoX-2B and is comparable to the 5B version in some aspects.For sea-wave prompts, the proposed model and the 5B version generate reasonable waves, whereas CogVideoX-2B produces an almost static surface.
- Prompt faithfulness is imperfect for some descriptions, such as “salt desert.”The authors suggest that more high-quality caption data could address this issue.
C.4 TOY EXPERIMENT OF COUPLING NOISE
The toy experiment compares the proposed coupled sampling with random coupling for piecewise flow matching. Coupled sampling produces straighter trajectories by organizing endpoint directions and reducing intersections.
- Coupled sampling produces much straighter flow trajectories than random coupling in the toy experiment.The experiment maps a few data points to a uniform distribution within separate time windows.
- Independent endpoint sampling creates random trajectory directions and intersections, compromising flow straightness.
- Coupling endpoint sampling organizes trajectories into possibly parallel paths with fewer intersections.This organization explains the improved straightness observed in the toy experiment.
D LIMITATIONS
The method is limited to autoregressive generation and cannot support keyframe or video interpolation. Its temporal pyramid can sometimes cause subtle long-term subject inconsistency, while training-data choices constrain prompt length and scene-transition learning.
- Temporal pyramid compression can sometimes produce subtle subject inconsistency, especially over long durations.The authors identify better temporal compression methods as important for broader applicability.
- Without prompt rewriting, the experiments focus on relatively short prompts.
- Data filtering prevented the model from learning scene transitions during training.The authors suggest adding a scene-director model as a possible remedy.