Source-linked AI summary
Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, Song Han
TL;DR
Video diffusion Transformers are costly because 3D full attention grows quadratically with context length, limiting practical generation. Sparse VideoGen is a training-free framework that profiles dynamic spatial and temporal sparsity and applies hardware-efficient sparse inference. It achieves end-to-end speedups while preserving video quality.
Problem
3D full attention makes video diffusion expensive because its computation grows quadratically with context length, limiting practical video generation.
Method
SVG uses online profiling to classify dynamic attention heads as spatial or temporal and applies corresponding hardware-efficient sparse attention without retraining.
Results
2.33× end-to-end speedup is achieved while maintaining high visual quality, with PSNR up to 29 on representative open video generation models.
Takeaways & Limitations
SVG provides practical efficiency gains for video diffusion while preserving generation quality and remaining compatible with FP8 quantization.
Takeaways & Limitations
Temporal sparsity is hardware-inefficient, and determining the optimal sparse pattern directly with full attention provides no practical efficiency benefit.
Abstract
from arXiv · showhide
Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen
1. Introduction
Video DiTs generate high-quality, temporally consistent videos but remain computationally expensive because 3D full attention scales quadratically with context length. Sparse VideoGen exploits two attention sparsity patterns through training-free profiling and efficient inference, delivering speedups while preserving quality.
- Motivation: HunyuanVideo requires almost an hour on an NVIDIA A100 GPU to generate a 5-second video, with 3D full attention exceeding 80% of end-to-end runtime.Attention becomes more dominant as resolution and frame count increase because its computational complexity is quadratic in context length.
- Observation: Video attention exhibits two inherent patterns: Spatial Heads focus within frames, while Temporal Heads attend across frames at corresponding spatial positions.These patterns contribute to spatial structure and temporal consistency, respectively.
- Method: SVG dynamically classifies each attention head using online profiling and applies the corresponding sparse attention computation.The profiling samples a small portion of queries, compares spatial and temporal sparse attention against full attention, and selects the lower-error pattern.
- Method: SVG addresses changing sparsity patterns across denoising steps and input data without requiring additional training.A head can switch between spatial and temporal behavior for different prompts, motivating on-the-fly identification.
- Results: 2.33× end-to-end speedup is achieved while maintaining high visual quality, with PSNR up to 29 on representative open video generation models.The evaluated models include CogVideoX-v1.5-I2V, CogVideoX-v1.5-T2V, and HunyuanVideo-T2V.
2. Related Work
Related approaches reduce diffusion cost through fewer denoising steps, compression, system optimization, or sparse and low-bit attention. SVG differs by operating directly on pretrained video models and remaining compatible with complementary compression and attention techniques.
- Decreasing the denoising steps: Few-step diffusion methods reduce sampling cost through ODE solvers, consistency losses, or distillation, but distillation approaches require expensive retraining or fine-tuning.SVG instead uses off-the-shelf pretrained models without additional training.
- Diffusion model compression: SVG is orthogonal to quantization, efficient architectures, and high-compression autoencoders, so these techniques can provide additional gains when combined.The paper specifically reports compatibility with FP8 quantization.
- Efficient system implementation: System-level methods such as batching and caching can improve throughput, but their output quality often drops below a PSNR of 22.The passage contrasts this with SVG preserving PSNR above 30.
- Sparse attention in LLMs: Sparse attention methods developed for LLMs identify locality or heavy-hitter token patterns, but video diffusion models require video-specific spatial and temporal sparsity.SVG's approach is built around these two video attention patterns.
3. Motivation and Analysis
Video DiT attention contains spatial and temporal sparsity patterns that can reduce computation while preserving accuracy. However, online pattern selection and hardware implementation are necessary because patterns vary and temporal sparsity is noncontiguous.
- Spatial Head: Spatial Heads primarily attend to tokens within the same or adjacent frames, producing a block-wise attention layout for spatial consistency.The block size corresponds to the number of tokens per frame.
- Temporal Head: Temporal Heads attend to tokens at the same spatial position across frames, producing a slash-wise layout that supports temporal consistency.With L tokens per frame, corresponding positions are separated by a stride of L.
- Shared tokens: SVG includes text prompts and first-frame tokens in both sparse patterns because they receive significant attention from both head types.This observation aligns with prior investigations cited by the paper.
- Pattern identification: SVG identifies each head’s pattern by sampling input rows, computing full, spatial, and temporal attention, and selecting the sparse result with lower MSE.The implementation samples a subset of queries and compares each sparse output with full attention.
- Computation savings: 31% sparsity for both spatial and temporal heads is reported on CogVideoX-v1.5-T2V while maintaining an average of 29.99 PSNR.The paper uses this result as evidence that sparse attention can reduce computation without degrading the reported quality metric.
- Practical constraint: Temporal sparsity is theoretically beneficial but cannot directly translate into real speedup because its pattern is hardware-inefficient.SVG therefore requires a practical hardware-oriented solution beyond the sparsity pattern itself.
4. Methodology
SVG is a training-free framework that exploits dynamic spatial and temporal sparsity in 3D full attention, combining online profiling with hardware-aware inference optimizations.
- Online profiling: Online profiling samples input rows and selects the spatial or temporal pattern with lower MSE against full attention.Profiling 1% of tokens achieves up to 31.1 PSNR with only 3% runtime overhead.
- Hardware-efficient layout: Temporal sparsity is initially non-contiguous, limiting Tensor Core utilization because hardware accelerators require contiguous elements.The temporal layout has stride L, the number of tokens per frame.
- Hardware-efficient layout: SVG transposes token-major tensors into frame-major tensors so tokens across frames become contiguous for block sparse attention.The transformation preserves mathematically equivalent attention outputs.
- System implementation: Customized CUDA, Triton, and FlashInfer kernels optimize QK-norm, RoPE, profiling, layout transformation, and sparse attention.FP8 quantization is also incorporated into sparse attention for additional efficiency.
5. Experiment
Experiments evaluate SVG on open 720p video diffusion models using quality, efficiency, visualization, and sensitivity analyses. SVG improves speed while preserving or surpassing baseline generation quality.
- Setup: SVG is evaluated on CogVideoX-v1.5-I2V, CogVideoX-v1.5-T2V, and HunyuanVideo-T2V at 720p resolution.CogVideoX uses 11 frames with 4080 tokens per frame, while HunyuanVideo uses 33 frames with 3600 tokens per frame.
- Setup: Quality evaluation uses PSNR, LPIPS, SSIM, and VBench Score to assess similarity and video quality.The benchmarks use VBench-based prompts and the Penguin Video Benchmark prompt for HunyuanVideo.
- Quality evaluation: SVG consistently outperforms all baselines in PSNR, SSIM, and LPIPS while achieving higher end-to-end speedup.Compared baselines include DiTFastAttn, MInference, Temporal-only attention, and PAB.
- Quality evaluation: 29.55 average PSNR on HunyuanVideo and 29.99 on CogVideoX-v1.5-T2V demonstrate high fidelity in SVG outputs.Figure 6 reports high visual consistency with original implementations across scene changes and object interactions.
- Quality evaluation: SVG adaptively maintains spatial and temporal consistency, whereas MInference and PAB substantially reduce quality through inadequate temporal modeling or result reuse.The comparison attributes MInference’s weakness to mean-pooling block sparsity and PAB’s to skipping 3D full-attention computation.
- Efficiency evaluation: 2.28× average end-to-end speedup is achieved while maintaining the highest generation quality, with sparse attention contributing 1.81×.For HunyuanVideo, runtime falls from 2253 seconds to 968 seconds, a total 2.33× speedup.
- Efficiency evaluation: Customized QK-norm and RoPE kernels provide average speedups of 7.4× and 15.5×, respectively.The customized sparse-attention kernel is evaluated against theoretical speedups across sparsity levels.
6. Conclusion
SVG accelerates video diffusion transformers by exploiting inherent spatial and temporal attention sparsity. Its training-free profiling and efficient inference system deliver speedups on representative models without losing quality.
- Conclusion: SVG classifies attention heads into spatial and temporal patterns and uses them for end-to-end efficiency improvements.The framework combines online profiling with an efficient inference system.
- Conclusion: SVG demonstrates prominent end-to-end speedup without losing quality on CogVideoX-v1.5 and HunyuanVideo.These results are reported on representative open video diffusion transformers.
Impact Statement
The paper identifies no societal consequences requiring specific emphasis.
- Impact Statement: The authors state that their work may have potential societal consequences but do not identify any that require specific highlighting.
A.1. Efficient Diffusion Models
Efficient diffusion-model research targets fewer denoising steps, smaller models, and more efficient system implementations. However, many approaches require retraining or fine-tuning, while the cited comparison reports lower PSNR for prior strategies than for the proposed method.
- Diffusion-model efficiency efforts primarily reduce denoising steps, compress model size, or optimize system implementation.
- Decreasing the denoising steps: Many diffusion models require numerous sampling steps because they learn data-distribution gradients through stochastic differential equations.
- Video-generation speedups that require retraining or fine-tuning are often impractical because of the computational expense.
- Diffusion Model Compression: Quantization improves diffusion-model efficiency by compressing model weights, including W8A8 and timestep-aware W4A8 strategies.
- Efficient System Implementation: System-level methods use dynamic batching, caching, or hybrid strategies to improve diffusion-model throughput.
- PSNR above 30 is reported for the proposed method, compared with below 22 for previous approaches, indicating higher output fidelity.
A.2. Efficient Attention
Sparse-attention research reduces attention cost by focusing computation on selected token subsets, while linear and low-bit attention address efficiency through different mechanisms. Sparse VideoGen is described as orthogonal to both and compatible with FP8 attention.
- Sparse Attention in LLM: Sparse-attention studies in language models identify concentration on initial, local, or dominant tokens to reduce computation.
- Linear and Low-bit Attention: Linear attention reduces traditional attention’s quadratic complexity to linear complexity.
- Sparse VideoGen is orthogonal to linear and low-bit attention and can integrate with FP8 attention for additional efficiency.
B. Visualization of the generated videos
The visualization comparison covers Dense Attention and Sparse VideoGen on HunyuanVideo and Wan 2.1 for 720p text-to-video and image-to-video generation. The reported comparison indicates similar quality and high pixel-level fidelity for Sparse VideoGen.
- The visualization study compares Dense Attention and Sparse VideoGen on HunyuanVideo and Wan 2.1 at 720p resolution.
- The comparison includes both text-to-video and image-to-video generation settings.
- Sparse VideoGen preserves high pixel-level fidelity with generation quality similar to dense attention.