PaperlayerView exact revision on arXiv

Source-linked AI summary

Radial Attention: $O(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation

Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, Song Han

arXiv:2506.19852v2cs.CVcs.AIcs.LG

TL;DR

Long-video diffusion is limited by the token growth and quadratic cost of dense attention. The paper introduces Radial Attention, a static sparse pattern based on spatiotemporal energy decay, and combines it with lightweight LoRA fine-tuning for longer sequences. It preserves video quality while delivering speed and tuning-cost reductions across default and extended video lengths.

  • Problem

    Long-video diffusion training and inference are computationally prohibitive because video token counts increase with temporal extent and dense attention scales quadratically.

  • Method

    Radial Attention models spatiotemporal energy decay with a static sparse mask of O(n log n) complexity and supports longer-video adaptation through LoRA fine-tuning.

  • Results

    Radial Attention preserves video quality and achieves up to 1.9× speedup at default lengths, while 4× longer videos receive up to 4.4× training-cost reduction and 3.7× inference speedup.

  • Takeaways & Limitations

    The method provides a scalable approach to high-quality long-video generation using sub-quadratic attention and minimal LoRA fine-tuning.

  • Takeaways & Limitations

    The exponential-decay assumption simplifies spatiotemporal dependencies, and the method remains quadratic with respect to resolution.

Abstract

from arXiv · show

Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporal Energy Decay in video diffusion models: post-softmax attention scores diminish as spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, we propose Radial Attention, a scalable sparse attention mechanism with $\mathcal{O}(n \log n)$ complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard $\mathcal{O}(n^2)$ dense attention and more expressive than linear attention. Specifically, Radial Attention employs a simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient LoRA-based fine-tuning. Extensive experiments show that Radial Attention maintains video quality across Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9$\times$ speedup over the original dense attention. With minimal tuning, it enables video generation up to 4$\times$ longer while reducing training costs by up to 4.4$\times$ compared to direct fine-tuning and accelerating inference by up to 3.7$\times$ compared to dense attention inference. Code is released at \href{https://github.com/mit-han-lab/radial-attention}{https://github.com/mit-han-lab/radial-attention}.

1 Introduction

Long-video diffusion is constrained by quadratic attention costs. Radial Attention uses observed spatiotemporal energy decay to create a static, sub-quadratic pattern that supports efficient longer-video adaptation.

  • Long-video training and inference are computationally prohibitive because video token counts grow with time and self-attention scales quadratically.
  • Spatiotemporal Energy Decay describes post-softmax attention scores diminishing as spatial or temporal token distance increases.The paper models this decay exponentially in both distances.
  • Radial Attention uses a static mask that unifies spatial and temporal attention while reducing attention complexity to O(n log n).Its compute density decays with the modeled energy decay.
  • Radial Attention prunes token relations without changing softmax attention, enabling lightweight LoRA adaptation of pre-trained models to longer sequences.The length-extension LoRA remains compatible with existing style LoRAs.
  • 1.9× speedup is achieved at default video lengths, while 4× longer videos receive up to 4.4× lower tuning costs and 3.7× faster inference without sacrificing quality.

2 Related Work

Related work accelerates video generation through model, hardware, and attention-pattern changes, but long-video extension and scalable sparse attention remain difficult. Radial Attention is positioned as a simpler, block-friendly alternative to several less scalable or less expressive approaches.

  • Video diffusion models: Video diffusion evolved from temporally augmented 2D UNets toward DiT backbones and decoupled spatial-temporal attention.
  • Efficient video generation: Image-diffusion acceleration methods such as distillation, caching, quantization, and distributed inference also apply to video, where 3D dense attention remains a bottleneck.
  • Long video generation: RIFLEx enables 2× length extrapolation through RoPE-frequency modification, but longer videos can suffer quality degradation such as blurring.
  • Attention with O(n log n) complexity: Earlier O(n log n) attention methods include hashing, hierarchical, multiresolution, multipole, and logarithmic-sparsity designs, but are often hardware-unfriendly or poorly scalable.
  • Attention with O(n log n) complexity: Radial Attention differs by using a simple, block-friendly static attention design.

3 Preliminary

Video diffusion uses dense 3D attention to model spatiotemporal interactions, but its quadratic cost becomes prohibitive as token counts grow. Sparse attention reduces this burden by masking selected token pairs, with a remaining challenge of combining expressiveness, training applicability, and efficiency.

  • A 5-second 720p HunyuanVideo generation requires approximately 110K latent tokens, reflecting the cost of video’s additional temporal dimension.
  • 3D dense attention computes interactions between all token pairs, while the QK^T operation has O(n^2) time and memory complexity.
  • FlashAttention reduces memory overhead but does not remove quadratic time scaling, leaving long or high-resolution video attention as a bottleneck.
  • Sparse attention restricts interactions by adding a mask M with entries −∞ or 0 to the attention logits.Entries set to −∞ are ignored by softmax.
  • Static masks are less expressive, whereas dynamic masks improve fidelity but add online decision overhead and do not apply to training.

4 Method

Radial Attention observes that post-softmax attention scores decay with spatial and temporal distance, then converts this pattern into a static sparse mask with exponentially decreasing compute density. The resulting method has O(n log n) complexity, preserves important local interactions, and supports efficient long-video adaptation.

  • 4.1 Spatiotemporal Energy Decay in Attention: Spatiotemporal Energy Decay describes attention scores diminishing as spatial or temporal distance increases, with regression suggesting an exponential pattern.Spatial and temporal attention exhibit different relative decay rates, motivating a unified treatment.
  • 4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay: Temporal compute density halves across successive bands, while band widths double so each band contributes bounded total computation.The central band retains full density; increasingly distant bands use lower density and broader diagonal structure.
  • 4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay: Radial Attention translates energy decay into compute-density decay using one static mask that preserves nearby spatial interactions while sparsifying distant frame interactions.The mask reduces diagonal widths with temporal distance and adds an attention sink for quality.
  • 4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay: Radial Attention scales as O(n log n) for long videos with fixed resolution and reduces attention computation by 9× for 4× longer videos versus dense attention.The method computes attention over hardware-friendly 128 × 128 blocks.
  • 4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay: At default video lengths, Radial Attention matches SVG’s video fidelity, outperforms STA and PA on PSNR, SSIM, and LPIPS, and achieves 1.8× speedup on HunyuanVideo and Wan2.1-14B.The comparison uses the same computation budget and a single H100 GPU.
  • 4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay: Its masked attention error decreases exponentially with larger spatial and temporal decay rates, and empirical comparisons report smaller errors than SVG.This analysis links the mask’s approximation quality to the observed decay parameters.

5 Experiments

Experiments evaluate Radial Attention on three video diffusion models using quality, similarity, efficiency, and long-video metrics. Across default and extended lengths, it preserves quality while reducing inference and tuning costs.

  • Setups: Experiments benchmark Radial Attention on Mochi 1, HunyuanVideo, and Wan2.1 using Vision Reward, PSNR, SSIM, LPIPS, and VBench-long.The models contain 10, 13, and 14 billion parameters, respectively.
  • Main Results: Under equal PFLOP budgets, Radial Attention preserves dense-attention video quality, outperforms STA and PA on similarity metrics, and matches SVG.The comparison uses PSNR, SSIM, and LPIPS on HunyuanVideo and Wan2.1 at default lengths.
  • Main Results: 1.9× and 1.8× end-to-end speedups are achieved for HunyuanVideo and Wan2.1, respectively, with 1.8× and 1.7× fewer PFLOPs.These measurements use a single H100 and the same system optimization as SVG.
  • Long Video Generation: At 4× video length, Radial Attention preserves quality with up to 4.4× lower tuning costs and 3.7× faster inference.The method reaches quality on par with LoRA fine-tuned dense attention, while untuned models degrade substantially at extended lengths.
  • Ablation Study & Analyses: Radial Attention remains compatible with existing style LoRAs and allows its long-video LoRA weights to be merged with them.Compatibility is reported for both default-length and longer-video settings.
  • Ablation Study & Analyses: With Radial Attention, LoRA fine-tuning matches or outperforms full fine-tuning as video length increases, unlike dense attention at shorter extensions.The comparison uses Vision Reward across increasing video lengths.
  • Ablation Study & Analyses: Radial Attention attains attention-output MSE 3.9 × 10^-3, below SVG’s 4.4 × 10^-3 and STA’s 1.5 × 10^-2.Lower MSE indicates closer preservation of the original attention output.
  • Ablation Study & Analyses: Exponential fits to average attention-decay curves achieve R2 above 0.985.This supports the use of exponential functions to model the observed decay.

6 Conclusion & Discussion

The paper proposes Radial Attention as an O(n log n) sparse mechanism for efficient long-video generation. It reports high fidelity at default length and preserved quality with substantial training and inference savings up to 4× longer videos, while noting resolution-related limitations.

  • Conclusion: Radial Attention models Spatiotemporal Energy Decay with a unified sparse attention pattern of O(n log n) complexity.The method is designed for efficient long-video generation.
  • Conclusion: At default video length, Radial Attention achieves up to 1.9× speedup with high fidelity.The conclusion summarizes this result across the paper’s evaluated models.
  • Limitations: The exponential-decay assumption simplifies complex spatiotemporal dependencies, and the method remains quadratic with respect to resolution.The paper identifies more efficient attention mechanisms and pre-training strategies for long, high-resolution videos as future work.

Changelog

The changelog records an initial preprint release followed by a NeurIPS 2025 camera-ready revision that compressed and rearranged the paper.

  • Changelog: V1 records the initial preprint release.
  • Changelog: V2 records the NeurIPS 2025 camera-ready version, with the paper compressed to 10 pages and sections rearranged.

A Derivations and Proofs

The appendix derives Radial Attention’s mask complexity and approximation behavior. Its geometry keeps dense local and sink regions, shrinks spatial bandwidth with temporal distance, and yields O(n log n) complexity under exponential-decay modeling.

  • A.1 Complexity: The complexity proof bounds the number of zeros in the attention mask because masked-attention cost is proportional to that count.The resulting upper bound scales as O(n log n).
  • A.1 Complexity: Dense attention is retained in the central band and attention sink, where the sink lets every token attend to the first frame.The central band covers neighboring frames.
  • A.1 Complexity: Outside the central band, the mask uses logarithmic bands with spatial diagonal widths that shrink with temporal distance.Separate regions handle diagonal widths at least one and below one.
  • A.1 Complexity: The aggregate mask bound is at most 4s²f·⌊log2 f⌋ and scales as O(n log n) with the number of frames.Here n = fs, with s tokens per frame and f frames.
  • A.2 Error Bound: The approximation analysis assumes exponential decay in temporal and spatial distance, with α and β representing temporal and spatial decay rates.The analysis also extends the temporal grid to integers while keeping the spatial grid finite.
  • A.2 Error Bound: The exact and masked softmax rows normalize retained and unretained attention separately, enabling a total-variation error analysis.The analysis characterizes how decay rates affect approximation error.
  • A.2 Error Bound: For temporal offset Δt, the mask bandwidth is w(Δt) = s 2^⌊log2 max(Δt,1)⌋, and retained tokens satisfy a spatial-distance condition.The resulting errors arise from spatial tails inside retained frames and frames skipped by subsampling.

B Additional Implementation Details

The experiments detail Radial Attention’s implementation settings and compare it with baselines across video quality, training, and inference efficiency. Results show strong quality preservation and consistent superiority over the Harmonic Series baseline.

  • The implementation details describe the evaluation settings and report comparisons across video quality, training, and inference efficiency.
  • Experiments compare Radial Attention with baselines on video quality, training speed, and inference speed across HunyuanVideo, Mochi 1, and Wan2.1-14B.The evaluations include default-length and extended-length generation settings.
  • Radial Attention achieves higher PSNR than STA on HunyuanVideo and Wan2.1-14B while maintaining the fidelity of original videos.The comparison uses 768p settings with 117 frames for HunyuanVideo and 69 frames for Wan2.1-14B.
  • Radial Attention consistently outperforms the O(n log n) Harmonic Series baseline across all reported metrics.

C.3 LoRA Compatibility Visual Results

Extended-length LoRA remains compatible with existing style LoRAs while preserving visual quality, although limited training data can introduce a slight style bias.

  • Combining the extended-length LoRA with existing style LoRAs preserves visual quality and enables longer-video generation.
  • A relatively small training dataset may cause subtle style differences when the extended-length LoRA is merged with a style LoRA.The paper attributes this discrepancy primarily to interaction between the length-extension LoRA’s slight style bias and the existing style LoRA.
  • The authors expect a more comprehensive training dataset to help mitigate the style discrepancy.

D.1 Ablation on Initial Dense-Attention Steps

Ablations identify two full-attention warmup steps for 4× longer inference and two initial dense layers during training as the strongest configurations under matched computation.

  • D.1 Ablation on Initial Dense-Attention Steps: 12 warmup steps achieve the best performance across all metrics for default-length Wan2.1-14B generation.All settings match overall computation by adjusting Radial Attention sparsity.
  • D.1 Ablation on Initial Dense-Attention Steps: Two warmup steps achieve the highest Vision Reward for 4× longer HunyuanVideo generation.The compared configurations use matched computation.
  • D.1 Ablation on Initial Dense-Attention Steps: Using two initial full-attention layers during training delivers the best video quality under the same computation budget.The ablation compares zero, one, two, and three dense layers on HunyuanVideo.
  • D.1 Ablation on Initial Dense-Attention Steps: Radial Attention’s sparsity pattern outperforms Harmonic Series Decay Attention in quantitative comparisons.Harmonic Series uses a diagonal width inversely proportional to distance from the main diagonal.
  • D.1 Ablation on Initial Dense-Attention Steps: The paper concludes that Radial Attention reduces computational costs and supports longer-video generation with minimal fine-tuning while maintaining quality.

F License

The supplied material lists licenses for several assets and includes detailed visual and quantitative comparisons across models, baselines, prompts, and generation lengths.

  • Wan 2.1, Mochi 1, Diffusers, and STA are licensed under Apache-2.0.
  • HunyuanVideo has a separate license, while SVG and OpenVid-1M have no explicit license.
  • For the reported Wan2.1-14B LoRA comparison, Radial Attention achieves VisionReward 0.165, 2.0× inference speedup, and 1.9× training speedup.
  • Figure E reports the highest Vision Reward for Radial Attention at 2× default length, while Original Wan2.1-14B is blurred and Dense Attention has inconsistent figures.
  • Figure F reports 4× length extension with a Vision Reward comparable to the original-length LoRA video and compatibility with existing style LoRAs.
Loading 2506.19852v2…