PaperlayerView exact revision on arXiv

Source-linked AI summary

Radial Attention: $O(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation

Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujun Lin, Lvmin Zhang, Songlin Yang, Jinbo Hu, Kelly Peng, Maneesh Agrawala, Ion Stoica, Kurt Keutzer, Song Han

arXiv:2506.19852v2cs.CVcs.AIcs.LG

TL;DR

Long-video generation is computationally costly because video tokens grow with time while self-attention scales quadratically. Radial Attention uses spatiotemporal energy decay to sparsify attention, preserving quality for videos up to 4× longer while reducing training costs by up to 4.4× and accelerating inference by up to 3.7× with minimal LoRA fine-tuning.

  • Problem

    The growth in video token counts and quadratic self-attention make training and inference on long videos computationally prohibitive.

    Source: §1 · p. 2

  • Method

    Radial Attention models decreasing attention scores with spatiotemporal distance and uses a static sparse mask with O(n log n) complexity, alongside LoRA fine-tuning for longer sequences.

    Source: §1 · p. 2

  • Results

    For videos up to 4× longer, Radial Attention preserves quality while delivering up to 4.4× training and 3.7× inference speedups with minimal LoRA fine-tuning.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Radial Attention contributes toward scalable, high-quality video generation and offers a foundation for efficient long-range attention in broader sequence modeling tasks.

    Source: §6 · p. 10

    The exponential-decay assumption simplifies complex spatiotemporal dependencies, and the method remains quadratic in resolution.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporal Energy Decay in video diffusion models: post-softmax attention scores diminish as spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, we propose Radial Attention, a scalable sparse attention mechanism with $\mathcal{O}(n \log n)$ complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard $\mathcal{O}(n^2)$ dense attention and more expressive than linear attention. Specifically, Radial Attention employs a simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient LoRA-based fine-tuning. Extensive experiments show that Radial Attention maintains video quality across Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9$\times$ speedup over the original dense attention. With minimal tuning, it enables video generation up to 4$\times$ longer while reducing training costs by up to 4.4$\times$ compared to direct fine-tuning and accelerating inference by up to 3.7$\times$ compared to dense attention inference. Code is released at \href{https://github.com/mit-han-lab/radial-attention}{https://github.com/mit-han-lab/radial-attention}.

1 Introduction

Radial Attention addresses the prohibitive cost of long-video self-attention with a static sparse mask based on spatiotemporal energy decay. It also enables longer-video adaptation with lightweight fine-tuning while maintaining video quality.

1 Introduction

  • Video’s larger token counts and quadratic self-attention scaling make training and inference on long videos computationally prohibitive, restricting model scalability.Source: §1 · p. 2
  • At default length, Radial Attention achieves up to 1.9× speedup on Wan2.1-14B and HunyuanVideo; at 4× longer length, it reduces tuning costs by up to 4.4× and accelerates inference by up to 3.7× without sacrificing quality.For a 509-frame 720p HunyuanVideo, it reduces attention computation by 9×.Source: Figure 2 · p. 2, §1 · p. 2
  • Prior approaches either profile heads to choose spatial or temporal sparse masks, risking misclassification on unseen longer-video distributions, or replace softmax with linear attention that may require substantial architectural changes.The passage notes that modest fine-tuning is typically insufficient to recover original video quality with linear alternatives.Source: §1 · p. 2
  • Radial Attention models post-softmax attention-score decay with spatial and temporal distance, using a static mask whose spatial window shrinks exponentially with temporal distance for O(n log n) complexity.Tokens attend to others at similar spatial locations, translating energy decay into computation-density decay while retaining the softmax attention mechanism.Source: §1 · p. 2
  • Because it prunes token relations without modifying softmax attention, Radial Attention supports longer-sequence adaptation of pretrained video diffusion models with lightweight fine-tuning such as LoRA.The length-extension LoRA is compatible with existing style LoRAs.Source: §1 · p. 2

2 Related Work

Prior work spans dense video attention, general acceleration techniques, and sparse methods, but the paper argues these approaches leave challenges in efficient long-video generation. Radial Attention is positioned as a video-structured, static O(n log n) alternative that supports longer generation with LoRA fine-tuning.

2 Related Work

  • Video diffusion progressed from temporal-module 2D UNets to DiT backbones, with recent models adopting costly 3D dense attention for joint spatiotemporal modeling [14] [15] [16] [18] [20] [21] [5] [1] [7] [6].Latte [19] introduced decoupled spatial and temporal attention; dense attention’s cost grows quadratically with frame count.Source: §2 · p. 3
  • Video acceleration also uses timestep distillation, caching, quantization, and distributed inference [23] [24] [25] [26] [27] [28] [30] [31] [32], while attention-quantization kernels target inference speed [33] [34] [35] [36] [37].Because video models rely on 3D dense attention, attention layers can become the computational bottleneck.Source: §2 · p. 3
  • LLM sparse patterns such as Long LoRA [39] and PowerAttention [45] disregard video structure, while STA [47] and SVG [8] face fixed-receptive-field and unseen-distribution limits.Radial Attention instead uses a static O(n log n) pattern across heads to accelerate training and inference and extend generation length.Source: §2 · p. 3
  • Long-video alternatives include RIFLEx’s 2× RoPE extrapolation [50], segment stitching [52], autoregressive Framepack [53], and linear attention [10] [9] [54] [55] [56] [57] [58], which can degrade local detail.RIFLEx can produce blurring at longer lengths; the paper positions Radial Attention between O(n^2) dense and O(n) linear attention and says LoRA enables longer-video fine-tuning with minimal overhead.Source: §2 · p. 3
  • Prior O(n log n) methods use hashing, hierarchical attention, refinement, multipole methods, or logarithmic time-series sparsity, whereas Radial uses a block-friendly static mask.The passage characterizes these earlier methods as often hardware-unfriendly and limited in scalability.Source: §2 · pp. 3–4

3 Preliminary

Video diffusion operates on large latent token sequences, making dense attention costly for long or high-resolution videos. Sparse attention reduces interactions, but existing static and dynamic schemes trade expressiveness against overhead and training applicability.

3 Preliminary

  • Generating a 5-second 720p HunyuanVideo [1] video requires approximately 110K latent tokens, while excessive latent compression degrades video quality [3].Video latents are progressively denoised from Gaussian noise and decoded into the final video.Source: §3 · p. 4
  • Dense attention computes all token-pair interactions at O(n^2) time and memory, which remains a bottleneck for long or high-resolution videos despite FlashAttention reducing memory overhead.The temporal dimension substantially increases video token counts.Source: §3 · p. 4
  • Sparse attention masks token interactions, but static patterns such as STA [47] are less expressive, while dynamic SVG [8] incurs online decision overhead and does not apply to training.The paper motivates a static pattern that can match dynamic methods’ expressiveness and support training.Source: §3 · p. 4

4 Method

Radial Attention converts spatiotemporal attention-score decay into a sparse compute pattern, with O(n log n) complexity and LoRA-based fine-tuning for extending pre-trained models to longer videos.

4.1 Spatiotemporal Energy Decay in Attention

  • Attention scores decay with spatial and temporal distance, with spatial attention showing stronger temporal decay and temporal attention showing stronger spatial decay.The authors call this Spatiotemporal Energy Decay; regression analysis indicates the decay closely follows an exponential distribution.Source: §4.1 · pp. 4–5, Figure 4 · p. 5

4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay

  • Radial Attention uses full compute density in the central temporal band, halves density in each successive band, and narrows spatial diagonals as frame distance grows.When diagonal width falls below one, the method reduces diagonal frequency; its final mask also incorporates an attention sink.Source: §4.2 · p. 5, Figure 5 · p. 6
  • Radial Attention unifies spatial and temporal attention in SVG [8], reducing computation for distant, lower-relevance frames and reallocating it toward nearer frames.Its central band captures dense spatial interactions, while SVG [8] does not account for temporal decay.Source: §4.2 · p. 6
  • At fixed resolution, complexity scales as O(n log n), cutting attention computation 9× versus dense attention for 4× longer videos; the attention-error bound falls exponentially as decay rates α and β increase.Source: §4.2 · pp. 6–7
  • At default length and equal compute budget, Radial Attention beats STA and PA in PSNR, SSIM, and LPIPS, matches SVG fidelity, and achieves 1.8× speedup on HunyuanVideo and Wan2.1-14B on one H100 GPU.Source: Table 1 · p. 7

4.3 Low-Rank Adaptation for Long Videos

  • LoRA fine-tuning of query, key, value, and output projections enables efficient long-video adaptation while keeping pre-trained weights largely intact.Radial Attention reduces training complexity to O(n log n), allowing minimal fine-tuning with lower memory and compute overhead.Source: §4.3 · p. 7

5 Experiments

Experiments evaluate Radial Attention on three text-to-video models at default and extended lengths, finding quality retention alongside inference and training savings. Ablations assess its attention fidelity, exponential-decay fit, and compatibility with LoRA adaptation.

5.1 Setups

  • Evaluations cover Mochi 1, HunyuanVideo, and Wan2.1, using Vision Reward, default-length PSNR, SSIM, and LPIPS, plus VBench-long for extended videos.VBench-long measures subject consistency, aesthetic quality, and imaging quality.Source: §5.1 · p. 7

5.2 Main Results

  • Up to 4.4× training-cost savings and 3.7× inference speedup accompany quality on par with LoRA-fine-tuned dense attention at extended video lengths.Original models degrade notably at longer lengths; Wan2.1-14B is reported only at 2× extrapolation because of its higher computational and memory costs.Source: Table 2 · p. 8, §5.2 · pp. 9–10
  • On a single H100, Radial Attention achieves 1.9× end-to-end speedup on HunyuanVideo and 1.8× on Wan 2.1 while preserving dense-attention quality and matching SVG.It consistently outperforms STA and PA on PSNR, SSIM, and LPIPS under the same compute budget.Source: §5.2 · p. 9

5.3 Ablation Study & Analyses

  • The exponential decay fit achieves R2 > 0.985 on average attention decay curves.The fitted model is y = exp(−ax + b).Source: Figure 8 · p. 10, §5.3 · p. 10

5.2 Main Results

  • Radial Attention remains compatible with pre-trained task-specific LoRAs, including style LoRAs merged for both default-length and longer-video generation.The paper reports compatibility by directly merging the Radial Attention long-video LoRA weights with style LoRAs.Source: §5.2 · p. 10

5.3 Ablation Study & Analyses

  • With Radial Attention, LoRA fine-tuning matches or outperforms full fine-tuning in Vision Reward as video length increases, unlike dense-attention LoRA before 4× extension.Dense-attention LoRA lags behind full fine-tuning until 4× length extension.Source: §5.3 · p. 10
  • Radial Attention’s average attention-output MSE on Wan2.1-14B is 3.9 × 10−3, versus 4.4 × 10−3 for SVG and 1.5 × 10−2 for STA.Lower MSE indicates better preservation of attention fidelity.Source: §5.3 · p. 10

6 Conclusion & Discussion

Radial Attention uses Spatiotemporal Energy Decay to enable efficient long-video generation while preserving quality. Its limitations include a simplifying decay assumption and quadratic complexity with respect to resolution.

6 Conclusion & Discussion

  • At default video length, Radial Attention achieves up to a 1.9× speedup with high fidelity; for videos up to 4× longer, minimal LoRA fine-tuning preserves quality and delivers up to 4.4× and 3.7× speedups in training and inference, respectively.The comparison is against the original dense attention at default length and dense attention inference for longer videos.Source: §6 · p. 10
  • Radial Attention is an O(n log n) sparse attention method motivated by Spatiotemporal Energy Decay in video diffusion models.The paper presents it as a unified attention pattern with sub-quadratic complexity for efficient long-video generation.Source: §6 · p. 10
  • The exponential-decay assumption simplifies natural-video dependencies, and the method remains quadratic with respect to resolution, limiting support for long, high-resolution videos.The paper identifies deeper modeling of data structure, more efficient attention, and pre-training strategies as future work.Source: §6 · p. 10

Changelog

The changelog records an initial preprint release in V1, followed by a V2 NeurIPS 2025 camera-ready version that compresses the paper to 10 pages and rearranges its sections.

  • V1 was the initial preprint release; V2 was the NeurIPS 2025 camera-ready version, compressing the paper to 10 pages and rearranging its sections.Source: Changelog · p. 11

A Derivations and Proofs … Notation

The appendix derives an O(n log n) upper bound for Radial Attention’s mask and formulates an asymptotic approximation-error bound under spatial and temporal attention-decay assumptions. It also defines the masked-softmax error and the geometry that determines which tokens are retained.

A.1 Complexity

  • Dense attention covers the central band and attention sink, whose f sink blocks and at most 3f central-band blocks bound the region’s nonzero entries.The attention sink lets every token attend to all tokens in the first frame.Source: §A.1 · p. 16
  • For bands with diagonal width at least 1, the mask uses a temporal-offset-dependent spatial condition to determine allowed attention.These bands exclude the central band.Source: §A.1 · p. 16
  • For bands with diagonal width below 1, Radial Attention reduces diagonal frequency; the derivation bounds the number of bands meeting this condition.The condition is stated as s 2⌊log2 max(|i−j|,1)⌋ < 1.Source: §A.1 · p. 16
  • The mask has at most 4s²f⌊log2 f⌋ ≤ 4sn(log2 n − log2 s) zeros, giving O(n log n) scaling with frame count f.The derivation combines bounds for the central region and the other diagonal bands.Source: §A.1 · p. 17

A.2 Error Bound

  • The error analysis bounds a single attention-matrix row and uses relative exponential decay, with α and β representing temporal and spatial decay rates.It extends the temporal index to an infinite grid while keeping the spatial grid finite.Source: §A.2 · p. 17, Assumptions · p. 17

Assumptions

  • Under the stated decay assumption, the analysis relates approximation error to spatial and temporal attention decay rates.The error bound is framed as an asymptotic bound for Radial Attention relative to that decay.Source: §A.2 · p. 17, Assumptions · p. 17, Notation · p. 18

Notation

  • The total-variation comparison uses exact and masked softmax rows; its error includes spatial tails in kept frames and frames skipped by subsampling.The mask retains spatial indices within a temporal-offset-dependent bandwidth for selected frames.Source: Notation · pp. 17–18

B Additional Implementation Details … C.2 Longer-video Length

The appendix reports that Radial Attention preserves video quality in default- and longer-video settings, and gives LoRA fine-tuning and evaluation details. It also compares Radial Attention with Harmonic Series and tests warmup length.

B Additional Implementation Details

  • LoRA fine-tuning used 8 H100 GPUs, rank 128, and dense attention in the first two DiT blocks.Global batch size was 1 with sequence parallelism for HunyuanVideo and Mochi 1, and 8 for Wan2.1; training steps varied by model and target length.Source: B Additional Implementation Details · p. 18

C Visualization of the generated videos

C.1 Default Video Length

  • At 768p and 117 frames for HunyuanVideo and 768p and 69 frames for Wan2.1-14B, Radial Attention has higher PSNR than STA and maintains original-video fidelity.The comparison is shown in Figures A and B.Source: §C.1 · p. 18

C.2 Longer-video Length

  • Radial Attention consistently outperforms Harmonic Series, another O(n log n) attention baseline, across all metrics.The comparison is reported in Table A.Source: §C.2 · p. 19
  • For default-length Wan2.1-14B generation, a 12-step full-attention warmup achieves the best performance across all metrics.The warmup-step ablation is reported in Table B.Source: §C.2 · p. 19
  • At 4× length for HunyuanVideo and Mochi 1 and 2× for Wan2.1-14B, Radial Attention achieves the highest average Vision Reward against baselines while preserving video quality.The comparisons use 21-second, 509-frame HunyuanVideo videos, 22-second, 667-frame Mochi 1 videos, and 10-second, 161-frame Wan2.1-14B videos at default resolutions.Source: §C.2 · pp. 18–19

C.3 LoRA Compatibility Visual Results · D Ablations on Initial Dense-Attention Layers and Steps

The extended-length LoRA can be combined with style LoRAs to preserve visual quality while generating longer videos, though small-dataset training may cause subtle style differences. The ablations examine dense-attention settings under a fixed computation budget and the O(n log n) mask design.

C.3 LoRA Compatibility Visual Results

  • Combining the extended-length LoRA with existing style LoRAs preserves visual quality while enabling longer-video generation.Figure F illustrates the compatibility results.Source: §C.3 · p. 19
  • The merged LoRA’s generated content style differs subtly from that of the original LoRAs.The passage attributes this discrepancy primarily to the relatively small extended-length LoRA training dataset, whose slight style bias may interact with the style LoRA.Source: §C.3 · p. 19
  • Training the length-extension LoRA on a more comprehensive dataset is expected to help mitigate the style discrepancy.This is presented as an expectation, not a reported experimental result.Source: §C.3 · p. 19

D Ablations on Initial Dense-Attention Layers and Steps

D.1 Ablation on Initial Dense-Attention Steps … E Broader Impacts

The ablations identify effective dense-attention warmups and layers, while comparing Radial Attention’s sparsity pattern with Harmonic Series Decay Attention; the broader impacts section describes potential benefits and risks.

D.1 Ablation on Initial Dense-Attention Steps

  • With computation matched, 12 dense-attention warmup steps (the first 25% of timesteps) achieve the best video quality across all metrics for default-length generation.This warmup is applied to all methods.Source: §D.1 · p. 19
  • For 4× longer HunyuanVideo generation, 2 inference warmup steps achieve the highest Vision Reward under matched computation.The warmup uses full attention for the first 2 steps.Source: §D.1 · p. 19, Table C · p. 20

D.2 Ablation on Initial Dense-Attention Layers

  • Under the same computation budget, using 2 initial full-attention layers during training delivers the best video quality and highest Vision Reward.The ablation compares 0, 1, 2, or 3 dense layers on HunyuanVideo.Source: §D.2 · p. 19, Table D · p. 20

D.3 Comparison with Other O(n log n) Sparsity Patterns

  • The sparsity-pattern ablation reports that Radial Attention is superior to Harmonic Series Decay Attention, whose diagonal width decreases inversely with distance from the main diagonal.The comparison evaluates the two O(n log n) attention masks.Source: §D.3 · p. 20

E Broader Impacts

  • Radial Attention’s reduced computational costs and O(n log n) self-attention may enable quality video creation with less energy use, including for education and creative arts.The paper frames these as potential benefits for greener AI applications and wider access to generative models.Source: E Broader Impacts · p. 20
  • The paper warns of deepfake misuse, misinformation, and a wider digital divide, and advocates responsible deployment, ethical standards, detection methods, safeguards, and proper code and model licenses.Source: E Broader Impacts · p. 20

F License

The appendix lists licenses for used assets and reports Radial Attention comparisons at extended video lengths, including compatibility with existing style LoRAs.

  • Wan 2.1 [7], Mochi 1, Diffusers, and STA [47] use Apache-2.0; SVG [8] and OpenVid-1M have no explicit license, and the HunyuanVideo [1] license is referenced but not specified here.Source: F License · p. 20
  • At 4× default length, Radial Attention achieves the best Vision Reward on HunyuanVideo and Mochi 1, while the compared methods show blurred, distorted, or inconsistent videos.The comparisons are at 21 seconds and 509 frames for HunyuanVideo, and 22 seconds and 667 frames for Mochi 1.Source: Figure C · p. 23, Figure D · p. 24
  • At 2× default length, Radial Attention LoRA achieves the highest Vision Reward among Wan2.1-14B baselines; one reported run has VisionReward 0.165, Latency 2847s, Inference Speedup 2.0×, and Training Speedup 1.9×.Figure E compares methods at 10 seconds and 161 frames.Source: F License · p. 25
  • Radial Attention LoRA is compatible with existing style LoRAs on HunyuanVideo, extending video length by 4× while maintaining vision reward comparable to an original-length LoRA video.Source: F License · p. 25

Related papers

Loading 2506.19852v2…