PaperlayerView exact revision on arXiv

Source-linked AI summary

SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer

Tongcheng Fang, Hanling Zhang, Ruiqi Xie, Zhuo Han, Xin Tao, Tianchen Zhao, Pengfei Wan, Wenbo Ding, Wanli Ouyang, Xuefei Ning, Yu Wang

arXiv:2601.16515v2cs.CV

TL;DR

Long video sequences make full attention costly, while sparse methods face a trade-off between attainable sparsity, quality, and tuning resources. SALAD addresses this by adding a lightweight linear-attention branch regulated by multi-level static-dynamic scaling. It reaches up to 90% sparsity and 1.52–2.03× inference speedup while preserving quality comparable to full attention.

  • Problem

    Full attention is costly for long video sequences, while training-free sparse attention reaches limited sparsity and training-based methods require substantial data and computation.

  • Method

    SALAD parallels sparse attention with a lightweight linear-attention branch and regulates its contribution using Multi-level Static-Dynamic Scaling Strategy.

  • Results

    Up to 90% sparsity and 1.52–2.03× inference speedup are achieved across models and sequence lengths while maintaining generation quality comparable to full attention.

  • Takeaways & Limitations

    SALAD provides high-sparsity attention with reported quality comparable to full attention and efficient tuning using 2,000 video samples and fewer than 1,600 steps.

  • Takeaways & Limitations

    SALAD’s acceleration depends on kernel efficiency, and the released VMoBA kernel limits the observed speedup of Top-K sparse attention.

Abstract

from arXiv · show

Diffusion Transformers have demonstrated remarkable performance in video generation. However, their long input sequences incur substantial latency due to the quadratic complexity of full attention. Various sparse attention mechanisms have been proposed. Training-free approaches are limited to moderate sparsity and thus yield only modest acceleration, whereas training-based methods can reach much higher sparsity but demand substantial data and computation. In this work, we propose SALAD, introducing a lightweight linear attention branch in parallel with the sparse attention. Leveraging a Multi-level Static-Dynamic Scaling Strategy to balance the two branches, our method attains up to 90% sparsity and 1.52-2.03x inference speedup across different models and sequence lengths, while maintaining generation quality comparable to the full attention baseline. Moreover, our finetuning process is highly efficient, requiring only 2,000 video samples, fewer than 1,600 training steps, and no more than 30 GPU hours with a batch size of 8.

1 Introduction

Long video sequences make full attention costly, while existing sparse approaches trade quality, sparsity, and training overhead. SALAD adds a controlled linear-attention branch to sparse attention, reaching high sparsity and speedup with quality comparable to full attention.

  • Motivation: Quadratic attention complexity makes high-resolution, long-duration video generation computationally expensive.Sparse attention reduces computation by restricting each query to selected keys and values.
  • Limitations of Existing Methods: 40%–60% sparsity is typical for training-free methods, whereas training-based methods reach 80%–95% but require substantial data and computation.VMoBA, for example, requires approximately 182 GPU hours on Koala-36M.
  • Limitations of Existing Methods: LoRA improves sparse-model quality but leaves artifacts and textual or temporal inconsistencies associated with restricted cross-token interactions.A LoRA-tuned model initially generates two dogs for a prompt describing one dog, which later merge.
  • SALAD: SALAD parallels sparse attention with a lightweight linear branch to restore neglected global token mixing at O(N) complexity.The sparse branch models most of the sequence while the linear branch supplements critical global information.
  • SALAD: SALAD uses Multi-level Static-Dynamic Scaling Strategy to regulate the linear branch across Transformer blocks, inputs, and diffusion timesteps.The method makes the linear branch auxiliary rather than allowing it to dominate the sparse-attention output.

2 Related Work

Video diffusion models increasingly process complete videos as unified token sequences with full attention, improving generation quality but increasing memory and computation. Related work also includes linear attention and LoRA-based recovery for compressed models.

  • Video Diffusion Models: Recent video diffusion models process entire videos as unified token sequences and apply full attention across all tokens.Earlier spatial-temporal factorization limited expressive capacity, while unified processing improved generation quality.
  • Video Diffusion Models: Linear attention reduces the cost of modeling long video sequences by transforming attention complexity from quadratic to linear.This addresses the computational burden of long token sequences.
  • LoRA for Model Compression Recovery: LoRA freezes pretrained weights and injects trainable low-rank matrices, supporting parameter-efficient adaptation and model-compression recovery.Prior extensions apply LoRA to additional components for long-context adaptation under local attention.
  • LoRA for Model Compression Recovery: The paper finds that LoRA does not fully recover ultra-sparse Video Diffusion Transformer performance and presents SALAD as an alternative tuning approach.This motivates combining sparse attention with a linear-attention branch.

3 Preliminary

The preliminary section describes full attention, masked sparse attention, and static or dynamic sparsity patterns. It also introduces sliding-window, spatial-temporal sliding-window, Top-K, and ReLU-based linear attention.

  • Attention: Full attention computes softmax(QK^T / sqrt(d_k))V, whose attention matrix requires O(N^2d) memory and computation.This quadratic cost becomes prohibitive for large sequence length N.
  • Sparse Attention: Sparse attention adds a mask M to the attention logits, restricting interactions to selected token pairs.The mask contains -∞ and 0 entries to suppress or retain query–key interactions.
  • Sparse Attention: Static sparse masks are predefined before inference, whereas dynamic masks are constructed during inference to adapt to input data.The paper explores static sliding-window attention and dynamic Top-K attention.
  • Sparse Attention Variants: Sliding-window attention uses sequence locality, with spatial-temporal reordering producing ST-SWA for video tokens.Its sparse mask can be derived through calibration on a small data subset.
  • Sparse Attention Variants: Top-K attention retains the k highest-importance keys for each query, while Top-P retains a variable number exceeding cumulative importance threshold p.The paper uses the VMoBA implementation for Top-K sparse attention.
  • Linear Attention: ReLU-based linear attention applies a feature map ϕ to queries and keys to obtain linear-complexity attention computation.ReLU is identified as a common choice for ϕ(·).

4 Methods

SALAD augments sparse attention with a parallel linear attention branch and regulates its contribution through multi-level static-dynamic scaling. The design preserves sparse attention as the primary path while using linear attention to restore global interactions and adaptively complement sparse outputs.

  • SALAD Architecture: SALAD adds a lightweight linear attention branch in parallel with sparse attention, sharing Q, K, and V and residually integrating the branches.The projection matrices are initialized from pretrained weights and fine-tuned with LoRA.
  • Multi-level Static-Dynamic Scaling: SALAD uses layer-wise static scaling through a zero-initialized projection and input- and timestep-aware dynamic scaling to control the linear branch.The dynamic module constrains scaling factors to s ∈ (0, 1) with a sigmoid and adds minimal inference overhead.
  • Rank Imbalance: Directly summing sparse and linear attention outputs is suboptimal because the lower-rank linear branch can suppress informative high-rank sparse-attention structure.The analysis therefore assigns sparse attention the primary sequence-modeling role and linear attention an auxiliary role.
  • SALAD Architecture: The linear branch provides global token interaction, whereas sparse attention limits each query to a subset of tokens.This complementary interaction pattern motivates combining the two attention mechanisms.
  • Multi-level Static-Dynamic Scaling: Moderately reducing the linear-branch scale can improve some dimensions, but excessively small values degrade performance and the best scale varies across inputs.These observations motivate dynamic rather than exclusively static control of the branch magnitude.
  • Scaling Effects: The combined output retains a rank comparable to sparse attention while the branches exhibit lower subspace overlap after scaling.This indicates that the linear branch complements rather than replaces the sparse branch’s high-rank structures.
  • Branch Dropping: Branches with very small scaling factors can be dynamically pruned using timestep-wise thresholds calibrated on a small validation prompt set.Dropping a small portion of such branches can slightly improve generation quality.

5 Experiments

SALAD is evaluated across models, sequence lengths, sparse-attention baselines, and scaling designs. It reaches high sparsity with competitive quality and speed while requiring lightweight tuning, and ablations identify dynamic scaling and the linear branch as important components.

  • Setup: Experiments cover Wan2.1-1.3B at 30k and 100k tokens and Wan2.1-14B at 480p, using VBench and Vision Reward metrics.Training-based methods use 2,000 Mixkit videos; SALAD uses batch size 8 and up to 1,600 steps in the reported 1.3B settings.
  • Main Results: 1.72× speedup at 30k tokens accompanies 90% overall sparsity and video quality comparable to, or surpassing, the dense model on VBench.SALAD consistently outperforms prior approaches across the four reported evaluation metrics in the 30k-token comparison.
  • Main Results: 2.03× speedup at 100k tokens is achieved while preserving performance comparable to the full-attention baseline.A preliminary Wan2.1-14B experiment reports about 1.59× speedup with performance close to full attention.
  • Limitations: Observed acceleration depends on the efficiency of the released CUDA kernels used by SALAD and its sparse-attention extensions.The released Top-K kernel based on VMoBA is not fully optimized, limiting its observed speedup.
  • Main Results: SALAD outperforms almost all tested LoRA variants on SC, BC, and IQ while maintaining comparable TC with fewer trainable parameters than rank-256 LoRA.Increasing LoRA parameters does not consistently improve video quality, with the best performance at rank 128.
  • Ablation Study: Dynamic scaling improves performance across all metrics by coordinating the sparse and linear branches, while constant scaling helps but remains inferior to adaptive values.The detached variant suggests the improvement primarily comes from dynamic scaling rather than gradient flow to the input.
  • Ablation Study: The sparse branch primarily restores core generation quality, while the linear branch adds complementary token interactions that improve fine details and coherence.Disabling the linear branch after training is compared against full SALAD and LoRA-only tuning, which can retain missing objects and duplicated subjects under high sparsity.

6 Conclusion

SALAD combines ultra-sparse attention with a lightweight linear attention branch to recover cross-token information lost under sparsity. Multi-level static-dynamic scaling balances the branches, producing high sparsity and faster inference while preserving generation quality comparable to full attention.

  • Conclusion: SALAD uses a parallel linear attention branch to complement sparse attention and capture low-rank global information missed by ultra-sparse attention.The method is designed for Diffusion Transformers and targets cross-token information loss.
  • Conclusion: The Multi-level Static-Dynamic Scaling Strategy regulates the balance between sparse and linear attention branches.The scaling strategy controls the auxiliary branch’s contribution across the model.
  • Conclusion: 90% sparsity and 1.52-2.03× inference speedup are achieved across models and sequence lengths while maintaining generation quality comparable to full attention.These are the reported overall performance ranges of SALAD.

A.1 Sparse Attention Implementation Details

The implementation aligns spatial or temporal locality with sliding-window attention through headwise token permutations. It also profiles window sizes and describes Top-K block selection, while scaling analysis examines output magnitude and effective rank.

  • Spatial-Temporal Sliding Window Attention: Video attention exhibits spatial locality within frames and temporal locality across neighboring frames, but flattening separates temporal neighbors in the sequence.Spatially adjacent tokens remain consecutive, whereas same-location tokens across frames are separated by Hh × Wh positions.
  • Spatial-Temporal Sliding Window Attention: A headwise token permutation groups temporal neighbors by spatial location so sliding-window attention can capture local temporal dependencies.The permutation realigns temporal locality with the attention map’s diagonal structure.
  • Spatial-Temporal Sliding Window Attention: Window sizes are selected greedily using eight VBench prompts, choosing the smallest window whose induced error stays below a predefined threshold.The implementation uses headwise permutation and sliding-window kernels from SVG and DiTFastAttnV2.
  • Top-K Block Sparse Attention: Top-K block sparse attention partitions tokens into blocks, scores query-to-block similarities using block means, and selects the top k blocks.This dynamic selection targets key and value blocks likely to have high attention scores.
  • Scaling Analysis: Figure 8 evaluates attention-output norms and effective rank at timestep t = 20 for Wan2.1-1.3B across Transformer blocks.The figure examines the effect of multi-level scaling on linear-attention regulation and rank restoring.

A.2 Detailed Analysis of Multi-level Static-Dynamic Scaling

SALAD’s scaling strategy regulates the linear branch to prevent rank erosion and encourage complementary information alongside sparse attention. Analysis shows that scaling suppresses the branch’s magnitude, restores effective rank, reduces subspace overlap, and enriches weaker spectral components.

  • Linear Attention Regulation: Naive branch summation causes rank erosion because a lower-rank linear output contributes a magnitude comparable to the sparse branch.The fused representation can have lower effective rank than sparse attention alone.
  • Linear Attention Regulation: The effective rank of the fused representation gradually recovers as static projection and dynamic scaling progressively suppress the linear branch.Static projection reduces its energy first, while input-timestep-aware scaling suppresses it further.
  • Linear Attention Regulation: Scaling increases the rank of the combined representation by regulating the linear branch’s contribution and mitigating its adverse impact on sparse outputs.Layer-wise static projection learns attenuation even without an explicit down-weighting constraint during training.
  • Subspace Decoupling: The scaling strategy reduces branch subspace overlap, encouraging sparse and linear attention to encode complementary rather than redundant information.Overlap is measured using the average squared cosine similarity between their top-k PCA subspaces.
  • Spectral Tail Revitalization: Scaled linear attention increases both leading and tail-end singular values, enriching weaker spectral components without substantially changing overall effective rank.Naive summation instead disproportionately amplifies the dominant singular value and reinforces redundant directions.
  • Spectral Tail Revitalization: Scaling limits spectral distortion by reducing the perturbation magnitude of the linear branch before aggregation.The paper interprets this through Weyl’s inequality, which bounds singular-value changes by the perturbation’s spectral norm.

A.3 Details of Post-training Branch Dropping

SALAD can prune linear-attention branches after tuning using timestep-aware scaling values. Their stable, largely prompt-independent distributions support lightweight calibration and reliable branch dropping with limited quality loss.

  • Scaling-Value Analysis: Dynamic scaling values reflect each block’s relative linear-branch contribution for a prompt and denoising timestep.They can be computed with negligible inference overhead and indicate block-wise demand for the branch.
  • Scaling-Value Analysis: Scaling-value quantiles shift toward smaller values at later timesteps and remain highly consistent across prompts at matched timesteps.This supports treating the statistics as primarily timestep-dependent rather than prompt-dependent.
  • Drop Strategies: A shared timestep-wise threshold can drop M/N of linear branches, with thresholds calibrated using a small calibration set.The threshold controls the fraction of skipped branches across N blocks.
  • Drop Strategies: A coarse threshold set at timestep 0 can be reused across all diffusion timesteps while producing stage-dependent dropping ratios.Because scaling values vary naturally over denoising, the constant threshold adapts branch dropping to timestep demand.
  • Drop Strategies: The coarse-grained strategy matches SALAD without dropping while providing a slight efficiency gain.The reported ablation uses a VBench subset.
  • Drop Strategies: Randomly dropping the same proportion performs noticeably worse, supporting dynamic scaling values as indicators for safe branch pruning.The coarse variant drops about 50% of branches on average across timesteps.

A.4 Ablation on the Non-Linear Function in the Input-Timestep-Aware Scaling Module

The Input-Timestep-Aware Scaling Module uses a nonlinear dynamic scaler, with sigmoid selected to constrain gating values and improve SALAD performance. In the ablation, sigmoid outperforms ReLU and Tanh.

  • Nonlinear Function Ablation: Constraining dynamic scaling values to [0, 1] is sufficient and beneficial for SALAD.The sigmoid choice increases the scaling module’s expressive capacity while enforcing the reported gating range.
  • Nonlinear Function Ablation: Sigmoid yields the best performance among the evaluated nonlinear functions, while ReLU and Tanh underperform.The ablation supports using sigmoid for the Dynamic Scaler.

A.5 Details of Non-Shared Sparse-Linear Attention

The non-shared architecture gives sparse and linear branches independent parameters, but SALAD’s shared-weight design achieves comparable performance with greater parameter efficiency. Additional experiments show SALAD’s gains are not explained simply by increasing LoRA rank, and tuning remains lightweight across settings.

  • Architecture Comparison: Non-shared sparse-linear attention uses independent parameter sets, making it less parameter-efficient than shared-weight SALAD.The non-shared variant can still be adapted to the SALAD formulation.
  • Architecture Comparison: Adding the Dynamic Scaler and zero-initialization substantially improves all reported metrics for the non-shared variant.The comparison follows the same training configuration on a common VBench subset.
  • Architecture Comparison: Shared- and non-shared-weight SALAD achieve comparable overall performance, with each variant leading on different metric subsets.Non-shared SALAD is slightly higher on SC and TC, while shared SALAD is better on BC and IQ.
  • Architecture Comparison: SALAD’s shared-weight formulation is adopted to avoid the additional memory and latency overhead of non-shared parameters.Its comparable performance and parameter efficiency motivate the final design choice.
  • Comparison with LoRA: Increasing LoRA rank does not consistently improve generation quality, whereas SALAD remains competitive or better across most metrics.The best LoRA configuration generally uses a moderate rank such as r = 128, while larger ranks can degrade performance.
  • Comparison with LoRA: SALAD retains only its persistent inference parameters after LoRA weights are merged into the base model.Thus, the reported 4.99% overhead refers to SALAD’s inference-time parameters rather than all trainable parameters.
  • Training Efficiency: SALAD tuning uses 2,000 videos, batch size 8, and settings ranging from 600 to 1,600 steps across evaluated model and resolution configurations.The reported costs are approximately 20 GPU hours at 480p and 30 GPU hours at 720p.
  • Training Efficiency: SALAD achieves comparable or superior results to dense performance at a fraction of the resource cost of larger-scale sparse-adaptation approaches.The comparison includes methods trained on substantially more videos or larger compute configurations.
Loading 2601.16515v2…