PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoNSA: Native Sparse Attention Scales Video Understanding

Enxin Song, Wenhao Chai, Shusheng Yang, Ethan Armand, Xiaojun Shan, Haiyang Xu, Jianwen Xie, Zhuowen Tu

arXiv:2510.02295v2cs.CVcs.AIcs.LG

TL;DR

Long-video understanding in multimodal language models is constrained by context length, while denser frame sampling increases token and computation costs. VideoNSA applies learnable hybrid sparse attention to video while preserving dense attention for text, and reports gains across long-video, temporal, and spatial tasks.

  • Problem

    Long-video MLLMs face context limits: sampling more frames can improve accuracy but increases token and computational costs.

    Source: §1 · p. 1

  • Method

    VideoNSA adapts learnable, hardware-aware Native Sparse Attention to video tokens through three complementary branches while retaining dense attention for text.

    Source: §1 · p. 2

  • Results

    VideoNSA reports improved performance across long-video understanding, temporal reasoning, and spatial understanding tasks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Analyses report task-dependent branch use and learned sparse attention weights that remain beneficial even under dense attention settings.

    Source: §1 · p. 2

Abstract

from arXiv · show

Video understanding in multimodal language models remains limited by context length: models often miss key transition frames and struggle to maintain coherence across long time scales. To address this, we adapt Native Sparse Attention (NSA) to video-language models. Our method, VideoNSA, adapts Qwen2.5-VL through end-to-end training on a 216K video instruction dataset. We employ a hardware-aware hybrid approach to attention, preserving dense attention for text, while employing NSA for video. Compared to token-compression and training-free sparse baselines, VideoNSA achieves improved performance on long-video understanding, temporal reasoning, and spatial benchmarks. Further ablation analysis reveals four key findings: (1) reliable scaling to 128K tokens; (2) an optimal global-local attention allocation at a fixed budget; (3) task-dependent branch usage patterns; and (4) the learnable combined sparse attention help induce dynamic attention sinks. Project Page: https://enxinsong.com/VideoNSA-web/, Code: https://github.com/Espere-1119-Song/VideoNSA

1 INTRODUCTION

Video understanding faces a context-length tradeoff: sampling more frames can improve accuracy but increases token and computation costs. VideoNSA applies learnable sparse attention to video, reports improved performance with a small attention budget, and analyzes its scaling and branch behavior.

1 INTRODUCTION

  • More frame sampling can improve accuracy, but adds tokens and computation and pushes against context limits; compression can lose information on complex reasoning tasks.The introduction contrasts compression with sparse attention, which preserves tokens while focusing computation on relevant dependencies.Source: §1 · p. 1
  • At 128K context, VideoNSA uses only 3.6% of the attention budget while improving performance across various tasks.The paper presents VideoNSA as a learnable, hardware-aware sparse attention framework tailored for video understanding.Source: §1 · p. 2
  • Analyses report extrapolation beyond training length, task-dependent temporal-density and spatial-resolution tradeoffs, shifting branch weights, a compression bottleneck, and branch-specific attention sinks.Learned sparse weights also remain beneficial under dense attention settings.Source: §1 · p. 2
  • VideoNSA combines global and local attention through three complementary branches, dynamically allocating information and attention budgets across tasks.The contributions identify compression, salient-block selection, and sliding-window attention as the branches.Source: §1 · p. 2

2 VIDEONSA

VideoNSA adapts Native Sparse Attention for video using complementary sparse branches and a hybrid design that applies sparse attention to vision tokens while retaining standard attention for text. It is trained end-to-end on filtered video-instruction data.

2.1 PRELIMINARIES

  • Native Sparse Attention constructs a query-specific, information-dense KV subset using a learnable gate over compression, salient-block selection, and sliding-window branches.Compression averages sequential key blocks, selection retains top-ranked blocks, and the sliding window keeps recent keys.Source: §2.1 · p. 2

2.2 ARCHITECTURE

  • VideoNSA splits vision and text tokens per layer, applies gated three-branch NSA to frame-level vision blocks, and retains standard GQA for text to preserve instruction following.The vision block size equals the number of tokens per frame, whose representations are formed by averaging the block's tokens.Source: §2.2 · pp. 3–4

2.3 TRAINING RECIPE

  • End-to-end training uses 216K filtered question-answer pairs from LLaVA-Video-178K, with up to 36K tokens per instance and 4600 H100 GPU hours.The data filters retain videos with 350–550 frames sampled at 4 fps; training limits each frame to 50,176 pixels.Source: §2.3 · p. 4

3 EXPERIMENTS

Across long-video understanding, temporal reasoning, and spatial understanding, VideoNSA reports strengths over compression methods and competitive results against sparse baselines. Ablations find that the full three-branch system outperforms single- and two-branch combinations.

3.1 EFFECTIVENESS ON VIDEO UNDERSTANDING

  • VideoNSA narrows the gap to state-of-the-art on long-video benchmarks, leads Tomato accuracy, and matches the strongest sparse baselines on spatial reasoning while surpassing compression methods.The reported advantages include order-sensitive temporal reasoning and ultra-long videos, including 10-hour LongTimeScope videos.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 ABLATION STUDY

  • Single-branch models degrade significantly, and even two-branch combinations remain inferior to full VideoNSA with dynamic gating.Table 2 reports the branch-selection ablation across tasks.Source: Table 2 · p. 5, §3.2 · p. 5

4 SCALING ANALYSIS AND FINDINGS

VideoNSA scales beyond its 36K-token training length, while performance depends on task-specific context and attention allocation. Analyses also characterize branch routing, latency, attention sinks, and transferring sparse-trained weights to dense inference.

Finding 1

  • Sparse-trained weights let dense inference recover and surpass the baseline on several benchmarks, but the effect is limited on LongVideoBench.VideoNSA significantly outperforms Dense-NSA on most tasks, highlighting the importance of runtime sparsity and dynamic gating.Source: Table 3 · p. 5, §4 · p. 6

Finding 2

  • 128K-token contexts outperform the 36K-token training length across benchmarks, but the preferred spatial-temporal allocation varies by task.LongVideoBench favors more tokens per frame; Tomato and TimeScope favor more frames, while VSIBench preferences vary with context length.Source: §4 · p. 6, Figure 2 · p. 6

Finding 3

  • VideoNSA reaches leading performance with 3.6% of full attention, and configurations near the training allocation generally perform better than simply enlarging the budget.Across most benchmarks, increasing global attention tends to outperform increasing local attention.Source: §4 · p. 7, Figure 3 · p. 7

Finding 4

  • Compression retains relatively high gate weights across most layers, while selection and sliding-window contributions diminish in the final layers.Gate preferences remain diverse across heads and depth; selection and sliding-window gates can exceed compression in early and middle layers.Source: §4 · pp. 7–8, Figure 4 · p. 8
  • Selection and sliding-window gates become more similar across heads in middle layers, whereas compression remains largely head-independent.Inter-head similarity is weak across all gates in the initial and final layers.Source: §4 · p. 8, Figure 5 · p. 8

Finding 5

  • The compression branch dominates inference latency as context grows, while selection and sliding-window branches contribute relatively little at longer contexts.Measured wall-clock latency departs from theoretical complexity estimates because of hardware parallelism, memory access patterns, and kernel launch overheads.Source: §4 · p. 8, Figure 6 · p. 8

Finding 6

  • Dynamic gating yields a 0.3% overall attention-sink ratio, despite the compression branch producing the most sinks.Selection yields almost no sinks; VideoNSA shows smoother temporal coverage than dense Flash Attention, whose sink ratio grows in deeper layers.Source: §4 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9

5 CONCLUSION

VideoNSA combines three sparse-attention branches in a hardware-aware hybrid model. The paper reports gains across video tasks while noting that prefill remains the primary bottleneck.

5 CONCLUSION

  • VideoNSA combines block-wise compression, salient block selection, and a sliding window, outperforming existing methods on long-video understanding, temporal reasoning, and spatial understanding.The approach preserves critical information while achieving near-linear scalability in efficiency and memory; prefill remains the primary bottleneck.Source: §5 · p. 10

7 ETHICS STATEMENT

The ethics statement describes dataset privacy, potential bias, and misuse safeguards.

7 ETHICS STATEMENT

  • The work uses publicly available datasets, considers bias in automatic answer generation, and reports safeguards against misuse such as generating misleading answers.The authors also state a commitment to further improving bias evaluation and minimization.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

The reproducibility statement points to supplementary implementation details and benchmark settings.

8 REPRODUCIBILITY STATEMENT

  • The supplementary material describes the model architecture, training procedures, hyperparameters, and evaluation benchmark settings.Implementation details are in Section B and benchmark settings are in Section C.Source: §8 · p. 10

9 THE USE OF LARGE LANGUAGE MODELS

VideoNSA is presented as a learnable, hardware-aware sparse attention approach for video, and the supplementary sections report training settings, benchmark results, and analyses of its branches. Across long-video, temporal, spatial, and additional benchmarks, the reported results are competitive with baselines and stronger on several task-specific comparisons.

A.2 SPARSE ATTENTION MECHANISM

  • VideoNSA unifies block compression, salient-block selection, and sliding-window attention under learnable gates for video’s long, redundant spatio-temporal context.The related-work discussion contrasts this design with methods primarily optimized for text or short multimodal contexts.Source: §A.1 · p. 19, §A.2 · p. 20

B DETAILED TRAINING SETTINGS

D MORE RESULTS ON LONG-FORM VIDEO BENCHMARKS

E MORE RESULTS ON TEMPORAL REASONING BENCHMARKS

F MORE RESULTS ON SPATIAL UNDERSTANDING BENCHMARKS

  • On VSIBench, VideoNSA is comparable overall to baselines and significantly outperforms them on object-relative direction, route planning, and object-size estimation.Table 11 reports overall accuracy and task-specific scores.Source: F MORE RESULTS ON SPATIAL… · p. 22, Table 11 · p. 23

G RESULTS ON ADDITIONAL VIDEO UNDERSTANDING BENCHMARKS

  • Additional LSDBench and VideoEvalPro experiments report a consistent VideoNSA advantage over training-free sparse-attention baselines across multiple video-understanding tasks.The supplied passages identify these benchmarks but do not provide their table values.Source: G RESULTS ON ADDITIONAL VIDEO… · p. 22

H VISUALIZATION OF ATTENTION PATTERN IN EACH BRANCH

  • The three attention branches have complementary roles: compression preserves salient information, selection highlights task-relevant regions, and the sliding window covers local temporal dependencies.The paper describes their combined roles as shaping the final attention output.Source: H VISUALIZATION OF ATTENTION PATTERN… · p. 23

I MORE RESULTS ON BRANCH COMBINATION

  • Single-branch models degrade substantially, and two-branch combinations remain inferior to full VideoNSA, supporting integration of all three branches with dynamic gating.The detailed branch-combination results are placed in Appendix I.Source: I MORE RESULTS ON BRANCH COMBINATION · p. 23

J MORE RESULTS ON INFORMATION SCALING STUDY

The information-scaling analysis reports that LongTimeScope and MLVU improve when extended from their 32K training context to 64K, with allocation preferences differing by benchmark. Additional context-length comparisons report that VideoNSA remains stable or stronger at 128K, unlike the base model’s frequent decline from 64K.

J MORE RESULTS ON INFORMATION SCALING STUDY

K ADDITIONAL CONTEXT-LENGTH SCALING RESULTS OF QWEN2.5-VL

L MORE RESULTS ON ATTENTION SCALING STUDY

The attention-scaling analysis reports strong results with a small attention budget and finds that equal-budget allocations favor global attention over larger local windows. The accompanying theoretical discussion links context-length scaling to routing stability and RoPE geometry, while noting a positional-encoding boundary at extreme distances.

L MORE RESULTS ON ATTENTION SCALING STUDY

  • On LongVideoBench and TimeScope, performance peaks near the training allocation or favors more global blocks, while larger local windows can reduce accuracy; strong results persist at 3.6% or less of full attention cost.The reported comparison uses equal-attention-budget configurations; TimeScope is described as especially sensitive to larger sliding windows.Source: L MORE RESULTS ON ATTENTION SCALING… · pp. 25–26

M THEORETICAL FOUNDATIONS OF SCALING BEHAVIOR

  • With context length increased through denser sampling, the sparse mask remains unchanged, and the paper argues that the learned routing structure is therefore preserved as extra tokens provide auxiliary evidence.The discussion says this keeps routing deviation small and can maintain or improve performance at longer contexts.Source: M THEORETICAL FOUNDATIONS OF SCALING… · p. 28
  • Changing the attention budget changes the visible-token mask; newly visible candidates can dilute or compete with existing key tokens and destabilize learned routing paths.The paper links this mechanism to increased routing deviation and potentially degraded performance.Source: M THEORETICAL FOUNDATIONS OF SCALING… · pp. 28, 30
  • RoPE relates position differences to rotation phase, supporting extrapolation at moderate context increases but risking phase aliasing when effective phase differences become excessive.At extreme distances, high-frequency components lose discriminative resolution and token separability is reduced.Source: M THEORETICAL FOUNDATIONS OF SCALING… · pp. 31–32

O MORE INTER-HEAD GATE SIMILARITES VISUALIZATION

P BENCHMARK-LEVEL GATING ANALYSIS AND PCA VISUALIZATION

The gating analysis reports that VideoNSA’s routing patterns vary with video content, and that replacing dynamic gates with static layer averages reduces performance across benchmarks.

P BENCHMARK-LEVEL GATING ANALYSIS AND PCA VISUALIZATION

  • Gate vectors form three distinct benchmark-specific clusters across compression, selection, and sliding-window branches, indicating routing strategies conditioned on visual content rather than layer depth alone.The videos represent multi-shot transitions, high-frequency motion, and complex spatial layouts.Source: P BENCHMARK-LEVEL GATING ANALYSIS AND… · pp. 33–34
  • Replacing each layer’s gate with a static value averaged from a 1K training subset reduces performance on all six benchmarks, especially tasks requiring long-range temporal integration.The passage presents this comparison as evidence that dynamic gating is essential.Source: P BENCHMARK-LEVEL GATING ANALYSIS AND… · p. 34

Q ADDITIONAL ANALYSIS OF TRAINING AND INFERENCE EFFICIENCY

This section examines VideoNSA’s training and inference efficiency, including how kernel overhead, implementation, settings, and modality relate to latency, attention-sink patterns, and sparse routing. It also reports head-level maximum and sink-count readings from dense-attention visualizations.

T DISCUSSION ON MODALITY-SPECIFIC SPARSITY

Q ADDITIONAL ANALYSIS OF TRAINING AND INFERENCE EFFICIENCY

  • 2.05 PFLOPs (0.24× Flash Attention) is NSA’s ideal cost; in practice, VideoNSA latency grows more slowly than dense attention and is competitive among sparse baselines.VideoNSA requires approximately 4600 H100 GPU hours versus 5280 for the dense baseline, or 0.87× its training cost. Query-head padding to 64 to meet Triton kernel requirements adds computation and memory-access overhead, so practical efficiency falls short of the ideal FLOPs advantage.Source: Q ADDITIONAL ANALYSIS OF TRAINING AND… · p. 34, Figure 26 · p. 37, Table 35 · p. 37

T DISCUSSION ON MODALITY-SPECIFIC SPARSITY

  • Text-only NSA shows early cross-head correlations that later decorrelate for selection and sliding-window gates, while VideoNSA has weaker overall correlations and only a few mid-layer clusters.The passages interpret this contrast as VideoNSA adapting routing to video’s spatiotemporal redundancy and multi-scale structure rather than collapsing into one pathway.Source: T DISCUSSION ON MODALITY-SPECIFIC… · pp. 37–38, Figure 29 · p. 39

R ADDITIONAL ANALYSIS ON CMP LATENCY BOTTLENECK

  • Changing CMP block size yields only modest latency improvements: smaller blocks incur memory-access overhead, while larger blocks require more computation per block.Block-size scaling affects latency only moderately within a narrow range and does not change the overall scaling trend.Source: R ADDITIONAL ANALYSIS ON CMP LATENCY… · p. 35, Table 36 · p. 38
  • flash-nsa runs about twice as fast as nsa-impl in the forward pass and up to six times faster in the backward pass, making kernel efficiency the dominant latency factor.Kernel efficiency includes memory-access patterns and kernel design; the passage also notes work on NSA optimization for TPUs.Source: R ADDITIONAL ANALYSIS ON CMP LATENCY… · p. 35, Table 37 · p. 38

S MORE ANALYSIS ABOUT ATTENTION SINKS ON VARIOUS SPARSE ATTENTION SETTINGS

  • Across sparse settings, smaller compression blocks sharpen initial sinks, larger blocks diffuse them, and larger windows lower sink density while broadening coverage; selection sinks remain sparse.The training window configuration w = 256 strikes a middle ground and shows sparse periodic clusters in the mid-to-late sequence.Source: S MORE ANALYSIS ABOUT ATTENTION SINKS… · p. 35, Figure 27 · p. 38

U DENSE ATTENTION SINK VISUALIZATION

  • In QWEN2.5-VL Flash Attention analysis (116720 tokens), layer-0 head 21 has max 3.738 and 3 sinks; layer-14 heads 1 and 2 have max 0.947/0.850 and 127/147 sinks.Source: U DENSE ATTENTION SINK VISUALIZATION · p. 40
  • Head 18 recorded max 1.093 with 118 sinks, while head 22 recorded max 0.922 with 105 sinks.Other reported heads in this visualization include head 12 (max 1.015, 137 sinks) and head 27 (max 0.860, 101 sinks).Source: U DENSE ATTENTION SINK VISUALIZATION · p. 40
  • In the 116720-token layer 27 Flash Attention analysis, head 27 recorded max 6.926 with 10 sinks, while head 24 recorded max 6.862 with 6 sinks.Other listed readings include head 0 (max 5.929, 12 sinks) and head 22 (max 6.562, 8 sinks).Source: U DENSE ATTENTION SINK VISUALIZATION · p. 40

Related papers

Loading 2510.02295v2…