PaperlayerView exact revision on arXiv

Source-linked AI summary

SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection

Ricardo Pizarro, Roberto Valle, José M. Buenaposada, Luis M. Bergasa, Luis Baumela

arXiv:2610.11579v1cs.CV

TL;DR

Existing adapters do not reduce inference-time computation, and token selection disrupts the spatial grids their convolutions require. The paper introduces native sparse 2D convolutions for dynamically pruned tokens and integrates them into SV-TAD. With InternVideoNext-L, it reports state-of-the-art accuracy on THUMOS-14 and ActivityNet-1.3 at substantially lower compute.

  • Problem

    Convolutional adapters process dense token grids and do not reduce inference computation, while token selection disrupts the spatial structure they require, limiting its gains.

    Source: §1 · p. 2

  • Method

    The paper introduces native sparse 2D convolutions that use neighbor indices to let convolutional adapters operate directly on dynamically pruned tokens without dense-grid reconstruction.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    InternVideoNext-L sets new state of the art on THUMOS-14 and ActivityNet-1.3 with 51% and 59% fewer TFLOPs, respectively.

    Source: §5 · p. 16

  • Takeaways & Limitations

    Auxiliary landmark supervision improves the ATTACH baseline by +2.41% mAP, showing the framework's support for multitask fine-grained assembly detection.

    Source: §1 · p. 3, §5 · p. 16

    The sparse kernel is faster than dense convolution only below an approximately 55% keep rate, and the implementation targets NVIDIA GPUs.

    Source: §5 · p. 16

Abstract

from arXiv · show

To adapt billion-parameter Vision Transformers for long-video understanding, recent methods freeze the backbone and train lightweight convolutional modules. While effective for parameter-efficient training, existing adapters do not reduce inference-time computation, leaving scalability with respect to video length largely unaddressed. Token selection can reduce attention cost by pruning redundant tokens, but it breaks the spatial grid structure required by convolutional adapters. This forces an expensive dense reconstruction, nullifying much of the potential speedup. We address this by introducing native sparse 2D convolutions, a primitive that allows these adapters, for the first time, to operate directly and efficiently on dynamically pruned token sets. We integrate this primitive into SV-TAD, an adapter framework for temporal action detection, reducing VideoMAEv2-L computation by up to 64% and achieving 2.2x faster inference, while maintaining state-of-the-art accuracy on THUMOS-14 and ActivityNet-1.3. When scaled to InternVideoNext-L, our approach surpasses the previous state of the art at roughly half its computational cost. Moreover, the sparse formulation naturally supports auxiliary task tokens, which improves fine-grained assembly detection on ATTACH.

1 Introduction

Existing convolutional adapters remain dense at inference, while token selection disrupts the grids they require. SV-TAD combines token selection with sparse convolutional adapters, reporting lower compute and faster inference while maintaining detection accuracy.

1 Introduction

  • Adapters' dense processing scales with the full video sequence, and reconstructing grids after pruning diminishes token sparsification's computational gains.Existing sparse 3D convolution engines are also incompatible with dynamic token selection's format and host-side operations.Source: §1 · p. 2

2 Related Work

  • Token selection reduces self-attention cost while preserving representations, but its irregular token sets leave convolutional adapters without their required spatial structure.Prior token-selection methods focus on attention and do not specify how irregular sets interact with local operators.Source: §1 · p. 2, §2 · p. 4

1 Introduction

  • SV-TAD inserts sparse convolutional adapters into a frozen ViT backbone and progressively prunes token sequences with selection modules.This combines parameter-efficient adaptation with token selection.Source: §1 · pp. 2–3
  • SparseConv2D uses a precomputed neighbor index table and custom CUDA kernels to convolve directly over retained tokens, avoiding dense-grid reconstruction.The regular patch grid allows the neighbor table to be built in O(Nkept), enabling token reduction to reduce memory and computation proportionally.Source: §1 · pp. 2–3
  • 64% fewer TFLOPs with ViT-L and 2.2x faster inference over the state of the art on THUMOS-14, while maintaining mAP.Across backbone scales, SV-TAD matches or exceeds AdaTAD with 43%, 64%, and 51% fewer TFLOPs in the comparisons reported for THUMOS-14.Source: §1 · p. 3, Figure 1 · p. 3
  • +2.41% mAP over the ATTACH baseline with auxiliary landmark supervision demonstrates support for multitask learning in fine-grained assembly detection.The sparse framework supports auxiliary task tokens.Source: §1 · p. 3

2 Related Work

  • Prior approaches leave the efficiency components disconnected: adapters use full grids, token selection omits local operators, and existing sparse engines mismatch ViT tensor formats and dynamic sparsity.SV-TAD combines sparse adapters, token selection, and native sparse 2D convolution without dense reconstruction or external sparse-tensor engines.Source: §2 · pp. 4–5

3 Method

SV-TAD combines token selection with native sparse convolutional adapters, processing retained visual tokens without reconstructing the dense spatial grid. Auxiliary tokens can guide selection and receive task-specific information through cross-attention.

3 Method

  • SV-TAD integrates native sparse convolution, token selection, and auxiliary cross-attention in adapters within a frozen ViT backbone.The sparse adapter is inserted after each Transformer block’s attention layer.Source: §3 · p. 5, §3.4 · p. 9

3.1 Token Selection

  • Attention-based selection progressively prunes visual tokens while preserving their original grid positions for sparse convolution.Selection is applied at evenly spaced layers, after early layers process all tokens.Source: §3.1 · p. 6

3.2 Native Sparse 2D Convolution

  • SparseConv2D uses a neighbor index table to gather each retained token’s local neighbors, avoiding dense reconstruction.For a 3 × 3 kernel, sparse convolution matches dense latency at about a 55% keep rate and is faster below that threshold.Source: §3.2 · p. 7

3.2.1 Custom CUDA Kernels.

  • Custom CUDA kernels implement forward and backward passes, with computation paths selected according to output-channel size.The implementation uses implicit GEMM via cuBLAS for at least 256 output channels and fused gather-and-GEMM tiled kernels for smaller channel counts.Source: §3.2.1 · p. 8

3.2.2 Complexity Analysis.

  • Sparse-convolution compute scales with the retained-token count, while memory is bounded by chunk size rather than total video length.At a 0.5 keep rate, throughput reaches 0.60 videos/s from 0.42 with a −1.13% average mAP change on THUMOS-14 using InternVideoNext-L.Source: §3.2.2 · pp. 8–9

3.3 Auxiliary Task Supervision

  • Auxiliary tokens aggregate task-specific visual information through cross-attention, supporting landmark-guided selection and pose heatmap prediction for assembly tasks.Only auxiliary tokens are updated by this cross-attention, leaving the visual-token path unchanged.Source: §3.3 · p. 9

3.4 Sparse Adapter Architecture

  • The bottleneck adapter down-projects features, applies sparse convolution and auxiliary cross-attention, then up-projects them into a learnably scaled residual.Before token selection, early layers use standard 2D convolution because sparse convolution offers no gains in the dense regime.Source: §3.4 · pp. 9–10

4 Experiments

SV-TAD combines sparse convolutional adapters with token selection to reduce computation across temporal action detection benchmarks. Experiments compare its efficiency, accuracy, and auxiliary-task support across datasets, convolution designs, and backbone scales.

4.1 Datasets and Setup

  • SV-TAD is evaluated on THUMOS-14, ActivityNet-1.3, and ATTACH, using frozen VideoMAE or InternVideoNext backbones, sparse adapters, token selection, and an ActionFormer head.Inputs are 768 frames at 224 × 224 for THUMOS-14 and ATTACH, and 192 frames at 160 × 160 for ActivityNet-1.3.Source: §4.1 · p. 11, §4 · p. 11

4.2 Efficiency Analysis

  • SV-TAD processes 1.57 videos/second versus AdaTAD’s 0.73, a 2.2x throughput improvement alongside a 64% TFLOPs reduction at kr=0.6.Training takes 3.01s per step versus 5.35s, and peak allocated memory is 9.23GB versus 13.30GB, a ∼31% reduction.Source: §4.2 · p. 11
  • At 6,144 frames, sparse convolution runs in 40ms versus dense convolution’s 70ms and uses ∼800MB versus ∼6GB of memory.At kr=0.3, the kernel speedup is 2× at 768 frames and 1.75× at 6,144 frames; chunked processing keeps sparse-convolution memory near-constant.Source: §4.2 · p. 12, Figure 5 · p. 12

4.4 Backbone-Matched Comparison

  • SV-TAD uses 10.29 TFLOPs, 43% fewer than AdaTAD’s 17.9, while accuracy is 72.44% vs. 72.15%.This backbone-matched THUMOS-14 comparison uses VideoMAEv2-B.Source: Table 3 · p. 13, §4.4 · p. 13

4.3 Convolution Comparison: Dense vs. Sparse

  • SparseConv2D exceeds DenseConv2D by +0.52% average mAP; DenseConv1D scores 69.60 vs. 69.35, a +0.25% advantage under scatter-gather reconstruction.The SparseConv2D–DenseConv2D gap shrinks to 0.10% in FP32 without token selection (kr=1).Source: Table 2 · p. 13, §4.3 · p. 13

4.5 Auxiliary Task Tokens Experiments

  • On ATTACH, adding auxiliary landmark supervision improves average mAP by +2.41% (18.69 vs. 16.28).The study frames ATTACH as a full temporal action detection task and uses a person split.Source: Table 4 · p. 14, §4.5 · p. 14

4.6 Comparison with State-of-the-Art

  • On THUMOS-14 with InternVideoNext-L, SV-TAD uses 59% fewer TFLOPs (34 vs. 83.59) and reaches 40.06% average mAP, +0.86% over VMAEv2-G’s 39.20%.With ViT-L on THUMOS-14, it uses 64% fewer TFLOPs (21.25 vs. 59.4) while accuracy is 73.47% vs. 73.50%; with a matched InternVideoNext-L backbone, it cuts compute from 95.3 to 34 TFLOPs and improves accuracy by +0.34%.Source: Table 5 · p. 15, §4.6 · p. 15

5 Conclusion

SV-TAD applies native sparse convolutions to pruned token sets, reducing computation while maintaining or improving detection accuracy across benchmarks. Results extend to fine-grained assembly detection and broader ViT convolutional modules, with implementation and pruning-rate boundaries.

5 Conclusion

  • Native sparse 2D convolution lets adapters operate on dynamically pruned tokens without reconstructing the dense spatial grid.This makes sparsity effective beyond attention layers.Source: §5 · p. 15
  • With ViT-L, SV-TAD matches top-performing accuracy at 64% fewer TFLOPs, and the efficiency gains grow with model capacity.The conclusion also reports state-of-the-art accuracy on both benchmarks with InternVideoNext-L.Source: §5 · pp. 15–16
  • SV-TAD sets new state of the art on THUMOS-14 and ActivityNet-1.3 with 51% and 59% fewer TFLOPs, respectively.It achieves the best accuracy while using roughly half the compute.Source: §5 · p. 16
  • The primitive also applies to spatial adapters, feature pyramid networks, and task-specific heads following token selection in Vision Transformers.Source: §5 · p. 16
  • The implementation targets NVIDIA GPUs, and sparse kernels become faster than dense ones only below an approximately 55% keep rate.The authors identify progressive pruning as a condition for the primitive’s greatest benefit.Source: §5 · p. 16

Supplemental Materials

The supplement collects implementation, experimental, and analysis details supporting the main paper.

Supplemental Materials

  • The supplement provides CUDA kernel implementation details, dataset descriptions, and full training hyperparameters.Source: Supplemental Materials · p. 21
  • It reports AdaTAD++ reproduction analysis and additional ablations on keep-rate sensitivity, numerical divergence, selection robustness, and generality beyond TAD.Source: Supplemental Materials · p. 21
  • It includes detailed ATTACH analysis and discusses token-selection incompatibility with standard convolutions, dense-versus-sparse comparisons, and native sparse 1D temporal kernels.Source: Supplemental Materials · p. 21

CUDA Kernel Implementation Details

The CUDA implementation selects between two kernel strategies based on output-channel dimension, both consuming neighbor indices and writing only retained positions. It combines fused sparse gathering with tiled or batched computation, while chunking bounds memory and complexity scales with retained tokens.

CUDA Kernel Implementation Details

  • Two complementary CUDA kernel strategies are selected by output-channel dimension; both use the neighbor index table and produce outputs only at kept positions.Neither strategy reconstructs the dense grid.Source: CUDA Kernel Implementation Details · p. 21

Implicit GEMM via cuBLAS (C ≥256)

  • For C ≥256, fused gathering, one strided batched GEMM, and fused reduction reduce the forward pass to three launches instead of 36.Chunked processing bounds peak memory independently of sequence length.Source: Implicit GEMM via cuBLAS (C ≥256) · p. 21

Custom Tiled Kernel (C < 256)

  • When Cout is small, cuBLAS launch overhead dominates compute, so hand-tuned CUDA kernels fuse gather and multiply–accumulate into one launch without intermediate buffers, using 32-token-by-32-output-channel tiles.The sparse kernels outperform dense baselines even under uniform random pruning, but reach peak throughput when retained tokens are spatially clustered.Source: Custom Tiled Kernel (C < 256) · pp. 21–22

Shared Kernel Infrastructure

  • All kernels use 128-bit vectorized memory loads and accumulate FP16/BF16 inputs in FP32 before bias addition and output casting.The fused operation writes results to global memory in one pass.Source: Shared Kernel Infrastructure · p. 22

S-I.4 Complexity Analysis

  • Sparse convolution costs O(Nkept · K2 · Cin · Cout), versus O(THW · K2 · Cin · Cout) for dense convolution; fixed-size chunks bound memory independently of sequence length.Layers after the final selection stage operate at the most aggressive sparsity, where per-layer gains are largest.Source: S-I.4 Complexity Analysis · p. 22

Dataset Details

The experiments cover three untrimmed or densely annotated video datasets, with dataset-specific sampling and train/evaluation protocols.

THUMOS-14

  • THUMOS-14 has 200 validation and 211 test videos across 20 sports classes; evaluation uses 768 sampled frames per video, split into 48 clips.Following AdaTAD, training uses the validation set and evaluation uses the test set.Source: THUMOS-14 · p. 23

ActivityNet-1.3

  • ActivityNet-1.3 spans 200 activity classes and approximately 849 hours; evaluation uses 4,728 validation videos after excluding 198 videos blocked on YouTube.The protocol samples 192 frames uniformly per video, using adaptive stride and 12 clips.Source: ActivityNet-1.3 · p. 23

ATTACH

  • ATTACH contains 51 bimanual assembly action classes across 1,200+ videos, with dense annotations and over 68% temporal overlap.Sampling uses 768 frames per video at stride 4, covering approximately 100 seconds.Source: ATTACH · p. 23

S-III

S-III specifies a frozen-backbone training pipeline with trainable sparse adapters and an ActionFormer head, alongside the training augmentation and detection-head settings.

Full Training Configuration

  • Training augmentation uses random temporal crops at a 0.9–1.0 crop ratio, horizontal flips with probability 0.5, and RandAugment [8].Source: Full Training Configuration · p. 23
  • The ActionFormer detection head uses six pyramid levels, regression ratios {1, 2, 4, 8, 16, 32}, focal classification loss, and DIoU regression loss.The focal-loss settings are γ=2 and α=0.25.Source: Full Training Configuration · p. 23

S-III.1

  • Training optimizes the sparse adapters and ActionFormer head while keeping the pretrained backbone frozen; optional ATTACH supervision jointly trains an auxiliary head.Without auxiliary supervision, training uses the detection loss alone; with it, the loss adds an MSE heatmap-regression term.Source: End-to-End Training and Inference · p. 24

AdaTAD++ Reproduction Analysis

The supplementary analysis reports an AdaTAD++ reproduction gap, examines numerical differences between dense and sparse convolutions, tests selector robustness, and evaluates transfer beyond temporal action detection.

S-IV

  • The AdaTAD++ reproduction achieves 69.95% versus 73.3% reported, a 3.35-point gap that the authors could not close despite extensive hyperparameter search.The authors cite missing hyperparameters, undocumented augmentation, and unspecified training dynamics; they used AdaTAD code as a base and applied the paper-described modifications.Source: AdaTAD++ Reproduction Analysis · p. 25

Dense vs. Sparse 2D Convolution: Numerical Divergence

  • +0.52% average mAP on THUMOS-14 for SparseConv2D over dense convolution is attributed to backend and gradient-path differences, not a methodological artifact.With no pruning and FP32, SparseConv2D reaches 69.79% and Dense 2DConv 69.89%, a 0.10% gap within run-to-run variance.Source: Dense vs. Sparse 2D Convolution:… · pp. 25–26

Robustness to the Token Selection Criterion

  • 72.43%/10.301 TFLOPs with an EViT-style selector and 68.06%/10.296 TFLOPs with a norm-based selector compare with 72.44%/10.297 TFLOPs for the default selector.The near-identical compute indicates SparseConv2D adds no overhead across these tested selection criteria; the accuracy difference is attributed to the criterion.Source: Robustness to the Token Selection… · p. 26

Generality Beyond Temporal Action Detection

  • 88.7% top-1 accuracy at 1.01 TFLOPs versus AIM’s 88.9% at 1.62 TFLOPs shows comparable accuracy with 37.5% less compute on Diving-48.The test replaces AIM’s dense spatial adapter with SparseConv2D and token selection while retaining the rest of the AIM recipe.Source: Generality Beyond Temporal Action… · p. 26

Detailed ATTACH Analysis

ATTACH’s overlapping bimanual actions challenge sparse detection, while landmark-supervised auxiliary tokens improve results; cross-attention alone is detrimental, and the design is conditional on informative auxiliary supervision.

S-VI

  • Over 68% of ATTACH annotations temporally overlap because of simultaneous bimanual actions, requiring representation of multiple concurrent activities per frame.Source: Detailed ATTACH Analysis · p. 26
  • 18.69% avg. mAP with K=17 pose-landmark queries surpasses AdaTAD by +0.85% while using 38% fewer FLOPs (11.17 vs. 17.90 TFLOPs).The auxiliary loss also guides token selection toward task-relevant regions such as hands and manipulated objects.Source: Table S3 · p. 27, Detailed ATTACH Analysis · p. 27
  • Adding cross-attention without auxiliary tokens reduces performance from 17.09% to 16.28% (−0.81%), as a single CLS query must represent multiple concurrent actions.This bottleneck is especially harmful on multilabel data with simultaneous hand actions.Source: Detailed ATTACH Analysis · p. 27
  • Cross-attention is beneficial only when paired with informative auxiliary tokens; without suitable supervision, the simpler No CrossAttn variant is preferable.The framework can incorporate domain-specific auxiliary tasks when supervision is available and remove them otherwise.Source: Detailed ATTACH Analysis · p. 27

Discussion

The discussion argues that sparse convolution reduces computation without increasing peak memory, explains why dense reconstruction is inefficient, and motivates a spatial rather than temporal sparse kernel.

S-VII.1

  • At 768 frames with VideoMAEv2-Base, SV-TAD and AdaTAD use nearly the same peak memory because frozen-backbone activations dominate and sparse kernels add only a small neighbor table.The table contains Nkept × 9 int32 entries, about 0.2 MB at kr=0.6.Source: Memory Consumption · p. 28

S-VII.2

  • Global TopK pruning yields variable token counts per frame and misaligned surviving positions, breaking the fixed spatial layout standard convolutions assume.Source: Incompatibility Between Token… · p. 28
  • At kr=0.6, dense reconstruction leaves about 40% of convolution FLOPs producing no useful output, while scatter-gather adds a latency floor that more pruning cannot remove.The workaround allocates memory proportional to the dense grid regardless of how many tokens remain.Source: Incompatibility Between Token… · pp. 28–29
  • SparseConv2D replaces allocate-scatter-convolve-gather with a fused kernel that uses a precomputed neighbor table, so memory and computation scale with Nkept rather than H×W×T.Source: Incompatibility Between Token… · p. 29

Why Spatial Rather Than Temporal Sparse Convolution

  • Despite a +0.25% mAP advantage for dense 1D temporal convolution in ablations, the paper favors spatial sparse convolution because temporal gathers have worse locality, tiling, and kernel utility.With global TopK at kr=0.6, a spatial position’s approximate survival probability in adjacent frames is 0.36.Source: Why Spatial Rather Than Temporal… · pp. 29–30
Loading 2610.11579v1…