PaperlayerView exact revision on arXiv

Source-linked AI summary

STORM: Token-Efficient Long Video Understanding for Multimodal LLMs

Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, Sungjin Ahn, Jan Kautz, Hongxu Yin, Yao Lu, Song Han, Wonmin Byeon

arXiv:2503.04130v4cs.CV

TL;DR

Many Video-LLMs encode frames independently, leaving temporal relationships to the LLM and making long-video processing costly. STORM adds a Mamba temporal encoder and token-reduction strategies; it reports state-of-the-art long-video benchmark results while improving computational efficiency.

  • Problem

    Many Video-LLMs encode frames independently, leaving temporal relationships for the LLM and relying on frame subsampling that can lose important information in long videos.

    Source: §1 · p. 1

  • Method

    STORM inserts a Mamba-based temporal encoder before the LLM and applies temporal or spatial pooling and training-free temporal sampling to reduce visual tokens.

    Source: §1 · pp. 1–2, §3.3 · p. 4, §3.4 · p. 5

  • Results

    STORM reports state-of-the-art results across long-video understanding benchmarks, including 72.5% accuracy on MLVU.

    Source: §4.2 · p. 7, §6 · pp. 11–12

  • Takeaways & Limitations

    The authors conclude that STORM combines long-video understanding with substantially improved computational efficiency.

    Source: §6 · pp. 11–12

    The authors speculate that pretrained LLMs’ effective context length is smaller than their nominal context length, hindering performance on very long sequences.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent advances in video-based multimodal large language models (Video-LLMs) have significantly improved video understanding by processing videos as sequences of image frames. However, many existing methods treat frames independently in the vision backbone, lacking explicit temporal modeling, which limits their ability to capture dynamic patterns and efficiently handle long videos. To address these limitations, we introduce STORM (Spatiotemporal TOken Reduction for Multimodal LLMs), a novel architecture incorporating a dedicated temporal encoder between the image encoder and the LLM. Our temporal encoder leverages the Mamba State Space Model to integrate temporal information into image tokens, generating enriched representations that preserve inter-frame dynamics across the entire video sequence. This enriched encoding not only enhances video reasoning capabilities but also enables effective token reduction strategies, including test-time sampling and training-based temporal and spatial pooling, substantially reducing computational demands on the LLM without sacrificing key temporal information. By integrating these techniques, our approach simultaneously reduces training and inference latency while improving performance, enabling efficient and robust video understanding over extended temporal contexts. Extensive evaluations show that STORM achieves state-of-the-art results across various long video understanding benchmarks (more than 5% improvement on MLVU and LongVideoBench) while reducing the computation costs by up to $8\times$ and the decoding latency by 2.4-2.9$\times$ for the fixed numbers of input frames. Project page is available at https://research.nvidia.com/labs/lpr/storm

1. Introduction

Many Video-LLMs encode frames independently, leaving temporal relationships to the LLM and making long-video processing costly. STORM adds a Mamba-based temporal encoder that supports token reduction while retaining temporal information.

1. Introduction

  • Independent frame encoding leaves temporal relationships for the LLM to infer, while naive frame subsampling can discard details needed for comprehensive video understanding.The passage contrasts this with LongVU, which selects tokens using queries but lacks a dedicated module to propagate spatiotemporal information before selection.Source: §1 · p. 1
  • STORM inserts a Mamba-based temporal encoder between the image encoder and LLM to integrate temporal dynamics early and enable downstream token reduction.The authors distinguish this design from Mamba video models that replace backbones without using Mamba to reduce video redundancy and compress visual tokens.Source: §1 · p. 1
  • Figure 1 presents free-form queries about complex long scenes and reports efficient, accurate understanding that outperforms existing methods across a wide range of tasks.The caption attributes this to capturing essential spatiotemporal cues while compressing redundant frame information.Source: Figure 1 · p. 2
  • The temporal encoder propagates information across the video so tokens carry temporal history, allowing fewer LLM tokens while preserving key information.The paper explores both training-free subsampling and training-based temporal and spatial compression.Source: §1 · pp. 1–2

2. Related Work

Related work spans visual-language architectures, long-video processing and token reduction, and Mamba-based video models. STORM combines a Mamba temporal module with multimodal LLMs to use temporal information for token compression.

2. Related Work

  • VLMs either integrate vision through cross-attention with frozen language-model weights or encode images as tokens that are concatenated with text for the language model.Video processing can extend the latter approach by treating videos as sequences of images; some systems instead use specialized video encoders or 3D convolutions.Source: §2 · p. 2
  • Long-video approaches include sequence-parallel systems and token-reduction methods, but direct temporal or spatial pooling often yields only marginal additional gains.The paper reports superior performance when pooling is combined with its temporal projector.Source: §2 · p. 2
  • Prior Mamba video models study video backbones and scanning designs, whereas STORM applies Mamba within a multimodal LLM to summarize history and enable visual-token compression.The related work describes Mamba models as efficient for long sequences, citing subquadratic computation complexity.Source: §2 · p. 3
  • BIMBA is identified as concurrent work exploring a similar long-video architecture and reporting similar empirical benefits.The authors say these independent findings support the hypothesis.Source: §2 · p. 3

3. Method

STORM places a Mamba-based temporal projector between visual encoding and the LLM, then reduces tokens through pooling or test-time temporal sampling. The method draws on state-space sequence modeling and targets the computation and context constraints of long-video processing.

3. Method

  • STORM combines temporal encoding with token compression to address the computational cost and context-length constraints of processing long videos.The method section covers both training-based and training-free compression along temporal and spatial dimensions.Source: §3 · p. 3, §3.3 · p. 4

3.1. Preliminaries

  • An SSM maps an input sequence to an output sequence recurrently, with a hidden state summarizing the input history; Mamba makes the state-space matrices input-dependent.The paper notes that Mamba’s hardware-aware parallel algorithm maintains SSM efficiency despite input-dependent matrices.Source: §3.1 · p. 3

3.2. Mamba-Based Temporal Projector

  • The temporal projector addresses independent frame processing by integrating information across frames, and its bidirectional Mamba layers scan within frames and across the sequence.The resulting tokens are enriched with temporal information before compression.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Training-Time Token Compression

  • Training-time compression averages enriched tokens across consecutive frames or pools spatially; temporal averaging reduces LLM input tokens with minimal loss of critical information.The section frames these methods as ways to reduce the cost and context burden of processing all frames.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4. Training-Free Temporal Token Sampling

  • At test time, uniform temporal token subsampling can follow temporal or spatial pooling because encoded tokens contain information from across the video.The authors report that subsampling maintains or can improve performance on video-understanding benchmarks, potentially by reducing noise from redundant frames.Source: §3.4 · p. 5

4. Experiments

STORM is evaluated against video-LLM baselines and across long-video benchmarks, with experiments examining token compression and extended frame inputs. STORM + T. Pooling outperforms existing models on the reported benchmarks while using fewer visual tokens.

4.1. Experiment Details

  • Experiments fine-tune a PaliGemma vision encoder and Qwen2-VL LLM, evaluate on four video benchmarks, and compare STORM variants with a VILA baseline trained on the same data and pipeline.The supervised fine-tuning mixture contains around 12.5M samples; long-video fine-tuning uses 128-frame inputs.Source: §4.1 · pp. 5–6

4.2. Results on Video Understanding Benchmarks

  • STORM + T. Pooling reaches 71.3% on MVBench, 72.5% on MLVU, 59.5% on LongVideoBench, and 63.4% on VideoMME, outperforming existing models with 25% of the original visual tokens.The comparison includes open-source models and proprietary-model results on the benchmarks specified in the passage.Source: Table 1 · p. 6, §4.2 · p. 7
  • Mamba improves VILA on three of four benchmarks, including a 2.3% VideoMME gain; test-time sampling cuts inference time by 43% and adds 0.8% on VideoMME.The passage attributes the sampling behavior to temporal information propagation that allows redundant tokens to be discarded.Source: §4.2 · p. 7
  • Temporal pooling supports 128-frame inputs at the 32-frame token count, improving VideoMME by 3.3%, LongVideoBench by 3.6%, MLVU by 2.3%, and MVBench by 1.8%.This variant reduces inference latency by 58.4%; adding temporal sampling reduces inference time by 65.5% and uses 12.5% of VILA’s visual tokens.Source: §4.2 · p. 7
  • With 128-frame inputs and the same token budget, temporal pooling improves both LongVideoBench and VideoMME, whereas spatial pooling degrades performance on both.At 32 frames, spatial pooling is effective on LongVideoBench and on par with temporal pooling on VideoMME.Source: §4.2 · p. 8, Table 3 · p. 8

5. Analysis

STORM’s temporal module supports token compression and longer video inputs, with analyses reporting benchmark gains alongside reductions in token use and latency. The section also examines training overhead, retained visual information, and streaming and architecture variants.

Long-context performance

  • +2.4% is the STORM–VILA performance gap at 128F, compared with +0.7% at 32F; STORM’s gains over its shorter-input versions are +2.6% from 8 to 32 frames and +2.5% from 32 to 128.The 32F and 128F comparison uses models first trained on 32-frame inputs and then fine-tuned on 128-frame inputs using a smaller long-video dataset.Source: Table 4 · p. 8, §5 · p. 8

Token-compression ablations

  • For 32-frame-trained models, test-time sampling reduces visual tokens to 50% and latency to 58.7% of uncompressed STORM; temporal pooling uses 25% of tokens and cuts latency to 42.7%.Temporal pooling is particularly effective on MVBench and MLVU but slightly degrades LongVideoBench and VideoMME; spatial pooling has a different benchmark trade-off.Source: §5 · pp. 8–9
  • STORM + T. Pooling + T. Sampling uses 12.5% of STORM’s visual tokens and 35.4% of its latency, while the combined pooling-and-sampling variant uses 3.13% of tokens and 29.5% of latency.The 3.13%-token variant retains 100% of VILA performance on VideoMME, 98.9% on LongVideoBench, 97.7% on MLVU, and 96.7% on MVBench. Temporal pooling also allows 128-frame context without additional LLM computation.Source: §5 · p. 9

Efficiency and long inputs

  • On 256-frame inputs, the LLM accounts for approximately 80% of total latency without compression; at a 4× compression ratio, its latency falls to roughly 35% of its original value.With up to 128 frames, test-time temporal sampling continues improving VideoMME performance while using a 50% token compression ratio.Source: §5 · p. 9, Figure 4 · p. 10

Training overhead

  • Without compression, the Mamba projector adds approximately 5% training latency: 20.1 hours for STORM versus 19.1 hours for VILA.The passage also reports a +3.6% MLVU gain with 4× fewer tokens.Source: §5 · p. 11

Retained visual information

  • Qualitative examples report that 4× compression retains task-relevant visual information, including in OCR-heavy tasks, while using 25% of visual tokens.The passage reports comparable performance to uncompressed versions on OCR-heavy tasks and better results than the baseline across nearly all VideoMME categories.Source: §5 · p. 11

Architecture and streaming generalization

  • STORM’s temporal module and compression are reported across configurations varying LLM, vision encoder, model size, and input resolution; a uni-directional variant also outperforms the VILA baseline in streaming tests.The uni-directional variant performs competitively with the bi-directional model at 32 frames, while the bi-directional model is stronger as video length increases.Source: §5 · p. 11

6. Conclusion

STORM combines a Mamba-based temporal encoder with token reduction to improve long-video understanding while preserving critical information in compressed inputs. Experiments show state-of-the-art benchmark results and improved computational efficiency.

6. Conclusion

  • The paper introduces STORM as a video-LLM model for enhancing long-video understanding.Source: §6 · p. 11
  • STORM integrates spatiotemporal dynamics into visual tokens early, enabling token compression while preserving critical information in compressed inputs.Its temporal encoder is Mamba-based.Source: §6 · p. 11
  • STORM achieves state-of-the-art results on long-video understanding benchmarks while substantially improving computational efficiency.The conclusion attributes these gains to a Mamba-based temporal encoder and efficient token reduction.Source: §6 · pp. 11–12

8. Appendix

The appendix reports qualitative comparisons, compression and task-level results, architecture and data details, latency measurements, and comparisons with other temporal-fusion strategies. Across these evaluations, STORM uses temporal information and token compression to handle longer video inputs efficiently.

8.2. Additional Results

  • Temporal pooling is reported as the most efficient and effective compression approach; compression improves VideoMME results across video lengths and task-specific accuracies.Temporal pooling especially improves object reasoning, test-time sampling improves attribute perception, and both methods improve temporal perception compared with VILA and uncompressed STORM.Source: §8.2 · p. 18, Table 9 · p. 20, Table 12 · p. 22

8.3. Architecture Details

  • STORM replaces the MLP projector with a linear layer and Mamba temporal module, then applies temporal and spatial token compression before the LLM.Temporal pooling and test-time sampling compress the temporal dimension; training-based spatial average pooling reduces frame-level tokens.Source: §8.3 · p. 18, Figure 6 · p. 19

8.4. Datasets Information

  • Training uses multimodal instruction-tuning mixtures; the long-video fine-tuning stage uses a smaller LLaVA-Video subset, while a separate ablation uses only videos of at least 128 frames.The long-video subset is approximately 11% of the full 1.35M-pair dataset; the long-video-only ablation contains about 360K pairs.Source: §8.4 · pp. 18–19, Table 8 · p. 20

8.5. Mamba Temporal Module Latency

  • The Mamba Temporal Module adds no more than 3% of total latency compared with vanilla VILA in the reported single-DGX A100-80G experiments.Source: §8.5 · p. 19, Figure 8 · p. 21

8.6. Inference Details

  • Inference frame counts are selected from 8 to 512 for the best overall accuracy for each task and setup.Table 9 reports 4× compression for spatial and temporal pooling and 2× additional compression for temporal sampling.Source: §8.6 · p. 19, Table 9 · p. 20, Table 15 · p. 24

8.7. STORM vs Other Temporal Fusion Strategies.

  • TSM and SlowFast fail to yield meaningful gains over VILA, whereas STORM significantly outperforms the compared fusion variants.The early comparisons did not consistently match input-frame counts, and the settings were generally favorable to TSM and SlowFast.Source: §8.7 · p. 20, Table 14 · p. 23

8.2. Additional Results

  • STORM benefits from the larger, more diverse training dataset, while baseline VILA degrades on several benchmarks with that dataset.The comparison is between the full LLaVA-Video dataset (about 1.35M pairs) and its long-video-only subset (about 360K pairs).Source: §8.2 · p. 18, Table 11 · p. 22

8.1. Qualitative Results

  • STORM produces the most detailed, coherent film summary versus LongVILA and LongVU, while temporal sampling makes its 128-frame input equivalent to 32 frames in token count.With equal LLM token counts, 128 frames also capture events missed by 32-frame inputs and support reasoning across temporal events.Source: §8.1 · p. 17, Figure 9 · p. 23, Figure 10 · p. 24
Loading 2503.04130v4…