Accelerating Streaming Video Large Language Models via Hierarchical Token Compression

Yiyu Wang; Xuyang Liu; Xiyan Gui; Xinying Lin; Boxue Yang; Chenfei Liao; Tailai Chen; Linfeng Zhang

Source-linked AI summary

Accelerating Streaming Video Large Language Models via Hierarchical Token Compression

Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, Linfeng Zhang

arXiv:2512.00891v2Nov 30, 2025cs.CV

TL;DR

Streaming VideoLLMs are difficult to deploy in real time because dense visual-token processing makes existing systems too slow, with ViT encoding and LLM prefilling creating major overhead. STC jointly accelerates both stages through feature caching and dual-relevance token pruning, achieving superior performance-efficiency trade-offs while preserving causal, query- and future-agnostic streaming operation.

Problem
Dense visual-token processing makes existing VideoLLMs too slow for latency-sensitive real-time streaming applications.
Method
STC is a plug-and-play framework that caches features from temporally redundant frames and prunes visual tokens using spatial and temporal relevance before LLM prefilling.
Results
Extensive experiments show that STC achieves superior performance-efficiency trade-offs across streaming video understanding settings.
Takeaways & Limitations
STC provides a practical, retraining-free approach for reducing redundant computation across both ViT encoding and LLM prefilling in real-time streaming VideoLLMs.
Takeaways & Limitations
STC is designed for query- and future-agnostic streaming, preserving causal integrity under real-time constraints.

Abstract

from arXiv · show

Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks, but they face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams. In streaming video scenarios, the primary bottleneck lies in the Vision Transformer (ViT) encoding stage, where redundant processing of temporally similar frames leads to inefficiency. Additionally, inflated token sequences during LLM pre-filling further exacerbate latency and memory overhead. To address these challenges, we propose \textbf{S}treaming \textbf{T}oken \textbf{C}ompression (\textbf{STC}), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs, optimizing both ViT encoding and LLM pre-filling stages to accelerate processing. STC introduces two token-level accelerators: \textbf{STC-Cacher}, which reduces ViT encoding overhead by caching and reusing features from temporally similar frames, and \textbf{STC-Pruner}, which compresses the visual token sequence before it enters the LLM, preserving only the most salient tokens based on both spatial and temporal relevance. Extensive experiments on four baseline streaming VideoLLMs across five benchmarks demonstrate that STC outperforms other compression methods. Notably, STC retains up to \textbf{99\%} of accuracy on the ReKV framework while reducing ViT encoding latency and LLM pre-filling latency by \textbf{24.5\%} and \textbf{45.3\%}.

1. Introduction

Streaming VideoLLMs are difficult to deploy in real time because dense visual tokens and repeated ViT processing create substantial latency. STC addresses these bottlenecks with causal caching and token pruning across the ViT and LLM stages.

1. Introduction: Streaming video understanding requires continuous frame processing and low latency, but dense visual-token computation limits real-time deployment.The setting is motivated by applications such as live sports commentary and augmented-reality glasses.
1. Introduction: ViT encoding in video understanding costs 2-3 times more than in image understanding, while a 32-frame video can produce 6,272 visual tokens for LLM prefilling.The comparison cites Qwen2-VL and LLaVA-OV, and contrasts video processing with roughly 1,900 image visual tokens.
1. Introduction: Streaming compression cannot rely on complete videos or future user instructions, making global-feature and instruction-aware pruning ineffective in this setting.Streaming systems must make compression decisions causally as frames arrive.
1. Introduction: STC jointly accelerates both stages through STC-Cacher’s selective feature reuse and STC-Pruner’s spatial-temporal token selection.The framework caches features from temporally similar frames and removes redundant visual tokens before LLM prefilling.
1. Introduction: 99% accuracy is retained on ReKV while LLM prefill latency is reduced by 45.3%.The contribution summary reports STC’s headline efficiency–accuracy result.

2. Related Work

Prior streaming VideoLLMs still encode every frame and feed dense token sequences to the LLM, limiting real-time efficiency. STC is presented as a streaming-native design that jointly addresses ViT encoding and LLM prefilling.

2. Related Work: Existing streaming approaches process each frame through ViT and feed dense token sequences into the LLM, creating high computational and memory costs.These costs limit real-time deployment despite the effectiveness of existing streaming paradigms.
2. Related Work: Prior VideoLLM compression methods include offline token merging or selection, while KV-cache methods reduce decoding memory overhead.Offline methods assume access to the full video and may operate during or after feature extraction.
2.2. Token Compression for VideoLLMs: Prior work either overlooks streaming’s causal nature or optimizes only one stage, whereas STC jointly optimizes ViT encoding and LLM prefilling.The framework uses a streaming-native two-stage design intended to maximize redundancy removal while respecting temporal causality.
2.2. Token Compression for VideoLLMs: STC’s two-stage overview pairs STC-Cacher for ViT computational redundancy with STC-Pruner for LLM prefilling sequence reduction.The modules target distinct stages of streaming VideoLLM inference.

3. Methodology

STC accelerates streaming VideoLLMs by addressing temporal redundancy in ViT encoding and long-context redundancy during LLM prefilling with two complementary, query- and future-agnostic token compressors.

Overview: The framework targets two streaming bottlenecks: repeated ViT processing of similar frames and redundant long visual-token sequences whose self-attention cost is O(N^2).Its two components jointly reduce computational overhead while preserving causal, query- and future-agnostic processing.
STC-Cacher: STC-Cacher identifies dynamic tokens through similarity between current and cached Key projections, then performs selective attention and scatter-updates cached outputs.The method computes attention for selected tokens and inserts updated values into cached matrices rather than executing a full forward pass.
STC-Cacher: STC-Cacher reduces ViT computation by caching reference-frame representations and selectively recomputing dynamic tokens while reusing cached states for static content.Reference frames receive full computation; subsequent frames bypass most computation using cached information.
STC-Pruner: STC-Pruner computes novelty from joint dissimilarity to both anchors and passes only the highest-scoring top-k tokens to the LLM.After processing, the current spatial anchor updates the history buffer for the next time step.
STC-Pruner: STC-Pruner reduces LLM prefilling cost by scoring visual tokens against temporal and spatial context anchors, retaining tokens that are novel relative to both.The Temporal Context Anchor summarizes historical context, while the Spatial Context Anchor represents the current frame’s global context.

4. Experiments

Experiments evaluate STC across streaming and long-video benchmarks, showing strong performance, efficiency gains, and complementary module behavior. Ablations identify effective token-dynamics metrics, feature reuse, and joint spatial-temporal scoring.

Main Comparisons: STC outperforms existing methods on all evaluated benchmarks, improving over VidCom2 by 1.6 on both OVO-Bench and StreamingBench.STC-Pruner also achieves state-of-the-art results across three offline long-video benchmarks.
Main Comparisons: STC reduces ReKV ViT encoding latency by 24.5% and LLM prefilling latency by 45.3% on OVO-Bench.The framework also outperforms ToMe by 5.6, 5.8, and an average of 4.1 across the reported benchmark groups.
Main Comparisons: STC-Cacher and STC-Pruner jointly support plug-and-play acceleration for online and offline-to-online VideoLLMs.STC-Cacher accelerates ViT encoding, while the combined framework also accelerates LLM prefilling for ReKV.
Ablation Studies and Analysis: Cosine similarity is the default STC-Cacher dynamics metric because it consistently performs best, while all tested metrics outperform ToMe.Key-states provide the strongest feature basis for identifying dynamic tokens.
Ablation Studies and Analysis: Reusing both attention and MLP activations outperforms reusing either pathway alone, preserving positional/contextual and channel/semantic information.Attention-only and MLP-only reuse each underperform the combined strategy.
Ablation Studies and Analysis: Joint STC-Pruner scoring over spatial and temporal anchors yields the most robust results because single-axis scoring misses either inter-frame novelty or intra-frame redundancy.The joint approach is designed to retain tokens important for complex reasoning.

5. Conclusion

STC is a plug-and-play framework that accelerates real-time streaming video understanding by optimizing both ViT encoding and LLM prefilling. Its two complementary modules reduce redundant computation and visual-token sequence length without retraining.

5. Conclusion: STC jointly optimizes ViT encoding and LLM prefilling through STC-Cacher and STC-Pruner for real-time streaming video understanding.The framework reduces redundant ViT computation and compresses tokens before they enter the LLM.
5. Conclusion: STC integrates into existing streaming VideoLLMs without retraining, providing a practical and scalable solution for latency-sensitive deployment.

Supplementary Material

The supplementary material expands the experimental documentation, ablations, implementation pseudocode, and visualizations for STC.

Supplementary Material: The appendix details benchmarks, model architectures, baseline methods, additional ablations, pseudocode, and visualizations.Additional analyses cover model backbones, cache update intervals, and hyperparameter sensitivity.

A. Detailed Experiment Settings

The detailed experiment settings describe the benchmarks, VideoLLM model families, and baseline frameworks used to evaluate STC. They cover streaming and offline-to-online video understanding configurations.

Detailed Experiment Settings: Evaluation uses OVO-Bench and StreamingBench for streaming understanding, plus EgoSchema, MLVU-dev, and VideoMME for long-video understanding.The benchmarks span temporal awareness, real-time interaction, and long-form comprehension tasks.
Detailed Experiment Settings: STC is evaluated on end-to-end online VideoLLMs and the offline-to-online ReKV framework.The online model descriptions include LiveCC and Dispider, while ReKV continuously processes streams and retrieves query-relevant KV caches.
Detailed Experiment Settings: ReKV separates continuous video encoding from question answering by storing processed KV caches and retrieving only query-relevant context.Its sliding-window attention reduces encoding overhead during streaming.

B. Additional Ablation Studies

Additional ablations examine STC across model architectures, feature-reuse choices, dynamic-token criteria, and cache-update intervals. They show that frequent cache updates are important for preserving performance over changing video content.

B.2. Additional Ablation Studies: The additional ablation studies include comparisons across model architectures and feature-reuse choices in STC-Cacher.The experiments compare attention and MLP features and evaluate alternative feature types for dynamic-token selection.
B.2. Additional Ablation Studies: Frequent cache updates yield better performance, while a static cache causes a sharp drop consistent with feature drift.The ablation varies the update interval from frame-by-frame updates to no updates.

B.2.1. Token Retention Ratios

Token-retention ablations evaluate Cacher and Pruner ratios and the balance between spatial and temporal relevance. Performance is generally robust, but frequent refreshing and sufficient spatial emphasis better preserve dynamic and semantic information.

B.2.1. Token Retention Ratios: Higher Cacher update ratios generally improve performance by refreshing more tokens and preserving dynamic information during rapid changes.The analysis evaluates the Cacher update ratio R_Cacher in Table 9(a).
B.2.1. Token Retention Ratios: Performance remains robust across joint Cacher and Pruner ratio combinations, indicating that the framework does not depend on narrow hyperparameter tuning.The joint effect of R_Cacher and R_Pruner is analyzed in Table 9(b).
B.2.1. Token Retention Ratios: Balanced or slightly spatial-weighted scores perform best, whereas heavily emphasizing temporal scores degrades performance.The comparison varies α between atemporal and aspatial scores.

D. More Visualizations by STC-Cacher

STC-Cacher concentrates computation on temporally dynamic visual regions while reusing cached representations for static content. Its selective-computation procedure uses reference frames, dynamic-token selection, and cached updates across ViT layers.

D. More Visualizations by STC-Cacher: STC-Cacher concentrates computational overhead on temporally significant regions while compressing computation for visually static tokens.The visualization illustrates the intended allocation of computation across changing and unchanged regions.
D. More Visualizations by STC-Cacher: Reference frames receive a full ViT forward pass, and their intermediate representations are cached for subsequent-frame reuse.The algorithm initializes an output sequence and performs full computation at reference-frame intervals.
D. More Visualizations by STC-Cacher: For non-reference frames, STC-Cacher identifies dynamic tokens and applies selective attention and MLP updates instead of recomputing every token.The procedure uses layer-wise dynamic-token selection with scatter updates before appending the resulting tokens.
D. More Visualizations by STC-Cacher: STC-Pruner then scores visual tokens using spatial and temporal dissimilarity and retains the top-ranked subset for the LLM.Its retention count is determined by the pruning ratio, while history is updated for the next step.