FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging

Ziyang Fan; Keyu Chen; Ruilong Xing; Yulin Li; Li Jiang; Zhuotao Tian

Source-linked AI summary

FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging

Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, Zhuotao Tian

arXiv:2602.08024v1Feb 8, 2026cs.CVcs.AIcs.CLcs.LG

TL;DR

VLLMs process many visual tokens, creating computational inefficiency, while independently compressing spatial and temporal redundancy misses video’s dynamic spatiotemporal relationships. FlashVID combines representative token selection with tree-based joint merging; it preserves high accuracy under aggressive compression and enables substantially longer videos within the same budget.

Problem
VLLMs incur substantial computational and memory overhead from large visual-token sequences, while existing methods overlook coupled spatial-temporal redundancy in dynamic videos.
Method
FlashVID is a training-free framework combining ADTS for representative token filtering with TSTM for fine-grained spatiotemporal redundancy elimination.
Results
Across three VLLMs and five benchmarks, FlashVID retains 99.1% relative accuracy to vanilla LLaVA-OneVision while pruning 90% of visual tokens.
Takeaways & Limitations
FlashVID functions as a plug-and-play module that enables significantly longer video sequences under constrained computational budgets.
Takeaways & Limitations
Constraining redundancy-tree depth and breadth to prevent merging spatiotemporally distant tokens yielded negligible gains, so no such constraints are applied in practice.

Abstract

from arXiv · show

Although Video Large Language Models (VLLMs) have shown remarkable capabilities in video understanding, they are required to process high volumes of visual tokens, causing significant computational inefficiency. Existing VLLMs acceleration frameworks usually compress spatial and temporal redundancy independently, which overlooks the spatiotemporal relationships, thereby leading to suboptimal spatiotemporal compression. The highly correlated visual features are likely to change in spatial position, scale, orientation, and other attributes over time due to the dynamic nature of video. Building on this insight, we introduce FlashVID, a training-free inference acceleration framework for VLLMs. Specifically, FlashVID utilizes Attention and Diversity-based Token Selection (ADTS) to select the most representative tokens for basic video representation, then applies Tree-based Spatiotemporal Token Merging (TSTM) for fine-grained spatiotemporal redundancy elimination. Extensive experiments conducted on three representative VLLMs across five video understanding benchmarks demonstrate the effectiveness and generalization of our method. Notably, by retaining only 10% of visual tokens, FlashVID preserves 99.1% of the performance of LLaVA-OneVision. Consequently, FlashVID can serve as a training-free and plug-and-play module for extending long video frames, which enables a 10x increase in video frame input to Qwen2.5-VL, resulting in a relative improvement of 8.6% within the same computational budget. Code is available at https://github.com/Fanziyang-v/FlashVID.

1 INTRODUCTION

VLLMs face substantial computational overhead from large visual-token sequences, while existing video compression methods overlook dynamic spatiotemporal relationships. FlashVID addresses this gap with representative token selection and joint spatiotemporal merging, achieving strong efficiency and accuracy across models and benchmarks.

Large visual-token sequences impose substantial computational and memory overhead because attention scales quadratically with sequence length.
Video compression remains challenging because frame-wise strategies neglect temporal dynamics and correlations alongside within-frame spatial redundancy.
FlashVID combines ADTS for representative token selection with TSTM for joint spatial-temporal redundancy modeling and fine-grained merging.
99.1% relative accuracy to vanilla LLaVA-OneVision is retained while pruning 90% of visual tokens across five benchmarks and three VLLMs.
FlashVID is training-free and improves inference efficiency with negligible performance drop while supporting longer input sequences within constrained computational budgets.

2 BACKGROUND AND MOTIVATION

VLLM efficiency is constrained by visual-token processing, especially because visual tokens dominate sequence length and attention and FFNs drive backbone cost. Existing compression paradigms and rigid correspondence assumptions motivate FlashVID’s unified, structure-aware spatiotemporal design.

Efficiency bottleneck analysis: Visual tokens typically dominate VLLM sequence length, making their compression necessary for efficient inference.
Efficient inference paradigms: VLLM acceleration frameworks use Before-LLM, Inner-LLM, or Hybrid Compression, with hybrid methods balancing information retention and computational savings.
FlashVID overview: FlashVID integrates ADTS for informative, diverse token selection with TSTM for structure-aware spatiotemporal redundancy reduction.
Key observations: Semantically consistent elements can shift in position, scale, or appearance, making fixed spatial correspondence unreliable across frames.
Key observations: Spatial and temporal redundancy are coupled because redundant regions within a frame can persist across multiple frames.

3 METHODOLOGY

FlashVID compresses video tokens in two complementary stages: ADTS selects informative and diverse tokens within each frame, while TSTM merges redundancy across spatial and temporal dimensions.

Tree-based Spatiotemporal Token Merging: TSTM constructs spatiotemporal redundancy trees to model redundancy across frames and within frames.The method computes adjacent-frame feature similarity and progressively builds hierarchical token relationships.
Tree-based Spatiotemporal Token Merging: Tokens link to their most similar counterparts when similarity exceeds a merging threshold, then tokens within each tree are aggregated.Aggregation produces compact spatiotemporal representations while avoiding merges between dissimilar tokens.
Tree-based Spatiotemporal Token Merging: Constraining redundancy-tree depth and breadth yielded negligible gains, so FlashVID applies no such constraints in practice.The authors note that redundancy-tree quality remains critical for fine-grained compression.
Attention and Diversity-based Token Selection: ADTS selects spatiotemporally informative tokens by combining diversity with [CLS] attention and event-relevance calibration.It formulates frame-wise selection as a calibrated Max-Min Diversity Problem.
Attention and Diversity-based Token Selection: Frame-wise diversity is computed from pairwise feature dissimilarities, but diversity alone may miss the most informative visual tokens.The pairwise distance matrix supports diverse subset selection through the maximal minimum distance criterion.
Attention and Diversity-based Token Selection: [CLS] attention highlights informative tokens, while event relevance emphasizes tokens correlated with the current video context.For encoders without an explicit [CLS] token, attention information can be derived from the attention matrix.
Overall Compression Pipeline: FlashVID first selects informative tokens with ADTS and then merges redundant tokens through TSTM to produce compact visual features.The two-stage pipeline is summarized in Algorithm 1 and is designed to preserve critical visual information during compression.

4 EXPERIMENTS

Experiments across multiple benchmarks and VLLM architectures show that FlashVID consistently improves performance and efficiency across retention ratios. Ablations further support combining ADTS with TSTM for effective token compression.

Experimental Setup: FlashVID is evaluated across five video understanding benchmarks and three representative VLLMs with distinct architectures.The benchmarks cover varied video durations and complex scenarios, supporting evaluation of effectiveness and generalization.
Main Results: FlashVID consistently outperforms previous state-of-the-art methods across different retention ratios on LLaVA-OneVision and LLaVA-Video.On LLaVA-OneVision, it preserves 99.1% of vanilla accuracy at R = 10% and surpasses the full-token vanilla model at R ∈{25%, 20%, 15%}.
Main Results: Under a fixed token budget, FlashVID enables Qwen2.5-VL to process 10× more video frames while improving overall performance by 8.6%.Token compression allows longer temporal context than the vanilla 16-frame baseline within the same computational and memory budget.
Main Results: FlashVID significantly surpasses previous methods on Qwen2.5-VL under various retention ratios, demonstrating generalization across VLLMs.The comparison uses models with significantly different architectures and characteristics.
Ablation Studies: ADTS outperforms attention-only and diversity-only token selection, while combining [CLS] attention and event relevance calibration yields the best performance.These ablations compare ADTS with ATS and DTS and separately evaluate its calibration terms.
Ablation Studies: The best performance occurs at α = 0.7, showing that balanced integration of ADTS and TSTM is preferable to either module alone.Here α controls the retained-token ratio between ADTS and TSTM; α = 0 and α = 1 represent TSTM-only and ADTS-only settings.
Efficiency: FlashVID preserves 99.1% relative accuracy at R = 10% while delivering 6.3× prefilling and 2.1× TTFT speedups.The efficiency experiment uses LLaVA-OneVision on a single NVIDIA A100 GPU and compares against FastVID on VideoMME.

5 CONCLUSION

FlashVID is a training-free, plug-and-play framework that combines representative token filtering with fine-grained spatiotemporal redundancy elimination, improving VLLM efficiency and accuracy while enabling longer video sequences under constrained computation.

FlashVID combines Attention and Diversity-based Token Selection with Tree-based Spatiotemporal Token Merging to compress visual tokens while preserving essential visual information.The framework performs representative token filtering followed by fine-grained redundancy elimination.
Experiments across three VLLMs and five video understanding benchmarks show superior efficiency and accuracy.
FlashVID functions as a plug-and-play module for processing significantly longer video sequences under a constrained computational budget.

Supplementary Material

The supplementary material extends the evaluation across Qwen2.5-VL and additional experiments, including fixed-token-budget comparisons and ablations.

Table 7 compares state-of-the-art methods on Qwen2.5-VL, highlighting the best performance among methods with similar retention ratios.
Qwen2.5-VL comparisons under a fixed token budget report performance when processing 10× video frames, with an 8.6% overall improvement within the same computational memory budget.
Supplementary experiments evaluate FlashVID against prior methods on Qwen2.5-VL and provide additional results on LLaVA-Video.

A.1 ADDITIONAL EXPERIMENTS ON QWEN2.5-VL

Additional Qwen2.5-VL experiments show that FlashVID generalizes across retention ratios and benefits from extending temporal context under a fixed token budget.

FlashVID consistently surpasses previous state-of-the-art methods on Qwen2.5-VL across tested retention ratios.Results are reported for retention ratios R ∈ {20%, 10%}, with additional settings in the supplementary tables.
Under a fixed token budget, evaluating 5× and 10× frames shows that longer temporal context can be obtained without increasing computational cost.
Table 10 reports LLaVA-Video prefilling time and Time-To-First-Token in milliseconds for efficiency analysis.

A.2 ADDITIONAL EXPERIMENTS ON LLAVA-VIDEO

Additional LLaVA-Video experiments show strong performance across retention ratios and substantial inference-speed improvements over vanilla processing.

FlashVID consistently outperforms previous state-of-the-art methods on LLaVA-Video across different retention ratios.
5.3× prefilling and 1.9× Time-To-First-Token speedups are achieved over vanilla LLaVA-Video while maintaining 95.9% relative performance.

A.3 ADDITIONAL ABLATION STUDIES

The ablations examine TSTM’s merging threshold, tree constraints, and expansion factor. Results show that threshold-based control and a moderate expansion factor matter more than added depth or breadth constraints.

Merging threshold: Increasing Tτ weakens merging and preserves temporal details, whereas lowering Tτ strengthens compression but may merge less-correlated tokens and introduce noise.Tτ directly controls TSTM’s compression strength.
Tree constraints: Depth and breadth constraints do not improve performance in the reported LLaVA-OneVision ablations.The authors hypothesize that Tτ provides a similar effect.
Expansion factor: 99.1% relative accuracy is achieved when the expansion factor fe ∈{1.25, 1.30}; fe = 1.25 is selected for better efficiency.A large fe may be inefficient, while a low value may lose critical information.

B EVALUATION BENCHMARKS

The evaluation uses five video understanding benchmarks spanning varied durations, tasks, and temporal reasoning demands. The section also presents the token-merging algorithm’s inputs, constraints, and compressed output.

Benchmark suite: Five benchmarks evaluate video understanding across diverse durations and complex scenarios: VideoMME, EgoSchema, LongVideoBench, MVBench, and MLVU.The benchmark set is intended to assess effectiveness and generalization.
Benchmark suite: VideoMME contains 900 videos across six domains and 30 subcategories, with durations from 11 seconds to 1 hour.It includes 2,700 human-annotated multiple-choice question-answer pairs.
Benchmark suite: LongVideoBench contains 3,763 videos lasting 8 seconds to 1 hour and 6,678 questions requiring referring-reasoning over multimodal contexts.Questions span 17 fine-grained perception and relation types.
TSTM algorithm: The TSTM algorithm takes token sequences, a similarity function, a merging threshold, and tree constraints, then returns a compressed token set.It initializes tokens as roots, constructs candidate parent links across frames, prunes backward, and aggregates redundancy trees.

C IMPLEMENTATION DETAILS

Implementation evaluates FlashVID across three representative VLLMs and uses video partitioning, ADTS, TSTM, and inner-LLM pruning. Frame transitions are computed from pooled features, while token budgets are aligned for fair comparisons.

Evaluation setup: Experiments evaluate LLaVA-OneVision, LLaVA-Video, and Qwen2.5-VL, with baselines reimplemented in LMMs-Eval.The models have distinct architectures and characteristics.
FlashVID components: FlashVID combines ADTS and TSTM with video partitioning and inner-LLM pruning.These design choices support efficiency while preserving visual information.
Video partition: Video partitioning groups consecutive similar frames into segments using cosine similarity between adjacent frame embeddings.Frame embeddings are obtained by global average pooling of video features; low transition similarity indicates a scene change.
Token selection: ADTS identifies spatiotemporally informative tokens by combining diversity selection with [CLS] attention and event-relevance calibration.The selection procedure solves a calibrated max-min diversity problem.
Token budget: FlashVID aligns average visual-token counts across Transformer layers to enforce comparable computational cost between acceleration methods.The expansion factor fe relates tokens entering the LLM to average retained tokens, while inner-LLM pruning controls later retention.

E.2 QUALITATIVE ANALYSIS ON LLAVA-ONEVISION

Qualitative analyses show that FlashVID can preserve or improve answers under compression and support longer temporal context within the same budget. They also expose calibration benefits and a TSTM failure mode involving semantic confusion.

TSTM failure cases: TSTM can merge tokens from different entities with similar semantic information, causing semantic confusion in failure cases.This is a limitation of fine-grained spatiotemporal redundancy compression.
LLaVA-OneVision comparison: At R = 25%, FlashVID sometimes answers correctly when full-token LLaVA-OneVision answers incorrectly, illustrating a “less is more” pattern.The comparison is qualitative and concerns selected examples.
Longer temporal context: Qwen2.5-VL processes 160 frames instead of 16 with FlashVID under the same computational budget, using longer temporal context for more accurate predictions.This corresponds to a 10× increase in frames.
ADTS analysis: Event relevance calibration helps ADTS identify key visual tokens and improves performance on tasks requiring fine-grained understanding.The comparison evaluates ADTS with and without this calibration.
Compression design: FlashVID uses a hybrid compression paradigm that retains more visual tokens before the LLM and prunes at a relatively high layer.The pruning layer is set to K = 20 for the evaluated 7B-scale models.