PaperlayerView exact revision on arXiv

Source-linked AI summary

FastVID: Dynamic Density Pruning for Fast Video Large Language Models

Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, Guiguang Ding

arXiv:2503.11187v3cs.CV

TL;DR

Video LLM inference is costly because video-token volume creates substantial redundancy, while existing approaches do not adequately exploit spatiotemporal redundancy. FastVID segments videos temporally and prunes tokens by density within segments; it reports state-of-the-art performance across models and benchmarks, with substantial compression on LLaVA-OneVision-7B.

  • Problem

    Video-token volume drives substantial inference costs, and prior image-compression methods do not exploit temporal dependencies across frames.

    Source: §1 · p. 1

  • Method

    FastVID dynamically partitions videos into temporally ordered segments and applies density-based token pruning within each segment.

    Source: §1 · p. 2

  • Results

    FastVID achieves state-of-the-art performance across Video LLMs and benchmarks; on LLaVA-OneVision-7B it prunes 90.3% of video tokens, reduces FLOPs to 8.3%, accelerates prefill 7.1×, and maintains 98.0% of original accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FastVID maintains high performance under extreme compression, supporting practical deployment of fast Video LLMs.

    Source: §5 · p. 10

    On LLaVA-Video, Qwen2-VL, and Qwen2.5-VL, FastVID outperforms existing state-of-the-art baselines but shows a noticeable accuracy drop versus the original models.

    Source: §C.2 · p. 18

Abstract

from arXiv · show

Video Large Language Models have demonstrated strong video understanding capabilities, yet their practical deployment is hindered by substantial inference costs caused by redundant video tokens. Existing pruning techniques fail to effectively exploit the spatiotemporal redundancy present in video data. To bridge this gap, we perform a systematic analysis of video redundancy from two perspectives: temporal context and visual context. Leveraging these insights, we propose Dynamic Density Pruning for Fast Video LLMs termed FastVID. Specifically, FastVID dynamically partitions videos into temporally ordered segments to preserve temporal structure and applies a density-based token pruning strategy to maintain essential spatial and temporal information. Our method significantly reduces computational overhead while maintaining temporal and visual integrity. Extensive evaluations show that FastVID achieves state-of-the-art performance across various short- and long-video benchmarks on leading Video LLMs, including LLaVA-OneVision, LLaVA-Video, Qwen2-VL, and Qwen2.5-VL. Notably, on LLaVA-OneVision-7B, FastVID effectively prunes $\textbf{90.3%}$ of video tokens, reduces FLOPs to $\textbf{8.3%}$, and accelerates the LLM prefill stage by $\textbf{7.1}\times$, while maintaining $\textbf{98.0%}$ of the original accuracy. The code is available at https://github.com/LunarShen/FastVID.

1 Introduction

FastVID addresses the inference cost of video-token volume by targeting temporal and visual redundancy without additional training. It preserves temporal structure through ordered segmentation and visual context through density-based token selection, achieving strong compression with accuracy retention.

1 Introduction

  • High video-token volume drives substantial inference costs, while prior image-compression methods do not exploit temporal dependencies across frames.The paper analyzes redundancy from temporal-context and visual-context perspectives.Source: §1 · p. 1
  • FastVID segments videos into temporally ordered groups and prunes within, but not across, high-redundancy segments to preserve temporal structure.Disrupting frame order or omitting frames can lead to incorrect comprehension.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Density-based sampling selects tokens that are both representative and distinctive, preserving segment visual context while reducing redundancy.Density peaks act as anchors, unlike content-agnostic uniform sampling that can merge important details with background.Source: §1 · p. 2
  • FastVID combines Dynamic Temporal Segmentation with Density Spatiotemporal Pruning to reduce spatiotemporal redundancy while retaining essential semantics.The framework is designed for inference-time acceleration without additional training.Source: §1 · pp. 1–2
  • 90.3% of video tokens are pruned on LLaVA-OneVision-7B, with FLOPs reduced to 8.3%, prefill accelerated 7.1×, and 98.0% of original accuracy maintained across benchmarks.FastVID achieves state-of-the-art performance across four Video LLMs and four benchmarks.Source: §1 · p. 2

2 Related Work

Prior work spans general Video LLM architectures, training-time compression, and token-reduction methods developed for images or videos. The paper focuses on inference-time acceleration for general Video LLMs, where efficient pruning under large token reduction remains unresolved.

2 Related Work

  • General Video LLMs process raw video tokens or apply pooling; Video-LLaVA [22] uses shared projection layers, while LLaVA-OneVision [19] transfers from images and LLaVA-Video [57] uses synthetic instruction data.Some models add video positional designs, including newline tokens in LLaVA-Video [57].Source: §2 · pp. 2–3
  • Training-time compression methods reduce video-token counts through approaches such as progressive clustering in Chat-UniVi [18], cross-modal and interframe reduction in LongVU [37], and Perceiver Resampler [17] in Apollo [60].These methods aim to enable longer video sequences.Source: §2 · p. 3
  • The paper targets inference-time acceleration for general Video LLMs, which remain the dominant paradigm and include widely adopted LLaVA-OneVision and Qwen-VL models.Earlier token-compression approaches include fixed spatial-token merging in ToMe [5] and neighboring-clip merging in TempMe [35].Source: §2 · p. 3
  • Image-focused methods reduce spatial tokens with text relevance, attention pruning, token recycling, or visual-encoder reduction, but overlook temporal relationships across frames.Examples include FastV, LLaVA-PruMerge [32], SparseVLM [56], and VisionZip [51].Source: §2 · p. 3
  • Video compression methods include successive-layer merging and pruning in FrameFusion [13], cross-frame merging and KV-cache reduction in DyCoke [41], and clustering in PruneVID [16].The paper reports efficiency, token-reduction, or latency limitations for these approaches, leaving efficient and accurate pruning at large reductions unresolved.Source: §2 · p. 3

3 Methodology

FastVID reduces video tokens before LLM processing by segmenting videos into temporally ordered, high-redundancy groups and pruning within each segment. Its components combine density-based merging with attention-based selection to retain visual context and salient details while preserving temporal and visual integrity.

3.1 FastVID

  • FastVID reduces video tokens at the prefill stage and is designed for compatibility with FlashAttention, KV cache, multi-turn conversations, and plug-and-play integration.The paper presents this as a practical way to improve computational efficiency in existing Video LLMs.Source: §3.1 · p. 3
  • The pipeline applies Dynamic Temporal Segmentation to form ordered, high-redundancy segments, then performs Density Spatiotemporal Pruning within each segment before passing retained tokens and query tokens to the LLM.The pruning stage combines Density-based Token Merging (DTM) and Attention-based Token Selection (ATS).Source: §3.1 · pp. 3–4, §3.3 · p. 5

3.2 Dynamic Temporal Segmentation

  • DySeg adapts segment boundaries to video complexity, aiming to preserve temporal order and high within-segment similarity while using coarser partitions for simple scenes and finer ones for complex scenes.Fixed-interval segmentation can group visually dissimilar frames, while clustering can impose a fixed cluster count and disrupt temporal order.Source: §3.2 · p. 4
  • DySeg marks boundaries at the c−1 least-similar adjacent-frame transitions and at any transitions whose similarity falls below τ.The least-similar transitions provide finer segmentation when the threshold selects none; for complex videos, they are typically a subset of threshold-selected transitions.Source: §3.2 · p. 4

3.3 Density Spatiotemporal Pruning

  • STPrune retains rPN tokens per segment, allocating drPN to DTM and (1−d)rPN to ATS.Here P is the number of frames in the segment, N is the number of tokens per frame, r is the retention ratio, and d controls the allocation between modules.Source: §3.3 · p. 5
  • DTM selects density-peak tokens from sampled anchor frames, assigns segment tokens to their nearest anchors by cosine similarity, and aggregates assigned tokens into those anchors.The density score is ρ_i × δ_i; high-scoring tokens serve as representative and distinctive anchors.Source: §3.3 · p. 5
  • DTM updates anchor tokens while retaining their original positional information, which the paper identifies as important for preserving spatiotemporal structure.Anchor-centric aggregation is also described as emphasizing representative tokens and enhancing feature representation.Source: §3.3 · pp. 5–6
  • ATS selects the highest-[CLS]-attention tokens to preserve critical visual details, using a pretrained SigLIP head because the usual Video LLM encoder output omits the head that generates [CLS].The head has 15.2M parameters, which the paper describes as minimal computational overhead compared with the full Video LLM.Source: §3.3 · p. 6

4 Experiments

FastVID is evaluated across multiple video models and benchmarks, with comparisons, efficiency measurements, ablations, and qualitative analysis. The experiments report strong performance under compression and examine how segmentation and token-pruning choices affect results.

4.1 Experimental Settings

  • Evaluations cover MVBench, LongVideoBench, MLVU, and VideoMME, using LLaVA-OneVision, LLaVA-Video, Qwen2-VL, and Qwen2.5-VL.The experiments use different frame and retention settings across model families and compare against image- and video-compression baselines.Source: §4.1 · pp. 6–7

4.2 Comparisons with State-of-the-Art Methods

  • After pruning 90.3% of tokens, FastVID preserves 98.0% of vanilla LLaVA-OneVision performance; it achieves a 7.1× speedup while preserving 98.0% accuracy.The efficiency comparison measures prefill latency on VideoMME using an A100 GPU.Source: §4.2 · pp. 7–8, Table 5 · p. 8
  • FastVID outperforms baselines on LLaVA-Video and Qwen2-VL and Qwen2.5-VL; gains include 2.4 accuracy on Qwen2-VL’s long subset and a 3.5 improvement over PruneVID at comparable retention on Qwen2.5-VL.The comparisons span distinct model architectures and retention settings.Source: Table 3 · p. 7, Table 2 · p. 7, §4.2 · p. 8

4.3 Ablation Study

  • Fixed-interval segmentation outperforms cluster-based segmentation by 1.4%, while DySeg integrates temporal order and intra-segment similarity for 98.0% average performance.The comparison uses eight segments for each alternative segmentation method.Source: §4.3 · pp. 8–9, Table 8 · p. 8
  • At d = 0.4, combining DTM and ATS improves performance by 1.8 and 4.5 percentage points over ATS and DTM alone, respectively.ATS alone outperforms DTM alone, but the combined allocation performs best.Source: §4.3 · p. 9, Table 6 · p. 9
  • On LLaVA-Video, DTM beats other merging strategies by 2.9%; the SigLIP head improves ATS by 1.0, and FastVID generalizes to Qwen-VL without pretrained heads or [CLS] tokens.For Qwen-VL, attention scores are computed at the ViT final layer using pseudo [CLS] tokens formed by averaging patch tokens.Source: §4.3 · p. 9, Table 7 · p. 9, Table 9 · p. 9
  • FastVID extends sampled frames from 32 to 320 at a comparable total number of video tokens, consistently improving performance.The results indicate length extrapolation across different retention ratios.Source: Table 10 · p. 9, §4.3 · p. 10

4.4 Qualitative Results

  • DTM aggregates visually similar content across frames, while ATS-selected tokens capture salient [CLS] information but may lack broader visual context.Figure 5 visualizes a 40-token allocation: 16 tokens for DTM and 24 for ATS.Source: §4.4 · p. 10, Figure 5 · p. 10

5 Conclusion

The conclusion presents FastVID as an inference-time pruning framework that reduces spatiotemporal redundancy while retaining performance under extreme compression. The appendix outlines additional experimental material and a discussion of limitations and broader impacts.

5 Conclusion

  • FastVID dynamically partitions videos into temporally ordered segments and prunes tokens within each segment to reduce spatiotemporal redundancy while preserving essential semantics.The conclusion describes experiments across multiple Video LLMs and benchmarks as demonstrating generalization and effectiveness.Source: §5 · p. 10
  • The authors conclude that FastVID maintains high performance under extreme compression, supporting practical deployment of fast Video LLMs.Source: §5 · p. 10

Appendix

  • The appendix provides additional experimental settings and results, including baseline reproduction details, computational-cost estimation, hyperparameter ablations, and further quantitative and qualitative results.It also includes a discussion of limitations and potential broader impacts.Source: Appendix · p. 15

A Additional Experimental Settings

The appendix describes baseline-reproduction choices and gives a theoretical estimate of LLM FLOPs associated with video tokens. It also identifies the parameters examined in segmentation and token-merging ablations.

A.1 Reproduction Details of Compared Baselines

  • The reproduced baselines include FastV, an adapted VisionZip applied after pooling, DyCoke evaluated only at prefill, and input-only and full-prefill PruneVID variants.The text specifies each method’s implementation settings and comparison conditions.Source: §A.1 · p. 15

A.2 Computational Cost Estimation

  • Per-layer LLM FLOPs are estimated as 2nD(h_kv d) + 2nD^2 + 2n^2D + 3nDD′, using the number of video tokens and model dimensions.Here n is video-token count, D the hidden-state size, D′ the FFN intermediate size, h_kv the key/value-head count, and d the head dimension.Source: §A.2 · p. 15
  • Figure 6 examines c, the minimum segment count, and τ, the transition-similarity threshold, as joint controls of segmentation granularity.Source: Figure 6 · p. 16
  • Figure 7 examines p, the anchor-frame selection interval, and β, the merging weight for anchor tokens and their associated tokens.Source: Figure 7 · p. 16

B.1 Ablation study on c and τ in DySeg

The ablation varies DySeg’s minimum segment count c and transition-similarity threshold τ. Results favor moderate segmentation and a threshold that groups redundant frames while separating non-redundant ones.

B.1 Ablation study on c and τ in DySeg

  • Performance remains stable for 1 ≤ c ≤ 16, declines at c = 32, and is optimal at c = 8; τ = 0.9 achieves the best result by grouping redundant frames and separating non-redundant ones.At c = 1, segmentation relies solely on τ; at c = 32, each frame forms a segment.Source: §B.1 · p. 16

B.2 Ablation study on p and β in DTM

The ablation finds that p = 4 best captures spatiotemporal context, while anchor-centric assignment with β = 0.6 performs best. Additional tables report LLaVA-OneVision results on VideoMME and ANet-QA.

B.2 Ablation study on p and β in DTM

  • p = 4 yields the best performance by balancing the number of anchor frames against anchor-token allocation to capture spatiotemporal context.Larger p selects fewer anchor frames, and values that are too large draw all anchor tokens from the first frame, limiting temporal information.Source: §B.2 · p. 16
  • β = 0.6 gives optimal results with Anchor-centric Assignment, while β = 1.0 discards non-anchor tokens and removes segment visual context.At β = 0.0, matching tokens are averaged.Source: §B.2 · p. 16
  • Additional tables report LLaVA-OneVision overall performance on VideoMME and ANet-QA.Source: Table 11 · p. 17, Table 12 · p. 17

B.3 Additional Quantitative Results

FastVID is compared with pruning methods that retain full video-token processing in early layers and with LLaVA-PruMerge at a matched token budget. The passages report lower computation or better performance for FastVID in these comparisons.

B.3 Additional Quantitative Results

  • FastVID compresses tokens from the input stage and outperforms PyramidDrop under extreme pruning, whereas PyramidDrop incurs high FLOPs processing full video tokens in its first eight layers.PyramidDrop prunes at layers 8, 16, and 24 with retention ratios of 4.1%, 2.0%, and 1.0%, respectively.Source: §B.3 · p. 17
  • FastVID achieves lower computational cost and better performance than SparseVLM, which retains high FLOPs by processing all tokens in early layers.SparseVLM prunes at layers 3, 7, and 16 with retention ratios of 10.4%, 5.3%, and 2.9%, respectively.Source: §B.3 · p. 17
  • At the same 608-token budget (≈9.7% retention), FastVID significantly outperforms LLaVA-PruMerge.Both methods select tokens based on attention scores, while FastVID is designed to preserve temporal and visual context in video.Source: §B.3 · p. 17

B.4 Results for long-context generation

FastVID is evaluated for long-context generation on ANet-QA video captioning, with metrics computed using GPT-3.5-Turbo-0125.

B.4 Results for long-context generation

  • On ANet-QA video captioning, FastVID achieves 97.6% accuracy and 98.6% score, demonstrating effectiveness in long-context generation.The metrics are computed using GPT-3.5-Turbo-0125.Source: §B.4 · p. 17

B.5 Additional Qualitative Results

Qualitative examples show adaptive segmentation across different temporal dynamics, while the PruneVID comparison describes design differences and reports faster pruning and consistently better accuracy at similar compression ratios.

B.5 Additional Qualitative Results

  • Figures 8–10 use static and dynamic scenes to illustrate adaptive segmentation; merged-region boundaries loosely align with object shapes.The reported settings are c = 0, τ = 0.9, d = 0.4, p = 2, with an average of 10 tokens retained per frame.Source: §B.5 · p. 17
  • At high compression, query-agnostic pruning is less effective for detailed questions when relevant evidence occurs in only a few frames.The examples concern banana-eating and an entrance scene; many retained tokens may belong to query-irrelevant frames.Source: §B.5 · p. 17

C.1 Comparison with PruneVID

  • FastVID segments by frame-transition similarity and uses density-based merging that selects anchor tokens from anchor frames, adapts anchor-token counts to segment length, and preserves their positions.Anchor-centric aggregation emphasizes representative tokens; the positional preservation is noted as beneficial for Qwen-VL models using M-RoPE.Source: §C.1 · p. 18
  • FastVID pruning is 6.1× faster than PruneVID (6.1 ms vs. 37.2 ms), and it consistently outperforms PruneVID at similar compression ratios across the reported Video LLM settings.The comparisons include LLaVA-OneVision with fixed-frame sampling and Qwen-VL models with dynamic frame sampling.Source: §C.1 · p. 18

C.2 Limitations

FastVID retains comparable accuracy on LLaVA-OneVision with 8.3% of the FLOPs, but shows an accuracy drop against the original model on longer-frame LLaVA-Video and Qwen variants. Its query-agnostic pruning supports KV-cache reuse across dialogue turns, while inheriting potential societal impacts of the original models.

C.2 Limitations

  • With 768 frames per video, FastVID shows a noticeable accuracy drop against the original Qwen2-VL and Qwen2.5-VL models, despite outperforming existing SOTA baselines.On LLaVA-OneVision with 32 frames per video, it maintains comparable accuracy with 8.3% of the FLOPs.Source: §C.2 · p. 18
  • Aggressive query-agnostic pruning can retain few tokens relevant to the query over long temporal spans, contributing to performance drops.The authors suggest query-guided keyframe selection for long-frame Video LLMs as future work.Source: §C.2 · p. 18
  • The pruned KV cache can be reused across dialogue turns with different queries, making query-agnostic pruning more suitable for multi-turn conversations.Source: §C.2 · p. 18
  • The authors frame query-aware and query-agnostic methods as having different advantages, with selection depending on the application scenario.Source: §C.2 · p. 19

C.3 Broader Impacts

  • FastVID does not modify model parameters or train new models, but inherits potential societal impacts such as representational bias or misuse from the original models.The authors also describe its efficiency as supporting more practical deployment in resource-limited settings.Source: §C.3 · p. 19

Related papers

Loading 2503.11187v3…