PaperlayerView exact revision on arXiv

Source-linked AI summary

HoliTom: Holistic Token Merging for Fast Video Large Language Models

Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, Huan Wang

arXiv:2505.21334v3cs.CV

TL;DR

Video LLMs face costly inference from redundant visual tokens, while inner- and outer-LLM pruning leave computational and global spatio-temporal gaps. HoliTom is a training-free framework combining global outer-LLM temporal and spatio-temporal merging with robust inner-LLM similarity-based merging. On LLaVA-OneVision-7B, it reduces computation to 6.9% of original FLOPs while preserving 99.1% of performance, and improves TTFT and decoding throughput.

  • Problem

    Redundant video tokens make video LLM inference computationally expensive, while existing inner- and outer-LLM pruning methods incompletely address global spatio-temporal redundancy and their integration.

  • Method

    HoliTom is a training-free holistic framework combining global redundancy-aware temporal segmentation and spatio-temporal merging before the LLM with similarity-based token merging inside it.

  • Results

    6.9% of original FLOPs preserves 99.1% of original performance on LLaVA-OneVision-7B, with 2.28× lower TTFT and 1.32× higher decoding throughput.

  • Takeaways & Limitations

    HoliTom achieves a state-of-the-art efficiency-performance trade-off for efficient video LLM inference across LLaVA-OneVision and LLaVA-Video.

  • Takeaways & Limitations

    HoliTom is primarily designed for fixed-length clips, does not natively support arbitrary-length online streaming, and does not optimize vision-tower latency.

Abstract

from arXiv · show

Video large language models (video LLMs) excel at video comprehension but face significant computational inefficiency due to redundant video tokens. Existing token pruning methods offer solutions. However, approaches operating within the LLM (inner-LLM pruning), such as FastV, incur intrinsic computational overhead in shallow layers. In contrast, methods performing token pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy within individual frames or limited temporal windows, neglecting the crucial global temporal dynamics and correlations across longer video sequences. This leads to sub-optimal spatio-temporal reduction and does not leverage video compressibility fully. Crucially, the synergistic potential and mutual influence of combining these strategies remain unexplored. To further reduce redundancy, we introduce HoliTom, a novel training-free holistic token merging framework. HoliTom employs outer-LLM pruning through global redundancy-aware temporal segmentation, followed by spatial-temporal merging to reduce visual tokens by over 90%, significantly alleviating the LLM's computational burden. Complementing this, we introduce a robust inner-LLM token similarity-based merging approach, designed for superior performance and compatibility with outer-LLM pruning. Evaluations demonstrate our method's promising efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational costs to 6.9% of FLOPs while maintaining 99.1% of the original performance. Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a 1.32x acceleration in decoding throughput, highlighting the practical benefits of our integrated pruning approach for efficient video LLMs inference.

1 Introduction

Video LLM inference is costly because long videos generate many redundant tokens, while existing pruning strategies incompletely address global spatio-temporal redundancy and their combination. HoliTom integrates outer- and inner-LLM merging, achieving substantial compression and improved inference efficiency while preserving performance.

  • Motivation: Tens of thousands of video tokens and quadratic LLM attention make long-video inference computationally expensive.Prior compression and pruning methods have not fully balanced efficiency and performance.
  • Limitations of Existing Methods: Inner-LLM pruning wastes computation in early layers, whereas outer-LLM methods mainly address spatial redundancy or limited temporal windows.Existing approaches therefore do not fully capture global video dynamics or comprehensive spatio-temporal redundancy.
  • HoliTom: HoliTom combines global redundancy-aware temporal merging and spatio-temporal reduction before the LLM with similarity-based merging inside it.The outer step reduces visual tokens to less than 10% and the inner method is designed for integration with it.
  • Evaluation: 6.9% of original FLOPs preserves 99.1% of original performance, with 2.28× lower TTFT and 1.32× higher decoding throughput on LLaVA-OneVision-7B.These measurements quantify the framework’s efficiency-performance trade-off and inference acceleration.
  • Evaluation: HoliTom’s evaluations on LLaVA-OneVision and LLaVA-Video demonstrate a state-of-the-art efficiency-performance trade-off.The framework reduces computational costs and accelerates inference while preserving model performance.

2 Related Work

Video LLMs and multimodal language models reduce or organize visual tokens through aggregation, pruning, and token-compression strategies. Prior work targets spatial, temporal, or inner-LLM redundancy, motivating broader compression approaches.

  • Video Token Compression: Video LLMs improve video understanding and question answering, but numerous frame-derived tokens hinder inference efficiency and adoption.Existing examples include VideoLLaMA’s Q-Former aggregation and MovieChat’s video-token processing approach.
  • Token Compression: ToMe merges similar vision-transformer tokens, while TempMe merges adjacent video clips to reduce spatial and temporal redundancy.These methods represent separate token-compression strategies for visual data.
  • Token Compression: TESTA combines temporal and spatial aggregation modules and reports up to a 75% reduction in processed tokens.FastV and TopV instead prune unnecessary visual tokens within multimodal language-model processing.

3 Method

HoliTom reduces video-LLM inference cost through temporal segmentation and merging before the LLM, spatial selection and clustering, and similarity-based merging within the LLM. Its pipeline preserves informative content while compressing redundant visual tokens across multiple stages.

  • Background on Video LLMs Inference: Video LLM inference processes visual and text tokens before prefilling and autoregressive decoding with a KV cache.The KV cache stores key-value pairs during prefilling and updates them during decoding.
  • Global Redundancy-Aware Temporal Merging: Temporal merging detects persistent feature similarity across consecutive frames and merges later redundant occurrences into their first appearance.A feature is temporally redundant when its normalized similarity exceeds threshold τ.
  • Global Redundancy-Aware Temporal Merging: Dynamic programming selects consecutive video segments that maximize the total number of prunable features.The optimal segmentation is reconstructed by backtracking from the final dynamic-programming state.
  • Spatial Merging: For non-redundant temporal tokens, attention-based selection reshapes token importance onto the spatial grid and retains the highest-scoring representatives.Importance is computed from average within-frame attention received by each token.
  • Spatial Merging: For redundant temporal tokens, density-peak clustering selects centers, averages assigned features, and concatenates compressed and non-redundant features in spatial order.The clustering procedure preserves positional characteristics while reducing redundant features.
  • Inner-LLM Merging: Inner-LLM merging combines information from similar visual tokens instead of directly discarding them, reducing tokens while mitigating information loss.This selective merging is designed to preserve relevant features that pruning would otherwise remove.

4 Experimental Results

Experiments across video benchmarks, backbones, compression settings, frame counts, and ablations show that HoliTom provides a strong efficiency-performance trade-off. Its integrated temporal, spatial-temporal, and inner-LLM merging improves practical inference efficiency while preserving performance under aggressive compression.

  • Experimental setup: HoliTom is evaluated on MVBench, EgoSchema, LongVideoBench, and VideoMME, covering varied video lengths and complex scenarios.These benchmarks are used to assess effectiveness and generalization.
  • Experimental setup: HoliTom is compared with six training-free baselines, including FastV, PDrop, VisionZip, DyCoke, and PruneVid.The comparison spans inner-LLM and outer-LLM token compression approaches.
  • Inference cost: 2.9 TFLOPs are incurred by FastV's first two shallow layers in LLaVA-OneVision-7B before pruning can substantially reduce computation.The decoding phase contributes approximately 2% of total FLOPs, so the experiments primarily optimize prefilling.
  • Main results: 11.3% FLOPs with 96% performance is achieved on LLaVA-OneVision-72B, while LLaVA-Video-7B reaches 9.5% FLOPs with 95.8% performance.The results indicate that aggressive compression remains effective across backbones, although it is harder on these models than on LLaVA-OneVision-7B.
  • Main results: HoliTom outperforms FastV when pruning 50% at shallower layers and across varying pruning rates at equivalent layers after outer-LLM compression.The inner-LLM merging strategy preserves information more effectively than directly discarding tokens in this compressed context.
  • Scaling with more frames: 64 frames yield superior performance over the vanilla model, while 128 frames avoid vanilla-model context-length bottlenecks and support longer temporal context.At 16 frames, HoliTom remains slightly below vanilla but exceeds the other compression methods.
  • Efficiency results: 2.28× lower TTFT and 1.32× higher decoding throughput are achieved, while prefilling time falls to 13.7% of the original.Token preprocessing and variable tokens per frame introduce overhead, but HoliTom still attains the strongest reported inference gains.
  • Ablation studies: 100.7% baseline performance at 75.7% FLOPs is obtained by the temporal merging module across four benchmarks.The ablation reports that temporal merging preserves performance while reducing computation and helps mitigate degradation from aggressive spatial pruning.

5 Conclusion

HoliTom is a training-free holistic token merging framework that combines outer-LLM spatio-temporal reduction with compatible inner-LLM token merging. Across prominent video LLMs, it reduces computation and inference latency while preserving high performance for complex, long-form video understanding.

  • Conclusion: HoliTom integrates outer-LLM spatio-temporal reduction with robust inner-LLM token merging to handle redundant visual tokens.The framework is designed as a synergistic, training-free approach for video LLM inference.
  • Conclusion: 6.9% FLOPs, 99.1% accuracy, 2.28× TTFT reduction, and 1.32× throughput are reported for HoliTom.These results summarize its reported efficiency-performance trade-off and inference acceleration.

A Supplemental Implementation Details

The implementation evaluates HoliTom on LLaVA-OneVision and LLaVA-Video models using separate hardware for evaluation and inference. Computational comparisons use total prefilling FLOPs with baselines configured for comparable cost.

  • Implementation: HoliTom is implemented on LLaVA-OneVision-7B/72B and LLaVA-Video-7B, with evaluation on NVIDIA A100 GPUs and inference on an RTX A6000.These model and hardware settings define the reported experimental implementation.
  • Implementation: Total prefilling FLOPs are used as the primary computational-cost metric, with baselines configured for comparable FLOPs.The implementation details specify baseline-specific pruning configurations for fair comparison.

B.1 Experiments on Qwen2.5-VL with High Frame Sampling

Supplementary experiments with Qwen2.5-VL address higher frame counts and dynamic-resolution settings that challenge existing compression methods. HoliTom maintains strong performance across compression rates and benefits from processing more frames under controlled computational cost.

  • Experimental setting: Qwen2.5-VL supports FPS sampling up to 768 frames and dynamic resolution, creating longer input sequences for token compression methods.These capabilities differ from the fixed 32/64-frame inputs and static resolutions used by LLaVA-OV and LLaVA-Video.
  • Main results: HoliTom surpasses state-of-the-art methods across both token compression rates, especially for long videos.FastV and inner-LLM merging results are unavailable because full attention matrix computation caused out-of-memory errors.
  • Temporal pruning: MVBench has the highest temporal pruning ratio at τ = 0.65, while VideoMME has the lowest among the four benchmarks.The histogram annotates each benchmark's average temporal pruning ratio.
  • Fine-grained details: 10% to 25% compression rates improve performance on selected MVBench object subtasks.The reported subtasks are object existence, object interaction, and object shuffle.
  • Frame scaling: More input frames improve performance against a vanilla model with fewer frames while maintaining constant or reduced total FLOPs.As frame count rises, vision-encoder overhead becomes a non-negligible part of computation.

B.4 Supplemental Ablation Study on τ

The ablation examines how the threshold τ affects performance at a 10% pre-LLM retention ratio. Peak performance occurs around τ = 0.65.

  • Peak performance occurs around τ = 0.65 at a 10% pre-LLM retention ratio.The corresponding temporal-pruning-rate histogram is reported across four datasets.

B.5 Ablation Study on Merge Strategy

HoliTom combines attention-guided compression with similarity-guided DPC-KNN clustering in its spatio-temporal merging strategy. The mixed strategy achieves the best performance and is designed to integrate with outer-LLM pruning.

  • Merge Strategy: The mixed merge strategy achieves the best performance in the ablation study.
  • Merge Strategy: Attention-guided compression handles non-redundant tokens within frames, while DPC-KNN clusters redundant tokens formed from adjacent frames by feature similarity.Single-frame self-attention scores are treated as unsuitable merging metrics for tokens already merged across frames.
  • Merge Strategy: The inner-LLM merging strategy requires attention scores from one prefilling layer and is designed to work with optimized attention implementations such as Flash Attention.The paper describes this one-time attention-score operation as introducing negligible overhead relative to total inference cost.

D Limitations and Future Work

HoliTom is primarily designed for fixed-length video clips and does not natively support online, arbitrary-length streaming input. It also leaves vision-tower latency unoptimized, motivating further work on streaming, quantization, and vision-tower acceleration.

  • Limitations: HoliTom does not natively support online, arbitrary-length streaming video input.This constrains real-time processing and long-term context understanding.
  • Limitations: The approach does not optimize the latency of the vision tower.The paper identifies quantization and vision-tower acceleration methods as directions for further optimization.
  • Future Work: The paper connects reduced computational needs with broader access to advanced video AI and wider application.

F More Visualizations

The visualizations compare HoliTom and other methods on challenging video questions and qualitative generation. HoliTom retains accurate answers and detailed descriptions under substantial token reduction, including when retaining only 15% of input tokens.

  • Challenging Video Understanding: At 7.1T FLOPs, HoliTom identifies the stationary object as purple and the green ball as moving left.The displayed comparison includes LLaVA-OV-7B, FastV, VisionZip, and HoliTom without merging.
  • Challenging Video Understanding: HoliTom produces correct answers on challenging video understanding tasks.
  • Qualitative Generation: HoliTom generates high-quality, accurate detailed descriptions while retaining only 15% of input tokens.
Loading 2505.21334v3…