PaperlayerView exact revision on arXiv

Source-linked AI summary

EarlyTom: Early Token Compression Completes Fast Video Understanding

Hesong Wang, Xin Jin, Lu Lu, Chenhaowen Li, Jian Chen, Qiang Liu, Huan Wang

arXiv:2605.30010v1cs.CV

TL;DR

Processing massive visual-token volumes limits practical Video-LLM deployment, while existing compression largely overlooks the vision encoder. EarlyTom compresses tokens inside the encoder with decoupled spatial selection, reducing TTFT by up to 2.65× and FLOPs by 61% while maintaining comparable accuracy to the full-token baseline.

  • Problem

    Existing Video-LLM compression methods largely overlook the vision encoder, which accounts for a major portion of TTFT and limits efficient deployment.

  • Method

    EarlyTom is a training-free framework that merges frames inside the vision encoder and applies decoupled spatial token selection for low-latency compression.

  • Results

    Up to 2.65× lower TTFT and 61% fewer FLOPs are achieved while maintaining accuracy comparable to the full-token baseline.

  • Takeaways & Limitations

    EarlyTom substantially improves Video-LLM inference efficiency while preserving comparable accuracy, supporting more practical real-world deployment.

Abstract

from arXiv · show

Video large language models (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.

1. Introduction

EarlyTom addresses Video-LLM deployment inefficiency by compressing visual tokens inside the vision encoder, where encoding contributes substantially to TTFT. Its frame merging and decoupled token selection reduce latency and visual-token count while preserving comparable accuracy.

  • Motivation: 36.3% of baseline TTFT is consumed by vision encoding, rising to 55.8% for HoliTom and 68.4% for VisionZip.These figures show that vision encoding remains a major latency bottleneck, particularly in methods optimized for LLM prefill latency.
  • Motivation: Existing compression methods act after the vision encoder or inside the LLM, limiting their potential TTFT reduction.Inner-LLM methods provide limited TTFT reduction, while outer-LLM methods offer higher but still limited reduction.
  • Method: EarlyTom performs frame merging inside the vision encoder to compress redundant visual information during encoding with negligible overhead.This early-stage mechanism is designed to reduce visual tokens and time-to-first-token efficiently.
  • Method: A decoupled token selection strategy further reduces visual tokens with low latency and enables end-to-end acceleration without sacrificing accuracy.The strategy is co-designed at the system level to minimize latency while shrinking the vision-token sequence.
  • Results: 2.65× TTFT reduction is achieved on LLaVA-OneVision-7B with 10% token retention, while maintaining accuracy comparable to the full-token baseline.Experiments on LLaVA-OneVision-0.5B/7B report state-of-the-art acceleration with extremely fast TTFT and comparable accuracy.

2. Related Work

Prior work compresses visual tokens either within the vision encoder or projector, or after encoding and before the LLM. These methods use similarity, energy, temporal redundancy, dynamic pruning, and inference-time scaling to improve efficiency while preserving informative tokens.

  • Intra-encoder token compression: Intra-encoder methods compress tokens within the vision encoder or projector before feeding them to the language model.ToMe uses key-token similarity, while PiToMe preserves informative tokens with energy scores and merges large similar clusters.
  • Pre-LLM token compression: Pre-LLM methods treat compression after vision encoding and before the language model as a plug-and-play module.DyCoke combines cross-frame temporal compression with dynamic KV-cache pruning to remove temporal and spatial redundancy.
  • Video attention sinking: Top-K-based compression can overlook semantic information in other frames, limiting video context understanding under video attention sinking.Certain frames or regions consistently attract disproportionately high attention, while other frames may contain overlooked semantic information.
  • TTFT composition: 36.3% of baseline TTFT is spent on vision encoding, which remains the largest component for HoliTom at 55.8% and VisionZip at 68.4%.The corresponding vision-encoding times are 323 ms for the baseline, 324 ms for HoliTom, and 325 ms for VisionZip.
  • Inference-time scaling: Inference-time scaling laws show that, under fixed compute, visual reasoning favors larger LLMs paired with aggressive token compression.QueCC analyzes the trade-off between visual-token count and LLM size and proposes a query-aware method for extreme compression.

3. Method

EarlyTom is a training-free video-token compression framework with two stages: temporal frame compression inside the vision encoder and decoupled spatial token selection. It merges redundant frames early, then applies separate selection strategies to dynamic and static frames to improve compression while limiting feature-distribution bias.

  • Overview: EarlyTom combines inner–vision encoder frame compression with decoupled spatial token selection for additional compression without biasing visual features.The framework is training-free and targets efficient video LLM inference.
  • Inner-Vision Encoder Frame Compression: Frame segments are formed online using EMA-smoothed cosine similarity between corresponding spatial tokens, with boundaries triggered when similarity falls below τseg.For consecutive frames, similarity is averaged across corresponding spatial positions and updated over time.
  • Inner-Vision Encoder Frame Compression: Middle frames are merged only when pairwise similarity exceeds τmerge and is greater than the similarity of the next frame pair, preserving temporal consistency.The local-optimal strategy targets redundant frames within each segment, excluding the first and last frames.
  • Inner-Vision Encoder Frame Compression: Merged frame representations use similarity-based weighting to concentrate features around semantically important content and reduce ambiguity from uneven temporal variation.Each frame pair is weighted by its similarity with the following frame.
  • Decoupled Spatial Token Selection: Dynamic frames use global Top-K attention selection, while static frames use local-window Top-K selection to preserve their original distribution and mitigate sink-token bias.Head and tail frames within segments are designated dynamic, middle frames static; global selection preserves motion-sensitive tokens, whereas local windows reduce distribution shifts.
  • Decoupled Spatial Token Selection: Local-window selection produces compressed static frames whose distribution is closer to the original, mitigating negative effects from vision-sink bias.The method selects the maximum-attention token within each equal-sized local window.

4. Experiments

Experiments evaluate EarlyTom on four video-understanding benchmarks and system-efficiency metrics, comparing it with mainstream training-free token-compression methods. Results show that early compression inside the vision encoder improves TTFT, FLOPs, and throughput while preserving accuracy near the full-token baseline across backbones and ablations.

  • Performance comparison with state-of-the-art methods: EarlyTom compresses tokens inside the vision encoder, addressing the bottleneck left by late-stage methods whose TTFT remains 458–661 ms at 10–25% retention.Prior methods’ throughput fluctuates between 27.5 and 32.1 tokens/s under these conditions.
  • Performance comparison with state-of-the-art methods: 36.5T FLOPs at 25% retention is substantially below the full-token baseline of 82.6T, while jointly reducing vision-encoding and LLM-prefill costs.The experiments report FLOPs, TTFT, and throughput, with efficiency results measured on a single NVIDIA A100 GPU.
  • Accuracy comparison with state-of-the-art methods: More than 96% average accuracy relative to the full-token baseline is maintained across four video-understanding benchmarks.The benchmarks are MVBench, EgoSchema, LongVideoBench, and VideoMME.
  • Comparison across different backbones: On LLaVA-OV-0.5B, EarlyTom delivers substantial TTFT and FLOPs reductions and throughput improvements while keeping accuracy within a narrow margin across four benchmarks.These results support EarlyTom’s architecture-agnostic acceleration without sacrificing quality.
  • Ablation studies: 20% retention with spatial token selection matches temporal merging’s 98.8% baseline accuracy at approximately 73.9% retention, while joint merging and selection achieves 58.8 accuracy.Starting frame merging at layer 6 provides the best balance between accuracy and throughput; local window sampling combines top-K informativeness with random-sampling efficiency.

5. Conclusion

EarlyTom is a training-free token compression framework for fast Video-LLM inference that combines early-stage frame merging within the vision encoder with decoupled spatial token selection. It achieves substantial efficiency gains while maintaining accuracy comparable to the full-token baseline.

  • Conclusion: EarlyTom is a training-free token compression framework for fast Video LLM inference.It performs compression during inference without training.
  • Conclusion: 2.65× reduction in TTFT is achieved through early-stage frame merging within the vision encoder and decoupled spatial token selection.The framework combines both mechanisms to improve inference efficiency.
  • Conclusion: 61% reduction in FLOPs is achieved while maintaining accuracy comparable to the full-token baseline.These results demonstrate EarlyTom’s effectiveness and efficiency for fast Video-LLM inference.

EarlyTom · Overview

The supplementary material provides additional experiments, latency analyses, visualizations, and implementation details for EarlyTom. It also evaluates EarlyTom’s generalizability on the LLaVA-Video-7B benchmark across different backbones.

  • Overview: The supplementary material adds quantitative experiments beyond the main paper.These experiments are included because of page limitations in the main paper.
  • Overview: It provides detailed latency analyses of EarlyTom.
  • Overview: It includes qualitative visualizations of the method’s behavior.
  • Overview: The supplementary material reports implementation details for EarlyTom.
  • Overview: Section A evaluates EarlyTom’s generalizability on a different video-LLM architecture.
  • Overview: The evaluation uses extensive efficiency and accuracy results from the LLaVA-Video-7B benchmark.
  • Overview: These results verify EarlyTom’s robustness across different backbones.

A. Generalizability Analysis on LLaVA-Video and Qwen2.5-VL

EarlyTom generalizes to LLaVA-Video-7B and Qwen2.5-VL-7B, reducing computational costs through in-encoder frame merging while preserving competitive video-understanding accuracy. Its spatial selection and frame-merging components are important under aggressive compression.

  • Generalizability: EarlyTom generalizes its evaluation to the LLaVA-Video-7B and Qwen2.5-VL-7B models.The evaluation tests the framework’s broad applicability beyond the original model.
  • Efficiency analysis: 35.1% FLOPs ratio and 947.4 ms TTFT at 15% retention yield a 6.8× speedup over the 6429.3 ms full-token baseline.The reductions come from performing frame merging directly within the vision encoder.
  • Accuracy and trade-off: 56.43% average score demonstrates competitive accuracy on standard video-understanding benchmarks with substantially reduced computational overhead.The results support an efficient inference solution balancing high throughput with reliable model performance.
  • Ablation studies: Removing decoupled spatial token selection reduces performance from 62.2% to 61.4%, highlighting its critical role under aggressive compression.The ablation study also identifies weighted frame merging as essential for optimal performance.

B. Detailed Analysis of TTFT Latency Decomposition

TTFT is decomposed into vision encoding, visual token processing, LLM prefill, and system overhead across retention rates and two LLaVA-OneVision scales. The analysis shows EarlyTom’s in-encoder compression reduces encoding latency, while comparison-method overhead can limit gains, especially on the smaller model.

  • Latency decomposition: TTFT is decomposed into Vision Encoding, Visual Token Processing, LLM Prefill, and System Overhead at 10%, 15%, 20%, and 25% retention rates.The decomposition is evaluated for LLaVA-OneVision-7B and LLaVA-OneVision-0.5B.
  • LLaVA-OneVision-7B: Vision encoding dominates total latency for the Baseline, HoliTom, and VisionZip on LLaVA-OneVision-7B.HoliTom and VisionZip reduce LLM prefill latency through token reduction but do not address the vision encoder’s computational cost.
  • LLaVA-OneVision-7B: EarlyTom compresses redundancy within the vision encoder, substantially reducing encoding latency and achieving the lowest total latency on LLaVA-OneVision-7B.HoliTom’s Visual Token Processing overhead partially offsets gains from reduced prefill time.
  • LLaVA-OneVision-0.5B: 0.90× speedup at 10% retention: HoliTom has higher total latency than the Baseline on LLaVA-OneVision-0.5B.The time saved in LLM prefill is insufficient to outweigh the extra computational overhead introduced by comparison methods on the lightweight model.

C. Visualization of the Attention Sink Phenomenon Across Diverse Video Samples · D. Pseudocode of EarlyTom

Visualizations across diverse videos reveal persistent attention-sink tokens that can distort attention-based compression, motivating EarlyTom’s decoupled spatial selection. The accompanying pseudocode specifies EarlyTom’s frame-merging and spatial-selection components for implementation.

  • C. Visualization of the Attention Sink Phenomenon Across Diverse Video Samples: Figure 6 presents SigLIP vision-encoder attention heatmaps across a diverse set of video samples.The visualizations substantiate the paper’s analysis of the Attention Sink phenomenon.
  • C. Visualization of the Attention Sink Phenomenon Across Diverse Video Samples: Across all examples, vertical heatmap stripes show spatial tokens retaining exceptionally high attention throughout each video sequence.These tokens are identified as attention sinks.
  • C. Visualization of the Attention Sink Phenomenon Across Diverse Video Samples: Attention sinks act as static feature-space attractors that dominate attention distributions despite changing visual content in dynamic frames.This pattern appears consistently across the displayed examples.
  • C. Visualization of the Attention Sink Phenomenon Across Diverse Video Samples: Ranking tokens solely by attention magnitude may favor static sinks and overlook less prominent but semantically rich dynamic features.This distribution characteristic motivates EarlyTom’s decoupled spatial token selection strategy.
  • C. Visualization of the Attention Sink Phenomenon Across Diverse Video Samples: EarlyTom distinguishes static frames, where sinks are stable, from dynamic frames when applying decoupled spatial token selection.The strategy is designed to address sink-induced selection bias.
  • D. Pseudocode of EarlyTom: The section provides pseudocode for EarlyTom’s two core components to facilitate implementation.Algorithm 1 covers inner-vision encoder frame merging, while Algorithm 2 covers decoupled spatial token selection.
  • D. Pseudocode of EarlyTom: Algorithm 1 performs adaptive streaming segmentation and weighted merging to reduce temporal redundancy during inner-vision encoder frame merging.This describes the frame-merging process in the vision encoder.
  • D. Pseudocode of EarlyTom: Algorithm 2 processes dynamic and static frames through distinct selection mechanisms to preserve balanced spatial information.The pseudocode details the decoupled spatial token selection strategy.

E. Future Work

EarlyTom identifies system-level optimization as an open direction because VLM inference is mainly dominated by prefilling, while existing efficient-inference methods focus primarily on algorithmic improvements. It also notes that jointly leveraging system design and algorithmic techniques remains unresolved.

  • Future Work: VLM inference budgets are mainly dominated by the prefill stage.EarlyTom highlights prefilling as the principal source of inference cost.
  • Future Work: Existing efficient-inference methods primarily pursue algorithm-level improvements rather than system-level optimizations.The passage contrasts algorithmic techniques with system-level optimization.
  • Future Work: Jointly leveraging system design and algorithmic techniques in a heterogeneous manner remains an open problem.The paper identifies this combination as an unresolved research direction.
Loading 2605.30010v1…