PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline

Xiao Zhang, Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Shichao Kan

arXiv:2609.35394v1cs.CV

TL;DR

Long videos produce many visual tokens, making Video LLM inference costly, and it is unclear how much compression performance requires elaborate selection mechanisms. SimpleCluster applies position-aware cross-frame clustering and represents clusters with mean features; it performs competitively across models and retention ratios, while stronger performance is associated with better feature-distribution preservation.

  • Problem

    Long videos produce many visual tokens that make Video LLM inference costly, and it remains unclear how much compression performance depends on elaborate selection rather than preserving structure in visual representations.

    Source: §1 · p. 1

  • Method

    SimpleCluster is a training-free baseline that uses position-aware cross-frame clustering and represents each cluster with the mean of its original visual features.

    Source: §1 · p. 2

  • Results

    SimpleCluster is competitive or superior across Video LLMs and retention ratios; at 1% retention it retains 88.3%, 84.8%, and 85.8% of corresponding uncompressed-model performance.

    Source: §4.2 · p. 5, §6 · p. 9

  • Takeaways & Limitations

    Stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially global coverage.

    Source: §1 · p. 2

    The study does not fully characterize when more sophisticated compression mechanisms provide additional advantages.

    Source: §7 · p. 9

Abstract

from arXiv · show

Video Large Language Models (Video LLMs) have achieved remarkable progress in video understanding, but their inference efficiency is constrained by the large number of visual tokens produced by long videos. Recent video token compression methods increasingly introduce sophisticated strategies for token selection, pruning, and merging. This raises a fundamental question: how much of compression performance can be obtained by simply preserving the structure encoded in the visual representations? We investigate this question with SimpleCluster, a simple and training-free baseline that performs position-aware cross-frame clustering in the visual feature space and represents each cluster using the mean of its original visual features. Extensive experiments across four video understanding benchmarks and three representative Video LLMs show that SimpleCluster achieves competitive or superior performance over recent compression methods across a wide range of token retention ratios, with particularly strong robustness under extremely low retention rates (e.g., 1%). To understand this behavior, we analyze the feature space preserved by different compression methods in terms of local approximation fidelity and global coverage. The results show that stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially its global coverage. These findings highlight feature-space preservation as an important consideration for video token compression under highly constrained token budgets. Our code is available at https://github.com/xiaozhang79/SimpleCluster.

1 INTRODUCTION

Long videos generate many visual tokens, making Video LLM inference costly and motivating compression that preserves understanding performance. The paper asks whether simple feature-space structure preservation can rival elaborate selection mechanisms, and studies SimpleCluster and feature-space preservation to address that question.

1 INTRODUCTION

  • Long videos are costly to process because dense spatial-temporal representations produce many visual tokens, motivating compression that reduces sequence length while preserving downstream understanding.Source: §1 · p. 1
  • The paper asks how much compression performance depends on elaborate token-selection mechanisms versus preserving structure already encoded in visual representations.Source: §1 · pp. 1–2
  • SimpleCluster is competitive or superior to recent compression methods across Video LLMs and retention ratios, with strong robustness under aggressive compression.The comparisons cover three representative Video LLMs and four video understanding benchmarks.Source: Figure 1 · p. 2, §1 · p. 2
  • Stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially its global coverage.SimpleCluster maintains broader feature-space coverage while preserving local feature fidelity, providing an empirical explanation for its robustness under aggressive compression.Source: §1 · p. 2

2 RELATED WORK

Video LLMs extend visual-language understanding to temporal perception and reasoning, but redundant video tokens create inference overhead. Video compression methods have expanded from image-token reduction to increasingly varied strategies that exploit temporal redundancy, raising questions about the necessity of complex designs.

2.1 VIDEO LARGE LANGUAGE MODELS

  • Video LLMs extend multimodal visual-language capabilities to temporal perception and reasoning, while redundant visual tokens contribute to substantial inference overhead.The section describes LLaVA-OneVision, LLaVA-Video, and InternVL3 as representative Video LLMs.Source: §2.1 · pp. 2–3

2.2 VISUAL TOKEN COMPRESSION

  • Visual token compression evolved from image-focused merging, pruning, and selection toward video methods that exploit temporal redundancy through varied compression strategies.The video methods include cross-frame merging, density-aware pruning, early-stage compression, and attention-guided aggregation.Source: §2.2 · p. 3
  • Because existing methods often use increasingly complex hand-crafted designs, the paper investigates simple visual feature-space clustering as a strong baseline.Source: §2.2 · p. 3

3 FROM COMPLEX VIDEO TOKEN COMPRESSION TO SIMPLECLUSTER

The paper frames video token compression as reducing a dense feature sequence to a fixed token budget while preserving its feature-space structure. SimpleCluster uses position-aware cross-frame clustering and cluster means to form the compact representation supplied to the LLM.

3.1 MOTIVATION

  • Compression transforms N visual features into a compact sequence of B features, with retention ratio r = B/N, while aiming to preserve feature-space structure.The input features come from T sampled frames and have dimension d.Source: §3.1 · p. 3

3.2 SIMPLECLUSTER: A SIMPLE FEATURE-SPACE CLUSTERING BASELINE

  • SimpleCluster uses frame, row, and column positions with 3D RoPE to guide cross-frame clustering, distinguishing visually similar tokens at different spatiotemporal locations.Position-aware transformed features define the clustering space, while the original projected visual features remain unchanged for aggregation.Source: §3.2 · p. 4
  • The method partitions tokens from sampled frames into B clusters by within-cluster cosine distance, then represents each cluster with the mean of its original visual features.The resulting B aggregated tokens are fed to the LLM for downstream video understanding.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

4 EXPERIMENTS

Experiments across four video benchmarks and three Video LLMs evaluate SimpleCluster against recent compression methods and examine its design and efficiency. SimpleCluster performs strongly across retention rates, while ablations and efficiency results assess the roles of its clustering choices and computational cost.

4.1 EXPERIMENTAL SETUP

  • Experiments cover four video benchmarks, three Video LLM backbones, and retention ratios from 1% to 25%, comparing SimpleCluster with five recent compression methods.The benchmarks are MVBench, EgoSchema, LongVideoBench, and VideoMME.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • At 5% retention, SimpleCluster scores 55.7, 55.6, and 53.7 on the three backbones, and at 1% retains 88.3%, 84.8%, and 85.8% of uncompressed performance.The scores correspond to LLaVA-OneVision-7B, LLaVA-Video-7B, and InternVL3-2B, respectively.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.3 COMPARISON WITH SIMPLE TOKEN COMPRESSION STRATEGIES

  • On LLaVA-OneVision-7B at 10% retention, SimpleCluster achieves the best overall performance among simple strategies, with an average score of 57.3.The comparison includes random sampling, pooling, and grid-sampling strategies; fixed spatial or temporal schemes may miss underlying feature structure.Source: §4.3 · pp. 5–6, Table 4 · p. 7

4.4 ABLATION STUDIES

  • Removing 3D RoPE decreases the average score from 57.3 to 56.6, indicating that spatiotemporal information in the clustering space provides useful structural cues for organizing visual tokens.The ablation is on LLaVA-OneVision-7B.Source: §4.4 · p. 7, Table 5 · p. 8
  • Cross-frame clustering improves average score from 56.1 to 57.3 over per-frame clustering by organizing visual tokens across frames, exploiting redundancy, and allocating limited tokens according to the global feature distribution.Source: §4.4 · p. 7
  • For LLaVA-OneVision-7B, nearest-token selection and mean aggregation achieve comparable performance, suggesting cluster-assignment quality matters more than the specific aggregation operation, as reported in Table 5.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 EFFICIENCY ANALYSIS

  • At 10% retention, SimpleCluster uses 27.5T prefill FLOPs and 514.1 ms TTFT, reducing them by 73.2% and 45.5% versus the uncompressed model while retaining 98.1% of average performance.Compared with AOT, it further reduces TTFT by 26.2% and achieves a higher average score.Source: §4.5 · p. 8, Table 6 · p. 8

5 WHY DOES FEATURE-SPACE CLUSTERING WORK?

SimpleCluster preserves the original visual feature space more effectively than competing methods at both tested retention ratios, and its stronger preservation coincides with stronger downstream performance. Its advantage remains pronounced under more aggressive compression.

5 WHY DOES FEATURE-SPACE CLUSTERING WORK?

  • At 10% retention, SimpleCluster’s L2-NQE is 0.281 and cosine coverage is 59.7%; at 5%, they are 0.336 and 45.9%.At 10%, its L2-NQE is 31.0% lower than EarlyTom’s 0.407, and its coverage is 24.7 percentage points above FastVID’s 35.0%; at 5%, its metrics outperform the best competing values, and even beat all competitors at 10%.Source: §5 · p. 9, Table 7 · p. 9
  • At both 10% and 5% retention, SimpleCluster has the lowest L2-NQE, highest cosine coverage, and best average downstream scores of 57.3 and 55.7, respectively.Weaker feature preservation generally coincides with larger performance degradation; global coverage is particularly associated with stronger downstream understanding.Source: §5 · p. 9
  • When retention falls from 10% to 5%, SimpleCluster’s L2-NQE rises by 0.055, coverage retains 76.9% of its original level, and its downstream score drops 1.6 points.Competing methods show L2-NQE increases of 0.121–0.148, coverage retention of 42.5%–61.1%, and downstream score decreases of 2.4–5.6 points.Source: §5 · p. 9

6 CONCLUSION

SimpleCluster is a training-free baseline that compresses video tokens through position-aware cross-frame clustering in visual feature space. Across three Video LLMs and four benchmarks, it performs competitively or better across retention ratios, while stronger downstream performance is associated with better feature-space preservation.

6 CONCLUSION

  • SimpleCluster uses position-aware cross-frame clustering and achieves competitive or superior performance across three Video LLMs, four benchmarks, and a wide range of retention ratios.The feature-space analysis links stronger downstream performance with better preservation of local approximation fidelity and global coverage.Source: §6 · p. 9

7 LIMITATION

The paper frames its contribution as a strong baseline study, while noting that it does not fully characterize when more sophisticated compression designs offer additional advantages. It also reports using LLMs to improve manuscript clarity, grammar, and readability, followed by review and editing.

7 LIMITATION

  • The study is positioned as a strong baseline, and does not fully characterize when more sophisticated compression mechanisms provide additional advantages.Future work could investigate adaptive strategies balancing feature-space coverage, local fidelity, efficiency, generality, and computational cost.Source: §7 · p. 9

AI USE STATEMENT

  • The authors used LLMs to improve manuscript clarity, grammar, and readability, then reviewed and edited the content and take responsibility for it.Source: AI USE STATEMENT · p. 10

A EVALUATION BENCHMARKS

The evaluation covers benchmarks for temporal understanding, egocentric long-horizon reasoning, long-context comprehension, and broad video understanding. Additional results report SimpleCluster’s performance at 10% and 20% retention across three Video LLM backbones.

A EVALUATION BENCHMARKS

  • The four evaluation benchmarks span temporal understanding, egocentric video understanding, long-context comprehension, and multimodal retrieval.MVBench targets temporal understanding; EgoSchema evaluates long-horizon reasoning in egocentric video; LongVideoBench tests long-context video-language understanding; and VideoMME covers video understanding across domains and temporal scales.Source: A EVALUATION BENCHMARKS · p. 14

B ADDITIONAL EXPERIMENTAL RESULTS

  • Additional experiments report performance comparisons at 10% and 20% retention on LLaVA-OneVision-7B, LLaVA-Video-7B, and InternVL3-2B, plus inference efficiency on the latter two backbones.Source: B ADDITIONAL EXPERIMENTAL RESULTS · p. 14

B.1 PERFORMANCE RESULTS

  • At 20% retention, SimpleCluster scores 58.4 on LLaVA-Video-7B and 56.2 on InternVL3-2B; at 10%, it scores 57.3 and 55.1, respectively.It achieves the best average score at 10% on LLaVA-Video-7B and at both retention ratios on InternVL3-2B; on LLaVA-OneVision-7B, it is competitive at 20% and scores 57.3 at 10%.Source: §B.1 · p. 14

B.2 EFFICIENCY ANALYSIS

At 10% token retention, SimpleCluster reduces prefilling FLOPs and time to first token (TTFT) on LLaVA-Video-7B and InternVL3-2B while retaining most uncompressed-model performance.

B.2 EFFICIENCY ANALYSIS

  • At 10% retention, SimpleCluster reduces prefilling FLOPs by 71.2% and 66.5%, and TTFT by 37.1% and 20.1%, on the two backbones, respectively.The backbones are LLaVA-Video-7B and InternVL3-2B, in that order.Source: §B.2 · pp. 14–15, Table 11 · p. 16
  • At 10% retention, SimpleCluster retains 95.2% and 96.3% of uncompressed-model performance on LLaVA-Video-7B and InternVL3-2B, respectively.Source: §B.2 · pp. 14–15, Table 11 · p. 16
  • Table 11 presents an efficiency analysis of SimpleCluster on LLaVA-Video-7B, with the analysis conducted under a 10% token-retention setting.Source: Table 11 · p. 16

C ADDITIONAL ANALYSIS

The additional analysis explains SimpleCluster through clustering distortion and cross-frame budget sharing, connecting its feature-space organization to local fidelity and global coverage while noting that feature similarity is not query-specific importance.

C ADDITIONAL ANALYSIS

  • Position-aware clustering considers visual similarity and relative spatiotemporal positions, helping group related tokens while reducing merges across substantially different locations or times.Source: C ADDITIONAL ANALYSIS · p. 16
  • Joint cross-frame clustering lets visually related tokens from different frames share representatives, giving a shared budget flexibility to adapt to video feature distributions.The analysis links this flexibility to repeated objects, backgrounds, and scene patterns, and presents it as a theoretical explanation for the empirical benefit of cross-frame clustering.Source: C ADDITIONAL ANALYSIS · pp. 16–17
  • Each cluster is represented by the mean of its original visual features, while positional information guides grouping and the compressed features remain in the original projected space.For a fixed cluster, this mean minimizes squared reconstruction error.Source: C ADDITIONAL ANALYSIS · p. 17
  • As the token budget decreases, each representative summarizes more of the feature distribution, making adaptive allocation important; fixed spatial or temporal allocation may spend tokens on redundant regions.The analysis connects lower clustering distortion to local fidelity and adaptive cross-frame allocation to better coverage of distinct feature regions.Source: C ADDITIONAL ANALYSIS · pp. 16–17
  • Across compression methods and token budgets, stronger downstream performance is associated with better feature preservation, including lower L2-NQE and higher cosine coverage under low retention.The clustering objective captures feature similarity, not query-specific importance, so its relationship to downstream understanding is evaluated empirically.Source: C ADDITIONAL ANALYSIS · p. 17

D.1 REPRODUCTION DETAILS OF COMPARED METHODS

The paper describes reproduction controls and SimpleCluster implementation, then uses visualizations to show cross-frame cluster assignments and dense-feature coverage. The visual analyses concern feature-space organization and proximity, not object tracking, attention, or task relevance.

D.1 REPRODUCTION DETAILS OF COMPARED METHODS

  • Compared methods use official implementations where available, while backbone, frames, preprocessing, prompts, generation settings, and benchmark scoring are held consistent.The compared methods are VisionZip, FastVID, EarlyTom, and AOT; retention is measured over visual content tokens before the LLM, excluding structural tokens.Source: §D.1 · pp. 17–18

D.2 IMPLEMENTATION DETAILS OF SIMPLECLUSTER

  • SimpleCluster runs after the backbone’s native visual processing and before the LLM, groups all video tokens under a target budget, and uses position-aware features for assignments while retaining original features for aggregation.The partition has B groups, with B set from the target retention ratio; prototypes are updated from normalized means of position-aware features.Source: §D.2 · pp. 18–19
  • The implementation samples 32 frames for LLaVA-OneVision-7B and InternVL3-2B, and 64 for LLaVA-Video-7B; InternVL3-2B retains dynamic image tiling.The corresponding grids are 14 × 14, 13 × 13, and 16 × 16 per image tile, respectively.Source: §D.2 · p. 18

E.1 CROSS-FRAME CLUSTER ASSIGNMENTS

  • Across four videos, clusters include both temporally localized groups and groups spanning many frames, aggregating recurring features while separately representing transient or visually distinct content.Figure 4’s colored regions encode cluster assignments, not verified object tracks or token importance.Source: §E.1 · p. 19, Figure 4 · p. 21

E.2 DENSE-FEATURE COVERAGE

  • At nominal 5% retention, SimpleCluster shows more uniformly high dense-feature similarity across two videos, while competing methods show larger low-similarity regions and coverage gaps.Figure 5 uses a shared color scale; its heatmaps show maximum cosine similarity to compressed tokens, measuring representational proximity rather than attention or task relevance.Source: §E.2 · p. 20, Figure 5 · p. 22
Loading 2609.35394v1…