Source-linked AI summary
V-CoLA: Vision Token Compression with Linear Attention
Hao Jiang, Yiru Mao, Tianpeng Bu, Hao Zhou, Hongtao Duan, Wang Jing, Bowen Xu, Xin Chen, Lulu Hu, Bin Yang, Yongliang Tao, Minying Zhang
TL;DR
Vision tokens impose substantial computational and memory burdens, and prior compression methods designed for softmax attention struggle to transfer to hybrid architectures with linear attention. V-CoLA uses linear-attention signals to select and merge tokens; on Qwen3.5, it retains 99.5% of average performance with 50% of tokens after the first layer, with runtime speedups of 1.86× to 6.15×.
Problem
Vision tokens impose substantial computational and memory burdens, while prior compression methods relying on softmax-attention scores or evaluated only on softmax models face transfer challenges with hybrid architectures incorporating linear attention.
Source: §1 · p. 1Method
Results
99.5% of average performance is retained with 50% of tokens after the first layer, and over 88.0% with 12.5%; runtime speedup is 1.86× to 6.15×.
Source: §1 · p. 2Takeaways & Limitations
Pruning overhead accounts for only 1.6% of the forward-pass cost in the reported evaluations.
Source: §1 · p. 2Evaluation focuses primarily on hybrid VLMs interleaving linear- and softmax-attention layers; purely Mamba-based and fully linear-attention VLMs are not covered.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Vision-language models (VLMs) have demonstrated impressive capabilities but suffer from substantial computational overhead, as vision tokens dominate the input sequence. This motivates vision token compression as a key direction to alleviate the burden. However, with the emergence of hybrid architectures incorporating linear attention (\eg, Qwen3.5), prior methods designed for softmax attention struggle to generalize. Our analysis reveals that both attention- and similarity-based approaches suffer notable performance degradation, underscoring the urgent need for compression methods tailored to this regime. To this end, we propose \textbf{V-CoLA}, an efficient training-free token compression framework specifically designed for linear attention. V-CoLA introduces a novel \textit{uniqueness-aware importance criterion} for identifying critical vision tokens, coupled with an \textit{adaptive token merging strategy} that performs compression. All components are optimized at the implementation level to remain compatible with the chunk-wise parallelism of linear attention, ensuring strong practical value. Extensive experiments across multiple benchmarks demonstrate the superiority of V-CoLA: it achieves 99.5\% of the original performance with only 50.0\% of vision tokens, and over 88.0\% with as few as 12.5\%, while delivering a 1.86$\times$ to 6.15$\times$ prefill speedup.
1 Introduction
Vision tokens impose substantial computational and memory burdens, while prior compression methods struggle to transfer to hybrid architectures with linear attention. V-CoLA uses linear-attention signals to select and merge tokens, retaining high average performance at reduced token counts.
1 Introduction
- Vision tokens far outnumber linguistic tokens, imposing substantial computational and memory burdens that motivate token compression.Prior compression methods reduce vision-token counts to improve computational efficiency and inference speed.Source: §1 · p. 1
- Attention- and similarity-based compression methods fail to surpass even a random baseline when transferred to hybrid architectures.The analysis attributes this failure to linear attention’s bounded recurrent state.Source: §1 · pp. 1–2
- V-CoLA estimates token importance from linear-attention dynamics using long-term contribution and short-term uniqueness to identify important, nonredundant tokens.Its adaptive chunk-wise merging adjusts chunk sizes and local compression ratios to reduce information loss in critical regions.Source: §1 · p. 2
- Implementation-level computation optimizations make V-CoLA compatible with linear-attention acceleration algorithms through chunk-wise parallelism.The reported pruning overhead is 1.6% of forward-pass cost.Source: §1 · p. 2
- 99.5% of average performance is retained with 50% of tokens after the first layer, and over 88.0% with 12.5%; runtime speedup is 1.86× to 6.15×.Pruning overhead accounts for 1.6% of forward-pass cost.Source: §1 · p. 2
2 Analysis
The analysis explains why familiar compression signals transfer poorly to hybrid architectures: their attention patterns differ, and linear attention compresses history into a bounded recurrent state. That state creates an information ceiling and selectively retains information, offering an intrinsic signal for token importance.
2.1 Preliminary
- Softmax attention retains token-level access at quadratic cost, whereas linear attention uses a recurrent state with selective writing, erasing, and forgetting.Gated DeltaNet’s decay gate and update-strength gate govern the recurrent update.Source: §2.1 · p. 2
2.2 Failure of Existing Methods
- Attention- and similarity-based methods show virtually no advantage over random token pruning on hybrid architectures.The analysis connects this result to differences in attention patterns and latent-feature similarity.Source: §2.2 · p. 3
- Qwen3.5 retains vision attention into intermediate layers, unlike Qwen3-VL’s rapid shallow-layer decline; its similarity-based redundant-token selections are also less reliable.The attention comparison is shown in Figure 3, and the similarity comparison in Figure 4.Source: §2.2 · p. 3, Figure 3 · p. 3, Figure 4 · p. 3
2.3 Information Ceiling of Linear Attention
- The bounded state creates an information-preservation problem: information loss grows linearly with sequence length once L > d, especially relevant when high-resolution inputs yield L ≫ d.The analysis treats the state as a lossy encoder of the token sequence.Source: §2.3 · p. 3
- Selective state updates encode an intrinsic token-importance signal, and the information-ceiling argument extends to bounded-state recurrent backbones generally.The argument relies on data processing and a finite state capacity independent of sequence length.Source: §2.3 · p. 3
3 Method
Section 3 introduces a training-free compression framework for hybrid architectures with linear attention, deriving token importance from recurrence and combining uniqueness-aware scoring, adaptive chunk-wise merging, and early exit.
3 Method
- V-CoLA combines linear-attention token-importance signals with uniqueness-aware scoring and adaptive chunk-wise merging in a training-free compression framework.The framework also uses vision-token early exit in deep layers.Source: §3 · pp. 3–4, §3.4 · p. 6
3.1 Linear Attention as an Intrinsic Indicator
- Visual-information retention falls below 20% in deeper layers, while adjacent-layer importance-ranking correlation remains high shallow and declines with depth.These patterns motivate using shallow-layer selective retention to identify important or redundant tokens.Source: §3.1 · p. 4, Figure 6 · p. 4
3.2 Uniqueness-aware Token Importance
- Importance combines final-state reconstruction with short-term state change, using token uniqueness to complement retained contribution and reduce redundant selections.Nearby pseudo-query probes measure changes in the state after a token is absorbed.Source: §3.2 · p. 5
3.3 Adaptive Chunk-wise Token Merging
- The merging strategy partitions tokens into importance-aligned chunks of similar importance mass, then merges within each chunk independently.Dense-importance regions receive finer chunks, while less-informative regions receive coarser chunks.Source: §3.3 · pp. 5–6
3.4 Early Exiting of Vision Tokens
- Perplexity drops sharply and plateaus after layer 24, and MME results support discarding deep-layer vision tokens without losing multimodal reasoning performance.The proposed early-exit mechanism removes all vision tokens in deep layers to further reduce computation.Source: §3.4 · p. 6
4 Experiments
Experiments compare V-CoLA with existing compression methods on hybrid VLM benchmarks, then examine inference efficiency, component contributions, and transfer across models. V-CoLA leads the reported comparisons and achieves substantial prefill speedups with low merging overhead.
4.1 Setup
- Experiments use Qwen3.5-9B primarily, with Qwen3.5-27B and InfiniteVL for generalizability, across eight benchmarks and on NVIDIA A100 GPUs.Compression is generally applied at the end of layer 1, with V-CoLA early exit at layer 24; average token-remaining rates are aligned across methods.Source: §4.1 · p. 6
4.2 Main Results
- V-CoLA consistently outperforms the compared methods across benchmarks and compression rates on Qwen3.5-9B.The comparison includes FastV, SparseVLM, DART, VisionZip, and DTP.Source: §4.2 · p. 6, Table 1 · p. 7
- Prefill speedups range from 1.86× to 6.15×, while adaptive token merging adds 0.86 ms, approximately 0.1% of total prefill cost.The Extended Gated DeltaRule has comparable single-query latency to the original operator and is 3.85× faster under 4-query inference.Source: §4.2 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7
4.3 More Analysis
- The default λ = 0.1 performs better, and a four-query forward window improves robustness without gains from adding further queries.Results vary smoothly across the tested hyperparameters; λ values from 0.05 to 0.10 perform comparably, and the default configuration transfers across the three tested models.Source: §4.3 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 8
- Removing either adaptive token merging or early exit reduces performance; merging aggregates token information, while early exit reallocates redundant token budget toward shallow-layer understanding.The merging strategy can emulate hard pruning by tuning its temperature coefficient.Source: §4.3 · p. 8, Table 6 · p. 8
- With the same configuration, V-CoLA achieves leading results on Qwen3.5-27B and InfiniteVL, extending evaluation across two model scales and architecture families.Extending evaluation to recurrent backbones with compatible matrix-state readouts remains future work.Source: §4.3 · p. 8, Table 7 · p. 8
5 Related Work
Prior visual-token compression methods score or group tokens, but their assumptions can be challenged in hybrid linear-attention architectures. Linear-attention and related recurrent models compress history into bounded states, motivating architecture-aware importance estimation.
5.1 Visual Token Compression for MLLMs
- Visual-token compression scores tokens using attention or similarity, or groups them, but these approaches rely on softmax scores, reliable hidden-state similarity, or static encoder statistics.Prior Mamba-based compression ranks importance from intermediate outputs and is described as suboptimal.Source: §5.1 · pp. 8–9
5.2 Linear Attention Architectures
- Linear-attention, state-space, and recurrent architectures compress history into bounded states, while hybrid VLMs interleave linear recurrence with softmax attention.Token compression tailored to linear-attention VLMs remains largely unexplored, motivating architecture-aware importance estimation.Source: §5.2 · p. 9
6 Conclusion
The conclusion presents V-CoLA as a training-free compression framework for hybrid VLMs with linear attention. It estimates importance from linear-attention dynamics and combines adaptive chunk-wise merging with early exit, with implementation optimizations designed for chunk-wise parallelism.
6 Conclusion
- V-CoLA is a training-free vision-token compression framework tailored to hybrid VLMs with linear attention.Its importance estimates use linear-attention dynamics.Source: §6 · p. 9
- The framework combines an adaptive chunk-wise merging strategy with an early-exit mechanism for vision tokens.Its implementation optimizations are compatible with chunk-wise parallelism.Source: §6 · p. 9
- V-CoLA delivers state-of-the-art performance across diverse benchmarks while improving inference efficiency.Source: §6 · p. 9
7 Limitations
The study identifies two limitations: its evaluation centers on hybrid architectures, and its analysis and method design are mainly grounded in Gated DeltaNet.
7 Limitations
- The authors frame the reported performance and efficiency gains alongside limitations that motivate future work.The section presents architectural coverage and linear-attention formulation scope as areas for further exploration.Source: §7 · p. 9
- The evaluation focuses on hybrid VLMs such as Qwen3.5 and InfiniteVL, leaving purely Mamba-based and fully linear-attention VLMs unevaluated.Extending the method to those architectures may require revisiting how the importance criterion interacts with the absence of periodic softmax-attention layers.Source: §7 · p. 9
- The analysis and method design are mainly grounded in Gated DeltaNet, so generality across a broader range of linear-attention formulations remains to be assessed systematically.The reconstruction-based perspective may extend to other recurrent state-update rules, including other gated linear-attention variants.Source: §7 · p. 9
8 Ethical Considerations
The paper reports that its research uses public benchmarks and pretrained models without user-facing deployment or human-subject experiments. It also describes parallel implementation choices for V-CoLA's linear-attention components and adaptive merging.
8 Ethical Considerations
- Experiments use publicly available benchmarks and pretrained models for their intended research use, with no user studies, user-facing deployment, or human-subject experiments.The authors state that the resources contain no intended personally identifiable or sensitive information and that training-free V-CoLA adds no biases beyond those in the underlying models.Source: §8 · p. 9
A V-CoLA Implementation Parallelism
- Chunk-wise parallelism limits arbitrary state access and permits only one sequence of query, key, and value inputs in the standard forward pass.The implementation targets chunk-wise methods that combine inter-chunk recurrence with intra-chunk parallelism.Source: A V-CoLA Implementation Parallelism · p. 13
- The implementation extends Gated DeltaRule to inject multiple queries for output computation without involving them in state recurrence.Shifted input keys serve as pseudo queries, and a single forward pass produces pseudo outputs for calculating token uniqueness.Source: A V-CoLA Implementation Parallelism · p. 13
- Adaptive merging balances chunk importance by placing actual boundaries near equal-importance intervals, using parallel matrix computation for GPU efficiency.The boundary choice aims to remain statistically unbiased while addressing variable chunk sizes and independent weight normalization.Source: A V-CoLA Implementation Parallelism · p. 13
B Evaluation Benchmarks
The section describes benchmarks spanning multimodal perception, reasoning, text recognition, hallucination, dialogue, and small-object recognition. It also outlines computational costs for hybrid linear- and softmax-attention models.
B Evaluation Benchmarks
- MME, MMBench, GQA, ScienceQA, TextVQA, POPE, SeedBench, VizWiz, MMStar, MMDU, and V*-Bench cover multimodal perception, reasoning, and generation tasks.The benchmarks include text in images, hallucination assessment, multi-image dialogue, and small objects in crowded scenes.Source: B Evaluation Benchmarks · pp. 13–14
C FLOPs Analysis and Theoretical Speedup
- Qwen3.5's 3:1 hybrid architecture uses three Gated DeltaNet linear-attention layers and one gated softmax-attention layer per four consecutive layers.The analysis derives theoretical speedup from reducing vision tokens in this architecture.Source: C FLOPs Analysis and Theoretical Speedup · p. 14
C.1 Per-Layer Computational Cost
- Softmax attention cost grows quadratically with sequence length N, while linear-attention cost grows linearly; their crossover is N = c · dh, about 1024 when dh = 128.The estimate is stated for the per-layer computational-cost analysis.Source: §C.1 · p. 15
C.2 Total FLOPs for 3:1 Hybrid Architecture
- The total FLOPs expression for the 3:1 hybrid architecture contains both linear-in-N and quadratic-in-N components.The equation combines terms for the hybrid model's layer types and separates contributions by their sequence-length scaling.Source: §C.2 · p. 15
C.3 Theoretical Speedup from Token Reduction
The analysis models speedup after vision-token compression as a function of retention, sequence length, and the share of layers that benefit. It reports empirical speedups close to the theoretical estimates at two retention rates.
C.3 Theoretical Speedup from Token Reduction
- After compression at layer lc, the retained vision-token fraction r reduces the remaining sequence length to N′ = Nt + rNv, approximated by rN when Nv ≫ Nt.The per-layer speedup depends on the linear and quadratic sequence-length costs.Source: §C.3 · p. 15
- Speedup approaches 1/r when linear costs dominate and 1/r^2 when quadratic costs dominate, with the end-to-end value interpolating between these regimes.The fraction of compressed layers controls how many layers benefit from compression.Source: §C.3 · p. 15
- At r = 0.5 with 8K vision tokens, empirical speedup is 1.91× (theory ≈2.0×), and at r = 0.125, it is 6.15× (theory ≈7.0×).The reported empirical speedups closely match Equation 18.Source: §C.3 · p. 16
D Additional Evaluations
Additional evaluations test V-CoLA on long-form multi-image dialogue, fine-grained small-object recognition, and end-to-end inference latency. Results show preserved dialogue quality, benefits from intermediate-layer visual access, and latency gains that depend on how much total cost is spent on prefill.
Long-form multi-turn dialogue
- Within 1.4 points of the full model across all six MMDU dimensions, V-CoLA preserved dialogue quality even at a 12.5% token budget.MMDU samples contain up to 20 images, 27 turns, and 18K multimodal tokens.Source: Table 8 · p. 16, D Additional Evaluations · p. 16
Fine-grained small-object recognition
- 2.09 points higher overall accuracy at a matched 50% budget came from early exiting; layer-24 exit stayed within 1.05 points of full-model accuracy, versus a 2.62-point loss at layer 12.V*-Bench evaluates fine-grained recognition of easily overlooked objects in crowded, high-resolution scenes.Source: Table 9 · p. 16, D Additional Evaluations · p. 16
End-to-end latency
- 38.6–64.6% lower time-to-first-token and up to 1.733× lower end-to-end latency were measured with INT4 TensorRT Qwen3.5-9B; gains are smaller when prefill is a smaller share of total cost.On unoptimized PyTorch, prefill accounts for 29.4% and decoding for 58.8% of end-to-end latency.Source: Table 10 · p. 16, D Additional Evaluations · p. 16