PaperlayerView exact revision on arXiv

Source-linked AI summary

VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

Yupeng Zheng, Kai Zou, Bin Liu, Nenghai Yu

arXiv:2607.12756v2cs.CV

TL;DR

Visual token compression must reduce VLM inference costs while preserving visual semantics, but existing heuristic and external-module approaches have important limitations. VisCo reuses the pretrained VLM as an intrinsic compressor through lightweight adaptation, and experiments show consistent gains across compression ratios, including the extreme one-token setting.

  • Problem

    Visual token compression seeks to reduce the cost of processing dense visual tokens while preserving essential semantics, but existing approaches rely on heuristics or external modules requiring substantial retraining.

  • Method

    VisCo uses the pretrained VLM as an intrinsic, asymmetric autoencoder with lightweight encoding adaptation, hierarchical information transfer, and a frozen decoding backbone.

  • Results

    Across 3 VLM backbones and 6 benchmarks, VisCo consistently outperforms existing methods across compression ratios and remains effective with a single visual token.

  • Takeaways & Limitations

    Leveraging pretrained VLM priors enables robust visual token compression under tight token budgets with only lightweight training.

Abstract

from arXiv · show

Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression. Code is available at: \href{https://github.com/Zyvpeng/VisCo}{\textcolor{blue}

1 Introduction

VisCo addresses the computational and KV-cache burden of visual tokens by reusing pretrained VLMs as intrinsic compressors rather than relying on heuristic pruning or external modules. Its lightweight, hierarchical autoencoding framework preserves task-relevant semantics and performs strongly under aggressive compression.

  • Motivation: High-resolution images produce many visual tokens, substantially increasing computation and key–value cache storage and limiting VLM deployment in resource-constrained, real-time scenarios.
  • Prior limitations: Training-free attention-based methods estimate token importance heuristically but are vulnerable to attention shift, while external compression modules require substantial retraining.
  • VisCo: VisCo uses the pretrained VLM itself as an asymmetric intrinsic compressor, introducing learnable memory tokens while keeping the pretrained backbone unchanged.The framework performs visual token self-compression during the vision encoding stage with lightweight training.
  • VisCo: VisCo passes multi-granularity visual semantics hierarchically from encoder layers to the decoder under a reduced token budget.
  • Evaluation: 3 VLM backbones and 6 benchmarks show that VisCo consistently outperforms existing methods, especially at high compression ratios, while its representations capture complementary information beyond compression.

2 Related Work

Related work spans modular VLM architectures, built-in compression designs, training-free token pruning or merging, and training-based compression modules. Across these approaches, the central unresolved challenge is preserving semantic fidelity under aggressive compression without imposing costly adaptation or training.

  • VLM Architectures: Representative VLMs combine pretrained visual encoders and LLM backbones through lightweight modality alignment modules, often trained in two stages.Examples include LLaVA, BLIP-2, and InstructBLIP.
  • Built-in Compression: Built-in mechanisms such as Qwen2-VL’s dynamic resolution and InternVL’s token reduction lower overhead, but achieve limited compression and motivate dedicated methods.These designs aim to improve efficiency without sacrificing fine-grained semantics.
  • Training-Free Methods: Training-free methods prune or merge visual tokens using attention scores or similarity, but tight budgets can remove fine-grained details and global semantics.FastV and SparseVLM exemplify attention-based pruning, while VisionZip uses token merging to preserve more global semantics.
  • Training-Free Methods: Training-free methods remain competitive under mild compression, whereas aggressive compression makes preserving semantic fidelity the central challenge.At low compression ratios, even simple input downsampling can outperform existing methods.
  • Training-Based Methods: Training-based methods either add lightweight modules with limited aggressive-compression gains or use heavier pipelines that increase training cost and require VLM adaptation.More recent approaches operate within the VLM but still rely on costly alignment, instruction tuning, or specialized compression training.

3 Method

VisCo is an asymmetric, parameter-sharing VLM autoencoder that compresses visual tokens with learnable memory tokens while reusing the frozen pretrained VLM for decoding. It transfers layer-wise memory KV caches from encoding to decoding and trains end-to-end with lightweight adaptation and teacher forcing.

  • Architecture: VisCo adds a lightweight encoder module while keeping the entire pretrained VLM frozen during decoding.The trainable parameters are limited to LoRA adapters and learnable memory tokens.
  • Architecture: The encoder appends N_m learnable memory tokens to N_v visual tokens, with N_m ≪ N_v, forming the joint input sequence.Visual tokens are produced by resizing the image, applying the vision encoder, and using the multimodal projector.
  • Hierarchical compression: VisCo transfers hierarchical layer-wise memory KV caches from encoder to decoder instead of passing final encoder hidden states.Shallow layers preserve low-level visual cues, while deeper layers encode progressively more abstract semantic information; decoding uses only the collected memory KV caches.
  • Decoding: VisCo reuses the shared pretrained backbone with LoRA removed in decoding and directly populates the decoder KV cache with the encoded memory bank.Autoregressive generation then proceeds on top of this cache.
  • Training: Training uses end-to-end teacher forcing and conditional-likelihood maximization without additional pre-training objectives.The trainable parameters consist of the memory tokens and LoRA adapters, while encoder-side KV caches serve as hierarchical prefixes rather than Prefix-Tuning projections.

4 Experiments

The experiments comprehensively evaluate VisCo’s performance and efficiency across six benchmarks and three VLM models.

  • VisCo is evaluated for performance and efficiency on 6 benchmarks across 3 VLM models.

4.1 Implementation Details

VisCo is fine-tuned for one epoch on a ∼10% subset of LLaVA-665K using LoRA, then evaluated on LLaVA-1.5-7B and Qwen2-VL models, including a native 4× compression setting.

  • Training setup: VisCo is fine-tuned for one epoch on the first conversation turn from a ∼10% subset of LLaVA-665K.LoRA is applied to Q/V projections with rank = 64, α = 128, and a learning rate of 5 × 10−5.
  • Evaluation models: Evaluation covers LLaVA-1.5-7B and Qwen2-VL at 2B and 7B scales.Qwen2-VL’s native 4× compression provides a testbed for VisCo’s robustness on already-compact visual representations.

4.2 Datasets

VisCo is evaluated on six multimodal benchmarks using benchmark-specific standard metrics, with Avg summarizing the mean retained score ratio across them.

  • 4.2 Datasets: VisCo is evaluated on MME, MMB, MMB-CN, MMVet, GQA, and POPE.Reported metrics are MME perception score, MMB/MMB-CN and GQA accuracy, MMVet GPT-4 assisted score, and POPE F1.
  • 4.2 Datasets: Avg is the unweighted mean of the compressed/original retained score ratio across benchmarks.

4.3 Baselines

VisCo is compared with plug-and-play, fine-tuned, and stronger reference methods, covering diverse visual-token compression baselines and training settings.

  • Baseline categories: The comparison includes FastV, SparseVLM, DivPrune, and VisPruner as plug-and-play methods requiring no extra fine-tuning.Two additional baselines are fine-tuned: PruMerge+ for 1 epoch on full LLaVA-665K and VisionZip‡ for 1 epoch on the same 10% LLaVA-665K subset as VisCo.
  • Baseline categories: VoCo-LLaMA and Mat… are included as stronger reference methods.The supplied passage truncates the second reference name after “Mat”.

4.4 Main Results

VisCo achieves strong visual-token compression across LLaVA-1.5 and Qwen2-VL, preserving task performance even under aggressive and single-token compression. It consistently outperforms directly comparable baselines and remains competitive with stronger training-based references while avoiding their extensive retraining requirements.

  • Results on LLaVA-1.5: At 32 tokens from 576, VisCo preserves 91.8% of LLaVA-1.5’s original performance and surpasses VisPruner by 4.0 points in Avg.It also achieves the best results on four benchmarks.
  • Results on LLaVA-1.5: In Avg., VisCo surpasses PruMerge+ by 49.9 points and VisPruner by 36.5 points.The result suggests that a very small number of memory tokens can form compact yet effective visual representations.
  • Results on LLaVA-1.5: Under the 1-token setting, VisCo consistently outperforms MatryoshkaQuery and surpasses VoCo-LLaMA on GQA and MMB.VoCo-LLaMA and MatryoshkaQuery require substantially more large-scale retraining than VisCo’s lightweight fine-tuning setting.
  • Analysis: VisCo remains remarkably stable with denser visual information, whereas existing compression methods suffer severe performance degradation, especially on Qwen2-VL.These results support exploiting pretrained LLMs’ intrinsic capabilities for visual compression.

4.5 Ablation Study

VisCo remains effective at extreme visual-token compression, while ablations show that its hierarchical propagation and early memory tokens are central to performance. Additional results indicate that memory tokens encode complementary visual representations that can improve the original model.

  • Compression limits: At single-token compression, VisCo achieves accuracies of 50.8 on GQA and 52.8 on MMB, outperforming competing methods.Existing methods converge to similarly low accuracy as retained visual tokens decrease.
  • KV-cache propagation: Our-HierKVPass consistently outperforms Our-ValuePass, while the value-passing variant still surpasses other baselines, implicating both hierarchical propagation and the autoencoder architecture.The results support better exploitation of the LLM’s pretrained priors beyond hierarchical aggregation alone.
  • Complementary representations: Compressed tokens can outperform the original model on GQA and POPE, indicating that memory tokens provide complementary visual representations rather than merely summarizing original tokens.They encode visual information from a new perspective.
  • Memory-token positions: Dropping the first 18 of 36 memory tokens causes a substantial decline across five benchmarks, whereas dropping the last 18 has a much smaller effect.Attention visualizations show early tokens focus on main objects, while late tokens attend more sparsely to backgrounds and local details.

4.6 Efficiency Analysis

VisCo’s decoding acceleration offsets its added encoding overhead in long-response settings, while cached image representations provide efficiency benefits across repeated questions and reduce KV-cache memory. In multi-turn dialogue, VisCo reaches break-even at turn 3 and maintains advantages across evaluated scenarios.

  • Long-response efficiency: VisCo matches VisionZip’s decoding speed and outperforms Origin and FastV under both 128-token and 32-token settings on MMVet.The comparison uses LLaVA-1.5-7B with a fixed output length of 256 tokens.
  • Multi-turn efficiency: Turn 3 is VisCo’s break-even point for amortized average latency per dialogue turn on MME.Figure 7 reports per-turn latency and amortized average latency over dialogue turns.
  • Long-response efficiency: 0.2 s faster than FastV in both settings, VisCo offsets its compression overhead through decoding acceleration in long-response scenarios.VisCo also significantly reduces KV-cache storage.
  • Multi-turn efficiency: VisCo compresses each image once and reuses cached representations across dialogue turns, unlike FastV and SparseVLM, which recompress for every query.This reuse is especially beneficial when users ask multiple questions about the same image.
  • Overall efficiency: VisCo consistently reduces KV-cache memory across evaluated scenarios and shows clear advantages in long-response and single-image multi-question settings.Efficiency is evaluated on MMVet and MME using a single 48GB NVIDIA RTX 6000 GPU.

5 Conclusion

VisCo is a training-efficient visual token compression framework that uses the pretrained VLM itself as an intrinsic compressor. It applies lightweight adaptation while keeping the backbone intact, addressing degradation under aggressive compression and the retraining cost of external modules.

  • 5 Conclusion: VisCo leverages the pretrained VLM itself as an intrinsic compressor for visual token compression.The framework performs compression through lightweight adaptation rather than adding an external compression module.
  • 5 Conclusion: By keeping the backbone intact, VisCo avoids the severe degradation of training-free methods under aggressive compression.Its compression procedure uses lightweight adaptation while preserving the pretrained backbone.
  • 5 Conclusion: VisCo avoids the heavy retraining cost associated with external-module-based compression approaches.The method performs compression through lightweight adaptation instead of requiring substantial backbone retraining.
  • 5 Conclusion: Experiments covered 3 VLM backbones and 6 benchmarks, where VisCo consistently surpassed prior methods.The passage reports extensive evaluation across multiple backbones and benchmarks.
Loading 2607.12756v2…