PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-centric Token Compression in Large Language Model

Ling Xing, Alex Jinpeng Wang, Rui Yan, Xiangbo Shu, Jinhui Tang

arXiv:2502.00791v5cs.CLcs.CV

TL;DR

Longer contexts and larger LLMs make input compression increasingly necessary. VIST uses a slow-fast visual compression framework with frequency-informed supervision, and reports accuracy gains over CEPE alongside lower token, FLOP, and memory costs.

  • Problem

    Long-context tasks and growing LLM parameter counts make it necessary to compress inputs because LLMs cannot afford to reason over the desired information without shrinking them.

    Source: §1 · p. 1

  • Method

    VIST renders loosely relevant long context as images for a frozen vision encoder and trainable Resampler, while PVE masks high-frequency tokens to guide the Resampler toward informative content.

    Source: §1 · pp. 1–2

  • Results

    VIST gains 3.6% over CEPE across 11 in-context learning datasets and 5.7% across 3 open-domain question-answering datasets; it uses 2.3× fewer visual tokens for the same input.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    A lightweight vision encoder offers a cost-efficient alternative to full-scale LLM computation for compressing loosely relevant long contexts.

    Source: §1 · p. 2

Abstract

from arXiv · show

Real-world applications are stretching context windows to hundreds of thousand of tokens while Large Language Models (LLMs) swell from billions to trillions of parameters. This dual expansion send compute and memory costs skyrocketing, making token compression indispensable. We introduce Vision Centric Token Compression (Vist), a slow-fast compression framework that mirrors human reading: the fast path renders distant tokens into images, letting a frozen, lightweight vision encoder skim the low-salience context; the slow path feeds the proximal window into the LLM for fine-grained reasoning. A Probability-Informed Visual Enhancement (PVE) objective masks high-frequency tokens during training, steering the Resampler to concentrate on semantically rich regions-just as skilled reader gloss over function words. On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%. This method delivers remarkable results, outperforming the strongest text encoder-based compression method CEPE by 7.6% on average over benchmarks like TriviaQA, NQ, PopQA, NLUI, and CLIN, setting a new standard for token efficiency in LLMs. The project is at https://github.com/CSU-JPG/VIST.

1 Introduction

VIST addresses the cost of longer contexts and larger LLMs with a slow-fast framework that sends loosely relevant text through a visual encoder and reserves the LLM for reasoning. It reports accuracy gains over CEPE alongside lower token, FLOP, and memory costs.

1 Introduction

  • Longer contexts and larger LLMs make compression necessary because LLMs cannot afford to reason over the desired information without shrinking the input.The introduction describes parameter counts growing from billions to trillions while real-world tasks require inputs far beyond early thousand-token regimes [2] [4] [5] [6] [7].Source: §1 · p. 1
  • The framework renders loosely relevant long contexts as images for a frozen vision encoder and trainable Resampler, then sends the compressed visual tokens and main input tokens to the LLM.This slow-fast arrangement mirrors selective reading: distant, low-salience context is skimmed while nearby sentences receive deeper attention.Source: §1 · p. 1
  • PVE uses frequency-based masking to suppress high-frequency text tokens, guiding the Resampler toward informative content and reducing redundancy in visual tokens.VIST uses token frequency as a proxy for importance rather than LLM-computed token-level information entropy [13] [14] [16] [17].Source: §1 · p. 2
  • Visual text tokenization avoids vocabulary bottlenecks and reports fewer tokens for Japanese (62%), Korean (78%), and Chinese (27%).The paper also describes simpler tokenization and greater resilience to character-level noise as potential advantages over text tokenizers.Source: §1 · p. 2
  • VIST gains 3.6% over CEPE across 11 in-context learning datasets and 5.7% across 3 open-domain question-answering datasets; it also uses 2.3× fewer visual tokens for the same input.The comparison reports 16% lower FLOPs and 50% lower memory usage.Source: §1 · pp. 2–3

2 Related Work

Prior context-compression work either uses LLMs to compress or select tokens, or relies on lightweight text encoders and external memory. VIST instead uses visual representations to compress long text and is presented as compatible with existing strategies.

2 Related Work

  • LLM-based soft-prompt and selection methods process long inputs with heavy LLMs, while external-memory approaches store and retrieve prior context; VIST is described as orthogonal and combinable with these strategies.Selection methods remove redundant tokens using information entropy computed by LLMs [13] [14] [16] [17] [27].Source: §2 · p. 3
  • Unlike CEPE's lightweight text encoder and cross-attention, VIST compresses long text into visual tokens guided by semantically rich text tokens [21].Related tokenizer-free visual methods include Pixel [20], but the passage says they overlook long-text scenarios or rely on complicated training pipelines.Source: §2 · p. 3

3 Methodology

VIST compresses distant context into visual tokens for the LLM while preserving a raw-text window for fine-grained reasoning. Its PVE objective uses frequency-informed text supervision to help the Resampler extract semantically meaningful content from rendered text.

3.1 Overall Pipeline

  • VIST splits long input into distant text rendered as images for a fast visual path and a raw-text window for fine-grained LLM reasoning.The compressed visual tokens enter the LLM through cross-attention.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Vision-centric Implementation

  • Text is rendered as uniformly distributed RGB images, with empty patches masked from attention and loss computations.The number of images can vary with input length.Source: §3.2 · p. 4

3.3 Token Reduction

  • With 28 images and 64 visual tokens per image, VIST compresses 4,096 text tokens into 1,792 visual tokens for cross-attention.Image and token counts can be adjusted during training and inference to control the compression ratio.Source: §3.3 · p. 4

3.4 Probability-informed Visual Enhancement

  • PVE’s contrastive objective aligns Resampler visual features with text-token embeddings, including embeddings formed after frequency masking and pooling.This alignment is intended to bridge the semantic gap between visual and raw text tokens.Source: §3.1 · p. 3, §3.4 · p. 5
  • PVE uses corpus frequency as a proxy for token importance, making lower-importance tokens more likely to be masked at a 50% rate.The resulting text embeddings provide denser supervision for the Resampler’s extraction of semantic features.Source: §3.4 · p. 5

4 Experiment

The experiments evaluate VIST on long-context language modeling, in-context learning, and open-domain question answering, then examine compression and training choices. They also compare VIST with text- and vision-based baselines and test it with Mistral 7B.

4.2 Long-context Language Modeling

  • 14.973 PPL on Book at Te=Td=2048 was VIST’s lowest, while at 16k tokens it matched CEPE* with 16% fewer FLOPs and 50% less memory.VIST’s perplexity decreased consistently as input length increased; Table 1 also reports throughput relative to TinyLlama and compression ratio.Source: Table 1 · p. 6, §4.2 · p. 6

4.1 Experimental Setup

  • The evaluation compared TinyLlama-based VIST with Replug, Stream, CEPE*, and compression-matched ToMe† and FastV†.Training used a frozen ViT-L/14 encoder and a 1B-token RedPajama sample spanning seven domains, with 4,096 tokens assigned to the encoder and 512 to the LLM.Source: §4.1 · pp. 5–6

4.3 In-context Learning

  • Across 11 datasets, VIST outperformed CEPE* in average accuracy; increasing encoder demonstrations to ne=18 raised accuracy from 33.3% to 46.3%.VIST and CEPE* used two demonstrations in the decoder, while most demonstrations were processed by the lightweight encoder.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 Open-domain Question Answering

  • At ke=20 and kd=10, VIST scored 9.11 EM points higher than CEPE* on TriviaQA; fast-path-only VIST matched TinyLlama using the top 10 passages.TinyLlama’s 2,048-token context limit constrained how many passages it could process.Source: Table 3 · p. 7, §4.4 · pp. 7–8

4.5 Ablation Study

  • On Mistral 7B, VIST‡ outperformed CEPE‡ while processing 4,096 LCM tokens or 20 ICL demonstrations, versus 2,048 tokens or two demonstrations for Mistral.The paper presents this as an extension of VIST to another LLM.Source: Table 7 · p. 8, §4.5 · p. 9
  • 64 visual tokens performed best on four of five datasets, while 128 performed best only on MR, showing that more visual tokens did not consistently improve results.The study attributes the trade-off to fewer tokens risking lost details and more tokens retaining irrelevant information.Source: §4.5 · p. 8, Table 4 · p. 8
  • Removing frequency-based masking reduced ICL and open-domain QA performance; the compared masking strategies used a 50% masking ratio.The study describes frequency-based masking as improving the information density of text-token embeddings guiding the visual encoder.Source: §4.5 · p. 8, Table 5 · p. 8
  • 42.7 on SST5 came with 4,096-token encoder training, while MR peaked at 90.5 with 6,144 tokens; longer inputs generally helped QA and ICL.Training with 1,024 tokens performed comparably to 2,048 tokens in the reported observation.Source: Table 6 · p. 8, §4.5 · p. 9

5 Discussion

The discussion finds that token frequency tracks semantic importance: masking rare tokens disrupts text–visual alignment, while frequency-based masking improves alignment and preserves key content while filtering less relevant words.

5 Discussion

  • At masking ratios 0.0 to 0.4, masking rare tokens sharply increases distance, while masking frequent tokens has minimal impact or reduces it.The comparison uses training-set token frequencies to assess importance across Book, PG19, Proof, and Code.Source: §5 · p. 9
  • Across masking ratios 0.0 to 0.9, VIST with frequency-based masking has smaller text–visual semantic distances than VIST without it on Arxiv and Book.The distances sum cosine distances between text token embeddings and visual features across test samples.Source: §5 · p. 9
  • Masking the top 50% most frequent tokens preserves key nouns and mainly removes function words, supporting frequency-based masking as a way to filter less relevant noise.The passage describes function words such as “the” and “of” as serving grammatical roles rather than conveying core semantic meaning.Source: §5 · pp. 9–10

6 Conclusion

The conclusion presents VIST as a vision-encoder-based approach to longer LLM inputs, motivated by redundancy in text tokens. It reports that VIST surpasses text-encoder-based compression counterparts in performance and efficiency, while identifying broader downstream evaluation as future work.

6 Conclusion

  • VIST surpasses text-encoder-based token compression counterparts in both performance and efficiency.The authors plan to evaluate VIST across a broader range of downstream tasks and investigate text token redundancy more deeply.Source: §6 · p. 10
  • VIST applies a lightweight visual encoder framework to address the challenge of longer input sequences for in-context learning.The conclusion characterizes VIST as a vision-centric token expansion method.Source: §6 · p. 10
  • The authors identify significant text-token redundancy as further support for their vision-encoder-based approach.Source: §6 · p. 10

Related papers

Loading 2502.00791v5…