PaperlayerView exact revision on arXiv

Source-linked AI summary

CoViST: Visual Token Compression via Composable States

Qi Zhang, Xiandong Meng, Ronggang Wang, Siwei Ma

arXiv:2609.33397v1cs.AI

TL;DR

Most visual-token compression methods leave each retained token’s represented evidence and original spatial position implicit, especially across repeated reductions. CoViST carries these quantities in a composable state and incorporates them into decoder attention. On LLaVA-1.5-7B, it reports high performance retention at multiple token budgets and the highest retention among compared methods under their respective budget settings.

  • Problem

    Most existing compression methods leave the visual evidence represented by each retained token and its original spatial context implicit, a limitation that becomes more pronounced under progressive compression.

    Source: §1 · pp. 1–2

  • Method

    CoViST is a training-free framework that carries representative features, original positions, contribution weights, and selection metadata in a composable state incorporated into decoder attention.

    Source: §1 · p. 2

  • Results

    CoViST-Fixed retains 99.9%, 99.5%, and 98.1% at 192, 128, and 64 tokens, while CoViST-Pro retains 99.8%, 99.9%, and 99.1% at those layer-average budgets on LLaVA-1.5-7B.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    One state and composition rule support both fixed and progressive compression, and CoViST has the highest retention among compared methods on LLaVA-1.5-7B at every budget under both strategies.

    Source: §5 · p. 10

    Aggregate benchmark retention does not guarantee reliability, fairness, or suitability for safety-critical use; the method also inherits limitations of its underlying models and evaluation benchmarks.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Visual token compression lowers the inference cost of vision--language models by representing images with fewer tokens. However, most existing methods compress visual tokens to a reduced set, leaving the amount of visual evidence represented by each token and its original spatial context implicit. Therefore, the compressed representation does not explicitly encode how much visual information each representative carries or where it lies in the original image. This limitation arises even after a single reduction and becomes more pronounced when compression is repeated across decoder layers. To address this issue, we propose CoViST, a training-free framework that represents a compressed image as a composable visual state. Specifically, the state combines representative features with original positions, effective contribution weights, and reusable selection metadata. CoViST constructs this state through coverage-guided selection and conservation-based contribution composition, and explicitly incorporates its contribution and positional information into decoder attention. Each component of the state retains its interpretation under successive reductions, enabling the same formulation to support both fixed compression before prefill and progressive compression within the decoder. Experimental results on seven LLaVA-1.5-7B benchmarks show that CoViST-Fixed retains 99.9\%, 99.5\%, and 98.1\% of uncompressed performance at 192, 128, and 64 tokens, respectively, and CoViST-Pro retains 99.8\%, 99.9\%, and 99.1\% at the corresponding layer-average budgets, outperforming state-of-the-art methods under their respective budget settings. Code will be released publicly.

1 INTRODUCTION

Existing compression methods leave each retained token’s represented evidence and original position implicit, a limitation that compounds under progressive reduction. CoViST carries this information in a composable state, supporting fixed and progressive compression; the paper reports the highest retention among compared methods under both strategies’ budget conventions.

1 INTRODUCTION

  • Existing methods leave each retained token’s represented visual evidence and original spatial context implicit, with the limitation worsening when reductions repeat across decoder layers.Visual tokens are processed by every decoder layer and retained in the key–value cache during generation, contributing substantially to inference cost.Source: §1 · pp. 1–2
  • CoViST represents compressed images as composable states that retain representative features, original positions, effective contribution weights, and reusable selection metadata across reductions.Closure under reduction lets the same state formulation support fixed compression before prefill and progressive compression within the decoder.Source: §1 · p. 2
  • Coverage-Guided State Initialization selects representatives, Coupled State Composition transfers contributions under per-image conservation with cross-stage inheritance, and State-Conditioned Attention incorporates weights and positions.The framework is training-free and supports CoViST-Fixed, which compresses to K tokens before prefill, and CoViST-Pro, which compresses progressively in the decoder.Source: §1 · p. 2
  • CoViST-Fixed and CoViST-Pro attain the highest retention among compared methods on LLaVA-1.5-7B under their respective budget conventions, accelerate inference, and generalize to three other models.The reported generalization models are LLaVA-1.5-13B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B.Source: §1 · p. 2

2 RELATED WORK

Prior visual-token compression methods use varied selection signals and operate at different stages, but generally pass reduced sets onward without preserving each representative’s original contribution. CoViST instead carries evidence-conserved weights, content, and positions together across reductions.

2 RELATED WORK

  • Training-free methods select tokens using encoder attention, feature diversity, instruction relevance, or coverage, and reduce them before or within the decoder, once or repeatedly.Related work also includes hybrid methods that reduce tokens in both locations and training-based methods that learn compact representations.Source: §2 · pp. 2–3
  • Existing methods generally omit records of how many original tokens each retained token represents, and later stages reselect from current features rather than carrying that record forward.CoViST also uses a coverage criterion for initialization, but carries the representation record across reductions.Source: §2 · p. 3
  • Prior compensation methods address content, attention contribution, or position, but treat them separately; CoViST combines them in one state with per-head attention bias and inherited weights.Prior approaches include token merging for content, attention-logit scaling or eviction-loss methods for contribution, and position retention or calibration.Source: §2 · p. 3

3 METHOD

CoViST represents compressed images as composable visual states that preserve representative features, original positions, contribution weights, and selection metadata through repeated reductions. Coverage-guided initialization, conservation-based composition, and state-conditioned attention support both fixed and progressive compression.

3.1 COMPOSABLE VISUAL STATES

  • A composable state preserves each component’s interpretation under reduction and explicitly incorporates contribution weights and original positions into decoder attention.Further reductions use the current state without retrieving discarded tokens.Source: §3.1 · p. 4

3.2 COVERAGE-GUIDED STATE INITIALIZATION

  • CSI allocates representatives to cover visual and instruction-relevant evidence, using joint encoder and decoder-input affinities and an instruction residual over visual evidence.It balances concentrated and diffuse evidence, protects attention peaks and non-redundant instruction anchors, and reserves part of the budget for remaining coverage gaps.Source: §3.2 · p. 4

3.3 COUPLED STATE COMPOSITION

  • CSC transfers removed tokens’ similarity-weighted contributions to representatives under an evidence-weighted conservation constraint, while confidence-gated content updates rotate features toward a weighted centroid without changing their norm.Later reductions inherit accumulated weights; the composed state retains original positions and restricts similarities and anchors to surviving representatives.Source: §3.3 · p. 5

3.4 STATE-CONDITIONED ATTENTION

  • SCA retains representatives’ original rotary positions and adds a bias based on contribution weights and positions to visual logits before softmax.Closed-form gains derived from frozen projections require no annotation, calibration, or training, and can be recomputed for another model.Source: §3.4 · pp. 5–6, Figure 3 · p. 5

3.5 FIXED AND PROGRESSIVE EXECUTION

  • CoViST-Fixed reduces once before the decoder, whereas CoViST-Pro initializes a larger state and reduces it at two intermediate prefill layers; both apply SCA at every layer.At the first internal reduction, CoViST-Pro scores current representatives using visual attention and instruction similarity, with protected anchors and spatial quotas.Source: §3.5 · pp. 6–7

4 EXPERIMENTS

Across the evaluated budgets, CoViST retains performance while supporting fixed and progressive compression, generalizes to three other models, and reduces generation latency. Ablations identify contribution composition and state inheritance as important components.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses LLaVA-1.5-7B across seven benchmarks, with comparisons drawn from published results under their own budgets, protocols, and full-model references.The reported retention averages task-score ratios; layer-average budgets describe decoder schedules rather than a fixed token count at every layer.Source: §4.1 · p. 7

4.2 MAIN RESULTS ON LLAVA-1.5-7B

  • CoViST-Fixed retains 99.9%, 99.5%, and 98.1% at 192, 128, and 64 tokens; CoViST-Pro retains 99.8%, 99.9%, and 99.1% at corresponding layer-average budgets.Both strategies rank first or second at every budget on LLaVA-1.5-7B.Source: §4.2 · p. 7

4.3 GENERALIZATION TO OTHER MODELS

  • On LLaVA-1.5-13B, CoViST-Pro retains 100.2%, 99.8%, and 99.0%; CoViST-Fixed retains 99.8%, 99.6%, and 98.2%.On Qwen2.5-VL-7B, CoViST-Fixed retains 99.4%, 98.5%, and 96.0%; on LLaVA-NeXT-7B, CoViST-Pro retains 100.4%, 99.7%, and 97.5%.Source: §4.3 · pp. 8–9, Table 4 · p. 8
  • On LLaVA-1.5-13B, CoViST-Pro and CoViST-Fixed rank first and second at every budget; on Qwen2.5-VL-7B, Fixed ranks first at every budget.For LLaVA-NeXT-7B, CoViST-Pro ranks first at all three budgets, exceeding the strongest compared method by 0.5, 2.6, and 2.3 points.Source: §4.3 · pp. 8–9, Table 4 · p. 8

4.4 INFERENCE EFFICIENCY

  • At equal layer-average budgets, CoViST-Pro has higher retention and CoViST-Fixed lower latency; generation speedups reach 2.22× and 1.92×, respectively, at 160 tokens.Measurements use LLaVA-NeXT-7B on POPE and include state-construction costs.Source: §4.4 · p. 9, Table 5 · p. 9

4.5 ABLATION STUDY

  • Conservation-based contribution composition brings the largest single gain at every tested budget, while state inheritance raises CoViST-Pro retention from 99.37% to 100.00% at 128 tokens.The ablation also finds that restoring original positional span adds 0.5 points at 64 tokens, and that one shared configuration changes retention by at most 0.4 points across tested constants.Source: §4.5 · p. 9, Table 6 · p. 9
  • The complete composed state raises retention at 64 tokens from 96.35% to 98.10%, and inheritance adds 0.38 points at 64 tokens.Removing attention bias lowers MME by 68 points at 192 tokens; compressing positional span costs over 1.4 points at 192 and 128 tokens.Source: §4.5 · p. 9, Table 6 · p. 9

5 CONCLUSION

CoViST represents compressed images as composable visual states and reports strong retention across models alongside faster generation. The authors caution that aggregate benchmark retention does not establish reliability or suitability for safety-critical use.

5 CONCLUSION

  • CoViST uses a composable visual state, closed under reduction and explicitly incorporated into decoder attention, to support fixed and progressive compression.The framework combines coverage-based representative selection, conservation-constrained contribution weights with inheritance, and positional information in attention.Source: §5 · p. 10
  • The paper identifies its module specifications, configuration constants, evaluation protocols, external comparison sources, and runtime measurement protocol across the main text and appendices.Code and runtime environment details are slated for public release.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The study evaluates existing VLMs on public benchmarks and introduces no new dataset or human-subject study.The conclusion reports generalization to three other models and generation acceleration on LLaVA-NeXT-7B of up to 2.22×.Source: §5 · p. 10, ETHICS STATEMENT · p. 11
  • The authors state that aggregate retention is not a guarantee of reliability, fairness, or suitability for safety-critical use.Compression can change fine-grained perception and the distribution of model answers even when average benchmark performance is preserved.Source: ETHICS STATEMENT · p. 11

A ATTENTION-OUTPUT ANALYSIS AND CONSERVATION IDENTITY

The appendix analyzes how replacing visual-token groups affects attention output and how contribution weights remain conserved across reductions. It gives a local error analysis and characterizes contribution withheld by the weight cap.

A.1 SETTING AND STATEMENT

  • Under shared query and text keys and values, attention-output error is controlled by representative-value error and the discrepancy in normalized group weights.The analysis assumes bounded representative and text values; it treats each text token as an unchanged singleton group.Source: §A.1 · p. 18, §A.3 · pp. 18–19

A.2 PERTURBATION OF NORMALIZED WEIGHTS

  • The normalized-weight perturbation bound used in the proof applies to positive measures and does not depend on a particular compression algorithm.The proof applies the lemma with visual-group ratios bounded by e^-δ and e^δ, yielding an L1 bound of 2 tanh(δ/2).Source: §A.2 · p. 18, §A.3 · p. 19

A.3 PROOF OF PROPOSITION 1

  • The proof bounds the value term by η and the normalized-weight term by V times the L1 difference between reference and compressed weights.The value term vanishes on text groups; visual-group probabilities sum to at most one.Source: §A.3 · pp. 18–19

A.4 DECOMPOSITION OF THE CONTRIBUTION DISCREPANCY

  • The contribution discrepancy is decomposed into within-group logit variation, representative-logit perturbation, and effective-contribution mismatch.The analysis keeps each representative's original RoPE index; renumbering would add another rotation term, while content updates affect both error terms.Source: §A.4 · p. 19

A.5 CONSERVATION UNDER THE UPPER BOUND

  • The upper-bound identity confines withheld evidence-weighted contribution to representatives that reach the contribution cap, and it remains valid with inherited weights.At later reductions, conservation holds within each stage under that stage's evidence; this identity concerns contribution, not the full attention output.Source: §A.5 · p. 19

B IMPLEMENTATION DETAILS

The implementation details specify evidence construction, representative selection, contribution composition, and state-conditioned attention. Progressive selection reuses inherited state metadata and positions across reductions.

B IMPLEMENTATION DETAILS

  • The shared implementation configuration applies to both execution strategies and all four models, with schedules and constants specified separately.When Qwen2.5-VL lacks a CLIP text encoder, instruction evidence comes from its first decoder layer and uses no instruction anchors.Source: B IMPLEMENTATION DETAILS · pp. 19, 21, Table 7 · p. 20

B.1 CSI: VISUAL AND INSTRUCTION EVIDENCE

  • Visual evidence combines denoised encoder attention with feature distinctiveness, while instruction evidence is derived from entropy-filtered CLIP text features.The visual distinctiveness mixes global and local terms; instruction evidence contributes a bounded residual over visual evidence.Source: §B.1 · p. 21

B.2 CSI: COVERAGE WEIGHTS, ANCHORS, AND RESERVE

  • CSI balances evidence across 16 regions, protects attention-core and qualifying instruction-anchor tokens, then fills remaining positions through greedy coverage and a coverage-gap reserve.The reserve is 5%; the selection returns indices in original spatial order.Source: §B.2 · pp. 21–22

B.3 CSC: CONFIDENCE

  • CSC uses semantic similarity to assign removed tokens to representatives, while spatial and feature affinities contribute to confidence rather than choosing the correspondence.The confidence uses original image coordinates; the content update uses current hidden features without retrieving features removed earlier.Source: §B.3 · p. 22

B.4 SCA: HEAD GAINS AND POSITIONS

  • SCA derives per-head gains from frozen projection matrices, mapping relative frequency emphasis to the range 0.6–1.4 without head labels.Representatives retain original image indices for RoPE and distance factors, and the causal attention mask is preserved.Source: §B.4 · p. 22

B.5 PROGRESSIVE COMPOSITION AND INTERNAL SELECTION

  • Progressive reductions reuse surviving similarity metadata, original positions, inherited weights, current hidden states, and stage-specific evidence for selection and composition.Internal selection protects surviving anchors and allocates quotas to coordinate-based selection, stratified spatial selection, and a coverage-gap reserve.Source: §B.5 · pp. 22–23, Table 8 · p. 23

C SCHEDULES AND COMPUTATIONAL ACCOUNTING

CoViST-Pro distributes visual-token counts across decoder layers to meet target layer-average budgets. Its decoder cost depends on token exposure and schedule distribution, while additional pipeline components contribute to measured generation time.

C SCHEDULES AND COMPUTATIONAL ACCOUNTING

  • For LLaVA-1.5-7B, CoViST-Pro schedules 168, 112, and 56 tokens to achieve exact layer averages of 192, 128, and 64.The progressive schedule uses reductions at prescribed decoder layers; retained representatives depend on intermediate computation.Source: C SCHEDULES AND COMPUTATIONAL ACCOUNTING · p. 23
  • Decoder cost is approximated from sequence length, hidden size, feed-forward size, and token count at each layer.Equal target K approximately aligns terms linear in sequence length, but not the sum of squared layer sequence lengths.Source: C SCHEDULES AND COMPUTATIONAL ACCOUNTING · p. 23
  • CoViST-Pro's visual-only quadratic decoder term is about 1.25 times CoViST-Fixed's across the three reported schedules.Measured generation times also include encoders, initial pairwise similarities, internal selection, and bias construction; the initial pairwise kernel has quadratic storage in original patch count.Source: C SCHEDULES AND COMPUTATIONAL ACCOUNTING · p. 23

D.1 TASKS, METRICS, AND RETENTION

The task and metric definitions specify how CoViST performance is assessed on LLaVA-1.5-7B. Retention is an equal-weight average of six task-specific ratios to the uncompressed reference.

D.1 TASKS, METRICS, AND RETENTION

  • Retention RPC (%) averages six task-specific ratios to the uncompressed reference across the reported LLaVA-1.5-7B benchmarks.The protocol uses task-specific measures including GQA exact match, POPE F1, MME aggregates, and OCR-augmented TextVQA.Source: §D.1 · p. 23, Table 9 · p. 24

D.2 CROSS-MODEL IMPLEMENTATION

The cross-model appendix specifies architecture-specific interfaces for LLaVA-NeXT and Qwen2.5-VL, then tests controlled interventions, position span, and inherited state during progressive compression.

D.2 CROSS-MODEL IMPLEMENTATION

  • For LLaVA-NeXT, compression follows image-feature packing; 48 newline markers retain their order and positions, are excluded from selection, and are added to the content-token budget.The base view and high-resolution grid contribute 576 and 48×48 tokens; CoViST-Fixed delivers K + 48 visual tokens.Source: §D.2 · p. 24
  • Qwen2.5-VL selection uses visual attention and language-block projections for evidence and instruction relevance, while preserving temporal, height, and width positions for attention correction.Descriptors are restored from window order and averaged within native merger groups before selection.Source: §D.2 · p. 24

D.3 CONTROLLED INTERVENTIONS

  • The paired MME intervention holds representative indices, token order, anchors, prompts, and positions fixed while separately disabling content composition or attention bias.Disabling content composition leaves effective weights unchanged; disabling attention bias removes its additional attention contribution.Source: §D.3 · p. 24, Table 12 · p. 25

D.4 CUMULATIVE STUDY AND POSITION SPAN

  • The full 576-position span is best at 192 and 128 tokens and within 0.1 point of the best at 64; compressed spans lose most on MME perception, ScienceQA, and TextVQA.The comparison varies position span around the main configuration; the full span preserves original grid geometry.Source: §D.4 · p. 26, Table 13 · p. 26

D.5 STATE INHERITANCE UNDER COVIST-PRO

  • At 64 tokens, the final 16 representatives average contribution weight 6.86 with inherited state, versus unit weights when internal reductions reset contributions and discard inherited anchors.The two conditions share the first stage and schedule, differing only at the internal reductions.Source: §D.5 · p. 26

D.6 SENSITIVITY

The appendix reports sensitivity to configuration changes and documents how comparison sources, task coverage, and budget semantics are handled.

D.6 SENSITIVITY

  • Every one-at-a-time CoViST-Fixed variation stays within 0.4 points of the main configuration at each budget, while nearby CoViST-Pro first-stage gains stay within 0.15 points.The first-stage gains of 0.75 and 0.85 are included; the authors consequently use one shared configuration across budgets.Source: §D.6 · pp. 26–27, Table 14 · p. 26, Table 15 · p. 26

E.1 SOURCES, COVERAGE, AND NORMALIZATION

  • Retention comparisons average six task ratios from Table 9, not published averages over other task suites, and external results are normalized to their source-specific full-model references.For example, SCoRe uses a ScienceQA reference of 66.8 while most other sources use 69.5.Source: §E.1 · p. 27

E.2 COMPLETE LLAVA-1.5-7B COMPARISONS

  • Tables 16 and 17 provide six task scores across budgets for pre-decoder fixed counts and decoder-prefill compression, with distinct panels for MME-PC, MME-P, RPC, and RP.The tables preserve source labels and distinguish layer-average, post-pruning, and sample-average budget conventions.Source: Table 16 · p. 27, §E.2 · pp. 27, 29, 31, Table 17 · p. 30

E.3 COMPRESSION LOCATION AND BUDGET SEMANTICS

  • Comparison eligibility excludes missing-score imputation and settings outside the fixed-count or specified budget conventions; several other-backbone, FLOPs-based, and incomplete-task reports are omitted.Layer schedules are averaged across every decoder layer before and after pruning; a sample-average final count does not determine that layer average.Source: §E.3 · p. 32

E.4 CROSS-MODEL SOURCES AND PER-TASK COMPARISONS

Cross-model comparisons retain source-specific evaluation conventions and define separate aggregates for different task subsets; a related-work table compares which correction quantities methods modify.

E.4 CROSS-MODEL SOURCES AND PER-TASK COMPARISONS

  • Cross-model comparisons normalize each external row by its source’s full model; our rows use our own full model, as sources report VQAv2 test-dev 80.0 versus our validation split 78.3.Source: §E.4 · p. 34
  • The appendix draws cross-model rows from specified source reports and keeps separately defined task aggregates distinct rather than combining their task sets.The two Qwen2.5-VL R4 panels use different four-task subsets; their TextVQA prompt conventions also differ.Source: §E.4 · pp. 34–35, Table S4
  • Only pre-decoder fixed-count configurations enter these appendix panels; missing task scores are not imputed, and dynamic-resolution keep ratios are not converted to fixed counts.Methods that compress inside the language model under layer-average budgets are compared in the main tables instead.Source: §E.4 · p. 35

F DESIGN COMPARISON WITH RELATED CORRECTIONS

  • The design comparison organizes attention-level corrections by representative weight, weight bound, head dependence, position treatment, inheritance, and feature treatment.Its entries summarize the cited papers and mark unreported aspects as “n.r.”Source: F DESIGN COMPARISON WITH RELATED… · p. 35

G RUNTIME MEASUREMENT PROTOCOL

The runtime protocol defines workload accounting, timing boundaries, and efficiency metrics for LLaVA-NeXT measurements, while comparison tables specify their own model and task scopes.

G RUNTIME MEASUREMENT PROTOCOL

  • The workload uses 672 × 672 inputs yielding 2,880 content tokens and 48 newline tokens before compression, with every newline token retained.CoViST-Pro visual-token counts are layer averages, not final-stage counts.Source: G RUNTIME MEASUREMENT PROTOCOL · pp. 35, 37, Table 27 · p. 39
  • The NeXT and Qwen2.5-VL comparison tables use source-defined panels and task subsets, with separate aggregates and budget conventions stated for each.The Qwen2.5-VL R3 aggregate covers three tasks, while its two R4 columns cover distinct four-task subsets.Source: Table 23 · p. 36, Table 24 · p. 37, Table 25 · p. 38
  • Measurements use one A800-SXM4-80GB GPU, FP16, SDPA, and 100 POPE prompts timed across three repetitions with greedy generation and KV caching.The three repetitions of every prompt produce identical outputs.Source: G RUNTIME MEASUREMENT PROTOCOL · p. 37
  • TTFT ends at the first output token, while total generation time includes visual encoding, compression, attention modulation, and autoregressive generation but excludes loading and preprocessing.The prefill KV size counts keys and values across decoder layers, including newline tokens; generation speedup divides uncompressed mean time by compressed mean time.Source: G RUNTIME MEASUREMENT PROTOCOL · p. 37
Loading 2609.33397v1…