Just Respect the Semantics: Image-Token Compression via Pre-LLM Semantic Variable-Resolution Tokenization and In-LLM Relevance Pruning

• Image-token compression must reduce decoder computation while preserving meaning because tokens can represent object parts, object boundaries, or just a few characters in a sign.Source: §1 · p. 1
• Semantic fungibility means tokens for a coherent concept can be summarized, while mixed-object regions and literal text have lower fungibility and require finer distinctions.Source: §1 · pp. 1–2
• Varest combines object- and text-guided variable-resolution tokenization with learned [REL] pruning after early decoder processing; optional adaptive halting can further reduce visual-token propagation.Source: §1 · p. 2
• Across three VLM backbones and 13 benchmarks, Varest achieves the highest mean performance retention among 15 baselines at every evaluated budget, with similar or lower computational cost.Source: §1 · p. 2

2. Method

• Given an image and prompt, Varest forms full-coverage regions totaling Km image tokens, processes them with the prompt and four [REL] registers, then retains Kp tokens at the decoder midpoint.Adaptive halting can remove all image tokens after layer 8, even before midpoint pruning, in which case [REL] pruning is skipped.Source: §2 · p. 2

2.1. Semantic Variable-Resolution Tokenization

• Object-boundary and textness cues guide merging: text cells remain at native resolution when budget allows, while weakly separated content is merged more coarsely and every raw token contributes to a region.The budget sets semantic granularity, and the boundary signal also incorporates text-border evidence to discourage merging across text transitions.Source: §2.1 · pp. 2–3

2.2. [REL] Relevance Pruning

• Four learned [REL] registers score contextualized image tokens after the first L/2 decoder layers and retain the Kp highest-scoring tokens for the remaining layers.The registers can access image and prompt but not target answer; selection is trained with a straight-through sigmoid surrogate.Source: §2.2 · p. 3

2.3. Optional Adaptive Halting

• Optional training-free adaptive halting uses intermediate answer-token predictions and removes image tokens and [REL] registers at the first layer meeting warm-up, three-layer stability, and entropy-confidence criteria.Halting is disallowed before layer 8; the top prediction must agree across three consecutive layers and entropy must be at most 1 nat.Source: §2.3 · p. 4

2.4. Training and Budget Control

• Joint adaptation trains the registers, scoring head, projector, position tags, LoRA adapters, and vocabulary head across merging and pruning budgets, while keeping localizers frozen and requiring no relevance labels.The reported image-token budget is the mean number entering a decoder layer; for LLaVA-NeXT, the requested ratio is applied to each image’s native sequence length.Source: §2.4 · p. 4

• Compression methods use semantic, saliency, or query cues: image-only cues do not explicitly account for object boundaries or text detail, while decoder-attention query cues depend on processing whose reliability can vary by layer.Source: §3 · p. 4
• CORE uses object masks for pooling but does not explicitly allocate finer resolution to text within an object; Varest separates detected text from its surrounding object and preserves native patch-grid resolution when budget permits.Source: §3 · pp. 4–5
• Across decoder depth, Varest learns relevance scores for contextualized semantic regions through [REL] registers rather than using attention weights as an importance proxy.The cited discussion notes that attention weights may correlate poorly with gradient-based importance and that different attention distributions can yield similar predictions.Source: §3 · p. 5

4.1. Qualitative Analysis

• At K = 32, Varest identifies text cells and combines them with merged-object context to answer correctly at very small budgets.Qualitative examples show that Varest can answer correctly even when [REL] does not retain the named tokens, consistent with information being absorbed before pruning.Source: §4.1 · pp. 5–6, Figure 6 · p. 6

4.2. Quantitative Analysis

• 86.10–92.33% mean retention at 5.56% of the original layer-averaged token budget; Varest leads all 15 baselines at every budget across three backbones.Retention is normalized against each uncompressed backbone and averaged over 15 metrics.Source: Figure 5 · p. 6, §4.2 · p. 6, Table 1 · p. 7

4.4. Efficiency Analysis

• Varest improves retention over HiDrop by 3.79–6.39 percentage points while using 1.0–4.7% fewer FLOPs; adaptive halting further trades 3.0–5.1% fewer FLOPs for 0.30–0.48 points of retention loss.The Figure 7 comparison averages retention and measured computation over four budgets; both settings use the same checkpoint.Source: §4.4 · pp. 6–7, Figure 7 · p. 7

4.3. Adaptive Halting

• Adaptive halting cuts visual-token propagation by 8.6–12.9% with mean score losses of 0.10–0.21 points; on LLaVA-1.5-13B, 28.8% of inputs halt before selection.Score changes are averaged equally over 15 metrics and four budgets on native 0–100 scales; early withdrawal contributes to the larger savings on LLaVA-1.5-13B.Source: §4.3 · p. 6, Table 2 · p. 7

4.5. Ablation Studies

• At the smallest budgets, removing object localization lowers retention by 5.76–11.99 points, while removing both localizers lowers it by 5.99–14.28 points; text-localizer effects vary by benchmark.Without text localization, TextVQA drops 1.21–2.24 native score points at the smallest budgets, while effects on other benchmarks vary.Source: §4.5 · pp. 7–8, Table 3 · p. 8
• At Km = 64, replacing object-boundary ordering lowers reconstruction IoU from 68.43% to 45.01%, while zeroing textness lowers individual answer-cell retention from 43.2% to 17.1%.Removing both cues lowers text preservation further to 7.9%.Source: §4.5 · p. 8, Table 5 · p. 8
• The joint model gains 2.41–3.26 retention points over the separately trained tokenizer-only arm at the smallest budgets, and learned selection beats random selection at every budget.Oracle Selection remains within 0.71 retention points of the Varest reference at every budget.Source: §4.5 · p. 8, Table 4 · p. 8

5. Discussion

• Efficient implementation is also required to translate Varest’s computational savings into lower latency.The discussion states that the method is compatible with FlashAttention [9], but compatibility alone does not guarantee lower latency.Source: §5 · p. 8
• Adapting Varest to Qwen-VL, InternVL, and medical images remains future work requiring domain-specific validation.The discussion suggests that healthy organs could be treated like objects and lesions like text.Source: §5 · p. 8
• The acknowledgements identify VarestBench and its dataset as an efficiency benchmark developed by CC and MT.The same passage lists the contributors’ roles in implementation, training, visualization, and baseline experiments.Source: §5 · p. 9

Supplementary Material

• Varest combines boundary- and textness-guided partitioning, mean-pooled region tokens with learned position tags, and midpoint selection by four learned [REL] registers.The overview also describes optional adaptive halting when intermediate answer predictions become stable and confident.Source: Figure 8 · p. 11

A. Method and Training Details

• The appendix details the method architecture and components, then presents training protocols and baseline adjustments and evaluation protocols.These topics are organized in Sections A.1, A.2, and A.3, respectively.Source: A. Method and Training Details · p. 11

A.1.1. Localizer Architecture and Field Alignment

• The object localizer predicts right- and lower-neighbor boundary strengths, while the grayscale text localizer predicts per-cell textness; both are lightweight and remain frozen during VLM adaptation and inference.The object localizer uses RGB inputs and DINOv3 supervision; the text localizer uses grayscale inputs and EasyOCR supervision.Source: §A.1.1 · pp. 11–12

A.1.2. Token-Budget Allocation

• Text and non-text cells receive separate region quotas, preserving individual text cells when the budget allows and allocating remaining capacity to non-text content.The quota procedure aims to attain the requested region count when sufficient cells are available; Figure 12 shows text retained at native resolution across budgets.Source: §A.1.2 · p. 14, Figure 12 · p. 14

A.1.3. Region-Merging Procedure

• Within each text or non-text group, neighboring cells are merged by lowest mean shared-boundary strength until its allocated region count is reached, preserving coverage of every grid cell.The groups are processed independently, so merging does not cross the text/non-text split.Source: §A.1.3 · pp. 14–15

A.1.4. Region Encoding and Backbone Integration

• Each region’s token is the mean of its projected patch features, augmented with a learned tag encoding its normalized centroid and extent.For LLaVA-NeXT, regions are encoded separately on the base and stitched high-resolution views.Source: §A.1.4 · pp. 15–16

A.1.5. Decoder-Stage Selection

• At the decoder midpoint, four learned [REL] registers select the Kp image tokens retained for later layers; training uses attention masking and a straight-through surrogate, while inference uses hard top-k selection.At inference, retained tokens keep their original order and position tags, while discarded image tokens and registers are physically removed.Source: §A.1.5 · pp. 16–17

A.1.6. Adaptive Halting Implementation

• During prefill, adaptive halting removes image tokens and registers when three consecutive top predictions agree and current readout entropy is at most 1, taking precedence over midpoint pruning.If no halting trigger occurs, standard [REL] pruning proceeds; halting is not reevaluated during generation.Source: §A.1.5 · p. 17, §A.1.6 · p. 18

A.2. Training Details

• The object localizer is distilled from DINOv3 neighbor-feature cosine distances, while the text localizer learns from EasyOCR cell labels using positive-weighted binary cross-entropy.Text-free negatives are excluded when estimating the positive-class weight.Source: §A.2 · pp. 18–19
• Joint adaptation uses answer-token cross-entropy on the backbones’ instruction-tuning data, sampling merging and pruning budgets during training.The reported training times are 54, 61, and 92 hours on one H100 for LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-NeXT-8B, respectively.Source: §A.2 · p. 19

A.3. Evaluation Settings

• Evaluation settings specify benchmark splits, scoring rules, and task prompts; generation uses greedy decoding, one beam, and KV caching.The listed evaluation sets are used without an additional sample limit.Source: §A.3 · p. 19

A.4. Compute Resources

• 35/45 words are read correctly by the unmerged model in the word-reading comparison.Table 8 compares text and background averaging interventions and mean replacement with mean-and-delete; all 45 words enter each result.Source: Table 8 · p. 20

B. Additional Experiments

• Additional experiments examine how merging affects answers and how averaging affects word reading and retained text resolution across budgets.Figure 17 illustrates selected reading-error examples with all 576 tokens retained; Table 8 reports the full cohort.Source: B. Additional Experiments · p. 20, Figure 17 · p. 20

Semantic compatibility and answering

• Merging two people into one token preserves the answer “Blue,” but including the snowboard in that token changes it to “White.”This example does not imply that every within-category merge preserves all task-relevant details.Source: §B.1 · p. 20

Boundary guidance across budgets

• At K_m = 32, learned boundaries achieve 54.21% region IoU versus 25.74% for CLIP-B; at K_m = 224, they achieve 80.13% versus 76.28%.The larger advantage under tight compression indicates boundary guidance matters most when regions summarize larger portions of the image.Source: §B.1 · pp. 20–21

Boundary leakage at tight budgets

• At K_m = 51, learned regions still cross object boundaries; in one selected example, mask IoU rises from 56% to 89% at K_m = 358.Boundary guidance is a useful allocation cue, but the available budget limits which distinctions can be preserved.Source: §B.1 · p. 21

B.2. Text Resolution Across Budgets

• Averaging alone can change a unit or letter, although not every text merge changes reading; equal-size background interventions preserve the unmerged score.The evaluation uses 45 of 48 TextOCR word crops, with disjoint horizontal pairs averaged; mean replacement and mean-and-delete separate averaging from token removal.Source: §B.2 · p. 21
• Textness guidance preserves finer resolution for answer-text cells as budget grows, with a larger benefit on TextVQA than DocVQA at low budgets.These measurements track unmerged answer-text cells, not reading accuracy.Source: §B.2 · p. 21
• At tight budgets, flagged text cells can outnumber allocated tokens and be merged; with sufficient budget, each flagged cell can receive its own region.The number of flagged cells also varies with the budget-dependent text threshold.Source: §B.2 · p. 21
• Missed text, such as an unflagged logo, receives no text-specific allocation, while false positives can assign text tokens to non-text shapes.These cases distinguish localization errors from budget-driven merging of correctly detected text.Source: §B.2 · p. 21

C. Further Quantitative Results

• The paper directs readers to further quantitative results comparing per-backbone baselines and evaluating object- and text-localizer prediction.The cited section pointer does not report those results here.Source: C. Further Quantitative Results · p. 21

C.1. Per-Benchmark Performance

• Varest records 39 metric wins on LLaVA-1.5-7B, 39 on LLaVA-1.5-13B, and 47 on LLaVA-NeXT-8B across the reported budgets.The runner-up counts are 19 wins for DUET-VLM on 7B, 10 for HiDrop on 13B, and 6 for HiDrop on NeXT-8B.Source: §C.1 · pp. 21–22

D. Adaptive Halting

• The further-results discussion includes information-horizon comparisons by benchmark and backbone, with horizons varying across tasks and backbones.Text-, chart-, and document-heavy tasks require late visual access in LLaVA, while grounding tasks reach the final decoder layer in Qwen.Source: Figure 18 · p. 22, D. Adaptive Halting · p. 22

C.2. Object-Localizer Accuracy

• The object localizer is within 0.1 AP and 0.8 ODS-F of its DINOv3-L/16 teacher on the native grid, and matches it at 51.0 vs. 50.5 AP and identical 51.1 ODS-F on tokenizer cells.Both learned fields far outperform Sobel gradient, by about 33 AP on the native grid and 30 AP on tokenizer cells.Source: §C.2 · p. 22

C.3. Text-Localizer Accuracy

• On 1,432 held-out text-bearing images, the text localizer and text-only version achieve 97.1 vs. 72.8 AUC over Sobel edge density.Training with text-free negatives reduces false detections: only 0.4–1.6% of text-free images receive a false text cell, versus 10.9–46.5% for the text-only version.Source: §C.3 · p. 22

D.1. Information Horizon Analysis

• A performance-based horizon is the earliest withdrawal point that keeps relative score loss within tolerance; dropping after layer l preserves access through l and removes visual tokens afterward.The analysis searches horizons for ϵ ∈ {1%, 2.5%, 5%} on 12 benchmarks comprising 14 targets, using LLaVA-1.5-7B and Qwen2.5-VL-7B.Source: §D.1 · pp. 22–23

D.2. Task-wise Halting Depth and Performance

• Figure 19 relates adaptive halt depths to benchmark-level 1% horizons and answer-score changes; its saved-work measure counts image-token processing across decoder layers, not whole-model speedup.Table 13 reports task-wise score changes against the same checkpoint without halting, averaged over four budgets.Source: Figure 19 · p. 23, §D.2 · p. 24, Table 13 · p. 24

D.1.1. LLaVA-1.5-7B

• For LLaVA-1.5-7B, the 1% horizon ranges from L10 on AI2D to L31 on ChartQA, with a median of L18 across 14 targets.Relaxing tolerance moves the median to L15.5 at both 2.5% and 5%; ChartQA and DocVQA still require late visual access at every tolerance.Source: §D.1.1 · p. 24

D.1.2. Qwen2.5-VL-7B

• For Qwen2.5-VL-7B, the horizon median is 21 at 1%, 20.5 at 2.5%, and 19 at 5% across satisfied targets.At 1%, horizons range from L16 to L27; RefCOCO, RefCOCO+, and RefCOCOg remain unchanged across tolerances.Source: §D.1.2 · p. 24

E.1. VarestBench and Measurement Protocol

• VarestBench contains 180 short-answer VQA cases across five datasets, with separate sweeps varying prompt length and reference-answer length.Cases run independently at batch size 1 on one H100 in fp16; FLOPs and memory are measured in separate passes.Source: §E.1 · p. 25

E.2. Computational Complexity

• The theoretical count covers image tokens in LLM prefill, whereas measured FLOPs count inference operations across the vision encoder, compression modules, prefill, and decoding.For Varest, the theoretical layer count is K_m before [REL], K_p after it, and zero after adaptive halting; layer-averaged budgets make its theoretical cost close to baselines at matched budgets.Source: §E.2 · p. 25
• Measured FLOPs are 2–9× theoretical; Varest is within 1.3% of HiDrop on both LLaVA-1.5 models and 3–8% below it on LLaVA-NeXT-8B.Fixed costs limit savings at small budgets: from K = 224 to K = 32 on LLaVA-1.5-7B, theoretical FLOPs fall about 7× but measured FLOPs only 2.4×; adaptive halting reduces measured FLOPs by 1–7%.Source: §E.2 · p. 25

F. Further Ablation Studies

• Varest slightly exceeds full-token VRAM usage because its localizers add model weights.The section introduces further ablations of localization, selection, and decoder-stage choices; Tables 16–17 describe selection-policy and spatial-localizer interventions.Source: §E.2 · p. 25, F. Further Ablation Studies · p. 25, Table 16 · p. 26

F.1. Spatial Information in the Localizer Fields

• Shuffled textness, shuffled boundaries, and CLIP-based boundary replacement reduce performance, showing that trained models benefit from image-specific spatial guidance.Shuffled textness is more harmful than zero textness on every backbone; on NeXT, the joint model loses 0.87 points with shuffled textness versus 0.02 with zero textness. Boundary guidance matters more under aggressive compression, and the comparison with −[REL] does not isolate [REL] alone.Source: §F.1 · pp. 26–27

F.2. Budget-Dependent Text Threshold

• Budget-dependent text thresholds improve mean retention by 1.13–2.43 points at every budget, raising the four-budget average from 94.45% to 96.24%.Across budgets, RefCOCO, RefCOCO+, and RefCOCOg improve by 2.42, 2.49, and 2.21 native score points, while TextVQA declines by 0.50 points, illustrating the allocation tradeoff.Source: §F.2 · p. 27, Table 18 · p. 27

F.3.1. Selection Policy at a Fixed Depth

• Random selection particularly harms reading tasks, while oracle selection reveals larger task-specific opportunities than its aggregate gains suggest.On NeXT, random-selection losses average 5.17 points on TextVQA, 16.59 on DocVQA, and 16.65 on ChartQA; oracle selection improves DocVQA by 2.40 points on average, but gold-answer attention does not guarantee a better retained set.Source: §F.3.1 · p. 27

F.3.2. Selection Depth at a Matched Budget

• Earlier pruning reduces mean performance under both retention ratios, with larger losses on grounding; oracle selection recovers 3.89 of the 5.13-point grounding gap at ρ = 1/4.The comparison uses a separately trained eight-layer pruning variant against midpoint Varest at matched layer-averaged budgets.Source: Table 19 · p. 27, §F.3.2 · p. 28

F.4.1. Sensitivity to Prompt Changes; F.4.2. Capacity for Prompt-Conditioned Selection

• Varest achieves 67.44% RefCOCO and 62.56% RefCOCOg grounding accuracy despite near-zero prompt-selection swap gains; supervised readout training nevertheless produces swap gains of +0.395 and +0.394.The supervised readout tests establish prompt-conditioned ranking capacity, not downstream answering improvement.Source: §F.4.1 · p. 28, §F.4.2 · p. 28

F.4.3. Effect of Explicit Prompt Supervision

• Prompt-supervised selector refinement changes aggregate native scores by +0.00 on 7B and +0.02 on 13B, with gains at some budgets and losses at others.Across the refinement runs, increasing prompt sensitivity does not yield a consistent performance improvement.Source: §F.4.3 · pp. 28–29, Table 22 · p. 29

F.4.4. Availability and Value Across Decoder Depth

• On RefCOCO, prompt-selection sensitivity rises as answer-informed benefit falls: on 7B, A increases from 0.098 at ℓ = 8 to 0.407 at ℓ = 16 while V falls from 0.0633 to 0.0031.On 13B, A reaches 0.515 at ℓ = 18 when V is 0.0006; prompt-attention selection reduces answer NLL by 0.0161 and 0.0253 at ℓ = 11 and 14 on 7B, but only 0.0003 at ℓ = 16.Source: §F.4.4 · pp. 29–30, Figure 20 · p. 29

F.4.5. Remaining Dependence on Specific Image Regions

• Reading tasks retain regional dependence later than RefCOCO: at ℓ = 20 on 7B, deletion contrast S is 0.4260 for TextVQA and 0.1717 for DocVQA.At ℓ = 16, DocVQA’s answer-informed advantage is 0.0491 versus 0.0031 on RefCOCO; reducing TextVQA’s retained count from 73 to 18 raises V(16) from 0.0174 to 0.3107. These sweeps hold stage counts fixed rather than matching layer-averaged computation.Source: Figure 20 · p. 29, §F.4.5 · p. 30