Source-linked AI summary
Query Independent Variable Rate Visual Token Coding
Hongbo Zhang, Zihao Yang, Liuyang Song, Daqian Yang, Haoyang Yao, Yan Wen, Zhengtao Yao
TL;DR
Visual-token compression usually selects tokens using criteria that can depend on the question, a mismatch when one representation must serve repeated or different queries. RDTok keeps every token and allocates bits across measured distortion–rate curves without text input. It improves on query-independent baselines, matches attention-ranked pruning on its tuned question, and in a shown transfer case answers a different question better.
Problem
Most visual-token compression selects tokens, often using query-dependent criteria, although representations may be cached, split across devices, or queried repeatedly.
Source: §1 · p. 1Method
RDTok keeps every token and uses exact integer rate–distortion optimisation to allocate a fixed bit budget across measured token curves, without text input.
Source: §1 · p. 1, §2.1 · p. 1, §2.2 · p. 3Results
Across two datasets and capacities, RDTok improves over query-independent baselines; on COCO captions, output KL falls 37–47% versus uniform rate at 4B.
Source: §1 · p. 1, §3.2 · pp. 3–4Takeaways & Limitations
RDTok matches attention-ranked pruning on the question it was tuned for and, in a shown case, answers a different question better when the representation is reused.
Source: §3.2 · p. 4Feature-MSE allocation does not directly capture output KL, while output-KL weighting depends on the query and does not transfer to the next question.
Source: §2.3 · p. 3
Abstract
from arXiv · showhide
Visual-token compression for vision--language models is posed almost entirely as a selection problem: decide which tokens to keep and discard the rest. The criteria that work best rank tokens by the attention the language model pays them, which makes the ranking a function of the question being asked. That is invisible in a single-turn benchmark and decisive whenever a compressed representation is written once and read many times, as when it is cached across the turns of a conversation or transmitted between a device and a server. We take the other half of the classical transform-coding toolkit instead: keep every token and vary its rate. A transform code exposes each token's measured distortion--rate curve, and a fixed bit budget is distributed across tokens by exact integer rate--distortion optimisation on those curves. No text enters the pipeline, so one compressed representation serves any query. At equal bit budgets, on two datasets and two capacities, it preserves the model's output distribution and its answers better than uniform-rate coding, the closed-form water-fill and distortion-ranked pruning. It matches attention-ranked pruning on the question pruning was tuned for, and overtakes it once the compressed image must answer a different question about the same image.
1. INTRODUCTION
Visual-token compression is usually framed as deciding which tokens to discard, often using query-dependent criteria that may not suit representations reused across questions. This work instead keeps every token and allocates a fixed bit budget across measured distortion–rate curves, enabling query-independent coding.
1. INTRODUCTION
- Prior compression work mostly prunes or merges tokens, and even budgeted approaches do not vary bits spent on surviving tokens.The paper’s alternative keeps every token and varies its rate.Source: §1 · p. 1
- Representations reused across conversation turns, device-server splits, or many corpus queries cannot rely on rankings computed for a single question.The encoder does not know the question, so a ranking from one question may be wrong for the next.Source: §1 · p. 1
- RDTok allocates a fixed bit budget exactly across measured token distortion–rate curves, without tuned parameters or text input.The paper also evaluates a transfer protocol using two independent human questions for each ChartQA test image.Source: §1 · p. 1
- The authors report gains over uniform-rate coding and pruning at equal bit budgets across two datasets and two model capacities.Their transfer protocol measures the cost of conditioning an allocation on a query when the representation must serve another question.Source: §1 · p. 1
2. METHOD
RDTok transforms visual tokens into measured distortion–rate curves, then allocates a fixed bit budget across them exactly. Its allocation uses image-side curves rather than a query-dependent score.
2.1. Transform and quantiser
- A PCA transform and coefficient quantisers produce each token’s measured distortion–rate curve on a grid of 64-bit increments.The basis is fit on images disjoint from evaluation images; the grid has 16 increments, with rate zero as the dropped-token analogue.Source: §2.1 · pp. 1–2
2.2. Allocation across tokens
- Exact dynamic programming allocates a fixed total rate across tokens to minimise distortion on their measured curves, without using a per-token score.The comparison methods use the same total budget, making the comparison equal-rate by construction.Source: §2.2 · p. 2
- The closed-form allocation assumes a parametric decay that measured curves often violate, and it is not scale invariant.Only 59.6% of marginal-gain differences are non-increasing, and the closed form requires a normalised importance score.Source: §2.2 · p. 3
- Pruning is the two-rate restriction of the allocation problem; among its variants, attention scoring is the only one that reads the query.The attention score sums the mass text positions send to each token; the method also stores each token’s rate index.Source: §2.2 · p. 3
2.3. Why the objective stays on the source side
- Weighting source distortion by output KL measures a property of the question, so it does not transfer to another question about the same image.The passage states that attention scoring has the same query-transfer defect.Source: §2.3 · p. 3
3. EXPERIMENTS
The evaluation compares variable-rate allocation with uniform-rate coding and pruning under equal bit budgets, then tests whether query-independent representations transfer across questions. It covers two datasets, two model capacities, and an analysis of allocation behavior and distortion.
3.1. Protocol
- The protocol tests 4B and 8B Qwen3-VL-Instruct models on ChartQA and COCO captions, with 256, 384, or 512 bits per token and equal total bits across arms.These rates correspond to 160×, 106×, and 80× compression of fp16 features; evaluation uses 288 images per cell.Source: §3 · p. 3, §3.1 · p. 3
3.2. Main results
- RDTok lowers output KL against every query-independent baseline in all ChartQA cells, leading on the other reported quantities in all but a few COCO cells.The comparison uses equal bit budgets; the table covers ChartQA at both capacities.Source: Table 1 · p. 3, §3.2 · p. 3
- The closed-form water-fill also varies rates but tracks uniform rate because its assumed parametric curves collapse the allocation onto three levels.The comparison indicates that measured distortion–rate curves, rather than rate variation alone, are the key design choice.Source: §3.2 · p. 3
- Attention-ranked pruning matches RDTok on task accuracy, while RDTok leads on output KL and preservation; in the illustrated case, only RDTok answers correctly.At 160×, RDTok spreads the budget over sixteen levels, while pruning discards three quarters of the tokens.Source: Figure 3 · p. 4, §3.2 · p. 4
3.3. Query transfer
- On held-out questions, RDTok leads attention-ranked pruning at every rate and capacity; pruning loses roughly half its fidelity at smaller capacity and more than doubles divergence at larger capacity.The transfer test uses two independent human questions per ChartQA image; the larger model’s attention is more query-specific, increasing the value of not conditioning allocation on the question.Source: Figure 4 · p. 4, §3.3 · p. 4
3.2. Main results
- On COCO captions, RDTok lowers output KL against all three query-independent baselines at both capacities, with every comparison reaching p < 10^-3.Against uniform rate, the reductions are 37–47% at 4B and 43–45% at 8B; against closed-form water-fill, 34–43% and 40–43%; against distortion-ranked pruning, 15–27% and 9–37%.Source: §3.2 · pp. 3–4
3.4. Analysis
- Perturbations with identical total squared error produce output KL differences of 2.9–14×, showing that feature MSE is not a sufficient statistic for output KL.Concentrating error on a quarter of tokens is several times worse than spreading it; the worst perturbations affect high-norm tokens or leading PCA directions.Source: §3.4 · p. 4