Beyond Selection: Token Parameterization for Extreme Visual Token Compression

1 Introduction

• At dozens of tokens, naive reduction can degrade grounding and compositional understanding, while selection can drop rare evidence and learned resampling adds cost and training complexity.Selection and resampling represent different trade-offs: selection is brittle at extreme compression, whereas learned resampling can require extra attention computation, parameters, and staged training.Source: §1 · p. 1
• At matched budgets, Braco leads aggregate Acc.–cost trade-off at 25/16/9 tokens and matches a competitive prior at 36×–64× compression while running ∼36% faster.At 16 tokens, its compressor has 16.6× lower latency and 78.8× fewer FLOPs than that prior compressor.Source: Figure 1 · p. 2, §1 · p. 3
• The paper treats extreme compression as a parameterization problem, separating task-relevant information in the retained subspace from coordinate choices affecting optimization and cross-modal alignment.It formalizes these dimensions as compressibility and learnability diagnostics.Source: §1 · p. 2
• Braco combines transform-basis truncation, input-independent basis-coordinate embeddings, subspace-preserving re-parameterization, and learned spatial residual tokens to recover localized details.The residual tokens are produced by lightweight sparse pooling.Source: §1 · pp. 2–3
• 95.2% Acc. at 23× and 93.2–94.0% Acc. at 36×–64× compression compare with the 576-token Vanilla upper bound, with prefill FLOPs reduced to 1.15–1.37T.On larger inputs, Braco retains 90.8% Acc. at 182× versus 68.9% for a comparable prior.Source: §1 · p. 3

2 Related Work

• Patch pruning and token merging are cheap and effective at moderate compression but can lose task-critical evidence at very small budgets; learned importance signals can also be unstable.Learned interfaces can achieve strong accuracy but add attention-style computation and alignment complexity.Source: §2 · p. 3
• Learned resamplers and query modules form compact latent tokens, but their attention computation and alignment complexity add costs to the compression interface.Braco instead emphasizes a fixed, lightweight visual interface whose cost can be measured independently of downstream prompting.Source: §2 · p. 3
• Transform-domain methods truncate high-frequency coefficients and reconstruct coarser grids, while Braco treats the retained representation as an interface the LLM must align to.This motivates evaluating both compressibility and coordinate organization, alongside stable basis-coordinate identity and spatial residuals.Source: §2 · p. 3

3.1 Token parameterization

• Braco maps an N × N visual-token grid through an orthonormal basis, retains structured coordinates, and can reorganize those coordinates without changing the retained subspace.The basis and retained index set determine the subspace; the optional orthogonal matrix changes only its coordinates.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Token compression objectives

• Compressibility measures retained task-relevant information, while learnability evaluates how coordinate choices within a fixed subspace affect downstream optimization.The compressibility objective combines normalized energy retention with a task-direction readability score; learnability includes statistical-conditioning and geometric penalties.Source: §3.2 · pp. 4–5
• Coefficient coordinates are favored when statistical conditioning dominates at very small backbone budgets, while coarse-grid coordinates become preferable when geometric compatibility dominates.This comparison keeps the retained subspace fixed and makes the coordinate preference budget-dependent.Source: §3.2 · p. 5

3.3 Specializing the functionals to our design space

• Braco retains a C × C low-frequency DCT block because DCT offers the most favorable empirical combination of energy concentration, task readability, and implementation simplicity in its diagnostics.It adds input-independent embeddings on retained transform coordinates to restore stable indexing cues after the basis change.Source: §3.3 · p. 5
• The spatial residual branch addresses localized evidence that low-pass truncation can drop, using a small residual budget pooled from the original spatial grid.The paper motivates this branch because sparse high-frequency spatial details can be diffuse in an incoherent transform basis.Source: §3.3 · pp. 5–6

3.4 The overall framework

• The final interface concatenates a structured DCT backbone and learned residual tokens, with total budget K = Kb + Kr, then normalizes and projects the tokens into the LLM hidden size.A lightweight TokenLearner-style scorer produces sparsemax weight maps over the original grid for the residual tokens.Source: §3.4 · p. 6

4.3 End-to-end comparison

• Braco leads the aggregate accuracy–cost frontier at 25, 16, and 9 tokens, and stays within 0.2 Acc. of QueCC at 4 tokens with lower cost.Against 576-token Vanilla, it retains 91.2–95.2 Vanilla-normalized Acc. across 4–25 tokens while reducing full-pipeline prefill FLOPs from 8.67T to 1.09–1.37T.Source: Table 1 · p. 7, §4.3 · pp. 8–9

4.1 Compressibility under basis choice and structured truncation

• DCT and Haar retain more energy than spatial or random bases under structured truncation, while DCT also scores higher than spatial on CelebA linear probes at K=1 and K=4.At K=32, DCT/Haar retain ≈0.51 energy versus ≈0.04 for spatial/random; under magnitude truncation, the gap largely disappears.Source: §4.1 · pp. 6–7, Figure 4 · p. 8

4.2 Learnability under subspace-preserving coordinate organizations

• With the same low-frequency DCT subspace, vanilla coordinates alone reach the dev cross-entropy target at Kb=4, while idct reaches it 18% and 60% faster at Kb=16 and Kb=64.The comparison holds the retained subspace fixed and measures pretraining time-to-threshold on held-out dev data.Source: Figure 3 · p. 6, §4.2 · pp. 7–8

4.3 End-to-end comparison

• At 16 tokens, Braco matches QueCC’s accuracy with 16.6× lower compressor latency and 78.8× fewer compressor FLOPs.The pre-/post-projector split indicates the comparison is not a boundary artifact.Source: Table 2 · p. 8, §4.3 · p. 9
• At 2880 input tokens, Braco reduces full-pipeline FLOPs from 40.57T to 3.45T and latency from 261.08ms to 44.36ms, while retaining 98.1 Acc.With Qwen2.5-3B, it retains 90.8 Acc. at 3645 input tokens (182× compression).Source: §4.3 · p. 9

4.4 Ablations

• At 9 tokens, hybrid c2s5 reaches 93.2 Acc., above pure-backbone c3s0 at 89.8 and residual-only c0s9 at 92.0.At 16 tokens, basis substitutions cost 2.3–3.0 Acc. points, coordinate substitutions 2.5–3.1 points, and embedding substitutions 0.8–0.9 points.Source: §4.4 · p. 9, Figure 5 · p. 9

• Braco disentangles compressibility from learnability and combines a structured backbone, basis-coordinate embeddings, budget-dependent coordinate organization, and sparse-pooled spatial residuals for a strong accuracy–efficiency trade-off.Source: §5 · p. 9

• The retained-energy quantity measures the squared Frobenius norm of the transformed token grid restricted to the fixed retained coordinates.The notation identifies Xi as a visual-token grid, UB as the token-lattice transform, PS as selection of retained indices, and K as the number retained.Source: A Derivation of the energy-retention… · p. 14
• Taking expectation over token grids makes the trace of the matrix second moment equal to expected retained energy under the fixed index set S.Source: A Derivation of the energy-retention… · p. 14
• Dividing by total expected token energy gives a dimensionless retained-energy score comparable across basis–truncation pairs for the same token-field distribution.Source: A Derivation of the energy-retention… · p. 14
• Orthogonal re-parameterization inside the retained subspace changes its coordinates but leaves the energy-retention score unchanged.Source: A Derivation of the energy-retention… · p. 14

B Task-readability term in the compressibility functional

• Readability measures the fraction of task-direction power retained by a subspace, complementing energy retention, which alone does not establish that task-relevant evidence survives.The compressibility score combines readability with energy retention.Source: B Task-readability term in the… · pp. 14–15

C Statistical conditioning term

• Orthogonal coordinate rotations preserve total token variance while changing token correlations and scale balance; the conditioning surrogate penalizes correlations and deviations from uniform per-token variance.The statistical term is normalized by K^2 in the budget-aware learnability objective, while β controls its trade-off with geometric compatibility.Source: C Statistical conditioning term · p. 15, D Final learnability objective · p. 15

D Final learnability objective

• The final learnability objective normalizes the statistical-conditioning term by K^2 to keep it comparable across budgets, while β tunes its weight against geometric compatibility.The geometric term remains unscaled because it is already normalized by construction.Source: D Final learnability objective · p. 15

E Choosing S: the C × C low-pass block

• The deployable low-pass rule retains a C × C block of transform-lattice coordinates, giving a backbone token count Kb = C^2.The selected lattice coordinates are mapped to a length-Kb index set using a fixed flattening order.Source: E Choosing S: the C × C low-pass block · p. 15

F Baselines and qualitative implications

• Under the same structured low-pass rule, DCT approximately diagonalizes token fields with dominant short-range correlations, concentrating energy at low frequencies and increasing energy retention and readability.The fixed low-pass block also limits the Haar basis because it does not implement the tree-structured selections typically favored by Haar.Source: F Baselines and qualitative implications · p. 16

G Basis-coordinate embedding details

• Structured truncation removes explicit spatial coordinates, so Braco adds an input-independent embedding on the retained transform lattice before serialization.The embedding supplies a stable coordinate code for retained transform tokens.Source: G Basis-coordinate embedding details · p. 16

H Coordinate organization candidates

• The two-candidate surrogate favors vanilla coordinates at very small backbones and idct once geometric compatibility outweighs the statistical-conditioning gap; this threshold is a design heuristic, not a theorem over all rotations.The main experiments use vanilla at Kb ∈ {1, 4, 9} and idct at Kb = 16; calibration comparisons also vary by encoder and dataset.Source: H Coordinate organization candidates · p. 17

I.1 Residual-domain motivation

• Low-pass DCT truncation efficiently captures global structure but drops localized details, which can dominate task error even when the retained subspace has high average compressibility.For sparse spatial residuals, the best m DCT coefficients retain only a small fraction of residual energy when mµ^2s ≪ L, motivating a spatial branch.Source: §I.1 · pp. 17–18

I.2 Residual-pooling implementation

• A lightweight scorer produces residual-slot logits, optionally augmented with a normalized gradient-energy bias, and sparsemax yields spatial weights for weighted-sum residual tokens.The sparsemax temperature is annealed during training to encourage sharper spatial selection; token scales are then harmonized before projection to the LLM hidden size.Source: §I.2 · p. 18

J.1 Compression diagnostics and operating regime

• The diagnostic maps show that a structured low-frequency DCT block covers the concentrated energy region, whereas the same structured block in spatial or random coordinates misses most evidence.The appendix protocol measures retained energy on frozen visual-token grids and uses CelebA linear probes as a separate readability diagnostic; end-to-end VLM benchmarks provide the multimodal task evidence.Source: §J.1 · pp. 18–19
• The reported extreme-budget regime uses K ∈ {25, 16, 9, 4}, corresponding to 23×–144× compression from the 576-token visual interface; K=4 is the smallest comparable operating point.K=1 and K=2 are omitted because key baselines do not cleanly support comparison at those budgets.Source: §J.1 · pp. 19–20

J.2 Main Braco configurations

• As the token budget increases, Braco expands the low-frequency backbone while keeping a small residual budget; only the 25-token setting uses idct organization.The configuration table records the exact Braco instantiations used for the main results.Source: §J.2 · p. 20, Table 4 · p. 20

J.3 Compression-module cost

• Residual pooling dominates Braco’s stage-level compressor cost, while transform and basis-coordinate embedding costs are small and similar across representative allocations.The residual-pooling stage remains at the GFLOP scale; standalone compressor overhead changes only mildly from 4 to 25 retained tokens.Source: §J.3 · pp. 20–21, Table 5 · p. 20, Table 7 · p. 21
• Adding five residual tokens to the same C=2 backbone raises accuracy from 85.7 to 93.2, while adding that backbone to five residual-only tokens raises accuracy from 88.2 to 93.2 at essentially unchanged module FLOPs and latency.Residual-only variants are weaker and not cheaper overall than the hybrid backbone–residual allocation.Source: §J.3 · p. 20, Table 6 · p. 20

J.4 Larger-input generalization and training time

• At 36× compression on 2880-token Vicuna-7B inputs, Braco reaches 98.1 normalized Acc. versus QueCC’s 97.7, with 3.45T versus 3.99T FLOPs and 44.36ms versus 66.73ms latency.The uncompressed full-pipeline cost rises from 8.67T FLOPs and 67.25ms at 576 tokens to 40.57T FLOPs and 261.08ms at 2880 tokens.Source: §J.4 · p. 21
• At 46×/64× compression on Qwen2.5-3B, Braco retains 93.1/92.6 Acc. and remains faster than the corresponding uncompressed baselines.QueCC reaches 77.2/74.0 Acc. at 729/1024 input tokens and 68.9 Acc. at 3645 tokens; its latency exceeds the uncompressed baseline at 729 and 1024 tokens.Source: §J.4 · pp. 21–22
• At 182× compression on Qwen2.5-3B, Braco retains 90.8 Acc. while reducing full-pipeline cost from 25.91T FLOPs and 240.66ms to 3.50T FLOPs and 88.64ms.The larger-input pipeline slices high-resolution images into multiple visual views before compression.Source: §J.4 · pp. 21–22
• Braco’s representative training runs are no slower than QueCC and are often faster under the shared two-stage recipe.The corresponding run budgets are 55.2 versus 61.6 A100 GPU-hours at 576 tokens and 68 versus 74.4 A100 GPU-hours at 2880 tokens for Braco and QueCC, respectively.Source: §J.4 · p. 22, Table 9 · p. 22, §J.5 · p. 23

J.5 Hardware scaling and measurement protocol

• On A100 and A800, Braco has lower latency than Vanilla as batch size grows, while Vanilla runs out of memory at batch size 128 and above.The batch-scaling curves report mean latency and standard-deviation bands; their protocol supports within-curve scaling comparisons.Source: Figure 6 · p. 23, Table 10 · p. 23, §J.5 · p. 24

J.6 Functional rankings and cross-encoder calibration

• The compressibility score C has mean within-budget Spearman correlation ρ = 0.93 with held-out MLP accuracy, versus 0.67 for E and 0.80 for R, and selects the best basis at all six budgets.Its basis ranking agrees with end-to-end ordering in the fully retrained 16-token ablation and the 9-token comparison.Source: §J.6 · p. 24
• The learnability score L_learn has mean within-budget Spearman correlation ρ = 0.88 with held-out dev-loss AUC; iDCT reaches the target 18% and 60% faster at 16 and 64 tokens.The criterion’s coefficient-versus-iDCT choice matches the final-accuracy winner at 4, 9, and 16 tokens, while fixed random-rotation controls do not.Source: §J.6 · p. 24
• Across three encoders, calibration thresholds are identical across LLaVA-Pretrain resamples and shift by at most one scanned square budget under DocVQA distribution shift.The procedure uses unlabeled images and supports adapting the crossover estimate to the visual-interface training distribution or a changed deployment domain.Source: §J.6 · p. 24, Table 11 · p. 24

J.7 Robustness to training seeds

• The repeated-training comparison uses three matched seeds per available method and budget, sharing the base checkpoint, data order, and training schedule within each seed.Table 12 reports sample mean ± sample standard deviation; QueCC lacks a 25-token setting and TokenPacker is absent from the four-token comparison.Source: §J.7 · p. 24, Table 12 · p. 25
• Across three matched runs, Braco leads QueCC by 0.24/0.27 Acc. points at 16/9 tokens and TokenPacker by 0.94/0.42/1.81 points at 25/16/9 tokens.At four tokens, the methods are effectively tied, with a difference of −0.07; Braco retains the latency advantage reported in Table 1.Source: Table 12 · p. 25, §J.7 · p. 25

K Limitations

• Braco is evaluated primarily on LLaVA-family vision-encoder-to-LLM pipelines and standard single-image multimodal benchmarks; new backbones, video, localization, or dynamic-token settings may require a calibration sweep.The remaining accuracy comparisons beyond key Braco–QueCC and Braco–TokenPacker settings use one trained checkpoint per setting.Source: K Limitations · p. 25

L Broader Impacts

• The paper notes that greater efficiency may lower multimodal energy costs but could also facilitate surveillance or automated content understanding, and recommends responsible-deployment practices.It introduces no new data sources or user-facing interaction mechanisms; the primary risks mirror those of underlying VLMs.Source: L Broader Impacts · p. 25

M Existing Assets and Licenses

• Third-party models, data, benchmarks, and baseline implementations are used under their original licenses or terms, and the paper does not redistribute weights or dataset files.The cited assets include LLaVA, Vicuna, Qwen2.5-3B, and benchmark releases.Source: M Existing Assets and Licenses · pp. 25–26