PaperlayerView exact revision on arXiv

Source-linked AI summary

HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantization

Nenad Banfic

arXiv:2609.35800v1cs.LGcs.CV

TL;DR

Low-bit KV-cache quantization saves storage but can sharply affect VLM accuracy. HeadGuard augments a base quantizer with an offline mask protecting selected image KV heads in BF16. Across models and benchmarks, it recovers accuracy on weaker quantizers, with keys-only protection offering a lower modeled storage cost.

  • Problem

    Low-bit KV-cache quantization reduces storage, but accuracy effects vary sharply across VLMs, motivating a comparison of head protection with changing the base quantizer or increasing precision.

    Source: §1 · p. 1

  • Method

    HeadGuard ranks physical KV heads offline using image- and output-sensitivity scores, then keeps selected image keys and optionally values in BF16 while the base quantizes unprotected entries.

    Source: §1 · p. 1, §3 · pp. 2–3

  • Results

    At a one-eighth head budget, K+V HeadGuard raises six-task, eight-model mean accuracy over uniform by 14.11 versus 12.65 percentage points at 2 bits for POPE versus MMBench calibration, respectively.

    Source: §5.6 · p. 7

  • Takeaways & Limitations

    Keys-only protection retains substantial recovery at lower modeled storage cost, and HeadGuard can improve generated answers and caption fidelity to BF16 outputs.

    Source: §6 · p. 7

    Low-bit quantization is simulated rather than stored in packed caches, so end-to-end memory use and speed are not measured; long-form and multi-turn generation are also unevaluated.

    Source: §4 · p. 4, §6 · p. 8

Abstract

from arXiv · show

Low-bit key-value (KV) cache quantization saves storage but can sharply degrade vision-language model (VLM) accuracy. We introduce HeadGuard, a composable head-protection method that augments a base KV-cache quantizer with a fixed high-precision mask. Image-sensitivity and output-sensitivity scores select physical KV heads offline, with approximately 1/8 protected in the main experiments; their image keys and optionally values remain in bfloat16 (BF16), while the base quantizes unprotected image entries. Across eight VLMs, three base quantizers, and eight benchmarks (six discriminative and two generative), HeadGuard recovers a substantial fraction of lost accuracy on weaker quantizers, with the strongest gains for Qwen and InternVL. At 2 bits, the six-task discriminative mean over eight models rises from 0.436 to 0.580 on the weakest base; protection can also improve generated answers and caption fidelity to BF16 outputs. Mean accuracy gains persist across all three quantizers with both tested calibration datasets. Keys-only protection retains substantial recovery at lower modeled storage cost. Evaluated through simulated quantization, HeadGuard offers a composable way to improve low-bit VLM accuracy without replacing the underlying quantizer.

1 INTRODUCTION

Low-bit KV-cache quantization can reduce storage while producing sharply different accuracy effects across VLMs. HeadGuard asks when protecting image KV heads helps and introduces an offline, reusable mask to augment a base quantizer.

1 INTRODUCTION

  • Low-bit quantization reduces KV-cache storage, but per-token 2-bit min–max gives POPE accuracy of 0.832 for LLaVA-NeXT and 0.496 for Qwen2.5-VL.Image tokens often dominate a VLM’s KV cache.Source: §1 · p. 1
  • The paper asks when protecting image KV heads helps compared with changing the base quantizer or increasing precision, while keeping model weights in BF16.The BF16 reference also keeps the full KV cache in BF16.Source: §1 · p. 1
  • HeadGuard ranks heads offline by image- and output-sensitivity, then keeps selected image keys and optionally values in BF16 while the base quantizes the rest.The fixed mask is reused across tasks without weight updates or inference-time gradients.Source: §1 · p. 1

2 RELATED WORK

Related approaches allocate precision, retain tokens, or reduce visual tokens, while HeadGuard focuses on image KV precision in VLMs. The paper also distinguishes its shared-harness base implementations and simplified controls from the complete published methods.

2 RELATED WORK

  • KIVI quantizes keys per channel and values per token [Liu et al., 2024d], while KVQuant handles outliers [Hooper et al., 2024] and AKVQ-VL uses token saliency and Hadamard transforms [Su et al., 2025].The paper’s KIVI-style and AKVQ-style bases implement selected arithmetic components, not the complete algorithms or serving kernels.Source: §2 · p. 2
  • KVmix and RateQuant allocate precision using sensitivity signals for text LLMs, whereas HeadGuard selects image KV heads in VLMs [Li et al., 2026; Zuo et al., 2026].Unlike RateQuant, HeadGuard does not fit quantizer-specific distortion curves; it preserves selected image keys and optionally values in BF16.Source: §2 · p. 2
  • HeadKV, DuoAttention, and SparseMM allocate per-head token budgets, while HeadGuard’s attention-mass and visual-response controls are simplified proxies for comparing ranking signals [Fu et al., 2025b; Xiao et al., 2025].The controls use the same calibration data and BF16 protection budget.Source: §2 · p. 2
  • DART removes visual tokens judged redundant, whereas HeadGuard keeps all tokens and changes only their KV precision [Wen et al., 2025].DART’s experiments report that attention-score token selection can perform worse than random selection.Source: §2 · p. 2

3 HEADGUARD

HeadGuard scores physical KV heads offline and protects a fixed subset from low-bit quantization. Its image- and output-sensitivity signals are used to rank heads, not to update model weights or quantify actual quantization error.

3 HEADGUARD

  • HeadGuard takes a frozen VLM, labeled calibration set, head budget, and base quantizer, then scores heads offline and bypasses quantization for selected image-token KV entries.The selected set remains fixed.Source: §3 · p. 2
  • The image gate scales image-position attention weights without renormalization before the value matrix, while the output gate scales the whole head output before the shared projection.With the output gate at 1, the image gate changes only the image contribution; with the image gate at 1, the output gate scales the whole head output.Source: §3 · p. 2
  • Image sensitivity is the mean absolute gradient of correct-answer log-probability to the image gate, while output sensitivity is the corresponding prompt-text loss gradient to the output gate.The image tokens remain inputs but are excluded as prediction targets for the prompt-text loss.Source: §3 · p. 2
  • Both scores are local first-order ranking signals computed with one forward and two backward passes per calibration example, without updating gates or model weights.They are not measures of faithful visual reasoning or actual quantization error.Source: §3 · p. 3
  • For physical KV heads, HeadGuard averages scores across shared query heads, standardizes each score across heads, sums them, and protects the top K.The main K+V and keys-only experiments use K = round(NKV/8), while the capped sweep varies the protected fraction.Source: §3 · p. 3
  • K+V protection keeps selected image keys and values in BF16; keys-only protection leaves selected image values quantized, while text KV remains in BF16.The base quantizer must allow quantization to be skipped for individual KV heads.Source: §3 · p. 3

4 EVALUATION SCOPE AND PROTOCOL

The evaluation spans eight VLMs, six discriminative tasks, and two generation tasks, with fixed calibration and multiple quantization and protection protocols. Low-bit quantization is simulated, so the experiments assess accuracy rather than packed-cache memory savings or speed.

4 EVALUATION SCOPE AND PROTOCOL

  • The study evaluates eight listed VLMs on six discriminative tasks plus TextVQA and COCO captioning, usually calibrating each model and protocol on 100 labeled POPE examples.The calibration comparison also uses 100 labeled MMBench examples.Source: §4 · p. 3
  • Main K+V and keys-only protocols use a fixed 1/8 head budget; the supporting budget/precision ablation varies bit-width, protected fraction, and calibration dataset.The main generative evaluations use 1,000 TextVQA and 200 captioning examples, while capped sweeps use 200 examples for each task.Source: §4 · p. 3
  • The bases are per-token min–max, KIVI-style, and AKVQ-style; AKVQ-style also preserves a nominal 2% of image tokens per KV head by key norm.The large sweeps simulate quantization by rounding and reconstructing floating-point values rather than storing a packed low-bit cache.Source: §4 · pp. 3–4
  • Head-selection comparisons include image-only and output-only sensitivity, activation magnitude, image-attention mass, visual-response norm, and random selection.The selectors protect the same number of physical KV heads.Source: §4 · p. 4
  • Discriminative scores use candidate-answer logits, TextVQA uses normalized answer matching, and caption ROUGE-L is measured against BF16 outputs rather than human captions.The generative scores are not averaged with classification accuracy; calibration and evaluation examples are disjoint.Source: §4 · p. 4
  • Image sizes and tile counts are bounded by model-specific preprocessing, and Gemma’s long image inputs are not tested.Gemma’s image-cropping mode is disabled, and its image length is capped.Source: §4 · p. 4

5 WHEN DOES HEAD PROTECTION HELP?

Head protection recovers low-bit accuracy unevenly across models and quantizers, with stronger gains on weaker bases. Keys-only protection retains much of that recovery at lower modeled storage cost, and protection can also improve generated answers and captions.

5.1 RECOVERY DEPENDS ON THE MODEL AND PRECISION

  • At 2 bits, the six-task mean rises from 0.436 to 0.580 with K+V HeadGuard, but neither Gemma model recovers from min–max degradation.Qwen2.5-VL rises from 0.360 to 0.747 and InternVL3 from 0.362 to 0.688; at 3 bits, the overall mean rises from 0.564 to 0.724.Source: §5.1 · p. 4, Table 1 · p. 5

5.2 BASE QUALITY MATTERS MORE THAN RECOVERY ALONE

  • At 2 bits, HeadGuard gains 14.37 points on min–max, 12.73 on AKVQ-style, and 0.85 on KIVI-style; a repaired weak base can still trail a stronger one.Protected min–max trails unprotected KIVI-style by 10.67 points, so head protection is a targeted repair rather than a substitute for choosing a strong base.Source: Table 2 · p. 5, §5.2 · p. 5

5.3 THE IMAGE-SENSITIVITY INCREMENT

  • At 2-bit min–max, adding image sensitivity raises six-task recovery from output-only’s 13.34 to 14.37 points; at 3 bits, hybrid and output-only recover 15.96 and 15.97 points.On POPE, hybrid selection beats output-only by 1.8–2.9 percentage points for five models, with 95% confidence intervals excluding zero.Source: §5.3 · pp. 5–6

5.4 KEYS-ONLY PROTECTION RETAINS SUBSTANTIAL RECOVERY

  • Keys-only protection yields mean 2-bit gains of 14.56 points on min–max, 13.95 on AKVQ-style, and 0.57 on KIVI-style, similar to K+V mean accuracy.For 2-bit min–max, modeled image-cache storage is 3.11 bits per element for K-only versus 3.97 for K+V, about 22% less.Source: §5.4 · p. 6, Table 3 · p. 6

5.5 PROTECTION CAN ALSO HELP GENERATION

  • With 2-bit min–max, K+V protection raises Qwen2.5-VL’s TextVQA score from 0.015 to 0.687; captioning mean ROUGE-L against BF16 output rises from 0.171 to 0.312.Keys-only protection raises Qwen2.5-VL’s TextVQA score from 0.009 to 0.690 on min–max but lowers it from 0.733 to 0.724 on KIVI-style.Source: §5.5 · p. 6

5.6 ABLATION: BUDGET, PRECISION, AND CALIBRATION

  • Larger protected fractions improve Qwen2-VL and InternVL3, while Gemma benefits more from raising min–max precision from 2 to 3 bits than from expanding the 2-bit budget to 1/4.At the one-eighth budget, K+V min–max gains over uniform are 14.11 versus 12.65 points at 2 bits and 15.79 versus 14.39 at 3 bits, for POPE versus MMBench calibration, respectively.Source: §5.6 · p. 7, Figure 1 · p. 7

6 DISCUSSION AND CONCLUSION

HeadGuard uses a reusable offline mask to protect selected image KV heads, recovering low-bit accuracy particularly strongly for Qwen and InternVL. The authors frame it as a targeted accuracy–storage trade-off and note important evaluation boundaries.

6 DISCUSSION AND CONCLUSION

  • HeadGuard substantially recovers low-bit VLM accuracy, with particularly strong recovery for Qwen and InternVL on weak bases.Its fixed offline mask is reused across tasks and requires neither weight updates nor inference-time gradients.Source: §6 · p. 7
  • Output sensitivity explains most recovery, while image sensitivity's additional benefit varies by model and calibration.The protection mask retains all tokens and is applied without replacing the underlying quantizer.Source: §6 · p. 7
  • Benefits extend to generated answers and caption fidelity to BF16 outputs, while keys-only protection retains substantial recovery at lower modeled storage cost.The authors recommend choosing a strong base quantizer and adding selective protection when its accuracy benefit justifies the modeled storage cost.Source: §6 · p. 7
  • Evaluation uses fixed image-size limits, custom scoring, simplified KIVI-style and AKVQ-style bases, and only 100 examples for each calibration set.The study simulates quantization and estimates storage; end-to-end memory and speed with packed caches, long-form and multi-turn generation, and larger calibration sets remain untested.Source: §6 · p. 8

REPRODUCIBILITY STATEMENT

The reproducibility statement describes released materials for reproducing reported aggregates and clarifies how hardware was controlled within evaluation protocols.

REPRODUCIBILITY STATEMENT

  • The code supplement includes evaluation and scoring code, frozen calibration scores, reference results, aggregation and bootstrap code, pinned dependencies, and unit tests for custom metrics.The analysis code reproduces reported aggregates from bundled reference results.Source: REPRODUCIBILITY STATEMENT · p. 8
  • Sections 3 and 4 specify the method, data splits, scoring, and averaging conventions, and data shuffling uses a fixed seed.The implementation details also include experiment-specific runners and figure-rendering code.Source: REPRODUCIBILITY STATEMENT · p. 8
  • Within each protocol, all models and measurements used the same GPU, and HeadGuard gains compare the same model and quantization method with and without protection.Absolute accuracy comparisons across protocols describe trends rather than quantify HeadGuard's gains.Source: REPRODUCIBILITY STATEMENT · p. 8

ETHICS STATEMENT

The ethics statement notes that the study uses public models and benchmarks without collecting new human data, and cautions against treating benchmark improvements as evidence of high-stakes safety.

ETHICS STATEMENT

  • The study collects no new human data, and improved benchmark accuracy does not establish safety in high-stakes deployment.The authors also note that quantized VLMs can hallucinate and that model and dataset licenses and privacy limitations still apply.Source: ETHICS STATEMENT · p. 8

AI USE STATEMENT

The appendix identifies the scope of its K+V and keys-only evaluation tables, alongside an AI-use disclosure.

AI USE STATEMENT

  • The authors disclose using generative AI assistants for code, figures, manuscript drafting and editing, and reference formatting, while stating they were not used to generate data or design the methodology.Authors reviewed and validated the code, results, figures, claims, conclusions, and interpretations.Source: AI USE STATEMENT · p. 8

APPENDIX

  • The appendix directs readers to Section 4 for the detailed results' protocols.The passage does not specify those protocols.Source: APPENDIX · p. 11

A KEY-AND-VALUE PROTECTION AT TWO AND THREE BITS

B KEYS-ONLY DISCRIMINATIVE EVALUATION

  • Table 6 reports keys-only protection at 2 bits across all six discriminative tasks, pairing unprotected accuracy with keys-only HeadGuard accuracy.The table gives per-model means on the full eligible discriminative splits, separately from K+V evaluation.Source: B KEYS-ONLY DISCRIMINATIVE EVALUATION · p. 11, Table 6 · p. 12

C POPE UNCERTAINTY IN FULL EVALUATION

POPE serves as both a calibration and evaluation benchmark, with uncertainty estimated on eligible evaluation questions after excluding calibration and diagnostic items. The reported paired image-cluster intervals express HeadGuard gains in percentage points, not absolute accuracy.

C POPE UNCERTAINTY IN FULL EVALUATION

D GENERATIVE RESULTS

The generative evaluations report task scores separately from the six-task accuracy mean, while the capped sweep also compares calibration and head-budget settings. At 2-bit min–max with a one-eighth head budget, HeadGuard’s mean accuracy gain is 14.11 points with POPE calibration and 12.65 with MMBench calibration.

D GENERATIVE RESULTS

  • Generative results report TextVQA and caption scores separately; caption fidelity measures agreement with BF16 output, not human-caption quality.K+V results cover all model–base pairs at both precisions, while K-only results cover TextVQA and caption pairs.Source: D GENERATIVE RESULTS · p. 14, Table 12 · p. 14, Table 13 · p. 15

E CAPPED BUDGET SWEEP

F CAPPED GENERATIVE EVALUATION

G ADDITIONAL CAPPED HEAD BUDGETS

  • Additional capped evaluations vary protected-head fractions across 1/16, 1/12, and 1/4, reporting six-task accuracy, TextVQA, and caption fidelity separately.The nominal fraction is rounded for each model, and each metric’s mean covers all eight models.Source: G ADDITIONAL CAPPED HEAD BUDGETS · p. 22

Related papers

Loading 2609.35800v1…