Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Eviction

Chenyu Zhou; Qiliang Jiang; Shuning Wu; Xu Zhou

Source-linked AI summary

Pixel Decodability Is Not a Compression Signal: Causally Evaluating Importance Proxies for Visual KV-Cache Eviction

Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou

arXiv:2609.13012v1Jul 27, 2026cs.CV

TL;DR

Visual KV caches may retain pixel-decodable content without using it to answer questions. The paper measures retention and causal use with pixel inversion and single-unit KV ablation, finding that retention is not a competitive compression signal in the tested setting.

Problem
The paper asks whether pixel-decodable retention tracks the visual KV units that a VLM causally uses to answer questions.
Method
It measures retention with learned pixel inversion and utilization with single-super-patch KV ablation, relating them within images in a preregistered held-out design.
Results
Retention shows no positive relationship with causal utilization, whereas attention weakly tracks utilization and serves as the design’s positive control.
Takeaways & Limitations
Pixel-decodable retention is an informational axis distinct from the functional axis and is not a competitive KV-compression signal in this setting.
Takeaways & Limitations
The conclusions are scoped to two VLMs, TextVQA, tested granularities, and a primary decoder layer, while single-unit ablation may miss redundantly distributed importance.

Abstract

from arXiv · show

Vision-language models retain a substantial amount of pixel-decodable visual content in their visual key-value cache. We show, in our setting, that this retention is task-inert: across our preregistered tests, how much a unit retains never positively tracks whether the computation that answers the question causally relies on it. We measure retention with a learned pixel-inversion decoder and causal use with single-super-patch KV ablation, the teacher-forced drop in gold-answer log-probability, and relate the two within images under a preregistered, sign-calibrated, held-out design. Retention is decoupled from attention and, in a well-powered null, from causal utilization. Utilization is not inert to every proxy: attention weakly but significantly tracks it, the only signal we find that does and the design's positive control. We characterize pixel-decodable retention as an informational axis of the visual KV cache, orthogonal to the functional one. How much task-inert content a cache holds differs by architecture in our model pair: the encoder-free model retains 2.7 times more than the encoder-based one. The engineering consequence is a controlled negative result. At super-patch granularity, deconfounded pixel-decodable retention ranks KV eviction no better than random; at token granularity it acquires only a weak inverse-importance signal at larger budgets, dominated at every budget by attention magnitude. In our setting, pixel-decodable reconstructability is not a competitive KV-compression signal at any granularity we test.

1 Introduction

The paper separates pixel-decodable retention, attention, and causal utilization in visual KV caches, testing whether common importance proxies track what the model uses. It finds retention is informational rather than functional, while attention provides a weak positive control.

1 Introduction: Retention does not predict causal utilization: partial correlations are −0.009 for Gemma and −0.052 for InternVL, with confidence intervals including zero.The preregistered test was powered at approximately 0.9 for ρ = 0.15, and neither model showed the hypothesized positive relationship.
1 Introduction: Attention weakly predicts utilization at +0.11 to +0.12, confirming that the causal readout detects a genuine functional signal.Because the same estimator detects attention’s signal, the retention null is not attributed to an insensitive instrument.
1 Introduction: The study defines pixel-decodable retention as an informational cache axis orthogonal to the functional axis of attention and causal utilization.Retention measures decodable image content, whereas utilization measures the teacher-forced gold-answer log-probability drop after KV ablation.
1 Introduction: The encoder-free model retains 2.68× more task-inert pixel content than the encoder-based model in the studied pair.The comparison is between Gemma-4-12B and InternVL3.5-8B.
1 Introduction: The contribution is a controlled negative result against treating pixel-level reconstructability as a visual KV-eviction importance signal.The work extends prior privacy-oriented reconstruction analysis by testing whether reconstructed content is causally used.

2 Related Work

The related work distinguishes decodable information from causal use and situates this study between visual KV reconstruction, cache compression, and attention-based pruning. The paper’s novelty is a causal, three-axis evaluation of reconstructability as a compression signal.

2 Related Work: Prior probing work shows that recoverable representation information can be causally inert, motivating a direct distinction between decodability and use.Examples span vision-transformer attributes and language-model structural information.
2 Related Work: Text-only cache studies use related retention, accessibility, and utilization terminology, but differ in tokens, probes, accessibility measures, and utilization readouts.Here retention uses an independent learned pixel-inversion decoder and utilization uses single-super-patch KV ablation with teacher-forced log-probability.
2 Related Work: Several VLM compression methods use reconstructability or value-vector regression to identify redundant cache units, making reconstructability’s importance assumption an open empirical question.The paper tests that assumption causally rather than inferring importance from downstream compression behavior alone.
2 Related Work: Earlier pixel-inversion work measured what visual KV caches reconstruct, whereas this paper tests whether reconstructed content predicts causal utilization.The present study adds single-unit KV ablation, attention, a three-axis analysis, and compression evaluation.
2 Related Work: Attention-based pruning is a stronger proxy than retention in this study but remains weak and can carry spatial biases on peripheral-evidence tasks.The related evaluation literature complements the paper’s finding that attention is useful but not a strong predictor.

3 Method

The method measures retention, attention, and causal utilization per visual super-patch, then relates them within images using preregistered, sign-calibrated, held-out rank partial correlations. Learned pixel inversion supplies the informational measure, while KV ablation supplies the causal measure.

3 Method: The study aggregates each model’s visual tokens into a 4 × 4 = 16 super-patch grid and estimates all quantities within images to remove image-level confounds.Gemma uses variable-resolution grids and InternVL uses a 16 × 16 token grid before aggregation.
3 Method: Retention is the shuffle-adjusted excess reconstruction quality obtained by decoding an image from a super-patch’s visual KV at primary layer ℓ = 0.The image-only prefill keeps the KV tied to encoded visual content rather than question-conditioned state.
3 Method: Utilization is Δtask(u) = log p(gold | full KV) − log p(gold | KV \ u), summed over gold-answer tokens, so positive values indicate causal load-bearing.The intervention masks one super-patch’s key/value positions and measures the teacher-forced log-probability change.
3 Method: Attention is computed as question-token attention mass to each super-patch, averaged across heads and selected full-attention layers.The method averages layers 5, 11, and 17, then sums token mass assigned to each super-patch.
3 Method: The primary estimand is a within-image rank partial correlation between retention and utilization, residualized for attention, pixel-energy covariates, and spatial position.This isolates the association beyond texture, attention, and centrality-related explanations.
3 Method: The sign is fixed on a disjoint 48-image calibration split and tested on 72 validation images, with decoder-noise correction and preregistered power near 0.9 for ρ = 0.15.The held-out design prevents post-hoc sign selection; cumulative eviction provides an aggregate analysis less dependent on per-unit reliability.

4 Results

Across held-out, preregistered analyses, pixel-decodable retention is unrelated to causal utilization and attention, while attention provides the only detectable utilization signal. Retention also fails as a compression signal against random eviction at super-patch granularity and adds no consistent benefit to attention-guided eviction.

4.1 Retention is decoupled from attention (C1): Retention and attention are weakly negatively correlated within images, with ρ values of −0.068 for Gemma and −0.088 for InternVL.Both values are far below the preregistered proxy threshold of 0.5, so decodable pixel content is not an attention proxy.
4.2 Retention is decoupled from causal utilization (C2, well-powered null): Retention does not causally predict utilization: partial ρ is −0.009 for Gemma and −0.052 for InternVL, with both confidence intervals including zero.The design was powered at approximately 0.9 for ρ = 0.15, and the same pipeline detects a positive attention–utilization association.
4.3 Attention weakly but genuinely tracks utilization: Attention weakly but significantly tracks utilization, with partial ρ values of +0.113 for Gemma and +0.120 for InternVL.Pixel-energy controls do not predict utilization, making attention the only measured proxy with a statistically detectable relationship to utilization.
4.4 Retention differs across the architecture pair (C3): The encoder-free Gemma retains 2.68× more decodable pixel information per unit than encoder-based InternVL, but this is a model-pair contrast rather than an architecture-class claim.The validation 95% CI is [2.37, 3.06], and the comparison is confounded by other differences between the two models.
4.5 Compression implication: retention is not a pruning signal: Deconfounded retention is no better than random for super-patch eviction, while attention beats random at every tested budget.Retention–random contrasts include zero at every budget, whereas FastV–random contrasts exclude zero throughout; combined retention and attention shows no consistent gain over FastV.
4.5 Compression implication: retention is not a pruning signal: At token granularity, attention dominates deconfounded retention at every budget, which shows only a weak inverse-importance signal at larger budgets.The token-granularity protocol uses each image’s actual visual-token count and equal deletion counts across strategies.

5 Discussion

The discussion interprets pixel-decodable retention as informational rather than functional and argues that its compression value is limited in the tested setting. It also scopes the conclusions to specific models, task, granularity, and measurement choices.

Interpretation: Retention measures recoverable pixel content, whereas utilization measures whether the model’s answer computation depends causally on the unit.These quantities can diverge because recoverable content need not be routed through the unit during the forward computation.
From probing principle to cache-compression evidence: The retention–utilization dissociation operationalizes decodable-versus-causal analysis as a preregistered, well-powered null with attention as a positive control.This distinguishes specific inertness of pixel decodability from a blanket absence of proxy–utilization relationships.
Implications for KV-cache compression: Deconfounded pixel-decodable retention ranks eviction no better than random at super-patch granularity, while attention outperforms it at every tested granularity.At token granularity, retention provides only a weak inverse-importance signal; neither retention nor attention is a strong proxy.
Implications for KV-cache compression: The work’s practical implication is to steer visual-cache compression away from pixel-level decodability while leaving feature-space reconstruction methods outside the negative claim.The authors suggest causal or task-aware signals may be needed to exceed the low ceiling of single-proxy pruning.
Limitations: The conclusions are scoped to two VLMs, TextVQA, super-patch analysis, and a primary decoder layer, with single-unit ablation potentially missing redundant importance.The architecture contrast uses one model per class, and diffuse-dependence tasks may show different relationships.

A Post-hoc exploratory: cross-question stability of the attention axis

A post-hoc analysis of validation images with two official TextVQA questions finds that attention-based retained sets are highly stable across questions. However, weak cross-question utilization agreement and measurement reliability limit stronger conclusions about query-specific routing.

Scope: The analysis is exploratory and post-hoc, using only the 34 validation images carrying exactly two official questions.It was conducted after the preregistered study and was not preregistered.
Cross-question stability: Attention-based retained sets show high cross-question Jaccard overlap, reaching 0.87–0.95 for Gemma and 0.74–0.94 for InternVL across token budgets.The same pattern appears at super-patch granularity, against lower random-set expectations.
Interpretation: Cross-question causal-utilization maps correlate only weakly, but measurement reliability prevents concluding that utilization is query-specific.Same-question and cross-model consistency are of comparable magnitude, so unit-level noise remains plausible.
Control analysis: The second-question attention control recovers from 0.029 to 0.101 among high-agreement questions, matching the first-question baseline when gold targets are clean.This indicates the instrument behaves normally for cleaner answer annotations.
Reading: The observed stability supports interpreting attention-based eviction as selecting largely query-independent visual salience, while region-divergent queries remain untested.Whether this pattern persists in settings such as multi-page document QA is left open.

B Instrument details: the pixel-inversion decoder and sinv

The pixel-inversion decoder maps visual KV tokens back into pixel space to measure recoverable content. The shuffle-adjusted score sinv isolates position-specific decodability for each super-patch.

Decoder: The inversion decoder projects each 4096-dimensional KV token to a 48-channel spatial cell and uses convolutional upsampling to reconstruct a three-channel image.It is trained with a pixel MSE objective, with approximately 3.37M parameters per layer.
Reconstruction score and sinv: For super-patch u, sinv(u) is the intact-KV regional gradient correlation minus the average correlation after three within-image random KV-position permutations.The score is dimensionless and measures excess position-specific decodable content.
Reconstruction score and sinv: Gradient correlation compares concatenated horizontal and vertical gradients of the decoded and ground-truth images over each super-patch’s pixel region.The shuffle baseline removes content attributable to arbitrary KV material at the output location.

C Criterion validation: localizing OCR-grounded answer regions

The ablation-based utilization score is validated against independently defined OCR-grounded answer regions on both models. Retention instead shows an inverted association with answer location, while robustness checks largely preserve these patterns.

Criterion validation: The protocol matches OCR answers using exact, subsequence, or edit-distance rules and maps matched boxes to super-patches by center point.The implementation also included ten normalized ground-truth answer variants, reported separately from the preregistration-faithful surface.
Criterion validation: 0.791/0.766 gold-only AUCs show that Δtask localizes answer regions on Gemma and InternVL, respectively.The expanded-surface AUCs are 0.723/0.714, and all four intervals exclude 0.5.
Criterion validation: 3.23 and 4.16 paired contrasts show higher expanded-surface Δtask in answer than non-answer regions for Gemma and InternVL.The readout is therefore anchored to an external OCR criterion rather than only internal consistency.
Criterion validation: Retention has inverted answer-location discrimination, with expanded-surface AUCs of 0.393 for Gemma and 0.415 for InternVL.The association points away from answer regions rather than supporting retention as an importance proxy, while remaining distinct from the continuous retention–utilization null.
Robustness: Removing the edit-distance tier preserves the expanded-surface direction, while the reduced-sample IoU analysis widens the InternVL interval to include 0.5.Independent rank-sum recomputation exactly agreed with the gold-only AUC cells.