PaperlayerView exact revision on arXiv

Source-linked AI summary

AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

Sara Abdali, Jongwoo Ko, Pashmina Cameron

arXiv:2610.06927v1cs.LGcs.AI

TL;DR

Long-context KV caches consume memory, while common training-free approaches save space by irreversibly evicting tokens. AttSVD retains all positions using prompt-specific, attention-guided low-rank factors and two decode modes; it remains competitive with dense caching and eviction baselines at reduced memory.

  • Problem

    KV caches grow with context length and dominate long-context memory, while many training-free compression methods irreversibly evict tokens.

    Source: §3 · p. 2

  • Method

    AttSVD uses an online, per-prompt attention-aware SVD, adaptive per-matrix ranks, and accumulating or streaming decode-time caching while retaining every token.

    Source: §1 · p. 1

  • Results

    At 50% cache, AttSVD matches or exceeds eviction and raw-SVD comparators across agentic evaluation and LongBench; AttSVD-Stream reaches 4.07 mean judge / 62.5% pass@4.

    Source: §4.1 · p. 7, §4.2 · p. 8

  • Takeaways & Limitations

    The retained factors provide per-head measures of effective-rank compressibility and attention geometry alongside compressed storage.

    Source: §5 · p. 9

    The interpretability quantities are geometric rather than semantic: singular directions mix position and content and describe prompts and heads, not predictions.

    Source: §3.6 · p. 6

Abstract

from arXiv · show

The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free remedies evict low-importance tokens, an irreversible choice along the sequence axis. We instead keep every token and store it more cheaply along the "feature" axis. We therefore propose AttSVD, a new "interpretable" low-rank compression whose basis is derived from each prompt's own attention geometry: an online, per-prompt truncated SVD that keeps only the directions attention actually reads, cutting persistent per-head KV memory in proportion to the retained rank. We propose two decode-time caching strategies, accumulating and streaming, for short and long generation regimes. Furthermore, we propose two refinements that make compression adaptive. A per-matrix energy rule sizes the logit space and the attention mass independently. An attention-aware basis truncates only in the spaces attention actually reads, preserving both the attention logits and the attention output. The same factors also provide free, per-head interpretability insights into the effective rank and the geometry attention consumes. Across multiple models, on both an agentic benchmark and the full LongBench suite AttSVD stays on par with the dense cache while using up to 50% of the KV-cache memory.

1 INTRODUCTION

AttSVD replaces irreversible token eviction with prompt-specific low-rank storage that retains every position. Its adaptive, attention-aware factors and two decode-time modes target different generation regimes while also exposing per-head diagnostics.

1 INTRODUCTION

  • Eviction methods drop tokens permanently, whereas reconstruction methods retain all positions and store them approximately.A wrongly evicted token cannot be attended to again.Source: §1 · p. 1
  • AttSVD computes an online, per-prompt factorization in the subspace attention reads, retaining all token positions without calibration or training.The basis comes from the prompt’s own activations and is optimal for its attention logits and mass at rank r.Source: §1 · p. 1
  • A dynamic per-matrix rank rule sizes keys and values separately, while an attention-aware basis preserves both attention logits and output.The method is exact at full rank.Source: §1 · p. 1
  • Accumulating mode keeps generated tokens exact for short-decode settings, while streaming mode compresses aged-out tokens for long generations.The prompt-derived ranks and attention-mass profile also provide per-head diagnostics without calibration.Source: §1 · p. 2

2 RELATED WORK

Prior low-rank KV methods commonly reuse a basis learned offline or from calibration, while OjaKV tracks an approximate calibration-seeded basis. AttSVD instead derives an exact, attention-guided basis for each prompt and adapts rank independently for each matrix.

2 RELATED WORK

  • Prior methods reuse offline weight decompositions, calibration-derived bases, or shared or trained bases; OjaKV instead tracks an approximate incremental-PCA basis seeded from calibration.OjaKV uses attention to select full-rank tokens, not to shape its basis.Source: §2 · p. 2
  • AttSVD computes an exact per-prompt truncated SVD at prefill without calibration or training, then freezes the basis during generation.This contrasts with OjaKV’s seeded, incremental estimate.Source: §2 · p. 2
  • AttSVD weights keys by query covariance and values by attention mass, aligning truncation with logits and output rather than variance alone.Its spectral-energy rule also selects a rank independently for each matrix.Source: §2 · p. 2

3 METHOD

AttSVD retains every token in a cheaper feature-axis representation, using prompt-specific attention geometry to guide compression and decode-time cache strategies.

3.1 PROPOSED ATTSVD

  • AttSVD keeps all token positions in compressed factors plus a small exact recent-token cache, rather than evicting tokens.The method is training-free and retains tokens in a cheaper form.Source: §3 · p. 2, Figure 1 · p. 3

3.2 PER-PROMPT AND ATTENTION-AWARE LOW-RANK COMPRESSION

  • Attention-aware truncation weights keys by query geometry and values by per-key attention mass, preserving the attention logits and output.For values, the method row-weights V by the square root of attention mass before applying standard truncated SVD.Source: §3.2 · p. 4

3.3 TWO KV-CACHE COMPRESSION REGIMES TAILORED TO PREFILL AND DECODE LENGTH

  • Prefill compresses the prompt bulk while retaining a small exact tail; accumulating keeps generated tokens exact, whereas streaming compresses aged-out chunks.Streaming maintains a bounded exact window and compresses chunks from their exact values, so compression error does not compound across generation.Source: §3.3 · pp. 4–5

3.4 CHOOSING THE RANK: COMPRESSION-RATIO OR PER-MATRIX ENERGY

  • A shared compression ratio sets a predictable common rank, while a per-matrix energy rule selects separate key and value ranks from their spectra.The default energy tolerance is 0.99, and ranks can be aggregated across heads or capped.Source: §3.4 · pp. 5–6

3.5 MEMORY COST

  • For long contexts with shared rank r, the persistent cache ratio approaches r/d; streaming bounds the exact buffer independently of generation length.Streaming trades extra decode-time SVDs for a bounded exact buffer; its compressed blocks approach the same r/d storage ratio when chunk length is large relative to d.Source: §3.5 · p. 6

3.6 INTERPRETABILITY: WHAT THE RETAINED BASIS EXPOSES

  • The retained ranks, query-subspace spectrum, and attention-mass profile provide per-head geometric diagnostics, including which heads have more compressible context.These diagnostics are geometric rather than semantic: basis vectors mix position and content and do not explain a prediction.Source: §3.6 · p. 6

4 EXPERIMENTS

The experiments compare AttSVD with dense-cache, eviction, and low-rank baselines on an agentic benchmark and LongBench. At 50% cache, AttSVD remains close to dense-cache quality, while per-prompt, per-matrix ranks also deliver near-exact LongBench quality.

4 EXPERIMENTS

  • Evaluation uses 184 agentic prompts and the full LongBench suite of 1,750 examples across 9 tasks, with methods compared at matched cache budgets.The agentic benchmark reports mean judge score and pass@4; LongBench reports native per-task metrics and their average.Source: §4 · p. 7

4.1 ATTSVD VS. EVICTION METHODS

  • On Phi-4-14B LongBench at 50% cache, AttSVD-Accum scores 36.0 versus KVzip 35.4, H2O 34.9, and KeyDiff 33.5.On Qwen3-8B, it ties the strongest eviction methods and stays within 0.7 of the dense cache.Source: §4.1 · p. 7
  • At 50% cache, AttSVD-Stream scores 4.07 mean judge / 62.5% pass@4 and AttSVD-Accum scores 4.01 / 59.2% on the agentic benchmark.Figure 2 reports results for Phi-4-14B and Qwen3-8B; the paper reports Qwen3 gaps among Full, AttSVD, H2O, and KVzip within the 95% confidence interval.Source: §4.1 · p. 7, Figure 2 · p. 8

4.2 ATTSVD VS. OTHER LOW-RANK COMPRESSION METHODS

  • Across the rank range, AttSVD follows the dense cache more closely than raw-KV SVD; at r=64, raw-KV SVD scores 3.79/41.3% on the agentic benchmark.AttSVD adds +0.13 to 0.29 mean judge and +7 to 18 pass@4 points over raw-KV SVD at matched memory.Source: Figure 3 · p. 8, §4.2 · p. 8
  • On Llama-3.1-8B-Instruct, AttSVD-Accum averages 48.5 versus the dense cache’s 48.5 at 80% cache and 47.8 versus 48.5 at 60%.It leads KQ-SVD by 1.1 to 2.6 points and both AttSVD schedules lead OjaKV, which scores 42.9 at 80% and 30.5 at 60%.Source: §4.2 · p. 8, Table 3 · p. 9

4.3 DYNAMIC ENERGY-RULE RANK

  • On the agentic benchmark, dynamic energy ranks match the dense cache from τ=0.99 onward; by τ=0.999, mean judge scores are 4.098 versus 4.145.The reported pass@4 values at τ=0.999 are 64.7% versus 66.8%.Source: §4.3 · p. 8, Figure 4 · p. 9
  • On Phi-4-14B LongBench, τ=0.997 selects average ranks r_eK ≈65 and r_eV ≈127; at ≈75% cache, the average is 36.5 versus 36.7 for dense.The same tolerance is applied to each matrix’s own spectrum, allowing each prompt to set its ranks.Source: §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSION

AttSVD retains every token while storing it in an instance-optimal, per-prompt low-rank subspace. Its adaptive ranks, attention-aware basis, and decode schedules support cache compression, while the factors also expose attention-related diagnostics.

5 CONCLUSION

  • AttSVD retains every token and uses an online, per-prompt SVD whose persistent cache shrinks to ≈r/d of dense.The method is exact at full rank and reconstructs lazily at decode.Source: §5 · p. 9
  • Adaptive per-matrix ranks and an attention-aware objective allocate compression according to the directions attention reads.The paper also provides accumulating and streaming decode modes for different generation scenarios.Source: §5 · p. 9
  • The factors provide per-head interpretability through effective-rank maps, query subspaces, and attention-mass profiles without calibration.Across several models, AttSVD matches dense-cache performance on the agentic benchmark and LongBench at roughly half the KV-cache memory.Source: §5 · p. 9

USE OF AI TOOLS

The authors describe the assistive uses of AI tools and state that authors conceived, verified, and approved the technical content and conclusions.

USE OF AI TOOLS

  • AI tools assisted with clarifying initial statements, drafting LATEX sections, improving figures and tables, and correcting grammar.Source: USE OF AI TOOLS · p. 10
  • The tools also helped make sentences more salient and served as a coding assistant for implementing the authors’ ideas.Source: USE OF AI TOOLS · p. 10
  • The authors state that they conceived, verified, and approved all technical content, claims, and conclusions.Source: USE OF AI TOOLS · p. 10

ETHICS STATEMENT

The authors characterize AttSVD as a training-free, inference-time method operating on frozen-model KV caches. They state that existing model biases and fairness limitations are inherited unchanged.

ETHICS STATEMENT

  • AttSVD operates at inference time on the key and value caches of a frozen pretrained model and introduces no new training signal.Source: ETHICS STATEMENT · p. 10
  • The authors state that biases and fairness limitations already present in the underlying models are inherited unchanged.Source: ETHICS STATEMENT · p. 10
  • The authors report no competing interests and no external funding that could have influenced the findings.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper documents its algorithms, proofs, and experimental setup, and describes how AttSVD differs from prior KV-cache compression and reconstruction methods.

REPRODUCIBILITY STATEMENT

  • The method and experimental setup are documented, with pseudocode for the algorithms, a proof of attention-aware truncation, and core code plus unit tests released as supplementary material.The full repository is deferred until acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXTENDED RELATED WORK

  • Eviction methods score tokens for retention or use non-binary retain, approximate, and evict allocations; the paper compares against H2O, KeyDiff, and KVzip.Table 5 groups SVD and low-rank KV-cache methods by basis source.Source: A EXTENDED RELATED WORK · p. 13, Table 5 · p. 13
  • Prior methods use bases from weights, calibration, or training, while per-prompt alternatives include key-only compression, approximate running bases, and fixed value maps.AttSVD instead derives its per-prompt low-rank basis from that prompt’s attention.Source: A EXTENDED RELATED WORK · p. 13

B PROOF OF EQ. (3) (ATTENTION-AWARE KEY TRUNCATION)

The proof transforms attention-weighted key and value approximation problems into ordinary truncated SVDs. The resulting metrics target the query geometry for keys and per-key attention mass for values.

B PROOF OF EQ. (3) (ATTENTION-AWARE KEY TRUNCATION)

  • Weighting key reconstruction by the query Gram matrix makes its objective equal to attention-logit error, and a change of variables reduces the rank-constrained problem to ordinary truncated SVD.The truncated SVD yields a global minimizer, unique when the rth singular value exceeds the next one.Source: B PROOF OF EQ. (3) (ATTENTION-AWARE… · pp. 13–14
  • For values, weighting rows by per-key attention mass turns the output-error objective into a truncated SVD of the mass-weighted values, followed by rescaling.This uses the diagonal per-key mass metric rather than the exact row metric.Source: B PROOF OF EQ. (3) (ATTENTION-AWARE… · p. 14
  • At 50% cache, diagonal mass weighting lowers relative value output error from 0.154 to 0.012, capturing most of the attainable gain.The exact row metric is costly to form, requires dense attention, and can be singular when queries are fewer than keys.Source: B PROOF OF EQ. (3) (ATTENTION-AWARE… · p. 14

C ALGORITHMS

AttSVD factors the prompt bulk with attention-guided key and value bases, then uses either an exact growing decode tail or streaming compression of aged-out chunks. On LongBench, streaming remains within run-to-run noise of accumulating while bounding that tail.

C ALGORITHMS

  • Accumulating keeps generated tokens exact, whereas streaming factors aged-out chunks into frozen low-rank blocks and keeps an exact sliding window.The accumulating exact tail grows with generation length; streaming bounds the exact buffer.Source: C ALGORITHMS · p. 15, §D.1 · p. 16
  • The prompt bulk is factored once into attention-aware key and value factors; an energy rule can select separate ranks from each matrix’s spectrum.Keys use query geometry, while values use attention mass; streaming chunks without queries use unweighted truncated SVD.Source: C ALGORITHMS · p. 15

D.1 STREAMING VS. ACCUMULATING DECODE

  • Streaming stays within 0.3 points of accumulating on average across LongBench budgets, while replacing its growing exact decode tail with a bounded window.The streaming-minus-accumulating gap is ∆∈[−0.29, +0.24], and streaming follows the dense cache (36.7) down to the 50% budget.Source: Table 6 · p. 16, §D.1 · p. 16

D EXPERIMENTAL SETUP AND EXTENDED EXPERIMENTS

  • At full rank, decomposition reproduces dense-attention output bit-for-bit, and at 32K context the measured KV-cache ratio matches r/d to the decimal.At shorter contexts, fixed factor and exact-tail terms make the ratio slightly larger.Source: D EXPERIMENTAL SETUP AND EXTENDED… · p. 16

D.2 STREAMING-WINDOW ABLATION ON A LONG-GENERATION TASK

The window ablation tests streaming where decode history actually overflows, while the synthetic-head experiment compares attention-aware truncation with raw SVD when relevant directions are misaligned with dominant singular directions.

D.2 STREAMING-WINDOW ABLATION ON A LONG-GENERATION TASK

  • On GovReport at 80% cache, ROUGE-L varies by under 0.15 as the exact window shrinks from 256 to 16 tokens and stays level with the accumulating cache.All variants share the same attention-aware prefill factorization.Source: Table 7 · p. 16, §D.2 · p. 17

D.3.1 ADVERSARIAL CASE ON A CONTROLLED SYNTHETIC HEAD

  • At rank 4 on an adversarial synthetic head, attention-aware truncation reduces relative key-logit error from 0.98 to 0.27; mass-weighted value error falls from 0.92 to 0.64 at rank 8 and reaches 0.00 at rank 16.The setup misaligns query-relevant key directions with dominant singular directions, challenging energy-ordered SVD.Source: §D.3.1 · p. 17

D.3.2 REAL-WORLD INTERPRETABILITY ANALYSIS

A real SmolLM2-135M head illustrates how attention-aware factorization changes which directions and tokens are retained. Its key/value spectra, query-relevant errors, and token-weighted reconstruction connect the method’s design to per-head structure and benchmark results.

Real-world interpretability analysis

  • The key spectrum is steep, with about 67% of energy in its leading factor and 99.5% in the top 32, while the value spectrum puts about 36% in its leading factor.The flatter value spectrum indicates values are less compressible than keys, motivating separate per-matrix ranks.Source: §D.3.2 · p. 17
  • At 50% cache, AttSVD reaches 4.067 / 62.5% on the Phi-4-14B agentic benchmark, within 0.08 of dense, while improving mean judge by +0.13 to 0.29 and pass@4 by +7 to 18 points over raw-KV SVD.The reported result is for the 50% cache setting.Source: §D.3.2 · p. 18
  • Attention-mass weighting reconstructs heavily attended tokens, including the first-token sink, lowering relative value reconstruction error from 0.154 to 0.012 at 50% cache.The factor panels also show per-token weights and top-contributing tokens for each factor.Source: §D.3.2 · p. 18, Figure 6 · p. 19
  • At 50% cache, attention-aware keys nearly match the full-attention target, and query-relevant logit error is 0.013 versus 0.141 for raw-KV SVD.Across budgets, query-relevant logit error stays more than 10× lower; the factor-level and real-head visualizations use SmolLM2-135M for legibility, while the effect also holds on larger models in the main experiments.Source: §D.3.2 · pp. 17–18, Figure 7 · p. 18
Loading 2610.06927v1…