PaperlayerView exact revision on arXiv

Source-linked AI summary

VFold: Symmetry-Aware Cross-Layer Value Cache Compression

Neha Verma, Sungwon Kim, Kenton Murray, Kevin Duh

arXiv:2610.12338v1cs.CLcs.LG

TL;DR

At long context lengths, KV caches can rival or exceed model-weight memory, while cross-layer compression for pre-existing models can involve decoding overhead or accuracy costs. VFOLD folds symmetry-aware alignment maps into attention weights and shares adjacent-layer value caches after attention. It reports >98% of full-cache performance at 25% total reduction and composes with 4-bit KIVI for 4.27× compression.

  • Problem

    At long context lengths, KV-cache memory can rival model-weight memory, while cross-layer cache sharing for pre-existing models has involved decoding overhead or accuracy costs.

    Source: §1 · pp. 1–2

  • Method

    VFOLD aligns adjacent layers’ value caches with function-preserving maps folded into attention weights, then merges them into a shared cache.

    Source: §1 · p. 2, §4 · pp. 4–5

  • Results

    At 25% total reduction, VFOLD retains >98% of full-cache performance on RULER and LongBench across three models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VFOLD’s recovered full value vectors let 4-bit KIVI apply without modification, simplifying their composition in practice.

    Source: §6.3 · p. 9

    The head-permutation setup has shared QKNorm RMSNorm weights across heads and applies RoPE identically across heads.

    Source: §4 · p. 5

Abstract

from arXiv · show

While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities. However, most existing techniques necessitate architectural changes to LLMs and incur substantial overhead. In this work, we propose a symmetry-aware value cache merging strategy that reduces cache memory while avoiding both harmful performance degradation and architectural overhead during decoding. Furthermore, we show that this approach can be exploited alongside existing cache compression techniques, composing with high-ratio quantization or key cache pruning to reach compression ratios that neither method reaches alone, with minimal additional cost. Ultimately, our findings reveal a major source of underutilized capacity in the value cache, offering a simple yet highly effective direction for scaling context windows under memory constraints.

1 INTRODUCTION

Long-context KV caches can rival or exceed model-weight memory, making cache compression important. VFOLD addresses gaps in cross-layer compression for pre-existing models by aligning and sharing adjacent layers’ value caches while preserving model function.

1 INTRODUCTION

  • For a 16GB Llama-3.1-8B model, a batch of eight 16K-token inputs also requires a 16GB KV cache.At sufficiently long contexts, the cache can far surpass the memory required by model parameters.Source: §1 · p. 1
  • KV cache compression includes eviction, quantization, low-rank representations, and sharing caches across layers.These approaches reduce cache size through different mechanisms, including discarding tokens, using fewer bits, or sharing representations.Source: §1 · p. 1
  • Cross-layer sharing has largely focused on models trained from scratch, while low-rank approaches for pre-existing models require always-on compression and decoding reconstruction.Pre-existing-model cache merging avoids that overhead, but MiniCache interpolates without re-projection and degrades heavily on long-context benchmarks.Source: §1 · pp. 1–2
  • VFOLD folds function-preserving alignment maps into attention weights, then merges adjacent-layer value caches after they have been used in attention.Halving the value cache yields a 25% total reduction while retaining >98% of full-cache performance on RULER and LongBench across three models.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Related work reduces cache memory through cross-layer sharing, low-rank compression, predictor modules, and symmetry-aware reparameterization. VFOLD applies foldable alignment maps to cache compression in pre-existing models.

2 RELATED WORK

  • Several cross-layer sharing methods establish cache-sharing patterns at initialization and pre-train with them, so they do not directly apply to standard per-layer pretrained models.Examples include Cross-Layer Attention, You Only Cache Once, Layer-Condensed KV Cache, and FusedKV.Source: §2 · p. 2
  • Methods for pre-existing models include cache merging, direct sharing, and low-rank compression, but MiniCache incurs a notable accuracy cost.MiniCache uses spherical interpolation and protects high-importance tokens; KVSharer directly shares dissimilar layers’ caches for prefill and generation (Yang et al., 2024).Source: §2 · p. 2
  • Predictor-based methods use trained modules to predict key and value vectors from prior vectors across layers, whereas VFOLD avoids predictor modules through directly foldable alignment maps.AQUA-KV uses predictors for quantization, and EchoKV for direct compression (Ji et al., 2026).Source: §2 · p. 3
  • Symmetry-aware reparameterization has been used for pruning, quantization, and model merging; VFOLD adapts mapping principles for cache compression.Examples include SliceGPT, QuaRot, SpinQuant, and RotateKV.Source: §2 · p. 3

3 PRELIMINARY ANALYSIS AND MOTIVATION

Adjacent-layer keys and values have low raw cosine similarity but high CKA similarity, while experiments indicate that values are more amenable than keys to cross-layer averaging. The analysis motivates aligning value caches before merging them.

3 PRELIMINARY ANALYSIS AND MOTIVATION

  • Adjacent-layer keys and values show low cosine similarity but high CKA similarity, suggesting similarity only up to a change of basis.The analysis uses Llama-3.1-8B-Instruct on GSM8K; CKA is invariant to rotations and scaling.Source: §3 · p. 3
  • For token-dimension eviction, each key/value pair is removed together, whereas other compression methods can reduce only keys or treat keys and values asymmetrically.The comparison tests averaging across non-overlapping layer pairs while preserving the first four tokens and a buffer of the last 128 tokens.Source: §3 · pp. 3–4
  • Values are substantially more amenable to merging than keys, despite both exhibiting high CKA similarity.The analysis compares GSM8K accuracy after averaging each cache type across adjacent layers.Source: §3 · p. 4, Table 1 · p. 4

4 VFOLD

VFOLD prepares adjacent value caches for sharing with offline, function-preserving alignment maps, then merges cache values at runtime. Alignment and averaging together produce merged caches that remain similar to both source caches.

4 VFOLD

  • VFOLD computes CCA projections for adjacent layers and derives a map between their cache spaces to maximize correlation.The maps are constructed from input tokens; CCA projects each cache into a space with maximum correlation.Source: §4 · p. 5
  • To preserve attention invariance, VFOLD pairs heads using value states and uses the Hungarian algorithm to minimize total alignment cost.Dimensions from different KV heads cannot be mixed; pairwise costs use Frobenius error.Source: §4 · p. 5
  • On held-out data, the aligned average reaches cosine similarities of approximately 0.85 to both source caches.Figure 3 compares the merged cache with layer ℓ and maps it back to compare with layer ℓ+1.Source: Figure 3 · p. 5, §4 · p. 5
  • VFOLD permutes KV heads with P and transforms each head with block-diagonal T, folding the maps into attention weights and undoing them in the output projection.The permutation setup relies on shared QKNorm RMSNorm weights across heads and identical RoPE across heads.Source: Figure 4 · p. 5, §4 · p. 5
  • Direct averaging reaches cosine similarity of approximately 0.7, while aligned merging is more similar to both source caches than averaging or alignment alone.The comparison motivates combining alignment with averaging.Source: §4 · p. 6
  • Because the folded model preserves function, approximation enters only after each value has been used in attention, and merging materialized caches reduces peak forward-pass memory.The current step is computed with full values before they are merged into the shared cache.Source: §4 · p. 6

5 EXPERIMENTAL SETTINGS

Experiments evaluate VFOLD on three instruction-tuned models using LongBench and RULER, with cross-layer compression baselines and two VFOLD variants. Calibration and token protection settings define the tested configuration.

  • Tests cover Llama-3.1-8B-Instruct, Mistral-Small-3.1-24B-Instruct, and Qwen3-8B, all using Grouped-Query Attention.Qwen3-8B also uses Query-Key Normalization.Source: §5 · p. 6
  • LongBench evaluates real-world long-context tasks across six areas, while RULER supplies synthetic long-context tasks at 16K tokens.RULER includes Needle-in-a-Haystack and aggregative tasks.Source: §5 · p. 6
  • The cross-layer baselines include MiniCache and CommonKV; CommonKV uses a shared low-rank subspace and is adapted for Qwen3-8B's QK-Normalization.MiniCache is also evaluated as MiniCache-V, applying the matched 25% total-cache reduction to values alone.Source: §5 · pp. 6–7
  • VFOLD is tested with folded alignment maps and cache averaging, or with averaging alone; calibration uses 128 Wikitext-2 samples of 2048 tokens.The setup protects four attention sinks and a 128-token sliding window; tokens enter the shared cache as they leave that window.Source: §5 · p. 7

6 RESULTS

At 50% value-cache reduction, VFOLD retains near-full-cache performance and compares favorably with cross-layer baselines. Efficiency measurements and compositions with key pruning or quantization assess its overhead and compatibility with other compression methods.

6.1 PERFORMANCE ON LONG-CONTEXT BENCHMARKS

  • At 50% value-cache reduction (25% total KV cache), VFOLD retains at least 98.3% of full-cache performance on every model and leads baselines in five of six settings.The exception is Mistral-Small-3.1-24B-Instruct on LongBench, where VFOLD and CommonKV are within 1.2% of full-cache performance.Source: §6.1 · pp. 7–8
  • VFOLD outperforms MiniCache-V by a substantial margin on both RULER and LongBench, while alignment improves overall scores by over 9 percentage points on average on RULER.On CWE, alignment raises Llama-3.1-8B-Instruct from 2.2 to 75.3; even VFOLD-NAIVE surpasses CommonKV on RULER for that model.Source: §6.1 · p. 8

6.2 EFFICIENCY BENCHMARKS

  • VFOLD leaves TTFT essentially unchanged (781 vs. 786 ms), adds the least decoding overhead among compressed methods, and fits 38 versus 33 full-cache sequences.The measurements use an 8K-token prompt and 256 generated tokens on an A100-80G; the remaining 6.8 ms over full cache likely comes from the protected-token sliding buffer.Source: Table 4 · p. 8, §6.2 · p. 8

6.3 COMPOSITION WITH OTHER COMPRESSION METHODS

  • Combining VFOLD with ThinK reaches 50% cache compression, a ratio neither value-only nor key-only pruning can reach alone, without a detrimental performance drop.Combined performance loss is roughly additive, with a slight exception for Mistral on RULER.Source: Table 5 · p. 9, §6.3 · p. 9
  • With 4-bit KIVI, VFOLD raises compression from 3.2× to 4.27×; the combination stays within 0.3 points of VFOLD on every model and benchmark.KIVI+VFOLD slightly outperforms VFOLD in three of six cases, and KIVI applies without modification because VFOLD recovers full value vectors.Source: Figure 5 · p. 9, §6.3 · p. 9

7 CONCLUSION

VFOLD folds alignment maps into model weights so adjacent-layer value caches can be shared with negligible performance degradation while leaving attention unchanged. In grouped-query attention, head permutations and alignment maps must expand across the associated query heads.

7 CONCLUSION

  • VFOLD folds low-cost alignment maps into attention weights, enabling value-cache sharing with negligible performance degradation while leaving the model’s function unchanged.Decoding modifies only the cache, not attention.Source: §7 · p. 9

A VFOLD MAP DETAILS FOR GROUPED QUERY ATTENTION

  • In GQA, head assignment applies to stored KV heads, and permuting one KV head requires moving all g associated query heads with it.The permutation is expanded by a Kronecker product at the appropriate block size.Source: A VFOLD MAP DETAILS FOR GROUPED QUERY… · p. 13
  • Tq repeats each KV head’s map T for each of its g query heads, using block-diagonal blocks Ig⊗T.The Kronecker product moves query-head blocks as units, maintaining alignment with their corresponding KV heads.Source: A VFOLD MAP DETAILS FOR GROUPED QUERY… · p. 13

B CCA AND ITS INVERTIBILITY

The CCA alignment maps are full-rank maps computed for matched KV heads, and their numerical inverses are reported as stable with negligible output-product differences after folding.

B CCA AND ITS INVERTIBILITY

  • For each pair of matched KV heads, CCA computes a full-rank dh × dh map; the inverse is needed to maintain functional equivalence when folding it into the model.The implementation uses a numerical inverse, with λ = 10^-4 for numerical stability of the inverse square root.Source: B CCA AND ITS INVERTIBILITY · p. 13
  • For Llama-3.1-8B-Instruct with 2-cache groups, the maps are well-conditioned, their numerical inverses are stable and close to exact, and folded value-output product distances are negligible.The table reports condition numbers, inverse error, and value-output distance.Source: B CCA AND ITS INVERTIBILITY · p. 13, Table 6 · p. 14

C CACHE SIMILARITIES ON WIKITEXT

On Wikitext calibration data, aligned cache similarities are close to 0.9; the passage suggests the maps generalize sufficiently or are not highly sensitive to the tested distribution shift.

C CACHE SIMILARITIES ON WIKITEXT

  • Aligned-cache average cosine similarity is closer to 0.9 on Wikitext calibration data, compared with 0.85 on held-out data.Figure 6 reports cosine similarities between cache representations.Source: C CACHE SIMILARITIES ON WIKITEXT · p. 14
  • The result suggests Wikitext calibration data is general enough to fit alignment maps, and/or that the maps are not highly sensitive to the shift to GSM8K test data.The GSM8K comparison is described as a distribution shift from alignment data.Source: C CACHE SIMILARITIES ON WIKITEXT · p. 14

VFOLD TO MORE LAYERS

VFOLD extends to groups of four layers by aligning each layer to a common reference and incrementally averaging the aligned caches. At this compression ratio, its benchmark results vary by model, while alignment remains especially important on RULER.

VFOLD TO MORE LAYERS

  • For k layers, VFOLD aligns each non-reference layer directly to a middle-layer reference, then averages the aligned caches to reduce value-cache size to 1/k.Using a common reference rather than chaining pairwise maps avoids compounding alignment error; the k − 1 alignments are independent and preserve the model’s function.Source: VFOLD TO MORE LAYERS · p. 14
  • At this compression ratio, MiniCache-V is omitted because its SLERP cache-combination method is not specified for groups larger than two.For Qwen, grouping starts at layer 8 rather than layer 9 to allow grouping by two.Source: VFOLD TO MORE LAYERS · pp. 14–15
  • On LongBench at k = 4, VFOLD retains 96.4%, 98.4%, and 97.8% of full-cache performance on Llama, Mistral, and Qwen, respectively.Against CommonKV, VFOLD is +1.2 on Llama and +1.1 on Qwen, but -0.7 on Mistral.Source: Table 7 · p. 15, VFOLD TO MORE LAYERS · p. 15
  • The average is computed incrementally, combining two layers at a time so only one additional cache is buffered during execution.This reduces peak activation memory during inference.Source: VFOLD TO MORE LAYERS · p. 15
  • The incremental-mean update initializes the buffer with the first cache and updates it as each subsequent cache is added, yielding the group average.The request’s equation specifies αi = i/(i + 1).Source: VFOLD TO MORE LAYERS · p. 15
  • On RULER at k = 4, VFOLD beats CommonKV on Llama (83.3 vs. 63.7), but trails on Mistral (−0.6) and Qwen (−2.3).CommonKV scores 4.0 vs. 89.4 on Llama’s hardest multi-key task (MK3); on the remaining eleven tasks, VFOLD is within 0.4 points of CommonKV or ahead.Source: Table 8 · p. 16, VFOLD TO MORE LAYERS · p. 16
  • At k = 4, aligned averaging avoids the RULER collapse seen with VFOLD-NAIVE, whose scores are 17.3, 56.9, and 18.1 on Llama, Mistral, and Qwen.The VFOLD–VFOLD-NAIVE gap widens on RULER from 4.0–12.8 points at k = 2 to 37.2–67.1 points at k = 4; MiniCache degrades severely at this ratio on both benchmarks.Source: VFOLD TO MORE LAYERS · p. 16

E COMMONKV SETTINGS

  • CommonKV’s k = 4 comparison uses group size 4, with the number of groups selected to keep rank around 0.7 while matching the compression ratio.CommonKV experiments leave the first four sink tokens and 128 recent tokens unmerged.Source: Table 9 · p. 16, E COMMONKV SETTINGS · p. 16

F TIMING EXPERIMENTS DETAILS

Efficiency measurements used a single-GPU setup and randomly sampled prompts. The section defines TTFT and TPOT and reports timing means across timed trials.

F TIMING EXPERIMENTS DETAILS

  • Measurements used one NVIDIA A100-80GB GPU with PyTorch 2.5.1, CUDA 12.1, float16, and SDPA attention.Prompts had length 8192 and were generated by sampling token IDs uniformly over the vocabulary, followed by 256 greedy decoding tokens.Source: F TIMING EXPERIMENTS DETAILS · p. 17
  • TTFT measures elapsed time from prefill start to sampling the first token, while TPOT averages the 255 decoding steps after prefill.The first decoding step is computed from prefill.Source: F TIMING EXPERIMENTS DETAILS · p. 17
  • Timing results are means from two warm-up trials and three timed trials.Source: F TIMING EXPERIMENTS DETAILS · p. 17

Related papers

Loading 2610.12338v1…