PaperlayerView exact revision on arXiv

Source-linked AI summary

SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention

Zihan Teng, Jiayu Zhao, Wentao Ren, Minhao Fan, Tianrui Ma, Song Chen, Weichen Liu

arXiv:2610.02953v1cs.LG

TL;DR

Long-context inference increases KV-cache memory pressure, while token-wise and feature-wise compression leave open challenges around information loss and Key reconstruction. SlimKV combines low-rank beacon representations with selective beacon K-RoPE removal for latent-space attention. It reports LongBench gains under compression and faster attention and decoding.

  • Problem

    Long-context KV-cache growth pressures memory, while existing compression approaches leave open information-loss and Key-reconstruction challenges.

    Source: §1 · pp. 1–2

  • Method

    SlimKV jointly compresses tokens and features with layer-adaptive low-rank beacon KV representations and removes K-RoPE only from beacon Keys for latent-space attention.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    SlimKV shows clear advantages under high compression, leads at lower ratios while retaining over 96% of the uncompressed model’s LongBench score, and reaches up to 7.34× attention and 3.38× end-to-end decoding speedup.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments on LongBench, Needle-in-a-Haystack, and decoding efficiency demonstrate a stronger accuracy-efficiency trade-off than prior KV-cache compression methods.

    Source: §6 · p. 9

    SlimKV is mainly evaluated on RoPE-based language models, so its behavior under other positional encoding schemes remains to be examined.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Long-context LLM serving is increasingly bottlenecked by KV-cache memory, especially in resource-constrained scenarios. Among existing KV-cache compression strategies, token-wise methods reduce cached states but risk information loss through eviction or condensation, while feature-wise methods reduce per-token KV dimensions but can require full-dimensional reconstruction to apply positional embedding, limiting decoding speedups. We introduce SlimKV, a question-agnostic joint token-feature KV-cache compression method. SlimKV uses low-rank-aware training to compress long contexts into beacon memory states with latent KV representations, together with layer-adaptive rank allocation. We further uncover a positional asymmetry: removing key-side RoPE affects beacon and raw tokens differently, with much smaller degradation for beacon tokens. Exploiting this asymmetry, SlimKV trains beacon KV projections under a K-RoPE-free constraint and enables latent-space attention during decoding, mitigating reconstruction latency. On LongBench, SlimKV outperforms baselines at 16x/32x compression and remains leading at 4x/8x, where it retains over 96% of the uncompressed model's score. Needle-in-a-Haystack confirms robustness across evidence positions, and efficiency evaluation shows up to 7.34x attention speedup and 3.38x end-to-end decoding speedup over the uncompressed model at 128K length.

1 Introduction

Long-context inference faces growing KV-cache memory pressure, while token-wise and feature-wise compression each have limitations. SlimKV combines both approaches and uses beacon-specific K-RoPE removal to enable latent-space attention; its reported results span accuracy, positional robustness, and decoding efficiency.

1 Introduction

  • Token-wise compression can lose information through deletion, while feature-wise compression can require full-dimensional Key reconstruction before RoPE.The paper frames these limits as motivation for joint token-feature compression that avoids the reconstruction bottleneck.Source: §1 · pp. 1–2
  • Removing key-side RoPE degrades raw-token Keys more than beacon Keys, so SlimKV removes it only from beacon Keys to enable reconstruction-free latent-space attention.RoPE remains for raw-token Keys and all Queries.Source: §1 · p. 2
  • The paper evaluates SlimKV on LongBench, Needle-in-a-Haystack, and long-context decoding latency to assess accuracy, positional robustness, and efficiency.The central question is whether joint token-feature compression can overcome single-axis limits without reconstruction.Source: §1 · p. 2
  • SlimKV jointly compresses tokens and features by learning compact beacon KV representations and allocating layer-adaptive ranks under a fixed feature budget.Beacon states summarize historical tokens in latent KV space.Source: §1 · p. 2
  • SlimKV shows clear advantages under high compression, leads at lower ratios while retaining over 96% of the uncompressed model’s LongBench score, and reaches up to 7.34× attention and 3.38× end-to-end decoding speedup.The evaluation also covers Needle-in-a-Haystack and long-context decoding latency.Source: §1 · p. 2

2 Related Work

Prior work compresses KV caches along token or feature dimensions, with trade-offs involving information loss, capacity, prefill computation, and reconstruction latency. SlimKV’s overview combines beacon-based token compression with latent KV representations.

2 Related Work

  • Token-wise methods reduce historical states through prompt or KV compression, but deletion can lose useful information and aggressive compression can still face capacity limits.KV-level compression reduces decoding cache cost but not prefill computation for the original long prompt.Source: §2 · p. 2
  • SlimKV’s overview shows low-rank beacon KV projections and latent beacon memories cached after prefill for latent-space attention during decoding.This avoids full-dimensional reconstruction during decoding.Source: Figure 3 · p. 3
  • Feature-wise methods shrink per-token representations or share heads, but post-hoc low-rank Keys can require full-space reconstruction before RoPE and attention scoring.Latent-KV architectures such as MLA typically require pretraining from scratch and are less applicable to existing GQA backbones.Source: §2 · p. 3

3 Joint Token-Feature Compression

SlimKV combines progressive beacon-based token compression with low-rank feature compression, caching compact latent beacon Keys and Values. It adapts projection ranks across layers using raw-token KV spectra and a floor-constrained equal-energy allocation.

3.1 Overview

  • SlimKV inserts beacon tokens at fixed intervals, lets each chunk absorb context into previous beacon memories, then discards raw-token KV states after prefill.It extends this progressive compression pipeline to cache only latent beacon Keys and Values under a global feature budget.Source: §3.1 · p. 3

3.2 Low-Rank Beacon Training

  • During inference, SlimKV stores latent beacon Keys and Values in the KV cache rather than their full-dimensional forms.The cached states are the compressed beacon memories produced by the chunk-wise pipeline.Source: §3.1 · p. 3, §3.2 · p. 3
  • Beacon projections are trained directly at their target low ranks, with the backbone frozen and only beacon projection parameters updated.Beacon hidden states are mapped into layer-specific latent spaces instead of producing full-dimensional Keys and Values.Source: §3.2 · p. 3

3.3 Layer-Adaptive Rank Allocation

  • SlimKV estimates layer-specific beacon ranks from pretrained raw-token KV singular-value spectra, whose rank patterns are reported to be highly consistent with beacon projections.It initializes each low-rank beacon projection with the truncated SVD of its raw-token counterpart.Source: §3.3 · p. 4, Figure 4 · p. 4
  • A floor-constrained equal-energy criterion allocates ranks across layers, and unused budget goes to layers with the largest marginal energy gain.The rank floor addresses the risk that pure equal-energy allocation assigns very small ranks to concentrated spectra.Source: §3.3 · p. 4

4 From Compression to Acceleration: Latent-Space Attention

SlimKV uses RoPE-aware beacon generation and exploits the smaller impact of removing key-side RoPE from beacon Keys to support latent-space attention. This avoids full-Key reconstruction, whose sequence-dependent cost can dominate long-context decoding.

4.1 RoPE-aware Beacon Generation

  • RoPE-aware beacon queries attend to RoPE-applied raw Keys, allowing beacon states to absorb coarse positional cues despite lacking explicit K-RoPE.The attention weights encode relations between beacon and raw-token positions.Source: §4.1 · p. 4

4.2 Positional Asymmetry

  • Removing K-RoPE degrades beacon Keys less than raw-token Keys, and RoPE-free beacon states reconstruct random sequences above chance.The reconstruction test uses length-16,384 sequences over 46 symbols and starts from 8× total compression.Source: §4.2 · p. 5, Figure 6 · p. 5

4.3 Reconstruction-free Latent-Space Attention

  • SlimKV removes K-RoPE from beacon Keys during training, enabling latent-space attention to compute scores and aggregate Values without full-Key reconstruction.Queries retain RoPE; the Value path is handled analogously in latent space.Source: §4.3 · p. 5

4.4 Cost Analysis

  • The reconstruction baseline materializes full-dimensional KV states at O(NbD2/k) cost, while SlimKV replaces this sequence-dependent cost with O(gD2/k) per-step projection overhead.Both approaches retain O(NbD) attention computation; the analysis concerns arithmetic cost, with empirical latency evaluated separately.Source: §4.4 · p. 5

5 Experiments

Across the evaluated compression settings, SlimKV leads LongBench comparisons and maintains retrieval across needle positions, while experiments also assess larger-model scalability, decoding efficiency, and design ablations. The evaluation uses frozen backbones and trains only beacon-branch QKV projections.

5.1 Experimental Setup

  • Evaluation uses Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct with frozen backbones, training only beacon-branch QKV projections on a two-stage data recipe.Training uses autoregressive loss on non-beacon tokens; beacon tokens serve as compressed memory and are excluded from supervision.Source: §5.1 · p. 6

5.2 Effectiveness Evaluation

  • At high compression, SlimKV has the best average LongBench score across both backbones; at 4× and 8×, it preserves over 96% of Full KV’s average score and leads in most settings.The 4× feature-compression setting strikes the best balance in most high-compression settings.Source: Table 4 · p. 6, §5.2 · p. 7
  • SlimKV maintains retrieval across context lengths and needle depths, including information beyond Qwen2.5-14B’s 32K context window.The Figure 7 evaluation reports performance across depths under 16× and 32× compression, and beyond-window retrieval at 8×.Source: Figure 7 · p. 7, §5.2 · p. 8

5.3 Scalability and Parameter Analysis

  • On Qwen3-30B-A3B-Instruct-2507, SlimKV-f4 retains competitive LongBench performance with 434M trainable parameters, 1.45% of the model size.The study describes this as preliminary transfer to a larger MoE model.Source: §5.3 · p. 8, Table 5 · p. 8

5.4 Efficiency Evaluation

  • At 11.5K tokens, SlimKV t4f4 and t8f4 achieve 1.03× and 1.37× TTFT speedups, reducing peak memory by 2.71× and 2.82×, respectively.The uncompressed model ran out of memory at longer contexts, so TTFT and peak memory were compared at its longest feasible length.Source: §5.4 · p. 8, Table 6 · p. 8

5.5 Ablation Studies

  • Ablations show that removing beacon K-RoPE alone degrades accuracy; low-rank beacon training and layer-adaptive ranks contribute to the full design’s advantage over Activation Beacon.The ablation uses Llama-3.1-8B-Instruct at 32× total compression.Source: §5.5 · p. 8, Table 7 · p. 8

6 Conclusion

SlimKV jointly compresses token and feature dimensions through layer-adaptive low-rank beacon KV states. Removing key-side RoPE only from beacon Keys enables reconstruction-free latent-space attention, and experiments report a stronger accuracy-efficiency trade-off than prior compression methods.

6 Conclusion

  • SlimKV combines question-agnostic token-feature compression, layer-adaptive low-rank beacon KV representations, and beacon-only key-side RoPE removal for reconstruction-free latent-space attention.Source: §6 · p. 9
  • Evaluations on LongBench, Needle-in-a-Haystack, and decoding efficiency show a stronger accuracy-efficiency trade-off than prior KV-cache compression methods.Source: §6 · p. 9

Limitations

The paper identifies positional-encoding coverage as a limitation: SlimKV is mainly evaluated on RoPE-based language models, and its behavior under other positional schemes remains to be examined.

Limitations

  • SlimKV is mainly evaluated on RoPE-based LLMs; behavior under other positional encoding schemes remains unexamined.Source: Limitations · p. 9

A.1 Quantitative Analysis of Layer-wise Rank Consistency

Beacon and raw K/V projections show similar layer-wise rank profiles across models, projections, and compression ratios, supporting raw-projection spectra as a proxy for allocating Beacon ranks.

A.1 Quantitative Analysis of Layer-wise Rank Consistency

  • Table 8 compares average selected ranks, layer-order agreement via Spearman correlation, and mean and maximum absolute per-layer rank differences.Ranks use the same explained-variance criterion and identical average-rank budget for each target compression ratio.Source: Table 8 · p. 12, §A.1 · p. 12
  • Across models, projections, and compression ratios, Beacon and raw projections have consistently high rank-profile Spearman correlations and per-layer rank differences within one or two ranks in almost all cases.The authors therefore support using raw K/V projection spectra as a proxy for layer-adaptive rank allocation before Beacon training.Source: §A.1 · p. 12

A.2 Interpretation of Attention Similarity Metrics

The section defines complementary measures of attention-map similarity, comparing attended-key distributions, global map patterns, coarse structure, and relative-distance allocation.

A.2 Interpretation of Attention Similarity Metrics

  • Attention-map Pearson correlation flattens valid causal-region entries, and higher values indicate more similar global attention patterns.The comparison is between two averaged attention maps.Source: §A.2 · p. 12
  • Coarse-block Pearson correlation compares averaged valid entries within coarse blocks, with higher values indicating more similar macroscopic structures.The metric summarizes attention-map structure at a coarse scale.Source: §A.2 · p. 12
  • Lag-profile JS divergence compares normalized attention mass by relative distance ℓ=i−j, with lower values indicating more similar near-range and long-range allocation.The two profiles are formed from attention maps A and B over the valid causal region.Source: §A.2 · p. 12
  • Column-profile JS divergence compares normalized attention mass aggregated by key position, with lower values indicating more similar attended-key distributions.The profiles are formed over key position j.Source: §A.2 · p. 12

A.3 Derivation of Decoding Cost

SlimKV keeps beacon attention in compressed latent space, avoiding sequence-scaled full-KV reconstruction while preserving exact standard attention for a hybrid cache. The appendix also reports implementation conditions, efficiency measurements, task-wise results, positional-evidence evaluation, architecture comparisons, and license terms.

A.3 Derivation of Decoding Cost

  • SlimKV avoids the reconstruction baseline’s Nb-scaled full-KV reconstruction cost by keeping dominant attention computation in compressed latent space and applying low-rank transformations only to current-step query/output.The cost analysis compares arithmetic scaling; it does not itself predict measured wall-clock speedup because lower-order operations, launch overhead, memory layout, and bandwidth are omitted.Source: §A.3 · p. 13

A.3.1 Hybrid Attention with Raw-Token Tail

  • Hybrid attention processes beacon tokens in latent KV space and raw-token tail states in original feature space, then normalizes their concatenated logits together.This is mathematically equivalent to reconstructing beacon KV states and applying standard attention, while avoiding explicit full-dimensional reconstruction for beacon tokens.Source: §A.3.1 · pp. 13–14

A.4 Implementation Details

  • Training freezes the backbone and updates only beacon-branch QKV projections; runs use BF16 on four NVIDIA A100 80GB GPUs, with model-dependent training time and peak memory.Llama-3.1-8B-Instruct uses about 8 hours of pretraining and 6 hours of fine-tuning at about 50GB peak GPU memory; Qwen2.5-14B-Instruct uses about 14 and 11 hours, respectively, at about 78GB peak memory.Source: §A.4 · p. 14

A.5 Detailed Efficiency Results

  • Table 9 reports attention and end-to-end latency over 128 generated tokens, and SlimKV consistently reduces both relative to the reconstruction-based variant.Table 10 reports TTFT and peak allocated memory; higher compression lowers peak memory, while SlimKV and the reconstruction variant have nearly identical peak memory because reconstruction changes computation, not stored cache size. SlimKV’s TTFT improvements over reconstruction are modest and attributed mainly to avoiding key-side RoPE and full-Key materialization.Source: §A.5 · pp. 14–15, Table 9 · p. 14, Table 10 · p. 14

A.6 Task-wise Analysis on LongBench

  • LongBench gains are most consistent for document QA, while summarization gains are smaller but stable and few-shot and code trends vary by backbone and task.The authors characterize RoPE-free beacon memory as especially effective for semantic aggregation and retrieval-oriented tasks, with fine-grained order-sensitive tasks remaining more sensitive to compression design.Source: §A.6 · p. 15

A.7 Needle-in-a-Haystack Evaluation Details

  • Figure 8 gives retrieval-accuracy heatmaps by context length and needle depth, averaged over seeds 42, 43, and 44; SlimKV-f4 maintains more stable retrieval performance than Activation Beacon.The comparison covers Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct under 16× and 32× total compression.Source: §A.7 · p. 15, Figure 8 · p. 16

A.8 Comparison with MLA

  • Unlike MLA’s token-granular content latent and separate positional-Key branch, SlimKV jointly compresses tokens and K/V features and acquires beacon positional cues during RoPE-aware prefill.The text also states that SlimKV retrofits pretrained GQA-RoPE backbones by training only beacon projections, whereas MLA requires costly base-model pretraining and is tightly coupled to model architecture.Source: §A.8 · p. 15

A.9 Compatibility with ALiBi; A.10 Licenses and Intended Use of Artifacts

  • ALiBi is algebraically compatible with reconstruction-free latent-space attention because it adds positional bias after the QK product; the artifacts are reported as used under their respective licenses and intended research purposes.The appendix lists the model, corpus, benchmark, and judge license terms, including use of GPT-5.4-mini through the OpenAI API under its standard terms.Source: §A.9 · p. 15, §A.10 · p. 15

Related papers

Loading 2610.02953v1…