PaperlayerView exact revision on arXiv

Source-linked AI summary

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

Yaoyang Liu, Shuwei Shen, Jincheng Tang, Jinguang Liu, Jinbao Xue, Hailong Yang, Ke Zhang, Yinjun Wu

arXiv:2610.17966v1cs.LGcs.AI

TL;DR

Long-context inference makes the growing KV cache a major memory bottleneck, and current compression objectives and rank allocation leave attention behavior or global budgeting insufficiently addressed. BELT combines attention-aware low-rank compression with learned global rank allocation under a user-specified component budget. Across evaluated models and tasks, it improves quality under aggressive compression and reports higher decoding throughput at comparable accuracy.

  • Problem

    Long-context inference makes the KV cache a major memory bottleneck, while existing low-rank criteria may not preserve attention behavior and rank allocation under a prescribed global budget remains challenging.

    Source: §1 · pp. 1–2

  • Method

    BELT uses attention-output-motivated low-rank compression, including a key-side objective targeting attention distributions, and a learned allocator that selects a user-budgeted number of components.

    Source: §1 · p. 2

  • Results

    At 70% KV cache compression on Qwen3-32B, BELT improves average score over the strongest evaluated baseline by 23.3 points on LongBench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Under aggressive compression, BELT reports up to 3.26× faster decoding and 87% lower KV-cache memory at comparable accuracy.

    Source: §6 · p. 9

    The surrogate solutions exactly minimize their respective approximations, but the analysis provides no uniform guarantee over rank-constrained candidates and concerns fixed calibration inputs rather than final autoregressive generation preservation.

    Source: §3.1.3 · p. 5

Abstract

from arXiv · show

Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to 17.8x overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to 2.7x speedup for the attention module and 3.3x end-to-end decoding throughput.

1 Introduction

Long-context growth makes the KV cache a major inference bottleneck, while existing compression objectives can distort attention and rank allocation lacks a direct global-budget guarantee. BELT targets attention behavior with low-rank objectives and learns rank allocation under a user-specified budget; experiments report gains under aggressive compression.

1 Introduction

  • KV-cache memory grows linearly with sequence length and can exceed available GPU memory in long-context inference.The paper situates low-rank decomposition alongside token eviction and quantization as approaches to cache management.Source: §1 · p. 1
  • Existing reconstruction-based criteria do not explicitly model how key errors interact with queries and softmax, while pre-softmax logit preservation does not ensure attention-probability or output preservation.The introduction also identifies uniform rank allocation and learned thresholds without a prescribed global component budget as limitations of prior approaches.Source: §1 · pp. 1–2
  • BELT’s key-side objective targets attention-distribution preservation rather than raw-key reconstruction or pre-softmax logit reconstruction.It derives quadratic compression surrogates with exact weighted-SVD solutions and extends the construction to RoPE and GQA.Source: §1 · pp. 1–2
  • A learned global rank allocator uses differentiable soft top-B selection during training and hard selection of the user-specified B components at deployment.It distributes ranks across attention heads according to compression sensitivity while preserving an ordered prefix within each KV block.Source: §1 · pp. 2–3
  • At 70% KV cache compression on Qwen3-32B, BELT improves average score over the strongest evaluated baseline by 23.3 points on LongBench and 13.8 points on RULER.With 3-bit KV quantization, BELT reaches 17.8× KV cache compression within 2 points of uncompressed zero-shot accuracy.Source: §1 · p. 2

2 Preliminaries

The preliminaries define attention through query-dependent probabilities over cached keys and values, then describe low-rank compression and its inference path. Compression stores lower-dimensional cache representations, which are projected back for attention, with value-side weight absorption available to streamline computation.

2.1 Attention and KV Cache

  • For a decoder attention head, cached keys and values are interacted with a query to compute attention probabilities and the final attention output; RoPE applies a position-dependent embedding to queries and keys.The framework compresses per-head KV cache and is applicable to MHA and GQA.Source: §2.1 · p. 3

2.2 Low-Rank KV Cache Compression

  • Low-rank compression approximates each key and value projection matrix with rank-r factors and stores compressed representations, reducing per-token cache memory from d_h to r.The factors are learned using a small calibration dataset.Source: §2.2 · p. 3

2.3 Inference with Compressed Cache

  • During inference, compressed representations are projected back for attention, and value-side weight absorption can precompute the fixed output projection.A similar absorption does not apply to keys because the up-projection cannot be absorbed into RoPE or merged with the query projection.Source: §2.3 · pp. 3–4

3 The BELT Framework

BELT seeks low-rank projections that preserve the original model’s attention output, splitting the joint key/value objective into separate problems. It combines a weighted-SVD value solution, a quadratic surrogate for the nonlinear key objective, global rank allocation, and optional quantization.

3.1.1 Decomposition objective

  • BELT minimizes attention-output deviation by decoupling the joint key/value approximation into separate value- and key-side objectives.Source: §3.1.1 · p. 4

3.1.2 Solving LV (c

  • The value-side objective is transformed into a regularized surrogate with a closed-form weighted-SVD minimizer, a step closely related to A3 [22].Source: §3.1.2 · pp. 4–5

3.1.3 Solving LK(c

  • For keys, BELT approximates the nonlinear softmax objective with Taylor expansion and Kronecker decomposition, then obtains a closed-form surrogate minimizer.Source: §3.1.3 · p. 5
  • The key surrogate is empirically close to the original objective at returned low-rank solutions under evaluated settings, without a uniform guarantee across rank-constrained candidates.Source: §3.1.3 · p. 5

3.2 Learned Rank Allocation across heads

  • Across heterogeneous blocks, BELT learns global component scores: training enforces relaxed gate mass, while deployment retains exactly B components after reserving one per block.Source: §3.2 · p. 5

3.3 Integration with Quantization

  • BELT integrates with Palu and STAR-KV quantizers by fusing their orthogonal transforms into the low-rank factors offline, with no retraining required.Source: §3.3 · p. 6

4 Experiments

BELT is evaluated across model architectures, compression rates, and short- and long-context tasks, with system measurements and quantization tests. The results report quality preservation under aggressive compression alongside faster decoding and lower cache memory.

4.1 Experiment Settings

  • Evaluations cover four LLMs spanning MHA and GQA, RoPE and NoPE, with perplexity, zero-shot, and long-context benchmarks.The baselines include general low-rank methods and KV-specific methods; S2VD is excluded because its code is unavailable.Source: §4.1 · p. 6

4.2 Accuracy Analysis

  • BELT consistently achieves the best perplexity and zero-shot accuracy on two LLaMA models at 50% and 70% compression, with marginal degradation from uncompressed models.Several baselines degrade sharply or collapse at 70% compression.Source: §4.2 · p. 6
  • BELT achieves the best average long-context score at both compression rates, and at 50% surpasses uncompressed LLaMA-3.1-8B.Retrieval-intensive tasks are most compression-sensitive, while summarization is relatively insensitive.Source: §4.2 · p. 6, Figure 2 · p. 7
  • Fine-tuning improves all methods, but baselines still fall behind BELT; direct fine-tuning achieves substantially lower accuracy than BELT+FT.The authors suggest the difference may be attributable to a local minimum issue.Source: §4.2 · p. 6

4.3 System-level Performance Analysis

  • BELT consistently outperforms SDPA and the baselines in attention speedup, with its advantage growing at longer contexts; STAR-Q further boosts speedup.BELT at 70% compression surpasses all baselines at 50%, while ERC-SVD is slower than the uncompressed baseline.Source: §4.3 · p. 7, Figure 3 · p. 8
  • BELT with STAR-Q yields substantially larger decoding throughput gains than SDPA, while STAR-KV improves throughput only modestly.The throughput comparison uses BELT at 70% compression and baselines at 50%; without STAR-Q, BELT still achieves considerable speedups.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Quantization.

  • On Qwen3-32B, 3-bit Palu-Q with BELT reaches 17.8× cache compression at 69.36% average zero-shot accuracy.Palu-Q and STAR-Q with BELT at 70% compression reach 13.3× and 16.7× compression at 70.12% and 70.28% accuracy, respectively.Source: §4.4 · p. 8, Table 2 · p. 8

5 Related Work

Prior work compresses model projections or KV caches through low-rank decomposition, token selection, and quantization. BELT distinguishes itself by deriving low-rank weighting from attention behavior and allocating rank and bit-width under a shared memory budget.

Low-rank decomposition for general LLM compression

  • General LLM low-rank methods reconstruct layers or weights, whereas BELT derives its weighting from attention outputs and attention-distribution curvature.The cited general methods include SVD-LLM, ASVD [12], Zero Sum SVD [13], ERC-SVD [14], and S2VD.Source: §5 · p. 9

General KV-cache compression

  • Token selection reduces sequence length and quantization reduces precision, while low-rank compression reduces hidden dimension; BELT's joint allocator budgets rank and bit-width together.The cited token-selection methods are [6] [41] [42] [43], and quantization methods are [45] [7] [47].Source: §5 · p. 9

6 conclusion

BELT aligns low-rank KV compression with attention outputs and uses global rank allocation under a hard budget. Experiments report accuracy and efficiency gains under aggressive compression, including faster decoding and lower memory at comparable accuracy.

6 conclusion

  • BELT addresses reconstruction-focused objectives and rank allocation that collapses to uniform thresholds with attention-aligned compression and a global hard-budget allocator.The conclusion reports up to 3.26× faster decoding and 87% lower memory at comparable accuracy.Source: §6 · p. 9

A Derivation of LV (c WV ) and LK(c WK)

The appendix derives separate value- and key-side criteria from attention-output error, then develops regularized low-rank solutions and introduces the shared-query-head setting of grouped-query attention.

A Derivation of LV (c WV ) and LK(c WK)

  • The value criterion isolates output error from compressing values with original attention fixed, while the key criterion seeks a value-independent, distribution-preserving surrogate.The key-output term is initially coupled to the compressed value projection.Source: A Derivation of LV (c WV ) and LK(c WK) · p. 14

B.1.1 Proof of Lemma 1

  • Adding positive-definite regularization converts the value objective into a weighted Frobenius form with an O(ϵ) correction for fixed calibration statistics.The correction terms are nonnegative.Source: §B.1.1 · pp. 15–16

B.1.2 Proof of Theorem 1

  • The regularized rank-constrained value problem reduces to truncated SVD after an invertible change of variables, yielding a global minimizer.The change of variables preserves the matrix rank.Source: §B.1.2 · p. 16

B.2.1 Proof of Theorem 2

  • For keys, the nonlinear objective is approximated by a quadratic surrogate whose errors include Taylor, Kronecker-approximation, and regularization contributions.The Taylor remainder bound depends on the logit perturbation.Source: §B.2.1 · pp. 16, 19

B.2.2 Corollary for decomposing key tensor

  • The rank-r key approximation is obtained from the components associated with the largest singular values of the transformed key matrix.The result is a global minimizer of the stated surrogate.Source: §B.2.2 · p. 19

B.2.3 Overall algorithm for computing L≈

  • The key algorithm fits a rank-1 Kronecker approximation, then solves the regularized weighted low-rank problem by truncated SVD without forming the full quadratic matrix.Alternating least squares makes the fitting step scalable.Source: §B.2.3 · pp. 19–20

C Extension to Grouped-Query Attention

  • Grouped-query attention requires one shared pair of low-rank projections per KV group that accounts for the behavior of all query heads in that group.The derivations condition on calibration inputs and query vectors.Source: C Extension to Grouped-Query Attention · p. 20

C.1 Output-Error Decomposition

The grouped-query extension decomposes output error, aggregates shared-key behavior across heads, and constructs tractable shared-value and low-rank surrogates with stated bounds and cache accounting.

C.1 Output-Error Decomposition

  • The grouped-query output-error bound is conditional on a uniform row-norm bound, which a rank constraint alone does not guarantee.Without that condition, the stated conditional upper bound does not follow from rank alone.Source: §C.1 · p. 21

C.2 Shared-Key Compression

  • Shared-key compression aggregates the attention KL divergences of every associated query head before fitting one low-rank approximation for the shared projection.The Kronecker fitting and low-rank optimization are performed once per shared key projection.Source: §C.2 · p. 21

C.3 Shared-Value Compression

  • The exact shared-value metric includes cross-head interactions, so it is generally not a single Kronecker product and requires an approximation for weighted-SVD optimization.A headwise alternative that discards output cross terms also does not generally produce one exact two-sided weighted Frobenius objective.Source: §C.3 · p. 22

C.4 Common Kronecker Surrogate and Closed-Form Solution

  • Approximating the grouped metric by positive-semidefinite Kronecker factors yields a regularized surrogate with a global rank-r minimizer.The solution follows from an invertible rank-preserving transformation and truncated SVD.Source: §C.4 · pp. 22–23
  • The closed-form solution exactly minimizes the constructed surrogate, not necessarily the original grouped objective.The key construction additionally incurs the KL Taylor remainder.Source: §C.4 · p. 23

C.5 Multiple Groups and Cache Accounting

  • Independent optimization of KV groups need not minimize total layer-output error because the expansion also contains cross-group terms.Each group is compressed separately.Source: §C.5 · p. 23
  • A shared KV group stores one compressed key and value representation per token, counting their latent coordinates once regardless of query-head count.The global rank budget is imposed over KV groups.Source: §C.5 · p. 23

D.1 RoPE-Aware Attention and KV Cache

  • RoPE rotates queries and keys before the attention dot product, while disabling RoPE reduces the attention vector to the NoPE case.The relative rotation depends on query and key positions.Source: §D.1 · pp. 23–24

D.2 Extension of Lemma 1 and Theorem 1

The appendix extends the value and key constructions to RoPE and compares BELT’s key metric with covariance-weighted logit reconstruction. The comparison centers on softmax curvature and attention-weighted context statistics.

D.2 Extension of Lemma 1 and Theorem 1

  • With RoPE, the regularized fixed-attention value surrogate retains the weighted-SVD solution, but it does not minimize the joint key–value objective.The additional error from changing attention is accounted for separately by the key-output term.Source: §D.2 · pp. 24–25

D.3 Extension of Theorem 2

  • The RoPE-aware key analysis defines a second-order quadratic metric and bounds its approximation error, including nonnegative regularization contributions.The key construction also includes a KL Taylor remainder.Source: §D.3 · p. 25, Then the surrogate · pp. 26–27

Then the surrogate

  • The regularized RoPE key surrogate is minimized by transforming the key matrix, taking its top-r truncated SVD, and applying the inverse transformation.The transformation preserves rank.Source: Then the surrogate · p. 28

E Mathematical Comparison with A3

  • BELT and A3 both use analytical low-rank approximation, but differ in the key-compression objective: attention distributions for BELT and pre-softmax logits for A3 [22].The distinction is the approximation objective and induced metric, not the truncated-SVD operation.Source: E Mathematical Comparison with A3 · p. 28

E.1 Notation and the Pre-Softmax Attention Operator

  • The comparison represents pre-softmax logits as a bilinear attention operator and writes calibration logits as a matrix.The notation uses query and context representations.Source: §E.1 · p. 28

E.2 A3: Covariance-Weighted Logit Approximation

  • A3’s covariance weighting accounts for calibration token statistics, but its logit reconstruction error remains Euclidean and does not weight perturbations by their softmax effects.For a fixed query, the error sums squared logit perturbations.Source: §E.2 · p. 29

E.3 Expressing Logit Reconstruction in Key-Weight Space

  • With query-dependent causal supports, the key-weight metric is generally a sum of Kronecker products rather than one product.A single covariance-factor construction therefore corresponds to its particular calibration weighting and support treatment.Source: §E.3 · p. 29

E.4 BELT: Attention-Distribution Curvature

  • BELT weights key perturbations by query-dependent softmax curvature and uses an attention-weighted centered context covariance instead of the uncentered context second moment.The contrasted logit-reconstruction metric uses identity weighting on logit perturbations.Source: §E.4 · p. 30

E.5 What the Different Metrics Preserve

BELT’s key-side metric weights relative-logit changes by the original attention distribution, with RoPE-aware sensitivity and a tractable approximation. The section also describes budget-constrained rank allocation, quantization integration, and shared runtime kernels.

E.5 What the Different Metrics Preserve

  • Attention-distribution curvature ignores uniform logit shifts and weights relative-logit changes by the original attention probabilities, unlike Euclidean logit reconstruction.Locally low-cost directions are not necessarily harmless under large perturbations, so the Taylor remainder matters.Source: §E.5 · pp. 30–31

E.6 RoPE and Position-Dependent Key Sensitivity

  • RoPE makes key sensitivity depend on relative position, and BELT incorporates those rotations while retaining softmax-curvature weighting rather than identity weighting.The comparison concerns a RoPE-aware logit-reconstruction criterion, not an asserted A3 implementation.Source: §E.6 · p. 31

E.7 Tractability and the Scope of Optimality

  • BELT approximates its post-softmax distribution-preservation metric to obtain a tractable weighted-SVD solution, but does not establish a global minimum of the original attention KL.The approximation involves Taylor, Kronecker, and regularization errors; attention KL also does not directly encode value vectors or downstream network sensitivity.Source: §E.7 · p. 31

F.1 Budget-Constrained Gates and Their Gradients

  • Deployment selects exactly the prescribed component budget while preserving a prefix in each block; relaxed gate mass during training does not guarantee an optimal hard allocation.The first component of each block is reserved, and the remaining components are selected by score.Source: §F.1 · p. 33

F.2 Integration with Quantization

  • BELT’s compressed cache works with existing low-rank quantizers, whose rotations spread dominant coordinates before per-token quantization without adding inference computation.STAR-Q’s leading-coordinate quantization is compatible with BELT’s prefix-preserving allocation.Source: §F.2 · p. 33

G Additional experimental settings

  • System measurements use shared generalized Triton kernels for different layouts, rebuilding keys with RoPE and applying attention weights directly to compressed values.The latency setup uses random cache values matching each checkpoint’s layout and ranks.Source: G Additional experimental settings · p. 33

H.2 Additional ablation studies

The ablations test BELT without RoPE, isolate rank and cache-side effects, and vary calibration data. Additional analyses examine approximation bounds and STAR-KV threshold behavior.

H.2 Additional ablation studies

  • BELT on ALiBi-based MPT-7B has accuracy losses close to LLaMA-2-7B at both compression rates, supporting its use without RoPE.The evaluation uses 2048-token sequence lengths; the table reports perplexity on 2048-token windows.Source: §H.2 · p. 33, Table 4 · p. 34
  • Across WikiText-2, FineWeb-Edu, and C4 calibration corpora, accuracy varies by only 0.26 points and consistently exceeds the uncompressed model.Perplexity follows the calibration domain, while the learned rank allocation preserves its structure.Source: §H.2 · p. 34, Table 7 · p. 36
  • At 70% compression, value-only compression degrades accuracy and perplexity more than key-only compression, despite removing less cache; combined degradation roughly equals the sum of the individual losses.The experiment uses BELT checkpoints without retraining; its findings indicate values are more compression-sensitive and keys tolerate lower ranks.Source: §H.2 · p. 34, Table 5 · p. 35
  • With fixed bases and equal cache budgets, BELT’s learned allocation has the lowest perplexity and highest average accuracy at both tested compression rates.At 70% compression it improves average accuracy by over 10 points versus Static, while retaining 39–41% of ranks in keys; Static uses uniform fractions, and Layer-wise search is sensitivity-based.Source: §H.2 · p. 33, Table 6 · p. 35

I Why star-KV fails?

  • The analysis suggests STAR-KV’s learned thresholds become nearly identical across heads, regardless of their eigenvalue distributions.The empirical studies report that the relevant ρ value is small at convergence.Source: I Why star-KV fails? · pp. 35–36

H.3 Empirical verification for theoretical claims

  • The evaluated approximation-error terms are small, and the Kronecker-plus-Taylor error sum is close to the true gap between the surrogate and key-side objective.The assessment is reported as empirical support for Theorem 2’s analysis.Source: §H.3 · p. 35, Table 8 · p. 36

I.1 Proof of Theorem 3

The appendix characterizes threshold concentration at stationary points, gives a checkpoint-level bound away from stationarity, and specifies evaluation conditions. It also limits the interpretation of these results and situates BELT among related KV-compression methods.

I.1 Proof of Theorem 3

  • At finite stationary points, the exact identity holds; ρ < 1 is sufficient for the stated upper bound but is not necessary for all gradients to be positive.The empirical ρ range is reported as 0.5–0.72, with the associated bound described as small.Source: §I.1 · p. 37

I.2 A Measurable Bound Away from Stationarity

  • The checkpoint-level concentration bound applies when the mean signed distillation gradient is positive and ρ + ε < 1; outside that condition it gives no finite certificate, not proof of dispersed thresholds.At exact stationarity, ε = 0 and the bound recovers Theorem 3.Source: §I.2 · p. 37

I.3 Empirical Evaluation

  • Checkpoint statistics require gradients averaged before dispersion is computed, with averaging and weighting matched to the training objective; subset calibration data does not certify the full objective without estimation-error control.Relative statistics are inapplicable when the mean gradient is nonpositive and need care when it is near zero.Source: §I.3 · p. 38

I.4 Scope and Interpretation

  • Theorem 3 characterizes stationary points rather than convergence, and its identity does not establish that gradient balancing causes threshold concentration.The result also does not imply concentration independent of regularization strength or uniform ranks across different singular spectra.Source: §I.4 · p. 38

J Other related works

  • BELT weights low-rank approximation error by attention behavior and enforces a hard model-wide rank budget, unlike reconstruction-centered methods and independent threshold learning.The related-work comparison includes KV-specific methods such as LoRC [17], PALU, A3 [22], and STAR-KV.Source: J Other related works · p. 38

Related papers

Loading 2610.17966v1…