PaperlayerView exact revision on arXiv

Source-linked AI summary

E8 Lattice Key Quantization for Low-Bit KV-Cache Compression

Sasho Nedelkoski, Alexander Acker, Soeren Becker, Dominik Scheinert, Odej Kao

arXiv:2610.15384v1cs.LGcs.AI

TL;DR

Long-context LLM serving must store KV pairs for every context token, while scalar key quantization faces a quality trade-off at low bitrates. This paper applies E8 lattice vector quantization to 2-bit keys and table-based scoring; across five LLM families it improves the quality–memory trade-off over scalar k3v2 in most long-context perplexity settings without measurable decode-throughput penalty.

  • Problem

    KV-cache storage creates memory pressure for long-context LLM serving, while scalar quantization can degrade quality at very low bitrates and commonly uses 3-bit keys to preserve model quality.

    Source: §1 · pp. 1–2

  • Method

    The method encodes keys in 8-dimensional E8 lattice blocks and scores compressed keys through precomputed query-to-codeword tables rather than explicit key dequantization.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across five LLM families, E8 k2v2 improves the quality–memory trade-off over scalar k3v2 in most long-context perplexity settings, with no measurable decode-throughput penalty.

    Source: §1 · p. 2

  • Takeaways & Limitations

    E8 k2v2 achieves ~6× total KV compression relative to fp16, while the conclusion reports preserved downstream quality and single-sequence decode-throughput parity.

    Source: §1 · p. 2, §5 · p. 10

    The framework assumes head dimension d is divisible by 8 and generalizes only to dimensions satisfying that constraint.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scalar quantization methods achieve significant compression, they typically face a quality-fidelity trade-off, either suffering from performance instability at 2-bit precision or necessitating 3-bit keys to maintain model quality. In this work, we demonstrate that a uniform low-bit k2v2 configuration (2-bit keys and 2-bit values) can preserve near-baseline quality by using vector quantization (VQ) based on the $E_8$ lattice for KV keys. We show that structured VQ exploits correlations across adjacent dimensions to achieve signal-to-quantization-noise ratios (SQNR) higher than those of independent scalar quantization at the same key payload rate. To ensure practical utility, we adapt a table-based asymmetric scoring algorithm that avoids explicit codebook dequantization during attention computation and preserves decode-throughput parity with scalar baselines. Across five LLM families, our $E_8$ k2v2 approach yields \textasciitilde$6\times$ KV compression, roughly 20\% smaller than scalar k3v2 (3-bit keys and 2-bit values), while retaining near-baseline long-context perplexity. In our full-suite perplexity evaluation, $E_8$ improves over scalar k3v2 in 80\% of model--dataset--context settings. These results indicate that $E_8$ lattice VQ improves quality per byte while keeping throughput at parity, suggesting that structured key quantization is a useful primitive for KV-compression stacks. Overall, our work provides a low-overhead primitive for expanding the quality-memory frontier in long-context LLM inference.

1 Introduction

Long-context LLM inference stores KV pairs for every token, creating a memory bottleneck that can limit context length and batch size. This work uses E8 lattice vector quantization for 2-bit keys, with table-based scoring to preserve decode throughput while improving the quality–memory trade-off.

1 Introduction

  • KV-cache storage across a long context can consume tens of gigabytes and constrain batch size, context length, and deployment cost.At 128K tokens, even an 8B grouped-query model requires tens of gigabytes of KV cache.Source: §1 · pp. 1–2
  • The method partitions keys into 8-dimensional E8 lattice blocks to exploit inter-dimensional correlations that independent scalar quantization ignores.The E8 lattice's near-spherical Voronoi cells provide a shaping advantage over cubic scalar quantizers.Source: §1 · p. 2
  • At 2 bits/dim payload, E8 k2v2 achieves ~6× total KV compression relative to fp16, with 2.25 bits/dim effective key rate including norm metadata.Reported memory counts end-to-end KV-cache bytes per token, including indices/signs, norms, and value metadata.Source: §1 · p. 2
  • On 128-dimensional Gaussian vectors, E8 obtains 10.47 dB SQNR versus 7.37 dB for scalar 3-bit quantization, and attention KL divergence is 0.045 versus 0.091 for scalar k3v2 and 0.224 for KIVI k2v2.The attention-fidelity probe compares compressed attention distributions with fp16.Source: §1 · p. 2
  • Table-based asymmetric scoring precomputes query-block dot products with all 65,536 E8 codewords, then scores compressed keys by codeword-id lookup without explicit key dequantization.The measured serving path matches the throughput of optimized scalar decoders.Source: §1 · p. 2
  • Across five LLM families, E8 k2v2 improves the quality–memory trade-off over scalar k3v2 in most long-context perplexity settings, with lower perplexity than KIVI k2v2 and no measurable decode-throughput penalty.The evaluation spans Qwen2.5, Llama-3.1, Mistral, Falcon3, and Granite.Source: §1 · p. 2

2 Related work

Prior KV-cache compression primarily uses scalar or projection-based coding, while vector-quantization work motivates joint coding of key dimensions. This paper uses a fixed E8 codebook and compressed-domain table scoring, distinct from approaches that reduce the number of cached tokens.

2 Related work

  • KV-cache quantizers such as KIVI [Liu et al., 2024], KVQuant [Hooper et al., 2024], and TurboQuant [Zandieh et al., 2025] primarily use scalar quantization, while GEAR combines low-rank approximation with quantized residuals.TurboQuant k3v2 is the paper's primary scalar baseline.Source: §2 · p. 3
  • Coupled Quantization [Zhang et al., 2024], PM-KVQ [Liu et al., 2025], NestQuant [Savkin et al., 2025], and CommVQ [Li et al., 2025b] motivate joint coding or vector codebooks, whereas this method uses a fixed, non-learned E8 codebook with online encoding and table scoring.KV-cache activations are produced online and attention scores must be computed against a growing cache.Source: §2 · p. 3
  • Token-eviction systems such as StreamingLLM [Xiao et al., 2024], H2O, and SnapKV [Li et al., 2024] reduce cache length rather than bits per token, making them orthogonal to E8 coding.The paper notes E8 coding could be applied to retained tokens or a projected subspace.Source: §2 · p. 3
  • Lattice quantization theory [Zamir and Feder, 1996, Conway and Sloane, 1999] motivates structured codes through space-filling gain, and E8's eight-dimensional sphere packing and near-spherical Voronoi cells motivate its use.The paper states that, to its knowledge, this is the first application of E8 coding to runtime KV caches with online encoding and throughput-neutral compressed-domain attention scoring.Source: §2 · p. 3

3 Method

The method pairs an E8 vector-quantized key representation with table-based attention scoring, integrated into TurboQuant. It compares effective cache size using end-to-end accounting that includes quantization metadata.

3.1 Preliminary baseline

  • The scalar reference quantizes each key dimension independently, while the implemented TurboQuant k3v2 baseline uses rotated unit-norm keys, Lloyd-Max quantization, and QJL residual signs.Compression comparisons use effective cache bytes per token, including dequantization and compressed-domain scoring metadata.Source: §3.1 · pp. 3–4

3.2 E8 lattice vector quantization for keys

  • E8 encoding rotates each key with a shared orthogonal matrix, partitions it into 8D blocks, and stores each block's codeword index plus a vector norm at 2.25 bits/dim effective key rate.For d = 128, the stored key totals 288 bits: sixteen 16-bit indices plus a 32-bit norm.Source: §3.2 · p. 4
  • ~6× end-to-end KV-cache compression over fp16 is reported for E8 k2v2 under accounting that includes packed keys, values, and per-token metadata.For d = 128, the E8 key payload is 288 bits and the value payload is 256 bits before value metadata.Source: §3.2 · p. 4

3.3 Table-based asymmetric scoring

  • Table-based asymmetric scoring rotates and partitions each query, builds query-to-codeword lookup tables, and scores cached keys by gathering entries for their stored indices instead of dequantizing them.Batched quality evaluations use an equivalent dense reconstruction, while decode-throughput measurements exercise the table-based serving path.Source: §3.3 · pp. 4–5

3.4 Implementation details

  • The TurboQuant integration retains its value quantization and recent-token residual window, while the E8 key path omits QJL residual signs and uses a seeded per-layer rotation.Structured nearest-codeword searches use reduced absolute-value candidates and sign/parity-consistent index reconstruction with a ±0.25 branch selection.Source: §3.4 · p. 5

4 Experiments

The experiments compare E8 k2v2 with fp16, KIVI k2v2, and scalar k3v2 across five instruction-tuned models. They assess rate-distortion, long-context quality, downstream accuracy, throughput, and memory.

4.1 Setup

  • The evaluation spans Qwen2.5-14B, Mistral-7B, Llama-3.1-8B, Falcon3-7B, and Granite-3.1-8B, all using grouped-query attention.The models have head dimensions divisible by 8 and varying numbers of KV heads.Source: §4.1 · p. 5
  • The four tested configurations are fp16, KIVI k2v2, scalar k3v2, and E8 Lattice k2v2.Compressed configurations use 2-bit values and a 128-token recent-token residual buffer unless varied explicitly.Source: §4.1 · p. 5
  • Evaluation covers long-context perplexity, few-shot accuracy, retrieval, representation and attention fidelity, decode throughput, and memory consumption.Perplexity uses WikiText-2 and C4 at contexts 512, 2048, and 4096; throughput is measured for single-sequence decoding on one 80 GB A100.Source: §4.1 · p. 6
  • On 128-dimensional Gaussian vectors, E8 lattice VQ achieves SQNR between scalar 3-bit and 4-bit quantization at a comparable low key rate.Figure 1 presents the rate-distortion comparison.Source: Figure 1 · p. 6

4.2 Rate-distortion analysis

E8 improves the reported rate-distortion performance over scalar quantization at low key rates. Across the perplexity suite, it outperforms scalar k3v2 in most settings, with Falcon3-7B the exception.

4.2 Rate-distortion analysis

  • At 2 bits/dim, E8 reaches 10.47 dB SQNR, versus 7.37 dB for scalar 3-bit, despite a lower effective key rate.The reported E8 result falls between the implemented scalar 3-bit and 4-bit configurations; it is not an optimal scalar-quantizer lower bound.Source: §4.2 · p. 6

4.3 Perplexity

  • E8 k2v2 improves over scalar k3v2 in 24/30 model–dataset–context blocks, including every non-Falcon model and both corpora.Scalar k3v2 is best in all six Falcon rows, but E8 remains below KIVI in every Falcon row. At 4K context, Granite-3.1-8B has WikiText-2/C4 deltas of +0.30/+0.50 for E8 versus +0.69/+0.94 for scalar.Source: §4.3 · pp. 6–7

4.4 Memory compression

E8 k2v2 reduces the quantized KV-cache footprint relative to scalar k3v2. The reported compression and memory savings use end-to-end per-token accounting.

4.4 Memory compression

  • For the four Hkv = 8 models, E8 k2v2 achieves ∼6× effective compression versus ∼4.8× for scalar k3v2, making the compressed cache roughly 20% smaller.Falcon3-7B differs: its larger head dimension amortizes per-vector metadata, yielding 6.2× for E8 and 5.1× for scalar.Source: §4.4 · p. 7
  • For Llama-3.1-8B, E8 k2v2 uses 21.7 KB/token versus 27.3 KB/token for scalar k3v2, saving 5.6 KB per token.At 128K context, the accounting projects roughly 700 MB of additional memory saved per sequence.Source: §4.4 · p. 7

4.5 Decode throughput

Table-based asymmetric scoring lets E8 k2v2 match scalar decode throughput in the measured setting. The downstream evaluation reports parity with baselines, while residual-window size affects compressed-task performance.

4.5 Decode throughput

  • E8 k2v2 achieves decode-throughput parity with fp16, scalar k3v2, and KIVI kernels across the evaluated architectures and context lengths.The measurements are for single-sequence decoding on an A100; variance is within ±0.2 tok/s, and the table-based scoring avoids explicit key dequantization.Source: §4.5 · p. 7, Table 3 · p. 8

4.6 Downstream task accuracy

  • Across models, E8 k2v2 maintains parity with fp16 and scalar baselines on HellaSwag, ARC-Challenge, and WinoGrande accuracy.The few-shot demonstrations are largely quantized, while the recent-token residual window protects final local context.Source: §4.6 · pp. 7–8, Table 4 · p. 8
  • With the residual window removed (W=0), performance ordering reappears; as it reaches 128 tokens, compressed configurations rapidly converge to fp16.Larger windows remain practically indistinguishable from the fp16 baseline.Source: §4.6 · p. 8

4.7 Attention fidelity

E8 key compression preserves attention distributions more faithfully than the reported scalar baselines. Its per-layer cosine similarity and attention KL results show this advantage across model depth.

4.7 Attention fidelity

  • E8 at 2 bits/dim has average per-layer cosine similarity of roughly 0.954, compared with roughly 0.921 for scalar 3-bit keys.The narrow, smooth shaded range suggests lattice quantization error is not concentrated in a small subset of layers.Source: §4.7 · p. 8
  • E8 achieves attention KL divergence of 0.045, versus 0.091 for the 3-bit scalar baseline and greater than 0.200 for KIVI k2v2.The reported per-layer KL range is stable and low.Source: §4.7 · p. 8

4.8 Long-context retrieval robustness

E8 k2v2 retains single-needle retrieval accuracy across the evaluated context lengths and models. A residual-window sweep also shows that method differences diminish as the uncompressed window grows.

4.8 Long-context retrieval robustness

  • The retrieval test inserts a randomly placed fact into distractor contexts of 4K, 8K, and 16K tokens across five models.Source: §4.8 · p. 8
  • At residual window W = 128, compressed methods are nearly indistinguishable in average downstream accuracy, while differences are visible at W = 0.Source: Figure 2 · p. 9
  • E8 k2v2 achieves 100% retrieval accuracy across 4K, 8K, and 16K contexts in all five models, matching fp16, KIVI, and scalar methods.Source: §4.8 · pp. 8–9

4.9 Ablation studies

The ablations identify a stable intermediate E8 scale range and find no average-cosine benefit from changing precision in the first few layers. The reported initial-layer sweep is an implementation-level check, not evidence that extra early-layer capacity never helps.

4.9 Ablation studies

  • E8 scale calibration has a stable plateau from α = 8 to α = 15, while α ≤ 4 clips strongly and α ≥ 20 wastes codebook resolution.MSE and cosine similarity remain stable across the intermediate plateau.Source: §4.9 · p. 9
  • Mixed-precision variants that change the first few layers do not improve average cosine over uniform E8 k2, so the main configuration keeps a uniform key format.Source: §4.9 · pp. 9–10
  • The initial-layer precision sweep is only an implementation-level check because the E8 3-bit variant adds a QJL residual correction.It is not evidence that early layers never benefit from additional capacity.Source: §4.9 · p. 10

5 Conclusion

The paper presents E8 lattice VQ as a practical low-bit KV-key compression primitive, using table-based asymmetric scoring to avoid explicit key dequantization. It reports improved key fidelity and quality per byte with preserved downstream quality and single-sequence decode-throughput parity, while identifying larger-batch and prefill-heavy scoring as unresolved systems challenges.

5 Conclusion

  • E8 lattice VQ uses a fixed geometry-aware codebook and table-based asymmetric scoring to score cached keys from indices without explicit key dequantization.Source: §5 · p. 10
  • Across diverse LLM families and long-context evaluations, the design improves key fidelity and quality per byte while preserving downstream quality and single-sequence decode-throughput parity.Source: §5 · p. 10
  • Efficient compressed-domain scoring beyond the evaluated setting, especially for larger batches and prefill-heavy workloads, remains an important systems challenge.Source: §5 · p. 10

Related papers

Loading 2610.15384v1…