Source-linked AI summary
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, Ji-Rong Wen
TL;DR
KV caching accelerates LLM inference but incurs substantial memory costs, while existing compression can compromise precision or require calibration data. DecoQuant decomposes KV-cache matrices into local tensors for data-free low-bit quantization; it achieves better average scores than other methods and a 1.25x speedup with 4-bit KV cache for 6k-token output generation.
Problem
KV caching reduces redundant calculations but creates substantial memory and I/O costs, while existing compression can compromise precision or depend on calibration data.
Source: §1 · p. 1Method
Results
DecoQuant achieves better average scores than other methods; with 4-bit KV cache, it provides a 1.25x speedup when generating 6k output tokens.
Source: §3.3 · p. 5, §4.2 · p. 6Takeaways & Limitations
The method supports fully data-free compression, including nearly lossless 4-bit KV-cache quantization and 8-bit quantization of weights and activations.
Source: §1 · p. 2Performance may be influenced by hardware configurations, software dependencies, and environmental conditions, requiring further analysis for practical deployment.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Key-value~(KV) caching is an important technique to accelerate the inference of large language models~(LLMs), but incurs significant memory overhead. To compress the size of KV cache, existing methods often compromise precision or require extra data for calibration, limiting their practicality in LLM deployment. In this paper, we introduce \textbf{DecoQuant}, a novel data-free low-bit quantization technique based on tensor decomposition methods, to effectively compress KV cache. Our core idea is to adjust the outlier distribution of the original matrix by performing tensor decomposition, so that the quantization difficulties are migrated from the matrix to decomposed local tensors. Specially, we find that outliers mainly concentrate on small local tensors, while large tensors tend to have a narrower value range. Based on this finding, we propose to apply low-bit quantization to the large tensor, while maintaining high-precision representation for the small tensor. Furthermore, we utilize the proposed quantization method to compress the KV cache of LLMs to accelerate the inference and develop an efficient dequantization kernel tailored specifically for DecoQuant. Through extensive experiments, DecoQuant demonstrates remarkable efficiency gains, showcasing up to a $\sim$75\% reduction in memory footprint while maintaining comparable generation quality.
1 Introduction
KV caching speeds LLM inference but increases memory use, while existing compression approaches can lose information or rely on calibration data. DecoQuant uses matrix decomposition to make KV-cache quantization data-free and reduce outlier-related quantization error.
1 Introduction
- KV caches reduce redundant attention calculations but grow with long-content generation, increasing storage costs and I/O latency.These costs can become more severe when I/O spans multiple machines.Source: §1 · p. 1
- Token pruning can lose information, low-bit quantization can degrade performance through outliers, and calibration or training limits use in data-constrained settings.Source: §1 · p. 1
- DecoQuant decomposes activation matrices into local tensors, applying low-bit quantization to the larger tensor and retaining high precision for the smaller one.The paper also develops an efficient dequantization kernel tailored to this method.Source: §1 · p. 2
- DecoQuant achieves competitive performance with 4-bit KV-cache quantization and 8-bit quantization of both weights and activations.The approach is fully data-free, eliminating complex calibration mechanisms.Source: §1 · p. 2
2 Preliminary
The preliminary section describes KV caching as a way to avoid recomputing attention tensors, while noting that cache size grows with generation length. It also introduces quantization and MPO tensor decomposition, the building blocks for DecoQuant.
2 Preliminary
- KV caching stores previously seen attention keys and values to avoid recomputation, but its size grows linearly with generation length.The decoding phase dominates inference latency in long-text generation, making KV-cache memory compression important.Source: §2 · p. 2
- Quantization maps floating-point values to low-bit integers, but outliers make it difficult to choose a suitable quantization step size.Outliers are values that significantly deviate from the majority.Source: §2 · p. 2
- MPO tensor decomposition factorizes a matrix into a sequential product of local tensors.The method uses this decomposition to reorganize and aggregate information within tensors, helping distinguish outliers.Source: §2 · p. 3
- For a two-tensor decomposition, TL denotes the tensor with more parameters and TS the one with fewer parameters.The distinction provides an opportunity to separate outliers across local tensors.Source: §2 · p. 3
3 Methods
DecoQuant decomposes matrices to adjust outlier distributions, then quantizes the larger local tensor at low bit width while retaining higher precision for the smaller tensor. It applies this approach to KV-cache compression and uses fused dequantization kernels for inference.
3.1 DecoQuant: Matrix Quantization based on Decomposition
- MPO decomposition makes TL's value distribution narrower than the matrix's, so TL can use low-bit quantization while TS retains 16-bit precision.TL contains 99.4% of parameters and TS 0.6%, so higher precision for TS has a small overall cost.Source: §3.1 · pp. 3–4
3.2 Efficient Inference based on DecoQuant
- Prefill KV cache is decomposed offline; during decoding, DecoQuant runs after cache accumulation, and fused dequantization-GeMM kernels target data-movement latency.The example accumulation threshold is 1k.Source: §3.2 · p. 4
3.3 Discussion
- With 4-bit KV cache, DecoQuant reaches 1.25x speedup when generating 6k output tokens; compression ratio typically approximates B/16.Source: §3.3 · p. 5
- DecoQuant supports data-free quantization of weights, activations, or both, unlike methods such as GPTQ and AWQ that support only weights.LLM.int8() supports weights and activations together; RTN supports all three settings.Source: §3.3 · p. 5, Table 1 · p. 5
4 Experiments
Experiments evaluate DecoQuant on language modeling, long-context in-context learning, decomposition variants, and decoding efficiency. The reported results compare favorably with other quantization methods and show lower quantization error with MPO-based local-tensor quantization.
4.1 Experimental Setup
- Experiments cover LAMBADA, five in-context learning tasks, LLaMA and OPT models, and quantization from 2 to 16 bits.The setup also compares RTN and SmoothQuant; GPTQ and LLM.int8() are excluded because they cannot quantize output activations for KV-cache use.Source: §4.1 · p. 5
4.2 Main Results
- DecoQuant achieves better average scores than other methods on LAMBADA; 4-bit quantization often approaches 16-bit performance, while 2-bit performs worse but retains an advantage over other methods.All quantization methods reduce KV-cache size relative to FP16.Source: §4.2 · pp. 5–6, Table 2 · p. 6
4.3 Detailed Analysis
- Quantizing only the largest local tensor gives the lowest reconstruction error, followed by quantizing both local tensors; matrix quantization has the largest error.The comparison supports substituting local-tensor quantization for matrix quantization to mitigate activation quantization error.Source: Figure 4 · p. 6, §4.3 · pp. 6–7
- Longer MPO decompositions improve quantization, but gains diminish as length increases; the experiments use n = 2 for efficiency and recommend higher n for higher accuracy.The improvement from n = 2 to n = 3 exceeds that from n = 3 to n = 4.Source: Figure 4 · p. 6, §4.3 · p. 7
4.2 Main Results
- Across five in-context learning datasets, RTN trails DecoQuant on average and deteriorates at 10-shot despite comparable 2-shot performance.The evaluation varies the number of demonstrations.Source: §4.2 · p. 6, Table 3 · p. 7
4.3 Detailed Analysis
- At 4-bit precision, MPO has quantization error 40.9 versus 105.4 for SVD and 103.7 for QR.MPO also offers flexible tensor shapes, unlike QR and SVD, allowing quantization granularity to balance accuracy and performance.Source: Figure 5 · p. 7, §4.3 · p. 7
4.4 Analysis of the Efficiency
- For LLaMA 70B at sequence lengths of 1k–8k, DecoQuant reduces KV-cache memory and achieves lower decoding latency despite added computation.At 6k tokens, the uncompressed cache matches model size while the compressed cache remains under 30GB; saved communication overhead outweighs DecoQuant’s extra computation.Source: §4.4 · pp. 7–8, Figure 6 · p. 8
5 Related Work
Prior work applies tensor decomposition to model compression, representation, fine-tuning, and pretraining, but its role in parameter quantization has received less study. Related KV-cache approaches include quantization, token reduction, and soft-prompt compression, each with different constraints.
Tensor Decomposition for Language Models
- Tensor decomposition has been used for compression [Gao et al., 2020; Sun et al., 2020], fine-tuning [Gao et al., 2023; Liu et al., 2021], and pretraining [Gao et al., 2022; Liu et al., 2023a], while its role in parameter quantization remains less studied.The paper presents its contribution as addressing that gap.Source: §5 · p. 8
Quantization for LLMs
- Activation quantization is challenging because of outliers; methods may retain FP16 outliers [Dettmers et al., 2022], and many quantization approaches require calibration data that can be inaccessible in some applications.The paper addresses activation quantization without calibration.Source: §5 · p. 8
KV Cache Compression
- KV-cache compression approaches include quantization, token reduction such as H2O, and soft-prompt compression such as AutoCompressors [Chevalier et al., 2023]; DecoQuant keeps all tokens.The related-work discussion notes that removing tokens based only on existing context may omit tokens important to the future.Source: §5 · p. 8
6 Conclusion
The conclusion presents DecoQuant as a data-free KV-cache compression method that applies low-bit quantization to the large decomposed tensor while keeping the small tensor at 16-bit precision. It also describes a fused dequantization kernel and identifies communication-dominant inference as future work.
6 Conclusion
- DecoQuant quantizes the large local tensor at low bit width and retains 16-bit precision for the small tensor to reduce KV-cache quantization error.The conclusion attributes this to shifting quantization difficulty from the original matrix to the small local tensor.Source: §6 · p. 8
- A fused dequantization technique tailored to DecoQuant is developed to accelerate generation.Source: §6 · p. 8
- Future work will explore DecoQuant where communication overhead dominates inference, including Splitwise configurations that separate prefilling and decoding across nodes.Source: §6 · p. 9
7 Limitations
The limitations section notes that performance may depend on hardware, software, and environmental conditions, and calls for further analysis of their effects. It also flags potential bias and fairness concerns as deployment expands to edge devices.
7 Limitations
- Performance may be influenced by hardware configurations, software dependencies, and environmental conditions.Source: §7 · p. 9
- The authors say these external factors need thorough analysis to assess their impact on practical deployment and real-world applications.Source: §7 · p. 9
- Potential deployment on edge devices, including personal smartphones, may raise social concerns about bias and fairness.Source: §7 · p. 9
A.1 Analysis of Outliers
The outlier analysis compares interquartile ranges across tensor structures and layers. It reports narrower ranges for decomposed tensors than for the original matrix, with TL narrowest, followed by TS.
A.1 Analysis of Outliers
- The analysis examines IQRs in the 1st, 16th, and 31st layers and across both keys and values.Source: §A.1 · p. 10
- IQR is the range between the 25th and 75th percentiles, and outliers are defined as values beyond 1.5 times the IQR from the quartiles.Source: §A.1 · p. 10
- The IQR range is narrowest for TL, followed by TS, and both decomposed tensors have smaller numerical ranges than the original matrix.The authors state that this indicates the method can be applied to all key/value tensors.Source: §A.1 · p. 10
A.2 Details of the datasets
The dataset description uses tokens per demonstration to represent KV-cache length in in-context learning experiments. The selected datasets cover shorter, moderate, and longer contexts.
A.2 Details of the datasets
- In the in-context learning experiments, KV-cache length is represented by tokens per demonstration because demonstrations make up most of prefilling.The dataset table describes tokens per demonstration as the average example length.Source: §A.2 · p. 10, Table 5 · p. 11
- Table 5 gives dataset details, with tokens per demonstration indicating the average example length.Source: Table 5 · p. 11
- Five datasets span longer contexts (Boolq), shorter contexts (Mr and Subj), and moderate contexts (Ag_news and RTE).Source: §A.2 · p. 11