Source-linked AI summary
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
TL;DR
Existing post-training quantization methods perform well at 8 bits but degrade at lower precision, motivating quantization-aware training for LLMs. LLM-QAT uses data-free distillation from pretrained-model generations and quantizes weights, activations, and KV caches, achieving accurate 4-bit models with improvements over post-training methods in low-bit settings.
Problem
Post-training quantization methods degrade below 8 bits, while LLM QAT requires difficult-to-obtain training data and must address the memory cost of weights and KV caches.
Method
LLM-QAT generates data from the pretrained model and uses teacher predictions as labels for quantized-model distillation, including weights, activations, and KV-cache quantization.
Results
LLM-QAT holds up better than post-training quantization below 8 bits; in the 8-8-4 setting, 30B LLM-QAT reaches 69.7 average zero-shot accuracy versus 50.7 with SmoothQuant.
Takeaways & Limitations
The method demonstrates accurate 4-bit quantized LLMs and supports simultaneous KV-cache quantization, addressing memory and long-sequence throughput constraints.
Takeaways & Limitations
4-bit activation quantization did not work well in the tested settings, and 4-bit quantization lacked out-of-the-box hardware support.
Abstract
from arXiv · showhide
Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.
1 Introduction
LLM deployment is constrained by the computational and memory costs of large models, while existing quantization methods degrade below 8 bits. LLM-QAT addresses this with data-free distillation and quantizes weights, activations, and KV caches to enable lower-bit models.
- Large language models improve capabilities with scale but impose substantial computational costs and environmental burdens during deployment.
- A 65B LLaMA requires 65GB for weights alone, while KV caches can consume tens of GBs and bottleneck long-sequence throughput.
- 8-bit post-training quantization preserves accuracy, but state-of-the-art methods dramatically degrade when pushed below 8 bits.
- LLM-QAT uses generated data from the pretrained model for knowledge distillation, avoiding dependence on the original training data.
- The method distills 7B, 13B, and 30B LLaMA models with weights and KV caches quantized to 4 bits, while activations reach 6-bit precision.
- The work presents QAT for LLMs and demonstrates accurate 4-bit quantization while simultaneously quantizing weights, activations, and KV caches.
2 Method
LLM-QAT addresses two challenges in quantizing LLMs: preserving broad zero-shot capability without original training data and handling their outlier-rich distributions. It uses data-free next-token generation, selected quantizers for weights, activations, and KV cache, and logits distillation to train a quantized student.
- Data-free distillation: Data-free QAT generates training data from the pre-trained model to better synthesize the pre-training distribution than narrow fine-tuning datasets.The method iteratively generates next tokens, using stochastic sampling from the teacher’s SoftMax distribution to improve sentence diversity and student accuracy.
- Data-free distillation: Stochastic next-token sampling provides diverse sequences, while teacher predictions serve as soft labels because sampled tokens contain inherent noise.The student is trained with cross-entropy-based logits distillation from the full-precision teacher.
- Quantization preliminaries: Linear quantization is presented as either MinMax, which preserves all ranges, or clipping-based, which removes outliers to allocate more bits to intermediate values.The scale and zero-point may be calculated statistically or learned through gradients.
- KV-cache quantization: The method applies per-token quantization to both key and value tensors in the KV cache, which is updated by appending each current key and value.KV-cache quantization extends the QAT approach used for activations.
- Quantizer design: LLM-QAT uses symmetric MinMax quantization with per-token activations and per-channel weights because clipping outliers harms LLM performance.The method quantizes weights and input activations in fully connected linear layers.
3 Experiments
Experiments evaluate LLM-QAT against post-training quantization across model sizes, quantization settings, perplexity, and downstream tasks. Results show stronger low-bit robustness, with generated data, preserved outliers, and logit distillation improving performance under specific conditions.
- 3.2 Main Results: Experiments compare LLM-QAT with RTN, GPT-Q, and SmoothQuant across weight, activation, and KV-cache bit settings on LLaMA-7B/13B/30B.Evaluation covers zero-shot Common Sense Reasoning, WikiText2 and C4 perplexity, and few-shot TriviaQA and MMLU.
- 3.2 Main Results: 69.7 average zero-shot accuracy is achieved by 30B LLM-QAT in the 8-8-4 setting, versus 50.7 with SmoothQuant.In the 4-8-4 setting, LLM-QAT achieves 69.9, trailing full precision by 1.5 points on average.
- 3.2 Main Results: 4-bit LLM-QAT models are recommended for the best efficiency-accuracy tradeoff, including 4-8-4 30B outperforming an 8-bit LLaMA-13B.The authors also report that 8-bit quantization should be preferred over smaller full-precision models, with PTQ sufficient at that setting.
- 3.3.1 Data Choice: Generated data generalizes better than existing data, particularly on zero-shot tasks, while sampled generations improve performance across all tasks through greater diversity.WikiText can overfit its domain, whereas C4 transfers reasonably to WikiText but performs poorly on zero-shot inference.
- 3.3.2 Quantization Function: Clipping-based StatsQ and LSQ do not surpass MinMax quantization, reinforcing that preserving outliers is critical for large-language-model performance.For LLaMA, predominantly symmetric weight and activation distributions favor symmetric quantizers, though this may not generalize to models with GeLU layers.
- 3.3.3 Knowledge Distillation: Logit distillation outperforms next-token label training, whose sampled labels introduce randomness and noise.LLM-QAT also benefits from SmoothQuant for W4A4, but SmoothQuant does not improve and may harm W4A8 performance.
4 Related Works
Prior language-model compression work largely focused on quantization and data-generation strategies outside the full LLM-QAT setting. This paper introduces next-token data generation for data-free quantization-aware training of quantized language models.
- Recent language-model compression methods have primarily explored quantization, while classic approaches were developed mainly for convolutional neural networks.
- Data generation for quantization-aware training remains relatively unexplored compared with established vision-domain approaches.
- Vision methods fine-tune student networks using teacher-generated images by updating noise inputs from label gradients and accumulating reconstructed images.
- The proposed language-domain approach generates next-token data, providing a more natural and effective way to fine-tune quantized language models.
5 Conclusion and Limitations
The paper concludes that data-free quantization-aware training enables accurate 4-bit LLM quantization, while identifying hardware support and 4-bit activation quantization as unresolved boundaries.
- Accurate 4-bit quantization of LLMs is possible with the proposed data-free quantization-aware training technique.
- The training-data-agnostic distillation method is presented as broadly applicable, including potentially to models trained with instruction tuning or reinforcement learning.
- The work excludes hardware implementation because 4-bit quantization lacks out-of-the-box hardware support.
- The method supports 4-bit weights, 4-bit KV cache, and 8-bit activations, but not sufficient 4-bit activation quantization.
A.1 Few-shot Evaluation Results
The appendix reports few-shot evaluation settings for quantized models on TriviaQA and MMLU, while the supplied passages do not provide performance values or a supported KV-cache comparison.
- Table 7 reports 5-shot few-shot exact-match performance on TriviaQA and 5-shot accuracy on MMLU.
- The supplied table description identifies benchmark metrics but does not state the quantized models’ numerical results.
A.2 Memory consumption of KV cache
KV-cache memory consumption is evaluated across sequence lengths because the cache can exceed model size for long contexts, making compression important for such applications.
- KV-cache memory is computed for various sequence lengths to assess its storage growth in large language models.
- For longer sequences, KV-cache size can quickly exceed model size, motivating KV-cache compression as long-context applications expand.
A.3.1 Zero-shot Common Sense Reasoning tasks
This section covers zero-shot common-sense reasoning benchmarks spanning reading comprehension, physical and social understanding, pronoun resolution, science questions, and open-book question answering.
- BoolQ evaluates naturally occurring yes/no reading-comprehension questions using passages and answers with explanations.
- PIQA measures physical commonsense understanding in natural language.
- SIQA measures computational models’ social and emotional intelligence through multiple-choice question answering.
- HellaSwag tests physically situated commonsense inference with four-way multiple-choice problems that challenge language models.
- WinoGrande evaluates commonsense reasoning through 273 expert-crafted pronoun-resolution problems.
- ARC contains 7,787 natural science questions divided into Challenge and Easy sets, with the Challenge Set answered incorrectly by two baseline algorithms.
- OBQA evaluates open-book question answering by requiring combination of provided science facts with external broad common knowledge.
A.3.2 Few-shot Tasks
This section covers few-shot evaluation through closed-book question answering and broad multitask knowledge assessment, alongside language-modeling corpora used for perplexity measurement.
- TriviaQA is a closed-book question-answering benchmark with over 650K question-answer evidence triples from 95K authored question-answer pairs.
- MMLU measures multitask language understanding with multiple-choice questions across humanities, STEM, and social sciences.
- WikiText2 contains over 100 million tokens extracted from verified Good and Featured Wikipedia articles.
- C4 is a collection of clean, natural English text produced by filtering web-extracted Common Crawl content.