PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Ping Luo

arXiv:2407.11062v3cs.LGcs.AIcs.CL

TL;DR

LLMs require substantial memory, while QAT’s resource demands limit its practicality despite reducing quantization error. EfficientQAT combines block-wise full-parameter training with end-to-end quantization-parameter training, achieving strong low-bit performance across models and settings. For example, it produces a 2-bit Llama-2-70B model in 41 hours on one A100-80GB GPU with accuracy 69.48 versus 72.41 for full precision.

  • Problem

    LLMs’ extensive parameters create memory challenges, while QAT requires substantial training time, GPUs, and data despite reducing quantization error.

  • Method

    EfficientQAT uses Block-AP to train all parameters block-wise, followed by E2E-QP to train quantization parameters end-to-end.

  • Results

    EfficientQAT outperforms existing PTQ, QAT, and Q-PEFT methods across models and quantization levels, including a 2-bit Llama-2-70B model scoring 69.48 versus 72.41 for full precision.

  • Takeaways & Limitations

    EfficientQAT provides a memory- and training-efficient approach for applying low-bit quantization across base, instruction-tuned, and multimodal LLM scenarios.

  • Takeaways & Limitations

    A performance gap remains versus FP16, especially at 2 bits, and effective training depends on 4096 high-quality, diverse samples.

Abstract

from arXiv · show

Large language models (LLMs) are crucial in modern natural language processing and artificial intelligence. However, they face challenges in managing their significant memory requirements. Although quantization-aware training (QAT) offers a solution by reducing memory consumption through low-bit representations with minimal accuracy loss, it is impractical due to substantial training resources. To address this, we propose Efficient Quantization-Aware Training (EfficientQAT), a more feasible QAT algorithm. EfficientQAT involves two consecutive phases: Block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP). To the best of our knowledge, Block-AP is the first method to enable direct training of all parameters in a block-wise manner, reducing accuracy loss in low-bit scenarios by enhancing the solution space during optimization. E2E-QP then trains only the quantization parameters (step sizes) end-to-end, further improving the performance of quantized models by considering interactions among all sub-modules. Extensive experiments demonstrate that EfficientQAT outperforms previous quantization methods across a range of models, including base LLMs, instruction-tuned LLMs, and multimodal LLMs, with scales from 7B to 70B parameters at various quantization bits. For instance, EfficientQAT obtains a 2-bit Llama-2-70B model on a single A100-80GB GPU in 41 hours, with less than 3 points accuracy degradation compared to the full precision (69.48 vs. 72.41). Code is available at https://github.com/OpenGVLab/EfficientQAT.

1 Introduction

LLMs offer broad capabilities but create substantial memory and training-cost challenges. EfficientQAT addresses QAT’s resource demands through block-wise full-parameter training followed by end-to-end quantization-parameter training, achieving strong low-bit results.

  • LLMs’ extensive parameters create major memory-footprint and bandwidth challenges.
  • QAT reduces quantization error through quantization constraints, but its fully trainable, end-to-end design requires substantial time, GPUs, and training data.
  • Existing quantization methods improve efficiency by restricting trainable parameters or using fine-grained reconstruction, but can significantly degrade low-bit accuracy.
  • EfficientQAT combines Block-AP, which trains all parameters within each transformer block, with E2E-QP, which trains quantization parameters end-to-end.
  • 41 hours on one A100-80GB GPU produced a 2-bit Llama-2-70B model with accuracy 69.48 versus 72.41 for full precision.

2 Related Works

Prior work spans post-training quantization, QAT, and quantized parameter-efficient fine-tuning. These approaches reduce memory or training costs but involve different trade-offs in compression, accuracy, and deployment.

  • Post-Training Quantization of LLMs: Post-training quantization includes weight-only and weight-activation approaches for reducing memory demands and matrix-multiplication overhead.
  • Quantization-Aware Training of LLMs: QAT can outperform PTQ but has been less explored for LLMs because of significant training costs.
  • Quantization-Aware Training of LLMs: BitNet b1.58 and OneBit investigate extreme binary or ternary quantization, while BitNet b1.58 reports near-lossless performance only up to 3-billion-parameter models.
  • Quantized Parameter-Efficient Fine-Tuning of LLMs: Quantized parameter-efficient fine-tuning methods add LoRA modules after low-bit quantization, but merging them can return models to FP16 format.
  • Quantized Parameter-Efficient Fine-Tuning of LLMs: QA-LoRA avoids that outcome by redesigning LoRA to merge into zero points, while parameter restrictions can hinder recovery from quantization information loss.

3 EfficientQAT

EfficientQAT uses a two-stage strategy: Block-AP trains all parameters block by block, then E2E-QP trains quantization parameters across the quantized model. The design reduces training cost while preserving broad optimization flexibility.

  • EfficientQAT: EfficientQAT replaces naive end-to-end QAT with Block-AP followed by E2E-QP to reduce memory requirements.
  • Block-Wise Training of All Parameters: Block-AP starts from uniform quantization and represents quantization through integer weights, full-precision weights, scaling factors, and zero points.
  • Block-Wise Training of All Parameters: Quantization and dequantization are integrated into the computation graph, allowing gradient-descent optimization of the quantization-aware pipeline.
  • Block-Wise Training of All Parameters: Block-AP trains model weights, scaling factors, and zero points directly, unlike prior partial-training approaches that optimize selected auxiliary parameters.
  • Block-Wise Training of All Parameters: Block-AP’s unrestricted parameter training expands the solution space beyond rounding-based methods that constrain integral-weight updates to (−1, +1).
  • End-to-End Training of Quantization Parameters: E2E-QP initializes quantized weights from Block-AP, freezes them, and trains quantization parameters end-to-end, typically only step sizes to avoid memory overhead.
  • End-to-End Training of Quantization Parameters: E2E-QP uses only dequantization during training and enables 2-bit Llama-2-70B QAT with 34.2GB of memory.

4 Experiments

Experiments evaluate EfficientQAT across quantization, instruction-tuning, ablations, training efficiency, and inference speed. The results show strong accuracy and perplexity performance alongside reduced training cost and faster INT2 inference.

  • Accuracy results: EfficientQAT significantly outperforms prior uniform-quantization methods across Llama-2 and Llama-3 in 2-bit and 3-bit settings.It improves accuracy over AWQ by 3.26% in Llama-3-8B w3g128 and over DB-LLM by 9.02% in w2g64.
  • Perplexity results: EfficientQAT consistently achieves lower Wikitext2 and C4 perplexity than competing methods across Llama-2 and Llama-3 at 2 and 3 bits.Perplexity reductions versus DB-LLM are 0.37 points for Llama-2-7B and 4.19 points for Llama-3-8B.
  • Ablation analysis: Both Block-AP and E2E-QP improve performance, with their combination producing the best ablation results.Block-AP outperforms E2E-QP in the reported component analysis.
  • Ablation analysis: Increasing E2E-QP training samples reduces the training-validation loss gap from 1.07 to 0.06 and raises average zero-shot accuracy from 57.14% to 58.99%.The study selects 4096 samples because accuracy shows no significant improvement beyond that size while the loss gap remains small.
  • Training efficiency: 4.8 hours and 40.9 hours are required to train the 7B and 70B models, respectively, while EfficientQAT takes only 50% of DB-LLM’s tuning time.A 70B model can be quantized on one A100-80GB GPU, whereas other compared methods require at least four.
  • Inference efficiency: 2.9x to 4.4x forward-pass speedups are achieved for EfficientQAT INT2 quantization through BitBLAS.Standard uniform quantization also supports deployment through multiple inference toolboxes.

5 Conclusion

EfficientQAT completes QAT with improved memory and training-time efficiency while maintaining versatility and performance across models and quantization levels. Its standard uniform quantization also simplifies deployment with popular toolboxes.

  • Conclusion: EfficientQAT completes quantization-aware training with improved memory usage and training time.The conclusion characterizes the method as applicable across varied models and quantization levels.
  • Conclusion: EfficientQAT is reported to be superior to existing PTQ, QAT, and Q-PEFT methods in versatility and performance.The conclusion bases this claim on comprehensive testing across various models and quantization levels.
  • Conclusion: Standard uniform quantization simplifies deployment using popular toolboxes.This deployment property is presented as a practical characteristic of EfficientQAT.

6 Limitation

EfficientQAT still trails full-precision models, especially at 2 bits, and depends on diverse, high-quality calibration data. Its effectiveness may therefore be constrained in data-scarce or domain-specific applications.

  • EfficientQAT retains a performance gap from FP16 models, particularly in 2-bit settings.
  • 4096 training samples are required for effective training in both Block-AP and E2E-QP.
  • Quantized-model performance can vary significantly with training-data size and distribution.
  • The method may be less effective in data-scarce or domain-specific applications.

A Reproducibility Statement

The reproducibility materials describe gradient computation for Block-AP and report INT2 speedups using BitBLAS on an A100-80GB GPU.

  • Block-AP uses a straight-through estimator to compute gradients through the rounding operation.
  • Gradients are specified for the scaling factor, zero point, and full-precision weight during Block-AP.
  • 2.9x to 4.4x INT2 speedups are reported for forward-pass execution using BitBLAS.
  • The speed comparison evaluates FP16 linear-layer matrix-vector multiplication in PyTorch against relative INT2 speedups on an A100-80GB GPU.

D Results Source of Other Method.

The paper compares EfficientQAT with multiple PTQ and QAT methods, using a mixture of reported, replicated, repository-derived, and code-produced results. The section also documents quantized-model sizing and group-size evaluation.

  • EfficientQAT is compared with GPTQ, AWQ, OmniQ, AutoRound, QuIP#, AQLM, LLM-QAT, BitDistiller, PB-LLM, and DB-LLM.
  • Comparison results come from official repositories, source papers, replicated experiments, or the authors’ codebase, depending on the method.
  • Results for QA-LoRA, QLoRA, QLoRA with GPTQ, and IR-QLoRA are cited from their respective publications.
  • Quantized models are more compact, with model-size comparisons reported against FP16 models using compression ratios.
  • N-bit quantization assigns each group of g weights a shared FP16 step size and N-bit zero point, while only transformer-block linear layers are quantized.
  • Table 12 evaluates 2-bit Llama-2-7B performance under different quantization group sizes.

F Additional Ablation Analysis

Additional analyses examine quantization group size, calibration-data choice, and extension to vision-language models. The results identify practical trade-offs and show strong low-bit LVLM performance with remaining optimization needs.

  • Quantization Group Size: A group size of 64 balances 2-bit quantization performance, outperforming size 128 but slightly underperforming size 32.Compared with size 128, it achieves 0.31 lower perplexity and 0.64% higher accuracy; compared with size 32, the differences are 0.09 perplexity and 0.14% accuracy.
  • Training Dataset: Changing calibration data from RedPajama to WikiText2 causes only 0.26% and 1.28% accuracy declines for w3g128 and w2g64, respectively.
  • Training Dataset: Using C4 as the calibration dataset can increase average zero-shot accuracy by 0.2-0.3 points.
  • Training Dataset: The authors recommend diverse calibration datasets such as C4 or RedPajama for Block-AP.
  • Instruction Tuning for LVLMs: EfficientQAT starts from Block-AP-quantized Vicuna models and applies E2E-QP fine-tuning within the LLaVA training pipeline.
  • Instruction Tuning for LVLMs: EfficientQAT outperforms QLoRA in low-bit settings for LLaVA-1.5-7B and LLaVA-1.5-13B.
  • Instruction Tuning for LVLMs: The 2-bit LLaVA-1.5-13B model reaches an average score of 59.9, exceeding the 59.6 score of FP16 LLaVA-1.5-7B trained with LoRA.
  • Instruction Tuning for LVLMs: A slight performance decrease for 4-bit EfficientQAT and 16-bit QLoRA versus 16-bit LoRA indicates further Q-PEFT optimization is needed for LVLMs.

H Comparisons with the Same Number of Data Samples

The main experiments use 4096 samples, while OmniQuant and GPTQ use 128; matched-sample evaluations address this difference. Table 15 reports C4 perplexity for Llama-2-7B across training-sample counts.

  • 4096 samples are used for EfficientQAT’s main experiments, compared with 128 samples for OmniQuant and GPTQ.The comparison is additionally conducted with the same number of data samples.
  • OmniQuant and GPTQ stabilize at 128 samples and do not improve with additional data.
  • Table 15 evaluates C4 perplexity for Llama-2-7B under different training-sample counts.

I Full Results

The full-results section reports average accuracy across five zero-shot tasks and provides task-specific results for 3-bit and 2-bit quantization. Tables 16 and 17 list zero-shot accuracy for Llama 2 and Llama 3.

  • Average accuracy is reported across five zero-shot tasks.
  • Tables 16 and 17 provide task-specific accuracy results for 3-bit and 2-bit quantization, respectively.
  • Table 16 reports 3-bit Llama 2 and Llama 3 zero-shot accuracy using lm_eval v0.4.2.The table reports acc rather than acc_norm.
  • Table 17 reports 2-bit Llama 2 and Llama 3 zero-shot accuracy using lm_eval v0.4.2.The table reports acc rather than acc_norm.
Loading 2407.11062v3…