Source-linked AI summary
Q8BERT: Quantized 8Bit BERT
Ofir Zafrir, Guy Boudoukh, Peter Izsak, Moshe Wasserblat
TL;DR
Large pre-trained Transformer models deliver strong NLP performance but require substantial compute, memory, and power resources. This paper applies quantization-aware training during BERT fine-tuning, achieving a 4× smaller model with minimal reported accuracy loss and potential 8bit inference acceleration.
Problem
Large pre-trained Transformer models have many parameters and require substantial compute, memory, and power resources for production inference.
Method
The method applies quantization-aware training during BERT fine-tuning, using fake quantization and full-precision backpropagation through the Straight-Through Estimator.
Results
4× smaller memory footprint and less than 1% relative quantization error were reported, excluding the RTE task.
Takeaways & Limitations
The method supports efficient BERT inference with 8bit GEMM and is intended for low-latency NLP applications across hardware platforms.
Abstract
from arXiv · showhide
Recently, pre-trained Transformer based language models such as BERT and GPT, have shown great improvement in many Natural Language Processing (NLP) tasks. However, these models contain a large amount of parameters. The emergence of even larger and more accurate models such as GPT2 and Megatron, suggest a trend of large pre-trained Transformer models. However, using these large models in production environments is a complex task requiring a large amount of compute, memory and power resources. In this work we show how to perform quantization-aware training during the fine-tuning phase of BERT in order to compress BERT by $4\times$ with minimal accuracy loss. Furthermore, the produced quantized model can accelerate inference speed if it is optimized for 8bit Integer supporting hardware.
1 Introduction
Large pre-trained Transformer models achieve strong NLP results but impose substantial memory and compute demands. Q8BERT addresses this deployment challenge through quantization-aware fine-tuning for compact BERT models.
- Motivation: BERT-Base has 110M FP32 parameters, while BERT-Large has 334M, creating high memory footprints and heavy inference requirements.These requirements include substantial compute and bandwidth.
- Motivation: Model compression reduces the compute and memory resources required for inference.Prior work combined pruning, quantization, and Huffman encoding to compress VGG-16 by 49×.
- Contribution: Q8BERT applies quantization-aware training during BERT fine-tuning to target a strong compression-accuracy ratio.The approach is presented as a method for compressing BERT while retaining task performance.
- Contribution: The method is not exclusive to BERT and can be integrated into other large pre-trained Transformer models.This extends the stated method beyond the specific BERT architecture.
2 Method
The method uses symmetric 8bit linear quantization and quantization-aware training to expose BERT to quantization error during fine-tuning. Fake quantization and the Straight-Through Estimator enable quantized forward behavior with full-precision backpropagation.
- 2.1 Quantization Scheme: Symmetric linear quantization converts BERT weights and activations to 8bit integers.The scheme uses a scaling factor and a maximum representable quantized value.
- 2.1 Quantization Scheme: For b bits, the maximum quantized value is M = 2^b−1 −1; for 8 bits, M = 127.Scaling factors may be determined dynamically or from training and calibration statistics.
- 2.2 Quantization-Aware Training: Quantization-aware training adapts neural-network training to quantization at inference rather than applying quantization only after training.Fake quantization introduces rounding error during training, and STE estimates gradients through the nondifferentiable rounding operation.
- Evaluation: Table 1 compares FP32 baselines, 8bit QAT models, and 8bit dynamically quantized models across GLUE and SQuAD tasks.Scores use each task’s specified metric on its publicly available development set.
- 2.2 Quantization-Aware Training: Fake quantization and STE provide quantized inference during training while back-propagating at full precision.This lets FP32 weights learn to overcome the quantization error.
3 Implementation
Quantized BERT replaces its Embedding and Fully Connected layers with quantized implementations while retaining selected operations in FP32. Training simulates quantized computations before deployment.
- Quantized BERT: The implementation quantizes all BERT Embedding and Fully Connected layers to Int8.The quantized layers perform GEMM using fake-quantized inputs and weights, while bias accumulation remains untouched during training.
- Quantized BERT: Operations requiring higher precision, including Softmax, Layer Normalization, and GELU, remain in FP32.The implementation replaces the corresponding layers in a PyTorch-Transformers BERT implementation.
4 Evaluation
The evaluation measures quantized BERT on GLUE and SQuADv1.1, comparing quantization-aware training with dynamically quantized models. Quantization-aware training induces less than 1% relative error in internal testing, excluding RTE, while reducing model space by 4×.
- Evaluation datasets: GLUE and SQuADv1.1 provide evaluation tasks spanning natural language understanding and question answering.GLUE covers a wide array of NLP tasks, while SQuADv1.1 is a reading-comprehension dataset with span answers.
- Accuracy comparison: Table 2 reports accuracy reduction relative to baseline for Dynamically Quantized and Quantization-aware Trained models.Tasks marked Large were trained with the BERT-Large architecture.
- Quantization-aware training results: Less than 1% relative error is induced by quantization in internal testing, excluding RTE, while model space is reduced by 4×.The reported relative error is measured against the baseline model, and the stated compression applies to the model’s space capacity.
- Effect of quantization-aware training: Dynamic Quantization produces significantly worse results over all tasks than the quantization-aware approach.The comparison evaluates whether quantization-aware training is necessary by contrasting QAT with post-training Dynamic Quantization.
5 Related Work
Related work explores compression and acceleration of Transformer models through distillation, post-training quantization, specialized hardware, and optimized kernels.
- 8bit Transformer compression: Transformer-LT combined knowledge distillation with 8bit post-training quantization but lost 1 BLEU score versus its baseline.Other work applied 8bit post-training quantization to Transformer-LT and used Intel specialized 8bit hardware for inference acceleration.
- Hardware-oriented approaches: Habana Labs reported Quantized BERT performance using 16bit Integer arithmetic, providing 2× compression.NVIDIA measured BERT with 16bit Floating Point arithmetic and implemented optimized kernels to reduce memory bandwidth during inference.
- Post-training quantization: Sucik fine-tuned BERT on a custom dataset and then applied 8bit Integer post-training quantization.
6 Conclusions and Future Work
The paper concludes that 8bit quantization of BERT GEMM operations achieves minimum accuracy loss across varied NLP tasks. It identifies efficient inference and reduced memory footprint as deployment goals, while proposing further compression work.
- Conclusion: The method quantizes BERT GEMM operations to 8bit across a variety of NLP tasks with minimum accuracy loss.
- Practical implications: The authors aim for software developers to use the method for BERT compression and efficient inference with 8bit GEMM operations.
- Deployment scope: Efficient inference is intended to support low-latency NLP applications across hardware platforms from edge devices to data centers.
- Future work: Future work will apply and integrate additional model compression methods with the quantized BERT model.