Source-linked AI summary
Towards Accurate Post-Training Quantization for Vision Transformer
Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, Xianglong Liu
TL;DR
Very-low-bit post-training quantization can cause severe accuracy drops in vision transformers, while existing calibration metrics and Softmax quantization are poorly suited to that setting. APQ-ViT combines blockwise error calibration with a Softmax quantizer designed to preserve its power-law distribution. It reports improvements over existing methods across classification and detection, especially at W4A4.
Problem
At ultra-low bit widths, existing calibration metrics can mismeasure quantization effects and quantization can damage Softmax’s power-law distribution, alongside severe accuracy drops or crashes.
Source: §1 · p. 1, §3.2 · p. 4, §3.3 · p. 5Method
APQ-ViT combines blockwise calibration that prioritizes significant errors with Softmax quantization that preserves its power-law distribution.
Source: §1 · p. 2, §3.3 · p. 6Results
Takeaways & Limitations
The reported results cover image classification and object detection across multiple vision-transformer architectures, including ViT, DeiT, and Swin Transformer.
Source: §1 · p. 2, §4.4 · p. 8The quantization objective follows the assumption that quantized parameters should retain as much information as possible from their full-precision counterparts.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
Vision transformer emerges as a potential architecture for vision tasks. However, the intense computation and non-negligible delay hinder its application in the real world. As a widespread model compression technique, existing post-training quantization methods still cause severe performance drops. We find the main reasons lie in (1) the existing calibration metric is inaccurate in measuring the quantization influence for extremely low-bit representation, and (2) the existing quantization paradigm is unfriendly to the power-law distribution of Softmax. Based on these observations, we propose a novel Accurate Post-training Quantization framework for Vision Transformer, namely APQ-ViT. We first present a unified Bottom-elimination Blockwise Calibration scheme to optimize the calibration metric to perceive the overall quantization disturbance in a blockwise manner and prioritize the crucial quantization errors that influence more on the final output. Then, we design a Matthew-effect Preserving Quantization for Softmax to maintain the power-law character and keep the function of the attention mechanism. Comprehensive experiments on large-scale classification and detection datasets demonstrate that our APQ-ViT surpasses the existing post-training quantization methods by convincing margins, especially in lower bit-width settings (e.g., averagely up to 5.17% improvement for classification and 24.43% for detection on W4A4). We also highlight that APQ-ViT enjoys versatility and works well on diverse transformer variants.
1 INTRODUCTION
Vision transformers offer promise for vision tasks, but their computational cost motivates compression, and existing post-training quantization struggles at very low bit widths. APQ-ViT combines blockwise calibration with a Softmax-specific quantization strategy and reports gains across classification and detection.
1 INTRODUCTION
- Post-training quantization uses a small unlabeled calibration dataset without retraining, addressing the storage and inference costs of vision transformers.The introduction notes that ViT-L has 307M parameters and 190.7 GFLOPs during inference.Source: §1 · p. 1
- At ultra-low bit widths, existing calibration metrics can mismeasure quantization effects, while existing quantization can damage Softmax’s power-law distribution.These issues are associated with severe accuracy drops or model crashes.Source: §1 · pp. 1–2
- APQ-ViT’s Blockwise Bottom-elimination Calibration measures quantization loss across blocks and prioritizes significant errors by eliminating second-order gradients for trivial errors.The scheme is designed around the block-stacking architecture and can be generalized to other variants.Source: §1 · p. 2
- Matthew-effect Preserving Quantization prioritizes preserving Softmax’s power-law distribution for the attention mechanism over maximizing mutual information.The paper presents this as its Softmax-specific quantization strategy.Source: §1 · p. 2
- Up to 5.17% classification improvement and 24.43% detection improvement on W4A4 are reported for APQ-ViT over existing post-training quantization methods.The paper also reports performance across ViT, DeiT, and Swin Transformer variants.Source: §1 · p. 2
2 RELATED WORK
Related work describes architectural approaches to making vision transformers more efficient and quantization approaches that reduce model size and computation. It distinguishes training-based and training-free quantization, then outlines methods for selecting scales and calibrating layers or modules.
2.2 Quantization
- Figure 2 visualizes Hessian-guided optimal-scale loss at 8-, 6-, and 4-bit quantization, plus overall loss across candidates.Source: Figure 2 · p. 3
2.1 Vision Transformer
- Vision-transformer efficiency work includes attention variants, pruning, and replacing some self-attention modules with convolutions, while leaving parameters full precision.Swin Transformer is described as a competitive backbone, and the section also discusses MobileViT and distillation.Source: §2.1 · pp. 2–3
2.2 Quantization
- Quantization reduces model size and computational complexity; PTQ avoids training or fine-tuning, and existing methods search scales or optimize calibration strategies.Examples include Hessian-guided scale search, layerwise optimization, alternating scale optimization, and module-wise reconstruction.Source: §2.2 · p. 3
3 METHOD
APQ-ViT addresses low-bit vision-transformer quantization with blockwise calibration and a Softmax-specific quantizer. Its design captures errors across transformer blocks while preserving Softmax’s concentration on large values.
3.1 Preliminaries
- The calibration pipeline uses a second-order Hessian-guided metric to select weight and activation scaling factors, alternating their optimization over candidate values.The method evaluates quantization’s influence on task loss through a Taylor expansion.Source: §3.1 · pp. 3–4
3.2 Blockwise Bottom-elimination Calibration
- BBC calibrates transformer layers blockwise and removes the smallest absolute quantization errors so the metric prioritizes larger, prediction-relevant errors.Blockwise calibration lets the metric account for quantization in adjacent layers; bottom elimination focuses it on errors that affect the block output.Source: §3.2 · pp. 4–5
3.3 Matthew-effect Preserving Quantization
- MPQ quantizes Softmax outputs while preserving their Matthew-effect, allocating finer representations to sparse, large values instead of optimizing mutual information alone.The Softmax output is described as an asymmetric, extremely unbalanced power-law distribution; MPQ uses asymmetric linear quantization.Source: §3.3 · pp. 5–6
3.4 Framework of APQ-ViT
- APQ-ViT obtains each transformer block’s output and gradient, optimizes layers blockwise with a bottom-elimination metric, and inserts MPQ after Softmax.The process stores block outputs and gradients rather than all layer data, reducing storage to about 20% of the existing methods’ requirement.Source: §3.4 · p. 6
4 EXPERIMENT
The experiments assess APQ-ViT through ablations and comparisons on image classification and object detection across vision-transformer architectures. The reported gains are strongest at low bit-widths, with the two proposed components also improving performance when combined.
4 EXPERIMENT
- The experiment section evaluates proposed components by ablation and compares methods on classification and detection across vision-transformer architectures.The classification comparisons include ViT, DeiT, and Swin Transformer; detection uses Swin Transformer backbones with Mask RCNN and Cascade Mask RCNN.Source: §4 · p. 6, §4.3 · p. 7, §4.4 · p. 8
4.1 Settings
- The experiments use 32 ImageNet images for classification calibration and one COCO image for detection calibration; Softmax, LayerNorm, and GeLU remain full-precision.The first convolution and last classification layers are also not activation-quantized.Source: §4.1 · p. 7
4.2 Ablation Study
- In ablations, BBC raises ViT-B W4A4 accuracy to 38.40%, 7.71% above the traditional method, while MPQ improves DeiT-S W4A4 accuracy by 21.96% over Logarithmic and 21.24% over Segmental quantization.Jointly applying BBC and MPQ further improves performance, which the authors say demonstrates that the strategies work orthogonally.Source: §4.2 · p. 7
4.3 Comparison on Classification Task
- Under W4A4, APQ-ViT improves classification accuracy by up to 5.17% on average over PTQ4ViT across transformer variants, patch sizes, and input resolutions.The authors also report average improvements of 1.02% under W6A6 and 3.87% under W8A4; W8A8 has an average accuracy loss of about 0.23% versus full precision.Source: §4.3 · pp. 7–8, Table 3 · p. 7
4.4 Comparison on Object Detection Task
- Under W4A4, APQ-ViT improves detection results by an average of 24.43% over PTQ4ViT and 30.81% over BasePTQ.For Cascade Mask RCNN Swin-S, it achieves 44.7% APbox and 40.1% APmask, versus 26.7% and 26.6% for PTQ4ViT; the reported drops are 3.8% and 3.2%, respectively.Source: §4.4 · p. 8, Table 4 · p. 8
5 CONCLUSION
APQ-ViT combines blockwise calibration with Softmax quantization designed to preserve its power-law distribution. Experiments report improvements especially at lower bit widths, and the method works with diverse vision transformer variants.
5 CONCLUSION
- Averagely up to 5.17% improvement for classification and 24.43% for detection on W4A4, with prevailing improvements especially at lower bit-width settings.Experiments also report versatility across vision transformer variants, including DeiT and Swin Transformer.Source: §5 · p. 8
- Bottom-elimination Blockwise Calibration fixes overall quantization error blockwise and prioritizes errors that impact the final output.The scheme is presented as a unified calibration approach.Source: §5 · p. 8
- Matthew-effect Preserving Quantization maintains Softmax’s power-law distribution and keeps the attention mechanism’s function.Source: §5 · p. 8
- APQ-ViT works with diverse vision transformer variants, including DeiT and Swin Transformer.Source: §5 · p. 8