PaperlayerView exact revision on arXiv

Source-linked AI summary

PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization

Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, Guangyu Sun

arXiv:2111.12293v3cs.CV

TL;DR

Prior PTQ methods can lose more than 1% accuracy at 8-bit quantization on vision transformers, despite PTQ’s deployment advantages. PTQ4ViT uses twin uniform quantization for special activations and a Hessian-guided metric for scaling factors. On ImageNet classification, quantized vision transformers achieve near-lossless accuracy, with less than 0.5% drop at 8-bit quantization.

  • Problem

    PTQ methods perform poorly on vision transformers because their special activation distributions and common scaling-factor metrics are not well handled.

  • Method

    PTQ4ViT combines twin uniform quantization with a Hessian-guided metric to reduce quantization error and select scaling factors.

  • Results

    Less than 0.5% accuracy drop at 8-bit quantization is achieved for ViT, DeiT, and Swin on ImageNet classification.

  • Takeaways & Limitations

    The experiments demonstrate near-lossless ImageNet prediction accuracy, making PTQ acceptable for vision transformers.

  • Takeaways & Limitations

    Non-uniform quantization solves the activation-distribution problem but is inefficient on most hardware devices without specially designed hardware.

Abstract

from arXiv · show

Quantization is one of the most effective methods to compress neural networks, which has achieved great success on convolutional neural networks (CNNs). Recently, vision transformers have demonstrated great potential in computer vision. However, previous post-training quantization methods performed not well on vision transformer, resulting in more than 1% accuracy drop even in 8-bit quantization. Therefore, we analyze the problems of quantization on vision transformers. We observe the distributions of activation values after softmax and GELU functions are quite different from the Gaussian distribution. We also observe that common quantization metrics, such as MSE and cosine distance, are inaccurate to determine the optimal scaling factor. In this paper, we propose the twin uniform quantization method to reduce the quantization error on these activation values. And we propose to use a Hessian guided metric to evaluate different scaling factors, which improves the accuracy of calibration at a small cost. To enable the fast quantization of vision transformers, we develop an efficient framework, PTQ4ViT. Experiments show the quantized vision transformers achieve near-lossless prediction accuracy (less than 0.5% drop at 8-bit quantization) on the ImageNet classification task.

1 Introduction

Vision transformers offer global-information modeling but remain costly to deploy, motivating quantization. PTQ4ViT addresses vision-transformer quantization with twin uniform quantization and a Hessian-guided scaling metric, achieving near-lossless 8-bit ImageNet accuracy.

  • Vision transformers replace CNN convolution layers with self-attention modules that capture global information and show strong potential across computer-vision tasks.
  • Quantization compresses neural networks by converting floating-point values to low-bit-width integers, reducing memory consumption and computation cost.
  • PTQ enables fast quantization and deployment using unlabeled calibration images, avoiding QAT’s training data, optimization time, and hyper-parameter tuning.
  • More than 1% accuracy drop at 8-bit quantization motivates analyzing vision-transformer activations, including highly unbalanced post-softmax values.
  • Twin uniform quantization separately quantifies asymmetric activation ranges while constraining scaling factors and designing a hardware-processable data format.
  • The Hessian-guided metric replaces inaccurate local metrics for selecting scaling factors, and PTQ4ViT achieves less than 0.5% accuracy drop at 8-bit quantization on ImageNet.

2 Background and Related Work

Vision transformers use self-attention to capture global information, but their computational demands motivate compression methods such as quantization. This section introduces transformer structure, attention computation, quantization, and related approaches.

  • Vision Transformer: Vision transformers replace convolution layers with self-attention modules to use global information in computer vision.ViT replaces all convolution layers, and subsequent architectures extend this approach to downstream tasks.
  • Vision Transformer: A vision transformer processes patch vectors through blocks containing multi-head self-attention and a multilayer perceptron with GELU activation.The self-attention module extracts global information across image patches.
  • Vision Transformer: Self-attention computes patch relationships by applying softmax to scaled QK^T and multiplying by V, after which multiple heads are concatenated.The scaling uses the hidden size of each head.
  • Quantization: Quantization converts weights and activations from floating-point to lower-bit integers, reducing memory, data movement, computation, and energy consumption.Uniform symmetric quantization projects values to k-bit integers using a scaling factor, rounding, and clamping.
  • Quantization: Twin uniform quantization separately quantizes values in two ranges and uses shifts instead of extra storage and FP32 format transformations.Compared with a CNN-oriented multiple-range method, it avoids an extra range bit and FP32 alignment operations.
  • Related Work: QAT optimizes quantization parameters using labeled training data, whereas PTQ uses unlabeled calibration images for faster quantization and deployment.QAT can achieve lower accuracy drops but requires training data, optimization time, and hyperparameter tuning.

3 Method

PTQ4ViT addresses vision-transformer quantization through twin uniform quantization for atypical activations and a Hessian guided metric for scaling-factor selection.

  • Base PTQ for Vision Transformer: Base PTQ searches layer-wise activation and weight scaling factors by minimizing output distance after matrix-multiplication quantization.It quantizes A and B symmetrically and alternately optimizes their scaling factors over candidate search spaces.
  • Base PTQ for Vision Transformer: More than 1% accuracy drop occurs when base PTQ is applied to vision transformers, even though calibration uses unlabeled images.The reported base PTQ procedure searches optimal scaling factors layer-by-layer.
  • Quantization Problems: Post-softmax activations are concentrated near zero with few large attention-relevant values, while post-GELU activations have broad positive and narrow negative ranges.These distributions differ from the Gaussian assumption commonly used to determine quantization scales.
  • Twin Uniform Quantization: Twin uniform quantization separately quantizes two value ranges using distinct scaling factors, improving treatment of asymmetric post-GELU values and softmax activations.Its data format uses a range flag and k−1 unsigned bits, enabling processing on existing CPUs and GPUs.
  • Hessian Guided Metric: MSE, cosine distance, and Pearson correlation can select sub-optimal scales because their local output criteria disagree with task loss.For blocks.6.mlp.fc1:activation, these metrics favor around 0.4 Amax 2^k−1, whereas task loss favors around 0.75 Amax 2^k−1.
  • Hessian Guided Metric: The Hessian guided metric estimates quantization influence on task loss through a Taylor expansion and yields scaling factors closer to those indicated by cross-entropy.It improves scaling-factor selection without requiring repeated network executions to obtain the last-layer output.

4 Experiments

Experiments evaluate PTQ4ViT across vision transformer architectures, bit-widths, competing PTQ methods, and component ablations on ImageNet classification. PTQ4ViT generally improves accuracy over base and other PTQ methods, with especially strong results at 8 and 6 bits.

  • With 8-bit quantization, PTQ4ViT limits accuracy drops to less than 0.5% across ViT, DeiT, and Swin models.
  • At 6-bit quantization, PTQ4ViT reduces the average accuracy drop to 2.1%, compared with 9.8% for base PTQ.
  • Swin transformers show less than 0.15% accuracy drops at 8-bit quantization, possibly because windowed self-attention reduces post-softmax imbalance.
  • At 8-bit and 6-bit quantization, PTQ4ViT outperforms other evaluated PTQ methods by more than 1% prediction accuracy on average.
  • At 4-bit quantization, PTQ4ViT performs worse than Liu et al.’s mixed-precision method, indicating mixed precision matters at lower bit-widths.
  • On ViT-S/224, the Hessian guided metric alone improves accuracy by 0.46% at 8 bits and 6.96% at 6 bits, with further gains when combined with twin uniform quantization.
  • Using twin uniform quantization without the Hessian guided metric lowers ViT-S/224 top-1 accuracy from 81.00% to 79.25% at 8-bit quantization.

5 Conclusion

The paper identifies special post-softmax and post-GELU distributions and inaccurate scaling-factor metrics as obstacles to PTQ for vision transformers. It addresses them with twin uniform quantization, a Hessian guided metric, and the PTQ4ViT framework, achieving near-lossless ImageNet accuracy.

  • Post-softmax and post-GELU activations have special distributions, while common quantization metrics inaccurately determine optimal scaling factors.
  • Twin uniform quantization and a Hessian guided metric address these distributional and scaling-factor selection problems.
  • PTQ4ViT is an efficient post-training quantization framework for vision transformers that decreases quantization error and improves prediction accuracy at a small cost.
  • The experiments achieve near-lossless prediction accuracy on the ImageNet classification task, making PTQ acceptable for vision transformers.

6 Appendix

The appendix details PTQ4ViT’s efficient calibration, parallel quantization choice, Hessian-guided derivation, scaling-factor search, and calibration stability. Its metric approximates loss sensitivity using output perturbations and diagonal gradient information.

  • PTQ4ViT pre-computes layer outputs and gradients and batches scaling-factor candidates, enabling quantization of most vision transformers in several minutes with 32 calibration images.
  • The framework uses parallel rather than sequential quantization because sequential quantization degrades accuracy on small calibration datasets, while parallel quantization is robust.
  • The task-loss surrogate uses cross-entropy between quantized predictions and floating-point predictions because PTQ lacks labeled calibration data.
  • A Taylor expansion around the converged pretrained model ignores the near-zero first-order gradient term and retains the second-order Hessian term.
  • The Hessian-guided metric replaces the output Hessian with a Fisher Information Matrix approximation and keeps only diagonal elements to reduce computation.
  • Scaling factors are selected by minimizing the gradient-weighted squared output perturbation over candidate quantization scales.
  • At least 20 quantization intervals and an upper bound m of at least 15 are sufficient, while search rounds beyond one change accuracy by less than 0.05%.
  • Across 20 runs using 32 calibration images, accuracy fluctuations remain small for the reported ViT, DeiT, and Swin models.
Loading 2111.12293v3…