PaperlayerView exact revision on arXiv

Source-linked AI summary

FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer

Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, Shuchang Zhou

arXiv:2111.13824v4cs.CV

TL;DR

Fully quantized vision transformers suffer severe degradation because LayerNorm inputs vary sharply across channels and attention maps are extremely non-uniform. FQ-ViT addresses these issues with PTF and LIS, achieving comparable performance to full-precision models across vision-transformer tasks, including 50.8 mAP on Cascade Mask R-CNN with Swin-S.

  • Problem

    Fully quantized vision transformers remain difficult to deploy because transformer models have high computational overhead and their LayerNorm inputs exhibit severe inter-channel variation that causes large quantization error.

  • Method

    FQ-ViT uses Power-of-Two Factor for LayerNorm inputs and Log-Int-Softmax for non-uniform attention maps, enabling 4-bit attention quantization and BitShift-based integer inference.

  • Results

    FQ-ViT achieves nearly lossless quantization across vision-transformer classification and detection benchmarks, including 50.8 mAP on Cascade Mask R-CNN (Swin-S) with 8-bit weights/activations and 4-bit attention maps.

  • Takeaways & Limitations

    The method provides a higher baseline for fully quantized vision transformers while reducing hardware resource requirements and supporting research into even lower bit-widths.

Abstract

from arXiv · show

Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.

1 Introduction

Vision transformers offer strong computer-vision performance but impose substantial deployment costs, while existing quantization methods struggle with fully quantizing transformer-specific modules. FQ-ViT addresses LayerNorm and Softmax quantization using PTF and LIS, achieving comparable accuracy to full-precision models.

  • Motivation: Vision transformers deliver competitive performance across classification, detection, and segmentation but require more parameters and computation than CNNs.ViT-L has 307M parameters and 190.7G FLOPs while reaching 87.76% ImageNet accuracy with large-scale pre-training.
  • Quantization challenge: Existing quantization approaches mainly target CNNs and degrade substantially when quantizing LayerNorm and Softmax in vision transformers.Leaving these modules in floating point avoids degradation but requires floating-point hardware units and reduces inference speed.
  • Quantization challenge: LayerNorm inputs exhibit severe inter-channel variation, while attention maps have highly non-uniform values concentrated near zero with a few values near one.Some LayerNorm channel ranges exceed 40× the median, and most attention-map values fall between 0 and 0.01.
  • Proposed approach: FQ-ViT introduces PTF for LayerNorm inputs and LIS for Softmax, enabling fully quantized vision transformers with reduced quantization error and simpler inference.PTF uses channel-specific power-of-two factors, while LIS provides higher resolution for small attention values and supports integer Softmax inference.
  • Results: The method uses 8-bit weights and activations with 4-bit attention maps while achieving performance comparable to floating-point versions.The paper reports extensive experiments across transformer architectures on image classification and object detection.

2 Related Work

Vision transformers improve performance across computer-vision tasks but remain difficult to deploy because of their parameter count and computational overhead. Related work explores faster architectures, token pruning, and training-based or training-free quantization.

  • Transformer architectures: Vision-transformer architectures have demonstrated effectiveness in image classification, object detection, and semantic segmentation.Swin Transformer is described as surpassing state-of-the-art CNNs on almost traditional computer-vision tasks.
  • Efficient architectures: Architecture-design methods improve efficiency through down-sampling, patch descriptors, redesigned Attention-MLP blocks, or progressive token sparsification.LeViT and DynamicViT are presented as examples of faster or more computationally efficient transformer designs.
  • Quantization: Quantization methods are categorized as Quantization-Aware Training or Post-Training Quantization, trading training cost against deployment convenience.QAT can support aggressively low-bit quantization but requires training resources, whereas PTQ is training-free.

3 Proposed Method

The method targets transformer-specific quantization failures in LayerNorm and Softmax by combining PTF for channel variation with LIS for non-uniform attention maps and integer-only inference.

  • LayerNorm and Softmax challenges: LayerNorm and Softmax require special treatment because fully quantizing them causes severe degradation, while leaving them unquantized retains floating-point hardware costs.LayerNorm cannot be folded into the previous layer and must be quantized separately; fully quantized vision transformers therefore need methods for both modules.
  • LayerNorm and Softmax challenges: 622.5/15.5 channel-wise range ratios in Swin-B versus 21.6/4.2 in ResNet152 demonstrate stronger inter-channel variation in vision transformers.Layer-wise quantization shares parameters across channels, so these fluctuations produce intolerable quantization error.
  • Power-of-Two Factor: PTF equips each LayerNorm input channel with a distinct power-of-two factor while retaining layer-wise quantization parameters.BitShift combines the factors efficiently, avoiding floating-point calculations required by group-wise or channel-wise quantization.
  • Log-Int-Softmax: 8.69% top-1 accuracy on ImageNet with 4-bit uniform attention maps drops 63.05% from the 8-bit case for DeiT-T, whereas log2 quantization matches 8-bit uniform performance with 50% less memory.This motivates using lower-bit attention-map quantization while preserving model performance.
  • Log-Int-Softmax: 98.8% of ViT-L attention-map values are below 1/16, so log2 quantization allocates 12 bins to values that uniform 4-bit quantization represents with one bin.The resulting representation preserves more small-value resolution and attention-map order consistency.
  • Log-Int-Softmax: LIS combines log2 quantization with integer polynomial exponential approximation to provide faster, lower-consumption Softmax inference.Unlike full-precision Softmax, LIS keeps the entire quantized multi-head self-attention computation in the integer domain.

4 Experiments

Experiments show that FQ-ViT supports fully quantized vision transformers across ImageNet and COCO, including 4-bit attention maps, while retaining strong accuracy and reducing inference complexity.

  • Image Classification on ImageNet: 81.20% top-1 accuracy is achieved on DeiT-B with all modules quantized to 8-bit, while 80.85% remains with 4-bit attention maps.
  • Experimental Setup: Table 2 compares bbox mAP across state-of-the-art methods on COCO, while Table 3 evaluates PTF and LIS using full-precision and quantized ViT-B on ImageNet.Table 3 uses MinMax with 8-bit weights and activations as the baseline, with attention-map bit-width indicated by the index.
  • Object Detection on COCO: 47.2 mAP on Mask R-CNN and 50.8 mAP on Cascade Mask R-CNN are achieved with 8-bit weights and activations and 4-bit attention maps.
  • Ablation Studies: PTF and LIS outperform the ViT-B baseline and achieve almost lossless accuracy, while PTF adds few BitOPs and LIS reduces them.
  • Visualization of Quantized Attention Map: LIS preserves more low-activation texture and relative attention-map rank than uniform quantization, especially at 6-bit and 4-bit widths.Uniform quantization sharply degrades or deactivates attention regions at lower bit-widths, whereas LIS remains acceptable and similar to 8-bit.

5 Conclusions

The paper presents PTF and LIS as a method for fully quantizing vision transformers while maintaining performance comparable to full-precision models and reducing hardware requirements.

  • PTF addresses LayerNorm input channel variation, while LIS enables 4-bit attention-map quantization and replaces inference MatMul with BitShift.
  • Experiments show that fully quantized vision transformers achieve comparable performance with full-precision models.

A.1 Quantized Inference of LayerNorm

The LayerNorm inference procedure quantizes its input and output, computes statistics from shifted quantized activations, and performs the resulting operations with integer-only calculations.

  • The quantized LayerNorm formulation uses learned parameters γ and β together with input statistics µX and σX.
  • PTF quantizes LayerNorm input X and represents its quantized value XQ using a scalar scale, zero point, and channel-wise Power-of-Two Factor α.
  • The procedure shifts quantized activation XQ with PTF α before calculating LayerNorm statistics from the shifted activation.
  • The mean of X and X^2 is computed over C channels, then used to calculate the variance-related statistics.
  • The resulting LayerNorm statistics are obtained using integer-only calculations, after which quantized input and output are integrated into inference.

A.2 Quantized Inference of Softmax

The paper replaces floating-point Softmax operations with an integer-only pipeline based on i-exp, reverse Softmax, Log2 quantization, and BitShift acceleration. LIS preserves higher resolution for small attention values while enabling fully fixed-point inference.

  • Integer-only Exponential: The i-exp approximation differs from the exponential function by at most 1.9 × 10^-3, below the 8-bit quantization error of 3.9 × 10^-3.
  • Log-Int-Softmax: LIS first subtracts the maximum input, applies integer-only exponential, and quantizes reverse Softmax outputs with Log2.This keeps inputs non-positive and supports integer-domain processing.
  • Quantization: In 4-bit quantization, N equals 2^4 − 1 = 15 for the Log2-quantized Softmax output.
  • Integer Log2: Log2 of an integer is computed entirely with integer arithmetic using Find First One and bit-based rounding.For example, 0000 1101 1010 1100₂ produces M = 11, χ = 1, and rounded output 12.
  • BitShift Inference: The attention-map–value multiplication is converted to a BitShift operation using the output scale of the quantized values.This realizes fully fixed-point Softmax inference and accelerates attention-map and value computation.

B Hyperparameter K of Power-of-Two Factor

Experiments show that PTF accuracy nearly saturates at K = 3, which is therefore selected as the default value across models.

  • K = 3 nearly saturates top-1 accuracy, so the paper selects 3 as PTF’s default hyperparameter.The choice is intended to accommodate differing inter-channel variation across models.

C Inter-channel Variation for Vision Transformers and ResNets

The comparison examines channel-wise activation ranges in Vision Transformers and ResNets and finds serious inter-channel variation in Vision Transformers.

  • The figure compares channel-wise minimum and maximum values for Vision Transformers and ResNets.Vision Transformers use the input of the last LayerNorm, while ResNets use the output of the fourth stage.
  • Vision Transformers exhibit serious inter-channel variation in the compared activations.
Loading 2111.13824v4…