PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-Training Quantization for Vision Transformer

Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, Wen Gao

arXiv:2106.14156v1cs.CV

TL;DR

Vision transformers are powerful but costly to deploy because of their sophisticated architectures and large resource demands. This paper introduces a vision-transformer-specific post-training quantization method that optimizes low-bit intervals, preserves attention rankings, and assigns mixed precision using feature diversity. Across benchmark models and datasets, it outperforms state-of-the-art post-training methods, including achieving 81.29% top-1 accuracy for DeiT-B on ImageNet with about 8-bit quantization.

  • Problem

    Vision transformers have sophisticated architectures and large parameter counts that create memory and computation challenges on resource-constrained devices.

  • Method

    The method optimizes weight and input quantization intervals, adds ranking loss to preserve attention-value order, and assigns layer bit-widths using nuclear norms of attention maps and output features.

  • Results

    The proposed method outperforms conventional post-training quantization methods across benchmark models and datasets, reaching 81.29% top-1 accuracy for DeiT-B on ImageNet with about 8-bit quantization.

  • Takeaways & Limitations

    The results demonstrate an effective transformer-specific post-training quantization scheme for reducing vision-transformer memory and computation costs.

Abstract

from arXiv · show

Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81.29\% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization.

1 Introduction

Vision transformers deliver strong computer-vision representations but their large, sophisticated architectures create memory, computation, and latency challenges on resource-constrained devices. The paper therefore develops post-training quantization tailored to transformer attention and mixed precision.

  • Motivation: ViT-L contains 307M parameters and requires 64G FLOPs, making inference expensive for mobile phones and intelligent cars.These resource-constrained settings also impose low-latency requirements for real-time applications.
  • Motivation: Quantization reduces neural-network storage and computation by using lower-bit weights and inputs without changing the model architecture.Quantizing inputs and weights can convert floating-point operations into integer or bit operations.
  • Research gap: Training-aware quantization is unsuitable when full training data are unavailable or training costs are intolerable for edge devices.This motivates a post-training approach that operates without additional training or fine-tuning.
  • Research gap: Post-training quantization avoids fine-tuning, but existing methods largely target CNNs or recurrent networks and overlook vision-transformer attention.The paper frames this mismatch as motivation for transformer-specific post-training quantization.
  • Approach: The proposed method optimizes weight and input quantization intervals to preserve similarity between full-precision and quantized outputs.It introduces ranking loss for attention values, nuclear-norm-based mixed precision, alternative interval search, and bias correction.

2 Related Works

Related work covers transformer models for computer vision, quantization and compression methods developed for NLP transformers, and post-training quantization techniques for neural networks. These approaches include architectural, mixed-precision, ternarization, binarization, and clipping or tensor-distance strategies.

  • Scope: The paper positions its contribution within prior work on transformer applications, BERT compression, and post-training quantization methods.The related-work discussion connects these strands without changing the vision-transformer architecture.
  • Vision transformers: Vision-transformer research applied sequence-based transformers to image classification and related recognition benchmarks.The cited developments include autoregressive pixel prediction and ViT, which treats image patches as sequences.
  • Transformer compression: NLP transformer compression research explored edge-oriented attention changes and increasingly aggressive quantization for BERT.Reported approaches include mixed-precision, product quantization, ternarization with knowledge distillation, and weight binarization.
  • Post-training quantization: Post-training quantization methods optimize tensor error, analytically compute clipping ranges, or allocate bits per channel without training or fine-tuning.The cited examples include OMSE, ACIQ, and outlier channel splitting.

3 Methodology

The method performs post-training quantization by optimizing weight and input intervals while preserving output similarity and self-attention rankings. It further assigns mixed precision using layer sensitivity estimated from nuclear norms.

  • Quantization setup: The method quantizes transformer weights and inputs involved in matrix multiplications, while leaving softmax unquantized.Quantized weights include attention, MLP, and embedding projections.
  • Quantization setup: Uniform quantization restricts floating-point values to finite levels controlled by an interval, bit-width, and clipping operation.Values exceeding the quantized domain are clipped.
  • Similarity-aware quantization: Quantization intervals are optimized to maximize similarity between full-precision and quantized output feature maps using calibration data and Pearson correlation.The search alternately optimizes weight and input intervals until convergence or an iteration limit.
  • Ranking-aware quantization: A ranking loss preserves the relative order of self-attention map elements because quantization can change that order and degrade performance.The loss is pairwise and margin-based, with γ controlling its trade-off.
  • Bias correction: Bias correction subtracts expected output error from each layer’s bias so quantized outputs retain the full-precision mean.Expected errors are estimated using calibration data after each search iteration.
  • Mixed-precision quantization: Mixed precision assigns more bits to sensitive layers, using nuclear norms of attention maps and output features to estimate sensitivity.The nuclear norm is the sum of singular values and helps reduce the search space for bit-width settings.

4 Exprimental results

Experiments on image classification and object detection show that the proposed mixed-precision post-training quantization method preserves accuracy while reducing precision and computational costs. Ablation results further indicate that ranking-aware quantization, bias correction, and nuclear-norm-based mixed precision improve performance.

  • Experimental setup: The method is evaluated on ViT and DeiT image-classification models and DETR object-detection models across CIFAR-10, CIFAR-100, ImageNet, and COCO2017.Calibration uses 100 images for CIFAR-10 and CIFAR-100, and 1000 images for ImageNet and COCO2017.
  • Image classification: 6-bit quantization reduces memory by about 25% and computational costs by 44% compared with conventional post-training quantization, while outperforming the percentile-based method.On image classification, the proposed method achieves average gains of 2.95% on CIFAR-100 and 3.28% on ImageNet over the percentile-based method.
  • Image classification: 8-bit quantization of ViT-L is comparable to full precision on CIFAR-100 and ImageNet, while outperforming the percentile-based method by a large margin.The authors suggest that greater redundancy in ViT-L makes its performance degradation smaller than ViT-B’s.
  • Image classification: 81.29% Top-1 accuracy is achieved by the proposed DeiT-B model at 8-bit quantization, compared with 75.21% for the percentile-based model.At 6-bit quantization, the proposed DeiT-B model reaches 77.47% Top-1 accuracy versus 73.99% for the percentile-based model.
  • Object detection: For DETR, 6-bit quantization outperforms percentile-based, EasyQuant, and Bit-Split methods by 2.6, 1.1, and 1.2 mAP, respectively.The proposed mixed-precision method achieves mAP comparable to the full-precision model at 8-bit quantization.
  • Ablation study: On ImageNet with ViT-B, the full method reaches 76.98% Top-1 accuracy at 8-bit quantization, comparable to the full-precision model.Similarity-aware mixed precision alone reaches 76.26%, while ranking-aware quantization and bias correction provide further gains.

5 Conclusion

The paper develops a mixed-precision post-training quantization scheme for vision transformers, selecting layer bit-widths using attention-map and output-feature nuclear norms. It preserves similarity between quantized and original feature maps through optimized intervals, ranking loss, bias correction, and alternative searching, improving accuracy and memory costs over conventional post-training quantization.

  • The proposed scheme assigns variant bit-widths to transformer layers according to the nuclear norms of attention maps and output features.
  • It searches quantization intervals to preserve similarity between quantized and original feature maps.
  • A ranking loss maintains the relative order of attention values after quantization.
  • Bias correction reduces accumulated quantization error, while an alternative searching strategy optimizes each transformer layer's quantization interval.
  • The method outperforms conventional post-training quantization by a large margin in network accuracy and memory costs.
Loading 2106.14156v1…