RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers

Zhikai Li; Junrui Xiao; Lianwei Yang; Qingyi Gu

Source-linked AI summary

RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers

Zhikai Li, Junrui Xiao, Lianwei Yang, Qingyi Gu

arXiv:2212.08254v2Dec 16, 2022cs.CVcs.LG

TL;DR

Existing PTQ methods for ViTs often lose accuracy, particularly at low bit widths, despite PTQ’s practical calibration advantages. RepQ-ViT decouples quantization from inference with scale reparameterization for extreme activation distributions, and experiments show strong low-bit performance without hyperparameters or costly reconstruction.

Problem
Existing PTQ schemes for ViTs suffer non-trivial accuracy degradation, especially in low-bit quantization, while ViTs remain costly to deploy.
Method
RepQ-ViT uses complex quantizers during quantization and reparameterizes their scales into simple hardware-friendly quantizers for inference.
Results
RepQ-ViT significantly outperforms existing methods in low-bit quantization across image classification, object detection, and instance segmentation.
Takeaways & Limitations
RepQ-ViT improves the practicality of low-bit PTQ for ViTs without hyperparameters or expensive reconstruction procedures.
Takeaways & Limitations
For post-Softmax activations, direct log2 quantization can overly sparsely represent important attention scores, while LayerNorm scale reparameterization may incur slight accuracy loss.

Abstract

from arXiv · show

Post-training quantization (PTQ), which only requires a tiny dataset for calibration without end-to-end retraining, is a light and practical model compression technique. Recently, several PTQ schemes for vision transformers (ViTs) have been presented; unfortunately, they typically suffer from non-trivial accuracy degradation, especially in low-bit cases. In this paper, we propose RepQ-ViT, a novel PTQ framework for ViTs based on quantization scale reparameterization, to address the above issues. RepQ-ViT decouples the quantization and inference processes, where the former employs complex quantizers and the latter employs scale-reparameterized simplified quantizers. This ensures both accurate quantization and efficient inference, which distinguishes it from existing approaches that sacrifice quantization performance to meet the target hardware. More specifically, we focus on two components with extreme distributions: post-LayerNorm activations with severe inter-channel variation and post-Softmax activations with power-law features, and initially apply channel-wise quantization and log$\sqrt{2}$ quantization, respectively. Then, we reparameterize the scales to hardware-friendly layer-wise quantization and log2 quantization for inference, with only slight accuracy or computational costs. Extensive experiments are conducted on multiple vision tasks with different model variants, proving that RepQ-ViT, without hyperparameters and expensive reconstruction procedures, can outperform existing strong baselines and encouragingly improve the accuracy of 4-bit PTQ of ViTs to a usable level. Code is available at https://github.com/zkkli/RepQ-ViT.

1. Introduction

PTQ offers practical ViT compression without end-to-end retraining, but existing methods degrade substantially at low bit widths. RepQ-ViT decouples quantization from inference through scale reparameterization, preserving complex-distribution handling while enabling hardware-friendly inference.

PTQ calibrates quantization parameters with a tiny unlabeled dataset, avoiding retraining and reducing time and resource costs.
Existing PTQ methods for ViTs can lose more than 1% accuracy even at 8-bit quantization, while ultra-low-bit performance remains unsatisfactory.
Post-LayerNorm activations use channel-wise quantization before reparameterization to hardware-compatible layer-wise quantization.The reparameterization adjusts LayerNorm affine factors and the next layer’s weights.
Post-Softmax activations use log√2 quantization initially, then reparameterize scales to log2 quantization for bit-shifting inference.
RepQ-ViT decouples quantization and inference, using complex quantizers initially and scale-reparameterized simplified quantizers during inference.
Across image classification, object detection, and instance segmentation, RepQ-ViT outperforms existing baselines without hyperparameters or expensive reconstruction procedures.

2. Related Works

ViTs offer strong vision performance but impose substantial memory and computation costs, motivating compression. PTQ is attractive for rapid deployment, yet methods developed for CNNs or adapted to ViTs often perform poorly on ViT structures.

ViTs’ large matrix multiplications create substantial memory footprints and computational overheads in resource-constrained edge scenarios.
Lightweight ViT architectures reduce deployment costs but retain floating-point parameters, leaving room for further compression.
PTQ avoids the resource-intensive and time-consuming full-dataset retraining required by quantization-aware training.
CNN-oriented PTQ methods such as DFQ, AdaRound, and BRECQ perform poorly on ViTs because of their substantially different structures.
ViT-specific methods address attention-score ordering, LayerNorm, Softmax, and data-free calibration, but their approaches differ in quantizer design and scope.

3. Methodology

RepQ-ViT decouples quantization from inference, using complex quantizers to preserve distributions and scale reparameterization to obtain hardware-friendly inference quantizers. It targets post-LayerNorm and post-Softmax activations, whose extreme distributions challenge standard layer-wise and log2 quantization.

Framework Overview: RepQ-ViT separates quantization from inference by converting initially complex quantizers into simplified hardware-friendly quantizers through scale reparameterization.The framework overview and Algorithm 1 describe this decoupled pipeline.
Inference Quantizers: Standard uniform quantization is hardware-supported, while layer-wise activation quantization and channel-wise weight quantization provide the inference granularity used in the framework.Matrix-multiplication weights and inputs are quantized, whereas LayerNorm and Softmax remain floating-point operations.
LayerNorm Activations: Post-LayerNorm activations exhibit severe inter-channel variation, making unified layer-wise scales inaccurate; DeiT-S channels 300–350 have minimum, mean, and maximum ranges of 3.94, 7.11, and 22.2.Channel-wise quantization addresses this distribution but requires dedicated activation hardware and adds computational overhead.
LayerNorm Activations: RepQ-ViT first applies channel-wise quantization to post-LayerNorm activations, then reparameterizes it into layer-wise quantization by adjusting LayerNorm factors and subsequent weights.The reparameterized form retains the robustness of channel-wise quantization while enabling efficient layer-wise inference.
Softmax Activations: Post-Softmax activations follow an unbalanced power-law distribution, with 99.2% below 0.3 and sparse larger values that encode important patch correlations.These larger values are not treated as disposable outliers because they guide greater attention between patches.
Softmax Activations: The method transforms a log√2 quantizer for post-Softmax activations into a log2 quantizer, preserving the former’s accuracy while enabling bit-shifting inference with slight overhead.The reparameterized scale supports efficient de-quantization, including parity computation via the least significant bit on FPGAs.

4. Experiments

Experiments across ImageNet and COCO evaluate RepQ-ViT on low-bit classification, detection, and segmentation, including ablations of its scale reparameterization and calibration efficiency.

Experimental setup: RepQ-ViT is evaluated on ImageNet classification with ViT, DeiT, and Swin variants, and on COCO detection and instance segmentation with Swin-backed Mask R-CNN frameworks.
Experimental setup: 32 ImageNet samples and 1 COCO sample are used for calibration, with no reconstruction procedures required.Inference uses channel-wise weight quantization and layer-wise activation quantization.
ImageNet results: 81.27% and 82.79% accuracy are achieved for DeiT-B and Swin-S in W6/A6 quantization, with 0.53% and 0.44% losses, respectively.The quantized model size is compressed by 5.3 times.
COCO results: 12.4 box AP and 13.4 mask AP gains over APQ-ViT are obtained for Swin-T Mask R-CNN under W4/A4 quantization.For Cascade Mask R-CNN, the corresponding gains are 19.8 box AP and 17.0 mask AP.
COCO results: 50.0 box AP and 43.5 mask AP are achieved for Swin-T Cascade Mask R-CNN under W6/A6, only 0.4 and 0.2 below full precision.With Swin-S, the same framework achieves 51.4 box AP and 44.6 mask AP.
Ablation studies: 70.28% accuracy is obtained with channel-wise post-LayerNorm quantization, versus 33.17% with direct layer-wise quantization, while reparameterization loses 1.25%.The reparameterized version supports hardware-friendly inference while retaining high accuracy.
Ablation studies: 1.58% accuracy improvement over simple log2 quantizers is obtained by log√2 quantizers for Swin-S post-Softmax activations.Scale reparameterization preserves the same accuracy as log2 quantizers with only slight additional inference overhead.

5. Conclusions

RepQ-ViT decouples quantization from inference by bridging complex distribution-preserving quantizers and simple hardware-friendly quantizers through scale reparameterization. Experiments show strong low-bit performance, while future work targets broader activation coverage and improved power-law modeling.

5. Conclusions: RepQ-ViT initially uses channel-wise quantization for post-LayerNorm activations and log√2 quantization for post-Softmax activations, then reparameterizes them for inference.The corresponding inference quantizers are layer-wise and log2 quantization, respectively.
5. Conclusions: The framework uses complex quantizers during quantization and simple hardware-friendly quantizers during inference, connected by scale reparameterization.
5. Conclusions: RepQ-ViT significantly outperforms existing methods in low-bit quantization.
5. Conclusions: Future work can extend channel-wise-to-layer-wise reparameterization to more activations and combine log√2 with log2 quantization for better power-law description.