Source-linked AI summary
Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, Guodong Guo
TL;DR
Large vision transformers are expensive to deploy, and low-bit quantization can substantially reduce performance. Q-ViT addresses this problem with information rectification and distribution-guided distillation, achieving competitive fully quantized DeiT and Swin models with ultra-low-bit weights and activations.
Problem
Low-bit vision transformers remain underexplored, while ultra-low-bit quantization can cause substantial performance loss compared with full-precision models.
Method
Q-ViT retains full-precision attention distributions using IRM for entropy-based information rectification and DGD for attention-similarity distillation.
Results
Q-ViT achieves comparable performance with full-precision counterparts across DeiT and Swin backbones while using ultra-low-bit weights and activations.
Takeaways & Limitations
Q-ViT provides a promising path toward extreme compression of vision transformers with competitive performance.
Takeaways & Limitations
The method assumes that query and key distributions in ViT architectures follow Gaussian distributions under distilling supervision.
Abstract
from arXiv · showhide
The large pre-trained vision transformers (ViTs) have demonstrated remarkable performance on various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. Among the powerful compression approaches, quantization extremely reduces the computation and memory consumption by low-bit parameters and bit-wise operations. However, low-bit ViTs remain largely unexplored and usually suffer from a significant performance drop compared with the real-valued counterparts. In this work, through extensive empirical analysis, we first identify the bottleneck for severe performance drop comes from the information distortion of the low-bit quantized self-attention map. We then develop an information rectification module (IRM) and a distribution guided distillation (DGD) scheme for fully quantized vision transformers (Q-ViT) to effectively eliminate such distortion, leading to a fully quantized ViTs. We evaluate our methods on popular DeiT and Swin backbones. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, our Q-ViT can theoretically accelerates the ViT-S by 6.14x and achieves about 80.9% Top-1 accuracy, even surpassing the full-precision counterpart by 1.0% on ImageNet dataset. Our codes and models are attached on https://github.com/YanjingLi0202/Q-ViT
1 Introduction
Vision Transformers offer strong visual performance but are costly to deploy, while ultra-low-bit quantization remains underexplored and can cause severe accuracy loss. Q-ViT addresses this through information rectification and distribution-guided distillation, achieving competitive performance with full-precision models.
- ViT-H contains 632M parameters, uses 2528MB memory, and requires 162G FLOPs, limiting deployment on resource-constrained platforms.
- Ultra-low-bit post-training quantization is ineffective because it constrains models to sub-optimized performance without fine-tuning and causes significant performance reduction.
- QAT reduces performance loss at higher compression rates, but its application to low-bit vision transformers remains largely unexplored.
- Q-ViT retains full-precision attention distributions by using IRM during forward propagation and DGD during backward optimization.
- IRM maximizes information entropy in quantized attention representations, while DGD uses activations and similarity matrices for accurate optimization.
- Q-ViT outperforms the baseline by a large margin on ImageNet and achieves performance comparable to full-precision counterparts.
2 Related Work
Related work has established vision transformers for visual modeling and developed compression methods, including low-bit quantization, to reduce inference cost and memory use.
- Vision transformer: Vision transformers apply self-attention to capture long-distance visual relations without image-specific inductive bias.
- Vision transformer: ViT represents 16 × 16 image patches as token sequences and uses a class token for classification.
- Quantization: Quantized neural networks use 1∼4-bit weights and activations to accelerate inference and reduce memory usage.
- Quantization: Prior quantization methods reduce quantization error or accelerate training and inference through ternary or low-bit weights, activations, and gradients.
3 Baseline of Fully Quantized ViT
The baseline fully quantizes ViT representations during forward propagation and uses distillation during backward optimization, with quantized attention computed through bit-wise matrix operations.
- The baseline quantizes ViT weights and activations during forward propagation and applies distillation during backward optimization.
- Quantized ViT architecture: Neural-network quantization is introduced with asymmetric activation quantization and symmetric weight quantization.
- Quantized ViT architecture: The quantization function is applied in forward propagation, while a straight-through estimator retains gradient derivation during backward propagation.
- Quantized ViT architecture: Each transformer block contains Multi-Head Self-Attention and a Multi-Layer Perceptron, with attention depending on query, key, and value.
- Quantized ViT architecture: Quantized attention uses separate Q-Linear layers for query, key, and value computation.
- Training for Quantized ViT: Knowledge distillation bridges the performance gap between quantized models and their full-precision counterparts.
4 Proposed Q-ViT
Q-ViT addresses fully quantized ViT performance loss by correcting attention-information distortion during forward propagation and improving teacher-guided optimization during backward propagation. Its IRM maximizes quantized attention entropy, while DGD transfers attention-distribution knowledge through similarity-based distillation.
- Bottlenecks: The fully quantized baseline suffers forward information distortion in attention scores and backward distillation that provides a misleading optimization direction.The architecture bottleneck arises during forward propagation, while the optimization bottleneck arises during backward propagation.
- Bottlenecks: Quantizing query, key, value, and attention weight causes the largest accuracy drop, reaching 10.03% in the architecture analysis.Quantized MLP and MHSA linear-layer weights cause smaller drops of 1.78% and 4.26%, respectively.
- Distribution Guided Distillation: Q-ViT produces attention-distance behavior similar to the full-precision model, unlike the fully quantized baseline.The comparison uses full-precision DeiT-Small, a fully quantized baseline, and Q-ViT on the same input.
- Information Rectification in Q-Attention: IRM maximizes the entropy of quantized query and key representations to mitigate distribution degradation caused by finite-point quantization.The module reshapes query and key distributions with learnable parameters before quantization, restoring information in the attention mechanism.
- Distribution Guided Distillation: DGD distills patch-based query and key similarity matrices, supplying semantic-level supervision that improves optimization of the fully quantized ViT.The matrices use l2-normalized activations and provide supervision beyond element-level output-logit matching.
- Distribution Guided Distillation: DGD retains the query and key distributions of the full-precision counterparts in Q-ViT.The paper analyzes these distributions using query and key histograms and Gaussian probability-density curves.
5 Experiments
Experiments on ImageNet use DeiT and Swin backbones to evaluate Q-ViT, its components, and low-bit baselines. Q-ViT improves accuracy over matched baselines while enabling substantial compression across model sizes and bit widths.
- Experimental setup: ImageNet experiments evaluate Q-ViT with DeiT-S, DeiT-B, Swin-T, and Swin-S backbones against low-bit baselines, LSQ, and 8-bit post-training quantization.The study reports both component ablations and main quantization results.
- Ablation study: 3.8% improvement results when IRM and DGD are combined on the 2-bit ViT-S baseline.The baseline drops by 0.2%, 2.1%, and 11.7% at 2, 3, and 4 bits, while each component improves performance individually.
- Main results: 80.9% Top-1 accuracy is achieved by 4-bit Q-ViT on DeiT-S, compared with 78.1% for 8-bit VT-PTQ.The 2-bit DeiT-S model also compresses FLOPs by 21.5×, while improvements over matched 2/3/4-bit baselines are 3.9%, 1.5%, and 1.2%.
- Main results: 21×, 12×, and 7.6× FLOPs compression is reported for 2-, 3-, and 4-bit DeiT-B Q-ViT, respectively.The corresponding improvements over matched baselines are 3.8%, 1.7%, and 1.9%.
- Main results: 1.3% and 1.1% gains over full precision are reported for 4-bit Q-ViT on Swin-T and Swin-S, respectively.For Swin-S, the model also achieves 7.9× compression, while improvements over matched baselines span 4.3%, 1.8%, and 1.5%.
6 Conclusion
The paper presents Q-ViT as a fully quantized vision transformer that combines analysis of quantization bottlenecks with IRM and DGD to improve accuracy under extreme compression. Its models achieve competitive performance with full-precision counterparts using ultra-low-bit weights and activations.
- Conclusion: Q-ViT combines a theoretical analysis of fully quantized ViTs with Information Rectification Module and Distribution Guided Distillation methods.The methods target the central issues identified in fully quantized ViT training.
- Conclusion: Q-ViT achieves comparable performance with full-precision counterparts while using ultra-low-bit weights and activations.The paper presents this result as a path toward extreme compression of vision transformers.