PaperlayerView exact revision on arXiv

Source-linked AI summary

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference

Zhikai Li, Qingyi Gu

arXiv:2207.01405v4cs.CV

TL;DR

ViTs are costly to deploy, and existing integer-only pipelines do not directly support their nonlinear operations. I-ViT quantizes the full computational graph with lightweight integer methods, achieving similar or slightly higher accuracy and 3.72∼4.11× speedup over FP inference on an RTX 2080Ti GPU.

  • Problem

    ViT nonlinear operations such as Softmax, GELU, and LayerNorm cannot naively use CNN-oriented dyadic integer-only pipelines, limiting fully integer inference.

  • Method

    I-ViT quantizes the entire ViT computational graph, applying dyadic arithmetic to linear operations and lightweight integer-only approximations to nonlinear operations.

  • Results

    3.72∼4.11× speedup over the FP model was achieved on an RTX 2080Ti GPU, with similar or slightly higher accuracy across various benchmarks.

  • Takeaways & Limitations

    I-ViT supports practical integer-only ViT inference that can exploit GPU integer arithmetic units while preserving benchmark accuracy.

  • Takeaways & Limitations

    The reported future work targets dedicated integer-only hardware and extension to object detection and semantic segmentation, leaving those deployment settings outside the demonstrated scope.

Abstract

from arXiv · show

Vision Transformers (ViTs) have achieved state-of-the-art performance on various computer vision applications. However, these models have considerable storage and computational overheads, making their deployment and efficient inference on edge devices challenging. Quantization is a promising approach to reducing model complexity, and the dyadic arithmetic pipeline can allow the quantized models to perform efficient integer-only inference. Unfortunately, dyadic arithmetic is based on the homogeneity condition in convolutional neural networks, which is not applicable to the non-linear components in ViTs, making integer-only inference of ViTs an open issue. In this paper, we propose I-ViT, an integer-only quantization scheme for ViTs, to enable ViTs to perform the entire computational graph of inference with integer arithmetic and bit-shifting, and without any floating-point arithmetic. In I-ViT, linear operations (e.g., MatMul and Dense) follow the integer-only pipeline with dyadic arithmetic, and non-linear operations (e.g., Softmax, GELU, and LayerNorm) are approximated by the proposed light-weight integer-only arithmetic methods. More specifically, I-ViT applies the proposed Shiftmax and ShiftGELU, which are designed to use integer bit-shifting to approximate the corresponding floating-point operations. We evaluate I-ViT on various benchmark models and the results show that integer-only INT8 quantization achieves comparable (or even slightly higher) accuracy to the full-precision (FP) baseline. Furthermore, we utilize TVM for practical hardware deployment on the GPU's integer arithmetic units, achieving 3.72$\sim$4.11$\times$ inference speedup compared to the FP model. Code of both Pytorch and TVM is released at https://github.com/zkkli/I-ViT.

1. Introduction

I-ViT addresses the deployment costs of ViTs by quantizing their full computational graph for integer-only inference. It combines lightweight integer approximations for nonlinear operations with hardware deployment that delivers substantial speedups while preserving accuracy.

  • ViTs have higher memory, computation, and power costs than CNNs, hindering deployment and real-time inference on resource-constrained edge devices.
  • Dyadic integer-only pipelines support linear operations but do not directly apply to ViT nonlinearities such as Softmax, GELU, and LayerNorm.Leaving these operations in floating point introduces mixed-unit communication costs and limits inference speedup.
  • I-ViT quantizes the entire ViT computational graph, using dyadic arithmetic for linear operations and lightweight integer-only approximations for nonlinear operations.Shiftmax and ShiftGELU use bit-shifting, while I-LayerNorm computes square roots through integer iterations.
  • Shiftmax and ShiftGELU use integer bit-shifting to perform most nonlinear-operation arithmetic and exploit efficient hardware logic.
  • 3.72∼4.11× speedup over the FP model was achieved on an RTX 2080Ti GPU using TVM, while accuracy remained similar or slightly higher.The evaluation covered various models on the large-scale classification task.

2. Related Works

ViTs deliver strong vision performance but incur substantial memory and computation costs, motivating compression for practical deployment. Existing quantization and integer-only approaches remain limited because inference may retain floating-point operations or fail on ViT non-linearities.

  • Vision Transformers: ViTs achieve strong performance across classification, detection, segmentation, and video recognition, but their large architectures hinder resource-constrained deployment.The attention mechanism provides global receptive fields, while the resulting memory footprint and computational overhead motivate compression.
  • Model Quantization: Model quantization compresses models by converting floating-point parameters to low-precision values in a hardware-friendly manner.Prior work explores trainable clipping and step sizes, straight-through estimation, non-uniform, channel-wise, and mixed-precision strategies.
  • Model Quantization: ViT-specific quantization methods address attention maps, learnable bit-widths and scales, Hessian-guided scaling, and powers-of-two or logarithmic quantization.These methods target ViTs’ distinctive structures, including LayerNorm and Softmax.
  • Integer-only Quantization: Existing ViT quantization approaches still perform all or part of inference with dequantized floating-point parameters, limiting use of low-precision arithmetic units and model acceleration.This limitation prevents the approaches from fully exploiting efficient integer hardware.
  • Integer-only Quantization: Dyadic arithmetic enables integer-only inference for linear and piecewise-linear operations but does not directly apply to ViT non-linearities because it relies on homogeneity.Language-model methods use L1 LayerNorm or integer polynomial approximations, but high-order polynomials are inefficient and language-model distributions mismatch ViTs.
  • Integer-only Quantization: I-ViT addresses the gap with an integer-only computational graph combining dyadic linear operations with Shiftmax, ShiftGELU, and I-LayerNorm.Figure 3 describes integer-only processing, with INT8 data streams except for labeled INT32 streams.

3. Methodology

I-ViT quantizes ViT computation end to end: linear layers use dyadic integer arithmetic, while nonlinear components use lightweight integer-only approximations. Shiftmax and ShiftGELU rely mainly on bit-shifting, and I-LayerNorm uses bounded integer iterations for square-root computation.

  • 3.1. Overview: I-ViT targets the entire ViT computational graph, including multi-head self-attention and MLP modules, for integer-only inference.The scheme applies to the full block structure rather than only selected linear layers.
  • 3.3. Integer-only Softmax: Shiftmax: Because Softmax does not satisfy the homogeneity needed by dyadic arithmetic, I-ViT approximates it with Shiftmax using max-centering, base-2 conversion, and integer division.The exponential approximation uses binary approximations and bit-shifting, while scale factors can be removed through fraction reduction.
  • 3.2. Dyadic Arithmetic for Linear Operations: Linear operations such as embedding Conv, MatMul, and Dense follow dyadic arithmetic with integer multiplication and bit-shifting for requantization.The pipeline converts floating-point rescaling into dyadic numbers represented through positive integer multiplication and right shifts.
  • 3.3. Integer-only Softmax: Shiftmax: Shiftmax performs nearly all arithmetic with bit-shifting plus a small number of integer operations, while approximating only selected mathematical steps.The method uses one integer subtraction, summation, and division, with other transformations remaining equivalent.
  • 3.4. Integer-only GELU: ShiftGELU: ShiftGELU approximates GELU through a sigmoid formulation, representing 1.702 multiplication with integer shifts before integer division and rescaling.The binary approximation gives Ip = Ix + (Ix ≫ 1)+(Ix ≫3)+(Ix ≫4).
  • 3.5. Integer-only LayerNorm: I-LayerNorm: I-LayerNorm computes the required square root by integer iteration, using ten iterations to provide most convergence while ensuring constant latency.The stopping rule is changed from convergence-based termination to a fixed iteration count for hardware implementation.

4. Experiments

Experiments evaluate I-ViT across ImageNet models and RTX 2080Ti deployment, comparing accuracy, latency, and alternative approximations. I-ViT preserves accuracy while enabling integer-only inference and substantial acceleration, with benefits varying by approximation and hardware setting.

  • Evaluation Setup: I-ViT is evaluated on ViT, DeiT, and Swin models for ImageNet classification, with PyTorch quantization-aware fine-tuning and TVM-based deployment.Weights are initialized from pretrained floating-point models and fine-tuned using naive STE.
  • Latency Evaluation: 3.72∼4.11× speedup over the FP model is achieved on an RTX 2080Ti using TVM and Turing Tensor Cores.The comparison covers end-to-end latency for integer-only I-ViT inference and floating-point or alternative integer methods.
  • Accuracy Evaluation: 80.12% Top-1 accuracy on DeiT-S is achieved with 8-bit integer-only inference, 0.27% higher than the FP baseline.Across multiple ImageNet benchmarks, I-ViT maintains comparable or slightly higher accuracy than the FP baseline.
  • Latency Evaluation: 3.45× and 3.53× acceleration are obtained by FasterTransformer for DeiT-T and DeiT-S, respectively, below I-ViT's reported speedup range.FasterTransformer retains floating-point non-linear operations and incurs both floating-point-unit inefficiency and integer–floating-point data interaction overhead.
  • Ablation Studies: 0.95% and 1.15% accuracy losses result when Shiftmax and ShiftGELU are replaced by second-order polynomial approximations for DeiT-B and Swin-S.Polynomial GELU alone reduces Top-1 accuracy by 0.86% for DeiT-B and 0.91% for Swin-S compared with ShiftGELU.
  • Ablation Studies: I-ViT maintains a constant acceleration effect for DeiT-S across various batch sizes, although increasing batch size increases latency because full parallelism is unavailable.The authors identify non-optimal TVM and Turing Tensor Core support as a hardware and software boundary.

5. Conclusions

I-ViT is presented as an integer-only quantization scheme that covers the full ViT computational graph. Its bit-shifting approximations and deployment results support comparable accuracy and faster inference, while future work targets dedicated hardware and additional vision tasks.

  • Conclusions: I-ViT is presented as the first integer-only quantization scheme for ViTs, quantizing the entire computational graph for inference without floating-point operations.Linear operations use the dyadic arithmetic pipeline, while non-linear operations use lightweight integer-only approximations.
  • Conclusions: Shiftmax and ShiftGELU perform most arithmetic with bit-shifting to benefit from efficient hardware logic.The conclusion identifies these methods as central non-linear components of the proposed scheme.
  • Conclusions: 3.72∼4.11× speedup over the FP model is achieved on an RTX 2080Ti using TVM and Turing Tensor Cores, with similar or slightly higher accuracy across benchmarks.The reported outcome combines practical deployment latency with benchmark accuracy.
  • Conclusions: Future work considers dedicated integer-only hardware such as FPGAs and extension to object detection and semantic segmentation.These directions are stated as planned expansions beyond the evaluated classification setting.
Loading 2207.01405v4…