PaperlayerView exact revision on arXiv

Source-linked AI summary

I-BERT: Integer-only BERT Quantization

Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, Kurt Keutzer

arXiv:2101.01321v3cs.CL

TL;DR

Transformer models are expensive to deploy, while prior Transformer quantization methods still use floating-point arithmetic. I-BERT replaces the full inference pipeline, including nonlinear operations, with integer-only computation. On RoBERTa-Base/Large, it achieves comparable or slightly higher GLUE accuracy and up to 4.00× speedup versus a floating-point baseline.

  • Problem

    Transformer models have high memory, latency, and power costs, while prior Transformer quantization methods retain floating-point inference that cannot efficiently use integer-only hardware.

  • Method

    I-BERT uses integer-only quantization with lightweight polynomial approximations for GELU and Softmax and integer computation of LayerNorm.

  • Results

    I-BERT matches or exceeds baseline accuracy on RoBERTa-Base/Large GLUE evaluations and achieves up to 4.00× speedup on a Tesla T4 GPU.

  • Takeaways & Limitations

    I-BERT demonstrates end-to-end integer-only Transformer inference without floating-point calculation while preserving GLUE accuracy relative to the baseline.

  • Takeaways & Limitations

    Deployment on lower-end microprocessors without floating-point hardware awaits improved software support for quantized neural-network models.

Abstract

from arXiv · show

Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot efficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM processors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for nonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline. Furthermore, our preliminary implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4 GPU system as compared to FP32 inference. The framework has been developed in PyTorch and has been open-sourced.

1. Introduction

Transformer models achieve strong NLP accuracy but are costly to deploy because of their size and reliance on floating-point operations. I-BERT addresses this by quantizing Transformer inference, including nonlinear operations, entirely with integers.

  • BERT-Large contains 340M parameters, illustrating the scale of modern Transformer models and their deployment challenge.
  • Quantization reduces memory footprint by representing parameters and activations with low-bit precision such as INT8 instead of FP32.Integer-only quantization can also improve inference speed on processors with low-precision integer multiplication units.
  • Floating-point inference limits efficient use of integer-only hardware and can increase latency, power demands, and chip area.The motivation is especially relevant to ARM embedded processors and edge hardware.
  • Prior integer-only methods target CNNs with linear or piece-wise linear operators and do not directly handle Transformer nonlinearities such as GELU, Softmax, and LayerNorm.
  • I-BERT introduces integer-only kernels for Transformer models, including lightweight second-order polynomial approximations for GELU and Softmax.The reported maximum approximation errors are 1.8 × 10^-2 for GELU and 1.9 × 10^-3 for Softmax.
  • I-BERT computes Embedding and MatMul with INT8 multiplication and INT32 accumulation, applies nonlinear operations to INT32 results, and requantizes them to INT8.Parameters and activations remain integer-valued throughout the computational graph.

2. Related Work

Prior Transformer quantization methods generally retain floating-point operations or require dequantization, unlike I-BERT’s integer-only approach. I-BERT extends integer-only quantization to Transformer nonlinearities and is presented as compatible with other precision schemes.

  • Quantization: Quantization represents parameters and activations with low-bit precision, but most prior work in this area focuses on CNN models.
  • Quantization schemes: Figure 1 contrasts fully floating-point simulated quantization, partially integer simulated quantization, and I-BERT’s integer-only inference.Only the proposed scheme avoids floating-point arithmetic and dequantization throughout inference.
  • Transformer quantization: Prior Transformer quantization methods use simulated quantization, performing all or part of inference in floating point and dequantizing values for those operations.
  • Compatibility: I-BERT’s method is complementary to mixed-precision and lower-precision quantization approaches, despite focusing on uniform quantization.
  • Integer-only inference: Existing CNN integer-only methods cannot directly apply to Transformers because GELU, Softmax, and LayerNorm are nonlinear operators.I-BERT aims to extend integer-only quantization to Transformer models without accuracy drop.

3. Methodology

I-BERT uses uniform static quantization and integer-only approximations to make Transformer inference executable without floating-point arithmetic. Its method replaces nonlinear operations with lightweight approximations while retaining integer processing throughout the computational graph.

  • Integer-only quantization: Uniform symmetric quantization maps values to low-bit integers using clipping and a scaling factor, with static scales fixed during inference.The method favors uniform quantization over non-uniform schemes because lookup tables can create hardware overhead.
  • Polynomial approximation: Second-order polynomials provide integer-only approximations for nonlinear functions because they require only addition and multiplication.The method balances approximation accuracy against the computational and memory costs of higher-order polynomials.
  • Integer-only GELU: I-BERT replaces GELU’s floating-point sigmoid-based approximation with i-GELU, which closely matches GELU near the origin and has maximum error 1.8 × 10−2.Its average error is 8.2 × 10−3, compared with 3.1 × 10−2 for h-GELU; h-GELU can cause accuracy degradation of up to 2.2 percentage points.
  • Integer-only LayerNorm: LayerNorm computes its runtime statistics dynamically and obtains the standard deviation using an integer square-root algorithm.The input mean and standard deviation vary rapidly for NLP tasks, making runtime calculation necessary.

4. Results

I-BERT was evaluated on RoBERTa models using GLUE accuracy, INT8 latency deployment, and GELU ablations. It generally preserved or improved accuracy and substantially reduced inference latency, while lower-end microprocessor deployment remained dependent on improved software support.

  • Latency Evaluation: 3.08× and 3.56× faster INT8 inference was measured for BERT-Base and BERT-Large, respectively, with up to 4.00× speedup over FP32.Measurements used direct deployment on a Tesla T4 GPU with accelerated INT8 execution.
  • Latency Evaluation: Demonstrating I-BERT on lower-end microprocessors without floating-point hardware awaits improved software support for quantized neural-network models.
  • Ablation Studies: Replacing GELU with h-GELU reduced accuracy by 0.5 points on average and up to 1.1 points on RTE across the evaluated tasks.The degradation occurred on all downstream tasks except MRPC.
  • Ablation Studies: i-GELU outperformed h-GELU by 0.7 points on average and achieved comparable or slightly better accuracy than full-precision GELU.

5. Conclusions

The conclusion presents I-BERT as an integer-only Transformer quantization scheme and reports accuracy and latency benefits on evaluated models. It also identifies training-time use of the approximations as future work.

  • I-BERT performs entire Transformer inference with pure integer arithmetic by approximating nonlinear operations including GELU, Softmax, and LayerNorm.
  • I-BERT improves average GLUE score by 0.3/0.5 points over baseline for RoBERTa-Base/Large and achieves up to 4.00× speedup on a Tesla T4 GPU.
  • Future work includes evaluating the approximations during training and studying i-GELU’s performance relative to GELU.

A. Quantization Methods

The quantization methods describe uniform mappings from floating-point values to low-bit integers and explain why static quantization is used for efficient inference. Static quantization fixes activation scaling factors before deployment rather than calculating ranges at runtime.

  • Uniform quantization maps floating-point x within [x_min, x_max] to a b-bit integer q, clipping values outside the range.
  • Asymmetric quantization permits unequal clipping bounds but introduces a bias term that must be handled during computation.
  • Static quantization precomputes fixed activation ranges from training statistics, avoiding the runtime scan and overhead required by dynamic quantization.I-BERT fixes all scaling factors during inference for maximum efficiency.

B. Error Term of Eq. 3

The section explains polynomial interpolation error and notes that increasing the interpolation order generally reduces approximation error when interpolation points are properly selected.

  • The polynomial approximation exactly matches the target data at the interpolating points (x_j, f_j).
  • The interpolation error is expressed using a point ξ within the smallest interval containing the interpolation points.
  • For a properly selected set of interpolation points, increasing polynomial order generally reduces error and can guarantee a good approximation at sufficiently high order.

C. Experimental Details

I-BERT uses mixed integer precision: INT8 for matrix operations and embeddings, and INT32 for nonlinear operations to preserve accuracy.

  • INT8 MatMul operations accumulate results in INT32 precision.
  • The Embedding layer remains at INT8 precision.
  • GELU, Softmax, and LayerNorm use INT32 precision because higher precision was important to avoid accuracy degradation after quantization.

C.2. Training

The evaluation fine-tunes RoBERTa on GLUE tasks, then applies quantization-aware fine-tuning and reports the best quantized development-set accuracy.

  • RoBERTa is first trained separately on GLUE downstream tasks until achieving its best development-set result as the baseline accuracy.
  • Quantization-aware fine-tuning is then used to recover accuracy degradation caused by quantization.
  • The hyperparameter search varies learning rate, self-attention dropout, and fully-connected dropout.The learning-rate values are 5e−7, 1e−6, 1.5e−6, and 2e−6.
  • Fine-tuning runs for up to 6 epochs on larger datasets and 12 epochs on smaller datasets.
  • I-BERT accuracy is the best development-set accuracy achieved by the resulting quantized model.

C.3. Accuracy Evaluation on the GLUE Tasks

Accuracy evaluation follows GLUE’s standard task-specific metrics, averages multiple metrics within tasks, and reports both MNLI development-set variants while excluding WNLI.

  • QQP and MRPC use classification accuracy and F1 score, while STS-B uses Pearson and Spearman correlation.
  • CoLA uses Matthews Correlation Coefficient, and the remaining tasks use classification accuracy.
  • Tasks with multiple metrics are reported using the average of those metrics.
  • MNLI reports accuracy separately for MNLI-match in-domain evaluation and MNLI-mismatch cross-domain evaluation.
  • WNLI is excluded because its dataset is relatively small and its behavior is unstable.

C.4. Environment Setup for Latency Evaluation

Latency is evaluated for INT8 and FP32 BERT-Base and BERT-Large models on a single Tesla T4 GPU, without NVIDIA’s most efficient plugins.

  • TensorRT 7.2.1 deploys and tunes BERT-Base and BERT-Large in both INT8 and FP32 formats.
  • Experiments run on Google Cloud Platform with a single Tesla T4 GPU, CUDA 11.1, and cuDNN 8.0.
  • NVIDIA’s plugins optimize Transformer operations through operation fusion but cannot be modified to support I-BERT’s integer-only kernels.
  • The latency evaluation therefore omits full plugin utilization, leaving potential for further optimization.The authors estimate that plugin-based INT8 inference could provide a further ∼2× speedup.
Loading 2101.01321v3…