PaperlayerView exact revision on arXiv

Source-linked AI summary

GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, Andreas Moshovos

arXiv:2005.03842v2cs.LGcs.ARstat.ML

TL;DR

Attention-based NLP models are memory-bound by their large parameter sets, making efficient execution difficult. GOBO quantizes most parameters to 3 bits without retraining; it maintains MNLI accuracy while reducing model footprint and improving TPU performance.

  • Problem

    Attention-based language models are memory-bound by their numerous parameters, which dominate their footprint, accesses, and execution time.

    Source: §1 · p. 1

  • Method

    GOBO post-training quantizes a trained attention-based model’s parameters, representing the vast majority with 3 or 4 bits without retraining or fine-tuning.

    Source: §1 · p. 1

  • Results

    GOBO maintains MNLI accuracy with 99.9% of weights at 3 bits, reduces off-chip model footprint by 10× and yields 10× TPU performance; under iso-compute-area constraints, its accelerator is 7× faster and uses 3× less energy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GOBO’s plug-in-compatible off-chip compression boosts energy efficiency and off-chip capacity.

    Source: §8 · p. 13

Abstract

from arXiv · show

Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art BERT models and their variants to 3 bits while maintaining their accuracy. Unlike other quantization methods, GOBO does not require fine-tuning nor retraining to compensate for the quantization error. We present two practical hardware applications of GOBO. In the first GOBO reduces memory storage and traffic and as a result inference latency and energy consumption. This GOBO memory compression mechanism is plug-in compatible with many architectures; we demonstrate it with the TPU, Eyeriss, and an architecture using Tensor Cores-like units. Second, we present a co-designed hardware architecture that also reduces computation. Uniquely, the GOBO architecture maintains most of the weights in 3b even during computation, a property that: (1) makes the processing elements area efficient, allowing us to pack more compute power per unit area, (2) replaces most multiply-accumulations with additions, and (3) reduces the off-chip traffic by amplifying on-chip memory capacity.

1 INTRODUCTION

Attention-based NLP models are memory- and energy-intensive, and existing compaction methods can require fine-tuning or sacrifice accuracy. GOBO compresses trained models without refinement and supports both memory-focused inference and specialized computation.

GOBO approach

  • GOBO takes a trained model, preserves accuracy without retraining or fine-tuning, and decodes to the original transformer architecture with floating-point parameters.It can reduce model footprint, traffic, and energy, or enable specialized hardware that replaces most multiplications with additions.Source: §1 · p. 1

GOBO method

  • GOBO stores typical layer-wise Gaussian outliers as-is and represents the remaining weights with typically 8 centroids, encoding typically 99.9% as 3-bit indexes.The method uses a novel representative-value selection algorithm.Source: §1 · p. 2

Experimental findings

  • On MNLI, GOBO maintains accuracy while quantizing 99.9% of weights to 3 bits.MNLI is identified as the most challenging GLUE task.Source: §1 · p. 2
  • GOBO’s off-chip memory compression reduces model footprint by 10× and translates to 10× performance for TPU.Source: §1 · p. 2
  • The centroid-selection algorithm converges 9× faster than K-Means and consistently reduces the required centroids by half.Source: §1 · p. 2
  • Under iso-compute-area constraints, the GOBO accelerator is 7× faster on average and consumes 3× less energy than an accelerator with TensorCore-like units.Source: §1 · p. 2

2 THE BERT FAMILY OF NLP MODELS

BERT is a widely used attention-based model for NLP, and the section outlines its variants, evaluation tasks, architecture, and weight distributions that motivate GOBO’s outlier-aware quantization.

BERT-Large and BERT-Base

  • BERT uses pre-training followed by task-specific fine-tuning; BERT-Large typically achieves higher accuracy with 3.5× more parameters than BERT-Base.The passage reports that training BERT from scratch in under an hour required 1,472 V100 GPUs, each with 32GB of memory and 450W power consumption.Source: §2 · p. 2

Tasks

  • The study centers on MNLI, the most quantization-sensitive GLUE task, and also evaluates STS-B, SQuAD [15] on available fine-tuned models, and HAT translation.MNLI is used for all models; single-GPU fine-tuning for SQuAD was impractical, so SQuAD evaluation uses available fine-tuned BERT-Large and SpanBERT models.Source: §2 · p. 2

BERT Architecture

  • BERT-Base has 12 layers, 73 fully connected layers, and 110M parameters; BERT-Large has 24, 145, and 340M, respectively.Both use 32-bit floating-point activations and weights; weights dominate memory because BERT consists mostly of fully connected layers.Source: §2 · pp. 2–3

Per Layer Weight Distribution

  • BERT layer weights are Gaussian-like, with a tiny fraction of fringe weights having much larger magnitudes than the rest.The observed distribution motivates separating typical weights from outliers.Source: §2 · p. 3

BERT Derivatives

  • BERT variants include DistilBERT and RoBERTa [10], while Q8BERT and Q-BERT are quantized BERT methods used for comparison.DistilBERT uses knowledge distillation for a smaller architecture; RoBERTa changes training and embeddings while maintaining BERT’s architecture.Source: §2 · p. 3

Per Layer Weight Distribution

  • GOBO defines the Gaussian group as weights within 99.9% of the layer distribution and treats values outside it as outliers.The paper reports that precisely representing outliers while quantizing the rest to a few representative values (e.g., 8) maintains accuracy; using representative values for all weights either reduced compression or sacrificed accuracy.Source: §2 · p. 3

3 RELATED WORK

The related work distinguishes quantization, pruning, and distillation, positioning GOBO as a quantizer that compresses weights without fine-tuning while maintaining accuracy.

Model Quantization

  • NLP compression spans quantization, which reduces parameter bits; pruning, which zeros weights; and distillation, which trains a smaller student to mimic a teacher.Quantization may map weights directly to fixed-point or indirectly to dictionary indexes.Source: §3 · p. 3
  • GOBO avoids the fine-tuning required by Q8BERT [4] and Q-BERT [3], and Section 4.3 reports greater compression than each while maintaining accuracy.Q8BERT quantizes weights and activations to 8-bit fixed-point values, while Q-BERT uses per-layer-group dictionaries and 8-bit embeddings.Source: §3 · p. 3

Model Pruning

  • Gordon et al. found 30%-40% magnitude-based BERT pruning had minimal accuracy impact [19], but comparable to GOBO’s nearly 10× compression requires removing nearly 90% of weights, ignoring encoding overhead.Pruning requires fine-tuning to compensate for accuracy loss; GOBO-plus-pruning is identified as future work.Source: §3 · p. 3

Knowledge Distillation

  • Distillation trains students to mimic teachers through encoder behavior, fewer attention heads [23] [24] [25] or layers [26], logits, or alternative networks such as BiLSTM [28] and CNN [30].Structured pruning can remove model components, including attention heads [20] [21] or encoder units [22].Source: §3 · pp. 3–4
  • DistilBERT is about half the size of BERT-Base, and GOBO compresses it by 10× into a model 20× smaller than BERT-Base.Source: §3 · p. 4

4 GOBO QUANTIZATION

GOBO preserves a small fraction of outlier weights in FP32 and quantizes the remaining Gaussian weights using per-layer representative values. It combines this representation with distribution-aware outlier detection and iterative clustering, achieving substantial compression with limited accuracy loss across evaluated models and tasks.

4.1 Outlier Detection

  • GOBO identifies outliers in fine-tuned layers by modeling weights with a Gaussian distribution and selecting weights below a log-probability threshold of -4.It keeps outliers in FP32; typically fewer than 0.1% of weights are outliers, while Gaussian-group weights use 3b indexes into per-layer representative values.Source: §4.1 · p. 4, §4 · p. 4

4.2 “G” Group Weight Quantization

  • GOBO’s equal-population clusters concentrate resolution in dense weight regions, and its iterative L1-based reassignment converges in 7 iterations for 3b quantization.This initialization alone degrades BERT-Base GLUE accuracy by 10%; GOBO’s iterative method converges 9× faster than K-Means, and terminating on L1 yields higher inference accuracy using L2.Source: §4.2 · pp. 4–5

4.3 Accuracy and Compression Potential

  • With 3b BERT-Base MNLI weights, 0.1% outliers reduce accuracy loss from 8.3% to less than 1%, while 10% outliers improve it by only 0.25%.The outlier fraction is swept from 0% to 100% in Figure 3.Source: §4.3 · p. 5, Figure 3 · p. 6
  • On BERT-Base MNLI, GOBO achieves 9.8× compression with less than 0.7% accuracy loss, or 7.92× compression with no accuracy loss.Q-BERT achieves 6.5× compression at 0.56% accuracy loss or 7.81× at 1.04% loss; Q8BERT achieves 4× at 0.7% loss. GOBO quantization takes about 10 minutes, versus 9 hours for baseline fine-tuning and 102 hours for Q8BERT fine-tuning.Source: §4.3 · p. 5
  • For BERT-Base, GOBO’s 3b quantization loses 0.69% accuracy versus 1.36% for K-Means; on STS-B, GOBO loses none at 3b, while K-Means needs 4b and linear quantization 5b.GOBO maintains baseline accuracy at 4b, compared with 5b for K-Means and 6b for linear quantization; linear quantization incurs nearly 52% error at 3b.Source: §4.3 · pp. 5–6
  • For BERT-Large, GOBO loses less than 1% accuracy on SQuAD at 3b and none at 4b, and loses none on MNLI at 3b.DistilBERT has no or less than 1% accuracy loss at 4b or 3b, respectively, with models 16× or 21× smaller than BERT-Base; K-Means needs twice as many bins. RoBERTa loses 8% at 3b, but targeted 4b quantization of sensitive layers limits loss to 1.4%; RoBERTa-Large loses less than 1% with targeted 4b quantization.Source: §4.3 · p. 6
  • On HAT’s WMT’14 English-to-French task, GOBO scores 0.4 higher BLEU than K-Means, while adding less than 1% outlier footprint; embedding quantization throughout maintains accuracy with 4b embeddings but loses 0.2% at 3b.GOBO achieves 5.31× compression on SpanBERT, matching SQuADv2 baseline accuracy at 3b and incurring less than 1% error on SQuADv1.1; quantizing HAT embedding tables raises compression to 7.8×.Source: §4.3 · p. 6

5 MEMORY COMPRESSION

GOBO compresses off-chip weights while preserving sequential access, reducing memory traffic and energy while increasing effective memory capacity. Its 16×16 submatrix format handles outliers and enables efficient decompression.

5 MEMORY COMPRESSION

  • GOBO’s off-chip format supports sequential streaming of layer weights, reducing memory traffic and energy while increasing effective memory capacity.Weights are accessed sequentially in chunks large enough to maximize reuse, so the format need not support random accesses.Source: §5 · p. 6
  • GOBO stores weight indexes in original order at 3b per weight, including outliers, whose separate values replace dummy indexes without runtime re-alignment.Each 16×16 submatrix contains 256 weights divided into 16 blocks; outliers are encoded in submatrix and block order as block and weight offsets plus an FP32 value.Source: §5 · pp. 7–8
  • A decompression engine uses two sequential streams and lookup tables to produce FP32 weights, with outliers selectively overwriting centroid values.Because outliers are rare, the engine can process at most one per cycle; the header supplies metadata and the centroid table used to configure its lookup tables.Source: §5 · pp. 7–8
  • Submatrices of 256 weights or more achieve nearly maximum possible compression for weight quantizations from 2b to 6b (Figure 5b).The comparison uses a no-outlier representation as the measure of maximum possible compression.Source: §5 · p. 8
  • Changing dataflow without changing the layout requires only storing outlier counts and values in separate linear arrays, with an extra reference to locate each submatrix’s outliers.Cumulative counts index the outlier array, and consecutive count entries give the number of outliers in a submatrix.Source: §5 · p. 8

6 COMPUTE ACCELERATION

GOBO accelerates fully connected layers by accumulating activations according to quantized weight indices, then multiplying the per-index sums by their centroids. Its hardware implements this two-phase computation while supporting weight-index reuse and wider dictionaries.

6 COMPUTE ACCELERATION

  • GOBO accumulates each activation into a sum selected by its weight index, then multiplies those sums by the corresponding centroids.This replaces per-weight centroid decoding and multiply-accumulation with activation accumulation per index followed by a few centroid MACs.Source: §6 · p. 8
  • 96× fewer MAC operations: in a 768 × 768 FC layer with 3b weights, GOBO uses 8 centroid lookups and 8 MACs instead of 768 of each per output activation.It performs 768 activation accumulations across 8 accumulators before centroid processing.Source: §6 · pp. 8–9
  • Each GOBO tile uses 16 processing elements with an FP32 adder and 8-entry register file each, plus a shared unit for centroid MACs and outliers.Processing has an activation-accumulation phase followed by centroid processing; the latter takes 128 cycles, during which the PEs are idle.Source: §6 · p. 9
  • GOBO’s rearranged arithmetic uses 8 separate accumulators and the experiments demonstrate that it preserves accuracy.Accumulating before multiplying with each weight delays multiplication until the relevant activations have been accumulated.Source: §6 · p. 9
  • Pairing adjacent tiles supports 4b weight indexes, while multiplexing across multiple input words produces partial sums and reuses weights across words.The paired tiles’ output entries are added over 16 cycles; the described dataflow processes groups of weight columns, and rows when beneficial.Source: §6 · p. 10

7 EVALUATION

The evaluation measures GOBO’s memory-compression benefits and compares its accelerator with Tensor Cores and IBM DLP in performance and energy efficiency. GOBO improves both, while its gains vary with architecture and precision.

7.1 Memory Compression

  • With memory compression, TPU performance improves nearly 10×, Eyeriss speedup is nearly 7× on average, and Tensor Cores performance improves nearly 4×.Tensor Cores’ weights account for 84% of memory transactions; GOBO reduces their footprint by nearly 10×.Source: §7.1 · p. 11, Figure 7 · p. 12
  • Memory compression reduces energy consumption by 3.7× on average for Eyeriss and 1.6× on average for Tensor Cores, while TPU energy improves by 7%.For Tensor Cores, GOBO reduces off-chip memory’s share of total energy from 47% to 20%.Source: §7.1 · p. 11, Figure 7 · p. 12

7.2 GOBO Hardware Accelerator

  • GOBO is 7× faster than Tensor Cores for BERT-base and DistilBERT, and 21× more energy efficient than Tensor Cores on BERT-Base.These are results for the FP32 comparison; GOBO’s speedup is slightly lower for RoBERTa because some layers use 4-bit weights.Source: §7.2 · pp. 11–12, Figure 7 · p. 12, Table 12 · p. 12
  • GOBO’s FP32 accelerator is 7× faster than Tensor Cores for BERT-base and DistilBERT, and its speedup over Tensor Cores on large models caps at 5.5× with a 2MB buffer.It fits 768 GOBO tiles in the compute area of 128 Tensor Cores; encoded weights improve buffer use, while replacing multipliers with accumulators and register files enables more compute units per area.Source: §7.2 · p. 11, Figure 7 · p. 12, Table 10 · p. 12
  • Against IBM DLP, GOBO is 2.85× faster and uses 6.8× less energy; IBM DLP+ is 2.2× faster than DLP.GOBO’s tiles are 1.2× smaller than DLP’s, allowing 20% more tiles; PE utilization is 83% for GOBO and 75% for DLP+.Source: §7.2 · p. 12, Figure 7 · p. 12
  • With FP16, GOBO is 3.3× faster than Tensor Cores and uses 2.4× less energy; GOBO compression makes Tensor Cores 2.8× faster and 1.6× more energy efficient than Tensor Cores alone.Against IBM DLP, GOBO compression raises PE utilization from 34% to 75%; GOBO reaches 83%.Source: §7.2 · p. 12, Figure 7 · p. 12

8 CONCLUSION

GOBO is a post-training quantization method for attention-based NLP models that reduces model size with little or no effect on accuracy. It supports a plug-in compression approach and a dedicated accelerator that keeps weights quantized throughout computation.

8 CONCLUSION

  • GOBO reduces attention-based NLP model size, including parameters and embedding tables, with little or no effect on accuracy.It is a post-training quantization method.Source: §8 · pp. 12–13
  • A plug-in-compatible application uses GOBO to compress off-chip data, boosting energy efficiency and off-chip capacity.Source: §8 · p. 13
  • A dedicated accelerator uses GOBO quantization throughout and never expands weights into their original values directly.Source: §8 · p. 13

Related papers

Loading 2005.03842v2…