GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

Ali Hadi Zadeh; Isak Edo; Omar Mohamed Awad; Andreas Moshovos

Source-linked AI summary

GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, Andreas Moshovos

arXiv:2005.03842v2May 8, 2020cs.LGcs.ARstat.ML

TL;DR

Attention-based NLP models are memory-bound because their many parameters make efficient inference difficult. GOBO post-training quantizes most weights to 3 bits without fine-tuning, preserving accuracy while enabling memory compression and specialized computation. Its hardware applications reduce model footprint, improve performance, and lower energy consumption.

Problem
Attention-based NLP models are memory-bound because their massive parameter sets make inference costly in storage, traffic, computation, and energy.
Method
GOBO uses post-training, outlier-aware dictionary quantization and hardware designs that retain most weights in compact form during memory access and computation.
Results
GOBO maintains MNLI accuracy while quantizing 99.9% of weights to 3 bits, reduces model footprint by 10×, and averages 7× higher speed with 3× lower energy than a TensorCore-like accelerator under iso-compute-area constraints.
Takeaways & Limitations
GOBO provides plug-in-compatible memory compression and a specialized accelerator that reduce traffic and computation costs for attention-based NLP inference.

Abstract

from arXiv · show

Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art BERT models and their variants to 3 bits while maintaining their accuracy. Unlike other quantization methods, GOBO does not require fine-tuning nor retraining to compensate for the quantization error. We present two practical hardware applications of GOBO. In the first GOBO reduces memory storage and traffic and as a result inference latency and energy consumption. This GOBO memory compression mechanism is plug-in compatible with many architectures; we demonstrate it with the TPU, Eyeriss, and an architecture using Tensor Cores-like units. Second, we present a co-designed hardware architecture that also reduces computation. Uniquely, the GOBO architecture maintains most of the weights in 3b even during computation, a property that: (1) makes the processing elements area efficient, allowing us to pack more compute power per unit area, (2) replaces most multiply-accumulations with additions, and (3) reduces the off-chip traffic by amplifying on-chip memory capacity.

1 INTRODUCTION

Attention-based NLP models are memory- and compute-intensive, making energy-efficient execution important. GOBO post-training quantization compresses most weights to 3 bits while preserving accuracy and supports memory- and computation-efficient hardware implementations.

Motivation: Energy efficiency matters because modern platforms are constrained while deep learning workloads increasingly require computation and memory.Benefits include lower data-center energy footprint, increased mobile uptime, and more computation per unit time.
Motivation: BERT models are especially costly because their attention-layer weights dominate footprint, memory accesses, and execution time.BERT-Large has a 1.12GB footprint, while BERT-base uses 326MB; both use 32-bit floating-point inference parameters.
GOBO: GOBO accepts trained attention-based models and reduces parameter bitwidth without retraining or fine-tuning.This supports deployments where training data is unavailable or update deadlines are strict.
GOBO: GOBO stores most weights with 3 or 4 bits while preserving the original transformer architecture after decoding.It can compress off-chip and potentially on-chip data, reducing footprint, traffic, and energy while improving capacity and performance.
GOBO: 99.9% of weights can be represented as 3-bit centroid indexes, while typically fewer than 0.1% of per-layer weights are stored as outliers.GOBO stores outliers precisely and represents other weights using a small per-layer centroid dictionary, typically with eight values.
Results: 10× smaller model footprint translates to 10× TPU performance, while the GOBO accelerator averages 7× higher speed and 3× lower energy under iso-compute-area constraints.The evaluation covers BERT variants and other attention-based NLP models, including comparisons with Q8BERT and Q-BERT.

2 THE BERT FAMILY OF NLP MODELS

BERT is an attention-based model used across NLP tasks, with released base and large variants and derivatives that trade size, architecture, and accuracy. Its weights dominate memory footprint because fully connected layers stream them from off-chip memory.

BERT: BERT, or Bidirectional Encoder Representations from Transformers, is an attention-based model used for diverse NLP tasks.Applications include sentiment analysis, paraphrasing, sentence similarity detection, and question answering.
BERT: BERT-Base and BERT-Large are pretrained models, with BERT-Large typically achieving higher accuracy using 3.5× more parameters.Pretraining uses billions of unlabeled words, followed by task-specific fine-tuning for a few epochs.
Memory footprint: BERT weights dominate memory footprint and must be streamed from off-chip because BERT consists mostly of fully connected layers with relatively short hidden states.Embedding tables map raw inputs into the vectors used by the network.
BERT derivatives: BERT variants include DistilBERT, which uses knowledge distillation for a smaller similar architecture, and RoBERTa, which changes training and embeddings while retaining the architecture.The study also compares against Q8BERT and Q-BERT quantized variants.
GOBO motivation: GOBO separates each layer’s weights into Gaussian and outlier groups, quantizing the former and representing the latter precisely.The Gaussian group covers weights within 99.9% of the layer’s fitted distribution, while outliers fall outside it.

3 RELATED WORK

Prior NLP model-compression approaches include quantization, pruning, and knowledge distillation. GOBO differs from related quantization methods through post-training operation, fewer outliers, and per-layer dictionary compression for attention-based models.

Compression approaches: Model compression commonly uses quantization, pruning, or knowledge distillation.Quantization reduces parameter bitwidth while preserving architecture; pruning forces weights to zero; distillation trains a smaller student model.
Knowledge distillation: Knowledge distillation trains a smaller student that may use fewer attention heads, fewer encoder layers, or different network components.Examples include replacing attention layers with bidirectional LSTM architectures.
Quantization: Q8BERT fine-tunes weights and activations to 8-bit fixed-point values, leaving some operations in FP32.GOBO avoids fine-tuning and can perform most accelerator computations without decompressing weights.
Quantization: Q-BERT uses Hessian-guided fine-tuning, multiple per-group dictionaries, and 8-bit embeddings, whereas GOBO uses one dictionary per layer and 3-bit embeddings without fine-tuning.The paper reports higher GOBO compression than Q-BERT while maintaining accuracy.
Pruning: Pruning removes weights by forcing them to zero, and prior work found that 30%–40% of BERT weights could be pruned with minimal accuracy effect.MNLI was reported as the task most sensitive to pruning in that work.

4 GOBO QUANTIZATION

GOBO identifies rare outlier weights and represents the remaining Gaussian-distributed weights with a small nonlinear dictionary, typically using 3-bit indexes. Across evaluated attention-based models, this approach achieves substantial compression with little or no accuracy loss, while avoiding fine-tuning.

Quantization method: GOBO separates layer weights into Gaussian and outlier groups, preserving outliers in FP32 and quantizing the Gaussian group to representative values.Outliers are selected from the layer’s fitted Gaussian distribution; the Gaussian group is clustered into equal-population bins and refined iteratively.
Quantization method: Typically, less than 0.1% of weights are outliers, allowing roughly 99.9% of weights to be stored as 3-bit indexes to eight FP32 centroids.With 16 representative values, GOBO reports no accuracy loss; with eight, inference error remains below 1%.
Quantization method: GOBO’s iterative centroid procedure converges in about seven iterations for 3-bit quantization and terminates using an L1 objective while producing centroids that minimize L2 error.The method repeatedly reassigns weights to nearer centroids and updates centroids from cluster averages.
Accuracy and compression: On BERT-Base MNLI, 0.1% outliers reduce 3-bit quantization loss from 8.3% to below 1%, while increasing outliers to 10% improves accuracy by only 0.25%.The comparison sweeps the outlier fraction from 0% to 100%.
Accuracy and compression: GOBO compacts BERT-Base by 9.8× with less than 0.7% accuracy loss or by 7.92× with no accuracy loss, outperforming the reported compression-loss trade-offs of Q-BERT and Q8BERT.Q-BERT reaches 6.5× at 0.56% loss or 7.81× at 1.04% loss, while Q8BERT reaches 4× at 0.7% loss.
Accuracy and compression: Across additional models and tasks, GOBO maintains low error: 3-bit quantization gives no STS-B loss, below 1% BERT-Large SQuAD loss, and 5.31× SpanBERT compression.For BERT-Large SQuAD, 4-bit quantization produces no loss; SpanBERT 3-bit quantization matches SQuADv2 baseline accuracy and stays below 1% error on SQuADv1.1.

5 MEMORY COMPRESSION

GOBO compresses off-chip weights while preserving sequential access, reducing memory traffic and increasing effective capacity. Its layout stores most weights as low-bit indexes while handling sparse outliers separately.

Memory Compression: GOBO compresses weights in off-chip memory to reduce traffic and energy while increasing effective memory capacity.The method supports streaming accesses because layer weights are accessed sequentially in large chunks.
Memory Layout: The memory container comprises a header, quantized weights, and an outlier section.The header stores layer metadata, index width, and the centroid table; outliers retain their FP32 values and positions.
Memory Layout: GOBO stores weight indexes in the original order, using 3 bits per weight including outlier positions.Outlier indexes are ignored and replaced by values encoded in the outlier section, preserving relative weight positions.
Compression Analysis: Submatrices of 256 weights or more achieve nearly the maximum possible compression ratio across 2b–6b quantization widths.The comparison uses compression ratios with and without outlier overhead.
Flexible Access: Cumulative outlier counts can index a separate outlier array, allowing the layout to adapt when dataflow changes.The counts array C indexes the outlier array O without changing the main weight layout.

6 COMPUTE ACCELERATION

GOBO accelerates fully connected layers by accumulating activations by quantized weight index before applying centroid values. This replaces many per-weight operations with a small number of accumulator-based operations.

Compute Transformation: GOBO transforms per-weight multiply-accumulations into activation accumulation by weight index followed by centroid operations.The approach delays multiplication until activations sharing a centroid have been accumulated.
Compute Transformation: A 768 × 768 layer with 3b weights uses 8 centroid accumulators instead of 768 per-output MAC operations.The transformed computation performs 768 activation accumulations, 8 centroid lookups, and 8 MAC operations.
Compute Transformation: 96× fewer MAC operations are required in the 768 × 768, 3b example.GOBO performs 8 MAC operations after accumulation instead of 768 per-output MAC operations.
Tile Architecture: The GOBO tile uses 16 processing elements, each with an FP32 adder and an 8-entry register file.The shared processing unit includes a 16-entry output activation register file, while adjacent tiles support 4b indexes.
Outlier Processing: Outliers are processed during the accumulation phase, and extra outlier-processing cycles disable the processing elements.The shared processing unit reads outliers through a FIFO and processes multiple outliers serially when necessary.

7 EVALUATION

Evaluation measures GOBO’s memory-compression and accelerator designs using cycle-accurate simulation and postlayout hardware estimates. Results show substantial speedup and energy benefits across TPU, Eyeriss, Tensor Cores, and GOBO accelerator comparisons.

Methodology: The evaluation uses GOBO-quantized models, cycle-accurate simulation with DRAM modeling, and postlayout energy and area estimates.Implementations target a 65nm TSMC technology library at 1 GHz.
Memory Compression Evaluation: TPU performance improves nearly 10× with GOBO memory compression, while Eyeriss speedup reaches nearly 7× on average.The TPU benefits strongly because it is severely weight-memory-bound; Eyeriss has lower potential because it uses fewer MACs.
Memory Compression Evaluation: GOBO memory compression improves Eyeriss energy by 3.7× on average and Tensor Cores energy by 1.6× on average.For Tensor Cores, GOBO reduces off-chip memory’s share of total energy from 47% to 20%.
Evaluation Setup: Figure 7 compares original FP32 models with scaled FP16 configurations across BERT, DistilBERT, RoBERTa, and SpanBERT variants.The starred large-model configurations distinguish TC, TC+, and GOBO designs with different on-chip buffer sizes.
Accelerator Evaluation: Under iso-compute-area FP16 comparison, GOBO compression makes TC+ 2.8× faster than TC, while the GOBO accelerator reaches 3.3×.GOBO remains the most energy-efficient architecture in this comparison and uses 2.4× less energy than TC.

8 CONCLUSION

GOBO is a post-training quantization method that reduces model size with little or no accuracy effect. The paper applies it both as plug-in memory compression and within an accelerator that keeps weights quantized during computation.

Conclusion: GOBO significantly reduces model size, including parameters and embedding tables, with little or no effect on accuracy.The method is presented as post-training quantization for attention-based NLP models.
Conclusion: GOBO memory compression is plug-in compatible with other accelerators and boosts energy efficiency and off-chip capacity.The second application uses GOBO quantization throughout the accelerator and never directly expands weights into their original values.