PaperlayerView exact revision on arXiv

Source-linked AI summary

OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization

Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, Yuhao Zhu

arXiv:2304.07493v1cs.AR

TL;DR

Large language model growth makes inference costly, while outliers undermine low-precision quantization and complicate hardware designs. OliVe uses localized outlier-victim pair quantization, sacrificing adjacent normal values to accommodate outliers in memory-aligned encodings. The resulting accelerator surpasses GOBO by 4.5× in performance and reduces energy by 4.0×, while achieving high model accuracy.

  • Problem

    LLM outliers make existing low-precision quantization less effective and require globally coordinated, hardware-costly outlier handling.

  • Method

    OliVe locally pairs outliers with adjacent normal-value victims, pruning victims to make space for outliers in memory-aligned encodings.

  • Results

    4.5× performance improvement and 4.0× energy reduction over GOBO are reported, with 4-bit PTQ accuracy loss below 1% for weights and activations.

  • Takeaways & Limitations

    OliVe extends practical Transformer PTQ to 4-bit precision and integrates with existing accelerators such as tensor cores and systolic arrays.

  • Takeaways & Limitations

    OliVe is lossy compression rather than strictly lossless compression, and existing GPU simulators do not support data compression.

Abstract

from arXiv · show

Transformer-based large language models (LLMs) have achieved great success with the growing model size. LLMs' size grows by $240\times$ every two years, which outpaces the hardware progress and makes model inference increasingly costly. Model quantization is a promising approach to mitigate the widening gap between LLM size and hardware capacity. However, the existence of outliers, values with significant magnitudes, in LLMs makes existing quantization methods less effective. Prior outlier-aware quantization schemes adopt sparsity encoding techniques to separate outliers from normal values where the process requires global coordination (e.g., a global sparsity coordination list). This incurs complex encoding/decoding hardware logics and an extra orchestration controller for the computation between outlier and normal values. As such, it is not hardware-efficient and hence only achieves sub-optimal quantization benefits. We propose OliVe, an algorithm/architecture co-designed solution that adopts an outlier-victim pair (OVP) quantization and handles outlier values locally with low hardware overheads and high performance gains. The key insight of OliVe is that outliers are important while the normal values next to them are not. Thus those normal values (called victims) can be sacrificed to accommodate outliers. This enables a memory-aligned OVP encoding scheme, which can be efficiently integrated to the existing hardware accelerators like systolic array and tensor core. As a result, OliVe-based accelerator surpasses the existing outlier-aware accelerator, GOBO, by 4.5$\times$ speedup and 4.0$\times$ energy reduction, respectively, with a superior model accuracy.

1 INTRODUCTION

LLM growth is making inference increasingly costly, while outliers limit low-precision quantization and force higher bit-widths. OliVe addresses this with localized outlier-victim pairing that preserves outliers, sacrifices adjacent normal values, and integrates with existing hardware.

  • Motivation: 240× model-size growth every two years outpaces hardware progress of 3.1×, making LLM inference increasingly challenging and costly.OPT-175B has 175 billion parameters and cannot fit in an 80GB H100 GPU.
  • Motivation: Quantization reduces inference costs through low-precision data types, but LLM outliers make existing schemes less effective.For models exceeding roughly 6 billion parameters, fewer than 0.1% outliers can strongly affect performance.
  • Quantization Challenge: Indiscriminate clipping of outliers and normal values causes significant accuracy drops, so Transformer models commonly use 8-bit or 16-bit quantization.Outlier suppression still suffers significant accuracy loss at 4-bit precision.
  • Quantization Challenge: Prior outlier-aware architectures separate outliers globally using sparsity encoding, causing unaligned memory access, complex hardware, and extra computation orchestration.GOBO and OLAccel use coordinate lists, while other designs use block-sparse indices or bitmaps.
  • OliVe: OliVe locally pairs outliers with adjacent normal-value victims, pruning victims to create space for outliers and enabling memory-aligned encoding.The approach avoids explicit sparse indexing and is designed for integration with existing accelerators.
  • Results: OliVe reaches 4-bit post-training quantization for weights and activations with accuracy loss below 1%.Its 4-bit BERT and BART PTQ accuracies outperform the 6-bit PTQ results of outlier suppression.
  • Results: 4.5× performance improvement and 4.0× energy reduction over GOBO are reported for the OliVe-based accelerator.The accelerator also improves performance over OLAccel by 3.8× and reduces energy by 2.1×.

2 MOTIVATION: ALIGNED OUTLIER

Transformer outliers are substantially more significant than CNN outliers, making low-bit quantization difficult and motivating memory-aligned outlier handling. OliVe uses adjacent normal values as victims to accommodate outliers without sparse indexing, preserving alignment and supporting efficient accelerator integration.

  • 2.1 Outlier Matters: About 99.7% of values lie within 3σ, while most tensors contain fewer than 0.5% outliers, concentrating normal values into a narrow quantization range.Values above 6σ are extremely rare, enabling higher-resolution quantization for normal values.
  • 2.1 Outlier Matters: Transformer Max σ is larger than CNN Max σ by one order of magnitude, making retaining outliers important for quantization accuracy.Large Transformer models require higher-precision quantization than CNNs because their outliers are more significant.
  • 2.2 Outlier Is Unaligned: Prior outlier-aware accelerators use coordinate lists, block-sparse indices, or bitmaps, but sparsity-based encoding causes unaligned storage and memory accesses.These designs separate outliers from normal values and require specialized hardware to manage their locations.
  • 2.2 Outlier Is Unaligned: 55% and 71% area overheads arise from GOBO and OLAccel outlier controllers, respectively, which parse indices and orchestrate outlier computation.GOBO is additionally limited to DRAM-level compression and decompression on GPUs.
  • 2.3 Outlier and Victim Analysis: OliVe prunes normal values adjacent to outliers as victims, embedding outliers in their original locations without explicit sparse indexing.The resulting OVP mechanism aligns memory accesses and remains compatible with existing accelerator architectures.
  • 2.3 Outlier and Victim Analysis: About 99% of pairs are normal-normal, around 1% are outlier-normal, and fewer than 0.06% are outlier-outlier pairs.The low outlier-outlier frequency allows OliVe to retain most outliers while sacrificing adjacent victims.
  • 2.3 Outlier and Victim Analysis: Clipping about 1% of outliers causes unacceptable BERT accuracy loss, whereas pruning random normal values has almost no loss and pruning victims has negligible loss.The experiment uses BERT-base on GLUE with remaining values kept at FP32.

3 OUTLIER-VICTIM PAIR ENCODING

OliVe encodes adjacent values as locally distinguishable outlier-victim pairs while preserving globally aligned memory access. It uses adaptive-bias floating-point representations for outliers and supports multiple normal-value data types.

  • 3.1 OVP Encoding Algorithm: Memory-aligned OVP encoding distinguishes outliers locally, avoiding global coordination while preserving aligned memory access and low hardware overhead.The encoding is globally identical but locally distinguishable for outlier and normal values.
  • 3.1 OVP Encoding Algorithm: OVP encoding groups two consecutive values, pruning the normal value in an outlier-normal pair into a victim while retaining normal-normal and outlier-outlier pair handling.The encoder reads two values simultaneously and produces an OVP encoding using an outlier threshold.
  • 3.2 Data Type for Normal Values: Normal values use adaptive data types including int4, flint4, and int8, with one representation reserved as the outlier identifier.For int8, the range narrows from [−128, 127] to [−127, 127] by reserving 10000000₂.
  • 3.3 Data Type for Outliers: Abfloat: Abfloat encodes outliers with a biased exponent so encoded values skip the normal-value interval and provide greater outlier range.Fixed-point conversion uses shift operations, reducing implementation overhead compared with floating-point representation.
  • 3.3 Data Type for Outliers: Abfloat: E2M1 minimizes rounding error for the largest outliers across BERT-base, BERT-large, BART-base, and GPT2-XL, while signed E4M3 is adopted for 8-bit abfloat.Outlier encoding disables both abfloat zero encodings to avoid conflict with the outlier identifier.

4 OLIVE ARCHITECTURE

OliVe integrates lightweight OVP decoders and mixed-precision processing into GPU tensor cores and systolic arrays. Its memory-aligned format preserves GPU programming interfaces and reduces systolic-array decoder count.

  • 4 OLIVE ARCHITECTURE: OliVe integrates OVP decoders into GPU tensor cores and output-stationary systolic arrays while supporting mixed precision and mixed data types.The architecture adds lightweight decoding and arithmetic support for the OVP representation.
  • 4.2 Decoders: The OVP decoder reads one byte containing one value pair, identifies outliers locally, and converts values into unified exponent-integer pairs.Normal int4 values receive a zero exponent, while abfloat values are decoded into exponent-integer form.
  • 4.3 Systolic Array Integration: Systolic arrays place decoders only along array borders, requiring n+m rather than n×m decoders for an n×m array.This reuses the GPU decoder design while saving most decoder instances.
  • 4.4 OliVe MAC Unit: OliVe MAC units add a shifter and adder to process exponent-integer pairs within fixed-point multiply-accumulate operations.The final accumulated result can be stored with a 32-bit integer.
  • 4.5 Mixed Precision: The design supports int8 and 8-bit abfloat multiplication by decomposing each value into high and low 4-bit components processed by four 4-bit PEs.The abfloat path includes an additional exponent relative to int8.
  • 4.5 Mixed Precision: OliVe clips abfloat outliers to avoid int32 accumulator overflow, but experiments report no practical truncation because observed outliers remain below the clipping range.The clipping boundary is 2^15, approximately 768σ after normalization and quantization, whereas observed maxima do not exceed 325σ.
  • 4.1 GPU Tensor Core Integration: Memory alignment preserves the original GPU programming interface, allowing OVP-supported frameworks to replace integer instructions with OVP instructions such as mmaovp.This enables direct integration into existing GPU quantization frameworks.

5 EVALUATION

OliVe is evaluated for accuracy, performance, energy, and area across language models, datasets, GPUs, and systolic-array accelerators. It maintains low-bit accuracy while delivering substantial acceleration and energy reductions with small hardware overhead.

  • Evaluation setup: OliVe is evaluated on BERT, BART, GPT2-XL, BLOOM-7B1, and OPT-6.7B across GLUE, SQuAD, and language-model perplexity tasks.The study compares OliVe with quantization methods and accelerator baselines including GOBO, outlier suppression, Q8BERT, ANT, OLAccel, and AdaFloat.
  • Accuracy: 4-bit PTQ loses less than 1% on BERT-base across eight GLUE datasets, around 1% on BERT-large, and around 2% on BART-base.On GLUE, OliVe outperforms the studied 4-bit, 6-bit, and 8-bit PTQ and QAT methods, including under GOBO’s weight-only setting.
  • Accuracy: 4-bit PTQ loses less than 2% on BERT-base and around 3% on BART-base on SQuAD, outperforming outlier suppression’s 6-bit PTQ.
  • Accuracy: OliVe achieves nearly original perplexity with 8-bit PTQ and performance close to int8 with 4-bit PTQ on GPT2-XL and BLOOM-7B1.On OPT-6.7B, 8-bit PTQ has only a negligible perplexity increase, while int8 quantization degrades significantly; int4 and 4-bit ANT are 10-1000× worse than FP32 on reported comparisons.
  • GPU performance and energy: 4.5× speedup over GOBO, 2.7× over int8, and 2.4× over ANT are achieved on average on GPUs.OliVe quantizes both activations and weights without increasing memory-access overhead, while GOBO uses FP16 computation and weight-only quantization.
  • Accelerator performance and area: 4.8×, 3.8×, and 3.7× speedups over AdaFloat, OLAccel, and ANT, respectively, are achieved on average in the systolic-array evaluation.The 4-bit and 8-bit decoders introduce about 2.2% and 1.5% core-area overhead, respectively.
  • Energy and area: 3.7×, 2.1×, and 3.3× energy reductions over AdaFloat, OLAccel, and ANT, respectively, are achieved on average in the systolic-array evaluation.OliVe also achieves the lowest normalized energy in the GPU comparison and introduces 0.250% 4-bit-decoder or 0.166% 8-bit-decoder GPU-area overhead.

6 RELATED WORK AND DISCUSSION

The discussion places OliVe among DNN compression and acceleration methods, emphasizing quantization’s efficiency benefits and the hardware costs of outlier-aware designs. It also identifies lossy compression and unsupported simulator capabilities as boundaries for the work.

  • DNN acceleration and compression: DNN compression methods such as pruning and quantization exploit model redundancy, while DNN acceleration targets computation, compilation, scheduling, and data reuse.
  • Pruning: Pruning can reduce model size but significant reduction produces irregular memory accesses that hinder inference and training acceleration.
  • Quantization: PTQ converts FP32 models directly to lower-bit models without training data or a training pipeline, whereas QAT retrains models to adapt to quantization noise.
  • Quantization accelerators: Low-bit quantization reduces memory bandwidth requirements and computation burden, but prior outlier-aware accelerators incur overhead from unaligned memory accesses.OliVe instead handles outliers in a memory-aligned and hardware-friendly way.
  • Floating-point representations: AdaptivFloat adapts exponent bias to layer-specific dynamic ranges, whereas OliVe’s abfloat applies a uniform bias to skip normal-value ranges.
  • GPU architecture and discussion: OliVe’s quantization is orthogonal to Ampere structured sparsity, but current GPU simulators cannot support compute data compression, which remains future work.The paper characterizes DNN quantization as lossy compression and positions OliVe as complementary to Ampere’s compute data compression.

7 CONCLUSION

OliVe uses local outlier-victim pair quantization to preserve accuracy while reducing hardware overhead, and integrates with tensor-core and systolic-array accelerators. Its accelerator outperforms GOBO in speed and energy.

  • Conclusion: OliVe sacrifices normal values adjacent to essential outliers as victims, enabling local OVP encoding with low hardware overhead and high performance gains.
  • Conclusion: 4-bit OliVe achieves nearly original accuracy for commonly used language models and integrates efficiently into tensor-core and systolic-array accelerators.
  • Conclusion: 4.5× speedup and 4.0× energy reduction over GOBO are reported for OliVe-based accelerators.
Loading 2304.07493v1…