PaperlayerView exact revision on arXiv

Source-linked AI summary

ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization

Cong Guo, Chen Zhang, Jingwen Leng, Zihan Liu, Fan Yang, Yunxin Liu, Minyi Guo, Yuhao Zhu

arXiv:2208.14286v1cs.LG

TL;DR

DNN models are growing, but conventional integer and floating-point types can require more bits to preserve accuracy, and variable-length outlier quantization adds hardware overhead. ANT combines fixed-length, adaptive numeric types with per-tensor type selection and a unified processing element. Its accelerator reports 2.8× performance improvement and 2.5× energy reduction over BitFusion.

  • Problem

    Integer and floating-point types can require more bits to maintain DNN accuracy, while variable-length outlier quantization introduces unaligned memory access and complex, high-cost hardware.

    Source: §I · p. 1

  • Method

    ANT uses fixed-length flint to adapt to value importance within tensors, selects primitive types by tensor distribution, and uses TypeFusion to process differing primitive types.

    Source: §I · p. 2

  • Results

    ANT achieves 2.8× performance improvement and 2.5× energy reduction over BitFusion.

    Source: §I · p. 2

  • Takeaways & Limitations

    ANT combines aligned memory access and efficient low-bit computation with adaptation within and across tensors.

    Source: §I · p. 2

Abstract

from arXiv · show

Quantization is a technique to reduce the computation and memory cost of DNN models, which are getting increasingly large. Existing quantization solutions use fixed-point integer or floating-point types, which have limited benefits, as both require more bits to maintain the accuracy of original models. On the other hand, variable-length quantization uses low-bit quantization for normal values and high-precision for a fraction of outlier values. Even though this line of work brings algorithmic benefits, it also introduces significant hardware overheads due to variable-length encoding and decoding. In this work, we propose a fixed-length adaptive numerical data type called ANT to achieve low-bit quantization with tiny hardware overheads. Our data type ANT leverages two key innovations to exploit the intra-tensor and inter-tensor adaptive opportunities in DNN models. First, we propose a particular data type, flint, that combines the advantages of float and int for adapting to the importance of different values within a tensor. Second, we propose an adaptive framework that selects the best type for each tensor according to its distribution characteristics. We design a unified processing element architecture for ANT and show its ease of integration with existing DNN accelerators. Our design results in 2.8$\times$ speedup and 2.5$\times$ energy efficiency improvement over the state-of-the-art quantization accelerators.

I. INTRODUCTION

DNN models are growing faster than hardware, while conventional numerical types struggle to represent tensors whose values differ in distribution and importance. ANT combines fixed-length, within-tensor value adaptation with per-tensor type selection, aiming for low-bit quantization with modest hardware overhead.

I. INTRODUCTION

  • Integer and floating-point types can require more bits to maintain accuracy, while variable-length outlier quantization brings unaligned memory access and costly hardware.Outlier-aware schemes use low-precision values for frequent small values and higher-precision values for rare, large values.Source: §I · p. 1
  • ANT combines within-tensor value-importance adaptation with per-tensor distribution adaptation using fixed-length encodings.Its flint primitive combines float's large range with int's high precision for important value intervals; variable-length first-one coding encodes the exponent field.Source: §I · p. 2
  • The framework selects among int, float, flint, and PoT primitives to fit each tensor's distribution and minimize quantization error.Unlike prior approaches described here as exploiting only intra-tensor or inter-tensor adaptivity, ANT combines both.Source: §I · p. 2
  • TypeFusion handles different primitive types in input and weight tensors by decoding them to a unified format for an underlying float or int MAC unit.The decoder is the required modification to the existing MAC unit.Source: §I · p. 2
  • The 4-bit ANT PE supports 8-bit int with little modification, enabling over 90% of tensors to use 4-bit representation while maintaining original-model accuracy.The paper presents this as mixed-precision support.Source: §I · p. 2
  • 2.8× performance improvement and 2.5× energy reduction over the mixed-precision accelerator BitFusion.The ANT accelerator integrates with existing DNN accelerator architectures.Source: §I · p. 2

A. Quantization Metric

Quantization is framed as minimizing mean squared error (MSE) between original and quantized values. The section introduces fixed-length numerical types and mixed-precision quantization as ways to represent DNN tensors.

A. Quantization Metric

  • MSE measures the squared difference between original and quantized values, and quantization aims to reduce it.The error is considered with respect to the value distribution p(x).Source: A. Quantization Metric · p. 2

B. Fixed-length Quantization

  • Fixed-length quantization uses narrow integer or floating-point representations, while PoT retains a large range without mantissa bits.FP16 uses 5 exponent and 10 mantissa bits; BF16 and TF32 use 8E7M and 8E10M, respectively.Source: B. Fixed-length Quantization · pp. 2–3
  • The general quantization pipeline scales and clips values before applying Quant, then Dequant decodes them to the original type for quantization-aware training.The clipping thresholds are min and max; int quantization rounds to the nearest value.Source: B. Fixed-length Quantization · p. 3
  • Weight quantization uses per-channel scales, activations use per-tensor scales, and unsigned types efficiently represent post-ReLU activations.The framework also supports signed and unsigned numerical types.Source: B. Fixed-length Quantization · p. 3

C. Mixed-precision Quantization

  • Mixed-precision quantization assigns different bit widths to tensors or layers, matching precision to their importance and sensitivity.Tensor-wise designs may use temporal or spatial execution; an 8-bit multiplication takes four cycles on a 4-bit PE temporally, or one cycle with four 4-bit PEs spatially.Source: C. Mixed-precision Quantization · p. 3

D. Outlier-aware Quantization

DNN tensors have diverse value distributions, while quantization schemes trade quantization benefit against hardware overhead. ANT is presented as a fixed-length framework that exploits both within-tensor and across-tensor adaptivity.

D. Outlier-aware Quantization

  • Outlier-aware methods encode frequent normal values at low precision and rare outliers at high precision, but variable-length encoding causes unaligned memory access and added hardware complexity.OLAccel and GOBO use this approach.Source: D. Outlier-aware Quantization · p. 3

III. MOTIVATION: ADAPTIVE DATA TYPE

  • ANT targets low-bit quantization with hardware-friendly adaptivity, addressing the resources required by prior variable-bit-length accelerator designs.The paper presents a qualitative framework for analyzing the hardware overhead of quantization.Source: §III · p. 3

A. Opportunities for Adaptive Type

  • Tensor distributions vary: an early ResNet18 activation is closer to uniform, whereas a BERT activation has a long tail resembling a Laplace distribution.For four-bit quantization, int suits narrow-range uniform-like values, while PoT suits long-tailed Laplace-like values.Source: A. Opportunities for Adaptive Type · p. 4
  • Within a tensor, very small values may be pruned and very large values clipped, so the scheme can allocate fewer precisions to both extremes while preserving a broad range.This is the intra-tensor adaptivity opportunity, complementary to choosing types for different tensor distributions.Source: A. Opportunities for Adaptive Type · pp. 3–4

B. Quantization Architecture Analysis

  • ANT achieves the lowest average memory and computation bit widths for activation and weight tensors, with negligible area overhead.Table I compares average access and computation bits and decoder/controller area across 13 CNN and Transformer workloads.Source: Table I · p. 4, B. Quantization Architecture Analysis · pp. 4–5
  • Conventional int quantization has no decoder or controller area overhead but often needs 8-bit values; AdaFloat [78] also has aligned accesses but adds a bias decoder and retains rigid resolution.BitFusion exploits inter-tensor precision choices but remains limited by its int primitive; OLAccel and GOBO add decoder and controller complexity, while BiScaled uses an extra bit mask.Source: B. Quantization Architecture Analysis · pp. 4–5

IV. ADAPTIVE NUMERIC DATA TYPE

ANT combines intra-tensor adaptation through flint with a fixed-length representation, assigning precision according to value importance. Its encoding adapts exponent and mantissa allocation across intervals while retaining fixed bit width.

A. Intra-tensor ANT: Flint

  • Flint allocates fewer mantissa bits to extremely small and large values and more to middle-range values within a fixed-length representation.More exponent bits also expand the range available for large values.Source: A. Intra-tensor ANT: Flint · p. 5
  • First-one encoding places the exponent–mantissa boundary at the first 1 after the sign bit, allowing decoding with a leading-zero detector.The exponent and mantissa settings are constants for a given quantization bit width.Source: A. Intra-tensor ANT: Flint · pp. 5, 7
  • The 4-bit unsigned flint example represents 16 distinct values with a maximum value of 64 across eight intervals.The table gives the values for 4-bit unsigned flint with exponent bias −1.Source: A. Intra-tensor ANT: Flint · p. 5, Table II · p. 6
  • Flint combines int, float, and PoT behavior across value intervals, with the highest mantissa precision aligned to frequent Gaussian-like values.The first four intervals act like integers, middle intervals like floats, and the last two like PoT.Source: A. Intra-tensor ANT: Flint · p. 6
  • The element-wise flint encoder converts high-precision values to low-bit codes and is designed for lightweight hardware support, including dynamic activation quantization.Weight quantization can be performed offline; activation quantization requires hardware support.Source: A. Intra-tensor ANT: Flint · pp. 6–7

B. Inter-tensor ANT

ANT selects a primitive numeric type for each tensor according to its distribution. This lets tensors use different fixed-length formats while retaining aligned memory access.

B. Inter-tensor ANT

  • ANT chooses a tensor's numeric type according to its distribution, supporting int, float, flint, and PoT for inter-tensor adaptation.The framework selects the type intended to fit each tensor's distribution.Source: B. Inter-tensor ANT · p. 7
  • Int is suited to uniform-like distributions, float or PoT to Laplace-like distributions, and flint to Gaussian distributions.Flint provides its highest resolution for the values with the highest frequency.Source: B. Inter-tensor ANT · p. 7
  • Each tensor retains a fixed-length format regardless of primitive type, so ANT memory accesses are aligned.Source: B. Inter-tensor ANT · p. 7

C. ANT-based Quantization Framework

ANT quantization selects a primitive type for each tensor, then fine-tunes the model to recover accuracy loss. Its inference framework uses low-bit weights and inputs while retaining high-precision outputs for convolution and fully connected layers.

C. ANT-based Quantization Framework

  • ANT selects each tensor's primitive type by minimizing mean squared error among candidate quantization functions, with clipping ranges chosen by MSE minimization.The selection runs once per tensor before fine-tuning.Source: C. ANT-based Quantization Framework · p. 7
  • Weight types are selected from trained-model weights without training samples, while activation type selection uses about 100 training samples.Source: C. ANT-based Quantization Framework · p. 7
  • Mixed-precision ANT starts all layers at 4 bits and raises the bit width of the highest-MSE layer to 8 bits until model accuracy meets a preset threshold.Fine-tuning follows each bit-width adjustment.Source: C. ANT-based Quantization Framework · p. 7
  • Convolution and fully connected layers use low-bit weights and input activations but keep output activations high precision; activation-layer outputs can be quantized to low-bit values.The stated reasons for high-precision outputs are accumulation precision and the needs of following activation layers such as SoftMax and GeLU.Source: C. ANT-based Quantization Framework · pp. 7–8

V. TYPE-FUSION PROCESSING ELEMENT

TypeFusion supports ANT’s different primitive types and mixed-type operands by building on either float- or int-based MAC units. The int-based design decodes flint into integer components and adds the shift needed for its MAC operation.

V. TYPE-FUSION PROCESSING ELEMENT

  • TypeFusion supports MAC operations across different ANT primitive types, including cases where input and weight tensors use different types.It can be implemented on an existing float- or int-based MAC unit using a type decoder to convert operands to a unified format.Source: §V · p. 8

A. Float-based PE

  • The float-based PE treats int and PoT as special float formats and uses exponent and mantissa decoders to prepare operands for multiplication.Int has no exponent, while PoT has no mantissa; flint uses the same n-bit exponent and mantissa widths in different value intervals.Source: A. Float-based PE · p. 8
  • A float-based flint decoder uses a leading-zero detector and shifters to extract exponent and mantissa fields before conversion to real values.For example, flint 1110₂ decodes to 12₁₀.Source: A. Float-based PE · p. 8

B. Integer-based PE

  • The int-based PE represents flint using a base integer and exponent, allowing int and PoT to use the same representation.For 4-bit flint, Table III gives the value decomposition; int has zero exponent, while PoT has base integer one.Source: B. Integer-based PE · pp. 8–9, Table III · p. 8
  • Flint expands int’s value range using a simple left shifter and can be decoded into two integers, allowing it to pair with int quantization without extra overhead.The decoder is extensible to arbitrary bit-widths, and the sign bit can be combined with the decoded values for int multiplication.Source: B. Integer-based PE · p. 9
  • The int-based flint MAC multiplies base integers, adds exponents, then shifts the product to produce a 16-bit result for the accumulator.The product is compatible with the original 16-bit accumulator.Source: B. Integer-based PE · p. 9

C. Signed Number Support

The signed flint decoder reuses most of the unsigned decoder’s components. For int-based flint, the base integer is converted to two’s complement to match the signed integer MAC.

C. Signed Number Support

  • The signed flint decoder can reuse most unsigned-decoder components without affecting its critical path.The unsigned decoder’s critical path lies in the leading-zero detector.Source: C. Signed Number Support · p. 9
  • For signed int-based flint, the sign bit is separated from the remaining bits, which are decoded using the 3-bit flint equations.The 4-bit unsigned flint decoder function can be reused for this decoding.Source: C. Signed Number Support · p. 9
  • The base integer is converted to two’s complement for the signed integer MAC, while the critical path remains in the leading-zero detector.For float-based flint, the sign bit is attached to the decoded exponent and mantissa.Source: C. Signed Number Support · p. 9

D. Mixed-precision Support

ANT supports mixed precision by using 4-bit ANT processing elements to implement 8-bit integer operations, and up to 91% of tensors use 4-bit ANT. The paper also describes integrating ANT into systolic arrays and tensor cores while retaining aligned memory access.

D. Mixed-precision Support

  • Up to 91% of tensors use 4-bit ANT while a fraction use 8-bit int, enabling mixed-precision quantization to maintain original full-precision model accuracy.The 4-bit design supports 8-bit int operations using four ANT processing elements and an extra adder tree.Source: D. Mixed-precision Support · pp. 9–10, Figure 8 · p. 10

VI. ARCHITECTURE INTEGRATION

  • ANT integration targets existing DNN accelerators, including systolic arrays and tensor cores, with optimizations to limit TypeFusion PE overhead.The instruction set can be extended for the design.Source: §VI · p. 10

A. ANT and Dataflow Co-design

  • Output-stationary systolic arrays suit ANT because low-bit inputs and weights feed high-bit outputs; weight-stationary arrays also show close benefits.ANT decoders are placed between the on-chip buffer and array, so low-bit tensors remain stored in the buffers and are decoded before entering the array.Source: A. ANT and Dataflow Co-design · p. 10, Figure 9 · p. 10
  • For an n×n output-stationary array, decoders are placed only at the boundary, requiring 2n rather than n^2 decoders.The weight-stationary array needs n input decoders; the short inter-PE distances make added wiring overhead negligible.Source: A. ANT and Dataflow Co-design · p. 10
  • Mixed-precision systolic arrays reuse existing PE components; ANT adds decoders outside the array rather than new PE components.In 8-bit mode, an n×n array of 4-bit ANT PEs becomes an n/2×n/2 array of 8-bit int PEs.Source: A. ANT and Dataflow Co-design · p. 10
  • For weight-stationary dataflow, weights can be decoded before preloading, so each PE stores decoded exponent and integer values.Other output-stationary optimizations can also be applied.Source: A. ANT and Dataflow Co-design · p. 10
  • Tensor cores can adopt ANT by augmenting MAC units and adding operand decoders, without changing GPU memory hierarchy.Tensor cores already support mixed precision, and ANT retains aligned memory accesses.Source: A. ANT and Dataflow Co-design · pp. 10–11

B. Instruction Set Extension

The evaluation reports that mixed-precision ANT maintains original accuracy on CNNs and loses less than 1% on ViT and BERT, while using 4-bit ANT for most tensors. It also compares primitive types and prior quantization methods across accuracy and hardware measures.

B. Instruction Set Extension

  • ANT’s fixed-width PoT and flint types leave load/store instructions unchanged and require no memory-subsystem modification.The original programming model for convolutional and fully connected layers is also unchanged; each layer’s type is determined after quantization.Source: B. Instruction Set Extension · p. 11

A. Methodology

  • The evaluation compares ANT with BitFusion, OLAccel, BiScaled, AdaFloat, and GOBO, using accuracy, performance, area, and energy measures.The authors implement ANT in PyTorch and use RTL synthesis and a cycle-accurate simulator for accelerator evaluation.Source: A. Methodology · p. 11, §VII · p. 11
  • The evaluation uses CNNs and Transformer models, including ImageNet classification models and BERT-Base on GLUE; only three GLUE datasets are shown.The reported datasets are MNLI, CoLA, and SST-2; the others are described as having similar results.Source: A. Methodology · p. 11, Table IV · p. 11

B. Quantization Accuracy

  • Adding primitive types generally reduces quantization error; flint helps vision and NLP, while PoT matters especially for Transformer NLP tasks with large activations.Float has the least impact, with its role replaced by other primitive types.Source: B. Quantization Accuracy · p. 12, Figure 10 · p. 12
  • Fine-tuning is essential to recover accuracy toward original values, and 4-bit IP-F and FIP-F achieve the minimum accuracy loss.The authors select IP-F as final ANT because the float-based PE used by FIP-F consumes almost 3× the area of the int-based PE.Source: B. Quantization Accuracy · p. 12, Figure 11 · p. 12
  • ANT4-8 achieves original model accuracy on CNNs and less than 1% accuracy loss for ViT and BERT.The accuracy-loss comparison with fine-tuning is shown in Figure 12.Source: B. Quantization Accuracy · p. 12, Figure 12 · p. 12
  • Without fine-tuning, ANT offers better accuracy than 6-bit BiScaled, which the authors attribute to its intra- and inter-tensor adaptivity and additional exponent domains.Table V presents the comparison.Source: B. Quantization Accuracy · p. 12, Table V · p. 12
  • For BERT on MNLI, weight-only ANT achieves similar accuracy to weight-only GOBO, while ANT uses fixed-length encoding rather than GOBO’s variable-length scheme.Table VI gives the comparison.Source: B. Quantization Accuracy · p. 12, Table VI · p. 12

C. Area

The area analysis finds a small decoder overhead for ANT, and the performance evaluation reports latency and energy reductions against four quantization-accelerator baselines. ANT’s selected tensor types and comparison setup vary across models and accuracy constraints.

C. Area

  • The int-based decoder and PE are selected because float-based PEs have about 3× the area at similar accuracy; decoder overhead is about 0.2% for the systolic array.The area breakdown is presented for ANT and other baselines under a 28 nm process.Source: C. Area · p. 13, Table VII · p. 13

D. Performance and Energy

  • Mixed-precision ratios are adjusted for near-original accuracy in iso-accuracy and iso-area comparisons, except BiScaled; AdaFloat is evaluated only at 8 bits.BiScaled is compared only on VGG16 and ResNet50, which have more than 5% accuracy loss.Source: D. Performance and Energy · p. 13
  • CNNs and ViT use a significant portion of 4-bit flint tensors, while NLP Transformers use roughly equal portions of 4-bit flint and PoT tensors.ANT-OS and ANT-WS use the same quantization algorithm and therefore have the same ratio of data types.Source: D. Performance and Energy · p. 13
  • ANT averages 2.8×, 3.24×, 1.48×, and 4× speedup over BitFusion, OLAccel, BiScaled, and AdaFloat; ANT-OS achieves 2.53×, 1.93×, 1.6×, and 3.33× energy reduction against those baselines.The energy comparison includes static energy and dynamic energy from DRAM, on-chip buffer, and core.Source: D. Performance and Energy · p. 14

E. ANT Type Selection Analysis

ANT selects numerical types according to tensor distributions and uses flint to balance precision and range. Its evaluation reports lower tensor MSE from type selection, while the conclusion reports latency and energy improvements.

E. ANT Type Selection Analysis

  • Across the evaluated tensors, ANT chooses the 4-bit type with minimum MSE; signed 4-bit float and PoT overlap for several weight and activation tensors.The analysis uses ResNet-18 on ImageNet and BERT-Base on MNLI; ResNet-18 activations use unsigned types, while its weights and all BERT tensors use signed types.Source: E. ANT Type Selection Analysis · p. 14
  • CNN first-layer tensors are more uniform-like and favor int, later CNN tensors are closer to Gaussian and favor flint, while BERT tensors show mixed distributions and outlier-sensitive activations.The BERT distribution inspection uses its first two Transformer blocks as representatives; the reported trend is similar to the remaining blocks.Source: E. ANT Type Selection Analysis · p. 14

VIII. RELATED WORK

  • ANT uses fixed-length types for architectural integration, unlike outlier-aware designs whose variable-length encoding can require unaligned accesses and heavier hardware changes.Flint is distinguished from Posit by having no regime bit and an encoding/decoding process based on float or int.Source: §VIII · p. 15

IX. CONCLUSION

  • ANT reports 2.8× latency reduction and 2.5× energy improvement over state-of-the-art quantization accelerators.The conclusion attributes ANT’s approach to intra-tensor adaptation with flint and inter-tensor type selection.Source: §IX · p. 15

APPENDIX

The appendix documents ANT’s accuracy and simulator evaluations, the models and datasets, and the artifact workflow and execution environment.

A. Abstract

  • The experiments evaluate model accuracy and ANT simulator performance across image-classification and NLP models, with fine-tuning code and checkpoints provided for evaluation.The listed models include VGG16, ResNet-18, ResNet-50, Inception-V3, ViT, and BERT; checkpoint-based image-model evaluation can finish in one hour.Source: A. Abstract · p. 15

B. Artifact check-list (meta-information) •

  • The artifact checklist lists NVCC 11.3, GCC 7.5.0, CUDA 11.3, and PyTorch 1.11, and estimates about 20 GB of disk space.It estimates about 30 minutes to prepare the environment and about 50 hours to execute all experiments, while checkpoint-based fast evaluation takes about one hour.Source: B. Artifact check-list… · pp. 15–16

3) Software dependencies:

2) Hardware dependencies:

  • Fine-tuning uses either a server with one NVIDIA A100 40GB GPU or a server with four NVIDIA A10 24GB GPUs for distributed fine-tuning.Source: 2) Hardware dependencies · p. 16

4) Data sets and models:

  • ImageNet is used for five image-classification models, and BERT-Base is evaluated on GLUE, with MNLI, CoLA, and SST-2 results presented.Source: 4) Data sets and models · p. 16

D. Installation

E. Evaluation and expected results

  • The artifact separates model-accuracy evaluation in ant_quantization from simulator performance and energy evaluation in ant_simulator; expected results are released for Figures 12 and 13 and Table V.Source: E. Evaluation and expected results · p. 16

Related papers

Loading 2208.14286v1…