PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko

arXiv:1712.05877v1cs.LGstat.ML

TL;DR

Mobile deployment requires CNNs that balance accuracy with model size and latency, but existing quantization methods have limited evidence on efficient architectures and real hardware. This paper co-designs integer-only quantization and training for MobileNets, improving the latency–accuracy tradeoff on ARM CPUs while retaining near-original accuracy.

  • Problem

    Mobile CNN deployment requires low latency and small models, while prior quantization evidence was limited on already efficient architectures and real hardware.

  • Method

    The paper co-designs affine integer-only quantization for weights and activations with simulated-quantization training to preserve model accuracy.

  • Results

    The approach improves the latency–accuracy tradeoff for MobileNet-based ImageNet classification and COCO detection on ARM CPUs while restoring accuracy to near-identical levels.

  • Takeaways & Limitations

    Integer-arithmetic-only inference, combined with efficient architecture design, supports more accurate and efficient deployment of vision models on common ARM CPUs.

  • Takeaways & Limitations

    The evaluation focuses on already efficient MobileNet architectures and common ARM CPUs, limiting the demonstrated scope of the conclusions.

Abstract

from arXiv · show

The rising popularity of intelligent mobile devices and the daunting computational cost of deep learning-based models call for efficient and accurate on-device inference schemes. We propose a quantization scheme that allows inference to be carried out using integer-only arithmetic, which can be implemented more efficiently than floating point inference on commonly available integer-only hardware. We also co-design a training procedure to preserve end-to-end model accuracy post quantization. As a result, the proposed quantization scheme improves the tradeoff between accuracy and on-device latency. The improvements are significant even on MobileNets, a model family known for run-time efficiency, and are demonstrated in ImageNet classification and COCO detection on popular CPUs.

1. Introduction

The introduction identifies a gap in mobile CNN inference: existing quantization methods often lack meaningful baselines and verifiable real-hardware efficiency. The paper addresses this with 8-bit integer quantization, integer-only inference, co-designed training, and MobileNet benchmarks targeting improved latency–accuracy tradeoffs.

  • Motivation: Current state-of-the-art CNNs are poorly suited to mobile devices because architecture development has emphasized accuracy rather than model complexity and computational efficiency.Mobile deployment requires small model sizes and efficient computation on platforms including smartphones, AR/VR devices, and drones.
  • Limitations of prior work: Existing quantization methods convert CNN weights and/or activations from 32-bit floating point to lower bit-depth representations, but remain limited in latency–accuracy tradeoffs.The introduction focuses on quantization approaches including TWN, BNN, XNOR-net, and related methods.
  • Limitations of prior work: Prior quantization work often uses over-parameterized architectures such as AlexNet, VGG, and GoogleNet, making compression gains less meaningful than evaluation on efficient models.The paper identifies quantizing efficient architectures as a more meaningful challenge.
  • Limitations of prior work: Many methods do not demonstrate efficiency gains on real hardware, while 1-bit quantization can cause substantial performance degradation and overly restrict model representation.Weight-only quantization primarily targets storage, whereas binary, ternary, and bit-shift methods target computational efficiency through bit operations.
  • Contributions: The paper co-designs integer-only inference and quantized training, then evaluates MobileNet-based classification and detection systems on ARM CPUs to improve latency–accuracy tradeoffs.The experiments cover ImageNet classification and COCO object detection, while the inference framework targets integer-only hardware such as Qualcomm Hexagon and ARM NEON.
  • Contributions: The paper quantizes weights and activations as 8-bit integers, with only bias vectors represented as 32-bit integers.This quantization scheme is designed for integer-arithmetic-only inference.

2. Quantized Inference

The quantization scheme maps integer values to real numbers through affine parameters, enabling efficient integer-only inference while preserving exact zero representation and correspondence with floating-point training. Offline fixed-point multipliers and factored zero-point handling reduce inference overhead to core integer matrix multiplication.

  • Affine quantization: The scheme uses an affine mapping from quantized integers q to real values r, with scale S and zero-point Z as quantization parameters.Each activations array and weights array uses its own parameter set.
  • Affine quantization: The zero-point Z is the quantized value corresponding to real zero, ensuring r = 0 is exactly representable for efficient zero-padding.
  • Integer-only computation: The non-integer multiplier M is computed offline, normalized as M = M0·2^-n, and implemented as a fixed-point multiplier with M0 in [0.5, 1).Using int32, the integer representation nearest to 2^31M0 provides at least 30 bits of relative accuracy.
  • Fused operators: Bias addition uses an int32 bias with zero-point Zbias = 0 and scale Sbias equal to the product of weight and input-activation scales.

3. Training with simulated quantization

The paper addresses accuracy losses in small quantized models by simulating quantization during training while retaining floating-point parameters for backpropagation. The method places quantization at inference-equivalent points and uses layer-specific ranges and levels to prepare an optimized low-bit inference graph.

  • Motivation: Post-training quantization can cause significant accuracy drops in small models, including from weight ranges differing by more than 100×.Training in floating point followed by weight quantization works sufficiently well for large models with substantial representational capacity, but fails more often for small models.
  • Simulated quantization: The proposed method simulates quantization in the forward pass while backpropagation proceeds normally and weights and biases remain floating point.This allows parameters to be nudged by small amounts while training reflects the rounding behavior used during quantized inference.
  • Quantization placement: Weights are quantized before convolution, while activations are quantized at inference-equivalent points such as after nonlinearities and bypass merges.Batch-normalization parameters are folded into weights before weight quantization when batch normalization is used.
  • Quantization parameters: Each layer’s quantization uses a configured number of levels and clamping range, with n = 256 for 8 bit quantization.The quantization function rounds values within the range [a; b] to the specified number of levels.
  • Training workflow: Training inserts fake-quantization operations into a floating-point graph, trains until convergence, then creates and runs an optimized low-bit inference graph.The workflow distinguishes the training graph from the final inference graph used by the low-bit engine.
  • Batch normalization: For batch-normalized models, quantization must simulate folding and quantize weights after scaling by batch-normalization parameters.This reproduces the inference graph, where batch normalization is folded into convolutional or fully connected weights and biases.

4. Experiments

Experiments show that quantized training preserves accuracy across ImageNet classification and detection while improving the latency–accuracy tradeoff of integer-only models on mobile hardware. These benefits extend across ResNets, InceptionV3, MobileNets, and face detection.

  • 4.1 Quantized training: Integer-only quantized ResNets achieve accuracies within 2% of floating-point counterparts across network depths.ResNet50 comparisons also show similar accuracy to INQ while providing additional run-time improvements.
  • 4.1 Quantized training: 7-bit quantized InceptionV3 models achieve accuracies close to 8-bit models, while ReLU6 reduces accuracy degradation versus ReLU.ReLU6 provides a fixed [0, 6] activation range that is easier to quantize precisely.
  • 4.2 Latency–accuracy tradeoff: Integer-only MobileNets achieve higher accuracy than floating-point MobileNets under the same latency budget.On Snapdragon 835 LITTLE cores, the accuracy gap is approximately 10% at the 33ms latency required for 30 fps.
  • 4.2 Latency–accuracy tradeoff: Quantization reduces face-detector latency by close to 2× with an approximately 2% average-precision drop, enabling the 25% detector to run at approximately 36 fps.The floating-point model remains below real-time at approximately 23 fps on one Snapdragon 835 big core.
  • 4.2 Latency–accuracy tradeoff: Using 4 cores produces a 1.5 to 2.2× speedup for quantized face-detection models, with larger models benefiting more from multithreading.Quantized models also improve the latency–accuracy tradeoff on Snapdragon 821 despite that processor’s floating-point optimization.
  • Ablation study: Ablations indicate that weights are more sensitive than activations to reduced bit depth, while 8- and 7-bit models perform similarly to floating-point models.When total bit depths are equal, retaining more bits for both weights and activations is preferred.

5. Discussion · A. Appendix: Layer-specific details · A.1. Mathematical functions

The paper presents integer-only neural-network quantization, with quantization-aware training restoring accuracy near the original model while reducing model size and improving ARM CPU inference efficiency. Its mathematical functions use pure fixed-point arithmetic without lookup tables.

  • 5. Discussion: The proposed quantization scheme approximates floating-point neural-network computations using only integer arithmetic.
  • 5. Discussion: Quantization-simulating training restores model accuracy to near-identical levels as the original model.
  • 5. Discussion: The scheme reduces model size by 4×.
  • 5. Discussion: ARM NEON-based implementations improve inference efficiency.
  • 5. Discussion: The approach advances the state-of-the-art tradeoff between latency on common ARM CPUs and the accuracy of popular models.
  • A.1. Mathematical functions: Hyperbolic tangent, logistic, and softmax functions are implemented in pure fixed-point arithmetic without lookup tables.

A.2. Addition · A.3. Concatenation

Addition layers require integer rescaling before accumulation, making them more expensive than floating-point addition. Concatenation avoids lossy rescaling by requiring identical quantization parameters for all inputs and outputs.

  • A.2. Addition: Addition layers rescale one input onto the other’s scale before integer addition.The rescaling uses fixed-point multiplication by M = S1/S2.
  • A.2. Addition: The required rescaling makes quantized Addition layers more expensive than floating-point Addition layers.After rescaling, the actual addition is performed as a simple integer addition.
  • A.2. Addition: Addition therefore requires scale alignment before the two activation arrays can be added.The scale-alignment step precedes the integer addition operation.
  • A.3. Concatenation: Concatenation layers face the same general rescaling problem as Addition layers.Rescaling uint8 values would be lossy, conflicting with the intended lossless nature of concatenation.
  • A.3. Concatenation: Concatenation avoids lossy rescaling by requiring identical quantization parameters for all input activations.This requirement replaces implementation of lossy uint8 rescaling.
  • A.3. Concatenation: The same quantization parameters must apply to both the input and output activations of a Concatenation layer.Using shared parameters removes the need for rescaling.

B. Appendix: ARM NEON details · C. Appendix: Graph diagrams

The appendix specifies ARM NEON implementation details for fixed-point inference, including correct multiplication and shifting behavior, and describes SIMD-friendly accumulation techniques. It also notes an accuracy hazard in NEON rounding and provides implementation context for supported ARM instruction sets.

  • B. Appendix: ARM NEON details: The appendix targets both 64-bit and 32-bit ARM NEON instruction sets and assumes familiarity with assembly programming.Its instruction mnemonics refer to 64-bit ARM, while the discussion applies equally to 32-bit ARM instructions.
  • B. Appendix: ARM NEON details: SQRDMULH exactly implements the article’s fixed-point multiplications, whereas SQDMULH10 must not be used because it rounds incorrectly.The appendix emphasizes selecting the correctly-rounding instruction.
  • B. Appendix: ARM NEON details: The rounding-to-nearest right-shifts used in section 2.2 do not map exactly to any ARM NEON instruction.This instruction-level mismatch motivates care when implementing the intended rounding behavior.
  • B. Appendix: ARM NEON details: Pure-arithmetic, SIMD-ready, branch-free fixed-point implementations of tanh and logistic functions are available in gemmlowp, with NEON and SSE specializations.TensorFlow Lite shows how these implementations are called.
  • B. Appendix: ARM NEON details: gemmlowp’s fixed-point math uses fixed-point multiplications and ordinary non-saturating integer additions rather than general saturated arithmetic.The appendix also points to TensorFlow Lite implementations and its Converter for related functionality.
  • B. Appendix: ARM NEON details: NEON’s RSHL with a variable negative offset rounds ties upward rather than away from zero, creating an overall upward bias.For −12/23, RSHL produces −1 instead of the correct round-to-nearest result −2; this bias can significantly reduce end-to-end inference accuracy.
  • B. Appendix: ARM NEON details: For matrix-multiplication accumulation, quantized uint8 operands and zero-points are shifted to int8 by subtracting 128 before multiply-add operations.This transformation enables an efficient NEON implementation of the core accumulation.
  • B. Appendix: ARM NEON details: Avoiding quantized weight value −128 ensures products are below 2^14 in absolute value, allowing two products to accumulate in a local int16 accumulator.The resulting design can use an 8-way SIMD multiplication with SMULL on int8 operands before transferring results to int32 accumulation.

D. Experimental protocols · D.1. ResNet protocol

The ResNet protocol standardizes ImageNet preprocessing and asynchronous momentum-based training, including delayed activation quantization. It also references gemmlowp implementations and presents original and quantized computation graphs.

  • D.1. ResNet protocol: ImageNet images are resized with preserved aspect ratio until the smallest side is 256, then center-cropped to 224 × 224 and mean-subtracted per RGB channel.
  • D.1. ResNet protocol: Training uses TensorFlow’s momentum optimizer with momentum 0.9 and batch size 32.
  • D.1. ResNet protocol: The learning rate starts at 10^-5 and decays by 0.1 every 30 epochs in a staircase schedule.
  • D.1. ResNet protocol: Activation quantization is delayed for 500,000 steps, and training uses 50 asynchronous workers.Training stops after validation accuracy plateaus, normally after 100 epochs.
  • D.1. ResNet protocol: The integer-arithmetic implementation is provided in gemmlowp, including an optimized NEON kernel used by TensorFlow Lite.
  • D.1. ResNet protocol: The protocol section includes a simple graph depicting the original computation.
  • D.1. ResNet protocol: It also includes a simple graph depicting the quantized computation.

D.2. Inception protocol · D.3. COCO detection protocol

The Inception protocol used large-scale asynchronous ImageNet training with RMSProp, augmentation, and EMA averaging. The COCO detection protocol specified fixed-size preprocessing, RMSProp optimization, delayed activation quantization, and validation-based stopping.

  • D.2. Inception protocol: Inception training ran for approximately 10 million steps with batches of 32 samples and 50 asynchronous distributed workers.Training used ImageNet 2012 299 × 299 labeled images.
  • D.2. Inception protocol: Inception inputs used random crops, random horizontal flips, and random color distortion during training.The training data consisted of ImageNet 2012 299 × 299 images with labels.
  • D.2. Inception protocol: Inception optimization used RMSProp with an initial learning rate of 0.045, exponential and stepwise decay by factor 0.94 every 2 epochs, and specified momentum, decay, and epsilon settings.The optimizer used 0.9 momentum, 0.9 decay, and a 1.0 epsilon term.
  • D.2. Inception protocol: Inception trained parameters were exponential-moving-average averaged with decay 0.9999.The EMA decay was reported alongside the RMSProp parameters.
  • D.3. COCO detection protocol: COCO training randomly cropped and resized images to 320 × 320, while evaluation directly resized images to 320 × 320 and normalized inputs to [−1, 1].These preprocessing procedures differed between training and evaluation.
  • D.3. COCO detection protocol: COCO optimization used TensorFlow RMSprop with batch size 32, learning rate 4 × 10−3, staircase decay by factor 0.1 every 100 epochs, and 20 asynchronous workers.Training stopped after validation accuracy plateaued, normally after approximately 6 million steps.
  • D.3. COCO detection protocol: COCO activation quantization was delayed for 500, 000 steps, and training normally stopped after approximately 6 million steps when validation accuracy plateaued.The delay was attributed to reasons discussed in section 3.

D.4. Face detection and face attribute classification protocol

The protocol defines preprocessing and separate optimization procedures for face detection and attribute classification, then measures model latency on Pixel phones across LITTLE and big CPU cores. Inputs are resized to task-specific dimensions before evaluation.

  • Preprocessing: Random 1:1 Flickr-dataset crops are resized to 320×320 for face detection and 128×128 for attribute classification, horizontally flipped with 50% probability, and normalized to [−1, 1].
  • Face Detection Optimization: Face detection uses RMSprop with batch size 32, an initial learning rate of 4 × 10−3, staircase 0.1 decay every 100 epochs, and delayed activation quantization for 500,000 steps.Training uses 20 asynchronous workers and usually stops after approximately 3 million steps when validation accuracy plateaus.
  • Face Attribute Classification Optimization: Face attribute classification follows [10] using Adagrad, batch size 32, constant learning rate 0.1, and 12 asynchronous workers for 20 million steps.
  • Latency Measurements: Latency is measured by repeatedly running both models on random inputs for 100 seconds on Pixel and Pixel 2 phones using 1, 2, or 4 LITTLE and big cores.Reported runtimes cover the face detector at 320 × 320 inputs and the face attributes classifier at 128 × 128 inputs.
Loading 1712.05877v1…