PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Methods and Theories of Quantized Neural Networks

Yunhui Guo

arXiv:1808.04752v2cs.LGcs.NEstat.ML

TL;DR

Deep neural networks deliver state-of-the-art results but impose memory and energy demands that hinder deployment on mobile and embedded devices. This survey reviews quantized neural-network methods, theories, applications, challenges, and trends; reported results include accuracy comparable to full precision and substantial compression. Quantization also supports lower-energy bitwise computation, while training, structural-preservation, and theoretical challenges remain.

  • Problem

    Deep neural networks require substantial memory and energy, limiting deployment on mobile and embedded devices.

  • Method

    The paper surveys quantized neural-network methods, theories, applications, challenges, and future research directions.

  • Results

    Quantized networks can achieve accuracy similar to full-precision counterparts, including 98.8% accuracy for a binary network on MNIST and 16 ∼24 times compression with 1% classification-accuracy loss on ImageNet.

  • Takeaways & Limitations

    Quantization can reduce model size, accelerate computation, lower energy consumption, and support deep-learning deployment on mobile and embedded systems.

  • Takeaways & Limitations

    Rounding can sharply reduce network performance, while retaining real-valued references during training increases memory overhead.

Abstract

from arXiv · show

Deep neural networks are the state-of-the-art methods for many real-world tasks, such as computer vision, natural language processing and speech recognition. For all its popularity, deep neural networks are also criticized for consuming a lot of memory and draining battery life of devices during training and inference. This makes it hard to deploy these models on mobile or embedded devices which have tight resource constraints. Quantization is recognized as one of the most effective approaches to satisfy the extreme memory requirements that deep neural network models demand. Instead of adopting 32-bit floating point format to represent weights, quantized representations store weights using more compact formats such as integers or even binary numbers. Despite a possible degradation in predictive performance, quantization provides a potential solution to greatly reduce the model size and the energy consumption. In this survey, we give a thorough review of different aspects of quantized neural networks. Current challenges and trends of quantized neural networks are also discussed.

1 Introduction

Deep neural networks achieve state-of-the-art results across major real-world tasks but are difficult to deploy on resource-constrained devices because of their large memory requirements. This survey reviews quantization methods, approaches, challenges, and future trends.

  • Deep neural networks achieve state-of-the-art results in computer vision, natural language processing, and speech recognition.
  • AlexNet, VGG-Net, and ResNet-101 require 200MB, 500MB, and 200MB of memory, respectively.
  • Quantized neural networks represent weights, activations, or gradients with few bits, including 8-bit or 1-bit formats.
  • Quantization can shrink model size and accelerate training and inference.
  • The survey reviews quantization methods and approaches, then discusses challenges and future trends.

2 Background

The background introduces feed-forward, convolutional, and recurrent networks, then explains quantization as a way to address model size, computational cost, communication cost, and energy consumption. It distinguishes quantization strategies and components while noting accuracy comparable to full precision alongside unresolved training and theoretical challenges.

  • Feed-forward Neural Networks: Feed-forward neural networks contain input, hidden, and output layers, with hidden layers transforming inputs into representations for subsequent computation.
  • Feed-forward Neural Networks: A fully connected layer can require 128M memory for a 256 × 256 grayscale image connected to 512 hidden neurons.
  • Convolutional Neural Networks: CNNs use convolutional, pooling, and fully-connected layers; convolutional filters extract image features into activation maps.
  • Convolutional Neural Networks: Low-bitwidth filters and inputs enable bitwise dot products that can greatly accelerate convolutional computation.
  • Recurrent Neural Networks (RNNs) and LSTM: RNNs and LSTMs model sequence dynamics with loops that incorporate previous computations, but their millions of parameters hinder embedded and mobile deployment.
  • Quantized Neural Networks: Quantization methods are classified as deterministic or stochastic, depending on whether values map one-to-one or are sampled from discrete distributions.
  • Quantized Neural Networks: Quantized weights and activations reduce model size, while quantized gradients reduce communication cost during distributed training.
  • Quantized Neural Networks: Fixed codebooks use predefined values, whereas adaptive codebooks learn discrete values from data; examples include binary, ternary, and power-of-two networks.

3.1 Deterministic quantization

Deterministic quantization maps real-valued parameters to fixed discrete values through rounding, clustering, optimization, or related schemes. These methods offer compression and accuracy benefits but face convergence, computational, and control limitations.

  • Rounding: Binary rounding uses Sign(x) during forward propagation, while straight-through estimation supplies gradients during back-propagation.The same framework can binarize weights, activations, or gradients.
  • Rounding: Channel-wise binary scaling uses EF(|x|), the mean absolute weight value of each output channel; a constant scalar can replace it for all filters.This scaling is used to approximate real-valued filters with binary values.
  • Rounding: Rounding clips real values to a bounded interval and replaces them with the nearest uniformly spaced quantization point.The spacing is σ(k) = 2^(1−k).
  • Vector Quantization: Vector quantization clusters weights and replaces each weight with its group centroid, while extensions partition matrices, quantize residuals, or add error correction.The approach was systematically applied to neural-network compression by Gong et al. (2014).
  • Vector Quantization: 16 ∼24 times compression with only 1% loss of classification accuracy was reported on ImageNet using k-means quantization with state-of-the-art CNNs.Han et al. retrained the network afterward to fine-tune the quantized centroids.
  • Optimization-Based Quantization: Hessian-weighted k-means measures quantization-induced performance degradation, addressing uncontrolled accuracy loss and unconstrained compression ratios.It limits deviations especially for weights with large effects on network performance.
  • Optimization-Based Quantization: Optimization-based methods include binary, ternary, semi-discrete, and loss-aware formulations, with trade-offs involving assumptions, second-order computation, and implementation complexity.Semi-discrete decomposition varies compression ratio and performance loss through k.

3.2 Stochastic Quantization

Stochastic and probabilistic quantization represent discrete values through sampling or learned distributions rather than fixed one-to-one mappings. These approaches can provide flexibility, regularization, sparsity, or Bayesian interpretations, but introduce gradient, tractability, and model-scope challenges.

  • Random Rounding: Stochastic quantization samples discrete values from distributions parameterized by real-valued inputs, unlike deterministic one-to-one mappings.Random rounding can map values probabilistically to nearby discrete points.
  • Random Rounding: Unbiased random rounding satisfies E(Q(x)) = x, making the method equivalent to injecting noise into the training process.The injected noise can act as a regularizer and enable conditional computation.
  • Random Rounding: Gradient estimates for discrete neurons often have high variance, which may cause oscillations in the loss function during training.This is a central challenge of random rounding methods.
  • Probabilistic Quantization: Expectation Back-propagation trains binary or ternary networks from discrete priors, using mean-field approximation and the Central Limit Theorem for an otherwise intractable update.The approximation produces a tractable solution to the Bayesian update.
  • Probabilistic Quantization: Local reparameterization and the Central Limit Theorem approximate discrete multinomial distributions with smooth Gaussians so gradients can pass through discrete nodes.The underlying loss is nondifferentiable because of discreteness.
  • Variational Inference: Variational methods place a quantizing prior on weights and approximate the intractable posterior with variational inference.The objective balances predictive reconstruction with closeness to the prior.
  • Bayesian Quantization: Multi-spike-and-slab priors replace low-variance weights with nearby quantization targets and prune weights with large variance.The reported replacements occur without significant accuracy loss.
  • Bayesian Quantization: Probabilistic quantization can produce sparse models, but depends on carefully chosen priors, is often intractable, and does not cover some models such as recurrent neural networks.These are stated scope and practicality limitations of the framework.

3.3 Discussion

Quantization methods differ in their mappings, target components, and practical trade-offs. Deterministic methods are generally preferred for hardware acceleration because quantization levels can be specified in advance.

  • Discussion: Quantization techniques are broadly classified as deterministic or stochastic, and may target weights, activations, or gradients.Weights and activations reduce model size, while quantized gradients can reduce distributed-training communication costs.

4 Quantization of Network Components

Quantization can target weights, activations, or gradients, with distinct motivations and technical challenges. Weight and activation quantization reduce memory, while gradient quantization primarily reduces communication costs in distributed training.

  • Weight Quantization: Weight quantization reduces model size and includes layer-wise, group-wise, binary-basis, power-of-two, and end-to-end strategies.Incremental network quantization partitions weights, quantizes groups, and retrains to compensate for accuracy loss.
  • Weight Quantization: Quantized weights make convergence harder and can make back-propagation infeasible because discrete neurons block ordinary gradient propagation.Smaller learning rates and approximate gradient estimates are needed for stable training.
  • Activation Quantization: Quantized activations reduce memory and can replace inner products with binary operations, but non-differentiability creates gradient mismatch and vanishing-gradient difficulties.Reported approaches include 8-bit activation quantization, continuous operator approximations, and half-wave Gaussian quantization.
  • Gradient Quantization: Gradient quantization targets communication cost in distributed SGD, where exchanging full-precision gradients creates a performance bottleneck.Methods include 1-bit, threshold, QSGD, ternary, and random-rounding approaches; gradient magnitude and sign must both be considered.
  • Gradient Quantization: Naive gradient quantization can violate convergence conditions, so sophisticated methods and low-variance, unbiased estimates remain important.Quantized gradients can also reduce backward-pass computation in single-machine training.

5 A Comparison of Two Quantization Methodologies

Quantization methods differ according to whether their codebook is fixed in advance or learned from data. Fixed codebooks simplify representation but constrain optimization, whereas adaptive codebooks offer flexibility with additional representation cost.

  • Fixed codebook quantization: Fixed codebook quantization maps weights to predefined values such as {−1, 1}, {−1, 0, 1}, or power-of-two numbers.Power-of-two codebooks can make digital implementation faster.
  • Fixed codebook quantization: Fixed codebooks can make optimization difficult because a small codebook limits the searchable parameter space.Training often requires backward-step approximations to pass gradients through discrete neurons, creating gradient mismatch.
  • Adaptive codebook quantization: Adaptive codebook quantization learns discrete values from data using vector or probabilistic quantization.Vector quantization clusters real values by a distortion measure, while probabilistic quantization uses posterior weight distributions.
  • Adaptive codebook quantization: Adaptive quantization includes hard assignments and probability-based soft assignments, with soft quantization inspired by weight sharing.Adaptive methods are more flexible and can avoid ad hoc training modifications, but their codebooks may require more bits.

6 Quantized Neural Networks: Case Studies

The survey reviews quantization during and after training through representative methods including BinaryConnect, XNOR-Net, DoReFa-Net, DeepCompression, ECSQ, and INQ. These methods show that high compression and near full-precision accuracy are possible, while accuracy depends on bit-width, methodology, dataset, and architecture.

  • 6.1 Quantization During Training: BinaryConnect uses binary weights during forward and backward passes while retaining real-valued weights for parameter updates.It achieved near-state-of-the-art results on MNIST and CIFAR-10.
  • 6.1 Quantization During Training: XNOR-Net binarizes inputs, weights, activations, and gradients to accelerate training and inference, with a slight accuracy drop from gradient binarization.DoReFa-Net extends this direction with more sophisticated rounding and arbitrary bit-widths.
  • 6.2 Quantization After Training: DeepCompression combines pruning, discrete-weight quantization, and Huffman coding to reduce network memory requirements by 35× to 49× without accuracy degradation.Retraining remaining connections and quantized centroids compensates for accuracy loss.
  • 6.2 Quantization After Training: ECSQ uses second-order loss information, approximates the Hessian as diagonal, and Hessian-weights k-means clustering of weights.The method achieves nearly the same accuracy as a full-precision network on ImageNet.
  • 6.2 Quantization After Training: INQ alternates weight partition, group-wise quantization, and retraining, progressively quantizing all weights while retaining full-precision groups for compensation.The survey reports that quantized-network performance improved rapidly and can approach state-of-the-art ImageNet results with binarized weights.
  • 6.3 Performance Comparison of Different Quantized Neural Networks: More weight bits generally yield higher accuracy, and binary networks are harder to train than ternary networks.Across some datasets and architectures, quantized networks still show a performance gap from full-precision models.
  • 6.3 Performance Comparison of Different Quantized Neural Networks: Methods that quantize after training generally obtain better results than methods that quantize during training.Training-time quantization must retain full-precision weights, which can create substantial memory overhead.

7 Why Does Quantization Work?

Quantization can work because neural networks tolerate certain perturbations, while binarization preserves important geometric and statistical properties of full-precision computation. Theoretical analyses provide limited but useful explanations for this behavior.

  • Neural networks can tolerate noise, and low-precision operations may act as regularization without substantially harming performance.The passages characterize quantization noise as potentially beneficial for generalization, while presenting this as a possible explanation rather than a guarantee.
  • Convergence analysis shows that quantized-weight SGD and BinaryConnect can approach the loss of a full-precision network under smoothness assumptions.The supplied passage states a convergence rate linear in Δ under an L-Lipschitz smooth loss, but its excerpt ends before specifying the complete bound.
  • Binarization almost preserves the direction of full-precision high-dimensional vectors, with the angle to a random normal vector converging to 37°.
  • Binarized and full-precision weights produce approximately proportional activation dot products, preserving statistical properties of forward computation.The relation is expressed as a · w_b ∼ a · w_c, where the proportionality factor is scalar.
  • A figure compares angles between random vectors and their binarized versions against angles between two random vectors.The red curves represent vector–binarized-vector angles, while blue curves represent angles between two random vectors.

8 Future of Quantized Neural Networks

Future work should narrow the remaining performance gap, broaden quantization beyond the currently dominant settings, and strengthen both training methods and theoretical guidance.

  • Future directions: More sophisticated methods are needed because a substantial performance gap remains between quantized and full-precision networks.Existing work focuses mainly on feed-forward or convolutional networks and classification, although recurrent and other tasks have begun receiving attention.
  • Future directions: Quantization research should develop task-specific methods for recurrent networks, semantic segmentation, video processing, and other applications.
  • Future directions: A proposed direction is to use weight structure information to guide more sophisticated rounding when training quantized networks from scratch.
  • Future directions: Another proposed direction is a generally applicable quantization method spanning natural language processing, speech recognition, and diverse deep-learning models.
  • Future directions: Developing theoretical guidance for quantizing neural networks is identified as an additional research direction.

9 Conclusion

The survey reviews the origins, methods, theories, applications, challenges, and future directions of quantized neural networks. It concludes that quantization can promote deep-learning deployment on mobile and embedded systems and may have significant future impact.

  • The paper provides a comprehensive survey of recent progress in quantized neural networks, covering both theories and applications.
  • The survey traces quantized neural networks to their origins and presents many newly developed methods.
  • The paper identifies potential challenges, offers general advice, and proposes several future research directions.
  • Quantized neural networks promote deep-learning applications in mobile devices and embedded systems, with expected significant future impact.
Loading 1808.04752v2…