PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantized Convolutional Neural Networks for Mobile Devices

Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, Jian Cheng

arXiv:1512.06473v3cs.CV

TL;DR

CNNs offer strong vision performance but require substantial computation and memory, limiting deployment on resource-constrained devices. Q-CNN quantizes convolutional kernels and fully-connected weights while minimizing layer-response estimation error, achieving major efficiency and compression gains with small accuracy loss. The resulting models can run image classification on mobile devices within one second.

  • Problem

    CNN computation and memory demands make deployment difficult on devices without high-performance hardware.

  • Method

    Q-CNN quantizes convolutional and fully-connected parameters, minimizes layer-response estimation error, and trains with error correction to reduce accumulated quantization error.

  • Results

    Q-CNN achieves 4∼6× speed-up and 15∼20× compression with classification accuracy loss within one percentage point.

  • Takeaways & Limitations

    Quantized CNN models can classify images on mobile devices within one second while substantially reducing test-phase, storage, and memory costs.

  • Takeaways & Limitations

    Realistic speed-up depends on implementation conditions, with BLAS-enabled mobile measurements yielding slightly lower gains than theoretical estimates.

Abstract

from arXiv · show

Recently, convolutional neural networks (CNN) have demonstrated impressive performance in various computer vision tasks. However, high performance hardware is typically indispensable for the application of CNN models due to the high computation complexity, which prohibits their further extensions. In this paper, we propose an efficient framework, namely Quantized CNN, to simultaneously speed-up the computation and reduce the storage and memory overhead of CNN models. Both filter kernels in convolutional layers and weighting matrices in fully-connected layers are quantized, aiming at minimizing the estimation error of each layer's response. Extensive experiments on the ILSVRC-12 benchmark demonstrate 4~6x speed-up and 15~20x compression with merely one percentage loss of classification accuracy. With our quantized CNN model, even mobile devices can accurately classify images within one second.

1. Introduction

CNNs achieve strong vision performance but impose substantial computation and memory demands, motivating Q-CNN, a unified framework that quantizes network parameters to accelerate inference and compress models.

  • Motivation: 60M parameters and over 729M FLOPs are required by 8-layer AlexNet to classify one image.FLOPs count floating-point operations required for one-image classification.
  • Motivation: Convolutional layers dominate computation, whereas fully-connected layers contain most network parameters, creating two complementary efficiency targets.Prior methods commonly addressed either convolutional acceleration or fully-connected parameter compression.
  • Approach: Q-CNN simultaneously accelerates and compresses convolutional networks by quantizing parameters and estimating layer responses with approximate inner products.The framework targets both convolutional and fully-connected layers.
  • Approach: Q-CNN minimizes each layer’s response estimation error and uses previous estimation errors during training to limit accumulated error across quantized layers.This training strategy is intended to preserve model performance while quantizing multiple layers.
  • Results: 4∼6× speed-up and 15∼20× compression are achieved with classification accuracy loss within one percentage point.On ILSVRC-12, the framework generally achieves 4× acceleration and 15× compression or higher with less than 1% top-5 accuracy loss.
  • Results: Quantized CNNs can be implemented on mobile devices and classify an image within one second.The paper reports substantially improved test-phase efficiency on mobile platforms.

2. Preliminary

Q-CNN targets the distinct computational and parameter bottlenecks of convolutional and fully-connected layers by using product quantization for efficient inner-product computation.

  • Motivation: Convolutional computation dominates test-phase overhead, while fully-connected layers store most network parameters.Efficient deployment therefore requires both convolution acceleration and fully-connected parameter compression.
  • Layer computation: Both convolutional and fully-connected forward passes are formulated around inner products between layer inputs and parameters.The convolutional response sums inner products over positions within a receptive field.
  • Product quantization: Product quantization splits vectors into M sub-vectors and replaces each with a sub-codeword from a learned sub-codebook.The vector is represented by concatenated sub-codewords for efficient computation and storage.
  • Product quantization: Precomputed sub-vector–codeword inner products transform O(D) inner-product computation into M additions, where M ≤ D.This reduction applies when the required lookup values are computed in advance.
  • Related work: Existing quantization methods mainly compress fully-connected parameters and generally do not accelerate test-phase computation or reduce runtime memory.Some methods reconstruct parameters during testing, limiting their compression benefit to disk storage.

3. Quantized CNN

Q-CNN quantizes convolutional and fully-connected parameters to accelerate test-time computation while reducing storage. It minimizes layer-response estimation error and uses error correction to limit accumulated errors across layers.

  • Q-CNN introduces quantized test-phase computation for both convolutional and fully-connected layers.The framework uses product-quantized parameters and approximate inner products.
  • 3.1. Quantizing the Fully-connected Layer: Fully-connected responses use M subspaces, sub-codebooks, and look-up tables, reducing each response to M addition operations.Only sub-codebooks and quantization indices need storage.
  • 3.2. Quantizing the Convolutional Layer: Convolutional kernels are split along feature-map channels so precomputed inner products can be reused across overlapping sliding windows.Quantized kernels and look-up tables reduce response-map computation and storage complexity.
  • 3.3. Quantization with Error Correction: Layer-response quantization minimizes the discrepancy between desired and approximated responses rather than only minimizing parameter error.The objective uses the desired response and the response computed from quantized parameters.
  • 3.3.1 Error Correction for the Fully-connected Layer: Block coordinate descent alternates updates to sub-codebooks and sub-codeword assignments for the response-estimation objective.Sub-codebooks are updated with fixed assignments, and assignments are updated with fixed sub-codebooks.
  • 3.3.3 Error Correction for Multiple Layers: Layerwise training uses quantized-network inputs and original-network responses to suppress error accumulation across multiple quantized layers.The approach sequentially quantizes layers while preserving the original network’s desired responses.
  • 3.3.3 Error Correction for Multiple Layers: Back-propagation is not adopted because discrete sub-codeword assignments make gradient-based joint optimization difficult.The paper identifies joint back-propagation as a possible future extension.

4. Related Work

Prior work separately accelerates convolutional layers and compresses fully-connected layers. These approaches include low-rank or tensor decomposition, fixed-point representations, pruning, matrix factorization, and distillation.

  • Low-rank and tensor decomposition methods accelerate convolutional layers by transforming kernels or layers into lower-complexity representations.
  • Fixed-point representations of connection weights or activations can benefit computation from hardware acceleration.
  • Fully-connected parameters have been compressed through connection removal, matrix factorization, and smaller networks trained with dark knowledge.

5. Experiments

Experiments on MNIST and ILSVRC-12 evaluate Q-CNN across convolutional and fully-connected layers, comparing efficiency, compression, and classification accuracy with established baselines. Q-CNN generally achieves substantial acceleration or compression with small accuracy losses, and error correction improves preservation of classification performance.

  • Results on MNIST: Q-CNN (EC) provides higher compression with less performance degradation than all MNIST baselines on both 3-layer and 5-layer networks.Error correction is especially effective for the deeper network; performance is stable across five runs, with a standard deviation of 0.05%.
  • Quantizing the Convolutional Layer: 4.27× acceleration of all AlexNet convolutional layers initially increases top-1 and top-5 error rates by no more than 2.5%, falling below 1% after fine-tuning.The reported setting is C′_s = 8 and K = 128.
  • Quantizing the Fully-connected Layer: Q-CNN compresses CaffeNet’s first fully-connected layer with negligible performance loss within 0.4%, outperforming DPP and SVD.Error correction is particularly helpful at higher compression rates.
  • Results on ILSVRC-12: 4×-6× speed-up and 15×-20× compression are achieved across ILSVRC-12 networks with classification-accuracy losses generally within about 1%.For CNN-S, the reported result is 5.78× speed-up, 20.16× compression, and a 0.85% top-5 accuracy drop; VGG-16 reaches 4.06× speed-up, 20.34× compression, and a 0.58% top-5 error increase.
  • Results on Mobile Devices: On mobile-device experiments, Q-CNN delivers 3× speed-up for AlexNet, 4× for CNN-S, 20× storage compression, and one-quarter of the original runtime memory with at most a 1% top-5 accuracy loss.These results compare original and quantized models on time, storage, memory consumption, and top-5 classification error.

6. Conclusion

The paper presents a unified framework that accelerates and compresses CNNs through parameter quantization, with negligible classification-accuracy loss.

  • The framework simultaneously accelerates and compresses convolutional neural networks.
  • Quantized parameters enable efficient test-phase computation while maintaining classification accuracy.

Appendix A: Additional Results

Additional experiments examine settings beyond the main reported configurations, including quantizing convolutional layers in AlexNet and fully-connected layers in CaffeNet.

  • The additional experiments evaluate alternative layer-quantization settings in AlexNet and CaffeNet.

Quantizing Convolutional Layers in CaffeNet

The paper evaluates quantizing all convolutional layers in CaffeNet and reports speed-up and accuracy changes for CaffeNet and CNN-S.

  • All convolutional layers in CaffeNet are quantized and evaluated with error-correction fine-tuning.
  • 1.15% and 0.75% increases in CaffeNet top-1 and top-5 error rates are reported after fine-tuning.
  • 3.32× speed-up is reported for accelerating all CaffeNet convolutional layers without fine-tuning.
  • 3.69× speed-up is reported for accelerating all CNN-S convolutional layers without fine-tuning.

Quantizing Fully-connected Layers in AlexNet

The appendix reports experiments quantizing all fully-connected layers in AlexNet.

  • All fully-connected layers in AlexNet are quantized in the reported additional experiment.

Quantizing Fully-connected Layers in CNN-S

The CNN-S experiments quantize all fully-connected layers and compare compression rates with increases in top-1 and top-5 error rates without fine-tuning.

  • All fully-connected layers in CNN-S are quantized for the reported experiment.
  • 14.37× compression is reported for compressing all fully-connected layers in CNN-S without fine-tuning.
  • The CNN-S comparison evaluates both compression rates and increases in top-1/5 error rates.

Appendix B: Optimization in Section 3.3.2

The appendix formulates error-corrected convolutional-layer quantization as an optimization over sub-codebooks and assignment indicators. It solves the problem by alternating residual-based updates of assignments and sub-codewords, including exhaustive selection and least-squares updates.

  • Problem formulation: The convolutional-layer objective models input feature maps, response feature maps, spatial locations, and channel dimensions for N training images.Inputs are S_n ∈ R^(d_s×d_s×C_s), responses are T_n ∈ R^(d_t×d_t×C_t), and p_s, p_t, and p_k denote spatial locations.
  • Problem formulation: The error-correction formulation optimizes sub-codebooks and assignment indicators for quantizing convolutional kernels.D_m denotes the m-th sub-codebook, while B^(m)_{p_k} indicates the assigned sub-codeword at kernel location p_k.
  • Optimization procedure: Block coordinate descent updates each subspace using a residual feature map before optimizing its sub-codebook.For the m-th subspace, the method first defines a residual feature map and then rewrites the corresponding optimization problem.
  • Optimization procedure: With assignment indicators fixed, the method sequentially updates sub-codewords by computing residual feature maps for each sub-codeword.The procedure applies this update within the m-th sub-codebook, including the k-th sub-codeword.
  • Sub-codeword update: After fixing assignments, the sub-codeword update is transformed into a least-squares problem.Solving the least-squares problem updates the k-th sub-codeword.
  • Assignment update: At each kernel location, the method exhaustively tests all sub-codewords and selects the one minimizing the objective.The assignment indicator is one-hot, so only one of K entries is nonzero at each location.
Loading 1512.06473v3…