PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Quantization for Deep Neural Network

Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, Pascal Frossard

arXiv:1712.01048v1cs.LGstat.ML

TL;DR

DNN complexity creates computational and memory demands that hinder mobile deployment, while equal quantization bit-widths may underuse differences among layers. The paper measures each layer's effect on prediction accuracy and optimizes layer-wise bit-widths, achieving higher compression rates than equal-bit-width and SQNR-based approaches. Its analysis assumes noise transfers through layers under an almost linear transformation, with strong nonlinearity becoming relevant only when accuracy is already near zero.

  • Problem

    DNN computational and memory costs hinder mobile deployment, and assigning the same quantization bit-width across structurally different layers can be sub-optimal.

  • Method

    The framework measures how parameter quantization errors in individual layers affect overall prediction accuracy, then uses combined layer effects to select each layer's bit-width without exhaustive search.

  • Results

    20-40% higher compression rate is achieved than equal bit-width quantization at the same model prediction accuracy.

  • Takeaways & Limitations

    Layer-wise bit-width optimization provides more general and accurate quantization optimization than previous approaches across the evaluated models.

  • Takeaways & Limitations

    The analysis assumes noise transfers through layers under an almost linear transformation; nonlinear behavior is treated as irrelevant once accuracy is already near zero.

Abstract

from arXiv · show

In recent years Deep Neural Networks (DNNs) have been rapidly developed in various applications, together with increasingly complex architectures. The performance gain of these DNNs generally comes with high computational costs and large memory consumption, which may not be affordable for mobile platforms. Deep model quantization can be used for reducing the computation and memory costs of DNNs, and deploying complex DNNs on mobile equipment. In this work, we propose an optimization framework for deep model quantization. First, we propose a measurement to estimate the effect of parameter quantization errors in individual layers on the overall model prediction accuracy. Then, we propose an optimization process based on this measurement for finding optimal quantization bit-width for each layer. This is the first work that theoretically analyse the relationship between parameter quantization errors of individual layers and model accuracy. Our new quantization algorithm outperforms previous quantization optimization methods, and achieves 20-40% higher compression rate compared to equal bit-width quantization at the same model prediction accuracy.

Introduction

DNNs offer strong performance but increasingly strain mobile-device resources. The paper develops layer-wise quantization optimization to reduce model size while controlling accuracy degradation.

  • Mobile deployment is constrained by DNN storage, computation, energy, and real-time processing requirements.
  • Parameter quantization reduces model size and can make computation more efficient by storing and processing parameters at lower bit-widths.
  • Equal bit-width quantization can be sub-optimal because DNN layers differ in structure and quantization properties.
  • The proposed framework measures each layer's quantization effect on overall prediction accuracy and uses combined layer effects to select bit-widths without exhaustive search.
  • 20-40% higher compression rate is achieved than equal bit-width quantization at the same model prediction accuracy.
  • Reducing quantization bit-width by 1 bit increases expected squared quantization noise by 4 times, corresponding to 6dB/bit quantization efficiency.

Measurement for quantization noise

The paper maps layer-specific quantization noise into feature-space perturbations and relates those perturbations to classifier accuracy. This produces a layer-wise measurement for optimizing quantization bit-widths.

  • Quantization noise in a layer's weights induces noise in the DNN's final feature map, allowing weight-domain effects to be analyzed in feature space.
  • The softmax classifier has a linear decision boundary in the final feature vectors, enabling robustness analysis against weight-induced perturbations.
  • The adversarial noise norm represents the minimum perturbation required to cause misclassification, while random noise is related to accuracy degradation.
  • The quantity involving the layer robustness parameter t_i(∆acc) estimates accuracy degradation consistently across layers with different quantization effects.
  • Earlier-layer noise passes through more layers, produces lower-rank final-feature noise, and therefore has a lower robustness parameter t_i(∆acc).
  • The layer-wise measurements combine through the proposed method's linearity and additivity to rewrite the model quantization objective.

Linearity of the measurements

The paper models DNN responses as locally linear to sufficiently small quantization noise, including through nonlinear layers under specific assumptions.

  • Linearity assumptions: The model is locally linear to quantization noise when ||r_w||2 ≪ ||w||2.Scaling noise by α scales the resulting last-feature-vector noise by α.
  • Nonlinear layers: For ReLU, noise remains linear except when it crosses the zero point, which is unlikely when sufficiently small.ReLU is defined by z = ReLU(a) = max(0, a).
  • Nonlinear layers: Max-pooling remains linear to noise when sufficiently small perturbations do not change the ordering of its inputs.The layer selects the maximum among its kernel inputs.
  • Nonlinear layers: For Sigmoid and PReLU, small-noise linearity assumes smooth behavior over most input ranges and a very low probability of crossing nonlinear regions.These assumptions extend the analysis beyond ReLU and max-pooling.
  • Propagation: The resulting noise on the final feature vector is modeled by propagating quantization noise under the linearity assumption.The model treats weight quantization noise as the source of the propagated feature-vector noise.

Additivity of the measurements

The analysis treats layer-wise quantization effects as additive under independent noise, allowing separate measurements to predict the combined effect across layers.

  • Layer operations: For pairwise multiplication, quantized inputs and weights produce terms involving input-weight cross-products and individual noise contributions.The full expression includes (a + r_a) · (w + r_w).
  • Layer operations: The product noise term r_ai · r_wi,d can be ignored when ||r_a||2 ≪ ||a||2 and ||r_w||2 ≪ ||w||2.The bias-noise term r_bd can also be ignored when weights and biases use the same bit-width quantization.
  • Layer-wise noise: 24: Separate, independent noise on inputs and weights is equivalent to noise on both, and operation-induced noise can be placed directly on the output feature map.These two properties support later multi-layer analysis.
  • Multiple layers: For a two-layer module, the combined output-noise magnitude is analyzed through the additive relation between the noises from layers 1 and 2.Figure 2 compares noise applied to either layer separately with noise applied to both.
  • Multiple layers: The two-layer relation assumes independent propagated noises, justified because they arise from independent quantization noises r_W1 and r_W2.This independence is important for the proposed estimation method.
  • Multiple layers: The analysis extends the two-layer relation to N layers and predicts simultaneous noise effects from separate layer measurements.Eq. (12) measures layer i, while Eq. (20) combines effects across layers.
  • Optimization objective: The optimization seeks a small combined noise effect while minimizing model size under a specified accuracy-loss constraint.This frames adaptive bit-width selection as a compression optimization problem.

Adaptive quantization on multiple layers

The adaptive quantization problem is reformulated using the proposed noise measurements to optimize layer-wise bit-widths under accuracy and model-size objectives.

  • Optimization formulation: The quantization optimization is expressed using the layer-wise noise measurement and the combined multi-layer measurement.This substitutes the proposed measurements into the original optimization problem.
  • Optimization formulation: The detailed derivation of the optimality condition is provided in supplementary material.The main text presents the condition but not its full analysis.

Optimal bit-width for each layer

The procedure computes layer sensitivity and quantization-noise factors, fixes the first layer’s bit-width, and derives the remaining layer bit-widths from the optimality condition.

  • Compute layer factors: The procedure first calculates t_i by increasing layer-i weight noise until a target accuracy degradation is reached, then recording mean final-feature noise.The target is denoted Δacc.
  • Compute layer factors: The procedure calculates p_i by fixing a layer bit-width, recording mean final-feature noise, and applying Eq. (16).An example fixes b_i = 10.
  • Derive bit-widths: The procedure fixes the first-layer bit-width b_1 and derives each remaining b_i using Eq. (22).The example uses b_1 = 10 and the resulting expression includes p_1·e^−α·b_1.
  • Compression–accuracy trade-off: 39: Selecting a lower b_1 yields a higher compression rate together with greater accuracy degradation.Different first-layer bit-width choices therefore produce different quantization results.

Comparison with SQNR-based approach

The SQNR-based approach reaches optimal bit-widths under an equal-effect assumption, whereas this work evaluates its quantization optimization through empirical experiments.

  • The SQNR-based approach defines optimal bit-widths under its stated quantization condition.
  • The SQNR-based approach omits p_i and t_i because equal bit-widths are assumed to produce equal SQNR values and accuracy effects across layers.
  • The authors empirically evaluate the proposed bit-width optimization approach and validate assumptions from earlier sections.

Empirical results about measurements

The experiments test whether the proposed measurements behave as assumed and evaluate quantized models across four common ImageNet architectures.

  • The experiments validate the estimated accuracy measurement's relationship with accuracy, linearity, and additivity.
  • AlexNet, VGG-16, GoogleNet, and ResNet are quantized layer by layer with uniform quantization and tested on ImageNet's 50,000-image validation set.

Calculate ti

The method estimates layer-specific robustness thresholds from noise–accuracy relationships, then uses those measurements to support efficient quantization analysis in the low-noise regime.

  • Calculate t_i: The AlexNet example sets accuracy degradation to roughly half of the original 57% accuracy, or 28%, to calculate t_i values.
  • Calculate t_i: The t_i calculation can require O(τN|D|) forward passes, reducible to O(τN′|D|) when only the last N′ layers are evaluated.
  • Calculate t_i: Computing t_i takes around 15 minutes for AlexNet and around 6 hours for ResNet-50, but focusing on the last few layers can reduce this time.
  • Linearity of measurements: Weight-noise and activation-noise norms are approximately linear when quantization noise is small, while earlier layers become less linear under larger noise.
  • Linearity of measurements: Earlier-layer nonlinearity is associated with ReLU and max-pooling effects, but it occurs when model accuracy is already near zero.
  • Linearity of measurements: The additivity of activation-noise norms is accurate for small noise but fails for large noise, under the condition ||rW_i||2 ≪ ||W_i||2 for every layer.

Optimal bit-width for models

Experiments evaluate adaptive layer-wise quantization against SQNR-based and equal-bit-width methods, showing smaller models at matched accuracy degradation. Gains are largest for architectures with more diverse layer sizes and structures.

  • Experimental setup: The experiments evaluate AlexNet, VGG-16, GoogleNet, and ResNet-50 on the ImageNet validation set.Fully connected layers remain at 16 bits because the SQNR-based comparison applies only to convolutional layers.
  • Comparison: The proposed method outperforms SQNR-based quantization and produces smaller models at the same accuracy degradation across the tested networks.All three methods use uniform quantization for each layer; equal-bit-width quantization uses the same number of intervals in every layer.
  • Results: 30−40% smaller model size is achieved for AlexNet and VGG-16 at the same accuracy degradation.
  • Results: 15−20% smaller model size is achieved for GoogleNet and ResNet-50 at the same accuracy degradation.
  • Contribution: The framework theoretically relates quantization errors in individual layers to model accuracy and optimizes each layer’s bit-width without exhaustive search.

Supplementary Material: Adaptive Quantization for Deep Neural Network

The supplementary analysis models quantization as noise and connects feature-map perturbations to classifier accuracy. It then derives how layer structure and quantization noise affect the model’s output.

  • Quantization noise: Quantization is modeled as adding noise to each original weight, with uniform noise determined by the quantization interval.For a uniform quantizer, the noise has zero mean and variance B^2/12.
  • Quantization noise: Reducing bit-width by 1 bit increases the expected squared quantization noise by 4 times.The paper identifies this with a quantization efficiency of 6dB/bit.
  • Softmax classifier: The DNN is decomposed into a feature-mapping function followed by a softmax classifier whose decision selects the largest feature-vector coordinate.The softmax decision boundary is linear in the final feature vectors.
  • Robustness analysis: Lemma 1 bounds the norm of random feature-map noise relative to the classifier’s robustness, with a probability controlled by δ and the number of classes.
  • Accuracy relation: Limiting the final feature-map noise limits the probability of misclassification, which the analysis bounds below 2δ.
  • Layer analysis: For convolutional layers, output dimensions depend on input dimensions, kernel dimensions, channel counts, output depth, and stride.Fully connected layers are treated as a special case of convolutional layers when the spatial dimensions equal 1.
  • Layer analysis: The convolutional output noise contains contributions from weight noise, input-feature noise, their interaction, and bias noise.The interaction term is ignored when input and weight noise norms are small, and bias noise is ignored when weights and biases use the same bit-width.

Additivity of ∥rZ∥22

Noise from separately quantized layers is analyzed as approximately additive at the final feature map. This additivity supports an optimization that allocates layer-wise bit-widths according to their effects on accuracy and model size.

  • Noise propagation: Quantizing either layer in a two-layer module produces a distinct perturbation that propagates to the final feature map.
  • Additivity: When both layers are quantized, the final feature-map noise is approximately the sum of the perturbations caused by quantizing each layer separately.
  • Additivity: The additivity relation assumes independent perturbations from the two layers’ independent quantization noises.
  • Optimization: The optimal layer-wise bit-width is characterized by a KKT-derived condition in the quantization optimization problem.Algorithms compute the layer robustness parameters, noise terms, and bit-widths from this condition.
  • Optimization: For layer i, the noise norm is modeled as p_i · e^(-α·b_i), where p_i captures layer effects and b_i is its bit-width.The layer size s_i and robustness parameter t_i also enter the bit-width allocation condition.
  • Comparison: Unlike the SQNR-based formulation, the proposed optimization includes layer-specific parameters p_i and t_i, allowing unequal accuracy effects across layers.The SQNR-based method is described as a special case when equal bit-widths produce equal effects on model accuracy.
  • Experimental results: 40% less model size is achieved for AlexNet and VGG-16, while 15−20% less is achieved for GoogleNet and ResNet-50 at the same accuracy degradation.
Loading 1712.01048v1…