PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Quantized Nets: A Deeper Understanding

Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, Tom Goldstein

arXiv:1706.02379v3cs.LGcs.CVstat.ML

TL;DR

Deploying neural networks on low-power devices motivates training directly with coarse quantization, but prior work was largely empirical and low-precision training can stagnate. This paper theoretically analyzes stochastic rounding and BinaryConnect under convex and non-convex settings, showing that high-precision representations provide a greedy search phase that fully quantized methods lack.

  • Problem

    Training quantized networks is important for resource-limited devices, yet existing methods were studied largely empirically and small updates can be erased by quantization.

  • Method

    The paper analyzes stochastic rounding and BinaryConnect theoretically through convex convergence guarantees and non-convex behavioral analysis.

  • Results

    High-precision representations such as BinaryConnect support an annealing or greedy-search phase that fully quantized stochastic rounding lacks, while convex accuracy depends on quantization coarseness.

  • Takeaways & Limitations

    The analysis explains why maintaining real-valued weights can produce better minimizers and why purely low-precision training is difficult.

  • Takeaways & Limitations

    The convergence guarantees use convexity-related assumptions, and BC guarantees concern real-valued weights rather than the quantized weights themselves.

Abstract

from arXiv · show

Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized networks, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of these algorithms for non-convex problems, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the difficulty of training using low-precision arithmetic.

1 Introduction

Quantized training is motivated by the need to run neural-network learning on resource-constrained devices, but existing methods were largely empirical. The paper analyzes their convergence and explains why high-precision representations help non-convex optimization.

  • Low-power embedded systems constrain memory and power, and some lack basic floating-point arithmetic.
  • Quantized weights can improve memory and computation efficiency while potentially matching high-precision networks’ performance.
  • Small learning-rate updates can be erased by post-update binarization, causing naïve quantized training to stagnate.
  • The paper theoretically studies why quantized training methods succeed or fail rather than relying primarily on empirical evidence.
  • Under convexity, stochastic rounding and BinaryConnect achieve accuracy depending on quantization level, while BC’s floating-point representation supports an annealing property absent from fully quantized methods.

2 Background and Related Work

Prior work either quantizes pretrained models or trains quantized models from scratch. This paper focuses on the latter because it supports constrained-resource training and inference.

  • Quantization can reduce arithmetic to 8-bit fixed-point and, in the extreme, represent weights with one bit.
  • Previous approaches divide into quantizing pretrained models and training quantized models from scratch.
  • Training from scratch is relevant when both training and inference must operate under constrained resources.
  • Many methods retain full-precision weight copies for training, but floating-point arithmetic may be unavailable or undesirable on low-power devices.

3 Training Quantized Neural Nets

The paper formalizes quantized optimization by combining stochastic-gradient updates with quantizers and by distinguishing when weights are quantized. It contrasts fully quantized rounding with BinaryConnect’s full-precision update buffer.

  • The objective decomposes into component losses, as in neural-network training, and standard SGD updates weights using sampled stochastic gradients.
  • A quantization function maps real-valued weights to representable low-precision values, with optimization behavior determined by the quantizer and quantization timing.
  • Deterministic Rounding (R): Deterministic rounding snaps values to the nearest quantized level, with binary weights constrained to {−1, 1}.
  • Deterministic Rounding (R): If gradient updates are smaller than the quantization step, deterministic rounding loses gradient information and weights may remain unchanged.
  • Stochastic Rounding (SR): Stochastic rounding preserves the input in expectation by rounding up with a probability determined by its fractional position between quantization levels.
  • BinaryConnect (BC): BinaryConnect accumulates updates in a full-precision buffer and quantizes weights before gradient computations; deterministic rounding is common in practice.

4 Convergence Analysis

The convergence analysis shows that SR and BC reach quantization-dependent error floors under convex assumptions, while BC can behave better when accumulated weights exploit local quadratic structure. The analysis also relies on explicit smoothness, boundedness, and representation assumptions.

  • The analysis assumes differentiable convex component losses, a convex domain, bounded stochastic gradients, and sometimes a finite domain.
  • Stochastic Rounding (SR): Stochastic-rounding updates are rewritten with an explicit quantization-error term whose expected size decreases as the learning rate decreases.
  • Stochastic Rounding (SR): SR converges to an accuracy floor, and its error approaches high-precision performance linearly as the quantization step Δ becomes finer.
  • BinaryConnect (BC): BC analysis assumes Lipschitz-bounded Hessians; under smoothness and finite diameter, BC also converges only up to an error floor.
  • BinaryConnect (BC): For quadratic losses, BC’s accumulated real-valued weights can converge to the true minimizer, suggesting an advantage over SR near quadratic local minima.
  • BinaryConnect (BC): The BC convergence guarantees concern the real-valued buffer rather than quantized weights, whose values may fail to converge in the usual sense.

5 What About Non-Convex Problems?

For non-convex problems, shrinking the learning rate improves BC's concentration near minimizers but does not improve SR's long-term iterate quality. SR instead slows exploration while retaining a stationary distribution that is not concentrated on local minimizers.

  • Exploration-exploitation: SR lacks the usual exploration-exploitation tradeoff: smaller stepsizes slow the algorithm without improving the quality of its iterates.Unlike typical continuous-valued SGD, SR does not become progressively more likely to produce low-loss iterates as the stepsize decreases.
  • Transition behavior: For sufficiently small α, SR makes approximately the same conditional transitions as at α/10, but takes about ten times longer to make them.The transition decisions are effectively unchanged after conditioning on a parameter update; only their absolute probabilities decrease.
  • Empirical behavior: Shrinking α makes BC iterates concentrate aggressively on a minimizer, whereas SR iterates remain widely distributed and stagnate.This behavior appears in the toy problem as α decreases from 1 to 0.001.
  • Stationary behavior: SR's limiting stationary distribution remains positive on every state and therefore is not concentrated on local minimizers.The analysis models SR as a Markov chain whose transition probabilities depend on the current iterate and learning rate.
  • Mixing time: As α shrinks, SR's mixing time increases, so exploration slows even though no exploitation benefit is realized.The stationary distribution is nearly invariant to sufficiently small α because relative transition probabilities remain similar while absolute probabilities decrease.
  • Scope of analysis: The Markov-chain analysis does not require a finite state space; the transition operator may instead be a linear operator.The paper retains the term “matrix” as standard terminology.

6 Experiments

Experiments compare quantized training methods across image-classification models and datasets, showing that BC-ADAM approaches full-precision performance while SR-ADAM benefits from larger batches.

  • 6 Experiments: Experiments train binarized VGG-like and Residual networks across CIFAR-10, CIFAR-100, and ImageNet.Models include ResNet-56, WRN-56-2, VGG-9, VGG-BC, and ResNet-18.
  • 6 Experiments: BC-ADAM has comparable performance to full-precision ADAM, while SR-ADAM outperforms R-ADAM across the evaluated models and datasets.A performance gap remains between SR-ADAM and BC-ADAM.
  • 6 Experiments: SR-ADAM changes more weights than R-ADAM and BC-ADAM, reaching nearly 40% changed weights, whereas BC changes no more than 20%.Random reassignment of binary weights would produce 50% change, providing a reference for the observed exploration.
  • 6.1 A Way Forward: Big Batch Training: Larger batch sizes improve SR performance consistently, while BC performs best with small batches.Large-batch SR changes weights less aggressively and less frequently than small-batch SR.
  • 6.1 A Way Forward: Big Batch Training: The experiments suggest that big-batch methods may help lower-precision training exploit reduced precision and accelerate training.This implication follows the observed performance improvement for SR with larger batches.

7 Conclusion

The conclusion presents theoretical analyses of BinaryConnect and Stochastic Rounding, including convergence guarantees and distinct behavior on general non-convex problems.

  • 7 Conclusion: The paper proves convergence results for BC and SR that predict accuracy bounds depending on discretization coarseness.

Appendices

The appendices provide proofs of the main-paper lemmas and theorems together with additional experimental details and results.

  • Appendices: The appendices contain proofs of the lemmas and theorems presented in the main paper.
  • Appendices: They also include additional experimental details and results.

B Proof of Theorem 1

The proof develops convergence bounds by analyzing quantization error, stochastic-gradient variance, convexity, and diminishing step sizes, then applies the argument across theorem cases.

  • B Proof of Theorem 1: The convergence argument uses the update rule, the average iterate, and a telescoping sum over iterations.
  • B Proof of Theorem 1: The proof invokes bounded variance, zero-mean quantization error, and strong convexity to simplify the distance-to-optimum bound.
  • B Proof of Theorem 1: With stepsize α_t = 1/[µ(t + 1)], the analysis derives a decreasing-stepsize convergence relation.
  • B Proof of Theorem 1: Jensen’s inequality is used to pass from iterate-wise bounds to a bound involving the averaged iterate.
  • B Proof of Theorem 1: Additional theorem cases repeat the update-rule and expectation argument, combining intermediate inequalities into convergence results.
  • B Proof of Theorem 1: The final convergence statements are obtained after applying the corresponding expectation and quantization-error bounds.

F Proof of Theorem 5

The proof approximates the stochastic-rounding transition matrix by a nearest-neighbor Markov operator with uniformly small error, then transfers stationary-distribution behavior from the approximation to the original chain.

  • Operator construction: The associated operator ˜Tα is constructed by filling diagonal entries so its columns sum to one, with α0 ensuring non-negative entries for α < α0.This makes ˜Tα a valid Markov-chain transition operator.
  • Stationarity: The stationary distribution ˜π of ˜Tα0 is also stationary for ˜Tα when α < α0.The proof verifies this by substituting ˜π into the transition operator.
  • Transition structure: For small α, transitions changing multiple coordinates are vanishingly unlikely, while single-coordinate transitions determine the leading behavior.The proof therefore focuses on neighboring states differing in exactly one coordinate.
  • Transition approximation: Tα = ˜Tα + O(α2), because off-diagonal entries are uniformly approximated and diagonal entries follow from column normalization.The approximation error is uniform over state pairs.
  • Stationarity: For sufficiently small α, Tα has a stationary distribution that approaches ˜π.This follows by relating the leading eigenvectors of the perturbed and approximating operators.
  • State accessibility: The approximation chain assigns positive probability to every state because neighboring states are mutually accessible through non-zero transitions.Connectivity yields ˜π(x) > 0 for every state x.

H.1 Neural Net Architecture & Training Details

The experiments use VGG-like and residual classifiers with convolutional-weight binarization, and examine training curves and weight-decay effects across several image-classification settings.

  • Neural Net Architecture & Training Details: The study trains VGG-like and residual networks on CIFAR-10, CIFAR-100, and ImageNet.The evaluated models include VGG-9, VGG-BC, ResNet-56, WRN-56-2, and ResNet-18.
  • Neural Net Architecture & Training Details: Training uses minibatches, generally of size 128, while big-batch SR-ADAM uses sizes 512 or 1024 depending on the model.The experiments are implemented in Torch7 and run on NVIDIA GPUs without weight decay by default.
  • Neural Net Architecture & Training Details: Only convolutional-layer weights are quantized during training; linear layers remain unquantized because they offer less computational benefit.The stated rationale is lower fully connected-layer computation and parameter counts relative to convolutional layers.
  • Neural Net Architecture & Training Details: Figure 6 reports solid training-error and dashed testing-error curves for the evaluated network architectures.The curves correspond to the training and testing errors summarized in Table 1.
  • Neural Net Architecture & Training Details: BC-ADAM with 1e-5 weight decay performs worse than with zero weight decay.Weight decay shrinks wr toward zero and increases the distance between binary wb and real-valued wr.
  • Neural Net Architecture & Training Details: Weight decay cannot decay SR weights because ||wb||2 is identical across binarized networks.Thus, the regularization effect differs between BC-ADAM and SR.
Loading 1706.02379v3…