PaperlayerView exact revision on arXiv

Source-linked AI summary

Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks

Sambhav R. Jain, Albert Gural, Michael Wu, Chris H. Dick

arXiv:1903.08066v3cs.CVcs.AIcs.LG

TL;DR

The paper tackles accuracy loss and limited adaptability in low-precision quantization by learning thresholds with network-level gradients. TQT uses backpropagation under hardware-friendly quantization constraints, achieving floating-point or near-floating-point accuracy across many CNNs, including difficult networks, with rapid retraining. It also provides an automatic TensorFlow graph framework, while leaving broader quantization variants for future work.

  • Problem

    Low-precision quantization supports efficient DNN inference but reduces accuracy, while fixed calibrated thresholds cannot adapt during training or optimize the overall network loss.

  • Method

    TQT learns quantization thresholds with weights using backpropagation and gradient descent under uniform, symmetric, power-of-2-scaled, per-tensor quantization constraints.

  • Results

    The experiments achieve floating-point accuracy for 8-bit quantization and near-floating-point accuracy for 4-bit quantization for most networks, with all networks converging within 5 epochs.

  • Takeaways & Limitations

    TQT provides a fast-converging approach for high-accuracy quantization on fixed-point hardware and an automatic TensorFlow graph framework through Graffitist.

  • Takeaways & Limitations

    The study does not compare TQT with other state-of-the-art low-bitwidth schemes and leaves broader scaling and quantization variants for future work.

Abstract

from arXiv · show

We propose a method of training quantization thresholds (TQT) for uniform symmetric quantizers using standard backpropagation and gradient descent. Contrary to prior work, we show that a careful analysis of the straight-through estimator for threshold gradients allows for a natural range-precision trade-off leading to better optima. Our quantizers are constrained to use power-of-2 scale-factors and per-tensor scaling of weights and activations to make it amenable for hardware implementations. We present analytical support for the general robustness of our methods and empirically validate them on various CNNs for ImageNet classification. We are able to achieve near-floating-point accuracy on traditionally difficult networks such as MobileNets with less than 5 epochs of quantized (8-bit) retraining. Finally, we present Graffitist, a framework that enables automatic quantization of TensorFlow graphs for TQT (available at https://github.com/Xilinx/graffitist ).

1 INTRODUCTION

The paper addresses accuracy loss from low-precision quantization by learning quantization thresholds alongside weights using network-level gradients. TQT combines this training with hardware-oriented scaling constraints and an automatic TensorFlow quantization framework.

  • Low-precision quantization enables low-power, high-throughput DNN inference but reduces accuracy.
  • Fixed thresholds cannot adapt to changing weight and activation distributions or optimize directly for final network loss.
  • TQT treats quantization thresholds as learnable parameters trained with weights through standard backpropagation and gradient descent.
  • TQT trains thresholds to balance range and precision while supporting per-tensor and power-of-2 scaling for hardware implementation.
  • The paper provides fast-converging log-domain threshold training, analytical and empirical validation, and Graffitist for automatic TensorFlow graph quantization.

2 RELATED WORK

Related work increasingly targets hardware-friendly quantization, while existing threshold-training approaches use gradient formulations that may poorly balance range and precision. The paper positions TQT against these methods and distinguishes its practical constraints and training behavior.

  • Prior quantization research spans binary, ternary, and higher-bitwidth networks trained with straight-through estimators.
  • Hardware-oriented methods commonly use uniform, per-tensor, power-of-2-scaled, and symmetric quantization for efficient fixed-point computation.
  • TensorRT calibrates thresholds by local Kullback-Leibler divergence but uses floating-point scale factors and does not explore retraining.
  • QAT, FAT, and NICE train threshold-like parameters with clipped gradients, but the paper reports poor 8-bit performance on difficult networks such as MobileNets.
  • LSQ uses a similar gradient idea but directly learns scale factors, which the paper associates with stability issues and 90-epoch retraining requirements.

3 TRAINED QUANTIZATION THRESHOLDS

The paper defines a hardware-oriented symmetric quantizer with per-tensor, power-of-2 scaling and trains its thresholds and weights using backpropagation. Its threshold gradients balance clipping range against precision, while input gradients tighten weight and activation distributions.

  • 3.1 Quantizer Constraints: Symmetric quantization sets the zero-point to zero, allowing zero-point cross-terms to be dropped from matrix multiplication or convolution.This removes special handling associated with zero-points.
  • 3.1 Quantizer Constraints: The quantizer uses a single scale-factor for each weight or activation tensor and constrains it to a power of 2 for efficient fixed-point implementation.Power-of-2 scaling enables simple bit-shifts instead of fixed-point multiplication.
  • 3.2 Linear Quantizer - Forward Pass: The forward quantizer scales, rounds, saturates, and de-quantizes tensor values using a threshold-dependent scale-factor.The scale maps the next larger power-of-2 threshold to the largest supported quantized value, after which out-of-range values are clipped.
  • 3.3 Linear Quantizer - Backward Pass: The backward pass uses the straight-through estimator with round and ceil gradients set to 1 without treating those operations as identities.Thresholds are trained in the log-domain for improved stability.
  • 3.4 Interpretation of Gradients: Threshold gradients move clipping limits inward when most inputs lie within them and outward when most inputs lie outside them, producing a range-precision trade-off.At convergence, positive in-range and negative out-of-range gradient contributions cancel.
  • 3.4 Interpretation of Gradients: Jointly training thresholds and weights uses threshold gradients to optimize clipping and input gradients to nudge weights and activations toward tighter bounds.This mutual optimization is performed over the global network loss.

4 FRAMEWORK FOR TQT

Graffitist is an end-to-end TensorFlow framework that optimizes graphs for TQT quantization, supports static or retrain modes, and generates hardware-oriented inference graphs. Its implementation covers common layer types, fused kernels, and configurable INT8/INT4 precision modes.

  • Framework overview: Graffitist transforms TensorFlow graphs for TQT quantization and retraining, producing optimized output graphs through sequential pattern-matching transformations.The framework is experimental and continues to add operation types, layer topologies, graph optimizations, and compression techniques.
  • Quantization modes: Static mode calibrates per-tensor weight thresholds from maximum absolute values and activation thresholds by locally minimizing symmetric KL-J distance.Calibration proceeds in topological order so each layer receives quantized, fixed inputs before its own thresholds are selected.
  • Quantization modes: Retrain mode simultaneously trains quantization thresholds and weights against a global loss, with recovery achieved within 5 epochs of TQT retraining.Graffitist separately generates quantized training and inference graphs, with the latter modeling the target fixed-point implementation.
  • Precision modes: The framework uses INT8 as 8/8 weight/activation precision and INT4 as 4/8 precision, enabling 50% weight compression in INT4 mode.The 4/8 choice is guided primarily by available 4x8 multipliers and reduces the memory footprint for fetching weights.
  • Layer support: Graffitist quantizes compute, Leaky ReLU, and average-pool layers while leaving concat lossless through explicit input-scale merging.Leaky ReLU uses q16() before the α-multiply operation to retain higher internal precision.
  • Implementation: Fused quantization kernels reduce the memory overhead of native TensorFlow implementations and support training with larger batch sizes.The unfused implementation must store intermediate tensors for backward-pass gradient computation, limiting the maximum GPU batch size.

5 EXPERIMENTS

The experiments evaluate TQT and Graffitist on ImageNet CNNs under fixed-point-oriented quantization settings. The study compares pretrained floating-point, static, and retrained configurations while examining calibration, initialization, and training procedures.

  • Experimental setup: TQT is evaluated on five CNN families spanning VGG, Inception, ResNet, MobileNet, and DarkNet models trained and validated on ImageNet.The evaluated set includes 12 networks, using pretrained FP32 models and standard network-specific preprocessing.
  • Experimental scope: The study targets scalable INT8/INT4 quantization for generic fixed-point hardware and compares TQT retraining with static quantization and weight-only retraining.The paper does not compare TQT with other state-of-the-art low-bitwidth schemes because its simplifying constraints may not be ideal at lower bit-widths.
  • Calibration and initialization: Calibration uses 50 unlabeled validation images per network to initialize thresholds in both static and retrain modes.Weights use MAX when thresholds are not trained, while activations use KL-J distance calibration.
  • Calibration and initialization: Trained weight thresholds are initialized from standard deviations or weight-distribution percentiles rather than MAX.The experiments summarize the selected initialization scheme in Table 2.
  • Training procedure: The experiments use Adam with β1 = 0.9 and β2 = 0.999, learning rate 1e-2 for thresholds, and learning rate 1e-6 for weights.Learning rates are exponentially decayed, with threshold and weight schedules scaled according to batch size.
  • Training procedure: Thresholds are incrementally frozen after 1000 · (24/N) steps to reduce oscillations caused by crossing critical integer thresholds under power-of-2 scaling.Freezing proceeds every 50 steps in increasing order of absolute gradient magnitude.
  • Experimental setup: The comparison includes FP32 baselines, static INT8, FP32 weight-only retraining, INT8 weight-only retraining, INT8 weight-plus-threshold retraining, and INT4 weight-plus-threshold retraining.INT8 denotes 8/8 weight/activation precision, while INT4 denotes 4/8 precision.

6 DISCUSSION

TQT retraining converges within five epochs and recovers floating-point or near-floating-point accuracy across most tested networks, while trained thresholds expose a range–precision trade-off. The benefit is greatest for difficult networks and lower-precision settings, though INT4 remains below FP32 under the stated constraints.

  • Accuracy and convergence: All networks converge within 5 epochs, with reported accuracy variation bounded by approximately 0.2% from checkpoint selection, 0.1% from floating-point nondeterminism, and 0.05% from rounding.The table reports the best validation checkpoint and epoch count.
  • Accuracy and convergence: 8-bit quantization reaches floating-point accuracy for most networks, while 4-bit quantization reaches near-floating-point accuracy for most networks.Static quantization incurs higher loss than retrained methods.
  • Network-dependent benefits: For difficult networks such as MobileNets and DarkNets, TQT weight-and-threshold retraining yields up to 4% higher INT8 top-1 accuracy than weight-only training and can match FP32 accuracy.Easier networks such as VGGs, Inceptions, and ResNets often gain no added benefit from threshold retraining.
  • Network-dependent benefits: INT4 accuracy falls short of FP32, and the authors attribute this possibly to the quantization constraints and first/last layers not retaining full precision.First and last layers are quantized to a minimum of INT8 for mapping onto the same fixed-point hardware.
  • Range–precision trade-off: MobileNet depthwise-convolution weights move thresholds inward by up to 3 log-domain integer bins, favoring precision, whereas other layers may move thresholds outward to favor range.Figure 5 plots pre- and post-retraining distributions alongside initial and trained raw thresholds.
  • Range–precision trade-off: Larger positive threshold deviations occur with 8-bit than 4-bit retraining, consistent with favoring range when more precision bits are available.Figure 6 summarizes deviations from initialized to trained thresholds across networks.

7 CONCLUSION

The paper concludes that TQT supports accurate fixed-point inference under uniform, symmetric, power-of-2-scaled, per-tensor constraints, with log-domain training and Graffitist enabling practical deployment. It also identifies broader quantization settings as future directions rather than established results.

  • Contributions: TQT constrains quantization to uniform, symmetric, power-of-2-scaled, per-tensor forms to support generic fixed-point hardware.The paper presents this as the hardware-oriented scope of the method.
  • Contributions: The threshold-gradient formulation enables a range–precision trade-off, while log-domain threshold training with an adaptive optimizer provides fast convergence.The conclusion links these design choices to high-accuracy quantized networks.
  • Contributions: Graffitist provides automatic quantization and retraining of TensorFlow graphs using the proposed methods.The framework is presented as the paper’s implementation layer for applying TQT.
  • Empirical validation: The paper empirically validates TQT on standard ImageNet CNNs and reports state-of-the-art 8-bit MobileNet quantization results.The conclusion summarizes the experimental scope and reported MobileNet outcome.
  • Scope and future work: Future work includes testing non-power-of-2 scaling, per-channel quantization, and non-symmetric or non-uniform schemes beyond the ImageNet use case.The authors present these as relaxations or broader settings not covered by the current work.

B LOG THRESHOLD TRAINING

The paper trains thresholds in the log domain because direct threshold updates can become unstable when thresholds cross zero. This representation supports numerical stability and is part of the procedure’s convergence design.

  • Training requirements: Threshold training must address numerical stability, scale invariance, and convergence rather than applying raw-threshold gradients without modification.The paper frames these as the three main properties required for stable training.
  • Log-domain parameterization: The log-domain formulation is introduced as a stability-oriented parameterization for threshold updates.The surrounding discussion connects it to the paper’s requirements for stable and convergent optimization.
  • Log-domain parameterization: Training log2 t instead of t prevents updates from making thresholds negative, which would cause log2 t and the scale factor s to diverge.The log-domain variable has domain R and is convenient because log2 t already appears in the scale expression.

B.2 Scale Invariance

Raw and log threshold gradients lack the desired threshold- and input-scale invariance, while variance-normalized log gradients approximate these properties and improve convergence stability. Adam provides a simpler alternative because its adaptive updates supply gradient norming.

  • Scale invariance: Raw and log threshold gradients both fail to provide the desired threshold-scale and input-scale invariance.Figure 7 shows that moving from raw to log thresholds degrades both invariance properties rather than improving them.
  • Threshold scale invariance: Log-threshold updates become exponentially slower below the optimal integer threshold and faster, potentially unstable, above it.This asymmetry follows from the gradient magnitudes on the two sides of the power-of-2 jump.
  • Input scale invariance: Log-gradient magnitudes also depend on input scale, with threshold-adjusted magnitudes varying quadratically with that scale.Thus log-threshold training does not behave consistently across differently scaled input distributions.
  • Normed gradients: Normalizing log-threshold gradients by their bias-corrected moving-average variance closely approximates the desired scale-invariant gradients.The normalization also preserves the near-jump gradient-magnitude ratio relevant to convergence dynamics.
  • Optimization behavior: Normed log gradients with SGD converge well and remain within a single integer log-threshold band, unlike oscillatory raw-gradient training.Adam without explicit normalization also works well because its adaptive gradients provide built-in gradient norming.

B.3 Convergence

Power-of-2 scaling creates sharp threshold-gradient jumps and bang-bang dynamics around an optimal integer log-threshold. Normed log gradients with SGD bound jump sizes, while the paper uses Adam as a simpler adaptive-gradient implementation.

  • Threshold dynamics: Power-of-2 scaling creates sharp gradient jumps at integral log2 thresholds, producing bang-bang-like threshold updates.Downstream features do not expose intermediate non-power-of-2 scale factors, so the forward pass changes sharply at integer boundaries.
  • Threshold dynamics: Negative feedback drives the threshold to oscillate around a critical integer log2 t∗ where the gradient changes sign.The two sides of the boundary have approximately constant gradients within one integer threshold interval.
  • Normed SGD: Normed log-gradient SGD bounds each sawtooth jump by the learning rate α, enabling convergence within a threshold bin when α ≪ 1.The bound follows from the designed constraint |g̃_i| ≤ 1.
  • Adam implementation: Adam is used in practice because its adaptive updates provide gradient norming without redefining the gradients explicitly.The implementation uses unnormed log gradients with Adam rather than the analytically simpler normalized-gradient SGD scheme.
  • Adam guidelines: The recommended Adam settings are α = 0.01, β1 = 0.9, and β2 = 0.999, derived from bounds involving p and rg.The guidelines require α < 0.1/√p, 1/e < β1 < 1, and 1 − β2 ≪ 1/p when rg ≈ p.

C ANALYSIS OF ADAM CONVERGENCE

The Adam convergence analysis models periodic bang-bang threshold gradients to bound oscillation magnitude and estimate convergence time. Experiments support the predicted relationship between oscillation period and gradient ratio, while noise motivates conservative over-design.

  • Assumptions: The Adam analysis assumes periodic threshold-gradient cycles in which one gradient persists for one step and the other for T − 1 steps.The moving variance is treated as approximately constant when T ≪ 1/(1 − β2).
  • Derivation: Adam’s cycle analysis expresses threshold updates through exponentially smoothed means and variances before solving for zero net displacement.The derivation tracks m_i, v_i, and the accumulated Adam updates across one oscillation period.
  • Deviation bound: The worst-case analysis substitutes T ← rg and assumes rg ≫ 1 to derive an upper bound on the maximum threshold deviation.The critical point of the displacement is then used to determine the maximum excursion.
  • Gradient-ratio estimate: The gradient-ratio estimate is rg ≈ 6fp and is conservatively covered by a 10× over-design factor.This estimate comes from the fraction of inputs crossing a threshold boundary and the outer-to-inner gradient ratio.
  • Validation: For σ = 10^-2, the measured oscillation period is T ≈ 280 while the estimated gradient ratio is rg ≈ 272.Across the tested settings, Figure 9 supports the predicted order-of-magnitude relationship T ≈ rg.
  • Validation and limitations: The predicted maximum log-threshold deviations are α√rg = (1.6, 0.4, 0.7) from left to right, but noise can violate the one-step lower-bin assumption.When the assumption fails, especially for negative threshold deviations, convergence may require the recommended 10× safety margin.

D BEST OR MEAN VALIDATION

The evaluation compares cherry-picked best validation checkpoints with averages from five predetermined validations in the last epoch, while Figure 10 visualizes MobileNet distributions and threshold changes after TQT retraining.

  • Validation protocol: Validation is run every 1000 training steps, and the checkpoint with the best top-1 score is saved.This procedure was retained during five epochs of retraining because intermediate validation was inexpensive.
  • Validation protocol: The authors compare positive-biased best validation with the average of five validations taken at predetermined last-epoch steps for two networks.The comparison quantifies the potential effect of checkpoint cherry-picking.
  • Distribution analysis: Figure 10 compares MobileNet v1 weight and activation distributions before and after quantized TQT retraining, alongside initial and trained raw thresholds.Each quantized layer is annotated with bit-width b and threshold deviation d := ∆⌈log2 t⌉.
Loading 1903.08066v3…