PaperlayerView exact revision on arXiv

Source-linked AI summary

Trained Ternary Quantization

Chenzhuo Zhu, Song Han, Huizi Mao, William J. Dally

arXiv:1612.01064v3cs.LG

TL;DR

Large neural networks are difficult to deploy on mobile devices with limited power budgets, motivating compression that preserves accuracy. TTQ learns ternary weight values and assignments during training, and its models reach or surpass full-precision accuracy while reducing parameter size by at least 16x and exceeding prior ternary networks by 3% on ImageNet.

  • Problem

    Deploying large neural networks on mobile devices is difficult because their size and energy demands conflict with limited power budgets.

  • Method

    TTQ learns two scaling coefficients and ternary assignments through back-propagation while maintaining latent full-precision weights during training.

  • Results

    At least 16x parameter reduction accompanies accuracy that reaches or surpasses full-precision models, while ImageNet accuracy exceeds prior ternary networks by 3%.

  • Takeaways & Limitations

    The deployed network needs only ternary weights and scaling coefficients, supporting compact inference models with potential custom-hardware acceleration.

Abstract

from arXiv · show

Deep neural networks are widely used in machine learning applications. However, the deployment of large neural networks models can be difficult to deploy on mobile devices with limited power budgets. To solve this problem, we propose Trained Ternary Quantization (TTQ), a method that can reduce the precision of weights in neural networks to ternary values. This method has very little accuracy degradation and can even improve the accuracy of some models (32, 44, 56-layer ResNet) on CIFAR-10 and AlexNet on ImageNet. And our AlexNet model is trained from scratch, which means it's as easy as to train normal full precision model. We highlight our trained quantization method that can learn both ternary values and ternary assignment. During inference, only ternary values (2-bit weights) and scaling factors are needed, therefore our models are nearly 16x smaller than full-precision models. Our ternary models can also be viewed as sparse binary weight networks, which can potentially be accelerated with custom circuit. Experiments on CIFAR-10 show that the ternary models obtained by trained quantization method outperform full-precision models of ResNet-32,44,56 by 0.04%, 0.16%, 0.36%, respectively. On ImageNet, our model outperforms full-precision AlexNet model by 0.3% of Top-1 accuracy and outperforms previous ternary models by 3%.

1 INTRODUCTION

Trained Ternary Quantization (TTQ) learns ternary weights and their assignments, targeting substantial compression while preserving or improving accuracy. During inference, it uses ternary values and scaling factors, retaining sparse and low-multiplication benefits.

  • Method: Trained Ternary Quantization uses two trainable scaling coefficients per layer to quantize weights into asymmetric ternary values.The ternary values differ from traditional fixed or symmetric schemes because positive and negative magnitudes are learned separately.
  • Method: Latent full-precision weights and scaling coefficients receive gradients during training, enabling the model to learn ternary assignments and values.The latent weights are discarded at test time.
  • Results: TTQ achieves higher accuracy on both CIFAR-10 and ImageNet, including the reported AlexNet ImageNet comparisons.The introduction presents these results as evidence that ternary quantization can preserve or improve accuracy.
  • Inference: Inference retains few-multiplication benefits because layer scaling factors are fixed, with custom hardware requiring only two multiplications per activation.The hardware-specific operation is described as an inference advantage.

2 MOTIVATIONS

The paper motivates quantization as a way to make deep neural networks easier to deploy on battery-constrained devices. The central challenge is reducing model size and energy use without sacrificing performance.

  • Deployment constraints: Battery constraints make energy efficiency a bottleneck for deploying deep neural networks on mobile devices.The paper notes that current models can contain hundreds of millions of parameters.
  • Deployment constraints: Smaller models reduce overhead for client exports, over-the-air updates, app distribution, and on-device deployment.The motivation includes autonomous-driving updates and app-size limits.
  • Compression–performance trade-off: Quantizing 32-bit weights to one or two bits reduces model size and memory references, but compressed weights usually degrade performance.This compression–performance contradiction motivates trained ternary quantization.

3 RELATED WORK

Related work compresses neural networks through binary or ternary quantization, low-bit training, pruning, and clustered weights. These approaches differ in their quantization values, gradient treatment, scaling rules, and whether assignments are learned or fixed.

  • Binary quantization: Binary and ternary connections compress weights and can speed inference, while latent full-precision weights support back-propagation through nondifferentiable quantization.Binary neural networks require only quantized values during inference and can achieve 32x smaller models.
  • Low-bit training: DoReFa-Net quantizes weights, activations, and gradients with different bit widths, enabling low-bit multiplication algorithms or hardware to accelerate training and inference.Its related binary-weight method uses the mean absolute full-precision weight as a layer-wise scaling factor.
  • Ternary quantization: TWN adds zero as a third value and uses symmetric thresholds with a layer-wise scaling factor to form {-W_l, 0, +W_l}.The scaling factor and threshold are obtained by minimizing L2 distance between full-precision and ternary weights.
  • Ternary quantization: TWN reports validation accuracy very close to full-precision networks despite requiring an additional bit for ternary weights.Its gradients are calculated using the same back-propagation equation described for the preceding method.
  • Clustering and pruning: Deep Compression fine-tunes low-precision weights from a pretrained full-precision network using clustered weight groups whose assignments are fixed at the beginning.This distinguishes it from methods that use zero or symmetric thresholds for quantization.

4 METHOD

TTQ learns both ternary weight values and their assignments during training, then discards full-resolution weights for inference. Its procedure uses normalization, thresholding, trainable scaling coefficients, and gradient updates that allow layer-specific ternary behavior.

  • Quantization procedure: TTQ normalizes full-precision weights to [-1, +1], thresholds them into {-1, 0, +1}, and back-propagates gradients through the quantization process.The shared threshold factor t is the same across layers to reduce the search space.
  • Learned ternary assignments: Updating latent full-precision weights learns which ternary codebook value each weight receives, so assignments can change when weights cross thresholds.The latent weights are retained during training but discarded at inference.
  • Learned ternary values: Two independent, trainable scaling coefficients represent positive and negative ternary weights in each layer.Unlike prior methods, these scaling coefficients are learned together with the other model parameters.
  • Layer-wise behavior: Different layers evolve differently: the first quantized convolutional layer becomes less sparse, whereas the last convolutional and fully connected layers become more sparse.The first layer’s scaling-factor magnitudes decrease, while those of the later layers increase.
  • Benefits of trained factors: The learned scaling factors can provide asymmetry between positive and negative weights and act as learning-rate multipliers during back propagation.The paper identifies both effects as benefits of trained quantization factors.

5 EXPERIMENTS

TTQ is evaluated against full-precision and other low-bit networks on CIFAR-10 and ImageNet. The ternary models preserve or improve accuracy, including a 1.6% lower Top-1 error than a full-precision AlexNet baseline.

  • CIFAR-10: TTQ substantially improves accuracy over a binary-weight model while remaining close to the full-precision ResNet baseline on CIFAR-10.The comparison uses full-precision, binary-weight, and trained ternary ResNet models.
  • CIFAR-10: ResNet-32, ResNet-44, and ResNet-56 accuracy improves by 0.04%, 0.16%, and 0.36%, respectively, over full-precision models.The improvement increases with model depth.
  • ImageNet: The AlexNet ternary model is trained from scratch with full-precision weights retained only in the first convolutional and last fully connected layers.Other layer parameters are quantized to ternary values, and validation Top-1 and Top-5 error are reported.
  • ImageNet: 42.5% Top-1 error is achieved by ternary AlexNet, versus 46.1% for DoReFa-Net and 45.5% for TWN.The ternary model also reports 1.6% lower Top-1 error than the full-precision batch-normalized AlexNet baseline.
  • ImageNet: TTQ reduces the training–validation performance gap relative to DoReFa-Net and TWN in the AlexNet experiment.The authors interpret this pattern as indicating that trainable scaling coefficients help prevent overfitting to the training set.
  • ImageNet: TTQ obtains a substantial improvement over BWN and TWN on ResNet-18B, while the reported full-precision comparison uses Facebook’s implementation.The TTQ model is trained from scratch, whereas TWN is fine-tuned from a full-precision network.

6 DISCUSSION

The discussion examines TTQ’s compression and efficiency benefits through ternary sparsity. Accuracy is best at moderate sparsity, while ternary weights reduce storage and inference cost and retain recognizable filter patterns.

  • Compression and efficiency: 16× model-size savings result from using ternary weights instead of 32-bit weights.The method retains potential energy benefits from skipping zero-weight computations.
  • Trade-off between sparsity and accuracy: Minimum error occurs between 30% and 50% sparsity, while sparsity beyond 50% increases error by reducing model capacity too far.As sparsity rises from 0 to 0.5, both training and validation error decrease.
  • Trade-off between sparsity and accuracy: Using a threshold rather than fixed sparsity allows sparsity to vary across layers and produces better results.The threshold is the sole hyper-parameter introduced to reduce the search space.
  • Compression and efficiency: Ternary values and sparsity reduce memory references and multiplications, enabling lower inference speed and energy consumption than Deep Compression.The authors report higher accuracy alongside these efficiency benefits.
  • Compression and efficiency: Forward time is reduced to less than 30% of full-precision networks through sparsity in convolution layers, where most computations occur.This result uses layer-specific thresholds rather than a shared threshold across all quantized layers.
  • Kernel visualization: Ternary filters retain edge and corner detector patterns found in full-precision AlexNet despite their lower precision.The authors state that ternary filters can extract key features after a full-precision first convolution layer.
  • Kernel visualization: Empty filters and filters with a single non-zero value provide targets for further pruning to compress and accelerate the model.These redundant kernels are observed in convolution layers.

7 CONCLUSION

TTQ compresses neural-network weights to ternary values using trained scaling coefficients and back-propagation. The resulting networks reduce parameter size by at least 16× while matching or surpassing full-precision accuracy on CIFAR-10 and ImageNet, and exceeding prior ternary networks on ImageNet.

  • TTQ compresses network weights to ternary values using two trained scaling coefficients for each layer.
  • Scaling coefficients and latent full-precision weights are trained with back-propagation, while layer-wise thresholds quantize the weights.
  • During deployment, only ternary weights and scaling coefficients are needed, reducing parameter size by at least 16×.
  • TTQ reaches or surpasses full-precision model accuracy on CIFAR-10 and ImageNet.
  • On ImageNet, TTQ exceeds prior ternary networks by 3%.
Loading 1612.01064v3…