Source-linked AI summary
Ternary Weight Networks
Fengfu Li, Bin Liu, Xiaoxing Wang, Bo Zhang, Junchi Yan
TL;DR
Deploying deep neural networks on edge devices is difficult because their storage and computational demands exceed limited device resources. TWNs address this by constraining weights to +1, 0, and -1, with an efficient approximate optimization, achieving reduced memory and computation while retaining strong benchmark performance.
Problem
Deep neural networks often require considerable storage and computational power, challenging deployment on devices with limited storage, battery power, and computing capabilities.
Method
Ternary weight networks constrain weights to +1, 0, and -1, formulate an optimization problem, and use a simple approximate ternary function for efficient computation.
Results
TWNs reduce memory usage about 16x and computation about 2x, while achieving similar classification performance to full-precision networks on MNIST and CIFAR-10 and outperforming BPWNs by more than 10% mAP in detection.
Takeaways & Limitations
TWNs provide a balance between model compression and accuracy, with an extra zero state that does not require multiplication or accumulation and supports hardware-friendly computation.
Takeaways & Limitations
The approximate threshold solution assumes the weights are generated from a uniform or normal distribution, rather than solving the discrete optimization directly.
Abstract
from arXiv · showhide
We present a memory and computation efficient ternary weight networks (TWNs) - with weights constrained to +1, 0 and -1. The Euclidian distance between full (float or double) precision weights and the ternary weights along with a scaling factor is minimized in training stage. Besides, a threshold-based ternary function is optimized to get an approximated solution which can be fast and easily computed. TWNs have shown better expressive abilities than binary precision counterparts. Meanwhile, TWNs achieve up to 16$\times$ model compression rate and need fewer multiplications compared with the float32 precision counterparts. Extensive experiments on MNIST, CIFAR-10, and ImageNet datasets show that the TWNs achieve much better result than the Binary-Weight-Networks (BWNs) and the classification performance on MNIST and CIFAR-10 is very close to the full precision networks. We also verify our method on object detection task and show that TWNs significantly outperforms BWN by more than 10\% mAP on PASCAL VOC dataset. The pytorch version of source code is available at: https://github.com/Thinklab-SJTU/twns.
1. INTRODUCTION AND RELATED WORK
Deploying deep neural networks on edge devices is difficult because their storage and computational demands can exceed limited device resources. Prior binarization methods reduce these costs, while the paper introduces ternary quantization as a more expressive alternative.
- Deep neural networks can overburden the limited storage, battery power, and computational capabilities of wearable and edge devices.
- Binary weight and activation methods reduce computation by replacing many multiply-accumulate operations with simple accumulation and achieve up to 32× model compression.
- The paper presents a ternary weight quantization scheme that constrains weights to +1, 0, and -1 for reducing neural-network storage and computational costs.
- A threshold-based ternary function provides an approximated and universal solution for calculating ternary weights from raw neural networks.
- Experiments evaluate the proposed approach on public benchmarks for image classification and object detection.
2. TERNARY WEIGHT NETWORKS
Ternary Weight Networks constrain weights to +1, 0, and -1, balancing expressive capacity with reduced storage and computation. Training approximates full-precision weights using threshold-based ternarization while retaining full-precision parameters for updates.
- Expressive ability: TWNs constrain weights to +1, 0, and -1, providing more filter templates than binary weights.A 3×3 ternary filter has 19683 templates versus 512 for a binary filter.
- Model compression: 16× model compression is achieved with 2-bit weights compared with float32 counterparts.VGG-19 storage is reduced from approximately 500M to approximately 32M.
- Computational requirement: TWNs retain the binary networks’ multiply-accumulate count because zero terms require no accumulation.This preserves hardware-friendly computation while adding a zero state.
- Problem formulation: Training minimizes Euclidean distance between full-precision weights and scaled ternary weights under nonnegative scaling and ternary-value constraints.The approximation is W ≈ α W_tilde, with each ternary weight in {-1, 0, +1}.
- Threshold-based ternary function: A threshold-based ternary function assigns +1, 0, or -1 according to whether each weight is above, within, or below a positive threshold.The threshold-based approximation is introduced because direct optimization yields interdependent variables and no deterministic solution.
- Threshold-based ternary function: For a given threshold, the optimal scaling factor can be computed, while the threshold itself is approximated under distributional assumptions.The paper uses uniform or normal weight-distribution assumptions to obtain an efficient approximate solution.
- Training of Ternary-Weight-Networks: During training, ternarization is applied in forward and backward passes, while float32 weights are retained for parameter updates.Deployment keeps ternary weights and scaling factors, supporting the reported compression rate.
3. EXPERIMENTS AND DISCUSSION
Experiments compare TWNs with binary- and full-precision networks across classification and detection benchmarks. TWNs match full-precision performance on smaller classification datasets, narrow the ImageNet gap, converge stably, and outperform binary networks on PASCAL VOC detection.
- Experimental setup: TWNs are benchmarked against BPWNs and FPWNs on MNIST, CIFAR-10, ImageNet, and PASCAL VOC.The comparisons use classification and object detection tasks with shared architecture, regularization, learning-rate scaling, and optimization settings where specified.
- Classification results: On MNIST and CIFAR-10, TWNs achieve performance similar to FPWNs while outperforming BPWNs.
- Classification results: On ImageNet, TWNs and BPWNs perform below FPWNs, but TWNs have a smaller accuracy gap from FPWNs than BPWNs do.Increasing the backbone from ResNet18 to ResNet18B reduces the performance gaps between low-precision and full-precision networks.
- Training behavior: TWNs converge almost as quickly and stably as FPWNs, whereas BPWNs converge slowly and have less stable training loss.
- Detection results: More than 10% mAP separates TWNs from BPWNs on PASCAL VOC object detection.The models use YOLOv5-small, initialize from MS-COCO weights, and are fine-tuned for 150 epochs.
4. CONCLUSION
The paper concludes that TWNs provide an efficient ternary-weight design that balances accuracy, compression, and computational requirements. Experiments support their performance across public classification and detection benchmarks.
- TWNs reduce memory usage about 16x and computation about 2x while balancing accuracy and model compression rate.
- The proposed method combines an optimization formulation with a simple approximated ternary function.
- Empirical results on public benchmarks show superior performance for TWNs.