Source-linked AI summary
Resiliency of Deep Neural Networks under Quantization
Wonyong Sung, Sungho Shin, Kyuyeon Hwang
TL;DR
Small-bit direct quantization can degrade DNN performance, motivating analysis of whether retraining and added network complexity improve resilience. The paper varies FFDNN and CNN complexity, evaluates retrained low-precision networks, and introduces effective compression ratio. Retraining nearly closes the ternary-versus-floating-point gap in performance-saturated networks, while complexity-limited networks remain less resilient.
Problem
Direct quantization does not perform well at small bit widths, leaving the resilience of FFDNNs and CNNs under retrained low-precision weights to be examined.
Method
The study varies FFDNN units and layers, CNN feature maps and layers, and evaluates direct and retrain-based quantization alongside effective compression ratio.
Results
The floating-point versus retrained ternary-weight performance gap almost vanishes in performance-saturated networks but remains when DNN complexity is reduced.
Takeaways & Limitations
After retraining, larger networks with extreme ternary quantization can use less weight memory than smaller higher-precision networks at the same accuracy.
Takeaways & Limitations
The analysis includes an assumption that weights within each group are not dependent on one another.
Abstract
from arXiv · showhide
The complexity of deep neural network algorithms for hardware implementation can be much lowered by optimizing the word-length of weights and signals. Direct quantization of floating-point weights, however, does not show good performance when the number of bits assigned is small. Retraining of quantized networks has been developed to relieve this problem. In this work, the effects of retraining are analyzed for a feedforward deep neural network (FFDNN) and a convolutional neural network (CNN). The network complexity is controlled to know their effects on the resiliency of quantized networks by retraining. The complexity of the FFDNN is controlled by varying the unit size in each hidden layer and the number of layers, while that of the CNN is done by modifying the feature map configuration. We find that the performance gap between the floating-point and the retrain-based ternary (+1, 0, -1) weight neural networks exists with a fair amount in 'complexity limited' networks, but the discrepancy almost vanishes in fully complex networks whose capability is limited by the training data, rather than by the number of connections. This research shows that highly complex DNNs have the capability of absorbing the effects of severe weight quantization through retraining, but connection limited networks are less resilient. This paper also presents the effective compression ratio to guide the trade-off between the network size and the precision when the hardware resource is limited.
1 INTRODUCTION
The paper examines whether retraining can preserve DNN performance under very low-precision quantization, while varying network complexity to measure its effect on resilience and hardware efficiency.
- Hardware DNN implementations can reduce power consumption and system size compared with GPU implementations.FPGA-based CNN implementations are reported to achieve more than 10 times the power-consumption advantage.
- Low-precision weight representation is motivated by the large number of multiply-and-add operations and weights in neural networks.Reducing word length can make homogeneous MAC-based hardware arrays more efficient.
- The study tests whether retraining recovers FFDNN and CNN performance with ternary weights or 3-bit, 7-level weights.Bias values are not quantized.
- Network complexity is varied to measure its effect on the performance gap between floating-point and retrained low-precision networks.The study changes unit size and layer count for FFDNNs and feature-map configuration for CNNs.
- Experiments use TIMIT phoneme recognition and CIFAR-10 image classification, and introduce effective compression ratio for comparing precision-size trade-offs.The metric is intended to guide network size and word-length choices for efficient DNN hardware.
2 RELATED WORK
Prior work studied fixed-point and low-precision neural networks to reduce hardware cost, including ternary and binary weight designs.
- Early fixed-point studies used statistical quantization-noise models and simulation-based word-length optimization for nonlinear algorithms.
- Ternary coefficients were used in digital filters to eliminate multiplications, at the cost of higher quantization noise.
- Earlier neural-network simulations varied hidden-unit counts and found good integer-arithmetic results for some benchmarks but not Protein.
- Recent fixed-point DNN studies reported ternary FFDNN and CNN performances close to floating-point results and used them in low-power VLSI and FPGA implementations.
3 FIXED-POINT FFDNN AND CNN DESIGN
The paper defines FFDNN and CNN architectures, varies their complexity, and applies direct quantization followed by retraining to reduce hardware precision and memory costs.
- FFDNN design: The FFDNN propagates each layer’s signal by multiplying weights, adding biases, and applying an activation function.
- CNN design: The CNN uses three convolutional and pooling layers plus a 64-unit fully connected hidden layer, while varying convolutional feature-map counts.
- FFDNN design: The FFDNN uses four hidden layers, varies each layer from 32 to 1024 units, and also reduces the number of hidden layers.
- Hardware motivation: Reducing weight word length lowers multiplier gate requirements, weight-storage memory, and power associated with precision or off-chip memory access.
- Quantization procedure: Direct quantization performs poorly at small bit widths, whereas retraining quantized weights can achieve better fixed-point performance.
- Quantization procedure: Uniform quantization uses an odd number of levels, with the step size selected to minimize L2 quantization error.Seven levels represent weights from -3·∆ through +3·∆ in 3 bits.
- Quantization procedure: Retraining propagates quantized weights but updates maintained floating-point weights before requantizing at each iteration.
4 ANALYSIS OF QUANTIZATION EFFECTS
Direct quantization becomes less damaging as network complexity increases, while retraining can nearly close the gap to floating-point performance in sufficiently complex FFDNNs and CNNs. Smaller, connection-limited networks remain more sensitive to low-precision weights.
- Direct quantization: Quantization distortion decreases slowly as the number of inputs to a unit increases because independent distortions partially cancel.This model predicts reduced quantization effects when the anterior layer contains more units.
- Direct quantization: 6-bit direct quantization is sufficient for larger FFDNNs, whereas smaller FFDNNs require 8 bits for near-floating-point performance; CNNs show a similar trend.For CNNs, about 6 bits suffice when the feature-map configuration is 16-16-32 or larger.
- Retraining: Retraining makes the fixed-point and floating-point performance gap converge rapidly as FFDNN hidden-unit size increases.The floating-point network saturates near 1024 hidden units, where the TIMIT experiment enters a training-data-limited region; smaller networks retain a larger gap.
- Depth and complexity: Both FFDNN and CNN results show that retraining reduces quantization effects when the network contains redundant complexity.The CNN’s 7-level quantization gap is very small across all tested feature-map levels.
5 EFFECTIVE COMPRESSION RATIO
The paper introduces effective compression ratio (ECR) to compare quantized networks with floating-point networks achieving the same accuracy, and uses it to guide precision–size trade-offs. Retraining makes larger networks with extreme ternary quantization especially memory-efficient at matched accuracy.
- Design guideline: The guideline selects bit-width and layer size according to total weight bits, target accuracy, or network size.The analysis assumes 2n −1 quantization levels represented with n bits and applies the same quantization level count across layers.
- Design guideline: Figure 7 evaluates framewise phoneme error rate against total weight bits while varying uniform hidden-layer size and quantization precision.The figure supports choosing a hardware design by comparing accuracy and total stored weight bits.
- ECR definition: ECR compares quantized-weight memory with the 32-bit floating-point memory required for a network achieving the same accuracy.The effective uncompressed size is obtained by finding a floating-point network with matching accuracy and multiplying its parameter count by 32 bits.
- ECR results: 5-bit direct quantization gives the best ECR except for layer size 1024, whereas 2-bit retrained quantization performs best across the examined alternatives.The comparison covers multiple network sizes and quantization bit-widths.
- ECR results: After retraining, a larger network with 2-bit ternary weights uses less memory than smaller higher-bit networks at the same accuracy.This finding makes network size and precision jointly important when optimizing memory usage.
6 DISCUSSION
The discussion links quantization resilience to network complexity and applies the findings to hardware-efficient DNN design. Low-precision arithmetic is most useful when added capacity can compensate for a small reference network, but less useful after performance saturation.
- Complexity and resilience: Reduced complexity lowers quantization resilience when units, feature maps, or hidden layers are reduced.The discussion also identifies recurrent-network quantization sensitivity as a direction for related experiments.
- Hardware design: For hardware-limited designs, very low precision plus increased network complexity is advised when the reference DNN is relatively small.The recommendation is framed around using available hardware capacity to expand the network.
- Hardware design: At performance saturation, enlarging an already-large network may add little benefit; Figure 7b and Figure 9b show 6-bit quantization best at layer size 1,024.This marks a boundary on the strategy of trading precision for additional network capacity.
7 CONCLUSION
The study finds that retrained ternary-weight networks nearly match floating-point performance in performance-saturated DNNs, while compact networks show a larger gap. Thus, network redundancy supports resilience to severe quantization, whereas connection-limited networks are less resilient.
- Main conclusion: Ternary-weight performance nearly matches floating-point performance when DNN capability is limited by training data rather than connections.This applies to the analyzed FFDNN and CNN under retrain-based quantization.
- Main conclusion: Reducing units, feature maps, or hidden layers increases the performance gap between floating-point and ternary fixed-point networks.The conclusion treats these configuration changes as reductions in network complexity.
- Main conclusion: Large networks can absorb lowered precision through redundant representation capability, whereas very compact networks cannot do so as effectively.The conclusion contrasts performance-saturated networks with connection-limited networks.