Source-linked AI summary
Convolutional Neural Networks using Logarithmic Data Representation
Daisuke Miyashita, Edward H. Lee, Boris Murmann
TL;DR
Large CNNs require low-precision computation for deployment on mobile and embedded platforms, but reducing fixed-point precision to very low bitwidths can harm performance. The paper uses non-uniform base-2 logarithmic representation for weights, activations, and dot-products, reporting higher accuracy than linear quantization at equivalent resolutions while eliminating bulky digital multipliers. It also introduces logarithmic backpropagation for end-to-end training.
Problem
Large CNNs have growing computational and memory demands, motivating low-precision arithmetic and compression for mobile and embedded deployment.
Method
The paper represents weights and activations logarithmically, computes dot-products with bitshifts, and incorporates logarithmic representation into training.
Results
Higher classification accuracies than linear quantization are achieved at equivalent resolutions, while state-of-the-art networks use 3b activations and weights with almost no loss in classification performance.
Takeaways & Limitations
Logarithmic representation can reduce precision and eliminate bulky digital multipliers while maintaining classification performance within the reported evaluations.
Takeaways & Limitations
The approach assumes trained weights are non-uniformly distributed around 0 and uses ReLU activations concentrated near 0.
Abstract
from arXiv · showhide
Recent advances in convolutional neural networks have considered model complexity and hardware efficiency to enable deployment onto embedded systems and mobile devices. For example, it is now well-known that the arithmetic operations of deep networks can be encoded down to 8-bit fixed-point without significant deterioration in performance. However, further reduction in precision down to as low as 3-bit fixed-point results in significant losses in performance. In this paper we propose a new data representation that enables state-of-the-art networks to be encoded to 3 bits with negligible loss in classification performance. To perform this, we take advantage of the fact that the weights and activations in a trained network naturally have non-uniform distributions. Using non-uniform, base-2 logarithmic representation to encode weights, communicate activations, and perform dot-products enables networks to 1) achieve higher classification accuracies than fixed-point at the same resolution and 2) eliminate bulky digital multipliers. Finally, we propose an end-to-end training procedure that uses log representation at 5-bits, which achieves higher final test accuracy than linear at 5-bits.
1. Introduction
CNNs achieve state-of-the-art image classification but increasingly demand computation and memory that challenge mobile and embedded deployment. The paper proposes logarithmic representation and arithmetic to reduce precision while preserving accuracy and simplifying hardware.
- 11.3 billion floating-point multiply-and-add operations per forward-pass and 230 MB of weights illustrate the deployment burden of a 152-layer network.
- 8-bit fixed-point networks support efficient deployment, but further simplification motivates methods that tolerate low-precision arithmetic.
- Log-domain weights, activations, and dot-products provide a general representation for efficient digital-hardware execution with minimal layer constraints.
- Non-uniform weight and activation distributions motivate logarithmic encoding, while logarithmic representations capture larger dynamic ranges in fewer bits than fixed-point.
- Higher classification accuracies result from logarithmic rather than linear quantization at equivalent resolutions.
- 3b activations and weights achieve almost no loss in classification performance on state-of-the-art networks.
- The paper also generalizes base-2 arithmetic across bases and develops logarithmic backpropagation for efficient training.
2. Related work
Prior work reduced network precision, trained with low-precision arithmetic, and developed specialized hardware, but these approaches often target particular architectures or hardware capabilities.
- Convolutional layers in AlexNet can be encoded to as little as 5 bits without a significant accuracy penalty.
- Stochastic rounding and quantized back-propagation extended low-precision computation to training, including powers-of-two multiplies realized with bitshifts.
- Specialized FPGA and accelerator research examined real-time propagation, classification performance, energy efficiency, and memory-versus-compute trade-offs.
3. Concept and Motivation
The paper proposes computing neural-network dot products in the binary logarithmic domain, using quantized weights and inputs to replace multiplication with bit operations. It also compares linear- and log-domain accumulation, highlighting trade-offs in precision, dynamic range, and hardware complexity.
- Dot-product computation: Each convolutional and fully-connected layer reduces to dot products, which can be computed in the log-domain rather than conventional floating- or fixed-point arithmetic.The conventional operation uses n multiply-and-add operations.
- Proposed logarithmic representation: Quantized logarithmic weights and inputs enable multiplication through bitshifting, eliminating expensive digital multipliers.The log-domain values are quantized to integers before bitshifting.
- Quantization: Log-domain quantization exploits the non-uniform distributions of weights and activations, with flooring and nearest-integer rounding offered as hardware-friendly options.Nearest-integer rounding is described as more precise than flooring.
- Proposed logarithmic representation: The second method transforms both operands into the log-domain, adds their quantized logarithms, bitshifts 1 by the resulting integer, and sums across terms.Linear-domain addition corresponds to summing exponentials, while linear multiplication becomes log-addition.
- Accumulation: Linear-domain accumulation is simpler but needs larger bit widths for dynamic range, whereas log-domain accumulation is more complicated yet computable with bit-wise digital hardware operations.The log-domain approximation uses log2(1 + x) ≃ x for 0 ≤ x < 1.
4. Experiments of Proposed Methods
The experiments evaluate logarithmic quantization for activations, fully connected weights, convolutional weights, and end-to-end training using AlexNet, VGG16, and CIFAR10 settings. Across these evaluations, logarithmic representations reduce quantization error, preserve classification accuracy at low bitwidths, eliminate multipliers, and improve trained-network accuracy over linear representation.
- Experimental setup: The evaluation uses published AlexNet and VGG16 models on ILSVRC-2012, testing inference with variable bitwidths and full-scale ranges without fine-tuning.The study also evaluates end-to-end training with logarithmic representation on CIFAR10.
- Activations: 34.19 versus 102.89 mean absolute activation quantization error is obtained for logarithmic versus linear quantization.Logarithmic quantization represents smaller activation values more finely and equalizes activations across more output bins.
- Activations: 3-bit logarithmic activations are 6.2% better than 3-bit linear activations for VGG16, while 4-bit logarithmic activations lose no top-5 accuracy from float32.For AlexNet, 3-bit logarithmic activations perform 0.2% worse than 3-bit linear activations.
- Fully connected weights: Logarithmic computation produces a 0.4% gain over linear for AlexNet and a 0.2% decrease for VGG16 while avoiding multipliers.The comparison applies quantization jointly to fully connected weights and activations.
- Training with logarithmic representation: Final CIFAR10 test accuracies are 0.9379 for log-5b, 0.9253 for linear-5b, and 0.8862 for BinaryNet.The logarithmic network uses quantized gradients, whereas the linear-5b and BinaryNet comparisons use unquantized gradients.
5. Conclusion
The paper presents low-resolution logarithmic representations for weights and activations, motivated by their non-uniform distributions, and evaluates them on image classification networks.
- Log-domain weights and activations use low resolution while eliminating bulky digital multipliers.
- The method is evaluated on ILSVRC-2012 classification using pretrained AlexNet and VGG16 models.
- The paper also extends logarithmic representation to end-to-end training.