Source-linked AI summary
Hardware-oriented Approximation of Convolutional Neural Networks
Philipp Gysel, Mohammad Motamedi, Soheil Ghiasi
TL;DR
High computational complexity limits CNN deployment, motivating hardware-oriented approximation. Ristretto automatically quantizes and fine-tunes CNNs using fixed-point representations. CaffeNet can be quantized to 8-bit weights and outputs with accuracy change below 1% versus its 32-bit floating-point counterpart.
Problem
CNNs require extensive computation, parameters, and hardware resources, while the trade-off between bit-width reduction and accuracy loss lacks thorough investigation.
Method
Ristretto automatically analyzes layer weights and outputs, converts floating-point CNNs to fixed point, and optionally fine-tunes the quantized network.
Results
CaffeNet can be quantized to 8-bit for weights and layer outputs with accuracy change below 1% versus its 32-bit floating-point counterpart.
Takeaways & Limitations
Fixed-point approximation reduces hardware multiplication-unit size, memory bandwidth, and on-chip storage requirements without decompression overhead.
Takeaways & Limitations
The framework does not include more sophisticated compression schemes, such as network pruning or binarization, that could further reduce network size and computational complexity.
Abstract
from arXiv · showhide
High computational complexity hinders the widespread usage of Convolutional Neural Networks (CNNs), especially in mobile devices. Hardware accelerators are arguably the most promising approach for reducing both execution time and power consumption. One of the most important steps in accelerator development is hardware-oriented model approximation. In this paper we present Ristretto, a model approximation framework that analyzes a given CNN with respect to numerical resolution used in representing weights and outputs of convolutional and fully connected layers. Ristretto can condense models by using fixed point arithmetic and representation instead of floating point. Moreover, Ristretto fine-tunes the resulting fixed point network. Given a maximum error tolerance of 1%, Ristretto can successfully condense CaffeNet and SqueezeNet to 8-bit. The code for Ristretto is available.
1 INTRODUCTION
Deep CNNs achieve high classification accuracy but require millions of parameters and billions of operations. Ristretto addresses the unresolved trade-off between reducing numerical precision and preserving accuracy through automated, application-specific trimming.
- Deep CNNs contain millions of parameters and require billions of arithmetic operations.
- Convolutional layers contain over 90% of arithmetic operations, while fully connected layers contain over 90% of network parameters.
- Fixed point arithmetic reduces chip area, energy consumption, processing-element size, and memory requirements without decompression overhead.
- Ristretto automatically balances bit-width reduction against a specified maximum accuracy error through fast post-training analysis.
2 MIXED FIXED POINT PRECISION
Quantized convolutional and fully connected layers use mixed precision, assigning separate output and weight bit-widths while preserving sufficient accumulation width. The central design problem is balancing bit-width reduction against classification accuracy.
- Mixed precision assigns different bit-widths to different parts of a CNN.
- Layer activations are multiplied by weights and accumulated through a series of MAC operations to form outputs.
- m and n denote the bit-widths of layer outputs and weights, respectively.
- An adder tree grows from m+n+2 bits at its first level to m + n + lg2 x bits at its last level, where x is the number of multiplication operations per output.
- Each layer requires a balance between reducing output and weight bit-widths and maintaining classification accuracy.
3 DYNAMIC FIXED POINT
CNN components have substantially different dynamic ranges, limiting ordinary fixed-point representation. Dynamic fixed point groups values by shared fractional length and uses separate groups for layer outputs and weights.
- Large layers accumulate thousands of values, making network parameters much smaller than layer outputs.
- Dynamic fixed point addresses the limited dynamic-range coverage of ordinary fixed point.
- Values are grouped so numbers within each group share a constant fractional length.
- Each network layer uses separate fixed-point groups for outputs and weights because weights are normally significantly smaller.
- Dynamic fixed-point arithmetic can be implemented in hardware with bit shifters.
- The paper positions Ristretto as a fast, automated tool for condensing neural networks before accelerator deployment.
4 RISTRETTO: APPROXIMATION FRAMEWORK IN CAFFE
Ristretto takes a trained Caffe model, quantizes it into a fixed-point network, and optionally fine-tunes the result. Its workflow analyzes ranges and activations, then uses shadow weights to retrain discrete parameters with minimal intervention.
- Ristretto framework: Ristretto accepts a trained model and produces a condensed fixed-point network in Caffe format for hardware accelerators.
- Quantization flow: The quantization flow analyzes weight ranges, calibrates activation statistics, and determines fixed-point representations across five stages.
- Fine-tuning: Fine-tuning uses full-precision shadow weights while quantized weights perform forward propagation.
- Fine-tuning: Ristretto computes gradients using floating-point layer outputs during fine-tuning, while validation uses fixed-point layer outputs.
- Fine-tuning: Ristretto uses a learning rate an order of magnitude lower than the last full-precision training iteration and requires minimal human intervention.
- Fine-tuning: During fine-tuning, fixed-point conversion occurs for each batch, after which computation can proceed in floating point.
5 RESULTS
Ristretto evaluates fixed-point approximation across five CNNs, showing that dynamic fixed point preserves accuracy at reduced bit-widths and enables substantial hardware-resource savings.
- Impact of dynamic fixed point: Dynamic fixed point maintains stable CaffeNet accuracy at lower bit-widths than traditional fixed point.Traditional fixed point performs well down to 18-bit, but accuracy drops significantly with further reduction, whereas dynamic fixed point remains stable.
- Quantization of individual network parts: 8-bit convolution kernels and layer activations change accuracy by only 0.3% across all three evaluated networks.Fully connected weights are more sensitive, with a maximum absolute accuracy change of 0.9% at 8-bit.
- Quantization of individual network parts: LeNet weights reach 2-bit precision with an absolute accuracy change below 0.4%.This result is reported when quantizing the network weights while the other number categories remain in floating point.
- Fine-tuning of all considered network parts: The three ImageNet CNNs can be trimmed to 8-bit, with relative accuracy drops of 0.9% for CaffeNet and 2.3% for GoogLeNet after fine-tuning.LeNet and CIFAR-10 have relative accuracy drops below 0.6%, while SqueezeNet also remains below 0.6%.
- Fine-tuning of all considered network parts: Ristretto reduces SqueezeNet’s parameter size to less than 2 MB, supporting deployment in smart mobile systems.SqueezeNet is already designed as a small ImageNet CNN, and Ristretto further condenses it.
- Hardware implications: 8-bit and 4-bit approximations reduce multiplication-unit size by about one order of magnitude, memory bandwidth by 4–8X, and increase on-chip parameter capacity by 4–8X.These hardware-oriented benefits are reported across all five 32-bit floating-point networks.
6 CONCLUSION AND FUTURE WORK
Ristretto is a fast, automated Caffe-based framework that quantizes CNN weights and layer outputs to reduce hardware resource requirements while preserving accuracy. The authors identify further compression and complexity-reduction features for future work.
- CaffeNet can use 8-bit weights and layer outputs while keeping accuracy change below 1% relative to its 32-bit floating-point counterpart.
- Ristretto reduces memory requirements, processing-element area, and overall power consumption for hardware accelerators.
- The framework is in its first development stage, with shared weights, network pruning, and network binarization proposed as future features.