Source-linked AI summary
Mixed Precision Training of Convolutional Neural Networks using Integer Operations
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradeep Dubey, Jesus Corbal, Nikita Shustrov, Roma Dubtsov, Evarist Fomenko, Vadim Pirogov
TL;DR
Existing mixed-precision training evidence centers on FP16 and is limited for INT16 training of SOTA CNNs on large datasets. The paper introduces shared-exponent DFP training with integer operations and reports competitive accuracy with faster ImageNet-1K training.
Problem
Prior mixed-precision results were dominated by FP16, while INT16 studies had limited coverage of SOTA networks or large datasets.
Method
The paper uses shared-exponent INT16 tensors, INT16-to-INT32 FMA operations, DFP conversions, and software overflow management on general-purpose hardware.
Results
The CNNs match or exceed FP32 accuracy on ImageNet-1K without hyper-parameter changes or extra iterations, while ResNet-50 reaches 75.77% Top-1 accuracy and training achieves 1.8X speedup over FP32.
Takeaways & Limitations
DFP16 enables INT16 training of ResNet-50, GoogLeNet-v1, VGG-16, and AlexNet on ImageNet-1K using general-purpose integer hardware.
Takeaways & Limitations
The implementation uses a single shared exponent for each tensor, with exponent handling and precision management performed in software.
Abstract
from arXiv · showhide
The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half-precision
1 INTRODUCTION
The paper targets faster mixed-precision training by combining INT16 tensor arithmetic with software-managed precision and overflow handling. It applies this setup to ImageNet-1K CNN training while preserving FP32 training settings and accuracy.
- Up to 2X or more training speedup motivates replacing FP32 with half-precision arithmetic.
- INT16 training differs from FP16 through higher precision, lower dynamic range, and distinct accumulation, scaling, and overflow semantics.
- The proposed setup uses INT16 tensors with shared exponents, INT16-to-INT32 multiply-accumulate, adaptive down-conversion, and overflow management.
- Specialized low-precision FMA instructions target the GEMM-like and convolution-heavy parts of forward propagation, back propagation, and weight-gradient computation.
- The method matches or exceeds FP32 Top-1 accuracy on ImageNet-1K without changing hyper-parameters or iteration counts, including 75.77% for ResNet-50.
- The paper organizes its presentation around prior work, DFP representation, kernels and training operations, experiments, and conclusions.
2 RELATED WORK
Prior reduced-precision training spans floating-point and custom fixed-point representations, with earlier studies demonstrating both broad FP16 training and specialized fixed-point benefits.
- Reduced-precision deep learning uses standard floating-point formats alongside custom fixed-point schemes.
- Prior FP16 work used FP16 storage and computation with FP32 accumulation and retained FP32 master weights for updates.
- Custom fixed-point formats provide flexibility in precision and dynamic range, and dynamically scaled fixed point achieved up to 4X improvement over an aggressively tuned floating-point CPU implementation.
- Other sub-16-bit studies explored dynamic fixed point, binary weights, binary activations, and combinations retaining gradients or weights in full precision.
3 THE DYNAMIC FIXED POINT FORMAT
DFP represents tensors as integer values with shared exponents, enabling software-managed precision for integer neural-network operations. The paper defines conversion, arithmetic, and down-conversion primitives for this representation.
- 3 THE DYNAMIC FIXED POINT FORMAT: A DFP-P tensor pairs an integer tensor I with a shared exponent Es across all elements, where P specifies integer bit-width.
- 3 THE DYNAMIC FIXED POINT FORMAT: DFP-16 trades precision and dynamic range against float and half-float, while Blocked-DFP can increase effective range through multiple fine-grained exponents.
- 3 THE DYNAMIC FIXED POINT FORMAT: Figure 1 compares precision and dynamic range across IEEE-754 float, IEEE-754 half-float, and DFP-16.
- 3 THE DYNAMIC FIXED POINT FORMAT: The implementation uses one signed 8-bit shared exponent per tensor, 2’s-complement integers, and commodity integer hardware with exponent handling in software.
- 3.1 DFP TENSOR PRIMITIVES: Converting a floating-point tensor to DFP derives its shared exponent from the absolute maximum value’s exponent.
- 3.1 DFP TENSOR PRIMITIVES: The resulting DFP tensor represents each floating-point element as fn = in × 2^Es, linking integer values to the shared exponent.
- 3.1 DFP TENSOR PRIMITIVES: DFP multiplication and addition produce 32-bit integer tensors with new shared exponents, while FMA products and their sum share the product exponent.
- 3.1 DFP TENSOR PRIMITIVES: Down-conversion right-shifts a DFP-32 output by Rs bits to fit a DFP-16 tensor and computes a new shared exponent.
4 NEURAL NETWORK TRAINING USING DYNAMIC FIXED POINT
The training pipeline uses INT16 Dynamic Fixed Point convolution kernels with FP32 outputs and quantization between operations, while managing INT32 accumulator overflow through partial accumulation and conversion.
- Core compute kernels: Training uses INT16 kernels for forward propagation, back-propagation, and weight-gradient convolutions, integrated with floating-point conversions for other operations.The convolution kernels consume DFP-16 tensors and produce FP32 outputs that are quantized for subsequent layers.
- Core compute kernels: The AVX512_4VNNI instruction performs paired INT16 multiply-adds with horizontal accumulation into INT32 lanes.Its data layout is arranged to match the instruction’s two-way horizontal accumulation.
- Handling overflows: INT16 products can overflow INT32 accumulators after only three products, while training accumulation chains can exceed one million terms.This makes overflow management necessary, especially for weight-gradient computation.
- Handling overflows: The implementation converts partial INT32 results to FP32 and scales them before accumulating into the final FP32 output.The scale is 2^(E_inp+E_wt), derived from the shared DFP exponents.
- Handling overflows: Overflow-management overhead ranges from less than 1% in most cases to at most 3%.The added conversion instruction is the main difference from an ideal kernel without overflow handling.
- Handling overflows: The method constrains accumulation-chain length and shifts inputs by 1 bit to prevent overflow and catastrophic training errors.Partial INT32 accumulation is used for short chains before conversion to FP32.
5 EXPERIMENTS AND RESULTS
Experiments compare DFP16 and FP32 training on ImageNet-1K CNNs using the same framework and training configuration. DFP16 closely follows FP32 accuracy while delivering substantial throughput gains.
- Experimental setup: Experiments compare mixed-precision DFP16 with FP32 training for several ImageNet-1K CNNs on Intel Knights-Mill hardware.The evaluation uses modified and baseline BVLC Caffe implementations across up to 32 training nodes.
- Experimental setup: The baseline and DFP16 runs use the same batch size and hyper-parameter configuration, with models trained from scratch using synchronous SGD.The first convolution and fully connected layers remain in FP32 and account for about 5–10% of compute.
- Accuracy results: 75.77% Top-1 and 92.84% Top-5 accuracy are reported for ResNet-50 with mixed-precision DFP16.The authors identify these as the highest reported accuracies on ImageNet-1K classification with reduced-precision training.
- Accuracy results: DFP16 closely tracks FP32 convergence across ResNet-50, GoogLeNet-v1, VGG-16, and AlexNet.For GoogLeNet-v1 and AlexNet, early gaps relative to FP32 close after subsequent epochs, especially following learning-rate changes.
- Performance discussion: 276 images/sec and 1.8X speedup over FP32 are achieved for ResNet-50 with optimized mixed-precision training.Further SGD optimization raises throughput to 317 images/sec, while similarly tuned FP32 reaches 194 images/sec and yields a 1.6X DFP16 speedup.
6 CONCLUSIONS
The work demonstrates mixed-precision DFP16 integer training for large CNNs on ImageNet-1K, matching or exceeding FP32 accuracy while targeting computation, communication, and storage savings.
- Conclusions: DFP16 enables INT-based training for ResNet-50, GoogLeNet-v1, VGG-16, and AlexNet on ImageNet-1K.The scheme uses dynamic fixed-point representation, shared exponent management, and general-purpose hardware integer pipelines.
- Conclusions: The trained networks achieve on-par or better accuracy than FP32 baselines while potentially reducing computation, communication, and storage by 2×.
- Conclusions: The method is positioned for future extension beyond visual CNNs to RNNs, LSTMs, GANs, and a wider range of applications.