Source-linked AI summary
8-Bit Approximations for Parallelism in Deep Learning
Tim Dettmers
TL;DR
Parallel deep learning is constrained by communication bandwidth and latency, especially when scaling across GPUs and computers. The paper develops 8-bit approximations for transferred gradients and activations, finding preserved predictive performance and large-cluster speedups, while identifying batch-size-related scaling boundaries.
Problem
Communication bandwidth and latency bottleneck parallel deep learning, while data and model parallelism face scaling limitations related to batch size.
Method
The paper develops 8-bit approximation data types that compress 32-bit gradients and nonlinear activations for data and model parallelism.
Results
8-bit approximation preserves predictive performance across MNIST, CIFAR10, and ImageNet and achieves state-of-the-art speedups for model parallelism.
Takeaways & Limitations
8-bit approximation is an efficient method for parallelizing convolutional networks on very large GPU systems.
Takeaways & Limitations
Data parallelism does not scale indefinitely because small per-GPU mini-batches slow matrix operations, while larger batches slow convergence.
Abstract
from arXiv · showhide
The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.
1 INTRODUCTION
The paper targets communication bottlenecks in parallel deep learning by compressing transferred values, developing and evaluating 8-bit approximations for gradients and activations.
- Motivation: Communication is a major bottleneck because sequential backpropagation requires parameter updates to finish before the next optimization iteration.Parallel training therefore requires high-bandwidth, low-latency communication between GPUs and computers.
- Approach: The paper focuses on reducing the number or size of parameters transferred rather than overlapping communication with computation.This is identified as one of two major ways to improve parallel deep learning performance.
- Evaluation: 8-bit gradient approximation leaves error rates unchanged on MNIST, CIFAR10, and ImageNet for the evaluated parallel settings.The contribution specifically covers both model and data parallelism.
- Speedup: 50x and more speedup is predicted for 96 GPUs with 8-bit approximation, compared with up to 23x for 32-bit.The predictive model has about 1% relative error on known speedup data.
- Comparison: 8-bit approximation improves convergence rates in convolutional networks by circumventing problems with large batch sizes for GPU clusters.The paper compares its algorithm with similar work.
- Comparison: The paper claims that 8-bit approximation sets the state-of-the-art for model parallelism in general.
2 BACKGROUND
This section describes data and model parallelism, their scaling constraints, and GPU communication bottlenecks, then presents a hybrid sub-batch scheme that hides communication during computation.
- 2.1 DATA PARALLELISM: Data parallelism keeps the model constant, feeds different mini-batches to GPUs, and synchronizes gradients after each pass.It is efficient when models have few parameters or high computation per parameter.
- 2.1 DATA PARALLELISM: Data parallelism scales poorly with mini-batches below 128 per GPU and converges more slowly as batch size increases.These effects limit scaling through slow matrix operations and large-batch convergence.
- 2.2 MODEL PARALLELISM: Model parallelism keeps data constant while distributing layer parameters across GPUs and synchronizing outputs for every layer.It is efficient for layers with many parameters, such as fully connected layers.
- 2.2 MODEL PARALLELISM: Model parallelism performs poorly with larger mini-batches because the matrix synchronized across GPUs becomes larger.Its outputs must also remain numerically accurate because small deviations can produce large errors in later layers.
- 2.3 COMMUNICATION BOTTLENECKS: Naive four-GPU data parallelism can be slower than one GPU: synchronizing AlexNet’s 0.223GB of 32-bit parameters takes 128ms at 7GB/s.The network’s full forward-backward pass takes under 100ms on current-generation GPUs.
- 2.3 COMMUNICATION BOTTLENECKS: InfiniBand latency becomes unmanageable above 512 kilobytes for clusters with more than a dozen nodes, exceeding 0.5ms per message.The paper identifies latency as the biggest bottleneck in large-scale GPU clusters.
- Hybrid Parallelism: The hybrid scheme combines data parallelism in convolutional layers with model parallelism in fully connected layers using K sub-batches.Incoming sub-batch communication is hidden under forward-pass computation, and multiple updates can hide further communication.
3 8-BIT APPROXIMATION
The paper develops 8-bit data types for compressing gradients and activations in data- and model-parallel deep learning, balancing compact representation with approximation accuracy. It evaluates their implementation, predictive speedups, approximation errors, and training performance.
- Designing 8-bit data types: 8-bit representations reserve one sign bit and allocate the remaining bits to exponent and mantissa, but limited mantissa precision can produce large errors.With a 3-bit exponent, values ending in 2 to 2.499 are approximated by values ending in 2, yielding an average relative error of 22.5%.
- Designing 8-bit data types: Binary-tree mantissas broaden the representable range and reduce average relative error compared with direct mantissa precision.The tree bisects the interval (0.1, 1) according to the route through the tree.
- Designing 8-bit data types: Dynamic exponents further reduce approximation error by reallocating exponent bits, while sacrificing exponent range and some accuracy below 10^-3.The dynamic format can approximate 0.2345678 as 0.236719, but represents a maximum exponent of 10^-6 instead of 10^-7.
- Designing 8-bit data types: Exponent offsets are fitted separately because model-parallel activations can have larger, high-variance values, especially with piecewise-linear activation functions.The desirable offset is 10^2 to 10^4, depending mainly on the nonlinear activation function.
- Implementation and computational performance: Compression and decompression use GPU shared-memory lookup and binary search, averaging 1 and 0.5 nanoseconds per number on an NVIDIA GTX Titan.The implementation uses one thread per number for compression and decompression lookup.
4 COMPARISON TO OTHER METHODS
The paper compares 8-bit approximation with reduced-precision and quantization methods, finding distinct trade-offs across computation, model parallelism, and data parallelism.
- Dynamic fixed point uses runtime-adjusted exponents, but prior 10-bit computation and 12-bit update results were about 20% worse relative to state of the art.
- 8-bit dynamic binary trees approximate numbers better than linear quantization but cannot be used for fixed-point computation.
- Stochastic rounding was not tested for the 8-bit types but might improve performance and permit approximations using fewer than 8 bits.
- 1-bit quantization maintains cumulative error to stabilize gradients, but its immediate error is too high for stable, accurate model-parallel forward passes.
- 1-bit quantization performs well for medium-sized fully connected systems, while convolutional layers gain no advantage over 32 or 8 bits because communication can be hidden.
- Large systems using 1-bit quantization face rapidly increasing batch sizes that slow convergence; adaptive batch-size selection is used to mitigate this problem.
CONCLUSION
The paper concludes that 8-bit approximation can accelerate communication in parallel deep-learning training while retaining predictive performance. It also identifies future progress in hardware and algorithms as important for parallel computing.
- 8-bit approximation can considerably speed communication in parallel training on large GPU clusters while retaining predictive performance.
- The dynamic tree data type approximates random numbers better than other known data types, although approximation techniques perform similarly during training.
- Future advances may come from new hardware and algorithms that preserve backpropagation performance while improving parallelizability.
5 THEORETICAL MODELS FOR PARALLELISM
The theoretical models combine benchmarked computation and communication costs to estimate parallel speedups. Their analysis uses overlapping transfers and computation, validates predictions against known results, and extends the model to a 96-GPU cluster.
- AlexNet benchmarks measure convolutional kernels, pooling, and fully connected matrix multiplications with and without model parallelism.
- Data-parallel communication in convolutional layers can be completely hidden under computation when gradients from one layer synchronize during computation of the next.
- Model parallelism splits convolutional activities across K GPUs and transfers sub-batches while previous sub-batches undergo forward and backward computation.
- 8-bit transfers incur penalties only for some transfers because most overlap fully with matrix multiplication, while smaller model-parallel matrices save 3.16ms for 32-bit multiplication.
- The speedup model combines GPU count, fully connected and convolutional times, parallel fully connected time, and communication penalties.
- The model predicts a speedup within 1% of Krizhevsky’s actual speedup of 3.66, despite omitting buffer-stacking costs and using a different GPU.
- For 32 nodes and 96 GPUs, gradient synchronization takes about 1.9ms and does not bottleneck data parallelism in convolutional layers.
- In the cluster model, fully connected model parallelism induces slowdown because small matrix-multiplication times cannot offset communication latencies.