Source-linked AI summary
Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
Dipankar Das, Sasikanth Avancha, Dheevatsa Mudigere, Karthikeyan Vaidynathan, Srinivas Sridharan, Dhiraj Kalamkar, Bharat Kaul, Pradeep Dubey
TL;DR
Training large neural networks requires distributed computation, but synchronous SGD is difficult to scale efficiently without changing its behavior. The paper develops a system-level approach combining efficient single-node computation with optimized multinode strategies, achieving strong scaling and training throughput across CNN workloads. Its cloud results also show the practical boundary imposed by virtualized CPU and network resources.
Problem
Large neural networks require distributed training, while synchronous SGD is difficult to scale because each node processes a bounded minibatch.
Method
The paper combines efficient single-node computation with analytically designed multinode synchronous SGD, preserving hyperparameters, data, and algorithmic behavior.
Results
The approach extends scaling and time-to-solution for deep learning and achieves 90X scaling for 512-minibatch VGG-A on 128 nodes.
Takeaways & Limitations
Synchronous SGD can achieve high-throughput CPU training at scale while retaining convergence behavior identical to the single-node version.
Takeaways & Limitations
The AWS evaluation uses virtualized CPU and network resources over 10 Gigabit Ethernet, incurring higher overheads than the dedicated bare-metal HPC cluster.
Abstract
from arXiv · showhide
We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior. We perform a detailed analysis of scaling, and identify optimal design points for different networks. We demonstrate scaling of CNNs on 100s of nodes, and present what we believe to be record training throughputs. A 512 minibatch VGG-A CNN training run is scaled 90X on 128 nodes. Also 256 minibatch VGG-A and OverFeat-FAST networks are scaled 53X and 42X respectively on a 64 node cluster. We also demonstrate the generality of our approach via best-in-class 6.5X scaling for a 7-layer DNN on 16 nodes. Thereafter we attempt to democratize deep-learning by training on an Ethernet based AWS cluster and show ~14X scaling on 16 nodes.
1. Introduction
Large neural networks require distributed training to reach practical training times, but synchronous SGD is difficult to scale because each node processes only a bounded minibatch. This work analyzes and optimizes vanilla synchronous SGD without changing its algorithmic behavior, achieving strong CNN training performance.
- Motivation: Several Exaflops are required by the largest networks, making single-node or single-card implementations insufficient for training in hours or minutes.The paper motivates multinode training as necessary to address this computational gap.
- Approach: The authors scale vanilla synchronous SGD without changing hyperparameters, altering the algorithm, or compressing data.The approach differs from variants such as 1-bit SGD, elastic-SGD, and asynchronous SGD by preserving algorithmic behavior.
- Approach: The approach combines system balance equations with data-, model-, and hybrid-parallel strategies to identify designs suited to different neural-network layers.The paper also optimizes single-node computation through cache blocking, threading, register blocking, and instruction scheduling on x86 systems.
- Scope: The analysis is presented as generic and applicable to other non-x86 systems, including GPUs and accelerators.The paper positions this as a generality claim beyond the evaluated Xeon-based systems.
- Results: The implementation achieves approximately 90% efficiency for convolutional operations and 70% for fully connected layers on Intel Xeon E5-269Xv3 systems.The authors report best time-to-train for several CNNs and record training throughputs on Xeon-based systems.
2. Optimizing Computation in Neural Network Training
The paper treats neural-network training as repeated tensor computations whose performance depends on computation, memory traffic, and reuse. It optimizes these operations through cache blocking, vectorization, register blocking, and data layouts that improve memory and SIMD efficiency.
- Compute Patterns: Neural-network training can be represented as a task graph whose nodes are layer computations and whose edges are tensor data dependencies.This view motivates analyzing compute, memory bandwidth, threading, cache blocking, vectorization, and register blocking together.
- Compute Patterns: Convolution, backpropagation, and weight-gradient computation use closely related nested-loop structures with different multiply-and-accumulate operations.Their shared memory-access patterns allow one cache-blocking strategy to apply across all three operations.
- Cache Blocking: The bytes-to-FLOPs ratio measures memory traffic relative to computation using output, input, and weight footprints for a convolutional loop.The ratio becomes important when activations and weights do not fit entirely in the CPU cache hierarchy.
- Cache Blocking: Cache blocking is formulated as minimizing B/F while constraining the block size BS to fit within on-chip cache capacity.BS denotes block storage, CPB denotes computation per block, and Sizecache denotes available cache with double buffering considered.
- Vectorization and Register Blocking: The optimized forward-propagation loop combines cache blocking over input and output feature maps with register blocking over output spatial dimensions.The implementation also lays out data over SIMD-width groups to make innermost accesses contiguous and enable vectorized fused multiply-add operations.
- Cache Blocking: A 128 KB cache per thread maintains B/F ≤0.04 for most convolutional layers even with minibatch size 1.The paper also reports a B/F ratio of 0.54 for the OverFeat-FAST C5 example when using the stated layer dimensions.
- Vectorization and Register Blocking: Register blocking improves the ratio of vector fused-multiply-add operations to loads and stores while helping hide VFMA latency.The Xeon execution model used in the analysis supports 2 loads, 2 VFMAs, and 1 store per cycle.
3. Optimizing Communication
The paper analyzes computation–communication balance in synchronous SGD to determine scaling limits and choose among data, model, and hybrid parallelism. It shows that communication behavior depends strongly on feature-map size and partitioning strategy, motivating hybrid designs for different layers.
- Scaling analysis: Strong scaling partitions one synchronous SGD iteration across nodes while preserving equivalence to a single-node serial implementation.The analysis uses computation and communication balance equations to determine work-partitioning strategies.
- Data parallelism: Data-parallel communication-to-computation ratio depends only on output feature-map size and data-points assigned per node, not kernel size, feature-map counts, or stride.This ratio is derived assuming floating-point data and complete send/receive overlap.
- Data parallelism: Communication can overlap computation because weight gradients become available after backpropagation, while updated weights are needed only before the next iteration’s forward propagation.The overlap strategy estimates scalability across layers while accounting for an unavoidable first-layer communication bubble.
- Data parallelism: 128 nodes for OverFeat-FAST and 256 nodes for VGG-A are estimated for convolutional layers in a 256-minibatch training run, although fully connected layers scale less in practice.The estimates use two platforms: FDR InfiniBand and 10GigE Ethernet.
- Model parallelism: Model parallelism is generally favored only for large kernels and small minibatches in convolutional layers, but is typically better than data parallelism for fully connected layers when ofm > minibatch.The comparison follows the simplified condition ofm·kernelw·kernelh·(2 − overlap) > inputw·inputh·minibatch.
- Hybrid parallelism: Hybrid parallelism partitions both minibatch and feature-map dimensions, and for ofm=4096, minibatch=256, N=64 it reduces communication volume relative to pure model parallelism.The paper treats data and model parallelism as special cases of the hybrid scheme.
4. PCL-DNN Software Framework
PCL-DNN combines data handling, optimized x86 compute, and MPI-based communication modules for distributed deep-learning execution. The framework separates input preparation and communication from computation to avoid throughput bottlenecks.
- Framework components: PCL-DNN consists of data handling, an optimized x86 CNN/DNN compute library, and an MPI-based communications library.The compute library executes forward propagation, backpropagation, and weight updates.
- Data handling: The data handling module continuously preprocesses inputs without becoming a training or classification throughput bottleneck or competing for compute resources.It is required to keep preprocessed data continuously available to the compute library.
- Compute library: The compute library processes networks layer by layer and uses AVX2 vector instructions for efficient convolutional and fully connected operations on x86 hardware.It supports multiple filter sizes and strides.
- Communications library: The communications library runs on a dedicated thread to separate communication from compute and supports overlap of hybrid-parallel backpropagation communication with forward computation.Its design aims to prevent communication from bottlenecking the compute library.
5. Experimental Results
Experiments show that PCL-DNN achieves high single-node efficiency and strong synchronous-SGD scaling for CNNs and DNNs on dedicated CPU clusters and AWS EC2. Scaling preserves VGG-A convergence while extending to challenging speech-recognition workloads.
- Single-node Performance: 315 and 95 images/s were achieved for OverFeat-FAST and VGG-A scoring, while training reached approximately 90 and 30 images/s, respectively.Across minibatch sizes, VGG-A throughput remained nearly constant; smaller OverFeat training minibatches were slower because of load imbalance.
- Scaling Results: 90X scaling was achieved for 512-minibatch VGG-A on 128 Cori nodes, reaching 2510 images/s and 70% scaling efficiency.For a 256 minibatch, VGG-A scaled to 64 nodes with 82% efficiency.
- Scaling Results: VGG-A Top5 validation and training accuracy overlapped across 32-node and 64-node runs.The distributed implementation retained synchronous SGD and made no hyperparameter changes, so its convergence matched the single-node version.
- Scaling on Cloud: On 16 AWS EC2 nodes, OverFeat and VGG-A reached 1027 and 397 images/s, corresponding to 11.9X and 14.2X speedups.The experiments used virtualized CPU and network resources connected by 10 Gigabit Ethernet; VGG-A achieved better speedup because of its higher flops-per-network-byte requirement.
- Automatic Speech Recognition: PCL-DNN delivered 4600 frames/s for a 7-layer CD-DNN on one Xeon CPU and continued scaling to 29.5K frames/s on 16 nodes.The study describes DNN scaling as more challenging than CNN scaling because of higher communication-to-compute ratios.
6. Conclusions
The paper demonstrates high-throughput deep-learning training at scale using synchronous SGD on CPUs. It extends scaling and time-to-solution while providing detailed multinode training analysis.
- Conclusions: Synchronous SGD can train deep-learning models at scale with high throughput on CPUs.The work reports advances in scaling and time-to-solution together with detailed multinode training analysis.