PaperlayerView exact revision on arXiv

Source-linked AI summary

FireCaffe: near-linear acceleration of deep neural network training on compute clusters

Forrest N. Iandola, Khalid Ashraf, Matthew W. Moskewicz, Kurt Keutzer

arXiv:1511.00175v2cs.CV

TL;DR

Long training times for high-accuracy DNNs hinder experimentation with new architectures and motivate scalable training. FireCaffe reduces communication overhead through hardware, communication, and batch-size choices, achieving large speedups on 128 GPUs while preserving reported accuracy in key cases.

  • Problem

    High-accuracy DNNs can take weeks to train on a modern GPU, slowing experimentation with new architectures.

  • Method

    FireCaffe combines high-bandwidth interconnects, reduction-tree communication, and optionally larger batches with tuned hyperparameters to scale DNN training.

  • Results

    FireCaffe achieves 47x speedup for GoogLeNet and 39x for NiN on 128 GPUs.

  • Takeaways & Limitations

    Reducing communication overhead enables near-linear speedups across several leading DNN architectures without degrading accuracy in the reported key result.

Abstract

from arXiv · show

Long training times for high-accuracy deep neural networks (DNNs) impede research into new DNN architectures and slow the development of high-accuracy DNNs. In this paper we present FireCaffe, which successfully scales deep neural network training across a cluster of GPUs. We also present a number of best practices to aid in comparing advancements in methods for scaling and accelerating the training of deep neural networks. The speed and scalability of distributed algorithms is almost always limited by the overhead of communicating between servers; DNN training is not an exception to this rule. Therefore, the key consideration here is to reduce communication overhead wherever possible, while not degrading the accuracy of the DNN models that we train. Our approach has three key pillars. First, we select network hardware that achieves high bandwidth between GPU servers -- Infiniband or Cray interconnects are ideal for this. Second, we consider a number of communication algorithms, and we find that reduction trees are more efficient and scalable than the traditional parameter server approach. Third, we optionally increase the batch size to reduce the total quantity of communication during DNN training, and we identify hyperparameters that allow us to reproduce the small-batch accuracy while training with large batch sizes. When training GoogLeNet and Network-in-Network on ImageNet, we achieve a 47x and 39x speedup, respectively, when training on a cluster of 128 GPUs.

1. Introduction and Motivation

Rapid DNN architecture development is constrained by training times that can stretch to weeks, motivating scalable training methods such as FireCaffe. The paper focuses on reducing communication overhead while preserving accuracy, reporting speedups on large GPU clusters.

  • Motivation: Weeks-long training for high-accuracy DNNs makes training time a key bottleneck in developing new architectures.Architectural variations require experimentation to assess their impact, even when single-GPU primitives operate near peak computation.
  • Motivation: Researchers value rapid experiment turnaround and often accept more system resources to minimize elapsed training time.The cited patience threshold is a few days or a week, and experiment scale is affected by waiting time.
  • Broader impact: Long training times delay bringing DNN-based solutions to market when companies train on much larger image databases.The paper contrasts ImageNet-1k with internal databases containing billions of images and hundreds of thousands of labels.
  • Broader impact: Accelerated training is presented as relevant both to existing DNN applications and to real-time or newly enabled uses such as robotic reinforcement learning.The paper discusses incorporating new data into models in real time for reinforcement learning.
  • FireCaffe: FireCaffe scales DNN training across 128 GPUs by reducing communication overhead with fast interconnects, reduction trees, and parallelization choices.The paper reports speedups exceeding 40x compared with a single GPU.

2. Hardware for scalable DNN training

FireCaffe evaluates scalable DNN training on concrete hardware because interconnect bandwidth and latency determine how much scaling can occur before communication dominates. The study uses Titan, whose Cray Gemini network supports this evaluation.

  • Hardware selection: Concrete hardware is required to demonstrate and verify FireCaffe’s correctness and real-world scalability.The paper uses theoretical or scale models as useful complements, but selects hardware based on node-to-node communication speed.
  • Hardware selection: High-bandwidth, low-latency interconnects enable greater scale by reducing the likelihood that communication overhead dominates training.Cray and Mellanox interconnects are described as substantially faster than typical Ethernet connections.
  • Evaluation platform: FireCaffe uses Oak Ridge’s Titan supercomputer, which provides a Cray Gemini interconnect and one NVIDIA K20x GPU per server.Titan has 18,000 servers in total.
  • Evaluation platform: Training runs use 32 to 128 GPUs, while Titan’s extra capacity allows multiple runs to proceed concurrently.The paper reports approximately 27 PetaFLOPS/s of total computational capacity.

3. Preliminaries and terminology

DNN training alternates forward and backward propagation under batched SGD, while the preliminaries quantify communication-relevant weight and activation volumes. The terminology fixes the meanings of weights, data, workers, epochs, and performance metrics.

  • Training process: DNN training alternates forward classification of a batch with backward computation of weight and data gradients.Weight gradients update the model’s weights, and batched SGD is used for popular models such as GoogLeNet.
  • Data volumes: Equation 1 calculates the combined byte size of weights in convolutional and fully-connected layers.The weight-size expression multiplies channel count, filter count, filter dimensions, and 4 bytes per floating-point value.
  • Data volumes: Equation 2 calculates the combined byte size of activations produced by all layers, including the batch dimension.Activation height, width, and batch size appear alongside channel count, filter count, and 4 bytes per floating-point value.
  • Terminology: The paper uses weights, parameters, filters, and W interchangeably, and likewise treats nodes, workers, and GPU servers as synonyms.Activations and data are also used interchangeably as D; an epoch is one pass through the training data.
  • Terminology: Performance is reported through specific metrics, especially accuracy and training time, rather than as an ambiguous general term.Fully-connected layers are defined as a special case of convolutional layers when filter and activation dimensions match.

4. Parallelism strategies

The paper distinguishes model and data parallelism by what each worker holds and communicates, then selects data parallelism for convolutional vision models because it usually requires less communication. FireCaffe implements this strategy by aggregating locally computed weight gradients.

  • Parallelism definitions: Model parallelism partitions model parameters and exchanges data gradients and activations, whereas data parallelism partitions each batch and exchanges weight gradients.The two strategies are defined for batched SGD training across multiple servers.
  • Strategy selection: The preferred parallelism strategy depends strongly on DNN architecture and should minimize communication between servers.Fully-connected speech-recognition models and convolutional computer-vision models present different communication characteristics.
  • Strategy selection: Convolutional vision models typically favor data parallelism because weight gradients are much smaller than data gradients at typical batch sizes.The paper states that popular vision architectures such as GoogLeNet have this property.
  • FireCaffe data parallelism: In FireCaffe, every GPU stores the full model, processes a batch subset, computes local weight gradients, and aggregates them across workers.Gradient aggregation uses either a parameter server or a reduction tree.
  • Architecture comparison: Table 1 compares data and computation volumes for four DNN architectures using batch size 1024, while weight size remains batch-independent.TFLOPS denotes the quantity of computation to perform.

5. Choosing DNN architectures to accelerate

DNN architectures with fewer parameters can retain high accuracy while requiring less communication for data-parallel training, making them more suitable for scaling.

  • Architecture and scalability: 8x fewer parameters let Network-in-Network match AlexNet’s accuracy, while 10x fewer let GoogLeNet match VGG’s accuracy.In data-parallel training, fewer parameters mean fewer weight-gradient bytes sent by each GPU worker.
  • Architectural choices: GoogLeNet and Network-in-Network reduce parameters through more 1x1 filters and smaller or absent fully-connected layers.AlexNet and VGG each have more than 150MB of fully-connected parameters, whereas GoogLeNet uses smaller fully-connected layers and Network-in-Network has none.
  • Architecture and scalability: More parameters do not necessarily produce higher classification accuracy.Figure 2 compares total parameter size with top-5 ImageNet accuracy across popular DNN architectures.
  • Architecture and scalability: Models with fewer parameters are more amenable to scalable data-parallel training while still delivering high accuracy.FireCaffe therefore focuses on accelerating models such as Network-in-Network and GoogLeNet.

6. Implementing efficient data parallel training

FireCaffe implements synchronous data-parallel training by aggregating weight gradients across GPUs, comparing parameter servers with reduction trees to reduce communication overhead.

  • Gradient aggregation: Gradient aggregation across GPUs produces the same numerical results as single-GPU training.Each worker sums weight gradients for its batch subset, after which the gradients are summed across GPUs.
  • Parameter server: A parameter server receives gradients from all workers, sums them, and sends the summed gradients back to every worker.The workers then apply these updates to their local model copies.
  • Parameter server: 2x more workers require at least 2x more parameter-server communication time per gradient update.The parameter server handles |∇W| ∗p bytes of traffic, and Figure 4 experimentally confirms the linear scaling.
  • Gradient aggregation: Allreduce reduces each worker’s gradient vector by element-wise addition and broadcasts the resulting vector to all workers.In this work, each worker contributes a vector of length |∇W|.
  • Reduction tree: Reduction trees aggregate gradients collaboratively across nodes and distribute the gradient sums back down the tree.A binary tree uses branching factor 2 and depth log2(p), yielding serialized communication of 2log2(p).
  • Reduction tree: O(log(p)) reduction-tree communication scales more efficiently than the parameter server’s linear communication with p.The logarithm base depends on the tree’s branching factor, and Figure 4 confirms the more efficient scaling experimentally.

7. Evaluation of FireCaffe-accelerated training on ImageNet

The evaluation measures FireCaffe on ImageNet-1k using public-dataset reporting, disclosed training settings, and single-GPU speed and accuracy baselines. FireCaffe reaches substantial distributed-training speedups for NiN and GoogLeNet, with accuracy preserved for GoogLeNet.

  • Evaluation methodology: FireCaffe evaluates GoogLeNet and Network-in-Network on ImageNet-1k using up to 128 GPUs and reduction-tree data parallelism.The evaluation uses a publicly available dataset and explicitly examines methodology before results.
  • Evaluation methodology: A single-GPU run provides the baseline for reporting multi-GPU training speedups and accuracy.Multi-GPU models are trained until reaching the single-GPU accuracy baseline.
  • Midsized deep models: 58.9% top-1 ImageNet-1k accuracy is reached by both AlexNet after 100 epochs and NiN after 47 epochs, with each taking approximately 6 days.NiN iterations are more time-consuming than AlexNet iterations despite reaching the same reported accuracy.
  • Midsized deep models: 13x speedup reproduces single-GPU NiN accuracy in 11 hours on 32 GPUs using the reported batch size and learning-rate configuration.At 32 GPUs, communication time and computation are approximately equal for NiN training.
  • Midsized deep models: 23x speedup trains NiN in 6 hours on 32 GPUs with batch size 1024, while reducing final accuracy by 3 10 of a percentage point.With fixed epochs, the larger batch performs 4x fewer training iterations; on 128 GPUs, NiN reaches 39x speedup.
  • Ultra-deep models: 47x speedup over single-GPU GoogLeNet training is achieved on 128 GPUs while matching single-GPU accuracy.The initial batch size was insufficient to keep the GPU cluster saturated, motivating larger-batch experiments and learning-rate tuning.

8. Complementary approaches to accelerate DNN training

The paper surveys complementary strategies for accelerating distributed DNN training, including faster convolution, reduced communication, and post-training model compression. These methods involve tradeoffs, particularly between communication efficiency and accuracy.

  • Complementary approaches: Several surveyed acceleration techniques could be combined with FireCaffe to further accelerate DNN training.The section introduces complementary approaches rather than evaluating them as part of FireCaffe.
  • Convolution acceleration: More than 90% of floating-point operations in the discussed DNNs reside in convolution layers, making convolution acceleration important for GPU utilization.The surveyed convolution methods target GPU computation and communication behavior.
  • Communication reduction: Reducing communicated data can improve distributed-training speed and scalability, but aggressive gradient quantization may reduce accuracy versus a non-quantized baseline.FireCaffe uses 32-bit floating-point weight gradients, while the section discusses lower-precision communication as related work.
  • Post-training compression: PCA-based weight compression can reach up to 5x compression, albeit with a substantial reduction in classification accuracy.The cited studies apply dimensionality reduction after training rather than during distributed gradient communication.

9. Conclusions

The paper frames long DNN training times as a limitation on research and productization and presents FireCaffe’s three-pillar approach for scalable training. The system achieves near-linear speedups on leading architectures while maintaining high accuracy in the reported results.

  • Conclusions: Long DNN training times limit research progress and productization, while faster training supports larger datasets, quicker product development, and real-time applications.The paper cites robot self-learning as one real-time DNN-training use case.
  • Conclusions: FireCaffe combines high-bandwidth interconnects, reduction-tree communication, and optional larger batches with tuned hyperparameters.These pillars target communication overhead while preserving the accuracy associated with smaller batches.
  • Conclusions: 39x speedup is achieved for NiN and 47x for GoogLeNet on a 128-GPU cluster, described as near-linear scaling.The reported conclusion covers multiple leading DNN architectures and preserves high accuracy in the stated scope.

Appendix: Frequently Asked Questions about FireCaffe

The appendix answers practical questions about FireCaffe compatibility, architecture selection, deployment, data handling, determinism, and the system’s name. It emphasizes parameter-efficient architectures, compatible cluster layouts, shared data storage, and reproducible execution.

  • Compatibility: FireCaffe is compatible with Caffe DNN models and network definitions.The appendix answers this compatibility question affirmatively.
  • Architecture selection: NiN has 8x fewer parameters than AlexNet while providing similar ImageNet accuracy, reducing its communication cost accordingly.The appendix recommends architectures with fewer parameters for faster FireCaffe training.
  • Architecture design: Architectures designed for FireCaffe should economize on parameters; reducing convolution and pooling strides may improve accuracy without adding parameters or harming scalability.The stated design tradeoffs concern accuracy and training scalability.
  • Deployment: FireCaffe supports single-server and multi-server GPU layouts, including multiple GPUs per server, with identical GPUs preferred to avoid backpropagation stragglers.The appendix recommends homogeneous GPU collections for speed and utilization.
  • Data handling: Training data is ingested through LMDB databases stored on a distributed filesystem accessible to all workers.The format supports data items represented by vectors with labels.
  • Determinism: FireCaffe produces repeatable numerical results for a fixed random seed and GPU count, matching ordinary Caffe exactly when Dropout is disabled.With Dropout enabled, FireCaffe remains deterministic but randomization differs slightly from Caffe.
  • Naming: The FireCaffe name references the FireBox warehouse-scale-computing roadmap and its emphasis on low-latency network hardware.The appendix connects interconnect quality to distributed-system performance.
Loading 1511.00175v2…