PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking State-of-the-Art Deep Learning Software Tools

Shaohuai Shi, Qiang Wang, Pengfei Xu, Xiaowen Chu

arXiv:1608.07249v7cs.DCcs.LG

TL;DR

Different deep learning tools perform differently across network types and hardware, making software–hardware selection difficult. The paper benchmarks five GPU-accelerated tools across CPU, GPU, and multi-GPU settings, finding that GPUs provide substantial speedups but no single tool consistently leads. These results guide users and identify optimization opportunities for developers.

  • Problem

    Performance varies by software tool, neural-network type, and hardware platform, making it difficult for users to select an appropriate software–hardware pair.

  • Method

    The study benchmarks Caffe, CNTK, MXNet, TensorFlow, and Torch on FCNs, CNNs, and RNNs across CPU and GPU platforms, including distributed multi-GPU versions.

  • Results

    No single tool consistently outperforms the others, while all tested tools achieve significant speedups over CPU counterparts on GPUs.

  • Takeaways & Limitations

    The benchmarking results can guide end users’ hardware and software selection and identify opportunities for further performance optimization.

  • Takeaways & Limitations

    The findings apply only to the software versions and experimental platforms specified because the tools are frequently upgraded.

Abstract

from arXiv · show

Deep learning has been shown as a successful machine learning method for a variety of tasks, and its popularity results in numerous open-source deep learning software tools. Training a deep network is usually a very time-consuming process. To address the computational challenge in deep learning, many tools exploit hardware features such as multi-core CPUs and many-core GPUs to shorten the training time. However, different tools exhibit different features and running performance when training different types of deep networks on different hardware platforms, which makes it difficult for end users to select an appropriate pair of software and hardware. In this paper, we aim to make a comparative study of the state-of-the-art GPU-accelerated deep learning software tools, including Caffe, CNTK, MXNet, TensorFlow, and Torch. We first benchmark the running performance of these tools with three popular types of neural networks on two CPU platforms and three GPU platforms. We then benchmark some distributed versions on multiple GPUs. Our contribution is two-fold. First, for end users of deep learning tools, our benchmarking results can serve as a guide to selecting appropriate hardware platforms and software tools. Second, for software developers of deep learning tools, our in-depth analysis points out possible future directions to further optimize the running performance.

1. Introduction

Deep learning tools use hardware acceleration, but their performance varies across network types and hardware platforms, complicating software–hardware selection. The paper benchmarks five GPU-accelerated tools across neural networks and platforms to characterize these differences.

  • Motivation: Training performance differs across tools even for the same neural network and hardware because vendors use different optimization methods.Performance also changes substantially across network types and hardware platforms.
  • Study scope: The benchmark evaluates Caffe, CNTK, MXNet, TensorFlow, and Torch on FCNs, CNNs, and RNNs across CPU and GPU platforms.The study also considers distributed multi-GPU versions.
  • Benchmark findings: Many-core CPU performance generally scales poorly, with 16 cores often only slightly faster than 4 or 8 cores.TensorFlow shows relatively better CPU scalability than the other tools.
  • Scope boundary: The software findings apply only to the versions and experimental platforms specified because the tools are upgraded frequently.The authors plan to update the benchmarking results for newer software versions.
  • Benchmark findings: Single-GPU results are network-dependent: Caffe, CNTK, and Torch lead on FCNs, MXNet excels on CNNs, and CNTK is strongest for LSTM RNNs.CNTK’s LSTM time efficiency is reported as up to 5–10 times better than other tools.

2. Background and Related Work

The paper reviews FCNs, CNNs, and RNNs alongside major open-source deep learning tools and their CPU, GPU, and multi-GPU execution strategies. It frames distributed training scalability as important because single-GPU memory limits network size.

  • Neural network models: FCNs are feed-forward networks, whereas CNNs use shared convolutional kernels to reduce parameters and RNNs connect outputs to historical input sequences.LSTM is introduced for handling vanishing and exploding gradients in RNN training.
  • Software tools: Caffe, CNTK, MXNet, TensorFlow, and Torch are presented as active open-source GPU-accelerated deep learning tools.The background describes their differing framework designs and intended capabilities.
  • Software tools: CNTK’s reported multi-GPU benchmark on a four-layer fully connected network achieved about 1.5x speedup over the compared toolkits.This was an official result reported in December 2015.
  • Distributed training: Multi-GPU training partitions mini-batches across workers, synchronizes gradients, and updates a shared model using tool-specific communication mechanisms.Caffe uses tree reduction, CNTK uses MPI, MXNet sums gradients before updating, TensorFlow synchronously updates a CPU-side replica, and Torch aggregates gradients on the GPU.
  • Distributed training: Multiple-GPU support matters because a single GPU has limited memory that constrains the size of trainable networks.The study evaluates both synchronization time and convergence rate for data parallelization.

3. Experimental Methods

The experiments evaluate running time and convergence across neural-network types, software tools, and CPU/GPU configurations. They combine synthetic and real-world data, single- and multi-GPU settings, and tool-specific timing procedures.

  • Evaluation Metrics: The evaluation measures mini-batch iteration time for running performance and convergence rates for multi-GPU data parallelization.Average running speed is computed across numerous iterations and convergence rates are compared for multiple GPUs.
  • Evaluation Metrics: Synthetic data evaluates running time, while real-world data measures convergence rates.The experiments use separate data purposes to assess computational speed and training progress.
  • Measurement Procedure: Timing is obtained through tool-specific commands or timing functions, with CNTK excluding the first epoch because it includes disk I/O.Caffe, MXNet, TensorFlow, and Torch use iteration or epoch timing procedures, while CNTK omits the first epoch.
  • Measurement Procedure: Reported performance is based on the authors’ usage of flexible APIs and configuration options and may not represent each tool’s best achievable performance.Different APIs can perform the same operations, and optimization settings can trade efficiency for lower memory requirements.
  • Network Configurations: The benchmark covers FCNs, CNNs, and RNNs, including a 55-million-parameter FCN-S, AlexNet, ResNet-50, FCN-R, AlexNet-R, ResNet-56, and two LSTM layers with input length 32.Synthetic experiments use FCN-S, AlexNet, and ResNet-50; real-world experiments use MNIST and Cifar10 configurations, while RNN tests use two LSTM layers.
  • Hardware Configurations: The hardware benchmark uses quad-core and dual 8-core CPUs plus GTX 980, GTX 1080, and Tesla K80 GPUs.CPU tests vary thread counts, while GPU tests cover Maxwell, Pascal, and Kepler architectures.
  • Hardware Configurations: Multi-GPU experiments use two Tesla K80 cards, providing four GK210 GPUs for data parallelization tests.One K80 card supports two-GPU experiments, and two cards support four-GPU experiments.

4. Experimental Results

Performance varies substantially across tools, neural-network types, hardware platforms, and parallelization settings. CPU scaling is generally limited, while GPU and multi-GPU results show tool-specific strengths and convergence behavior.

  • CPU results: For CPU FCNs, Torch, CNTK, and Caffe generally outperform MXNet and TensorFlow at lower thread counts, while TensorFlow performs best with 32 threads.This pattern appears on the dual E5-2630 server, whereas several tools cannot scale to 8 threads on i7-3820 because of memory limitations.
  • CPU results: For CPU CNNs and ResNet, Caffe or Torch lead at smaller thread counts, while TensorFlow becomes competitive or best with more threads; TensorFlow lacks a ResNet result because of unsupported pooling.The leading tool therefore depends on network, CPU platform, and thread count.
  • CPU results: 5-10 times: CNTK outperforms Torch and TensorFlow on CPU LSTM, while MXNet does not support LSTM on the CPU version.TensorFlow is slightly better than Torch in these CPU tests.
  • Single GPU results: On single GPUs, Caffe, CNTK, and Torch lead FCNs, MXNet leads AlexNet-S and ResNet-50, and Torch or MXNet lead other CNN configurations.Memory limits qualify some results: MXNet runs out of memory on GTX980 for FCN-S, while MXNet and other tools can fail at larger mini-batches.
  • Single GPU results: For GPU LSTM, CNTK is fastest, taking about 65 milliseconds at mini-batch size 1024 on K80 versus about 1000, 767, and 5000 milliseconds for MXNet, TensorFlow, and Torch.This is the strongest reported single-GPU time difference among the tested networks.
  • Multi-GPU results: With multiple GPUs, throughput generally rises and convergence accelerates, but scaling differs: CNTK and MXNet lead FCN scaling, while TensorFlow may gain little beyond two GPUs.For AlexNet-R, all tools except TensorFlow achieve up to 40% speedup when GPUs are doubled; TensorFlow improves by about 30 percent.
  • Multi-GPU results: In multi-GPU comparisons, MXNet and Torch generally converge fastest, while tool efficiency depends on scale: MXNet leads with one GPU and Torch tends to lead with multiple GPUs.Caffe is generally slower to converge, although its convergence rate improves relative to TensorFlow as GPUs increase.

5. Discussion

The discussion links performance differences to hardware utilization, network type, and implementation choices. It also examines CPU scaling, GPU acceleration, and multi-GPU gradient aggregation.

  • Single-GPU performance: Caffe, CNTK, and Torch perform slightly better than MXNet and TensorFlow on FCNs with one GPU.The discussion attributes this comparison partly to whether implementations use cuBLAS transpose operations, which can be substantially slower.
  • Reported comparisons: Figures 15–18 report performance comparisons for LSTM, FCN-R, AlexNet-R, and ResNet-56 across GPU configurations.The multi-GPU figures use mini-batch sizes of 4096 for FCN-R, 1024 for AlexNet-R, and 128 for ResNet-56.
  • CNN performance: All tools use cuDNN for CNN convolutions, but parameter choices can select different GPU kernels; FFT often offers faster convolution through inner products.The discussion identifies FFT-based convolution as a potential optimization in many cases.
  • RNN performance: CNTK performs much better than TensorFlow and Torch on LSTM training, while TensorFlow and Torch achieve similar performance.The tools launch different sets of kernels and use different computational structures for LSTM training.
  • Multi-GPU performance: Multi-GPU scaling is strongly influenced by gradient aggregation and PCIe data transfer; MXNet reduces this overhead by aggregating gradients on the GPU.TensorFlow and Torch aggregate gradients and update the model on the CPU, with Torch using parallel CPU updates and scaling slightly better than TensorFlow.
  • GPU performance: GPU computing provides all tested tools with significant speedups over their CPU-only versions.The tested GPUs perform matrix multiplication and FFT substantially faster than CPUs.

6. Conclusion and Future Work

The conclusion reports broad GPU speedups but finds no universally best software tool. Future work expands the benchmark's software, hardware, and distributed-system coverage.

  • Conclusion: All tested tools achieve significant speedup over CPU counterparts by making effective use of GPUs.The conclusion presents GPU speedup as a general result across the evaluated tools.
  • Conclusion: No single software tool consistently outperforms the others, leaving opportunities for further performance optimization.The result supports tool- and workload-specific performance comparisons rather than one universal winner.
  • Future work: Future work will add more tools and hardware platforms, including Paddle, AMD GPUs, and Intel Xeon Phi.The planned additions broaden the benchmark beyond its current software and hardware set.
  • Future work: Future work will evaluate tool scalability on a high-performance GPU cluster.This extends the distributed evaluation beyond the reported platforms.

7. Appendix

The appendix records benchmark revisions, configuration changes, bug fixes, added experiments, and version updates made across successive releases.

  • Version 6: Version 6 added multi-GPU results, included MXNet, updated tools to latest major versions, and added MNIST and Cifar10 results.These changes expanded both the evaluated software and the experimental datasets.
  • Version 5: Version 5 revised a TensorFlow AlexNet configuration bug.The appendix lists this as a configuration correction.
  • Version 3: Version 3 corrected the CUDA version in Table 4 and re-tested the experiments.It also revised network configurations and removed extra AlexNet operations such as dropout.
  • Version 3: Version 3 modified the CNTK RNN setup by removing an extra LSTM classification task and using customized brain scripts.The change aligned the CNTK configuration more closely with the other tools.
Loading 1608.07249v7…