Source-linked AI summary
Comparative Study of Deep Learning Software Frameworks
Soheil Bahrampour, Naveen Ramakrishnan, Lukas Schott, Mohak Shah
TL;DR
Deep learning’s expanding use has produced multiple frameworks whose relative strengths are not uniform across architectures, hardware, and tasks. This paper compares five frameworks on extensibility, hardware utilization, and speed using single-machine CPU and GPU experiments. Theano and Torch are generally the most extensible, while performance leaders vary by architecture, hardware, and deployment or training setting.
Problem
Several deep learning frameworks facilitate implementation, but their comparative extensibility, hardware utilization, and speed across architectures and settings require evaluation.
Method
The paper benchmarks Caffe, Neon, TensorFlow, Theano, and Torch across architectures on a single machine using CPU and Nvidia Titan X GPU settings, measuring gradient computation and forward times.
Results
Theano and Torch are most extensible; Torch leads CPU performance and many GPU convolutional and fully connected cases, while Theano leads GPU LSTM performance and TensorFlow is less competitive on a single GPU.
Takeaways & Limitations
Framework suitability depends on architecture, hardware, and whether training or deployment is prioritized, with Torch and Theano often strongest across the tested settings.
Takeaways & Limitations
The conclusions are bounded by single-machine experiments and framework limitations, including unavailable AlexNet configurations caused by Theano’s padding support.
Abstract
from arXiv · showhide
Deep learning methods have resulted in significant performance improvements in several application domains and as such several software frameworks have been developed to facilitate their implementation. This paper presents a comparative study of five deep learning frameworks, namely Caffe, Neon, TensorFlow, Theano, and Torch, on three aspects: extensibility, hardware utilization, and speed. The study is performed on several types of deep learning architectures and we evaluate the performance of the above frameworks when employed on a single machine for both (multi-threaded) CPU and GPU (Nvidia Titan X) settings. The speed performance metrics used here include the gradient computation time, which is important during the training phase of deep networks, and the forward time, which is important from the deployment perspective of trained networks. For convolutional networks, we also report how each of these frameworks support various convolutional algorithms and their corresponding performance. From our experiments, we observe that Theano and Torch are the most easily extensible frameworks. We observe that Torch is best suited for any deep architecture on CPU, followed by Theano. It also achieves the best performance on the GPU for large convolutional and fully connected networks, followed closely by Neon. Theano achieves the best performance on GPU for training and deployment of LSTM networks. Caffe is the easiest for evaluating the performance of standard deep architectures. Finally, TensorFlow is a very flexible framework, similar to Theano, but its performance is currently not competitive compared to the other studied frameworks.
1. INTRODUCTION
The paper motivates comparing deep learning frameworks because growing use of diverse architectures has produced competing tools with different strengths. It evaluates extensibility, hardware utilization, and speed to clarify framework suitability and limitations.
- Deep learning frameworks support efficient development and implementation as deep learning methods gain popularity across application domains.
- Extensibility covers support for diverse architectures, training procedures, and convolutional algorithms.
- Hardware utilization measures how effectively frameworks use multi-threaded CPU or GPU resources.
- Speed is evaluated from both training and deployment perspectives.
- The study aims to help users and enterprises assess framework suitability by presenting strengths and current limitations.It also discusses limitations relevant to future framework development.
2. OVERVIEW OF THE DEEP LEARNING FRAMEWORKS
The overview places the studied frameworks within a broader ecosystem developed to create and test deep architectures. At the time of writing, widely used options included Caffe, Theano, Torch, Neon, TensorFlow, and several others.
- Deep learning’s growing application popularity stimulated academic and industry interest in frameworks for creating and testing diverse architectures.
- The paper identifies Caffe, Theano, Torch, Neon, and TensorFlow among the widely used deep learning frameworks at the time.
3. BENCHMARKING SETUP
The benchmark measures training and deployment speed across frameworks using gradient computation and forward times, while documenting hardware, software, and implementation conditions. Experiments run on one CPU/GPU machine and account for framework-specific compilation, library, and threading choices.
- Evaluation Metrics: The study uses forward time to measure deployment latency and gradient computation time to indicate training time.Gradient computation excludes parameter-update time.
- Scope Caveat: Framework activity can make the study’s observations obsolete as communities add features and functionality.
- Implementation Choices: Theano’s forward and gradient functions require compilation before execution, and those compilation times are reported separately.
- Experimental Environment: The experiments run on a single machine with an Intel Xeon CPU, Nvidia Titan X GPU, 32 GiB memory, and specified framework versions.
4. RESULTS AND DISCUSSIONS
The study benchmarks five frameworks across datasets, architectures, hardware settings, and convolution implementations, showing strong hardware- and workload-dependent differences in speed. GPU acceleration benefits convolutional networks, while framework rankings vary with batch size, architecture, and supported operations.
- Experimental scope: The evaluations cover stacked autoencoders and convolutional networks on MNIST and ImageNet, plus LSTM training on IMDB reviews.
- Experimental scope: The benchmark reports averaged gradient-computation and forward-pass times across CPU and GPU settings, with batch size 64 for the LeNet comparison.
- LeNet: Theano has the best GPU gradient-computation time for small LeNet batches, whereas Neon performs best for large batches.
- LeNet: Theano consistently achieves the minimum LeNet forward time, especially for large batches, while TensorFlow performs worst, particularly for small batches.
- AlexNet: GPU use provides a more significant speed-up for AlexNet than for LeNet, reaching at least 25× in the reported comparison.
- AlexNet: For AlexNet without LRN and grouping, Torch with fbcunn has the best gradient-computation performance, while Neon has the best forward-pass performance followed closely by Torch.
5. CONCLUSIONS
Across the five evaluated frameworks, Theano and Torch are most extensible, while performance leadership varies by architecture, hardware, and workload. Torch leads CPU performance and several GPU settings, Theano leads LSTM GPU performance, and TensorFlow remains flexible but less competitive on a single GPU.
- Theano and Torch are the most extensible frameworks for supporting diverse deep architectures and libraries.Theano’s symbolic differentiation is particularly useful for implementing non-standard deep architectures.
- Torch performs best for CPU-based training and deployment across the tested deep network architectures, followed by Theano.Neon has the worst performance in this CPU comparison.
- Torch is best for GPU deployment of trained convolutional and fully connected networks, followed by Theano.
- For GPU training of convolutional and fully connected networks, Theano is fastest on small networks, whereas Torch is fastest on larger networks; Neon is competitive for large convolutional networks.
- Theano delivers the best GPU performance for training and deployment of recurrent LSTM networks.
- TensorFlow is highly flexible for assigning graph components across homogeneous and heterogeneous devices, but its single-GPU performance is less competitive.
7. APPENDIX
The appendix describes the implementation characteristics of the five frameworks and reports experiment and compilation measurements for several deep-learning workloads. It covers framework backends, supported interfaces, network construction, and timing tables for autoencoders and Theano compilation.
- 7.1 Caffe: Caffe separates network architecture definitions from implementation and supports CPU or GPU exploration of layers and architectures.It provides command-line, Python, and Matlab interfaces and uses CUDA for GPU computation.
- 7.1 Caffe: Caffe supports configured directed acyclic graphs with implemented layers and loss functions, including convolutional, fully connected, and pooling layers.
- 7.2 TensorFlow: TensorFlow uses data-flow graphs whose nodes represent mathematical operations and whose edges carry multidimensional arrays.Its architecture supports CPU and GPU backends on desktop, server, or mobile platforms.
- 7.3 Theano: Theano is a Python symbolic manipulation library designed to improve execution and development time for machine-learning algorithms using repeated tensor computations.
- 7.3 Theano: Theano uses CUDA arrays and customized C and CUDA code generators tailored to input types, sizes, and shapes.
- 7.4 Torch: Torch provides strong CUDA and CPU backends, mature machine-learning and optimization packages, and neural-network libraries for arbitrary acyclic graphs with automatic differentiation.
- Experimental tables: Table 7 reports averaged stacked-autoencoder processing times for pre-training and fine-tuning, including gradient-computation and forward-pass measurements.
- 7.5 Neon: Neon provides customized CPU and GPU backends, with a NervanaGPU backend optimized for Nvidia Maxwell GPUs such as Titan X.