Source-linked AI summary
cuDNN: Efficient Primitives for Deep Learning
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, Evan Shelhamer
TL;DR
Deep-learning kernels are expensive to optimize and must be reworked as parallel architectures evolve, yet no BLAS-like library broadly addresses these primitives. cuDNN provides optimized, framework-integrable routines with low auxiliary-memory use, improving Caffe training time by 36% on a standard model while reducing memory consumption.
Problem
Deep-learning frameworks lack an analogous shared library for computational kernels, making optimization difficult and repetitive as processor architectures evolve.
Method
cuDNN provides maintained, optimized deep-learning primitives through an easy-to-integrate library designed for performance across architectures and minimal auxiliary memory.
Results
36% improvement in training time for 200 iterations was achieved after integrating cuDNN into Caffe for the bvlc reference caffenet model on an NVIDIA Tesla K40.
Takeaways & Limitations
cuDNN lets frameworks use optimized deep-learning routines without manually writing parallel code, while supporting performance portability across evolving GPU architectures.
Takeaways & Limitations
cuDNN convolution routines remain below matrix-multiplication performance, and planned extensions include additional convolution dimensions and multi-GPU training.
Abstract
from arXiv · showhide
We present a library of efficient implementations of deep learning primitives. Deep learning workloads are computationally intensive, and optimizing their kernels is difficult and time-consuming. As parallel architectures evolve, kernels must be reoptimized, which makes maintaining codebases difficult over time. Similar issues have long been addressed in the HPC community by libraries such as the Basic Linear Algebra Subroutines (BLAS). However, there is no analogous library for deep learning. Without such a library, researchers implementing deep learning workloads on parallel processors must create and optimize their own implementations of the main computational kernels, and this work must be repeated as new parallel processors emerge. To address this problem, we have created a library similar in intent to BLAS, with optimized routines for deep learning workloads. Our implementation contains routines for GPUs, although similarly to the BLAS library, these routines could be implemented for other platforms. The library is easy to integrate into existing frameworks, and provides optimized performance and memory usage. For example, integrating cuDNN into Caffe, a popular framework for convolutional networks, improves performance by 36% on a standard model while also reducing memory consumption.
1 Introduction
cuDNN addresses the difficulty of repeatedly optimizing deep-learning kernels by providing a maintained library of efficient routines that integrates with existing frameworks. Its design targets performance, memory efficiency, and portability across processor architectures.
- CNN kernels dominate training computation and require substantial architecture-specific optimization to achieve acceptable performance.These optimizations include scheduling data movement, placing data in on-chip memory, and register blocking.
- A shared optimized-routine library lets frameworks focus on higher-level concerns while library providers handle hardware-specific optimization.The intended benefit is performance portability as parallel architectures evolve.
- cuDNN provides a flexible C-language API that integrates into existing frameworks and is tested and maintained across processor architectures.The library also targets small mini-batch sizes and varied use cases.
- cuDNN minimizes auxiliary memory usage, freeing memory for larger models and datasets.
2 Library
The cuDNN library exposes low-level, framework-agnostic primitives through a flexible GPU-oriented API. Its library includes convolution and tensor operations designed to accommodate varied layouts, strides, and common neural-network computations.
- cuDNN uses lower-level computational primitives rather than a layer abstraction, simplifying integration with frameworks that have their own abstractions.The API performs operations on data in user-controlled buffers.
- The API supports forward and backward routines, single and double precision, convolution, pooling, activations, variable layouts and strides, and tensor transformations.It also supports indexing subsections of input images and manipulating 4D tensors.
- cuDNN requires input and output data to reside on the GPU while exposing a host-callable, thread-safe C-language API.Convolution routines use descriptors for layer attributes, tensors, filters, and arbitrary tensor strides.
- 2.1.1 Spatial Convolutions: Batched convolution maps input data and filters to a four-dimensional output tensor whose spatial dimensions depend on filter sizes, padding, and stride.The convolution inputs are minibatch data and filters, while stride can reduce the number of computed output pixels.
- 2.1.1 Spatial Convolutions: The forward convolution evaluates output elements over minibatch, feature-map, and spatial indices using zero-extended input data and an accessing function.Its computation forms a seven-way nested loop with four independent loops and three accumulation loops.
- 2.2 Other functions: Beyond convolution, cuDNN provides activation functions, numerically stable softmax, pooling, and tensor transformations to reduce framework-specific parallel code.These routines can support programs that train standard convolutional networks using cuDNN and cuBLAS.
3 Implementation
cuDNN implements convolution by combining matrix-multiplication efficiency with lazy on-chip lowering, avoiding auxiliary memory while maintaining robust performance across convolution configurations and GPU architectures.
- Implementation choices: FFT convolution was rejected because it requires substantial temporary memory and performs poorly for strided convolutions common in modern networks.Striding reduces work by uv, but pruned FFTs can be slower than dense FFT followed by subsampling.
- Implementation choices: Direct convolution was rejected because its many specialized implementations are difficult to maintain and can perform poorly outside particular parameter regimes.The cited example reports cuda-convnet2 performing poorly when batch size falls to 64 or below.
- Our approach: cuDNN avoids materializing the lowered data matrix in off-chip memory by lazily generating it in on-chip memory during tiled computation.The approach dynamically maps matrix-multiplication tiles to convolution elements as computation proceeds.
- Our approach: cuDNN convolution routines provide competitive performance with zero auxiliary memory required.This design targets performance close to matrix multiplication without consuming scarce GPU memory for auxiliary structures.
- Performance: 0.8× to 2.25× cuDNN performance relative to cuda-convnet2, with an advantage at smaller batch sizes, while performance versus Caffe ranges from 1.0× to 1.41×.With mini-batch size 16, cuDNN reaches 86% of maximum cited performance.
- Performance portability: 23-35% of peak on Tesla K40 and 30-51% of peak on GTX 980 demonstrate performance across Kepler and Maxwell GPU architectures.The paper presents this as performance portability without requiring users to retune code as architectures evolve.
4 Caffe Integration
cuDNN integrates into Caffe through its existing layer and memory interfaces with an almost purely additive patch, improving training performance while preserving the framework’s model and execution interfaces.
- Impact: cuDNN integration raises Caffe’s speed and memory efficiency without sacrificing its expression or modularity.The integration is presented within Caffe’s layer-based architecture and unified memory interface.
- Integration: The Caffe integration leaves the core framework unaltered and confines changes to layer definitions, descriptor helpers, implementations, and tests.The patch is described as almost purely additive.
- Integration: cuDNN layers follow Caffe’s setup, forward, and backward protocol, with handles and descriptors configured during setup.Forward and backward library calls occur in the corresponding layer methods.
- Convolution layer: Caffe+cuDNN convolution parallelizes group-convolution forward computation and backward gradients for bias, filter weights, and bottom data.The convolution layer also exploits cuDNN’s reduced memory consumption to speed execution.
- Performance: 36% improvement in training time for 200 iterations was measured on the bvlc reference caffenet model using cuDNN R1 on an NVIDIA Tesla K40.Table 4 reports the performance improvement from integrating cuDNN into Caffe.
- Deployment: A single compilation flag enables cuDNN layer implementations as Caffe’s default engine, while unsupported cases automatically fall back to standard Caffe functionality.The model schema and framework interfaces remain unchanged.
5 Baidu Integration
cuDNN was integrated into Baidu projects and improved convolutional-layer performance while supporting broader domains, lower memory use, and straightforward integration.
- cuDNN reduced memory consumption compared with matrix multiplication, enabling larger models and mini-batches.
- Asymmetric padding for non-square inputs made cuDNN useful in speech and language applications beyond image processing.
- cuDNN was simple to integrate because its flexible interface avoided changes to existing data-structure layouts.
6 Future Work
The authors identify remaining performance and functionality gaps and propose expanding cuDNN’s primitive coverage and multi-GPU support.
- Convolution routines remain competitive but still trail matrix multiplication, motivating further work to shrink the performance gap.
- Planned extensions include 1D and 3D convolutions for speech, language processing, and video applications.
- The authors also propose adding Local Receptive Field computations, which resemble convolutions but use untied weights.
- Another proposed direction is helping users accelerate training with multiple GPUs.
7 Conclusion
The paper presents cuDNN as a library of deep learning primitives, including convolution implementations and routines for training and evaluating complete networks without manually writing parallel code.
- cuDNN provides reliable convolution performance across a wide range of input sizes without requiring auxiliary memory.
- Its convolution implementation uses highly optimized matrix multiplication routines to provide high performance.
- cuDNN includes routines for training and evaluating complete deep neural networks without manually writing parallel code.
- The conclusion argues that such libraries will become increasingly valuable as parallel architectures continue to evolve.