PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks

Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, Tim Kraska

arXiv:1801.04380v1cs.DCcs.LG

TL;DR

Limited GPU DRAM restricts the deep and wide nonlinear networks practitioners can train, while existing frameworks do not dynamically balance memory provisioning and training speed. SuperNeurons introduces a dynamic GPU memory scheduling runtime with three memory optimizations and performance mechanisms. The paper reports layer-wise minimal peak memory and leading performance, including the ability to train substantially deeper networks.

  • Problem

    Limited GPU DRAM and nonlinear dependency variation restrict training, while existing frameworks lack dynamic scheduling that jointly provisions memory and optimizes speed.

  • Method

    SuperNeurons dynamically schedules tensor placement, movement, allocation, and deallocation using Liveness Analysis, Unified Tensor Pool, and Cost-Aware Recomputation.

  • Results

    SuperNeurons reduces peak memory to max(li), the maximal layer usage, and is reported to deliver leading performance among state-of-the-art GPU deep-learning systems.

  • Takeaways & Limitations

    The runtime creates opportunities to train deeper and wider neural architectures within limited GPU DRAM resources.

  • Takeaways & Limitations

    Existing frameworks’ static memory techniques can trade memory reduction against speed, and TensorFlow’s GPU-to-CPU transfers compromise at least 50% of communication speed.

Abstract

from arXiv · show

Going deeper and wider in neural architectures improves the accuracy, while the limited GPU DRAM places an undesired restriction on the network design domain. Deep Learning (DL) practitioners either need change to less desired network architectures, or nontrivially dissect a network across multiGPUs. These distract DL practitioners from concentrating on their original machine learning tasks. We present SuperNeurons: a dynamic GPU memory scheduling runtime to enable the network training far beyond the GPU DRAM capacity. SuperNeurons features 3 memory optimizations, \textit{Liveness Analysis}, \textit{Unified Tensor Pool}, and \textit{Cost-Aware Recomputation}, all together they effectively reduce the network-wide peak memory usage down to the maximal memory usage among layers. We also address the performance issues in those memory saving techniques. Given the limited GPU DRAM, SuperNeurons not only provisions the necessary memory for the training, but also dynamically allocates the memory for convolution workspaces to achieve the high performance. Evaluations against Caffe, Torch, MXNet and TensorFlow have demonstrated that SuperNeurons trains at least 3.2432 deeper network than current ones with the leading performance. Particularly, SuperNeurons can train ResNet2500 that has $10^4$ basic network layers on a 12GB K40c.

1 Introduction

SuperNeurons addresses GPU memory limits that constrain deep and wide neural-network training by dynamically scheduling tensor memory. Its three memory optimizations reduce peak usage to the maximal layer usage while performance optimizations preserve training efficiency.

  • Motivation: Deeper and wider neural architectures can improve generalization, but limited GPU DRAM restricts the network designs practitioners can train.Practitioners may need less-desired architectures or multi-GPU partitioning.
  • Approach: SuperNeurons is a dynamic GPU memory scheduling runtime for training deep nonlinear neural networks beyond physical GPU-memory limits.It orchestrates tensor placement, movement, allocation, and deallocation transparently to users.
  • Memory optimizations: Liveness Analysis, Unified Tensor Pool, and Cost-Aware Recomputation reduce network-wide peak memory to max(li), the maximal memory usage among layers.The methods recycle tensors, offload checkpoint tensors, and recompute selected results.
  • Performance: Performance optimizations amortize frequent memory operations through a preallocated GPU memory pool and address efficiency issues in tensor management.The runtime also dynamically allocates memory for convolution workspaces under limited GPU DRAM.
  • Contribution: SuperNeurons is presented as enabling deeper and wider architectures while maintaining leading GPU training performance.The paper identifies dynamic convolution-workspace allocation as part of this performance strategy.

2 Background and Motivation

Super-deep nonlinear networks create GPU-memory and dependency-management challenges. Existing frameworks use static techniques that inadequately handle nonlinear dependencies and may trade memory savings or communication efficiency against training speed.

  • Challenges: Super-deep nonlinear architectures require substantial computation and face limited GPU resident memory plus highly variable computational dependencies.These issues differ from the predictable sequential dependencies of linear networks.
  • Memory demand: 18.5GB and 44.3 GB are required by ResNet152 and Inception v4, respectively, at batch size 32.These requirements are similar to or exceed the resident memory of commercial GPUs.
  • Dependency variation: Join and fan connections create non-sequential dependencies, with joins linking layers and fans branching execution into multiple paths.DenseNet can use nondeterministic join connectivity, while fan connections create multiple execution branches.
  • Existing techniques: Static tensor reuse works well for linear networks but requires extra tensors for future dependencies in nonlinear networks.This limits its effectiveness for nonlinear training.
  • Framework limitations: TensorFlow’s GPU-to-CPU swapping compromises at least 50% of communication speed, while frameworks generally lack dynamic scheduling that jointly provisions memory and optimizes speed.The paper also identifies convolution-workspace allocation as a decisive factor in CNN training speed.

3 Design Methodologies

SuperNeurons combines dynamic tensor scheduling with three memory optimizations and supporting performance mechanisms. The design targets layer-wise minimal peak memory while managing external-memory transfers, recomputation, and convolution workspaces.

  • Design goal: SuperNeurons provisions training memory while seeking convolution workspaces within the constraint of native GPU memory size.This combines memory reduction with performance-oriented workspace allocation.
  • Notation and baseline: The baseline independently allocates a tensor for each memory request, while l_peak is defined as max(li) across network layers.Here, li denotes maximal memory usage among layers and N denotes network length.
  • Liveness Analysis: Liveness Analysis recycles tensors that are no longer needed during back-propagation and can provide up to 50% memory savings.Its frequent large-memory operations are supported by a preallocated heap.
  • Unified Tensor Pool: Unified Tensor Pool consolidates external memory pools, overlaps data transfers with computation, and uses a GPU Tensor Cache to reduce communication.Checkpoint tensors include compute-intensive layers such as fully connected and convolution layers.
  • Cost-Aware Recomputation: Cost-Aware Recomputation reduces peakm to max(li) by tracking checkpoint memory distributions and minimizing extra computation.The method ensures peakm ≤ max(li).

3.1 Prerequisites

SuperNeurons constructs execution routes for nonlinear neural networks by recursively exploring dependencies and coordinating forward and backward steps. Tensors are the basic memory scheduling units because cuDNN operates at layer granularity.

  • Tensor representation: A typical DNN layer computes on a four-dimensional tensor indexed by batch, channel, height, and width.
  • Tensor representation: Tensors are used as the basic memory scheduling unit because cuDNN operates at layer granularity.
  • Execution-route construction: Algorithm 1 recursively explores subsequent layers with depth-first search, pausing at joins until all prerequisite layers finish.A dependency counter in each layer tracks whether its inputs have been completed.
  • Execution-route construction: The execution route records network layers as ordered forward and backward steps for nonlinear architectures.In the illustrated route, nested fan structures are handled and prerequisite layers are identified before a later computation.

3.2 Liveness Analysis and Its Related Issues

Liveness Analysis tracks tensors required before and after each layer, freeing tensors once no future dependency needs them. Its memory savings reduce peak usage, but frequent allocation and deallocation create performance overhead.

  • Liveness Analysis: Liveness Analysis enables different tensors to reuse the same physical memory across different time partitions.The runtime implements a data-flow analysis for nonlinear networks with O(N^2) construction cost.
  • Analysis formulation: The analysis tracks live tensors with in and out sets before and after each layer’s computation.The method simplifies the analysis by assuming identical layer memory usage and that forward results are needed during back-propagation.
  • Liveness Analysis: The runtime removes tensors from each layer’s out set when no subsequent layer depends on them.For example, tensors t2 and t5 can be freed after FC7 because later computations no longer require them.
  • Memory impact: 50% memory is the maximum saving reported for Liveness Analysis relative to the baseline.
  • Performance issues: O(N^2) analysis and frequent tensor operations can impose substantial runtime overhead during training.ResNet50 spends 36.28% of training time on cudaMalloc and cudaFree, motivating a heap-based GPU memory pool.

3.3 Unified Tensor Pool(UTP) and Its Related Issues

Unified Tensor Pool extends GPU memory with external physical pools through transparent tensor movement, while selective offloading, prefetching, caching, and recomputation manage capacity and performance.

  • UTP abstraction: UTP consolidates GPU and external physical memory pools, asynchronously transferring tensors to reduce GPU DRAM pressure.The paper focuses on local CPU DRAM but states that the abstraction also applies to other CPU or GPU memory pools.
  • Offloading and prefetching: CONV tensors are offloaded because POOL, ACT, BN, and LRN consume over 50% of memory while accounting for about 20% of computation.Dropout, Softmax, and FC layers are not fruitful offloading targets because each uses less than 1% of total memory.
  • Offloading and prefetching: Asynchronous offloading transfers CONV outputs to pinned CPU memory and frees GPU memory after the transfer completes.An independent background thread checks transfer events so GPU-to-CPU movement can overlap with forward computation.
  • Offloading and prefetching: Asynchronous prefetching brings soon-to-be-reused tensors back to GPU DRAM during backward computation.The transfers are scheduled for tensors needed by the previous CONV layer and can overlap with backward computation.
  • Cost-aware recomputation: Cost-Aware Recomputation selects a speed-centric or memory-centric strategy according to a segment’s memory cost.The speed-centric strategy recomputes a segment once, whereas the memory-centric strategy recomputes dependencies for each backward layer with O(N^2) additional computation.
  • Caching tensors: The runtime triggers data transfers only when GPU DRAM is insufficient and otherwise caches tensors to maximize reuse.
  • Caching tensors: The LRU cache locks dependent tensors, promotes hits to the front, and offloads unlocked least-recently-used tensors on misses.LRU.out frees enough space for a new tensor by moving eligible tensors to CPU RAM.

3.4 Cost-Aware Recomputation

Cost-Aware Recomputation combines memory-centric and speed-centric recomputation to keep peak memory at the layerwise minimum while limiting extra computation.

  • Motivation: POOL, ACT, LRN and BN use about 50% of memory but less than 10% of forward time, making them candidates for recomputation.Freeing cheap-to-compute layer tensors exposes substantial memory savings with limited performance loss.
  • Baseline Strategies: The speed-centric strategy retains recomputed tensors for reuse, whereas the memory-centric strategy frees them to maximize memory savings.The speed-centric approach incurs O(N) extra computation; the memory-centric approach recomputes dependencies for each backward layer.
  • Cost-Aware Strategy: Cost-Aware Recomputation applies the speed-centric strategy when a segment’s memory cost is ≤lpeak and the memory-centric strategy otherwise.The runtime first computes lpeak = max(li) as the threshold for selecting between the strategies.
  • Trade-off: The cost-aware method matches the memory-centric strategy’s peak memory while keeping extra recomputations comparable to the speed-centric strategy.This combines the lower memory bound of the memory-centric strategy with the lower recomputation cost of the speed-centric strategy.
  • Guarantee: Cost-Aware Recomputation guarantees costb_k ≤ lpeak, so the final network-wide peak becomes max(li), the minimal layerwise peak.The bound follows because the final peak is max(costb_k) = lpeak.

3.5 Finding the Best Convolution Algorithm under the Memory Constraint

SuperNeurons dynamically allocates convolution workspaces after reserving memory for functional tensors, adapting workspace capacity to changing memory availability while preserving performance opportunities.

  • Workspace Importance: CONV layers account for over 50% of total computing time, and some cuDNN algorithms require temporary workspaces for maximal speed.Workspace memory is therefore important to high-performance training.
  • Evaluation Context: The runtime evaluates workspace allocation as part of the performance analysis of SuperNeurons.The supplied table caption summarizes extra recomputation and peak memory for the recomputation strategies, not workspace allocation.
  • Dynamic Allocation: Dynamic Conv Workspace Allocation profiles free GPU memory after memory-saving techniques and allocates the remaining space to convolution workspaces.Functional tensor allocations are prioritized because convolution workspaces do not affect functionality.

4 Evaluations

Evaluations measure SuperNeurons’ memory techniques, performance optimizations, and end-to-end framework performance across linear and nonlinear networks. The results show substantial memory reductions, scalable model capacity, and leading training speed.

  • Memory Optimizations: 31.9% improvement in peakm reduces AlexNet memory from 2189.437MB with 36 tensors to 1489.355MB with at most 17 tensors.The peak tensor count does not necessarily coincide with the peak memory location.
  • Memory Optimizations: 48.29% improvement over the baseline’s peakm yields a new peak of 1132.155 MB after Prefetching/Offloading atop Liveness Analysis.The peak shifts to POOL2 backward because CONV1–4 are offloaded and CONV5 is prefetched.
  • Memory Optimizations: 886.385MB is the measured max(li), and the three memory techniques reduce peakm to approximately 886MB at backward LRN1.This reaches the theoretical layerwise minimum because cuDNN must retain tensors within a layer.
  • Speed Optimizations: GPU Memory Pool speedups are more significant for nonlinear networks than for linear networks because nonlinear networks perform more memory operations.The pool amortizes intensive allocation and deallocation overhead by preallocating a large GPU-memory region.
  • Speed Optimizations: 33.33% performance loss occurs without Tensor Cache, which avoids communications at AlexNet batch sizes of 256 →896.Communication overhead increases with batch size without the cache and can outweigh computation.
  • Capacity: 3.2432x deeper ResNet than TensorFlow is supported, and SuperNeurons trains ResNet2500 with approximately 10^4 basic layers on a 12GB GPU.The reported depth improvements over Caffe, Torch, and MXNet are 12.9730x, 12.6316x, and 4.0000x, respectively.
  • End-to-End Performance: SuperNeurons consistently demonstrates leading speed on AlexNet, VGG16, ResNet50 →152, and Inception V4.The reported performance is attributed largely to the convolution workspaces supplied by the dynamic GPU memory scheduler.

5 Related Work

Prior approaches address GPU memory shortage through model parallelism, data movement, parameter reduction, and static framework techniques. Their limitations motivate dynamic GPU memory scheduling for deep-network training.

  • Evaluation Context: End-to-end framework evaluation is included as part of the paper’s assessment of its runtime design.The supplied figure passage identifies TITAN XP as the benchmark platform.
  • Parallelism: Model Parallelism partitions networks across machines or GPUs but demands substantial intra-network communication for synchronization.Data Parallelism is consequently common in high-performance training systems.
  • Data Movement: vDNN uses asynchronous CPU–GPU prefetching and offloading, but performance depends on the communication/computation ratio.Cheap layers such as POOL can make PCI-E transfers comparatively costly.
  • Parameter Reduction: Pruning and quantization reduce parameter memory, but parameters occupy only a negligible portion of training memory.These approaches are therefore limited for training-memory reduction despite deployment benefits.

6 Conclusion

SuperNeurons addresses GPU memory constraints in deep-network training with a dynamic scheduling runtime and three memory techniques. Its evaluations support deeper and wider neural architectures while targeting high performance.

  • SuperNeurons reduces peak memory to max(l_i), the minimal layer-wise granularity, using three memory techniques.
  • The runtime includes performance optimizations intended to maintain high performance during memory reduction.
  • Evaluations against state-of-the-art deep-learning frameworks demonstrate the runtime’s effectiveness and efficiency.
  • The approach creates opportunities to explore deeper and wider neural architectures.
Loading 1801.04380v1…