PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

Zhihao Jia, Sina Lin, Charles R. Qi, Alex Aiken

arXiv:1802.04924v2cs.LGcs.DCcs.NE

TL;DR

CNN training is increasingly compute-intensive, while a uniform data- or model-parallel strategy can underperform because different layers may prefer different strategies. Layer-wise parallelism jointly searches per-layer configurations; evaluation reports higher throughput, lower communication costs, and better scalability while maintaining network accuracy.

  • Problem

    CNN training has growing computational requirements, and applying one parallelization strategy to every layer can underperform because layer preferences differ.

    Source: §1 · p. 1

  • Method

    Layer-wise parallelism assigns individual configurations to layers and uses a cost model with graph search to jointly select a runtime-efficient strategy.

    Source: §1 · pp. 1–2

  • Results

    On 16 P100 GPUs across 4 nodes, layer-wise parallelism is 1.4-2.2× faster than state-of-the-art strategies, reduces communication costs by 1.3-23.0× versus data and model parallelism, and scales better.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Because its configurations perform the original network computation, layer-wise parallelism maintains original network accuracy while improving runtime performance.

    Source: §1 · p. 1, §7 · p. 8

    The formulation assumes devices process a layer independently by computing disjoint subsets of its output tensor.

    Source: §4 · p. 3

Abstract

from arXiv · show

The past few years have witnessed growth in the computational requirements for training deep convolutional neural networks. Current approaches parallelize training onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, these approaches result in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an individual parallelization strategy. We jointly optimize how each layer is parallelized by solving a graph search problem. Our evaluation shows that layer-wise parallelism outperforms state-of-the-art approaches by increasing training throughput, reducing communication costs, achieving better scalability to multiple GPUs, while maintaining original network accuracy.

1. Introduction

CNN training is increasingly compute-intensive, and applying one parallelization strategy across every layer can underperform because different layers may prefer different strategies. The paper introduces layer-wise parallelism and reports better throughput, communication costs, and scalability than state-of-the-art approaches.

1. Introduction

  • A single data- or model-parallel strategy can underperform because densely connected layers may prefer model parallelism while convolutional layers typically prefer data parallelism.Some layers may also benefit from parallelizing across multiple data dimensions.Source: §1 · p. 1
  • Layer-wise parallelism gives each layer an individual strategy; a cost model and graph-search algorithm jointly select configurations for runtime performance while preserving original network accuracy.The search space includes data and model parallelism as special cases.Source: §1 · pp. 1–2
  • On 16 P100 GPUs across 4 nodes, layer-wise parallelism is 1.4-2.2× faster than state-of-the-art strategies and reduces communication costs by 1.3-23.0× compared to data and model parallelism.The evaluation also reports better scalability than other parallelization strategies.Source: §1 · p. 2
  • For Inception-v3 scaled from 1 GPU to 16 GPUs, layer-wise parallelism obtains 15.5× speedup, while other strategies achieve at most 11.2×.Source: §1 · p. 2

2. Related Work

Related work spans data and model parallelism, a hybrid strategy for convolutional and fully connected layers, system-level communication optimizations, and network parameter reduction. The paper distinguishes its approach by seeking faster distributed training while preserving the original network accuracy.

2. Related Work

  • Data parallelism replicates the network and can bottleneck on parameter-heavy layers, whereas model parallelism reduces parameter-synchronization costs but exposes limited parallelism.Source: §2 · p. 2
  • OWT uses data parallelism for convolutional and pooling layers and model parallelism for fully connected layers, but layer-wise parallelism further reduces communication costs and improves performance over OWT.OWT is used as an experimental baseline.Source: §2 · p. 2
  • System-level methods optimize communication or overlap synchronization with backpropagation, but the cited systems remain based on data parallelism and limited by communication costs.Source: §2 · p. 2
  • Pruning and redundant-parameter elimination can improve runtime by reducing network parameters, but modify the network and may decrease accuracy; this paper instead maintains the original network accuracy.Source: §2 · p. 2

3. Hidden Dimensions in Parallelizing a Layer

Parallelization can use dimensions beyond the sample dimension, and layers can differ in which dimensions and degrees of parallelism suit them. The examples show that these choices can affect execution time and communication costs.

3. Hidden Dimensions in Parallelizing a Layer

  • CNN tensors include sample, height, width, and channel dimensions, and combinations of dimensions and their parallelism degrees can be considered for a layer.Table 1 lists parallelizable dimensions for different layers and defines length as a position in a 1D image.Source: §3 · p. 2, Table 1 · p. 3
  • For a 2D convolutional layer on 4 GPUs, data parallelism is suboptimal compared with parallelization across other dimensions.Figure 1 compares execution time for Conv8 in VGG-16; parallelizing in other dimensions preserves the same output, though GPUs may share input data.Source: §3 · p. 2, Figure 1 · p. 3
  • For the fully connected layer example, channel-dimension parallelism reduces communication costs by 12× compared with data parallelism.It removes parameter synchronization but adds input-tensor transfers.Source: §3 · p. 3, Figure 2 · p. 3
  • In Inception-v3, the convolutional layer performs best on 16 GPUs, while the fully connected layer performs best on 4 GPUs.Figure 3 compares computation and communication time across degrees of data parallelism.Source: §3 · p. 3, Figure 3 · p. 3

4. Problem Definition

The problem maps a neural-network computation graph onto a device graph and describes each layer’s parallelization through partitions of its output tensor. It seeks layer configurations that minimize per-iteration training time while preserving the original network’s computation.

4. Problem Definition

  • A device graph represents hardware devices and their connections, while a computation graph represents network layers and the tensors passed between them.Source: §4 · p. 3
  • Parallelizing a layer assigns devices disjoint subsets of its output tensor, using divisible output dimensions as parallelization axes.Source: §4 · p. 3
  • Configurations specify the degree of parallelism along each chosen dimension and assume equal partitions to balance device workloads.Figure 4 illustrates single-dimension and multi-dimension configurations for a 2D convolutional layer.Source: §4 · p. 3, Figure 4 · p. 4
  • Every configuration produces the same output, preserving the original network’s accuracy.Source: §4 · p. 4
  • The objective is to choose a layer-wise strategy S that minimizes per-iteration execution time t(G, D, S).Each strategy assigns a configuration to every layer.Source: §4 · p. 4

5. Method

The method uses a cost model and dynamic-programming-based graph search to optimize layer parallelization, under assumptions about computation predictability and communication bandwidth.

5. Method

  • The method combines a cost model with dynamic-programming-based graph search to find an optimal parallelization strategy under that model.Source: §5.1 · p. 4
  • Layer processing time is assumed to be predictable with low variance and largely independent of input contents.Source: §5.1 · p. 4
  • Tensor transfers are assumed to take size divided by link bandwidth, with communication bandwidth fully utilized.Source: §5.1 · p. 4

3. The runtime system has negligible overhead. A device

The method estimates computation, transfer, and synchronization costs, then reduces the computation graph to search for an optimal strategy. The reductions preserve optimality and make the search practical on the CNNs discussed.

3. The runtime system has negligible overhead. A device

  • Experiments indicate that dense-matrix-based CNN layers satisfy the processing-time assumption, and the implementation meets the bandwidth and overhead assumptions well enough for significant runtime gains.Source: §3 · p. 4
  • The model estimates layer processing, tensor-transfer, and parameter-synchronization costs; synchronization approximates communication time for moving gradients and updated parameters.Per-step execution time includes forward processing, back propagation, and parameter synchronization.Source: §3 · p. 4

5.2. Graph Search

  • Layer-specific strategies create exponentially many candidates, so the search uses CNN locality to motivate graph reductions that preserve optimal strategies.Source: §5.2 · p. 5
  • Node elimination replaces a single-in/single-out layer and its edges with a cost-defined bypass edge; edge elimination merges duplicate edges by summing transfer costs.Figure 5 depicts the node and edge elimination operations.Source: §5.2 · p. 5, Figure 5 · p. 5
  • After enumerating strategies on the reduced graph, the algorithm reverses the eliminations to recover an optimal strategy for the original graph.Theorems 1 and 2 establish that the reverse construction preserves optimality; Figure 6 illustrates reductions on an Inception module.Source: §5.2 · pp. 5–6, Figure 6 · p. 6
  • The algorithm runs efficiently on the listed CNNs, reducing AlexNet, VGG, Inception-v3, and ResNet to a final graph with two nodes.The complexity analysis gives O(EC3) for performing reductions and O(EC) for undoing them; Table 2 describes the analyzed quantities.Source: §5.2 · p. 6, Table 2 · p. 6
  • Algorithm 1 reduces execution time by orders of magnitude versus depth-first search on the original graph.Source: §5.2 · p. 6

6. Experiments

Experiments on AlexNet, VGG-16, and Inception-v3 compare layer-wise parallelism with alternative strategies, including throughput, communication, scalability, and cost-model accuracy. The analysis also describes how optimal layer configurations vary across a CNN.

6. Experiments

  • The experiments use AlexNet, VGG-16, and Inception-v3 on a four-node cluster of P100 GPUs, with data-parallel results checked against TensorFlow and PyTorch.The framework’s data-parallel performance is reported because it matched or exceeded TensorFlow and PyTorch on all three CNNs.Source: §6 · pp. 6–7

6.1. Runtime Performance

  • Layer-wise parallelism achieves speedups of 12.2×, 14.8×, and 15.5× for AlexNet, VGG-16, and Inception-v3, respectively, compared with 6.1×, 10.2×, and 11.2× from the best other strategy.Source: §6.1 · p. 7

6.2. Cost Model

  • The cost model’s estimated and measured per-step execution times differ by less than 10% in all experiments.Table 4 reports the relative differences between estimated and actual execution time.Source: §6.2 · p. 7, Table 4 · p. 7

6.1. Runtime Performance

  • Layer-wise parallelism increases throughput by up to 2.2×, 1.5×, and 1.4× for AlexNet, VGG-16, and Inception-v3, respectively.The plotted comparison covers different CNNs and device sets; higher throughput is better.Source: §6.1 · p. 7, Figure 7 · p. 8
  • Layer-wise parallelism reduces communication overhead by 1.2–2.5× versus OWT parallelism; OWT reduces it by 1.1–23.0× versus data and model parallelism.Figure 8 compares communication costs, for which lower is better.Source: §6.1 · p. 7, Figure 8 · p. 8

6.3. Analysis of Optimal Parallelization Strategies

  • Optimal configurations use data parallelism in early wide layers, reduce devices and use spatial parallelism deeper, then switch to model parallelism on fewer devices for dense layers.These choices reflect changing costs of moving tensors between layers and synchronizing gradients or parameters.Source: §6.3 · p. 8
  • For VGG-16 on four GPUs, the analyzed strategy moves from sample parallelism in early convolutional and pooling layers to height/width parallelism, then channel parallelism for fully connected layers.The strategy adaptively decreases the degrees of parallelism for fully connected layers.Source: Table 5 · p. 7, §6.3 · p. 8

7. Conclusion

The conclusion presents layer-wise parallelism as a way to configure each CNN layer individually, using a cost model and graph search to select a strategy. Experiments report better throughput, lower communication costs, and better scalability than state-of-the-art strategies.

7. Conclusion

  • Layer-wise parallelism lets each CNN layer use an individual parallelization configuration.The approach uses a cost model and dynamic-programming-based graph search to find a globally optimal strategy under that model.Source: §7 · p. 8
  • The cost model evaluates runtime performance, and dynamic programming-based graph search finds a globally optimal strategy under that model.Source: §7 · p. 8
  • Experiments report that layer-wise parallelism outperforms state-of-the-art strategies in throughput, communication costs, and scalability on larger numbers of devices.Source: §7 · p. 8

A. Node and Edge Eliminations

The appendix describes node and edge eliminations as graph simplifications used in the strategy-search proofs. The proof fragments compare costs on the original and modified graphs.

A. Node and Edge Eliminations

  • Node elimination removes a layer with one incoming and one outgoing edge, replacing those edges and the layer with a direct edge.Source: A. Node and Edge Eliminations · p. 10
  • Edge elimination replaces two edges between the same layers with one edge.Source: A. Node and Edge Eliminations · p. 10
  • For node elimination, the proof compares original-graph and modified-graph costs, accounting for the removed layer and its adjacent edges and the replacement edge.The proof states that unrelated cost functions cancel in the subtraction.Source: A. Node and Edge Eliminations · p. 10
Loading 1802.04924v2…