PaperlayerView exact revision on arXiv

Source-linked AI summary

Priority-based Parameter Propagation for Distributed DNN Training

Anand Jayarajan, Jinliang Wei, Garth Gibson, Alexandra Fedorova, Gennady Pekhimenko

arXiv:1905.03960v1cs.DCcs.LG

TL;DR

Distributed DNN training is limited by communication-heavy parameter synchronization. The paper introduces P3, which slices parameters and prioritizes synchronization to overlap communication with computation, improving throughput across several models.

  • Problem

    Data-parallel DNN training can become communication-bound because workers synchronize large gradient volumes over networks whose bandwidth limits scalability.

  • Method

    P3 splits parameters into independently synchronized slices and prioritizes their transmission according to when they are needed in subsequent iterations.

  • Results

    P3 improves training throughput by as much as 25% for ResNet-50, 38% for Sockeye, and 66% for VGG-19.

  • Takeaways & Limitations

    P3 provides higher bandwidth resiliency and better scalability than the baseline MXNet implementation.

Abstract

from arXiv · show

Data parallel training is widely used for scaling distributed deep neural network (DNN) training. However, the performance benefits are often limited by the communication-heavy parameter synchronization step. In this paper, we take advantage of the domain specific knowledge of DNN training and overlap parameter synchronization with computation in order to improve the training performance. We make two key observations: (1) the optimal data representation granularity for the communication may differ from that used by the underlying DNN model implementation and (2) different parameters can afford different synchronization delays. Based on these observations, we propose a new synchronization mechanism called Priority-based Parameter Propagation (P3). P3 synchronizes parameters at a finer granularity and schedules data transmission in such a way that the training process incurs minimal communication delay. We show that P3 can improve the training throughput of ResNet-50, Sockeye and VGG-19 by as much as 25%, 38% and 66% respectively on clusters with realistic network bandwidth

1 INTRODUCTION

Distributed DNN training is communication-bound because workers synchronize large gradient volumes, motivating P3’s finer-grained, priority-aware overlap of synchronization with computation.

  • Motivation: Data-parallel synchronous SGD distributes training across workers, which independently compute updates and synchronize them after each iteration.Each iteration includes forward propagation, backward propagation, and parameter synchronization.
  • Motivation: Hundreds of megabytes of gradients per iteration can make data-parallel training communication-bound and require high-bandwidth networks.Faster accelerators and larger DNNs further increase transmission rates and volumes.
  • Our Approach: Communication can overlap computation because gradients are generated from final to initial layers, while their parameters are consumed in reverse order during the next iteration.Final-layer gradients have larger generation-to-consumption gaps, enabling synchronization to overlap with both forward and backward propagation.
  • Our Approach: P3 is a synchronization mechanism based on parameter slicing and priority-based updates.It splits layers into smaller independently synchronized slices and prioritizes slices according to when they are needed again.
  • Our Approach: Layer-wise synchronization can underutilize resources in heavy models, while scheduling based on gradient generation and consumption can use bandwidth more efficiently.The paper identifies VGG and Sockeye as examples of heavy models affected by layer-wise granularity.
  • Results: P3 improves training performance over standard MXNet by as much as 25% for ResNet-50, 38% for Sockeye, and 66% for VGG-19.The implementation is evaluated against the standard MXNet implementation.

2 BACKGROUND

DNNs learn parameters through repeated forward and backward propagation, while data-parallel workers synchronize gradients through parameter servers whose network communication can limit scalability.

  • DNN Training: A DNN is a sequence of layers that transforms input vectors using associated parameters into prediction outputs.Training seeks parameter values that produce accurate predictions.
  • DNN Training: Each training iteration performs forward propagation to compute predictions and loss, followed by backward propagation to calculate layer gradients.The gradients quantify each parameter’s error contribution with respect to the loss.
  • Distributed Training: Data-parallel training distributes the dataset among workers that simultaneously train a shared DNN using synchronous SGD.Distributing the workload across machines reduces total training time.
  • Parameter Server: A parameter server stores current parameters, supplies them to workers, and aggregates their locally computed gradients for synchronization.Workers read parameter values before iterations and send gradients after local computation.
  • Parameter Server: Network communication between workers and the parameter server often becomes the bottleneck preventing linear scalability.Figure 3 depicts parameter-server-based data-parallel training in a four-node cluster.
  • Synchronization Limitations: Current frameworks can synchronize gradients at layer granularity, which may cause suboptimal utilization when layer sizes differ substantially.The paper notes that gradients are often generated as whole-layer representations.

3 LIMITATIONS OF PARAMETER

Current parameter synchronization limits distributed DNN training because layer-level, aggressive communication creates queuing delays and leaves network resources idle. These problems worsen when model layers have highly unequal parameter sizes, motivating finer-grained, computation-aware scheduling.

  • Aggressive synchronization: Aggressive synchronization sends each layer’s gradients immediately after backward propagation, inducing queuing delays between iterations.The resulting delay includes additional waiting from preceding layers.
  • Aggressive synchronization: Forward propagation leaves the network completely idle under the illustrated synchronization pattern.
  • Uneven parameter sizes: Heavy fully connected layers require more synchronization time and can induce queuing delays for lighter initial convolution layers.
  • Uneven parameter sizes: VGG-19’s single fully connected layer contains 71.5% of all parameters, disrupting layer-level communication-computation pipelining.Pipelining is effective only when layer sizes are more or less uniform.
  • Uneven parameter sizes: In the three-layer example, communication delay is dominated by the imbalanced second layer, leaving computing resources and bidirectional bandwidth underutilized.The example spends the final three time steps receiving parameter updates.
  • Design implications: DNN-specific scheduling can reduce delay by considering both when gradients are generated and when parameters are consumed, while finer synchronization granularity improves resource use.

4 P3: DESIGN AND IMPLEMENTATION

P3 redesigns parameter synchronization around independently synchronized parameter slices and priorities based on forward-processing order. Its implementation modifies MXNet’s KVStore worker-server pipeline to prioritize slices and broadcast updates directly.

  • P3 design: P3 combines parameter slicing with priority-based updates as its two core design components.
  • P3 design: P3 splits layers into smaller parameter slices that can be synchronized independently, improving overlap between transmission and parameter updates.The slicing example also completely utilizes bidirectional bandwidth.
  • P3 design: Slices inherit priorities from parent layers, with earlier forward-propagated layers receiving higher priority than later layers.During backward propagation, slices are issued according to these priorities.
  • System architecture: P3 modifies MXNet’s KVStore, whose KVWorkers send gradients and whose KVServers aggregate gradients, update parameters, and serve updated values.
  • Worker and server implementation: P3Worker splits each layer’s gradient matrix using a predefined maximum slice size and assigns slices to P3Servers round-robin.
  • Worker and server implementation: Priority queues let worker and server consumer threads transmit and process the highest-priority slices first.Priorities are carried in packet headers and used as server queue keys.
  • Worker and server implementation: P3Server broadcasts updated parameters immediately after receiving all updates, eliminating explicit notifications and pull requests to improve bidirectional bandwidth utilization.The change preserves correctness because workers otherwise issue a pull after every push.

5 EVALUATION

P3 is evaluated against MXNet’s baseline across bandwidth sensitivity, network utilization, scalability, accuracy, and parameter-slice granularity. It improves throughput most for models or settings constrained by communication, while preserving convergence accuracy.

  • Evaluation setup: P3 is evaluated on ResNet-50, InceptionV3, VGG-19, and Sockeye against the standard MXNet KVStore implementation.Training throughput is measured after warm-up and averaged over 1,000 iterations.
  • Bandwidth sensitivity: 26% more throughput is achieved by P3 than the baseline for ResNet-50 at 4Gbps, while InceptionV3 reaches an 18% maximum speedup.P3 maintains linear ResNet-50 throughput below the bandwidth point where the baseline begins to drop.
  • Bandwidth sensitivity: 66% speedup is achieved for VGG-19 at 15Gbps, while Sockeye improves by up to 38% with P3.Parameter slicing alone provides 49% speedup for VGG-19 at 30Gbps because these models contain very large layers.
  • Bandwidth sensitivity: P3 performs better than the baseline particularly when bandwidth is limited, but its speedup diminishes when communication greatly exceeds computation.This makes P3 more suitable for shared clusters where effective bandwidth is below peak capacity.
  • Network utilization: P3 reduces network idle time and uses bidirectional bandwidth more effectively than the baseline, especially for Sockeye.The improved utilization is identified as a key reason for Sockeye’s speedup despite its heavy initial layers.
  • Scalability: 61% higher VGG-19 throughput is obtained on an eight-machine cluster, while Sockeye throughput improves by up to 18% on eight nodes.ResNet-50 shows similar baseline and P3 performance on the 10Gbps scalability setup.
  • Training accuracy: P3 achieves better final accuracy than DGC in the reported ResNet-110 experiments, with DGC showing an average accuracy drop of 0.4%.P3 communicates full gradients and does not modify the original SGD algorithm.
  • Parameter granularity: Throughput peaks at a maximum slice granularity of 50,000 parameters before smaller slices incur excessive packet-synchronization overhead.The 50,000-parameter maximum is used in all experiments because it was empirically optimal.

6 RELATED WORK

The related-work discussion contrasts P3’s communication-overlap strategy with methods that reduce transmitted information. P3 retains full gradients, avoiding the accuracy trade-offs associated with lossy or stale updates.

  • Paper scope: The paper frames P3 as a framework-level response to limitations in data-parallel deep-learning distribution methods.Its stated focus is exploiting domain-specific characteristics of deep-learning models.
  • Domain-specific synchronization: P3 extends wait-free backpropagation by overlapping communication with both forward and backward propagation and adding parameter slicing.Poseidon independently synchronizes layers, whereas P3 further uses slicing to improve network-bandwidth utilization.
  • Communication reduction: Gradient quantization reduces communication by representing gradients with fewer bits, while sparse synchronization sends only selected parameters.Examples include 1-bit SGD, gradient dropping, and AdaComp’s adaptive compression ratio.
  • Accuracy trade-offs: These compression approaches trade training performance against model accuracy because value approximation or stale parameter updates introduce information loss.P3 instead sends the full gradient matrix on every iteration.
  • Comparison with DGC: DGC reports up to 600× compression and around 5× speedup in low-bandwidth networks, but the authors report difficulty reproducing its results and better accuracy with P3.P3 is presented as orthogonal to compression and potentially usable on top of compression mechanisms.

7 CONCLUSION

The paper concludes that P3 improves distributed DNN training by better utilizing network bandwidth and overlapping synchronization with computation. Across several models, it improves throughput while preserving model convergence, with an accompanying open-source artifact.

  • Conclusion: P3 addresses underutilized network bandwidth and high performance penalties under bandwidth limitations in data-parallel training.The method is implemented on MXNet and compared with the standard MXNet implementation.
  • Conclusion: 66% higher VGG-19 throughput, 38% higher Sockeye throughput, and 25% higher ResNet-50 throughput are reported with P3.These are the paper’s headline improvements across image-classification and machine-translation models.
  • Artifact: The artifact includes P3 source code, benchmark programs, installation scripts, and the evaluated model benchmarks.The source code is publicly available, with an archived artifact also provided.
  • Artifact: The benchmark suite covers ResNet-50, InceptionV3, VGG-19, and Sockeye using ImageNet1K and IWSLT15 data.ImageNet1K preparation is manual, whereas IWSLT15 is included in the artifact.
  • Experimental requirements: Experiments require multiple Nvidia-GPU machines connected by a high-bandwidth interconnect, with four machines and at least 10Gbps recommended.The workflow uses a warm-up period before throughput measurement.

A.3.4 Data sets

The artifact workflow requires installing P3 and the baseline across participating machines, preparing or obtaining the datasets, configuring hosts and SSH access, and then running model-specific benchmarks.

  • Data preparation: The dataset must be copied to the exact same directory location on all machines after code installation and preparation.ImageNet1K is converted to RecordIO format from class-organized image directories.
  • Data preparation: ImageNet1K preparation uses the provided im2rec commands, while the IWSLT15 dataset is included for Sockeye.The image dataset is used for ResNet-50, InceptionV3, and VGG-19.
  • Installation: CUDA and cuDNN must be installed before unpacking and building the P3 artifact on each machine.The documented installation command builds and installs the MXNet-based implementation.
  • Installation: The baseline is installed by repeating the installation process after switching to the baseline branch.This supports comparison between P3 and the baseline implementation.
  • Cluster configuration: One machine must be designated as master, with all participating hosts listed and passwordless SSH enabled from the master.The hosts file is updated on the master machine.
  • Execution: Benchmarks are launched with the model selector for ResNet, Inception, VGG, or Sockeye.The provided command is run after installation and cluster configuration.
  • Bandwidth experiments: Linux tc qdisc can limit each interface’s peak transmission rate for bandwidth experiments.The documented command specifies rate, latency, burst, and MTU parameters.

A.6 Evaluation and expected result

The evaluation scripts report training throughput for the tested models and support comparisons with the paper’s Section 5 measurements. They can be customized by changing mini-batch size and host-file location.

  • Training throughput is output directly and can be compared with measurements reported in the corresponding Section 5 figures.
  • Benchmark scripts support per-machine mini-batch-size and hosts-file customization.
  • The experiment script accepts model selections for ResNet, Inception, VGG, or Sockeye.
  • The document provides links describing submission, reviewing, and artifact-badging methodologies.

B.1 Network Utilization

The paper examines whether limitations observed in one framework also occur elsewhere. TensorFlow and Poseidon show poor network utilization under bandwidth constraints, paralleling MXNet’s behavior.

  • The section’s purpose is to show that the limitations described earlier extend to other frameworks.
  • TensorFlow and Poseidon also utilize network bandwidth poorly under constrained conditions, similar to MXNet.The measurements concern ResNet-50 and InceptionV3 on a four-node cluster.
  • Figures 13 and 14 report network-utilization measurements for ResNet-50 on TensorFlow and InceptionV3 on Poseidon.

B.2 Asynchronous SGD

The paper compares P3 with asynchronous SGD, which avoids waiting for all workers but uses stale updates. P3 achieves higher final accuracy and reaches 80% accuracy faster in the reported experiment.

  • ASGD updates parameters asynchronously, so workers avoid waiting for all participants but incur stale-parameter updates and reduced convergence rate.
  • Figure 15 presents the ASGD-versus-P3 comparison.
  • 93% final top-1 accuracy was achieved by P3, compared with 88% for ASGD on ResNet-110 and CIFAR-10.The experiment used a four-machine cluster and a 1Gbps network.
  • 6× faster was the approximate rate at which P3 reached 80% accuracy compared with ASGD.
Loading 1905.03960v1…