PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Study

Suyog Gupta, Wei Zhang, Fei Wang

arXiv:1509.04210v3stat.MLcs.DCcs.LGcs.NE

TL;DR

Distributed asynchronous training involves coupled choices that can trade model accuracy against runtime as learner count, mini-batch size, learning rate, and synchronization change. Rudra studies these interactions with parameter-server experiments, introducing staleness-aware learning-rate modulation and synchronization protocols across CIFAR10 and ImageNet. The results support reducing mini-batch size as learners increase to preserve accuracy while controlling runtime and staleness.

  • Problem

    Asynchronous scale-out deep learning has complex interactions between training hyperparameters and distributed design choices, making their effects on accuracy and runtime difficult to characterize.

  • Method

    Rudra empirically evaluates synchronization protocols, gradient staleness, mini-batch size, learning rates, and learner counts using parameter-server training on CIFAR10 and ImageNet.

  • Results

    The experiments show that learning-rate division by average gradient staleness improves convergence and test error, while 1-softsync minimizes staleness and runtime for a given test error.

  • Takeaways & Limitations

    Maintaining model accuracy as learners increase requires reducing mini-batch size per learner, implying an upper limit on exploitable parallelism for a given model.

Abstract

from arXiv · show

This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtime performance and model accuracy. We introduce a new learning rate modulation strategy to counter the effect of stale gradients and propose a new synchronization protocol that can effectively bound the staleness in gradients, improve runtime performance and achieve good model accuracy. Our empirical investigation reveals a principled approach for distributed training of neural networks: the mini-batch size per learner should be reduced as more learners are added to the system to preserve the model accuracy. We validate this approach using commonly-used image classification benchmarks: CIFAR10 and ImageNet.

I. INTRODUCTION

Distributed deep learning couples non-convex optimization with scale-out systems, creating interdependencies between training hyperparameters and implementation choices that affect accuracy and runtime. Rudra studies these interactions empirically using parameter-server training and image-classification benchmarks.

  • Motivation: Distributed asynchronous-SGD creates interdependencies among synchronization protocol, learner count, and training hyperparameters that affect accuracy and runtime.These choices include mini-batch size and learning rate, alongside gradient staleness and system design.
  • Approach: Rudra is a parameter-server framework designed to study these interdependencies through empirical evaluation on public image-classification benchmarks.
  • Contributions: The study quantifies gradient staleness, investigates its interaction with mini-batch size, learning rate, and learner count, and evaluates effects on accuracy and training time.
  • Contributions: Rudra introduces a learning-rate tuning strategy for stale updates and a synchronization protocol intended to reduce communication overhead while retaining accuracy and runtime performance.
  • Distributed training: The framework uses data-parallel learners that independently train on unique mini-batches while maintaining copies of the model.The parameter server accumulates gradients and updates shared weights.

III. DESIGN AND IMPLEMENTATION

Rudra defines several asynchronous synchronization strategies around a parameter server and uses timestamps to quantify gradient staleness. The n-softsync protocol collects a controlled number of gradients, empirically bounding staleness in homogeneous clusters.

  • Terminology: The parameter server holds model weights, collects learner gradients, and applies updates, while learners compute weight updates.
  • Terminology: Rudra tracks weights with scalar timestamps and defines gradient staleness as the difference between current and gradient timestamps.Average staleness is computed from the vector clock of gradients triggering a weight-timestamp advance.
  • Synchronization protocols: Hardsync averages one gradient from each learner before updating and broadcasting weights, so its staleness is always zero.
  • Synchronization protocols: Async learners independently push gradients and pull weights, but gradient staleness can become difficult to control.The cited discussion notes that staleness can reach hundreds in a Downpour SGD implementation.
  • Synchronization protocols: In n-softsync, the parameter server updates weights after collecting c = floor(λ/n) gradients; in homogeneous clusters, staleness is empirically bounded at 2n.

B. Rudra-base System Architecture

Rudra-base is a controlled parameter-server architecture for studying scale-out and hyperparameter effects. It uses timestamp-aware learner communication and adjusts learning rates for synchronization and staleness.

  • Architecture: Each learner pulls current weights and timestamps before training, then sends gradients with their timestamps to the parameter server.Timestamp checks can reduce unnecessary pullWeights traffic.
  • Architecture: Rudra-base implements hardsync and n-softsync while using a multithreaded parameter server to accumulate gradients and apply updates.
  • Learning-rate handling: In hardsync, the learning rate is multiplied by λµ/B, while n-softsync scales it by the reciprocal of gradient staleness.The paper reports that this n-softsync treatment can significantly improve model accuracy.
  • Implementation: Rudra-base is non-blocking except for gradient and weight transfers, which use blocking MPI calls to control how gradients are received and handled.The architecture is intended to isolate scale-out and hyperparameter effects.

C. Rudra-adv and Rudra-adv∗System Architecture

Rudra-adv replaces the centralized communication path with a tree-structured parameter-server group, while Rudra-adv* further overlaps communication with computation. These designs target large models and small mini-batches but trade runtime gains against gradient staleness.

  • Motivation: Rudra-base becomes difficult to scale for large models because concurrent transfers to one parameter server create substantial network contention.For a 300 MB model, 16 simultaneous sends can take over one second under link contention.
  • Rudra-adv: Rudra-adv forms a tree of parameter-server nodes that averages gradients locally, relays them upward, and broadcasts updated weights downward.
  • Rudra-adv*: Rudra-adv uses learner-side communication threads to send gradients and receive weights asynchronously, with computation and communication buffers for weight pulls.
  • Evaluation setup: The communication-overlap comparison uses Rudra-base, Rudra-adv, and Rudra-adv* with a 300 MB model and about 60 learners in an adversarial scenario.
  • Rudra-adv*: Rudra-adv* can almost completely overlap computation with communication and supports large models with the smallest possible mini-batch size.The paper connects this design to improved runtime performance while retaining good model accuracy.

A. Hardware and software environment

Rudra is evaluated on a POWER7 supercomputer using CIFAR10 and ImageNet image-classification benchmarks with convolutional neural networks. The experiments span the framework’s scale-out evaluation and associated software environment.

  • Hardware environment: Rudra runs on a POWER7 supercomputer whose nodes provide four eight-core processors, 128 GB memory, and 192 GB/s bidirectional interconnect bandwidth.Each node has a theoretical peak performance of 982 Gflop/s and memory bandwidth of 512 GB/s.
  • Software environment: The software environment uses Red Hat Enterprise Linux 6.4, IBM xlC 12.1, ESSL for BLAS routines, and IBM MPI 1.2.
  • Benchmark datasets: The evaluation uses CIFAR10 and ImageNet, extending the initial CIFAR10 scale-out exploration to ImageNet.CIFAR10 contains 60,000 RGB images, while ImageNet provides the larger benchmark setting described in the paper.
  • Network architectures: The CIFAR10 model is a three-convolutional-layer network with pooling, a fully connected layer, and a 10-way softmax output.
  • Network architectures: The ImageNet model has five convolutional layers, three fully connected layers, approximately 72 million parameters, and a 289 MB model size.It is trained with momentum-accelerated SGD using batch size 256 and momentum 0.9.

A. Stale gradients

The study examines how synchronization choices introduce gradient staleness and how learning-rate modulation can mitigate its effect. It evaluates these relationships through staleness measurements and runtime–accuracy tradeoffs.

  • Staleness mechanisms: Hardsync has zero gradient staleness, whereas n-softsync and asynchronous protocols introduce gradients computed from earlier weight timestamps.
  • Staleness mechanisms: With 30 learners, 1-softsync waits for 30 gradients and 2-softsync waits for 15 before updating the parameter-server weights.The resulting average-staleness behavior is measured in Figure 3(a).
  • Learning-rate modulation: For n-softsync, the learning rate is modulated using average gradient staleness to account for stale updates.The paper reports that this adjustment was beneficial and sometimes necessary in experiments.
  • Learning-rate modulation: Dividing the learning rate by average staleness improves convergence and lowers test error for n-softsync with 30 learners and minibatch size 128.
  • Runtime–accuracy tradeoffs: The hardsync tradeoff curves compare staleness, minibatch size, and learner count against a 17.9% CIFAR10 baseline test error.

B. (σ, µ, λ) tradeoff curves

The tradeoff curves expose how synchronization, minibatch size, learner count, and gradient staleness jointly determine training time and test error. Across protocols, reducing per-learner minibatch size as learners increase can recover accuracy, while 1-softsync offers the strongest runtime under these configurations.

  • Hyperparameter search over minibatch size, learner count, and staleness is costly because runtime and model quality must be optimized jointly.The tradeoff curves represent validation or test error against total wall-clock training time across configurations.
  • With hardsync, training time decreases as learner count increases, but test error rises unless minibatch size is reduced.Along the λ = 30 contour, reducing µ from 128 to 4 restores much of the error degradation while sacrificing part of the speed-up.
  • The 1-softsync protocol avoids λ-softsync’s runtime degradation at µ = 4 and λ = 30 by reducing parameter-server pullWeights traffic.For µ = 128, both softsync protocols show similar speed-ups through λ = 30; beyond 30 learners, parameter-server bottlenecks are expected to limit λ-softsync.
  • For hardsync, configurations with the same µλ product are expected to produce nearly the same test error.The equivalence follows because synchronized learners compute gradients using identical weights; small residual differences can arise from stochastic sampling and initialization.
  • When µλ ≈128, test error remains about 18–19% even as average gradient staleness varies from 1 to 30 under n-softsync.The bounded staleness keeps learners’ weight vectors approximately similar, and smaller updates improve this approximation.
  • The resulting guideline is to reduce each learner’s minibatch size as learners are added so that µλ stays constant, while modulating learning rates for stale gradients.The study also reports that 1-softsync has the smallest training time for any µλ, whereas test error increases monotonically with µλ.

D. Summary of results on CIFAR10 benchmark

On CIFAR10, the Rudra-base baseline reaches 17.9% test error after training 140 epochs, with a 22,392-second runtime.

  • 17.9% test error is achieved by the baseline configuration (σ, µ, λ) = (0, 128, 1).The configuration takes 22,392 seconds to finish training 140 epochs.

E. Results on ImageNet benchmark

On ImageNet, reducing the minibatch size enables many learners to preserve baseline accuracy while improving runtime, although more aggressive scaling can increase staleness and validation error. The fastest configurations trade accuracy for speed, while 1-softsync improves runtime over hardsync with slightly higher validation error than baseline.

  • ImageNet runtime and accuracy: 54 hours/epoch is reduced to ∼330 minutes/epoch with µ = 16 and λ = 18, while top-5 error remains 20.85%, matching baseline accuracy.This result uses the Rudra-base architecture with hardsync.
  • ImageNet runtime and accuracy: ∼270 minutes/epoch is achieved by 1-softsync with µ = 16 and 18 learners, reaching 45.63% top-1 and 22.08% top-5 accuracy after 30 epochs.This setup uses AdaGrad and warmstarting to stabilize and accelerate training.
  • Scaling learners: λ = 54 learners with µ = 4 trains at ∼212 minutes/epoch and reaches 46.09% top-1 and 22.44% top-5 error using Rudra-adv and 1-softsync.The Rudra-adv architecture uses a tree-structured parameter server group to alleviate bottlenecks.
  • Scaling learners: ∼125 minutes/epoch with Rudra-adv* increases top-1 validation error to 46.53%, while µ = 8 and λ = 54 reaches ∼96 minutes/epoch but exceeds 50% top-1 error.The results support reducing minibatch size per learner as the number of learners increases.
  • Training dynamics: Training speed ranks adv*-softsync > adv-softsync > base-softsync > base-hardsync, while non-baseline configurations show marginally higher validation error.Figure 8 compares top-1 validation-error evolution across the four configurations; the paper speculates that tuning AdaGrad’s initial learning rate may recover some accuracy.

VI. RELATED WORKS

Related work positions distributed deep learning at the intersection of machine learning and high-performance computing, emphasizing parameter-server scale-out and the accuracy effects of gradient staleness. Rudra studies these effects empirically and identifies smaller minibatches as a way to counter staleness across CIFAR10 and ImageNet.

  • GPU-based solutions accelerate matrix-heavy training but may be limited by the memory available for large models.
  • DistBelief and Adam use parameter-server architectures with asynchronous learner updates, while differing in system tuning and optimization details.
  • Parameter-server staleness negatively impacts model accuracy, motivating bounded-staleness and alternative optimization approaches.
  • Rudra empirically finds that smaller minibatches can counter staleness, with the heuristic transferring from CIFAR10 to ImageNet.

VII. CONCLUSION

The study divides the learning rate by average gradient staleness to improve convergence and test error, and finds that 1-softsync minimizes staleness and runtime while maintaining model accuracy.

  • Dividing the learning rate by average gradient staleness produces faster convergence and lower test error.
  • The 1-softsync protocol accumulates λ gradients before updating weights and is reported to minimize gradient staleness while achieving the lowest runtime.
Loading 1509.04210v3…