PaperlayerView exact revision on arXiv

Source-linked AI summary

SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat

arXiv:1910.00643v2cs.LGcs.DCmath.OCstat.ML

TL;DR

Communication-efficient distributed training reduces synchronization costs but can lower model quality, motivating a method that improves accuracy without giving up those efficiency gains. SlowMo periodically synchronizes workers and applies momentum over a base optimizer; experiments report consistent optimization and generalization improvements, while theory establishes convergence guarantees.

  • Problem

    Communication-efficient distributed methods can run faster than blocking ALLREDUCE methods but may produce less accurate models after the same number of updates.

  • Method

    SlowMo runs a base optimizer for multiple steps, periodically averages worker parameters, and applies a slow-momentum update.

  • Results

    SlowMo consistently improves optimization and generalization across image classification and machine translation tasks while maintaining similar communication efficiency.

  • Takeaways & Limitations

    SlowMo provides a general framework that improves communication-efficient training and yields convergence guarantees for smooth non-convex objectives.

  • Takeaways & Limitations

    The extent to which compression benefits parameter-averaging methods such as BMUF, D-PSGD, SGP, and OSGP remains unclear.

Abstract

from arXiv · show

Distributed optimization is essential for training large models on large datasets. Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods (e.g., using gossip algorithms) to decouple communications among workers. Although these methods run faster than AllReduce-based methods, which use blocking communication before every update, the resulting models may be less accurate after the same number of updates. Inspired by the BMUF method of Chen & Huo (2016), we propose a slow momentum (SlowMo) framework, where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm. Experiments on image classification and machine translation tasks demonstrate that SlowMo consistently yields improvements in optimization and generalization performance relative to the base optimizer, even when the additional overhead is amortized over many updates so that the SlowMo runtime is on par with that of the base optimizer. We provide theoretical convergence guarantees showing that SlowMo converges to a stationary point of smooth non-convex losses. Since BMUF can be expressed through the SlowMo framework, our results also correspond to the first theoretical convergence guarantees for BMUF.

1 INTRODUCTION

Communication-efficient distributed methods reduce synchronization costs but can sacrifice model quality. SlowMo adds periodic synchronization and momentum to improve optimization and generalization while retaining communication efficiency and theoretical convergence guarantees.

  • Motivation: Blocking ALLREDUCE communication before every optimizer step limits distributed training scalability and makes performance sensitive to stragglers.Distributed optimization is used for large models and datasets, but blocking synchronization creates substantial communication overhead.
  • Existing approaches: Decentralized training and Local SGD reduce communication overhead through approximate aggregation or multiple local updates between communication rounds.These approaches respectively decouple communication or reduce its frequency.
  • Existing approaches: Faster communication-efficient methods may produce lower training quality or generalization accuracy after the same number of iterations.The added optimization noise is identified as a reason these methods may not match large-minibatch methods.
  • Motivation: Momentum improves optimization and generalization empirically, but its integration with communication-efficient algorithms lacks consensus.Existing approaches commonly maintain separate, unsynchronized momentum buffers, while synchronization has also been studied.
  • Contribution: SlowMo periodically averages worker parameters after τ base-optimizer steps and applies a momentum update on top of local SGD or decentralized methods.The framework is evaluated across image classification and neural machine translation tasks.
  • Contribution: SLOWMO converges to a stationary point of smooth non-convex functions at rate O(1/√(mTτ)), and its framework includes BMUF.The result applies across a variety of base optimizers and gives BMUF its first theoretical convergence guarantees.

2 THE SLOW MOMENTUM (SLOWMO) FRAMEWORK

SlowMo is a nested framework that runs a base optimizer for τ inner steps, synchronizes worker parameters, and then applies slow momentum. By selecting the base algorithm and hyperparameters, it recovers methods including large-minibatch SGD, Local SGD, BMUF, and Lookahead.

  • Framework structure: SLOWMO builds on a base optimization algorithm and uses nested inner and outer loops across m workers.The framework targets stochastic optimization with locally available objectives and samples.
  • Framework structure: Within each outer iteration, workers perform τ base-optimizer steps, using either a non-communicating optimizer or a decentralized method such as SGP.The base optimizer determines the inner updates and may itself involve communication.
  • Synchronization and update: After the inner steps, workers use ALLREDUCE to average parameters and then perform a synchronized slow-momentum update.The resulting parameters and slow-momentum buffer are identical across workers after synchronization.
  • Synchronization and update: The slow-momentum buffer scales the parameter difference by 1/γt, making it invariant to changes in the fast learning rate.The outer update uses the product αγt of the slow and fast learning rates.
  • Recovering existing methods: SGD with τ = 1, α = 1 recovers large-minibatch SGD with momentum, while SGD with τ > 1, α = 1, β = 0 recovers Local SGD.Other base-optimizer and hyperparameter choices recover additional algorithms.
  • Recovering existing methods: With a communication-free base optimizer, τ > 1 and β > 0 recover BMUF; with m = 1 and β = 0, the framework recovers Lookahead.The latter uses SGD or Adam as its base optimizer and α ∈ (0, 1].
  • Empirical behavior: SLOWMO combined with SGP consistently improves optimization and generalization over SGP alone without a significant runtime increase.For a fixed iteration count, SLOWMO with SGP is also reported as superior to SLOWMO with SGD.

3 RELATED WORK

Prior work reduces communication through local updates, decentralized averaging, and compression, while theoretical guarantees often target different optimization settings. The paper positions SlowMo among these methods and identifies open questions about compression and accelerated momentum theory.

  • Communication-efficient training: Local SGD reduces communication by synchronizing parameters only after multiple optimizer steps, while related work also uses parameter-server averaging.This line of work predates the paper and includes postlocal SGD for distributed deep learning.
  • Decentralized optimization: Decentralized methods replace ALLREDUCE with peer-to-peer approximate averaging, reducing coupling but injecting noise because worker models are not exactly synchronized.D-PSGD, AD-PSGD, SGP, and OSGP differ in synchronization and asynchrony, with reported speedups over more coupled methods.
  • Theory: Existing decentralized convergence results mainly address smooth non-convex SGD without momentum, while related optimal-rate results may assume deterministic convex objectives.These settings do not directly match the smooth non-convex stochastic optimization setting studied here.
  • Related frameworks: Methods in the SLOWMO framework relate to distributed learning approaches that average locally optimized models and use the resulting direction in momentum.The framework uses the actual worker loss rather than an approximate loss in the cited comparison.
  • Open questions: The benefit of combining gradient compression with parameter-averaging methods such as BMUF, D-PSGD, SGP, and OSGP remains unclear.Combining SLOWMO with compression is identified as future work.
  • Open questions: Theory for momentum's benefits remains limited, and extending accelerated convergence results from serial settings to decentralized momentum is left open.The cited accelerated results cover restricted deterministic or strongly convex settings rather than the paper's full target setting.

4 EXPERIMENTAL RESULTS

Experiments across image classification and machine translation show that SLOWMO improves optimization and generalization across distributed training baselines, while adding negligible communication cost. Its performance depends on τ, with moderate values balancing speed and accuracy.

  • Optimization and Generalization Performance: SLOWMO consistently improves training loss and validation accuracy or BLEU across all evaluated tasks and baseline algorithms.The evaluation covers CIFAR-10, ImageNet, and WMT’16-En-De with Local SGD, Local Adam, SGP, and OSGP baselines.
  • Communication Cost: SLOWMO maintains nearly the same average iteration speed as SGP and OSGP because extra parameter averaging is amortized over τ = 48 iterations.For Local SGD and Local Adam, SLOWMO does not increase communication because those methods already average parameters every τ iterations.
  • Effects of τ: τ = 48 provides a good speed–accuracy tradeoff for ImageNet and WMT’16, while larger values remain better than the corresponding base optimizer in tested settings.Validation performance is non-monotonic in τ; τ = 96 for ImageNet and τ = 192 for WMT’16 still outperform SGP alone.
  • Comparison with Double-Averaging Momentum: 75.73% validation accuracy and 302 ms per iteration: SLOWMO-SGP outperforms double-averaging momentum at 75.54% and 402 ms per iteration.The comparison is reported on ImageNet.
  • Comparison with Double-Averaging Momentum: 73.24% validation accuracy and 282 ms per iteration: SLOWMO outperforms double-averaging with Local SGD at 72.04% and 405 ms per iteration.The Local SGD comparison uses τ = 12 for the baseline.

5 THEORETICAL RESULTS

The theoretical analysis establishes convergence guarantees for SLOWMO under smooth non-convex stochastic optimization, including linear speedup with more workers. It also recovers known rates for AR-SGD and provides guarantees for BMUF and Lookahead as special cases.

  • Assumptions: SLOWMO is analyzed for smooth non-convex objectives with worker-specific expected losses, stochastic-gradient variance, and bounded update-direction variance.The analysis uses a constant fast learning rate and assumes synchronized initialization with zero initial slow momentum.
  • General guarantee: Theorem 1 gives a convergence bound for SLOWMO when α, β, γ, τ, and T satisfy a specified parameter relationship and τT is sufficiently large.The bound is stated under Assumptions 1–3 and includes an objective-gap term, stochastic variance, and base-optimizer effects.
  • Special cases: SLOWMO recovers the AR-SGD rate in the τ = 1, α = 1, β = 0 special case and yields convergence corollaries for BMUF and Lookahead.BMUF corresponds to Local SGD with SLOWMO, while Lookahead corresponds to the single-node β = 0 case.
  • Base-optimizer effect: The base-optimizer term measures bias between the full-batch gradient and the averaged expected update, reflecting discrepancies among local models.For the considered base optimizers, this term can be related to existing distributed-optimization analyses.
  • Linear speedup: O(1/√(mTτ)) is the dominant rate after sufficiently many steps, so m workers require m times fewer steps to achieve a fixed error.This recovers the rate of AR-SGD and establishes a linear speedup in the number of workers.

6 FASTER SLOWMO: REMOVING THE PERIODIC ALLREDUCE

The paper removes the periodic ALLREDUCE from SGP-SLOWMO to reduce communication overhead. The resulting variant preserves iteration time and achieves nearly the same reported performance as standard SGP-SLOWMO.

  • Variant: SGP-SLOWMO-noaverage omits the exact parameter average during SLOWMO updates, addressing the additional communication cost incurred by SGP and OSGP.The variant is evaluated with α = 1, β = 0.6, and τ = 48.
  • Results: 75.78% versus 75.73% on ImageNet shows similar performance for SGP-SLOWMO-noaverage and SGP-SLOWMO.The comparison is reported for the ImageNet experiment.
  • Results: 2.11 versus 2.10 validation NLL on WMT’16 shows only a slight degradation without the exact average.Validation NLL is reported for SGP-SLOWMO-noaverage versus SGP-SLOWMO, respectively.
  • Runtime: 298 ms per iteration on ImageNet and 1227 ms on WMT’16 preserve the base algorithm’s iteration time because the variant requires no additional communication.The authors attribute the similar runtime to removing the extra communication step.
  • Interpretation: The results suggest that slow momentum updates contribute more to SLOWMO’s performance gain than momentum-buffer synchronization.The paper leaves further investigation of this variant for future work.

7 CONCLUDING REMARKS

The paper presents SLOWMO as a general momentum framework for communication-efficient distributed optimization. Across three deep learning tasks, it reports improved performance, convergence to stationary points, and the first convergence guarantee for BMUF.

  • Framework: SLOWMO can be applied on top of SGD and decentralized methods including SGP and asynchronous OSGP.The framework is designed for communication-efficient distributed optimization algorithms.
  • Empirical findings: Across three deep learning tasks, SLOWMO consistently improves optimization and generalization performance while maintaining similar communication efficiency.This is the paper’s main empirical conclusion relative to the corresponding baseline algorithms.
  • Theory: SLOWMO converges to a stationary point of smooth non-convex objective functions.This conclusion is supported by the paper’s theoretical convergence analysis.
  • Implications: Because BMUF is expressible as SLOWMO with Local SGD or Local Adam, the paper provides the first convergence guarantee for BMUF in the literature.The claim is stated as a consequence of the framework’s theoretical analysis.

A.1 IMPLEMENTATION DETAILS

The experiments use PyTorch-based implementations across CIFAR-10, ImageNet, and WMT’16 En-De, with task-specific models, optimization schedules, and SLOWMO hyperparameter searches. Reported WMT’16 results show improved validation NLL for SGP and Local Adam with SLOWMO.

  • Software: PyTorch 1.0 implementations use CUDA 9.2, CUDNN 7.3, and NCCL 2.2.13 across the reported experiments.ImageNet, WMT’16 En-De, SGP, OSGP, and CIFAR-10 implementations build on the cited public codebases.
  • ImageNet: ImageNet uses ResNet-50 for 90 epochs with a linearly scaled learning rate, decay at epochs 30, 60, and 80, Nesterov momentum 0.9, and weight decay 10^-4.The schedule follows the protocol suggested by Goyal et al. (2017).
  • WMT’16 En-De: WMT’16 En-De uses a six-block big Transformer with Adam, beta1 = 0.9, beta2 = 0.98, ϵ = 10^-8, and 25 training epochs.The learning-rate schedule follows Ott et al. (2018).
  • WMT’16 results: SLOWMO improves validation NLL for SGP and Local Adam on WMT’16 En-De.Table B.1 reports validation NLL, where lower values are better, with and without SLOWMO.

B ADDITIONAL EMPIRICAL RESULTS

On WMT’16 En-De, SLOWMO improves the validation NLL and BLEU score of both SGP and Local Adam.

  • SLOWMO improves validation NLL and BLEU score for SGP and Local Adam on WMT’16 En-De.

B.2 ADDITIONAL TRAINING CURVES

Additional training-curve experiments show that SLOWMO accelerates SGP convergence, while hyperparameter sweeps favor α = 1 and intermediate slow momentum values. On WMT’16 En-De, some larger-β settings substantially worsen validation loss.

  • Additional training curves: SLOWMO substantially improves the convergence speed of SGP.The corresponding training curves use α = 1 and τ = 12, with shaded min-max ranges across workers.
  • Hyperparameter impact: On CIFAR-10 with OSGP, α = 1 gives the highest validation accuracy for every fixed β, while the best β ranges from 0.4 to 0.8.
  • Hyperparameter impact: On WMT’16 En-De, α = 1 gives lower validation loss than α = 0.5 for fixed β with SGP or Local Adam.
  • Hyperparameter impact: For SLOWMO-Adam, validation loss is substantially worse with β > 0.5 and α = 1.0 or β > 0.7 and α = 0.5.These settings are omitted from the plotted results.

B.4 BASE OPTIMIZER MOMENTUM BUFFER STRATEGIES

The appendix compares strategies for handling base-optimizer momentum buffers across tasks. Resetting is preferred for ImageNet, whereas maintaining buffers is preferred for Adam on WMT’16 because resetting the second-moment buffer hurts optimization and averaging costs more communication.

  • Buffer strategies: SLOWMO can reset, maintain, or average base-optimizer buffers at each outer iteration.The base optimizer may use momentum buffers, including Adam’s first- and second-moment estimates.
  • ImageNet: On ImageNet, buffer strategies achieve similar training and validation performance, so reset is selected to avoid averaging’s higher communication overhead.
  • WMT’16 En-De: On WMT’16, resetting underperforms maintaining or averaging, while averaging achieves the best results at significantly higher communication cost.
  • WMT’16 En-De: For Adam on WMT’16, resetting the second-moment buffer to zero hurts optimization, so maintaining buffers is selected as the default.The passage attributes this to the importance of warming up Adam’s buffer.

D.3.3 FINAL RESULTS

The analysis derives convergence bounds for SLOWMO by relating its outer and inner updates to the base optimizer. In special cases, the bound reduces to vanilla mini-batch SGD, and sufficiently large iteration counts make the leading term determine the rate.

  • Final convergence bound: The final convergence bound upper-bounds the expected average squared gradient norm using the initial optimality gap and error terms.
  • Base-optimizer cases: For Local SGD, all worker nodes are averaged every τ iterations and the variance term is V = σ2/m.
  • Base-optimizer cases: For SGD with one worker, β = 0, the expected base update equals the gradient, and V = σ2.
  • Special cases: When α = 1 or τ = 1, the upper bound reduces to the vanilla mini-batch SGD case.
  • Rate behavior: If K is sufficiently large, the first term dominates the convergence rate.
Loading 1910.00643v2…