PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Decentralized Deep Learning by Dynamic Model Averaging

Michael Kamp, Linara Adilova, Joachim Sicking, Fabian Hüger, Peter Schlicht, Tim Wirtz, Stefan Wrobel

arXiv:1807.03210v2cs.LGcs.AIcs.DCstat.ML

TL;DR

Decentralized deep-learning applications can find data centralization or periodic gradient sharing infeasible, while non-convex model averaging can itself worsen performance. This paper adapts dynamic averaging to deep learning, triggering communication according to model divergence. It reports improved performance-communication trade-offs over periodic approaches, including stronger advantages under concept drift, and retains loss bounds with communication tied to learning-problem hardness.

  • Problem

    For some distributed applications, centralizing data or periodically sharing gradients requires infeasible communication, and non-convex model averaging can produce a worse model than any local model.

    Source: §1 · pp. 1–2

  • Method

    The paper adapts dynamic averaging to decentralized deep learning, using synchronization based on model divergence while evaluating predictive performance and communication.

    Source: §1 · p. 3, §2 · p. 4

  • Results

    Dynamic averaging outperforms periodic averaging in practice, including with learner subsampling, with its advantage amplified under concept drift.

    Source: §6 · p. 12

  • Takeaways & Limitations

    The protocol retains the underlying learning algorithm’s loss bounds with communication bounded by the hardness of the learning problem.

    Source: §7 · p. 14

    Without a bound on local-model quality, averaging can be arbitrarily bad, and whether robust methods apply to decentralized deep learning remains open.

    Source: §6 · p. 13

Abstract

from arXiv · show

We propose an efficient protocol for decentralized training of deep neural networks from distributed data sources. The proposed protocol allows to handle different phases of model training equally well and to quickly adapt to concept drifts. This leads to a reduction of communication by an order of magnitude compared to periodically communicating state-of-the-art approaches. Moreover, we derive a communication bound that scales well with the hardness of the serialized learning problem. The reduction in communication comes at almost no cost, as the predictive performance remains virtually unchanged. Indeed, the proposed protocol retains loss bounds of periodically averaging schemes. An extensive empirical evaluation validates major improvement of the trade-off between model performance and communication which could be beneficial for numerous decentralized learning applications, such as autonomous driving, or voice recognition and image classification on mobile phones.

1 Introduction

Periodic model averaging can spend communication regardless of its utility and cannot balance adaptation to concept drift against unnecessary communication between drifts. Dynamic averaging instead ties communication to model divergence, adapting a prior approach for convex objectives to deep learning’s non-convex setting.

1 Introduction

  • Centralizing distributed data or periodically sharing gradients can require infeasible communication, despite periodic gradient aggregation working well in tightly connected systems.Model averaging can instead send model parameters and avoid exchanging or centralizing privacy-sensitive data.Source: §1 · pp. 1–2
  • In non-convex problems, the average of local models can perform worse than every model being averaged.The passage illustrates this problem with deep learning model averaging, termed Federated Learning by McMahan et al..Source: §1 · p. 2, Figure 1.1 · p. 2
  • Periodic averaging communicates independently of utility, while concept drifts force a trade-off between slow adaptation and unnecessary communication.Communicating rarely can slow adaptation to a drift; communicating frequently can generate substantial communication between drifts.Source: §1 · p. 2
  • Dynamic averaging allocates communication by utility, avoiding it when learners suffer no loss and increasing it when they suffer large losses.The approach was proposed for online convex objectives and is adapted here to non-convex deep-learning objectives.Source: §1 · pp. 2–3
  • The paper evaluates dynamic averaging on classical deep-learning tasks and synthetic and real-world concept-drift tasks, including autonomous-driving in-fleet learning.Comparisons include periodically communicating schemes such as Federated Averaging.Source: §1 · p. 3

2 Preliminaries

The preliminaries define decentralized online learning through local learners, their synchronization protocol, and cumulative loss and communication measures. Efficiency requires both retaining serial-learning performance and bounding communication in relation to the serial learner’s loss.

2 Preliminaries

  • The setting has m local learners running the same algorithm on streaming batches drawn from a time-varying distribution, with cumulative loss as the learning objective.The setup includes online learning with B = 1 and mini-batch training with B > 1.Source: §2 · p. 3
  • A decentralized protocol pairs a learning algorithm with a synchronization operator that replaces local models with an aggregate; communication is measured cumulatively.For neural networks, the local models share an architecture, so their average is the average of their weights.Source: §2 · p. 4
  • Consistency means retaining the serial learner’s loss, while adaptiveness requires communication bounded linearly in learner count and serial loss.A protocol is efficient when it satisfies both criteria.Source: §2 · p. 4
  • Periodic protocols are consistent but not adaptive because their communication grows with learners and rounds independently of loss.The passage contrasts this with non-synchronizing protocols, which are adaptive but not consistent.Source: §2 · p. 4

3 Dynamic Averaging

Dynamic averaging synchronizes when model divergence warrants communication, while preserving the mean model and keeping divergence within a threshold. Local checks can trigger either selective balancing among learners or a full synchronization.

3 Dynamic Averaging

  • The protocol measures whether synchronization is worthwhile using model-configuration divergence: communication is less useful when models are already similar and more useful when they differ.Similarity can result from small changes since the previous averaging step or from learners reaching similar models.Source: §3 · p. 5
  • The dynamic averaging operator can omit synchronization when divergence is low; when applied, it preserves the mean model and bounds divergence by ∆.It may leave all models unchanged below the threshold or average only a subset to meet the divergence constraint.Source: §3 · p. 5
  • Learners check their distance from a shared reference model every b rounds, and the local conditions ensure global divergence stays within ∆ when none is violated.The last synchronization’s average is the first choice for the reference model; a closer reference to the true average makes the local conditions tighter.Source: §3 · p. 5
  • After a local violation, the coordinator can incrementally query learners and return an average if it lies within the safe zone, or synchronize all learners.If all nodes are queried, the result is equivalent to full synchronization; the balancing process bounds divergence and leaves the global mean unchanged.Source: §3 · pp. 5–6

4 Efficiency of Dynamic Averaging

The efficiency analysis relates continuous averaging with mini-batch SGD to serial mini-batch SGD, then establishes performance and communication properties for dynamic averaging. Its predictive-performance guarantee relies on a local-convexity condition, while its communication bound scales with learner count and serial loss.

4 Efficiency of Dynamic Averaging

  • Continuous averaging with m learners, batch size B, and mini-batch SGD is equivalent to serial mini-batch SGD with batch size mB and a learning rate m times smaller.The equivalence is stated for the continuous averaging protocol using mini-batch SGD.Source: §4 · p. 7
  • Dynamic averaging with SGD retains the serial mini-batch SGD loss bound, including for classic SGD with B = 1.The passage identifies continuous averaging as consistent on this basis.Source: §4 · p. 7
  • When the loss is locally convex within an O(∆)-radius around the current average, dynamic averaging with SGD has performance similar to periodically communicating protocols.The passage describes this as a non-trivial but realistic assumption [29, 20].Source: §4 · p. 7
  • The dynamic averaging communication bound is adaptive because model communication costs O(m) bytes for a fixed-size neural network.The bound is given for the protocol using SGD and divergence threshold ∆.Source: §4 · p. 8

5 Empirical Evaluation

Across MNIST, concept-drift, and deep-driving experiments, dynamic averaging retained comparable predictive performance with less communication than periodic approaches. It also adapted communication around concept drifts, while deep-driving results depended on choosing a suitable communication level.

5 Empirical Evaluation

  • The experiments used mini-batch SGD with synchronization operators whose hyperparameters were optimized on an independent dataset.The evaluation included MNIST CNN classification, a synthetic dataset with concept drifts, and autonomous-driving simulation.Source: §5 · p. 8

Dynamic Averaging for Training Deep Neural Networks

  • On MNIST, dynamic averaging with σ∆=0.7 reached performance comparable to periodic averaging with σb=1 using half the communication.The experiment compared cumulative error and communication across dynamic and periodic setups, plus non-synchronizing and serial baselines.Source: Dynamic Averaging for Training Deep… · p. 9, Figure 5.1 · p. 9
  • In the non-convex setting, periodic synchronization with σb=4 had higher cumulative loss than the non-synchronizing baseline.Local models can converge to different local minima, whose average may have higher loss than each local model.Source: Dynamic Averaging for Training Deep… · p. 9

Comparison of the Dynamic Averaging Protocol with FedAvg

Adaptivity to Concept Drift

Case Study on Deep Driving

  • In deep driving, a communication protocol could be outperformed by a dynamic protocol; too little communication and very high communication (σb=1 and σ∆=0.01) performed badly, while dynamic averaging with ∆=0.1 or ∆=0.3 performed similarly to the serial model.Source: Case Study on Deep Driving · p. 12

6 Discussion

Dynamic averaging is reported to outperform periodic averaging in practice, with its advantage amplified under concept drift, while also applying across learning algorithms. The discussion identifies scalability mechanisms and open issues involving outliers and threshold selection.

6 Discussion

  • Model averaging requires a common architecture for local models because they jointly train one global model, distinguishing it from ensemble methods.Distributed mini-batch SGD is also described as a strong parallel-learning approach for convex problems and as performing similarly to periodic averaging for deep networks.Source: §6 · p. 12
  • Dynamic averaging outperforms periodic averaging in practice, with the advantage amplified under concept drift and applicable across arbitrary learning algorithms.The paper also says its non-convex results extend earlier convex-case findings on retaining performance with less communication.Source: §6 · p. 12
  • Model averaging is susceptible to outliers: without a bound on local-model quality, their average can be arbitrarily bad.The discussion notes that robust alternatives can be computationally expensive or unsuitable for non-convex problems, leaving their application to decentralized deep learning open.Source: §6 · p. 13
  • The divergence threshold Δ is not intuitive to set; the suggested practice is to tune it on a small data subset for the desired performance–communication trade-off.Whether Δ can be adapted during learning in a theoretically sound way remains an open question.Source: §6 · p. 13
  • False alarms from local violations can increase communication as learner count grows, but balancing violations with a small learner subset partially compensates.In the worst case, dynamic averaging communicates as much as periodic averaging; the experiments nevertheless report good scaling with learner count.Source: §6 · p. 13, Figure 6.1 · p. 13
  • With frequent averaging, mild initialization heterogeneity can improve training, whereas large heterogeneity and one-round averaging can impair it.The compared MNIST performances vary by initialization heterogeneity ε and averaging rounds b, relative to homogeneous initialization with b = 1.Source: §6 · pp. 13–14, Figure 6.2 · p. 14

7 Conclusion

The conclusion presents dynamic averaging as a communication-efficient way to retain high predictive performance while adapting to concept drift, with loss bounds and communication tied to problem hardness. The appendix section supplies further experimental setup and evaluation details.

7 Conclusion

  • Dynamic averaging achieves similarly high predictive performance with substantially less communication than periodic averaging and adapts to concept drifts.The paper states that it retains the underlying learning algorithm’s loss bounds, with communication bounded by the hardness of the learning problem.Source: §7 · p. 14

A Details on the Empirical Evaluation

A.1 Experiments on MNIST

The MNIST experiments report that dynamic averaging can match predictive performance with less communication, including in comparison with FedAvg. A synthetic concept-drift experiment shows communication adapting after drifts as learner loss changes.

A.1 Experiments on MNIST

  • After 3000 examples, dynamic averaging reduces communication without greater loss and eventually achieves both lower cumulative error and lower cumulative communication than a similarly performing periodic protocol.During the first 500 examples, it invests more communication while both protocols have high loss, resulting in faster convergence and a lower increase in cumulative error.Source: §A.1 · p. 19, Figure A.1 · p. 19

A.2 Comparison with FedAvg

  • At σΔ=0.8, communication falls by more than 80% compared with σb=5; σΔ,b and σFedAvg protocols accompany slight cumulative-loss increases and slight model-accuracy decreases.Each learner is trained on 8000 training examples; accuracy is calculated on the last 100 training examples.Source: §A.2 · p. 20

A.3 Adaptivity to Concept Drift

  • After a synthetic concept drift, synchronizations are frequent and then decrease as learners’ instantaneous loss decreases.The experiment triggers drifts at random with probability 0.001 by generating a new graphical model.Source: §A.3 · p. 20

A.4 Deep Driving Experiments

Deep-driving experiments report similar driving performance to the serial baseline with substantially less communication than periodic averaging. Dynamic averaging’s advantage over periodic averaging also holds with ADAM and RMSprop.

A.4 Deep Driving Experiments

  • The simulator study records multiple drivers on a single track, while data collected across regions, times, or vehicles may have greater variability and heterogeneity.For the driving network, the front camera view is the input and the steering angle is the output.Source: §A.4 · pp. 21–22
  • Cross-region in-fleet data cannot be assumed iid; prior results suggest the approach can handle non-iid data, and analysis of this and other challenges remains future work.Source: §A.4 · p. 22
  • Dynamic averaging achieves driving performance similar to the serial baseline while requiring substantially less communication than periodic averaging.Communication is concentrated early in training and then considerably reduced, most visibly for Δ=0.1.Source: §A.4 · p. 24, Figure A.5 · p. 24

A.5 Black-Box Algorithms Evaluation

  • Dynamic averaging’s advantage over periodic averaging also holds with ADAM and RMSprop, supporting its use as a black-box synchronization protocol across learning algorithms.The comparison reports performance similar to the dynamic protocol paired with distributed mini-batch SGD.Source: §A.5 · p. 24, Figure A.6 · p. 25

A.6 Scale-out Experiments

The scale-out experiments report that dynamic synchronization can improve the loss–communication trade-off as learner count grows. The initialization study finds that averaging tolerates moderate heterogeneity but becomes sensitive to local mini-batch count at larger heterogeneity levels.

A.6 Scale-out Experiments

  • As learners saturate and no longer trigger local conditions, dynamic protocols’ advantage over periodic protocols becomes more pronounced.The experiment fixes examples per learner, so increasing the learner count also increases the training dataset.Source: §A.6 · p. 25
  • With m = 200, σ∆=0.3 requires less communication than σb=1; at m = 100, σ∆=0.7 already reaches much smaller cumulative loss.The comparison uses MNIST cumulative-loss and communication plots across m = 10, 100, and 200 learners.Source: Figure A.7 · p. 25, §A.6 · p. 26
  • In the worst case, dynamic averaging communicates as much as periodic averaging; in practice, it requires substantially less communication than FedAvg on the same learner count.The worst-case comparison is to FedAvg on a fraction C of m learners.Source: §A.6 · p. 26

A.7 Stability of the Dynamic Averaging Protocol Regarding Model Initializations

  • For initialization noise ϵ ∈ {1, 2, 3}, averaged models converge even with many local mini-batches, whereas averaging fails at ϵ = 20.The reported transition lies between ϵ = 5 and ϵ = 10, where convergence depends strongly on the number of local mini-batches.Source: Figure A.8 · p. 26, §A.7 · p. 27
  • The initialization experiment varies noise scale ϵ and local mini-batches b, but leaves learning-rate effects for future study.Noise is imposed on Xavier Glorot homogeneous initialization, and averaged-model performance is relative to ϵ = 0 and b = 1.Source: §A.7 · p. 27

B Details on Theory

The theory section defines regret relative to a reference model and establishes conditions under which dynamic averaging retains periodic averaging’s regret bound. For SGD, the argument relies on contraction, including local convexity assumptions for non-convex losses.

B Details on Theory

  • Under Theorem 5’s conditions, dynamic averaging’s average regret differs from periodic averaging by at most ϵ.The theorem requires an L-Lipschitz loss, a contraction update, b ≥ log^-1(2c^-1), and ∆ ≤ ϵ/2L.Source: B Details on Theory · pp. 28–29
  • The proof links regret to average pairwise model distances and shows that dynamic synchronization retains the distance bound of static synchronization.Regret is cumulative loss relative to a reference model, such as the best model in hindsight.Source: B Details on Theory · pp. 27–28
  • For convex losses, SGD is a contraction with a sufficiently small constant learning rate, η ≤ (ρL + λ)^-1.Here ρ is the data radius, L the loss’s Lipschitz constant, and λ the regularization parameter.Source: B Details on Theory · p. 29
  • For non-convex losses, the guarantee holds when losses are locally convex in a bounded region around the models’ span.The stated region has radius 2∆R around the dynamic-model average, with update magnitude bounded by R = ηL.Source: B Details on Theory · p. 29

C Federated Learning: Different Sampling Rates and non-iid Data

This section addresses federated-learning settings with non-iid data and unequal sampling rates. It describes weighted averaging for unequal sample counts, while noting that dynamic averaging’s behavior with non-iid data still needs further study.

C Federated Learning: Different Sampling Rates and non-iid Data

  • The theoretical analysis assumes fixed, balanced sampling and iid data, unlike the federated-learning setting described with non-iid data and unbalanced sampling.The cited federated-learning setting also includes massive distribution and limited communication infrastructure.Source: C Federated Learning: Different… · p. 30
  • Whether differences in local data distributions preserve dynamic averaging’s learning behavior remains unsubstantiated and requires further research.The discussion says this should hold, while identifying further research as necessary to substantiate it.Source: C Federated Learning: Different… · p. 30
  • For unequal sampling rates, the protocol weights each local model by its number of observed examples and uses the weighted average.The total sample count per round is N = Σ_i B_i; the method can be generalized to time-dependent sampling rates.Source: C Federated Learning: Different… · p. 31
Loading 1807.03210v2…