Source-linked AI summary
Gossip training for deep learning
Michael Blot, David Picard, Matthieu Cord, Nicolas Thome
TL;DR
Training deep convolutional networks on large datasets is slow, and distributed approaches can lose time synchronizing a central network. The paper introduces GoSGD, a gossip-based method for distributed SGD. On CIFAR-10, GoSGD reached the same training-loss score in about 4 hours that EASGD needed more than 7 hours to reach.
Problem
Deep CNNs with many parameters train slowly on large datasets, and central synchronization in distributed methods can waste computation time.
Source: §1 · p. 1Method
GoSGD names the optimization method combining gossip with stochastic gradient descent for deep architectures.
Source: §1 · p. 2Results
About 4 hours for GoSGD versus more than 7 hours for EASGD to reach the same training-loss score on CIFAR-10.
Source: §3 · p. 4Takeaways & Limitations
GoSGD is fully asynchronous and decentralized, avoiding idling while using pairwise exchanges over the faster CPI communication channel.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information between different threads inspired by gossip algorithms and showing good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized. We compared our method to the recent EASGD in \cite{elastic} on CIFAR-10 show encouraging results.
1 Introduction
Deep CNNs are slow to train on large datasets, while existing distributed approaches rely on central synchronization. GoSGD adapts gossip-style peer-to-peer information sharing to SGD for CNN training.
1 Introduction
- Deep CNNs have many parameters and train slowly on large datasets, limiting how many architectures can be tested.GPU computation accelerates training but does not remove this difficulty.Source: §1 · p. 1
- Existing distributed SGD methods exchange information through a central network, whose synchronization can waste distributed computation time.Consensus matters because neural-network workers can produce very different optimizations.Source: §1 · p. 1
- GoSGD combines SGD with gossip-style peer-to-peer sharing, yielding a fully asynchronous, decentralized method for deep learning.Gossip averaging is described as rapidly bringing agents toward consensus without a central variable.Source: §1 · pp. 1–2
2 Gossip Stochastic Gradient Descent
GoSGD distributes CNN training across workers that alternate local SGD updates with asynchronous gossip exchanges. Its test model averages worker weights, making worker consensus important to the gradient approximation.
2.1 GoSGD algorithm
- Workers alternate mini-batch SGD updates with push-only, sum-weight gossip exchanges, using a Bernoulli probability p to decide whether to share weights.A worker may send at most once per iteration but process multiple incoming weights sequentially before its next gradient update.Source: §2.1 · p. 2
- The gossip updates retain stochastic-vector weights and converge toward consensus exponentially fast in mixing updates when no gradient update occurs.The exchange probability p is the algorithm’s only adjustable parameter; p=0.01 already ensures very good consensus in the experiments.Source: §2.1 · p. 3
2.2 Test model
- The test model is the average of all workers’ weights, and its representation uses stochastic-vector coefficients with no null value.Keeping worker weights close to this consensus is needed for each worker’s mini-batch gradient estimate to align with the true gradient; averaging close workers’ gradients can improve the estimate using M times the number of draws.Source: §2.2 · p. 3
3 Experiments
The CIFAR-10 experiments compare GoSGD with MEASGD and a no-exchange baseline. With p=0.02, GoSGD reaches the same training loss in about 4 hours that EASGD reaches in more than 7 hours.
3 Experiments
- The CIFAR-10 comparison uses eight workers, batches of 128 images, a constant learning rate of 0.01, and weight decay of 10^-4.The EASGD baseline uses momentum 0.99 and α=0.887; p controls GoSGD exchange frequency.Source: §3 · p. 4
- At p=1, GoSGD makes better use of exchanges than EASGD when comparing loss against images processed per worker.The authors suggest this may indicate better consensus during training.Source: §3 · p. 4
- About 4 hours versus more than 7 hours: with p=0.02, GoSGD reaches the same training loss faster than EASGD.The comparison uses training loss plotted against elapsed time.Source: §3 · p. 4
4 Discussion
The discussion presents GoSGD as a fully asynchronous, decentralized scheme with pairwise exchanges, and identifies consensus-rate analysis as a possible extension.
4 Discussion
- GoSGD is fully asynchronous and decentralized, avoiding idling, and uses pairwise exchanges that benefit from the faster communication channel CPI.The authors experimentally validated the approach.Source: §4 · p. 4
- The authors identify deriving a consensus convergence rate for GoSGD as a possible extension of analyses available for many gossip algorithms.They propose examining how additional gradients affect gossip averaging and consensus.Source: §4 · p. 4
- The proposed future analysis could inform exchange-frequency optimization while limiting its impact on consensus between threads.The discussion frames this as a direction for further study.Source: §4 · p. 4