PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Use Large Mini-Batches, Use Local SGD

Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, Martin Jaggi

arXiv:1808.07217v6cs.LGstat.ML

TL;DR

Very large mini-batches improve distributed training efficiency but can harm generalization. This paper proposes post-local SGD and finds that it matches small-batch generalization while improving communication efficiency over large-batch alternatives.

  • Problem

    Very large mini-batch SGD can improve throughput but has limited generalization performance on unseen data.

  • Method

    The paper proposes post-local SGD and empirically studies communication-efficiency and generalization trade-offs across local SGD variants.

  • Results

    Post-local SGD outperforms large-batch SGD in generalization, matches small-batch SGD, and is more communication-efficient than mini-batch competitors.

  • Takeaways & Limitations

    Local SGD provides a simpler, more efficient alternative to large-batch training that improves generalization while retaining distributed-training benefits.

  • Takeaways & Limitations

    A theoretical study of local SGD generalization is beyond the scope of this work because large-batch generalization remains poorly understood theoretically.

Abstract

from arXiv · show

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a \emph{post-local} SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of \emph{local SGD} variants.

1 INTRODUCTION

Distributed deep-learning training requires communication-efficient parallel optimization without sacrificing generalization to unseen data. The paper therefore studies local SGD and proposes post-local SGD to address the generalization gap of large-batch training while retaining scalable efficiency.

  • Local SGD: Local SGD lets each worker perform H sequential mini-batch updates before communication, balancing computation against communication while changing the update dynamics relative to mini-batch SGD.When B = HBloc, local and mini-batch SGD have identical communication patterns and evaluate the same number of training examples, but their updates differ for H > 1.
  • Main Results: Figure 1 shows that post-local SGD matches the single-machine baseline’s test accuracy, while large-batch SGD matches training curves but generalizes worse.The comparison concerns A5 versus A1 and A2, respectively.
  • Motivation: Large-batch SGD increases throughput but very large overall batches are known to cause drastically decreased generalization performance.Reducing batch size can alleviate this issue but undermines training efficiency while maintaining parallelization.
  • Main Results: Post-local SGD starts local SGD from a model produced by large-batch SGD, closing the generalization gap and improving communication efficiency over mini-batch competitors.It remains less communication-efficient than local SGD in the direct comparison.
  • Contributions: The paper provides a comprehensive empirical study of how workers, local steps, and mini-batch sizes trade off communication efficiency and generalization.The study covers both communication efficiency and generalization across the two motivating scenarios.

2 RELATED WORK

Prior work links synchronized large-batch SGD to degraded training and test performance and relates this gap to sharper minima and generalization. Local SGD theory remains less developed, with studies examining convergence, worker scaling, delays, and how averaging frequency affects solution quality.

  • The generalization gap in large-batch training: Synchronized large-batch SGD enables distributed scaling but typically degrades training and test error, including for ImageNet batches above 10^3.
  • The generalization gap in large-batch training: Keskar et al. (2017) associate larger batches with sharper minima, whereas flat minima are preferred for better generalization; this interpretation remains debated by Dinh et al. (2017).
  • Local SGD and convergence theory: Local SGD has less-developed theoretical foundations than mini-batch SGD, with prior work studying one-shot averaging, general worker graphs, and whether K workers yield linear speedup.
  • Local SGD and convergence theory: Prior analyses cover local SGD convergence for strongly convex and smooth objectives, smooth non-convex objectives, stochastic-gradient decrement, and adversarial delays.
  • Local SGD and convergence theory: Empirical and theoretical studies suggest that more frequent averaging at the beginning of optimization can improve solution quality.

3 POST-LOCAL SGD AND HIERARCHICAL LOCAL SGD

This section introduces post-local SGD, which delays local updates until a second training phase, and hierarchical local SGD, which adapts communication and computation across system hierarchy levels.

  • Post-local SGD: Post-local SGD starts local SGD only after t1 initial mini-batch SGD steps, preserving the first phase while enabling communication-efficient training later.Its iteration-dependent schedule uses Hptq = 1 for t ≤ t1 and H for t > t1.
  • Post-local SGD: Post-local SGD uses small local mini-batches during warm-up and switches to effective batches of size HBloc, while reusing tuned large-batch learning-rate schedules without additional tuning.The first phase is identical to mini-batch SGD with B = Bloc.
  • Hierarchical Local SGD: Hierarchical local SGD runs local SGD as an inner loop at each hierarchy level, adapting computation and communication to heterogeneous bandwidths across chips, machines, racks, and data centers.The design targets optimal resource adaptivity across different system levels.

4 EXPERIMENTAL RESULTS

Experiments on CIFAR-10/100 and ImageNet show that local SGD improves communication efficiency and scalability over mini-batch SGD, while post-local SGD addresses large-batch generalization and can match or exceed small-batch performance. These findings hold across architectures, worker counts, batch sizes, and additional communication-compression or optimizer settings.

  • Scenario 1: At 92.48% mini-batch accuracy, local SGD can achieve 91.2% accuracy with a 2.59ˆ time-to-accuracy speedup when H=8.This comparison uses fixed Bloc and K=16, and the 91.2% result matches He et al. (2016a).
  • Scenario 1: Local SGD scales 2ˆ better than mini-batch SGD in time-to-accuracy as worker count increases, overcoming communication bottlenecks while retaining better generalization.Figure 1 varies K and H from 1 to 16 on CIFAR-10 with ResNet-20.
  • Scenario 1: Local SGD reaches 75% ImageNet top-1 accuracy with at least 1.5ˆ speedup on a 16 ˆ 2-GPU cluster using ResNet-50.The result demonstrates scalability to larger datasets and clusters while remaining competitive with large-batch ImageNet methods.
  • Scenario 2: Post-local SGD generalizes better and faster than large-batch mini-batch SGD across CIFAR architectures, while achieving the same or better performance than small mini-batch baselines.The study evaluates large batches with H=16 and H=32 across multiple architectures, with supporting results in Table 3, Table 5, Figure 3, and Appendix C.5.
  • Scenario 2: Post-local SGD remains effective across H, K, and larger batches, improves compressed communication methods, and outperforms LARS while avoiding a reported 2% large-batch generalization drop.The experiments cover CIFAR-100, ImageNet, language modeling, compression schemes, and batches up to KBloc=8192.

5 DISCUSSION AND INTERPRETATION

The discussion interprets local SGD as computation-free, controllable stochastic-noise injection that can improve large-batch generalization, while noting that the theoretical explanation remains incomplete. Evidence suggests post-local SGD favors lower-curvature solutions than large-batch SGD, consistent with better generalization.

  • Scope and limitations: The paper does not provide a profound theoretical study of local-SGD generalization, instead presenting the favorable experiments as motivation for future research.The authors frame the discussion as an interpretation of local SGD through stochastic-noise injection rather than a complete theory.
  • Implementation: Post-local SGD can be integrated with LARS without extra modification or parameter synchronization because LARS already uses layer-wise learning rates and distributed training.The implementation uses the NVIDIA Apex LARS code path for mixed precision and distributed PyTorch training.
  • Connecting Local Updates with Stochastic Noise Injection: The usual noise-ratio explanation breaks down for excessively large batches, where noise decreases relative to gradient effects and ρ no longer uniquely determines training dynamics.This breakdown is especially relevant for large batches and small datasets, helping explain persistent generalization difficulties.
  • Connecting Local Updates with Stochastic Noise Injection: Local updates inject structured stochastic noise without extra computation, increasing noise through K and H while using the same learning-rate-to-batch-size ratio as mini-batch SGD.During local updates, the effective noise variance is approximately KΣ(w), compared with Σ(w) for mini-batch SGD.
  • Curvature and generalization: Post-local SGD tends toward low-curvature minima with better generalization, whereas large-batch SGD tends toward high Hessian-spectrum solutions; interpolation and sharpness analyses support this pattern.Figure 4 evaluates Hessian spectra and interpolates minima from mini-batch and post-local SGD; Appendix Figure 13 visualizes model sharpness.

6 CONCLUSION

The paper extensively studies the trade-off between communication efficiency and generalization in local SGD, introducing post-local SGD as a variant that outperforms large-batch SGD and matches small-batch SGD generalization while converging to flatter minima.

  • 6 CONCLUSION: Post-local SGD outperforms large-batch SGD and matches small-batch SGD in generalization, while converging to flatter minima than traditional large-batch SGD.The work presents the first extensive study of communication-efficiency versus generalization-performance trade-offs for local SGD in distributed and heterogeneous environments.

Supplementary Material … A.4.2 IMAGENET

The supplementary appendix specifies datasets, architectures, large-batch learning schemes, and training procedures used to evaluate local and post-local SGD across CIFAR, WikiText-2, and ImageNet. The ImageNet protocol fixes a 90-epoch budget, equal data access across methods, and Goyal et al. (2017)-style learning-rate scaling and warmup.

  • A.1 DATASET: The appendix evaluates image classification on CIFAR-10/100 and ImageNet, using standard augmentation and channel-wise normalization for CIFAR data.
  • A.2 MODELS AND MODEL INITIALIZATION: Experiments cover ResNet-20, DenseNet-40-12, WideResNet-28-10, LSTM, and ResNet-50 to assess post-local SGD across vision and language tasks.
  • A.2 MODELS AND MODEL INITIALIZATION: The scaling ratio measures computation relative to communication, so local SGD is expected to offer greater advantages for models with smaller ratios.Computation is proportional to input-processing floating-point operations, whereas communication is proportional to model size or parameter count.
  • A.3 LARGE BATCH LEARNING SCHEMES: Large-batch learning follows Goyal et al. (2017) by linearly scaling the learning rate with batch size and gradually warming it up for five epochs.
  • A.4.1 CIFAR-10/CIFAR-100: CIFAR experiments give competing methods equal sample exposure, terminate at matched standalone-worker sample counts, and use architecture-specific epoch budgets.ResNet-20, DenseNet-40-12, and WideResNet-28-10 access 300, 300, and 250 epochs, respectively.
  • A.4.1 CIFAR-10/CIFAR-100: CIFAR optimization uses Nesterov momentum of 0.9, architecture-specific weight decay, and independently computed batch-normalization statistics for distributed workers.Local SGD otherwise uses the exact same optimization scheme as mini-batch SGD.
  • A.4 HYPERPARAMETER CHOICES AND TRAINING PROCEDURE, OVER DIFFERENT MODELS/DATASETS: Learning rates are grid-searched for each CIFAR batch size because no universal scaling rule works across batch sizes, tasks, and architectures, following Shallue et al. (2018).Large-batch SGD is warmed up for the first five epochs in both default-scheme and tuned-learning-rate experiments.
  • A.4.2 IMAGENET: ImageNet ResNet-50 training uses 90 data passes, global reshuffling each epoch, equal gradient access across methods, local batches of 128, and five-epoch warmup.Learning rates are scaled from the 0.1 base rate and 256-example base batch, then decayed by 10 at 30, 60, and 80 epochs.

A.5 SYSTEM PERFORMANCE EVALUATION … B.3 MORE RESULTS ON LOCAL SGD TRAINING

The evaluation shows that larger mini-batches improve single-GPU parallelism, while local SGD can substantially reduce convergence time on a convex problem. The algorithm performs local updates on each node before periodically aggregating gradients and synchronizing models.

  • A.5 SYSTEM PERFORMANCE EVALUATION: Figure 5 examines how the latency of transmitting data among CPU cores changes with the number of cores.It measures 100 MB all-reduce operations using PyTorch MPI on a 10 Gbps Kubernetes cluster with 48 cores per physical machine.
  • A.5 SYSTEM PERFORMANCE EVALUATION: Larger mini-batches provide better GPU parallelism when running forward and backward passes for ResNet20 on CIFAR-10.Table 7 evaluates the time for 4096 samples across different mini-batch sizes on a single GPU.
  • B.1 FORMAL DEFINITION OF THE LOCAL SGD ALGORITHM: Local SGD initializes synchronized models, performs H local mini-batch updates in parallel across K nodes, and periodically aggregates gradients.The procedure takes an initial model, training data, local batch size Bloc, step size η, optional momentum m, synchronization steps T, and local steps H.
  • B.2 NUMERICAL ILLUSTRATION OF LOCAL SGD ON A CONVEX PROBLEM: The convex illustration studies logistic regression on the w8a dataset and measures iterations until reaching target accuracy ϵ=0.005.Configurations of K, H, and Bloc use learning rates selected by grid search, with time counted through stochastic-gradient computations and communication rounds.
  • B.2 NUMERICAL ILLUSTRATION OF LOCAL SGD ON A CONVEX PROBLEM: More than 2× faster convergence with local SGD using (Bloc, H)=(16,16) than (64,1), and 3× faster than (256,1) for K=16.The comparison comes from the convex logistic-regression illustration in Figure 6(a).
  • B.2 NUMERICAL ILLUSTRATION OF LOCAL SGD ON A CONVEX PROBLEM: Local SGD achieves its best speedup with H=16 on a small number of workers, but its advantage diminishes as K becomes very large.This scaling behavior is shown in Figure 6(b).

B.3.1 TRAINING CIFAR-10 VIA LOCAL SGD … C.5 POST-LOCAL SGD TRAINING ON DIVERSE TASKS

Across CIFAR-10, ImageNet, and larger-batch settings, local SGD reduces communication while preserving or improving training and generalization, whereas post-local SGD injects noise after learning-rate decay to reach flatter, better-generalizing solutions. Practical studies show that momentum and warm-up variants offer limited or unclear benefits, while post-local SGD’s timing is important for efficiency and stability.

  • B.3.1 TRAINING CIFAR-10 VIA LOCAL SGD: On CIFAR-10, local SGD is more communication-efficient than mini-batch SGD while maintaining the same test accuracy and converging faster.With local mini-batch size Bloc = 128, all methods use the same total number of gradient computations.
  • B.3.1 TRAINING CIFAR-10 VIA LOCAL SGD: With H > 1, local SGD performs H times fewer global synchronizations while accessing the same number of samples, and its total-training-time advantage grows with H.Mini-batch SGD is the H = 1 special case with synchronization after every local update.
  • B.3.2 TRAINING IMAGENET VIA LOCAL SGD: For ImageNet ResNet-50, local SGD is evaluated over 90 data passes with H = 8, using a warm-up that doubles local steps until reaching the target.The local-step warm-up follows the theoretical initial-training assumption.
  • B.3.3 LOCAL SGD SCALES TO LARGER BATCH SIZES THAN MINI-BATCH SGD: Local SGD scales beyond the large-batch limit: with H = 2 on CIFAR-10, it trains and generalizes better in update steps while reducing communication cost.Prior empirical studies identify a regime where additional large-batch parallelism provides no benefit.
  • B.4.1 LOCAL SGD WITH MOMENTUM: Momentum variants remain difficult to tune theoretically, while CIFAR-10 experiments find only slight accuracy improvements from some global-momentum factors and no convincing benefit from local-step warm-up.The authors therefore consider local momentum and report that warm-up noise can degrade training quality even after stabilization.
  • C.1 THE ALGORITHM OF POST-LOCAL SGD: Post-local SGD uses frequent synchronization before the first learning-rate decay and reduced communication afterward, with eventual local steps H1 in its algorithmic schedule.The algorithm sets H(t) = 1 before the first decay and H(t) = H1 after it.
  • C.2 THE EFFECTIVENESS OF TURNING ON POST-LOCAL SGD AFTER THE FIRST LEARNING RATE DECAY: Post-local SGD should begin after the first learning-rate decay: delaying it to the second decay increases communication cost and risks convergence to sharper minima.The study examines both the sufficiency and necessity of injecting additional stochastic noise at that point.
  • C.4 UNDERSTANDING THE GENERALIZATION OF POST-LOCAL SGD: Post-local SGD reaches lower-curvature, flatter solutions with better generalization than large-batch or mini-batch SGD, including when trained from scratch or resumed from a checkpoint.The Hessian spectrum and one-dimensional interpolation analyses associate post-local SGD with low-curvature and flatter minima.

C.5.1 POST-LOCAL SGD TRAINING ON CIFAR-100 FOR GLOBAL MINI-BATCH SIZE KBLOC “4096

On CIFAR-100 with global mini-batch size 4096, fine-tuned large-batch SGD suffers severe quality loss, whereas post-local SGD closes or surpasses the small-batch generalization baseline. Longer training also alleviates large-batch optimization difficulty for ResNet-20.

  • Post-local SGD training on CIFAR-100: Post-local SGD closes the roughly 2% generalization gap of fine-tuned large-batch SGD or surpasses fine-tuned small-mini-batch baselines across three CIFAR-100 CNNs.The comparison uses default hyperparameters drawn from small-mini-batch and large-batch training schemes.
  • Post-local SGD training on CIFAR-100: Increasing ResNet-20 training from 300 to 400 or 500 epochs alleviates the optimization difficulty of large-batch training on CIFAR-100.This extends the analysis of Hoffer et al. (2017) and Shallue et al. (2018).

C.5.2 POST-LOCAL SGD TRAINING ON LANGUAGE MODELING · C.5.3 POST-LOCAL SGD TRAINING ON IMAGENET

Post-local SGD is evaluated as a large-batch training strategy for WikiText-2 language modeling and ImageNet classification. It provides a proof-of-concept for LSTM training and outperforms mini-batch SGD on ImageNet at batch sizes 4096 and 8192.

  • C.5.2 POST-LOCAL SGD TRAINING ON LANGUAGE MODELING: The language-modeling experiment uses a three-layer LSTM with hidden dimension 650, B_loc=64, 120 epochs, BPTT length 30, gradient clipping 0.4, and output dropout 0.4.Loss is averaged over examples and timesteps, and the learning rate decays after the algorithm accesses 50% and 75% of training samples.
  • C.5.2 POST-LOCAL SGD TRAINING ON LANGUAGE MODELING: On WikiText-2, the authors provide a proof-of-concept evaluation showing post-local SGD’s effectiveness for large-batch LSTM language-model training.The reported metric is validation perplexity, where lower values are better; Table 13 averages three runs and fine-tunes mini-batch SGD learning rates.
  • C.5.2 POST-LOCAL SGD TRAINING ON LANGUAGE MODELING: The WikiText-2 setup adapts Merity et al. (2018) and addresses the difficulty of scaling LSTM training because of its multiple hyperparameters.The authors note that most prior large-batch work focuses on computer vision rather than language modeling.
  • C.5.3 POST-LOCAL SGD TRAINING ON IMAGENET: The ImageNet evaluation uses standard ResNet-50 training configurations and measures post-local SGD after its transition at the first learning-rate decay.The experiments evaluate challenging ImageNet training under a 90-pass limit over the data.
  • C.5.3 POST-LOCAL SGD TRAINING ON IMAGENET: At batch sizes 4096 and 8192, post-local SGD achieves higher ImageNet accuracy than mini-batch SGD: 76.18 versus 75.87 and 75.65 versus 75.64, respectively.ResNet-50 training is limited to 90 passes, with post-local SGD beginning at the first learning-rate decay.
  • C.5.3 POST-LOCAL SGD TRAINING ON IMAGENET: ImageNet performance is evaluated after each complete access to the training data on an 8 × 4-GPU V100 Kubernetes cluster with 10 Gbps bandwidth.These experiments use a resource-limited platform rather than the paper’s main experimental platform.

C.5.4 POST-LOCAL SGD VS. OTHER NOISE INJECTION METHODS … D.2 THE ALGORITHM OF HIERARCHICAL LOCAL SGD

Post-local SGD outperforms isotropic noise injection and supports compressed training while retaining or improving generalization and communication efficiency. Hierarchical local SGD extends this strategy across heterogeneous communication levels by combining local, block, and global synchronization.

  • C.5.4 POST-LOCAL SGD VS. OTHER NOISE INJECTION METHODS: Against Wang & Joshi (2019), local SGD reaches better accuracy with less communication on the same CIFAR-10 task and K=4.Their evaluation does not cover the difficult large-batch scenario, whereas the comparison uses a smaller ResNet-20.
  • C.5.5 POST-LOCAL SGD WITH OTHER COMPRESSION SCHEMES: Post-local SGD integrates with sign-based compression, using local sign updates followed by synchronization, and significantly improves generalization while further reducing communication.The method adapts signSGD by averaging signs rather than using majority voting.
  • C.5.5 POST-LOCAL SGD WITH OTHER COMPRESSION SCHEMES: Distributed EF-signSGD with post-local SGD achieves performance similar to mini-batch SGD after hyper-parameter tuning under the same experimental setup.The extension adapts single-worker EF-signSGD to multiple workers and investigates alternative algorithmic designs.
  • C.5.5 POST-LOCAL SGD WITH OTHER COMPRESSION SCHEMES: Weight decay and Nesterov momentum significantly improve training and test performance for both compressed post-local SGD algorithms under a fixed epoch budget.These techniques are used for local model updates in both signSGD and EF-signSGD experiments.
  • D HIERARCHICAL LOCAL SGD: Hierarchical local SGD targets decentralized heterogeneous systems with multiple communication bandwidths by applying local SGD at each hierarchy level for adaptivity and performance.The motivating architecture includes fast within-node links and slower server, rack, cluster, and external-network connections.
  • D.1 THE ILLUSTRATION OF HIERARCHICAL LOCAL SGD: Hierarchical local SGD performs H local updates on each GPU, synchronizes within GPU-blocks, then performs global synchronization after Hb block updates.This structure matches computation–communication trade-offs across hierarchical systems.
  • D.2 THE ALGORITHM OF HIERARCHICAL LOCAL SGD: The hierarchical algorithm implements nested local-update, inner all-reduce, block-synchronization, outer all-reduce, and global-synchronization steps.Its inputs include local steps H, block steps Hb, total nodes K, and nodes K1 per GPU-block.

D.3 HIERARCHICAL LOCAL SGD TRAINING · D.3.1 THE PERFORMANCE OF HIERARCHICAL LOCAL SGD. · E COMMUNICATION SCHEMES

Hierarchical local SGD reduces communication costs in heterogeneous multi-server systems while improving scalability, test accuracy, and robustness to network delays. Its communication analysis shows that block steps are more decisive than local steps for reducing expensive cross-server synchronization.

  • D.3 HIERARCHICAL LOCAL SGD TRAINING: The experiments model heterogeneous systems by grouping GPUs across servers whose limited bandwidth constrains communication of large-model updates.They train ResNet-20 on CIFAR-10 using a procedure aligned with local SGD.
  • D.3.1 THE PERFORMANCE OF HIERARCHICAL LOCAL SGD.: Increasing local steps cannot indefinitely improve communication performance, whereas inner-node synchronization reduces heterogeneous synchronization difficulty and benefits larger clusters.The training-time study uses ResNet-20 on CIFAR-10 across eight nodes with two GPUs each.
  • D.3.1 THE PERFORMANCE OF HIERARCHICAL LOCAL SGD.: Increasing block steps reduces communication cost with trivial performance degradation, and sufficient block steps make hierarchical local SGD robust to network delays.This robustness is observed even in a two-server, four-GPU experiment with fixed H = 2.
  • D.3.1 THE PERFORMANCE OF HIERARCHICAL LOCAL SGD.: Hierarchical local SGD improves scaling and test accuracy over mini-batch SGD across heterogeneous cluster topologies when H·H_b = 16.The comparison uses the same training procedure and reports better generalization with sufficient block size and block updates.
  • E COMMUNICATION SCHEMES: The communication model assumes K devices uniformly distributed across K_1 servers, with local batch size B, H local steps, and H_b block steps over N samples.These parameters define the hierarchical local SGD training procedure analyzed in the communication section.
  • E COMMUNICATION SCHEMES: MPI all-reduce communication costs C·log_2 K per aggregation, where C combines message transmission time and network latency.The scheme uses general all-reduce algorithms, including recursive halving and doubling.
  • E COMMUNICATION SCHEMES: Hierarchical local SGD synchronizes within servers every H mini-batches but globally only after H_b block updates, making H_b more deterministic for communication reduction than H.The intra-server and cross-server costs differ because C_1, the within-server cost, is lower than C_2, the cross-server cost.
  • E COMMUNICATION SCHEMES: The hierarchical scheme is orthogonal to hardware-optimized gradient aggregation and can integrate with optimized all-reduce implementations.It targets aggregation costs in general distributed scenarios rather than replacing the underlying aggregation implementation (Goyal et al., 2017).

F DISCUSSION AND FUTURE WORK

The discussion identifies open questions around data locality, learning-rate scheduling, adaptive local-step selection, and topology-aware hierarchical local SGD. These issues concern theoretical gaps, untuned optimization choices, and system-dependent performance.

  • Data distribution patterns: Data-local training without global shuffling, relevant to extremely large or privacy-sensitive federated datasets, is not covered by current theoretical understanding.The experiments globally shuffle once per epoch and assign each worker a disjoint data portion.
  • Better learning rate scheduler for local SGD: Local SGD outperforms mini-batch SGD consistently, but existing large-batch learning-rate schemes were developed for mini-batch SGD rather than local SGD.For ImageNet, the experiments reused Goyal et al. (2017)'s large-batch configuration without local-SGD-specific tuning.
  • Adaptive local SGD: Adaptive choices of the local-step count H could potentially eliminate or simplify complex learning-rate schedules.The discussion also points to cyclic learning-rate schedules as a related direction.
  • Hierarchical local SGD design: Hierarchical local SGD is efficient for heterogeneous systems, but topology affects its gains: an 8 × 2-GPU configuration did not improve with more frequent inner-node synchronization.Sufficiently large GPU blocks can benefit from hierarchical block updates.
Loading 1808.07217v6…