PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Sameer Bibikar, Haris Vikalo, Zhangyang Wang, Xiaohan Chen

arXiv:2112.09824v1cs.LGcs.AIstat.ML

TL;DR

Resource-limited devices and bandwidth-constrained links make federated learning costly in both local computation and communication. FedDST trains and transmits sparse subnetworks dynamically; in non-iid CIFAR-10 experiments, it reports up to 3% better accuracy than FedAvgM with half the upload bandwidth. The paper's scope is element-wise unstructured sparsity.

  • Problem

    Federated learning must address the computational burden of complex models on resource-limited edge devices and the communication cost of updates over bandwidth-limited links.

    Source: Introduction · p. 1

  • Method

    FedDST dynamically trains sparse subnetworks on clients and transmits them instead of full models to reduce local computation and communication.

    Source: Introduction · pp. 1–2

  • Results

    Up to 3% better accuracy than FedAvgM with half the upload bandwidth is reported on non-iid CIFAR-10.

    Source: Conclusion and Broader Impacts · p. 7

  • Takeaways & Limitations

    FedDST's focus on communication efficiency and performance makes more federated-learning settings practical in the wild.

    Source: Conclusion and Broader Impacts · p. 7

    FedDST considers only element-wise unstructured sparsity, with practical local-training speedups and structured sparsity left for future work.

    Source: Communication and Local Training Savings · p. 5

Abstract

from arXiv · show

Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. Unfortunately, current deep networks remain not only too compute-heavy for inference and training on edge devices, but also too large for communicating updates over bandwidth-constrained networks. In this paper, we develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) by which complex neural networks can be deployed and trained with substantially improved efficiency in both on-device computation and in-network communication. At the core of FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network. With this scheme, "two birds are killed with one stone:" instead of full models, each client performs efficient training of its own sparse networks, and only sparse networks are transmitted between devices and the cloud. Furthermore, our results reveal that the dynamic sparsity during FL training more flexibly accommodates local heterogeneity in FL agents than the fixed, shared sparse masks. Moreover, dynamic sparsity naturally introduces an "in-time self-ensembling effect" into the training dynamics and improves the FL performance even over dense training. In a realistic and challenging non i.i.d. FL setting, FedDST consistently outperforms competing algorithms in our experiments: for instance, at any fixed upload data cap on non-iid CIFAR-10, it gains an impressive accuracy advantage of 10% over FedAvgM when given the same upload data cap; the accuracy gap remains 3% even when FedAvgM is given 2x the upload data cap, further demonstrating efficacy of FedDST. Code is available at: https://github.com/bibikar/feddst.

Introduction

Federated learning must make both local training and communication feasible on resource-limited devices, while keeping user data local. FedDST applies dynamic sparse training to federated learning, and the paper reports communication and performance gains in challenging non-iid settings.

Introduction

  • Edge FL must reduce both local computation and communication because complex models strain devices and client upload bandwidth is limited.Prior communication-efficient FL work has focused on update sparsity, client sampling, and other methods.Source: Introduction · p. 1
  • FedDST trains clients on sparse matching subnetworks and transmits those instead of full models, targeting computation and communication costs together.The framework is built around federated dynamic sparse training.Source: Introduction · pp. 1–2
  • Dynamic sparsity accommodates local heterogeneity more robustly than fixed shared masks and introduces an in-time self-ensembling effect associated with improved performance over dense training.The paper presents these as findings about dynamic sparsity during federated training.Source: Introduction · p. 2
  • On pathologically non-iid CIFAR-10, FedDST improves accuracy by 3% over FedAvgM (Hsu, Qi, and Brown 2019) while requiring only half the upload bandwidth.Source: Introduction · p. 2

Related Work

The related work spans federated optimization, neural-network pruning, dynamic sparse training, and pruning within federated learning. It positions FedDST against methods that use fixed or client-specific masks and explains its sparse-training and communication choices.

Federated Learning

  • FedAvg-style federated learning trains clients locally in communication rounds and aggregates their updates at a server, while FedOpt permits different client and server optimizers.Clients keep training data local; real-world FL also faces non-iid data, client heterogeneity, and limited edge resources.Source: Federated Learning · p. 2
  • Communication-efficient FL distinguishes sketched updates, compressed only during communication, from structured updates trained directly in a compressed representation.The passage notes relatively little prior work on pruning or weight readjustment throughout the full FL process.Source: Federated Learning · p. 2

Network Pruning

  • Conventional pruning removes connections from trained dense networks before fine-tuning, while single-shot methods select sparse networks at initialization using sensitivity or gradient-flow criteria.The cited pruning criteria also include magnitude, activation, and Taylor-based approaches.Source: Network Pruning · p. 2

Dynamic Sparse Training

  • Dynamic sparse training changes the selected subnetwork during training while keeping parameter count constant; SET and RigL use pruning and growth, and DST can yield temporal self-ensembling.The passages describe layer-wise sparsity distributions, including ERK, and connect parameter exploration to performance advantages over dense networks.Source: Dynamic Sparse Training · pp. 2–3

Pruning in Federated Learning

  • PruneFL adjusts masks using aggregated gradients and layer-time estimates, whereas LotteryFL maintains client-specific subnetworks and can reset pruned weights.PruneFL's estimates measure the time cost of retaining connections; LotteryFL prunes after a local validation threshold is met.Source: Pruning in Federated Learning · p. 3
  • FedDST maintains a global sparse model that changes over time, readjusts masks on clients and server, and avoids dense model or gradient transmissions and dense initial training.The authors contrast this with LotteryFL's local sparse models and PruneFL's periodic dense-gradient uploads.Source: Pruning in Federated Learning · p. 3
  • FedDST's algorithm takes client datasets, per-layer sparsities, and an update schedule as inputs, then initializes a server model before client sampling and transmission.The supplied algorithm passages also show local minibatch training and scheduled mask updates before updated networks are returned.Source: Pruning in Federated Learning · p. 3
  • Experiments report a generalization advantage over PruneFL, while fixed sparsity and infrequent mask transmission keep FedDST updates low-bandwidth.PruneFL sends dense gradients on mask-adjustment rounds, and LotteryFL sends dense models more frequently when accuracy thresholds are not met.Source: Pruning in Federated Learning · p. 3

Methodology

FedDST trains and communicates sparse subnetworks, with clients dynamically readjusting masks during federated training. The paper attributes performance gains, especially in highly non-i.i.d. settings, to a spatial-temporal ensembling effect.

FedDST: Overview of the General Framework

  • Clients train sparse networks and periodically prune and regrow weights, reallocating a decaying proportion α_r; the server aggregates client networks and masks for the next round.The reallocation rate follows a cosine-decay schedule, and α controls the tradeoff between mask-space exploration and client agreement.Source: FedDST: Overview of the General… · p. 4
  • Sparse weighted averaging ignores clients that pruned a parameter and uses weight magnitudes and client votes to address aggregation under heterogeneous data distributions.The method is inspired by FedAvg’s dataset-size-weighted averaging.Source: FedDST: Overview of the General… · p. 4
  • FedDST reports a spatial-temporal ensembling effect and performance improvements, particularly in highly non-i.i.d. settings.The proposed effect spans client-specific subnetworks and mask exploration over training time.Source: FedDST: Overview of the General… · p. 4

Communication and Local Training Savings

  • 5× lower upload cost than PruneFL at S = 0.8 during mask-readjustment rounds; FedDST caps any client's upload at (32(1 − S) + 1)n bits.The stated comparison concerns upload cost during mask-readjustment rounds.Source: Communication and Local Training Savings · p. 5
  • At 80% sparsity, CIFAR-10 forward computation requires 0.8 MFLOPs, versus 4.6 MFLOPs for the dense network.FedDST maintains sparse networks throughout training and skips FLOPs in both training and inference.Source: Communication and Local Training Savings · p. 5
  • FedDST considers element-wise unstructured sparsity; optimizing practical local-training speedups on FL hardware and incorporating structured sparsity remain future work.The paper notes that irregular access has traditionally made unstructured sparsity less translatable into hardware benefits.Source: Communication and Local Training Savings · p. 5

Experiments

In the non-iid experiments, FedDST performs best across tested upload limits and halves upload cost relative to FedAvgM. Additional experiments examine its compatibility with FedProx and its robustness to α and sparsity choices.

Experiments

  • Experiments use MNIST and CIFAR-10 with pathologically non-iid client data, and CIFAR-100 samples distributed across 400 clients using Dirichlet(0.1).For MNIST and CIFAR-10, each of 400 clients receives two classes and 20 training images per class.Source: Experiments · pp. 5–6

Main Results

  • Half the upload cost of FedAvgM yields the best performance at every tested upload limit; FedDST also significantly improves accuracy over the strong RandomMask baseline.The experiments report best test accuracy before each upload limit, averaged across 10 runs.Source: Main Results · p. 6
  • FedDST added to FedProx consistently improves performance across datasets, supporting its compatibility with other FL frameworks.The experiment adds FedProx’s proximal term and adjusts FedDST’s growth criterion.Source: Main Results · p. 6, Figure 1 · p. 6

Insensitivity to Variations in α

Performance at Different Sparsity Levels

  • FedDST is robust to sparsity variations, performs best at relatively high sparsities, and remains reasonable at untuned levels while converging faster than FedAvgM, especially early in training.The experiments therefore use sparsity 0.5 or 0.8.Source: Performance at Different Sparsity Levels · p. 7

Conclusion and Broader Impacts

FedDST is presented as communication-efficient federated learning for pathologically non-iid data. On non-iid CIFAR-10, it achieves up to 3% better accuracy than FedAvgM with half the upload bandwidth.

Conclusion and Broader Impacts

  • Up to 3% better accuracy than FedAvgM with half the upload bandwidth is reported on non-iid CIFAR-10.Source: Conclusion and Broader Impacts · p. 7
  • The conclusion reports robustness to sparsity variations and describes the results as promising for sparsity in difficult federated-learning settings.Source: Figure 3 · p. 7, Conclusion and Broader Impacts · p. 7
  • FedDST’s communication efficiency and performance are described as making more federated-learning settings practical in the wild, including settings involving private user data.The paper gives medical datasets restricted from exchange between clients as an example.Source: Conclusion and Broader Impacts · p. 7

Related papers

Loading 2112.09824v1…