PaperlayerView exact revision on arXiv

Source-linked AI summary

DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training

Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, Dacheng Tao

arXiv:2206.00187v1cs.LG

TL;DR

Personalized federated learning must handle heterogeneous client data without relying on vulnerable, communication-heavy central servers. Dis-PFL addresses this gap with decentralized peer-to-peer training, personalized sparse masks, and fixed sparsity throughout training. The paper reports improved personalized performance alongside lower communication and computation demands, while showing that sparsity requires careful tuning.

  • Problem

    Existing personalized federated learning methods mainly use centralized servers, creating server vulnerability and communication burdens, while dense-to-sparse training poorly fits heterogeneous hardware.

  • Method

    Dis-PFL uses personalized sparse masks in decentralized peer-to-peer learning, maintaining sparse local models through intersection-weight averaging, fixed-mask training, and gradient-based mask adjustment.

  • Results

    Dis-PFL increases averaged local test accuracy while reducing busiest-node communication, local computation cost, and communication rounds, and adapting to heterogeneous resources.

  • Takeaways & Limitations

    Dis-PFL provides a decentralized personalized federated learning design that jointly addresses data heterogeneity, client heterogeneity, and resource constraints.

  • Takeaways & Limitations

    Choosing the sparsity ratio requires balancing communication benefits against information exchange, training error, and generalization.

Abstract

from arXiv · show

Personalized federated learning is proposed to handle the data heterogeneity problem amongst clients by learning dedicated tailored local models for each user. However, existing works are often built in a centralized way, leading to high communication pressure and high vulnerability when a failure or an attack on the central server occurs. In this work, we propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL, which employs personalized sparse masks to customize sparse local models on the edge. To further save the communication and computation cost, we propose a decentralized sparse training technique, which means that each local model in Dis-PFL only maintains a fixed number of active parameters throughout the whole local training and peer-to-peer communication process. Comprehensive experiments demonstrate that Dis-PFL significantly saves the communication bottleneck for the busiest node among all clients and, at the same time, achieves higher model accuracy with less computation cost and communication rounds. Furthermore, we demonstrate that our method can easily adapt to heterogeneous local clients with varying computation complexities and achieves better personalized performances.

1. Introduction

The introduction motivates decentralized personalized federated learning by combining personalization for heterogeneous data with peer-to-peer communication and sparse, client-specific models. Dis-PFL uses decentralized sparse training to reduce resource demands while adapting to heterogeneous clients.

  • Data heterogeneity motivates personalized federated learning, which produces individual models instead of relying solely on a shared global model.
  • Centralized personalized FL depends on a server that creates failure, attack, privacy, and communication-burden vulnerabilities.
  • Dense-to-sparse personalization methods first train dense models, limiting their suitability for hardware-constrained and heterogeneous clients.
  • Dis-PFL combines customized client masks with decentralized communication to address both data and client heterogeneity.
  • Its decentralized sparse training averages only intersection weights, trains with fixed sparse masks, and adjusts masks using gradient information.
  • Experiments report higher local test accuracy, lower busiest-node communication and computation costs, and fewer rounds to reach the same target.

2. Related Work

Related work spans personalized federated learning, decentralized peer-to-peer optimization, and sparse neural networks. These areas respectively address client-specific models, server-free consensus, and reduced model parameterization.

  • Personalized federated learning targets individual models for each node, while federated learning methods address non-IID data through objectives, aggregation, and client selection.
  • Decentralized learning replaces the central server with peer-to-peer communication and can combine gossip averaging with SGD to reach consensus.
  • Decentralized methods also modify optimization for heterogeneous data, including adaptive momentum and other changes to consensus-oriented training.
  • Sparse neural-network methods include dense-to-sparse and sparse-to-sparse approaches, motivated by evidence that sparse models can match dense-model performance.

3. Dis-PFL algorithm

Dis-PFL formulates personalized federated learning with client-specific sparse masks and solves it through decentralized sparse training. Each client maintains and updates a sparse personalized model while exchanging information with neighbors, with sparsity also supporting generalization and heterogeneous resource constraints.

  • Problem formulation: Personalized federated learning seeks client-specific models by optimizing local risks, while communication and regularization let clients share information to improve generalization.The framework includes traditional federated learning as the special case where all client models are constrained to be equal.
  • Personalized sparse models: Each client uses a binary mask m_k to activate selected weights from a shared model w, producing a personalized sparse model w ⊙ m_k.A mask value of 1 keeps a weight active for client k; a value of 0 makes it dormant.
  • Decentralized sparse training: Dis-PFL keeps each client sparse throughout training to satisfy computing, memory, and communication constraints, rather than first training a dense model.Personalized sparsity is determined by each client’s capacity c_k, allowing different clients to use different masks.
  • Decentralized sparse training: Neighbor models are combined by weighted averaging over intersecting active parameters, then masked locally before sparse local training continues.This modified averaging process integrates models with different masks while preserving each client’s personalized sparse structure.
  • Decentralized sparse training: Mask updates prune small-magnitude weights and recover high-gradient coordinates, while gossip averaging supplies recovered coordinates with values for warm-up.The method combines modified gossip averaging, fixed sparse-mask training, and local mask searching.
  • Generalization analysis: The generalization analysis states that increasing sparsity, represented by a smaller β_k and β, yields a smaller bound on the train–test gap.The result is presented under bounded loss and sample-size assumptions for the model learned by Algorithm 1.

4. Experiments

Experiments evaluate Dis-PFL across datasets, non-IID partitions, communication topologies, and client constraints. The results report higher personalized accuracy, lower communication and local training costs, faster convergence, and adaptation to heterogeneous clients, while showing that sparsity requires careful tuning.

  • Experiment setup: Dis-PFL is evaluated on CIFAR-10, CIFAR-100, and Tiny-Imagenet under Dirichlet and another non-identical data-partition scenario.The experiments compare centralized and decentralized federated-learning baselines.
  • Main experiments evaluation: Dis-PFL outperforms centralized and decentralized baselines in personalized test accuracy across all three datasets and both non-IID partitions.Global-consensus methods can underperform separate local training, while dense personalized baselines remain below Dis-PFL.
  • Main experiments evaluation: Dis-PFL reduces peer-to-peer communication costs and local training FLOPS by maintaining sparse models throughout training, including memory-constrained settings.The reported cost reductions also indicate inference-speed and energy-consumption benefits.
  • Main experiments evaluation: Dis-PFL reaches target accuracy in significantly fewer communication rounds than other baselines, although fully connected topologies can converge faster than sparse topologies at equal rounds.The topology effect is attributed to more information being communicated per round in fully connected networks.
  • Experiments on client heterogeneous setting: Under heterogeneous client constraints, Dis-PFL adapts across sparsity groups and achieves better test accuracy than comparable 50%-parameter D-PSGD and D-PSGD-FT models.Models with different sparsities retain the ability to handle their own tasks across data partitions and backbones.
  • Discussion of the sparsity ratio: Sparsity-ratio experiments show a trade-off: greater sparsity can reduce communication and improve generalization in one regime, but excessive or insufficient sparsity degrades performance.A reasonable sparsity ratio therefore requires selecting a practical balance between communication benefits, training error, and mask personalization.

5. Conclusion

Dis-PFL addresses data and client heterogeneity in personalized federated learning through decentralized sparse training. The authors report reduced communication and local training costs, adaptation to heterogeneous clients, theoretical and experimental analysis of sparse masks, and extensive empirical efficacy.

  • Dis-PFL simultaneously tackles data heterogeneity and client heterogeneity in personalized federated learning.
  • Decentralized sparse training reduces the communication bottleneck and local training costs.
  • Dis-PFL adapts to clients with heterogeneous capabilities.
  • The work provides theoretical and experimental understanding of sparse masks and extensive experiments verifying Dis-PFL’s efficacy.

A. More details on algorithm implementation

The local mask-searching procedure updates each client’s sparse mask using a scheduled pruning rate, magnitude pruning, and gradient information. It takes the current model and mask as input and returns an updated mask.

  • The local mask-searching algorithm takes the current model and mask as input and produces a new mask.
  • It computes the current prune rate using cosine annealing with an initial pruning rate.
  • The procedure samples local data and backpropagates to obtain dense gradients.
  • For each layer, it prunes weights by magnitude and recovers weights using gradient information to obtain the updated mask.

B. Experiments

The experiments provide detailed comparisons between Dis-PFL and other baselines, with additional implementation details and extensive experimental results.

  • The experiments compare Dis-PFL’s performance against other baselines.
  • The section includes additional experimental details and extensive results.
  • The stated purpose is to evaluate the proposed Dis-PFL against alternative methods.

B.1. Datasets

The evaluation uses CIFAR-10, CIFAR-100, and Tiny-Imagenet with two non-IID data-partitioning methods. The Dirichlet partition controls label-distribution heterogeneity through its α parameter, while the second method is identified as pathological partitioning.

  • The experiments use CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets.CIFAR-10 has 10 classes and 5000 training samples each; CIFAR-100 has 100 classes and 500 samples each; Tiny-Imagenet has 200 classes and 500 samples each.
  • Training data are split across devices using two non-IID partitioning methods.
  • The Dirichlet partition creates uneven label distributions, with smaller α indicating higher data heterogeneity.
  • The second data-partitioning method is called pathological partitioning.

B.2. Model Architectures

The experiments use ResNet18 and VGG11 with group normalization, alongside standardized optimization and training settings. Baseline methods cover local, centralized personalized, sparse personalized, and decentralized federated learning approaches.

  • Model architectures: Evaluations use PyTorch implementations of ResNet18 and VGG11, replacing batch normalization with group normalization.The replacement is motivated by reported detrimental effects of batch normalization in federated learning.
  • Training settings: All methods use SGD with weight decay 0.0005, batch size 128, and a learning rate initialized at 0.1.The learning rate decays by 0.998 after each communication round.
  • Baselines: Local training serves as the direct personalized federated-learning solution, while FedAvg collaboratively trains a global model through weighted averaging.Local training uses five epochs as one communication round for consistency.
  • Baselines: The baseline suite includes FedAvg-FT, Ditto, FOMO, SubFedAvg, and D-PSGD, spanning fine-tuning, regularization, neighbor-based mixing, sparse personalization, and decentralized training.SubFedAvg uses dense-to-sparse training, beginning with a fully dense local model and pruning it iteratively.

B.5. More experiments results

Additional experiments evaluate communication efficiency and performance across datasets and decentralized topologies. The reported comparisons include target-accuracy communication rounds and decentralized evaluations on CIFAR-100 and Tiny-Imagenet.

  • Communication efficiency: Communication-round requirements to reach target accuracy are reported for CIFAR-10, CIFAR-100, and Tiny-Imagenet.The corresponding results appear in Tables 5, 6, and 7.
  • Communication efficiency: Table 5 reports averaged communication rounds to reach target accuracy on CIFAR-10.
  • Communication efficiency: Table 6 reports averaged communication rounds to reach target accuracy on CIFAR-100.
  • Communication efficiency: Table 7 reports averaged communication rounds to reach target accuracy on Tiny-Imagenet.
  • Decentralized topologies: Additional decentralized-topology performance comparisons are reported for CIFAR-100 and Tiny-Imagenet.These comparisons are presented in Tables 8 and 9.

B.6. Extended experiments on random clients dropping settings

The random-client-dropping experiments assess whether decentralized personalization remains effective when clients disconnect. Dis-PFL maintains decent personalized models across dropping probabilities, although higher disconnection probabilities slow convergence and may reduce final performance.

  • Random client dropping: Decentralized training can tolerate one or more clients dropping in a round without necessarily halting the overall training process.The experiment is motivated by the possibility that clients or a server fail to participate in every communication round.
  • Random client dropping: Dis-PFL provides fairly decent personalized models compared with local training across device-dropping probabilities.The evaluation is conducted in a fully connected topology, as shown in Figure 6.
  • Random client dropping: Higher device-disconnection probabilities may slow Dis-PFL convergence and affect the final personalized result.

C.3. Main proof

The proof treats Dis-PFL as an iterative decentralized learning process with sparse, partial gradient information and Gaussian noise. It derives per-step privacy bounds, extends them across iterations, and then obtains the final generalization result.

  • Proof setup: The proof first views Dis-PFL globally and assumes a shared initial consensus model formed by decentralized model fusion.Each client applies its personalized mask to this consensus model before local sparse training and mask searching.
  • Proof setup: At iteration t, decentralized learning optimizes a virtual consensus model using partial gradient information from clients because of sparse masks.The proof denotes the virtual consensus model by W_t and assumes bounded loss gradients.
  • Per-step privacy: The proof calculates each step’s differential privacy by modeling mini-batch sampling without replacement and additive Gaussian noise during local training.The mini-batch has size τ, and the privacy analysis considers adjacent sample sets.
  • Per-step privacy: The gradient-space diameter and the remaining-parameter proportion β bound the sparse local gradients used in the privacy analysis.The local training gradients are bounded using the clients’ differing sparsity and sparse masks.
  • From privacy to generalization: Combining the subsampling lemma yields a per-step privacy guarantee, while the iterative-learning lemma provides the overall differential-privacy guarantee.The final generalization result follows by applying the differential-privacy generalization lemma.
Loading 2206.00187v1…