Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity

1 INTRODUCTION

• Federated learning collaboratively optimizes a common objective across clients coordinated by a central server without explicitly sharing their data.The introduction also names scalability, data locality, ownership, and privacy as advantages over centralized learning.Source: §1 · p. 1
• Non-IID client data makes training a single model on the union of client datasets harder, motivating personalization alongside communication-efficiency work.The introduction notes that FedAvg can work with certain non-IID data, while other studies pursue personalization and communication-cost reduction.Source: §1 · p. 2

1.2 CONTRIBUTIONS

• The proposed framework targets statistical heterogeneity, communication cost, and personalization together through client-specific subnetworks found by hybrid and unstructured pruning.Clients are framed as participants seeking improved performance on their personalized target distributions, not merely sources of data and training.Source: §1.2 · p. 2

2 RELATED WORK

• Under statistical heterogeneity, some clients may gain no benefit from FL because the shared model can be less accurate than their local models (Yu et al. (2020); Hanzely & Richtárik (2020)).The section identifies improving their own model performance as clients’ main motivation to participate.Source: §2 · pp. 2–3
• Sending full model updates each round makes communication a bottleneck for large networks, especially because uplink speeds are typically slower than downlink speeds.Prior approaches described include structured or sketched updates and gradient compression (Koneˇcn`y et al. (2016); Lin et al. (2017)).Source: §2 · p. 3
• Most prior work measured global rather than individual-client accuracy, while personalization methods may affect privacy or require a separate second step.That second step personalizes a collaboratively trained global model using each client’s private data, potentially adding computational overhead (Jiang et al. (2019); Yu et al. (2020); Sim et al. (2019)).Source: §2 · p. 3

3 METHOD

• The method aims to reduce the inference costs and model sizes of high-capacity CNNs in FL, where computation and efficiency matter for resource-constrained platforms.It situates pruning among model-compression and inference-acceleration strategies, including progressive pruning followed by fine-tuning to restore performance (Frankle & Carbin (2018)).Source: §3 · p. 3

3.1 EFFICIENT LEARNING WITH PRUNING

• Experiments motivate the Client Subnetwork Observation: clients with similar or partially similar labels can have similar smaller subnetworks that improve their accuracy.The authors describe the costs of finding these subnetworks as cheap.Source: §3.1 · p. 3

3.2 EFFICIENT LEARNING WITH PRUNING

• Under some scheduling, a client’s masked, p-percentage-pruned network can reach higher test accuracy than its dense network.The comparison uses each client’s minimum-validation-loss training point and reports the relation Acc_k ≤ Acc′_k.Source: §3.1 · pp. 3–4

3.3 STRUCTURED, UNSTRUCTURED, AND HYBRID PRUNING

• The paper considers unstructured parameter pruning, structured channel pruning, and hybrid pruning, which combines channel- and parameter-level pruning.Unstructured pruning offers flexibility; channel pruning trades some flexibility for implementation ease and is most effective when networks are sufficiently deep.Source: §3.3 · p. 4

3.4 WHY LEARNING WITH PRUNING IN FL IS EFFICIENT

• Each round selects clients to download, train, prune, and upload models; Sub-FedAvg averages only the intersection of their remaining channels or parameters.The method’s rationale is that pruning removes commonly shared parameters while retaining personalized ones, and clients with label overlap share similar personalized parameters.Source: §3.4 · p. 4

3.5 ALGORITHM

• Unstructured pruning masks the lowest-magnitude parameters by a specified percentage, while structured pruning ranks channels using batch-normalization scaling factors.The unstructured routine applies masks subject to validation-accuracy, target-pruning-rate, and mask-distance thresholds; hybrid pruning applies filter and fully connected layer pruning independently when their respective constraints are met.Source: §3.5 · p. 5

4.1 EXPERIMENT SETTINGS

• Experiments use MNIST, CIFAR-10, EMNIST, and CIFAR-100 with two randomly assigned data shards per client; models are a CNN for MNIST/EMNIST and LeNet-5 for CIFAR datasets.Shards contain 250 examples, except CIFAR-100 shards of 125; experiments use 100 clients, batch size 10, five local epochs, and SGD with learning rate 0.01 and momentum 0.5.Source: §4.1 · pp. 5–6

4.2.1 ACCURACY VERSUS PRUNING PERCENTAGE

• Accuracy generally rises as pruning increases to a level of sparsity, then declines at around 50% pruning as further pruning removes personal parameters; almost 85% of clients show the same pattern.The figures cover sampled clients and averages over clients on LeNet-5/CIFAR-10, and averages on CIFAR-10, MNIST, and EMNIST.Source: Figure 1 · p. 6, Figure 2 · p. 8, §4.2.1 · p. 8

4.2 MAIN RESULTS

• Iterative pruning removes common parameters while retaining personalized ones; clients with overlapping labels are likely to share personal parameters and can average them through Sub-FedAvg.The authors describe this as a way for clients to find partners with similar labels.Source: §4.2 · pp. 7–8

4.2.2 COMMUNICATION EFFICIENCY

• Sub-FedAvg (Un) reduces required communication rounds by 2–10× across cases; after around 100 rounds, accuracy is 99.5% on MNIST, 99.3% on EMNIST, and 86% on CIFAR-10.For CIFAR-10, FedAvg requires about three times as many rounds as Sub-FedAvg (Un) to obtain the target accuracy.Source: Figure 3 · p. 8, §4.2.2 · p. 10

4.2 MAIN RESULTS

• In the low-data non-IID regime, Sub-FedAvg yields around 20% accuracy improvement over Standalone and around 30% over FedAvg, while traditional FedAvg performs worse than Standalone.The comparison covers the proposed structured/unstructured pruning approach against local-only training and traditional federation.Source: §4.2 · p. 7, Table 1 · p. 9

4.2.3 FLOP REDUCTION

• The proposed model reduces FLOPs by up to 2.4× versus state-of-the-art; on LeNet-5, pruning 50% of channels yields around 50% FLOP reduction and around 38% parameter savings.The FLOP analysis counts convolution operations and treats batch normalization and pooling as negligible.Source: §4.2 · p. 8, Table 2 · p. 10, §4.2.3 · p. 10

• The framework outperforms state-of-the-art algorithms on CIFAR-10/100, MNIST, and EMNIST.Source: §5 · p. 11
• It iteratively prunes neural-network parameters and channels, removing commonly shared parameters while retaining personalized ones.Source: §5 · p. 11
• The framework is also efficient in communication cost and FLOP count.Source: §5 · p. 11