Source-linked AI summary
LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
Ang Li, Jingwei Sun, Binghui Wang, Lin Duan, Sicheng Li, Yiran Chen, Hai Li
TL;DR
Non-IID data makes it difficult to train a shared federated model that generalizes well across clients, while communication efficiency remains a central challenge. LotteryFL learns personalized sparse subnetworks per client and communicates only those networks, significantly improving personalization and communication cost on non-IID datasets.
Problem
Statistical heterogeneity makes it difficult to train a shared federated model that generalizes well across clients, motivating personalized approaches to federated learning.
Method
LotteryFL learns a client-specific sparse Lottery Ticket Network and communicates only its parameters between clients and the server.
Results
LotteryFL significantly outperforms FedAvg and LG-FedAvg in personalization and communication cost across the evaluated non-IID datasets; on CIFAR-10, it achieves 89.70% accuracy, 14.18% and 43.5% higher than LG-FedAvg and FedAvg.
Takeaways & Limitations
LotteryFL provides a personalized and communication-efficient federated learning framework for non-IID settings, supported by purpose-built datasets and the Client-Wise Non-IID Index.
Takeaways & Limitations
The dataset construction assumes that each client lacks sufficient data to train a local model with the desired performance.
Abstract
from arXiv · showhide
Federated learning is a popular distributed machine learning paradigm with enhanced privacy. Its primary goal is learning a global model that offers good performance for the participants as many as possible. The technology is rapidly advancing with many unsolved challenges, among which statistical heterogeneity (i.e., non-IID) and communication efficiency are two critical ones that hinder the development of federated learning. In this work, we propose LotteryFL -- a personalized and communication-efficient federated learning framework via exploiting the Lottery Ticket hypothesis. In LotteryFL, each client learns a lottery ticket network (i.e., a subnetwork of the base model) by applying the Lottery Ticket hypothesis, and only these lottery networks will be communicated between the server and clients. Rather than learning a shared global model in classic federated learning, each client learns a personalized model via LotteryFL; the communication cost can be significantly reduced due to the compact size of lottery networks. To support the training and evaluation of our framework, we construct non-IID datasets based on MNIST, CIFAR-10 and EMNIST by taking feature distribution skew, label distribution skew and quantity skew into consideration. Experiments on these non-IID datasets demonstrate that LotteryFL significantly outperforms existing solutions in terms of personalization and communication cost.
1 Introduction
LotteryFL is a personalized, communication-efficient federated learning framework that discovers client-specific sparse lottery-ticket subnetworks under non-IID data. The work also introduces benchmark datasets and a Client-Wise Non-IID Index, with experiments showing improvements over FedAvg and LG-FedAvg [13] in personalization and communication cost.
- Motivation: LotteryFL addresses statistical heterogeneity, which makes it difficult for a shared global model to generalize well across non-IID clients.Federated learning keeps local data private while clients communicate model parameters through a central server.
- Our work: LotteryFL discovers a sparse lottery-ticket network for each client, enabling personalized models while reducing communication under non-IID settings.The framework exploits the Lottery Ticket hypothesis to identify subnetworks within a larger base model and communicates only these client-specific networks.
- Datasets: The authors construct non-IID datasets from MNIST, CIFAR-10, and EMNIST to support research on practical federated learning settings.They also define the Client-Wise Non-IID Index to quantitatively evaluate distribution differences across clients.
- Experiments: Experiments show that LotteryFL significantly outperforms FedAvg and LG-FedAvg [13] in both personalization and communication cost.The comparison is conducted on the designed non-IID datasets.
2 Related Work
Prior federated-learning research addresses non-IID personalization and communication efficiency separately, while LG-FedAvg [13] was the first method identified as tackling both simultaneously but uses an unrealistic setting. This paper targets both challenges under more realistic federated-learning conditions.
- Background: Federated learning seeks a global model that performs well for nearly all participants, but statistical heterogeneity and communication efficiency remain critical challenges [18] [3].
- Personalization: Existing personalization methods use meta-learning [4] [6], multi-task learning, or transfer learning [9] [10] to adapt global models to non-IID clients.
- Communication Efficiency: Communication-efficiency studies [11] reduce transmitted data by combining FedAvg with sparsification, quantization, sketching, or other compression techniques.
- Related Work Gap: LG-FedAvg [13] was the first method identified as addressing personalization and communication efficiency simultaneously, but its problem setting does not represent realistic federated learning.
3 Design of LotteryFL
LotteryFL combines the Lottery Ticket hypothesis with FedAvg so each client learns a data-dependent lottery ticket network (LTN), communicates only its LTN parameters, and ultimately obtains a personalized model. Clients iteratively prune and retrain subnetworks while the server aggregates only the corresponding LTN parameters.
- 3 Design of LotteryFL: Each client identifies an LTN by pruning the base model with its local data, and only LTN parameters are communicated instead of the full base model [14].The client’s local mask indicates the subnetwork selected through the Lottery Ticket hypothesis.
- Training algorithm: The server aggregates only client LTNs via FedAvg and updates the corresponding parameters, rather than aggregating complete models.This is the key distinction from standard FedAvg and reduces the communicated model components to the learned subnetworks.
- Training algorithm: In each communication round, the server randomly samples participating clients, which train their local masked models using local training data.Client data are split into training, validation, and test sets, and local optimization uses mini-batch training for E epochs.
- Training algorithm: When validation accuracy exceeds accthreshold before reaching rtarget, a client prunes small weights at rate rp to learn the next-round mask.The resulting mask incorporates data-dependent features, after which the LTN weights are re-initialized to their corresponding values in the initial model θ0.
- Training algorithm: After the predefined communication rounds, each client retains a personalized model θk.The pruning, local training, and LTN aggregation process repeats until the specified number of rounds is reached.
4 Non-IID Datasets
The section constructs practical non-IID datasets from MNIST, CIFAR-10, and EMNIST by modeling feature, label, and quantity skew, and introduces the Client-Wise Non-IID Index (CNI) to quantify cross-client heterogeneity.
- Dataset construction: The datasets model three major non-IID effects: feature distribution skew, label distribution skew, and quantity skew.Feature skew gives clients different features for the same label; label skew changes per-label amounts across clients; quantity skew makes label counts unbalanced within a client.
- Dataset construction: Two MNIST and CIFAR-10 configurations ensure clients have insufficient local data: n-class balanced represents feature skew, while n-class unbalanced combines feature and quantity skew.In both configurations, class choices can vary across clients and test data follows each client’s training distribution.
- Dataset construction: EMNIST is transformed into a non-IID dataset by grouping handwritten letters and digits by its By_Author writer attribute, assigning each client a specific writer’s images.Its shared image structure with MNIST preserves the original image format while representing a more challenging classification task.
- Client-Wise Non-IID Index: CNI quantifies non-IID severity across clients without requiring dataset-specific trained feature extractors and classifiers, unlike the prior Non-IID Index (NI) [20].It measures the distance between average class representations on one client and the corresponding representations across other clients, using normalized feature-space statistics and the ℓ2-norm.
- Client-Wise Non-IID Index: The section reports CNI values for the constructed non-IID datasets under different settings in Table 1.The supplied passage identifies the table’s scope but does not provide its cell values.
5 Evaluation
LotteryFL is evaluated on constructed non-IID MNIST, CIFAR-10, and EMNIST datasets for client personalization and communication cost. Across tested client-participation and data-volume settings, it achieves the best personalization with the lowest communication cost, while pruning increases the proportion of personalized parameters.
- Evaluation setup: The evaluation uses averaged client test accuracy for personalization and communicated data volume between clients and the server for communication cost.Experiments construct non-IID datasets from MNIST and CIFAR-10 using 2-class balanced and 2-class unbalanced configurations, while EMNIST is included in the reported client-participation experiment.
- Impact of the number of participating clients per round: LotteryFL achieves the best personalization with the lowest communication cost across MNIST, CIFAR-10, and EMNIST under varying numbers of participating clients per round.The comparison is reported in Table 2 for the 2-class balanced setting.
- Impact of the balance rate: LotteryFL significantly improves personalization and communication efficiency together under different balance rates.The experiment uses at most 20 samples for one class on each client and reports results in Table 4.
- Personalized parameter analysis: Increasing the target pruning rate raises each layer’s percentage of personalized parameters by removing commonly shared parameters while retaining local-data features.Personalized parameters are defined as parameters shared by fewer than 10% of clients, and their distributions are visualized in Figure 2.
6 Conclusion
LotteryFL is presented as a personalized, communication-efficient federated learning framework for non-IID settings, inspired by the Lottery Ticket hypothesis. The work also contributes datasets for non-IID federated learning and introduces CNI to quantify cross-client distribution heterogeneity.
- LotteryFL is designed as a personalized and communication-efficient federated learning framework for non-IID settings, inspired by the Lottery Ticket hypothesis.
- The authors construct and publish datasets supporting federated learning under non-IID settings to facilitate research on robust federated learning in more challenging environments.
- CNI is introduced as the first metric for quantitatively evaluating the degree of non-IID data distribution across clients.
Broader Impact
Federated learning can support privacy-sensitive collaboration among organizations such as hospitals by keeping patient data local, but prior work shows that private training data may still be recoverable [24].
- Broader Impact: Hospitals and other organizations can participate as federated-learning clients when privacy regulations or ethical constraints require patient data to remain local.Hospitals are presented as an example because they hold substantial patient data for intelligent healthcare.
- Broader Impact: Federated learning enables collaborative learning without requiring participating institutions to share their locally held data.The passage identifies FL as a promising solution for privacy-constrained applications.
- Broader Impact: However, prior work shows that private training data can feasibly be recovered in federated-learning settings [24].This caveat qualifies the privacy benefits of applying FL to sensitive institutional data.