Source-linked AI summary
Federated Learning with Personalization Layers
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, Sunav Choudhary
TL;DR
Federated learning must handle client data heterogeneity while keeping raw data local, but standard federated averaging is poorly suited to personalized deep learning. The paper proposes FedPer, which shares base layers and trains personalization layers locally, and reports effectiveness on non-identical CIFAR partitions and FLICKR-AES. Its scope includes assumptions about personalization being captured by client-specific weights and all devices remaining active throughout training.
Problem
Statistical heterogeneity across user devices complicates federated personalization, while limited per-user data makes isolated model training inadequate.
Method
FedPer decomposes deep feedforward networks into collaboratively trained base layers and locally trained personalization layers.
Results
FedPer is effective on non-identically partitioned CIFAR datasets and FLICKR-AES, whereas FedAvg is ineffective for the evaluated personalization tasks.
Takeaways & Limitations
Separating shared representation from private personalization can combat the ill-effects of statistical heterogeneity in federated personalization.
Takeaways & Limitations
The setup assumes client personalization is fully captured by personalization-layer weights and that all user devices remain active throughout training.
Abstract
from arXiv · showhide
The emerging paradigm of federated learning strives to enable collaborative training of machine learning models on the network edge without centrally aggregating raw data and hence, improving data privacy. This sharply deviates from traditional machine learning and necessitates the design of algorithms robust to various sources of heterogeneity. Specifically, statistical heterogeneity of data across user devices can severely degrade the performance of standard federated averaging for traditional machine learning applications like personalization with deep learning. This paper pro-posesFedPer, a base + personalization layer approach for federated training of deep feedforward neural networks, which can combat the ill-effects of statistical heterogeneity. We demonstrate effectiveness ofFedPerfor non-identical data partitions ofCIFARdatasetsand on a personalized image aesthetics dataset from Flickr.
1 Introduction
Federated learning keeps training data on user devices, but statistical heterogeneity makes personalization difficult for a single shared model. The paper proposes combining collaboratively trained base layers with locally trained personalization layers and evaluates this approach on CIFAR and FLICKR-AES.
- Federated learning trains models on devices using local data and occasional communication with an aggregating parameter server.
- Statistical heterogeneity across clients can substantially harm federated training for applications such as personalization, recommendation, and fraud detection.
- Personalization tasks may assign different labels to identical inputs for different users, requiring models that differ across users rather than one replicated global model.
- Limited samples per user require collaboration, while insufficient overlap between users’ rated items can make collaborative filtering unsuitable for personalized image tasks.
- FedPer trains shared base layers with federated averaging and keeps personalization layers local, allowing models to capture user-specific preferences.
- The approach is evaluated against standard federated learning on non-identically partitioned CIFAR-10/CIFAR-100 and the FLICKR-AES personalized image aesthetics dataset.
- Standard federated learning performs poorly on FLICKR-AES because identical client models cannot capture users’ personal preferences.
2 Related Work
Related work addresses federated heterogeneity through personalization, collaborative filtering, meta-learning, multi-task learning, and transfer learning. FedPer instead targets deep learning with a base-plus-personalization split while focusing on statistical heterogeneity and preserving compatibility with approaches to other federated challenges.
- Federated Learning: The paper restricts its novelty to statistical heterogeneity while remaining composable with methods addressing system heterogeneity, privacy leakage, and communication overhead.
- Federated Learning: FedPer learns overlapping local models by sharing base layers while retaining personalization layers, extending federated learning beyond one replicated global model.
- Multi-task and Transfer Learning: Healthcare-motivated federated transfer learning assumes global training can be separated from later local personalization, an assumption the paper says does not hold for personalized image tasks.
- Multi-task and Transfer Learning: The paper’s model split resembles prior federated deep-learning splitting principles, but its specific split is motivated and evaluated using different datasets and smaller data-volume distributions.
- Distributed Personalization and Recommendation: Federated personalization and recommendation require redesign because federated systems introduce statistical and communication heterogeneity absent from tightly controlled distributed settings.
- Distributed Personalization and Recommendation: Recent federated collaborative-filtering and meta-learning methods address recommendation but do not cover deep learning models.
3 Modeling and Algorithmic Setup
FedPer models each client with shared base layers and unique personalization layers, optimizing their weights through local training and federated aggregation. The setup formalizes personalized risk while using SGD locally and FedAvg-style aggregation for shared layers.
- Model setup: Each client model combines shared base-layer weights with client-specific personalization-layer weights and corresponding activation functions.The base layers are common across devices, while each device has its own personalization layers.
- Model setup: The personalized network performs a forward pass on each client by composing the base and personalization layers to produce a prediction.The paper denotes this client-specific forward operation by f.
- Objective: Training minimizes average personalized population risk over the shared base weights and all client-specific personalization weights.Because the client distributions are unknown, empirical risk based on each client’s samples is used as a proxy.
- Client training: FedPer updates client models locally with stochastic gradient descent between global aggregations, using specified decision variables, batch size, epochs, and learning rate.The client procedure initializes personalization weights locally and receives the shared base weights from the server.
- Server aggregation: The server aggregates updated base-layer weights across clients using sample-count-based weights while leaving personalization layers outside the federated averaging step.The server algorithm initializes, distributes, receives, and repeatedly aggregates the shared base weights.
4 Experiments
Experiments compare FedPer with FedAvg across non-identically partitioned CIFAR datasets and personalized FLICKR-AES data, varying architectures, personalization depth, and base-layer design. FedPer generally improves convergence, client-averaged accuracy, and cross-client consistency under heterogeneous data, while its best personalization depth depends on the dataset.
- Experimental Setup: The experiments use ResNet-34 and MobileNet-v1 across CIFAR-10, CIFAR-100, and FLICKR-AES, with heterogeneous client partitions and local user data splits.CIFAR experiments use 10 clients with class-restricted partitions, while FLICKR-AES experiments use 30 users as clients and per-user 80%/20% training/testing splits.
- Effect of Statistical Heterogeneity: FedPer improves convergence speed and steady-state client-averaged test accuracy over FedAvg on highly non-identical CIFAR-10 partitions.As partitions become more identical, FedPer’s performance approaches FedAvg; the same qualitative pattern holds on CIFAR-100.
- Effect of Statistical Heterogeneity: FedPer produces lower variation in MobileNet-v1 test accuracy across clients than FedAvg on the highly heterogeneous CIFAR-10 partition.The paper identifies this lower variation as important for fairness in learning.
- Effect of Personalization Layers: For CIFAR-10, KP = 2 achieves the best performance for both MobileNet-v1 and ResNet-34, whereas KP = 1 is best for CIFAR-100.There is no clear general correlation between the number of personalization layers and steady-state client-averaged accuracy beyond the benefit of having at least one personalization layer.
- Base-Layer Analysis: Replacing the base layers with linear layers causes a performance drop on CIFAR-100, indicating that the base layers are not redundant.The experiment tests whether personalization layers alone are sufficient for the task.
- Performance on FLICKR-AES: FedAvg is unfit for personalized image aesthetics on FLICKR-AES, while FedPer captures user preferences through locally trained personalization layers.The FedAvg MobileNet-v1 result is described as similar to random guessing, whereas FedPer provides evidence of modeling personalized tasks despite the models not being state-of-the-art.
5 Conclusion
FedPer separates shared base layers from locally trained personalization layers to address statistical heterogeneity, with empirical results showing advantages over FedAvg on CIFAR and FLICKR-AES.
- FedPer trains base layers collaboratively while training personalization layers locally to capture user-specific aspects.
- Base plus personalization layers help combat the ill-effects of statistical heterogeneity in federated learning.
- Empirical results on FLICKR-AES and CIFAR datasets demonstrate FedAvg’s ineffectiveness and FedPer’s effectiveness for personalization tasks.
A.1 Effect of Statistical Heterogeneity
Figures 11 and 12 present CIFAR-100 plots corresponding to the CIFAR-10 plots in Section 4.2. Figure 13 shows cross-client variation in test accuracies under identical data partitioning at k = 10.
- Figures 11 and 12 plot results on CIFAR-100.These figures correspond to the CIFAR-10 plots in Section 4.2.
- The CIFAR-100 plots in Figures 11 and 12 correspond to the CIFAR-10 plots in Section 4.2.
- Figure 13 reports cross-client variation in test accuracies with identical data partitioning at k = 10.
A.2 Effect of Personalization Layers
FedPer’s CIFAR-100 performance varies with the number of personalization layers, and one personalization layer appears to perform best in test accuracy.
- KP = 1 appears to achieve the best test accuracy for FedPer on CIFAR-100.
A.3 Effect of FineTuning
FedPer fine-tunes locally trained personalization layers after receiving updated base parameters, improving CIFAR-100 client accuracy but showing no noticeable FLICKR-AES effect.
- Each global round sends aggregated base-layer parameters to clients, which locally update their model parameters before returning updated base parameters.
- Fine-tuning personalization layers for one epoch while freezing base layers is intended to accommodate changes in the shared base parameters.
- Figure 13 presents MobileNet-v1 client-model performance variation on CIFAR-10 for FedAvg and FedPer.
- Figure 14 presents FedPer performance on a non-identical CIFAR-100 partition as the number of personalization layers changes.
- Fine-tuning improves MobileNet-v1 and ResNet-34 client accuracy on CIFAR-100.
- Fine-tuning personalization layers has no noticeable effect on MobileNet-v1 and ResNet-34 performance on FLICKR-AES.