Source-linked AI summary
Personalized Cross-Silo Federated Learning on Non-IID Data
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, Yong Zhang
TL;DR
Non-IID client data challenge global federated models in personalized cross-silo settings. The paper proposes FedAMP, which uses attentive message passing to promote pairwise collaboration among similar clients, and proves convergence for convex and non-convex models. Extensive benchmark experiments report superior performance for the proposed methods.
Problem
Non-IID client data can make global collaboration unable to achieve good performance for individual clients, motivating personalized pairwise collaboration.
Method
FedAMP gives each client a personalized cloud model and attentively passes messages among models with similar parameters using weighted combinations.
Results
The paper proves FedAMP convergence for convex and non-convex personalized models and reports superior performance in extensive benchmark experiments.
Takeaways & Limitations
Pairwise collaboration lets the method exploit similarity among clients with non-IID data rather than relying exclusively on one global model.
Takeaways & Limitations
Existing fine-tuning methods do not solve the non-IID personalization problem systematically in the reported experiments, including CIFAR100.
Abstract
from arXiv · showhide
Non-IID data present a tough challenge for federated learning. In this paper, we explore a novel idea of facilitating pairwise collaborations between clients with similar data. We propose FedAMP, a new method employing federated attentive message passing to facilitate similar clients to collaborate more. We establish the convergence of FedAMP for both convex and non-convex models, and propose a heuristic method to further improve the performance of FedAMP when clients adopt deep neural networks as personalized models. Our extensive experiments on benchmark data sets demonstrate the superior performance of the proposed methods.
1 Introduction
Non-IID client data makes one-size-fits-all global collaboration ineffective for personalized cross-silo federated learning. FedAMP instead promotes pairwise collaboration among similar clients through attentive message passing, with convergence guarantees and strong empirical performance.
- Motivation: Non-IID differences in data size, class distributions, and class-specific distributions often prevent a single global model from performing well for every client.The limitation is especially relevant in personalized cross-silo settings with tens or hundreds of clients.
- Motivation: The paper identifies the core bottleneck as assuming that one global model can fit clients whose data reflect different cultures, lifestyles, and environments.It proposes similarity-based pairwise collaboration as a better fit for heterogeneous clients.
- Method: FedAMP gives each client a personalized model and maintains a corresponding personalized cloud model rather than using one global cloud model.Client models are passed as messages to cloud models with similar parameters.
- Method: FedAMP updates each personalized cloud model through a weighted convex combination of received messages, adaptively strengthening collaboration among similar clients.The attentive mechanism iteratively encourages more collaboration between clients with similar model parameters.
- Contributions: The paper proves convergence for convex and non-convex personalized models and introduces a heuristic for deep-neural-network clients.Extensive benchmark experiments report superior performance for the proposed methods.
2 Related Works
Prior personalized federated-learning approaches commonly rely on one global model, which limits collaboration to all-client aggregation and local customization. This paper instead studies direct pairwise collaboration that exploits similarity among clients with non-IID data.
- Scope: Personalized federated learning addresses collaborative training of client-specific models when private client data are non-IID.The related work includes global federated learning, local customization, and multi-task federated learning.
- Global federated learning: Global federated learning trains one model over the union of client data, but non-IID data make strong personalized performance on every client difficult.The cited literature frames this as a challenge for global collaboration.
- Local customization: Most local customization methods still use one global model for all-client collaboration followed by client-specific customization.This framework does not support direct pairwise collaboration among clients with similar data.
- Multi-task collaboration: Primal-dual pairwise collaboration methods achieve strong performance for convex models but require strong duality and do not extend to deep neural networks.The cited method therefore has a model-class limitation.
- This work: FedAMP differs by directly exploring pairwise collaboration and is designed to exploit client similarity under non-IID data.The paper reports that this approach is particularly effective in that setting.
3 Personalized Federated Learning Problem
The paper formulates personalized federated learning as privacy-preserving joint training of client-specific models on non-IID data. Its objective combines each client’s private training loss with an attention-based regularizer that promotes adaptive collaboration among similar models.
- Problem formulation: Personalized federated learning trains personalized models collaboratively from clients’ non-IID private data without exposing those data to other clients or third parties.Each client’s model is evaluated against the distribution underlying its own data.
- Problem formulation: The formulation represents m clients with model parameters w1, ..., wm and private datasets D1, ..., Dm sampled from distinct distributions.The setup explicitly permits client-specific models and distributions.
- Objective: The optimization objective sums the individual training losses of all personalized models and adds a collaboration regularizer.The loss term lets each client train using its own private data.
- Attention mechanism: The attention-inducing function A(∥wi−wj∥2) measures model-parameter differences nonlinearly and controls collaboration between clients.The paper requires A to be increasing and concave, continuously differentiable, and to have a finite right-derivative limit at zero.
- Attention mechanism: The method uses the negative exponential function A(∥wi−wj∥2) = 1 − e^-∥wi−wj∥2/σ as its attention-inducing function.σ is a hyperparameter; smoothly clipped absolute deviation and minimax concave penalty are also given as examples.
- Attention mechanism: Using this attention function, the method adaptively encourages similar clients to collaborate more, which the paper reports can dramatically boost personalized federated-learning performance.The collaboration is implemented through attentive message passing.
4 Federated Attentive Message Passing
FedAMP implements personalized federated learning by alternating cloud-side attentive message passing with client-side personalized optimization. Its convex-combination messages assign greater influence to similar clients, creating iterative collaboration while preserving private training data.
- General optimization method: FedAMP alternates optimizing the inter-client collaboration term A(W) with optimizing the client objectives F(W) until a preset iteration limit.The cloud-side step computes U^k, followed by a proximal client-side update of W^k.
- FedAMP implementation: The client-server implementation maintains a personalized cloud model for each client instead of a single global model.Clients receive their own cloud model and locally optimize personalized parameters using private data.
- Attentive message passing: Each personalized cloud model is a convex combination of clients’ previous model parameters, with weights derived from the attention-inducing similarity function.The weights are non-negative and sum to one when the step size is sufficiently small.
- Privacy-preserving collaboration: Model-aggregation messages are passed using model parameters rather than private training data, so clients can collaborate without exposing their private datasets.A client computes its personalized update locally from its private objective and received cloud model.
- Adaptive collaboration: Because similarity weights increase for nearby model parameters, FedAMP creates a positive feedback loop that strengthens collaboration among similar clients.The resulting process adaptively and implicitly groups similar clients for more effective collaborations.
5 Convergence Analysis of FedAMP
The convergence analysis establishes guarantees for FedAMP under convex and smooth non-convex objectives. It provides global convergence to an optimum in the convex case and to a stationary point in the non-convex case.
- Convex objectives: Under convexity and the stated boundedness assumption, FedAMP converges globally to an optimal solution of the objective.The theorem also provides an iteration bound for finding an ε-optimal solution.
- Convex objectives: FedAMP needs at most O(ε^-2) iterations to find an ε-optimal solution when the objective is convex.The guarantee is stated for a solution satisfying G(f_W) − G* ≤ ε.
- Non-convex objectives: Under smooth non-convex objectives, Lipschitz gradients, and the stated assumption, FedAMP globally converges to a stationary point.The result applies when the step sizes satisfy the theorem’s conditions.
- Non-convex objectives: FedAMP needs at most O(ε^-4) iterations to find an ε-approximate stationary point for the smooth non-convex objective.The returned point satisfies ∥∇G(f_W)∥ ≤ ε.
6 HeurFedAMP: Heuristic Improvement of FedAMP on Deep Neural Networks
HeurFedAMP revises FedAMP’s message-weight computation for deep neural-network models, where Euclidean distance may be unreliable in high dimensions. It retains FedAMP’s optimization steps while using a heuristic self-attention-based weighting scheme.
- Reported outcome: HeurFedAMP is reported to improve FedAMP’s performance for deep neural-network personalized models because cosine similarity is more robust for high-dimensional parameters than Euclidean distance.The paper characterizes this as a heuristic improvement.
- Motivation: Deep neural networks create a high-dimensional setting in which Euclidean distance may no longer effectively measure differences between personalized model parameters.This can make the resulting message weights less effective for attentive collaboration.
- Heuristic method: HeurFedAMP follows FedAMP’s optimization steps but treats the self-message weight as a hyperparameter before computing inter-client message weights.The self-attention hyperparameter controls the proportion of a client’s message sent to its own personalized cloud model.
- Heuristic method: HeurFedAMP produces non-negative message weights that sum to one, so each personalized cloud model remains a convex combination of received messages.This preserves the aggregation structure of FedAMP.
7 Experiments
Experiments compare FedAMP and HeurFedAMP with personalized and global federated learning baselines across IID, pathological non-IID, and practical non-IID settings. The proposed methods are especially strong when clients have related but different data distributions, where attentive pairwise collaboration outperforms global aggregation.
- Experimental setup: Experiments compare FedAMP and HeurFedAMP with SCAFFOLD, APFL, fine-tuned FedAvg and FedProx, FedAvg, FedProx, and Separate across four benchmark data sets.The data sets are MNIST, FMNIST, EMNIST, and CIFAR100; performance is measured by best mean testing accuracy (BMTA).
- Experimental setup: The three evaluation settings are IID, pathological non-IID with two classes per client, and practical non-IID with groups of clients sharing similar data distributions.The practical setting is intended to reflect subgroups of companies with similar distributions that differ from those outside the subgroup.
- IID data setting: Under IID data, FedAvg and FedProx usually perform best, while FedAMP and HeurFedAMP perform much better than FedAvg-FT, FedProx-FT, and SCAFFOLD among personalized methods.Because clients are similar under IID data, FedAMP's attention assigns comparable weights to messages, producing global collaboration similar to FedAvg.
- Pathological non-IID data setting: Under pathological non-IID data, FedAvg and FedProx degrade substantially on FMNIST and EMNIST, and global models catastrophically damage performance on CIFAR100.Global aggregation of differently trained personalized models introduces instability, while FedAMP and HeurFedAMP avoid it by aggregating only similar personalized models.
- Practical non-IID data setting: Under practical non-IID data, FedAMP and HeurFedAMP match SCAFFOLD on MNIST and significantly outperform all other methods on FMNIST, EMNIST, and CIFAR100.They also achieve higher testing accuracy for more individual clients on the challenging data sets, with all reported Wilcoxon tests yielding p-values below 10^-4.
- Practical non-IID data setting: FedAMP and HeurFedAMP recover the three ground-truth collaboration groups in their collaboration-weight matrices, unlike methods based on a single global model.The groups are clients 0-9, 10-35, and 36-61, supporting the role of attentive pairwise collaboration in the practical setting.
8 Conclusions
FedAMP and HeurFedAMP use attentive message passing to strengthen collaboration among similar clients while preserving private data. The mechanism improves learning performance and captures collaboration groups that a single global model cannot represent.
- FedAMP and HeurFedAMP introduce attentive message passing to facilitate collaboration between clients without infringing data privacy.The mechanism enables similar clients to collaborate more strongly than dissimilar clients.
- Figure 2 shows the distribution of testing accuracy across all clients under the practical non-IID setting.
- Figure 3 visualizes FedAMP and HeurFedAMP collaboration weights using client IDs on both axes.
- The mechanism significantly improves learning performance.
Ethics Statement
Data-protection regulations restrict transmission of user data between sources, creating a challenge for data-intensive machine-learning tasks. FedAMP and HeurFedAMP are proposed to train models using private data across multiple data owners while addressing these constraints.
- Data-protection regulations, including Europe’s General Data Protection Regulation, strictly restrict user-data transmission between sources.
- These transmission restrictions are identified as a major challenge for data-intensive machine-learning tasks.
- FedAMP and HeurFedAMP are proposed to securely and efficiently train high-performance AI models using private data held by multiple data owners.