PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning

Jun Luo, Shandong Wu

arXiv:2110.08394v3cs.LG

TL;DR

Non-IID client data challenge one-model-fits-all federated learning, motivating personalized cross-silo FL. APPLE learns client-specific benefits from other clients’ models and regulates global versus local training, achieving state-of-the-art performance across the evaluated datasets and settings.

  • Problem

    Non-IID data distributions make a single global federated model inadequate for some cross-silo clients.

  • Method

    APPLE learns client-specific directed relationships over other clients’ core models and uses a proximal penalty to control global and local objectives.

  • Results

    APPLE achieves state-of-the-art performance across two benchmark and two medical imaging datasets under two non-IID settings.

  • Takeaways & Limitations

    APPLE enables clients to use more information from clients with similar distributions while mitigating potentially harmful transfer.

  • Takeaways & Limitations

    The cross-silo formulation assumes a relatively small number of stateful clients, while downloading all core models can still impose considerable communication cost.

Abstract

from arXiv · show

Conventional federated learning (FL) trains one global model for a federation of clients with decentralized data, reducing the privacy risk of centralized training. However, the distribution shift across non-IID datasets, often poses a challenge to this one-model-fits-all solution. Personalized FL aims to mitigate this issue systematically. In this work, we propose APPLE, a personalized cross-silo FL framework that adaptively learns how much each client can benefit from other clients' models. We also introduce a method to flexibly control the focus of training APPLE between global and local objectives. We empirically evaluate our method's convergence and generalization behaviors, and perform extensive experiments on two benchmark datasets and two medical imaging datasets under two non-IID settings. The results show that the proposed personalized FL framework, APPLE, achieves state-of-the-art performance compared to several other personalized FL approaches in the literature. The code is publicly available at https://github.com/ljaiverson/pFL-APPLE.

1 Introduction

Non-IID data can make one global federated model perform poorly across cross-silo clients, motivating personalized FL. APPLE adaptively weights other clients’ core models and controls global-local training, achieving state-of-the-art results across the evaluated settings.

  • Data heterogeneity across clients can cause federated models to perform poorly in particular silos, especially in cross-silo applications.The paper highlights differing user preferences, acquisition protocols, and demographics as sources of heterogeneity.
  • APPLE adaptively personalizes each client model by learning how much it can benefit from other clients’ models without accessing their data distributions.Each client uses a unique directed relationship vector to weight downloaded core models.
  • APPLE flexibly controls training between global and local objectives through a dynamic penalty on directed relationship vectors.This penalty is introduced as a mechanism for balancing collaboration and personalization.
  • APPLE achieves state-of-the-art performance against other personalized FL approaches on two benchmark and two medical imaging datasets under two non-IID settings.

2 Related Work

Related personalized FL methods address heterogeneous client distributions through fine-tuning, meta-learning, multi-task learning, clustering, or model interpolation. These approaches differ in whether they adapt a global model, learn client relationships, or combine local and global models.

  • Robust global-model methods address non-IID training through proximal penalties, gradient correction, or dynamically updated regularization.
  • Personalized FL permits different client models when data come from distinct distributions, addressing limitations of a single global consensus model.
  • Existing methods fine-tune global models or connect FL with meta-learning and multi-task learning to adapt across client distributions.
  • Interpolation-based methods construct client models by weighting global and local models, estimating personalized weights, or using attention-based prox-centers.

3 Adaptive Personalized Cross-Silo Federated Learning

APPLE formulates personalized cross-silo FL around client-specific models assembled from locally trained core models and weighted downloaded cores. It learns directed relationships while using a proximal, scheduled penalty to balance collaboration with personalization, subject to communication and cross-silo assumptions.

  • 3.1 Problem Formulation: APPLE’s cross-silo formulation assumes relatively few stateful clients and does not require client selection at the start of each round.
  • 3.2 Adaptively Learning to Personalize: APPLE forms each personalized model as a client-specific convex combination of downloaded core models using learnable directed relationship weights.Each client has its own DR vector, while downloaded core models are held fixed during local updates.
  • 3.2 Adaptively Learning to Personalize: Clients upload core models rather than personalized models, and DR vectors remain local, limiting what other parties can infer about personalization.
  • 3.3 Proximal Directed Relationships: A proximal DR penalty prevents the learned relationship matrix from collapsing toward identity and encourages collaboration between clients.Without constraint, heterogeneous data can make off-diagonal relationships too small and cause training to resemble individual learning.
  • 3.3 Proximal Directed Relationships: The penalty coefficient and loss scheduler control the transition between global and local objectives, with large proximal strength making FedAvg a special case.The scheduler decays with training rounds so models can first learn shared features and later emphasize personalization.
  • Downloading all clients’ core models can impose substantial communication overhead, motivating communication-reduction techniques.The worst-case per-round download cost can be N times that of methods downloading one model per client.

4 Experiments

APPLE is evaluated on four image datasets under pathological and practical non-IID settings, with convergence, client-level generalization, communication limits, and directed relationships examined. Across settings, APPLE achieves the strongest reported overall performance while adapting relationships to client similarity.

  • Experimental Results: APPLE achieves the highest BMCTA across all datasets and non-IID settings, outperforming the other compared personalized FL methods.The comparison covers pathological and practical non-IID settings; Table 1 reports BMCTA for all four datasets.
  • Experimental Results: Under pathological non-IID data, separate training can remain competitive, while direct FedAvg averaging and fine-tuning provide little boost over separate training.The setting gives clients few classes and little similarity in their data distributions, limiting the benefit of a shared global model.
  • Experimental Results: Under practical non-IID data, APPLE reaches state-of-the-art performance across all settings despite fine-tuning methods outperforming some personalized alternatives.The practical setting requires more careful integration of global information because clients have more complicated class distributions and local majority classes.
  • Experimental Results: With restricted downloads, APPLE outperforms FedFomo on CIFAR10 and OrganMNIST(axial) but not PathMNIST under pathological non-IID data, and wins across practical settings.The comparison varies the maximum downloaded models per client per round over M = 11, 7, 5, 2, 1.
  • Experimental Results: Selecting the top M core models by expected client benefit means that downloading fewer models does not necessarily reduce APPLE’s performance.The model-selection rule prioritizes core models with the highest estimated chance of benefiting each client.
  • Experimental Results: Directed relationships emphasize self-models and increase toward clients with similar data distributions, as illustrated by PathMNIST relationships p1,6 and p1,8.For CIFAR10, high within-class variance corresponds to large self-relationships and small relationships to other clients.

5 Conclusions

The paper concludes that APPLE adaptively quantifies how much clients benefit from one another and controls global-local training through a proximal directed-relationship penalty. Across four datasets and two non-IID settings, it shows superior performance, useful adaptation to distribution similarity, and state-of-the-art results under limited communication.

  • Conclusions: APPLE adaptively learns directed relationships that quantify how much each client can benefit from other clients’ models.The framework is evaluated for convergence and generalization across four image datasets under two non-IID settings.
  • Conclusions: The proximal directed-relationship penalty controls the training focus between global and local objectives.
  • Conclusions: APPLE shows overall superior effects over related personalized FL alternatives and reaches state-of-the-art performance under limited communication budgets.
  • Conclusions: The learned relationships favor clients with similar distributions while mitigating non-beneficial influences from clients with drastically different distributions.

A.1 Loss Scheduler for Proximal Directed Relationships

APPLE uses a monotonically decreasing loss scheduler to emphasize global information early and local training later. The scheduler reaches zero after a designated round and can transition between objectives at different rates.

  • Loss Scheduler: The loss scheduler λ(r) decreases monotonically from 1 to 0, shifting training emphasis from global information toward local training.The scheduler is defined over the current training round r, and its value remains zero after round L.
  • Loss Scheduler: The scheduler’s decay form is treated as a hyperparameter, with candidate schedules including an exponentially decreasing form using ϵ = 10^-3.The exponential schedule is described as producing a rapid transition from global to local training.
  • Loss Scheduler: Figure 7 compares the scheduler designs by showing how their values change across training rounds before reaching zero after L.

A.2 Core Model Selection under Limited

APPLE selects core models under a per-client communication budget by converting directed-relationship magnitudes into sampling probabilities. An exponential schedule increases confidence in these relationship-based contributions as training proceeds.

  • A.2 Core Model Selection under Limited: The shared power base depends on the current round and the mean number of downloads per core model.The base is defined using b(r), with rM/N representing the mean downloaded times over the first r rounds.
  • A.2 Core Model Selection under Limited: Core models are selected probabilistically from normalized powers based on each client's directed-relationship magnitudes.The exponents are |p_i,j|, so larger relationship magnitudes yield greater selection probability.
  • A.2 Core Model Selection under Limited: The exponential design reflects growing confidence in relationship magnitudes as training progresses.Early rounds give core models greater potential to update, so the relationship estimates are treated as less reliable initially.

B.1 Datasets

The experiments partition each dataset into pathological and practical non-IID settings, differing in how samples are assigned across clients and how local labels are distributed. Under the practical setting, APPLE and FedFomo show advantages over other compared personalized methods and similarly fast convergence.

  • B.1 Datasets: Datasets are divided into pathological and practical non-IID distributions for evaluation.Figures 4 and 5 characterize both where class images are assigned and each client's local label distribution.
  • B.1 Datasets: Practical non-IID experiments favor APPLE and FedFomo over other compared personalized methods, with similarly fast convergence.The practical setting includes varied categories, imbalanced majority classes, and client datasets of different sizes.

B.3 Visualization of Directed Relationships Throughout Training

The practical non-IID visualization examines how directed-relationship vectors evolve during training. APPLE can retain substantial relationships with clients lacking a shared majority class by optimizing the joint contribution of downloaded core models.

  • B.3 Visualization of Directed Relationships Throughout Training: Figure 8 visualizes directed-relationship-vector trajectories under the practical non-IID setting.The analysis notes that the 80% × 1, 10% × 1, 1% × 10 split makes relationships harder to infer directly from client data distributions.
  • B.3 Visualization of Directed Relationships Throughout Training: APPLE can assign a large relationship to a client without a shared majority class when its core model contributes beneficially.For OrganMNIST (axial), p1,8 can remain large even though clients 1 and 8 share no majority class.
  • B.3 Visualization of Directed Relationships Throughout Training: FedFomo may reduce another client's model weight when differing majority classes make its personalized model perform poorly locally.This validation-based weighting can fail to maximize the global information learned across clients.

B.4 Additional Implementation Details

The experiments use established image datasets, a four-layer CNN, and fixed-round local training with SGD. APPLE-specific scheduler, regularization, and communication settings are selected from stated hyperparameter grids.

  • B.4 Additional Implementation Details: For MedMNIST, the original training and test sets are combined and repartitioned into 80% training and 20% testing data.MNIST and CIFAR10 retain their pre-existing training and test sets.
  • B.4 Additional Implementation Details: The CNN contains two 5 × 5 convolutional layers followed by fully connected layers of 500 units and the number of classes.The final layer width matches the number of classes.
  • B.4 Additional Implementation Details: The model is trained for 160 rounds with 5 local epochs per round, batch size 256, and SGD momentum 0.9.Learning rates and decay values are selected from the reported grids.
  • B.4 Additional Implementation Details: APPLE tunes scheduler type, μ, and L from cosine or exponential scheduling, μ values, and training-round fractions.The candidate L values are 10%, 20%, and 30% of total training rounds.
Loading 2110.08394v3…