PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Personalized Federated Learning

Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi

arXiv:2003.13461v3cs.LGcs.DCstat.ML

TL;DR

Global-model-focused federated learning can limit local personalization, particularly when client data are heterogeneous. APFL mixes local and global models and adapts the mixing weight per client. The paper provides generalization and convergence analyses and reports experiments supporting its approach.

  • Problem

    A single global model may generalize poorly on individual clients’ data when client distributions are heterogeneous, limiting local personalization.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    APFL collaboratively learns a global model and client-specific models by adaptively mixing local and global models.

    Source: §4 · p. 7, §8 · p. 20

  • Results

    The paper reports experiments supporting its theoretical findings, and states that the local ERM model’s risk can be strictly worse than the personalized model’s.

    Source: §7 · p. 18, §8 · p. 20

  • Takeaways & Limitations

    The analysis identifies data-distribution diversity and per-device sample counts as key factors in the generalization of personalized models.

    Source: §8 · p. 20

    In the nonconvex setting, the convergence analysis assumes K = n, and alleviating this assumption remains open.

    Source: §5.2 · p. 12

Abstract

from arXiv · show

Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize. In this paper, we advocate an adaptive personalized federated learning (APFL) algorithm, where each client will train their local models while contributing to the global model. We derive the generalization bound of mixture of local and global models, and find the optimal mixing parameter. We also propose a communication-efficient optimization method to collaboratively learn the personalized models and analyze its convergence in both smooth strongly convex and nonconvex settings. The extensive experiments demonstrate the effectiveness of our personalization schema, as well as the correctness of established generalization theories.

1 Introduction

Federated global models can generalize poorly to individual clients as their data become more heterogeneous. APFL instead combines local and global models with a client-specific mixing weight and provides theory and experiments for this approach.

1 Introduction

  • As client data diversity increases, FedAvg and SCAFFOLD global models generalize poorly on local data, while the proposed personalized model is less affected.Figure 1 compares generalization and training losses on MNIST with logistic regression.Source: §1 · p. 1, Figure 1 · p. 2
  • APFL mixes optimal local and global models with an adaptive client-specific weight, and analyzes the mixture’s generalization and communication-efficient optimization.The analysis relates generalization to the mixing parameter, distribution divergence, and local and global training data.Source: §1 · p. 2
  • The paper reports experiments supporting its personalization approach and generalization theories, comparing against global and localized federated-learning models.The introduction states that the experiments demonstrate the approach’s superiority over those models.Source: §1 · p. 2

2 Related Work

Prior federated personalization work spans fine-tuning, multi-task learning, contextualization, regularization, and model interpolation. This paper situates its approach among interpolation methods and emphasizes adapting each client’s degree of personalization to data distribution.

Local fine-tuning

  • Local fine-tuning adapts a received global model using client data, but the cited discussion distinguishes minimizing optimization error from achieving personalized-model generalization.Other described variants include MAML-style updates and changing only a personalized layer.Source: §2 · p. 3

Multi-task learning, contextualization, and regularization

  • Other personalization approaches include multi-task learning and client clustering, context-dependent models, and regularizing differences between local and global models.The cited regularization work includes approaches applied to model predictions.Source: §2 · pp. 3–4

Personalization via model interpolation

  • Mansour et al. (2020) propose model interpolation with generalization guarantees, while this paper says their bound does not show the advantage of mixing models.The paper says its analysis makes the bound depend on the mixture parameter, data diversity, and optimal local and global models.Source: §2 · p. 4
  • The paper frames the degree of personalization as dependent on client data distributions and proposes adaptively tuning that degree for each client.It states that personalization can help balance shared and local knowledge under non-IID data, while IID data does not benefit from it.Source: §2 · p. 4

3 Personalized Federated Learning

The section formulates personalized federated learning as combining a global model trained over clients with client-specific learning. It gives a generalization bound and an optimal mixing rule that reflect data heterogeneity and sample counts.

3 Personalized Federated Learning

  • The global model minimizes empirical risk over the average of clients’ data distributions, while each client has its own distribution and local risk.The average distribution is defined as the mean of the client distributions.Source: §3 · p. 4

3.1 Personalized model

  • When client distributions are heterogeneous, the global model may generalize poorly locally; the personalized model mixes global and local models with an adaptive weight.The mixture aims to use shared information while limiting harm from distribution heterogeneity.Source: §3.1 · pp. 4–5
  • The mixed personalized model need not minimize the client’s empirical risk and can retain residual training risk on that client’s data.The local model is optimized while partially incorporating the global model.Source: §3.1 · p. 5

3.2 Generalization guarantees

  • The generalization bound depends on global and local sample counts and the divergence between a client distribution and the average distribution.The discussion says high divergence can make the global model hurt local generalization, while limited local samples can make a local model generalize poorly.Source: §3.2 · pp. 5–6
  • The optimal mixing parameter lies in [0, 1], increases toward local-model emphasis with greater distribution divergence, and favors more global-model contribution when local samples are scarce.With more local samples, the discussion says the optimal choice shifts toward the local model.Source: §3.2 · p. 6
  • Unlike the cited Mansour et al. (2020) bound, this paper’s bound depends on α_i, which makes the effect of model mixing visible in the analysis.The paper identifies this dependence as showing the advantage of the personalization scheme.Source: §3.2 · p. 6

4 Optimization Method

The optimization method jointly learns a shared model and client-specific models. Local Descent APFL alternates client-side updates with server aggregation, while clients adapt their mixture weights during optimization.

4 Optimization Method

  • The method jointly learns a global model and personalized models, with each client’s mixing parameter balancing local learning against the shared model according to distribution alignment.The section motivates a local descent approach to optimize both objectives simultaneously.Source: §4 · pp. 6–7

4.1 Local Descent APFL

  • Local Descent APFL has selected clients maintain global, local, and mixed models, perform τ local updates, and send their global-model versions for server averaging.The server broadcasts the aggregated model to another selected client set in the next round.Source: §4.1 · p. 7

4.2 Adaptive α update

  • Because the distribution distance needed for the theoretical optimum is usually unknown, clients adapt α_i empirically by gradient descent on their local objective.The update is based on the relationship between the model difference and the gradient at the personalized model; when personalized and global models are close, α changes little.Source: §4.2 · pp. 7–8

5 Convergence Analysis

For fixed α, the section analyzes Local Descent APFL under smooth strongly convex and nonconvex losses, using gradient diversity and, in the strongly convex case, the local–global optimality gap to characterize heterogeneity. It gives convergence guarantees for global and personalized models, with qualifications involving residual error and communication.

5.1 Strongly Convex Loss

  • With τ = T/K, the strongly convex convergence rate recovers O(1/KT), matching the rate of local SGD with periodic averaging.The obtained global-model rate matches FedAvg’s convergence rate.Source: §5.1 · p. 10
  • Under smooth strongly convex assumptions, Theorem 3 establishes personalized-model convergence for all clients with a lower bound on α_i and a specified diminishing learning rate.The theorem also specifies an averaging scheme for the iterates.Source: §5.1 · p. 10
  • The strongly convex residual error depends on gradient diversity and the local–global optimality gap, scaled by 1 − α_i.These quantities characterize heterogeneity across local functions and the gap between local and global optima.Source: §5 · p. 9, §5.1 · p. 11
  • Theorem 4 removes the lower-bound condition on α_i, but its residual error uses G instead of the more precise ζ_i and Δ_i quantities.The bounded-gradient assumption requires parameters in a bounded domain, implemented with projection.Source: §5.1 · p. 11

5.2 Nonconvex Loss

  • For nonconvex losses, τ = n^-1/4 T^1/4 yields n^3/4T^3/4 communication rounds, while the convergence-rate speedup applies only to partial terms and assumes K = n.Alleviating the K = n assumption and obtaining broader speedup remain open problems.Source: §5.2 · p. 12

6 Experiments

The experiments assess APFL’s optimization and generalization on training and local validation data, including whether personalized models improve clients’ validation performance.

6.1 Experimental setup

  • Experiments use logistic regression, CNN, and MLP models across MNIST, CIFAR10, EMNIST, and synthetic datasets with varied client heterogeneity.MNIST and CIFAR10 include manually partitioned non-IID data; EMNIST represents natural heterogeneity.Source: §6 · p. 13, §6.1 · pp. 13–14

6.2 Experimental results

  • Results report optimization on training data and generalization on local validation data, averaging localized and personalized model performance over online clients.Experiments use 100 clients except EMNIST, which uses 1,000; localized models receive τ = 10 local SGD steps unless otherwise stated.Source: §6.2 · p. 14

Strongly convex loss.

  • For non-IID MNIST, α = 0.25 gives the best generalization; as data approaches IID, personalization’s effect fades and global FedAvg or SCAFFOLD generalizes better.Increasing α lowers training loss, while global models have higher local training loss.Source: Strongly convex loss · pp. 14–15, Figure 2 · p. 15
  • Across sampling rates of 0.3, 0.5, and 0.7, APFL outperforms localized FedAvg in training and generalization; larger α can lessen sampling’s negative impact.Lower sampling rates hurt FedAvg’s training and generalization performance.Source: Strongly convex loss · p. 15, Figure 3 · p. 16

Adaptive α update.

  • Across synthetic datasets with different diversity levels, adaptive α yields nearly the same APFL generalization performance, while FedAvg has a large gap.APFL also outperforms FedAvg in training on the same datasets.Source: Adaptive α update · pp. 15–16, Figure 4 · p. 16

Nonconvex loss.

  • After 100 communication rounds on CIFAR10 with a CNN, APFL’s personalized model outperformed localized and global FedAvg and SCAFFOLD models in optimization and generalization; adaptive α achieved the best training loss, while α = 0.25 achieved the best generalization.Source: Table 1 · p. 17, Nonconvex loss · p. 17
  • On naturally heterogeneous EMNIST, APFL with adaptive α matches localized FedAvg’s training loss and achieves higher validation performance.The experiment uses an MLP and 1,000 clients.Source: Nonconvex loss · p. 17, Figure 5 · p. 17
  • 100 rounds: APFL models outperform FedAvg, perFedAvg, and pFedMe in training loss and generalization accuracy on the MNIST MLP comparison.Among APFL models, α = 0.75 has the lowest training loss, while adaptive α has the best validation accuracy.Source: Table 2 · p. 18, Nonconvex loss · p. 18

7 Discussion and Extensions

The discussion connects personalization to distributional and optimization differences between local and global models. It also considers when mixing helps, adaptation for newly joined clients, and a more difficult agnostic extension.

Connection between learning guarantee and convergence

  • The generalization analysis links distribution divergence to gradient diversity in the empirical-loss convergence rate, while local/global optimal-solution divergence also enters the discussion.The bound includes terms tied to the hypothesis class and data distributions.Source: §7 · p. 18

Why APFL is “Adaptive”

  • APFL's adaptive mixing responds to local-distribution drift: when drift is large, the global model contributes less to local generalization, so α should increase.The update of α depends on the correlation between the local gradient and the deviation between local and global models; experiments report better performance than fixed α.Source: §7 · p. 18

Comparison with local ERM model

  • The personalized model can have lower risk than local ERM, and the discussion identifies conditions under which mixing can benefit a local model.The cited discussion says the local sample count should be sufficiently large and divergence and global-model generalization error small for this benefit.Source: §7 · pp. 18–19

Personalization for new participant nodes

  • For three clients joining after 100 communication rounds with 97 clients, APFL personalization outperforms FedAvg fine-tuning in training loss and validation accuracy, including with increased diversity.The comparison uses five epochs of updates on synthetic data; the figure passage reports that increasing diversity makes personalization harder.Source: §7 · pp. 19–20, Figure 6 · p. 20

Agnostic global model

  • Agnostic APFL is left for future work because its global empirical risk is evaluated on a different domain and its resulting minimax problem complicates convergence analysis.The risk upper bound derived for hαi does not hold in that setting.Source: §7 · p. 20

8 Conclusion and Future Work

The paper concludes by presenting APFL as a mixture of local and global models, with generalization analysis, communication-reduced optimization, and experiments supporting the approach.

8 Conclusion and Future Work

  • APFL learns each personalized model as a mixture of local and global models.The paper gives generalization guarantees whose key factors include client-distribution diversity and per-device sample counts.Source: §8 · p. 20
  • The proposed communication-reduced optimization method is analyzed for smooth strongly convex and nonconvex functions.Source: §8 · p. 20
  • Experiments in a federated setting provide empirical support for the theoretical results.Source: §8 · p. 20

A Proof of Generalization Bound

The proof bounds the mixed model's risk on a client's distribution by decomposing it and controlling its excess risk. It also flags a gap in how the mixture's drift is bounded, then transitions to convergence analysis with client sampling.

A Proof of Generalization Bound

  • The mixed-model risk proof uses convexity to split the local risk into components, then bounds them using uniform VC-dimension error bounds and a bounded-loss assumption.The proof considers squared hinge loss for classification and MSE for regression; bounded loss can hold when data and hypothesis parameters are bounded.Source: A Proof of Generalization Bound · pp. 24–26
  • The proof characterizes the mixed model's drift from the local empirical risk minimizer through hypothesis capacity, but leaves a tighter drift bound for future work.λH(S) quantifies empirical loss discrepancy over S among pairs of hypotheses in H.Source: A Proof of Generalization Bound · p. 24

B Proof of Convergence

  • The convergence proof first treats full participation, then generalizes to uniform random sampling of K clients, introducing variance that depends on K.Source: B Proof of Convergence · p. 26

B.1 Proof without Sampling

The appendix establishes convergence results for local descent APFL without client sampling, using lemmas that bound gradient gaps and local/global model deviation. It then extends the proof pipeline to sampled clients, with sampling-size-dependent variance.

B.1 Proof without Sampling

  • Without sampling, the appendix states convergence theorems for both local and global models under Assumptions 1–3 and specified mixing-weight, learning-rate, and averaging choices.The passages provide theorem conditions but not the complete convergence expressions.Source: §B.1 · p. 26

B.1.1 Proof of Useful Lemmas

  • The useful lemmas bound the per-iteration gap between local and global gradients and the deviation between each local version of the global model and the global model.Source: §B.1.1 · p. 27

B.1.2 Proof of Theorem 8

B.1.3 Proof of Theorem 7

  • The local-model convergence proof bounds several terms, applies the deviation and gradient-gap lemmas, and telescopes the resulting inequality.Source: §B.1.3 · pp. 31–33

B.2 Proof with Sampling

  • The sampled-client proof follows the same pipeline as the no-sampling proof but adds variance that depends on sampling size K.Source: §B.2 · p. 35

B.2.1 Proof of Useful Lemmas

The appendix proves bounds on gradient disagreement and local-model deviation, then uses them in convergence proofs. For partial participation, it models client updates with an indicator and accounts for added variance through hyperparameter tuning.

B.2.1 Proof of Useful Lemmas

  • Lemma 5 bounds the gap between local and global gradients at each Algorithm 1 iteration.The supplied passage identifies the bound but does not include its expression.Source: §B.2.1 · p. 36
  • Lemma 6 bounds the deviation between each local version of the global model and the global model at each iteration.The bound is stated for Algorithm 1 with sampling; its expression is not included in the supplied passage.Source: §B.2.1 · p. 36

B.2.2 Proof of Theorem 2

  • Assuming each client’s objective function satisfies Assumption 3-2, the passage applies Algorithm 1 with learning rate ηt = 16 µ(t+a) and κ = L/µ.Source: §B.2.1 · p. 37

B.2.3 Proof of Theorem 3

  • For Theorem 3, partial participation updates K of n devices sampled uniformly at random, modeled with an indicator for each client’s selection.The proof describes the stochastic gradient as unbiased but notes added variance, addressed by tuning hyperparameters.Source: §B.2.3 · p. 38

C Proof of Convergence without Assumption on αi

This appendix section introduces the proof of Theorem 4 and then turns to convergence results for nonconvex functions, beginning with useful lemmas.

C Proof of Convergence without Assumption on αi

D Proof of Nonconvex Loss Convergence

D.1 Proof of Technical Lemmas

The technical lemmas underpin the proofs of Theorems 5 and 6. The nonconvex convergence proof combines these lemmas with bounds developed under smoothness and synchronization conditions.

D.1 Proof of Technical Lemmas

  • The technical lemmas derive bounds from update rules and smoothness, with some arguments summing over synchronization stages.The proofs also use expectation and telescoping sums; the supplied passages omit the lemmas’ full statements.Source: §D.1 · pp. 44, 46, 47, 48

D.2 Proof of Theorem 5

  • In the cited proof, the argument proceeds according to Lemma 9, then plugging in Lemma 10 is said to conclude the proof.Source: §D.2 · p. 49

Related papers

Loading 2003.13461v3…