PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Personalized Federated Learning

Alysa Ziying Tan, Han Yu, Lizhen Cui, Qiang Yang

arXiv:2103.00710v3cs.LGcs.AIcs.DC

TL;DR

Privacy-sensitive data are distributed across silos, while conventional FL struggles with heterogeneous data and insufficient personalization. This survey synthesizes PFL motivations, methods, challenges, datasets, metrics, and open directions through a taxonomy organized by personalization strategies. It concludes by identifying architectural design, realistic benchmarking, and trustworthy PFL as promising research trajectories.

  • Problem

    Federated learning must address heterogeneous client data and insufficient personalization while preserving privacy.

  • Method

    The survey proposes a taxonomy of PFL techniques organized by challenges and data-based, model-based, architecture-based, and similarity-based personalization strategies.

  • Results

    The survey synthesizes PFL ideas, challenges, opportunities, public datasets, evaluation metrics, and open research problems.

  • Takeaways & Limitations

    The survey offers a roadmap toward new PFL architectures, realistic benchmarking, and trustworthy personalized federated learning systems.

Abstract

from arXiv · show

In parallel with the rapid adoption of Artificial Intelligence (AI) empowered by advances in AI research, there have been growing awareness and concerns of data privacy. Recent significant developments in the data regulation landscape have prompted a seismic shift in interest towards privacy-preserving AI. This has contributed to the popularity of Federated Learning (FL), the leading paradigm for the training of machine learning models on data silos in a privacy-preserving manner. In this survey, we explore the domain of Personalized FL (PFL) to address the fundamental challenges of FL on heterogeneous data, a universal characteristic inherent in all real-world datasets. We analyze the key motivations for PFL and present a unique taxonomy of PFL techniques categorized according to the key challenges and personalization strategies in PFL. We highlight their key ideas, challenges and opportunities and envision promising future trajectories of research towards new PFL architectural design, realistic PFL benchmarking, and trustworthy PFL approaches.

I. INTRODUCTION

Privacy-sensitive data are increasingly distributed across isolated silos, motivating privacy-preserving AI and federated learning. Personalized federated learning extends FL to address heterogeneous data and limited personalization.

  • Motivation: Sensitive data from edge devices and organizations often remain isolated, while privacy regulations increase demand for privacy-preserving AI.Healthcare data are a prominent example because they are sensitive and distributed across institutions.
  • Federated learning: Federated learning collaboratively trains models across data silos while keeping training data local under a central parameter server.Clients range from individual devices to organizations, and model parameters rather than raw data are exchanged.
  • Federated learning: FL is commonly categorized as horizontal, vertical, or federated transfer learning according to how participants’ feature and sample spaces are distributed.Horizontal FL shares features across participants with different samples, whereas vertical FL uses overlapping samples with different features.
  • Taxonomy: The proposed PFL taxonomy distinguishes data-based, model-based, architecture-based, and similarity-based approaches.The taxonomy organizes personalization strategies for addressing FL limitations.
  • Motivation for PFL: PFL addresses FL’s poor convergence on heterogeneous data and lack of personalized solutions, which can reduce individual-client performance.These challenges arise because local data distributions may differ substantially across clients.

1) Poor Convergence on Heterogeneous Data:

Non-IID client data cause client drift and reduce FedAvg convergence, while a single global model may not personalize adequately for divergent users. The survey organizes PFL strategies and identifies evaluation and research directions for addressing these limitations.

  • Poor Convergence on Heterogeneous Data: FedAvg accuracy decreases on non-IID data because repeated local training and synchronization produce client drift.The resulting server updates can move away from the true global optimum, requiring careful hyperparameter tuning for stability.
  • Poor Convergence on Heterogeneous Data: Under non-IID data, averaged client updates may place the global model far from the true global optimum.The illustration contrasts an averaged model close to the optimum under IID data with one displaced under heterogeneous data.
  • Lack of Solution Personalization: A single globally shared model fits the average client and may generalize poorly to local distributions that differ substantially from the global distribution.Mobile-keyboard users with different demographic, linguistic, and cultural patterns may require tailored predictions.
  • Survey scope: The survey fills a shortage of comprehensive PFL reviews by presenting a systematic perspective on personalized models under statistical heterogeneity.It focuses specifically on PFL rather than general FL concepts, methods, or applications.
  • Survey contributions: Its taxonomy classifies personalization strategies as data-based, model-based, architecture-based, and similarity-based while highlighting assumptions and potential limitations.The survey also discusses public datasets, evaluation metrics, and future architectural, benchmarking, and trustworthy-PFL directions.

II. STRATEGIES FOR PERSONALIZED FEDERATED LEARNING

The survey organizes PFL methods around two strategies: personalizing a global model or learning individual personalized models. These strategies balance collaboration and generalization against client-specific performance.

  • Overview: The survey’s taxonomy organizes PFL methods according to key challenges and personalization strategies.This framework provides the basis for its systematic review of existing approaches.
  • Global Model Personalization: Global model personalization first trains one global FL model and then adapts it locally using each client’s data.This strategy targets performance degradation caused by client drift on non-IID data.
  • Learning Personalized Models: Learning personalized models modifies FL aggregation so that individual models are trained for different clients.These approaches use architecture-based or similarity-based personalization techniques.
  • Optimization objectives: Standard FL minimizes aggregated local losses using one global model, producing a common output for all clients without personalization.With heterogeneous client distributions, this average-loss objective can yield poor performance.
  • Optimization objectives: Local learning trains each client’s model without communication, providing personalization but limiting generalization because local datasets are small.Collaboration can provide stronger generalization by exploiting knowledge across clients.
  • Optimization objectives: PFL seeks a balance between FL’s collaboration and local learning’s personalization.The two extremes differ in knowledge sharing, generalization, and client-specific outputs.

III. STRATEGY I: GLOBAL MODEL PERSONALIZATION

Strategy I personalizes a globally shared FL model to address client drift under heterogeneous data. It includes data-based methods that modify participation or data distributions, alongside a taxonomy spanning data augmentation and client selection.

  • Strategy I trains a single global FL model and then personalizes it for each client through local adaptation.The approach follows a two-step process: FL training followed by additional training on each local dataset.
  • Data-based Approaches: Data-based approaches reduce statistical heterogeneity in client data distributions to mitigate client drift and improve global-model generalization.
  • Data Augmentation: Data augmentation uses synthetic over-sampling, under-sampling, or shared proxy data to make client data more statistically homogeneous.In FL, these methods can require data sharing or a proxy dataset representative of the overall distribution.
  • Client Selection: Client selection samples participating clients to obtain a more homogeneous data distribution and improve model generalization.Some strategies also account for data and resource heterogeneity in edge-computing settings.

B. Model-based Approaches

Model-based global model personalization preserves heterogeneous client information while improving convergence and personalization through regularization, meta-learning, and transfer learning.

  • Model-based approaches seek a strong global model that can produce better personalized models without removing the diversity of client behaviors.
  • Regularized Local Loss: Regularized local loss limits the impact of local updates, improving convergence stability and global-model generalization.Each client minimizes its local loss plus a regularization loss involving the global and local models.
  • Regularized Local Loss: Regularization can compare global and local models or use historical local snapshots to reduce client drift and weight divergence.FedProx uses a proximal term, while SCAFFOLD uses variance reduction; MOON aligns local and global representations and separates current from previous local representations.
  • Meta-learning: Meta-learning optimizes a global initialization that adapts to heterogeneous tasks after a few local gradient steps.Per-FedAvg builds on MAML, while pFedMe adds l2-norm regularization to balance personalization and generalization; second-order gradients can be computationally expensive.
  • Transfer Learning: Transfer learning reduces discrepancy between a trained global source-domain model and a local target-domain model by reusing generic features and fine-tuning task-specific layers.The surveyed procedure trains a global model, reuses it locally, and fine-tunes other layers with local data.
  • Data-based methods may lose information about client diversity and have unresolved privacy concerns, while model-based methods generally require only slight modifications to FedAvg.

IV. STRATEGY II: LEARNING PERSONALIZED MODELS

Strategy II addresses solution personalization by training individual personalized FL models rather than only adapting a single global model.

  • Strategy II modifies the FL model aggregation process and applies different learning paradigms to build personalized models.Its taxonomy divides approaches into architecture-based and similarity-based categories.

A. Architecture-based Approaches

Architecture-based PFL personalizes models through client-specific designs, principally by decoupling private parameters or using knowledge distillation.

  • Architecture-based approaches tailor model designs to clients through parameter decoupling and knowledge distillation.
  • Parameter Decoupling: Parameter decoupling keeps private client parameters local while sharing federated parameters, enabling task-specific representations.Common designs use shared base layers with private personalized layers or private feature representations.
  • Parameter Decoupling: Unlike split learning, parameter decoupling shares federated model parameters with clients, whereas split learning shares split-layer weights and gradients sequentially.Split learning can provide a privacy advantage but is less efficient because clients train sequentially.
  • Knowledge Distillation: Knowledge distillation supports diverse personalized architectures by transferring class-score or logit knowledge toward clients, the server, both, or neighboring clients.
  • Knowledge Distillation: FedMD lets clients use diverse models by repeatedly training on a public dataset using consensus class scores, then fine-tuning on private data.
  • Knowledge Distillation: FedDF groups clients with identical architectures for prototype-model training and then performs cross-architecture learning.This setting accommodates clients with different computational capabilities and model architectures.
  • Knowledge Distillation: Bidirectional distillation trains small edge models and a large server model, but uploading client ground-truth labels creates a potential privacy risk.

B. Similarity-based Approaches

Similarity-based PFL personalizes models by exploiting relationships among clients, including pairwise, group-level, and multi-center relationships. These approaches range from jointly learning related tasks to clustering clients or interpolating global and local models.

  • Similarity-based Approaches: Similarity-based PFL models client relationships so related clients learn similar personalized models through multi-task learning, model interpolation, or clustering.Multi-task learning and interpolation use pairwise relationships, whereas clustering models group-level relationships.
  • Multi-task Learning: Multi-task learning treats each FL client as a task, leveraging relationships in heterogeneous local data to improve generalization across related tasks.MOCHA extends distributed multi-task learning into federated learning using a primal-dual formulation.
  • Model Relationships: FedAMP maintains a personalized cloud model for each client as a weighted combination of local client models, then transfers it for local training.The personalized cloud model is recomputed from client models and used in each communication round.
  • Model Interpolation: Model interpolation balances generalization and personalization by penalizing divergence from the mean model, with λ = 0 yielding fully local learning.APFL instead learns a client-specific mixing parameter to combine global and local models adaptively.
  • Clustering: Clustering trains separate models for homogeneous client groups when client distributions differ substantially, but some methods incur high costs or require a fixed cluster count.Hierarchical clustering can require multiple communication rounds, while IFCA begins with a predetermined number K of models.
  • Architecture-based Approaches: Architecture-based approaches offer personalization through customized models, but parameter decoupling and knowledge distillation face design and capacity-gap challenges.Knowledge distillation may require a representative proxy dataset, while private-versus-federated parameter choices affect the balance between generalization and personalization.

V. PFL BENCHMARK & EVALUATION METRICS

PFL benchmarking uses datasets and evaluation designs that represent diverse tasks and non-IID conditions, but existing studies often create their own partitions instead of adopting shared benchmarks.

  • FL Benchmark Datasets: PFL benchmarking frameworks include FLBench, Edge AIBench, OARF, FedGraphNN, and LEAF, which covers six datasets across image, language, and sentiment tasks.LEAF supports both IID and non-IID settings.
  • PFL Experimental Evaluation Design: Despite available FL benchmarks, most PFL studies simulate non-IID data by repartitioning public datasets or constructing synthetic datasets.Common public datasets include MNIST, EMNIST, and CIFAR-100.
  • PFL Experimental Evaluation Design: PFL evaluation designs are summarized according to the types of non-IID settings simulated and the personalization methods used.The survey organizes these settings in Table III.

1) Quantity Skew:

Quantity skew represents heterogeneity in the amount of local data held by clients, reflecting unequal data availability in real-world federated environments.

  • Quantity Skew: Clients can hold local datasets of different sizes, with some possessing considerably more data than others.This heterogeneity arises from diverse usage patterns across clients.
  • Quantity Skew: Researchers simulate quantity skew by using imbalanced datasets directly or distributing data to clients according to a power law.Both approaches produce unequal local dataset sizes.

2) Feature Distribution Skew:

PFL studies model several forms of distributional heterogeneity and evaluate personalization across model, system, and trustworthy-AI dimensions. However, existing evaluations often cover only one skew type and emphasize accuracy over broader cost-benefit objectives.

  • Feature Distribution Skew: Feature distribution skew occurs when client feature distributions Pc(x) differ while P(y|x) remains shared across clients.It can be simulated with user-partitioned datasets or rotations.
  • Label Distribution Skew: Label distribution skew varies Pc(y) across clients while keeping P(x|y) unchanged, reflecting demographic or linguistic differences among users.Datasets are commonly partitioned by labels to create this setting.
  • Label Preference Skew: Label preference skew varies Pc(x|y) across clients while keeping P(y) constant, and can be simulated by swapping a proportion of labels.This setting represents differences in personal preferences affecting ground-truth labels.
  • Evaluation Scope: Most PFL studies evaluate only one non-IID type, with feature and label distribution skew most common and label preference skew largely limited to clustering methods.The survey calls for community-wide benchmark alignment and standardized evaluation design.
  • Model Performance Metrics: Aggregated accuracy cannot reflect individual personalized models, so evaluations also use client-level, distribution-based, and before-versus-after personalization metrics.Convergence is assessed through loss, communication rounds, local epochs, or formal convergence bounds.
  • Holistic Evaluation: PFL evaluation should jointly consider model performance, system costs, and trustworthy-AI attributes rather than focusing primarily on accuracy gains.Relevant system dimensions include communication, computation, scalability, and heterogeneity.

VI. PROMISING FUTURE RESEARCH DIRECTIONS

The survey identifies future PFL directions spanning architecture design, privacy-preserving heterogeneity analysis, adaptability, aggregation, and robustness. These directions target heterogeneous, dynamic, and operationally variable federated environments.

  • Future Research Directions: The survey envisions future PFL research on new architectures, realistic benchmarking, and trustworthy approaches as practical applications demand better personalization.These directions are presented as promising trajectories for the field.
  • Client Data Heterogeneity Analytics: Privacy-preserving analysis of client data heterogeneity remains an open problem because existing distribution metrics require access to raw data.Total Variation, 1-Wasserstein, and Earth Mover’s Distance quantify statistical heterogeneity but cannot be calculated without raw data.
  • Aggregation Procedure: Specialized aggregation procedures are needed because averaging may be unsuitable for complex PFL scenarios and remains insufficiently studied theoretically.A layer-wise matched averaging formulation has been proposed for CNN and LSTM architectures.
  • PFL Architecture Search: Neural Architecture Search could reduce manual effort in selecting architectures suited to heterogeneous client distributions.Federated neural architectures can be sensitive to hyperparameter choices and may perform poorly when not carefully tuned.
  • Spatial Adaptability: PFL systems must adapt spatially to new clients, dropouts, and stragglers under heterogeneous edge-device capabilities.Existing approaches commonly assume a fixed client pool, while communication-efficient methods such as gradient compression and asynchronous updates address related bottlenecks.
  • Temporal Adaptability: Temporal adaptability requires PFL systems to learn from non-stationary data affected by concept drift.Concept-drift handling involves detection, understanding, and adaptation; leveraging existing algorithms for PFL remains an open direction.

B. Opportunities for PFL Benchmarking

The survey identifies realistic data, realistic non-IID settings, and holistic metrics as priorities for meaningful PFL benchmarking, while outlining trustworthy and collaborative research directions.

  • Benchmarking: Realistic PFL datasets should include more modalities and a broader range of machine-learning tasks from real-world applications.
  • Benchmarking: Current PFL evaluations usually model only one non-IID setting, motivating deeper study and simulation of real-world distribution shifts such as temporal skew and adversarial attackers.
  • Benchmarking: Systematic evaluation should jointly consider model performance, system performance, and Trustworthy AI attributes through holistic cost-benefit analysis.
  • Open Collaboration: Future PFL research should promote collaboration among self-interested data owners, including incentive mechanisms based on game theory, pricing, and auctions.
  • Trustworthy PFL: Trustworthy PFL remains open across fairness, explainability, privacy, and robustness, including client-induced bias, privacy risks from explanations, and vulnerabilities in complex PFL protocols.The survey calls for frameworks balancing explainability and privacy and for defensive strategies against attacks.
  • Survey Roadmap: The survey organizes PFL techniques by challenges and personalization strategies, reviews datasets and metrics, and proposes future directions in architecture, realistic benchmarking, and trustworthy approaches.
Loading 2103.00710v3…