PaperlayerView exact revision on arXiv

Source-linked AI summary

Heterogeneous Federated Learning: State-of-the-art and Research Challenges

Mang Ye, Xiuwen Fang, Bo Du, Pong C. Yuen, Dacheng Tao

arXiv:2307.10616v2cs.LGcs.AIcs.CV

TL;DR

Existing federated-learning research largely assumes homogeneous models, whereas practical clients differ in data, architectures, networks, and hardware. This survey systematically analyzes HFL challenges, reviews methods through a three-level taxonomy, and discusses open directions, while noting the lack of widely recognized benchmarks.

  • Problem

    Practical federated learning involves heterogeneous data distributions, model architectures, network environments, and hardware, but existing work mainly focuses on homogeneous settings.

  • Method

    The survey summarizes HFL challenges, reviews recent papers, and classifies approaches into data-level, model-level, and server-level methods.

  • Results

    The survey provides a comprehensive taxonomy and systematic analysis of HFL challenges, existing techniques, and future research directions.

  • Takeaways & Limitations

    The taxonomy and discussions are intended to support understanding of HFL and promote further development in the field.

  • Takeaways & Limitations

    HFL currently lacks widely recognized benchmark datasets and benchmark testing frameworks.

Abstract

from arXiv · show

Federated learning (FL) has drawn increasing attention owing to its potential use in large-scale industrial applications. Existing federated learning works mainly focus on model homogeneous settings. However, practical federated learning typically faces the heterogeneity of data distributions, model architectures, network environments, and hardware devices among participant clients. Heterogeneous Federated Learning (HFL) is much more challenging, and corresponding solutions are diverse and complex. Therefore, a systematic survey on this topic about the research challenges and state-of-the-art is essential. In this survey, we firstly summarize the various research challenges in HFL from five aspects: statistical heterogeneity, model heterogeneity, communication heterogeneity, device heterogeneity, and additional challenges. In addition, recent advances in HFL are reviewed and a new taxonomy of existing HFL methods is proposed with an in-depth analysis of their pros and cons. We classify existing methods from three different levels according to the HFL procedure: data-level, model-level, and server-level. Finally, several critical and promising future research directions in HFL are discussed, which may facilitate further developments in this field. A periodically updated collection on HFL is available at https://github.com/marswhu/HFL_Survey.

1 INTRODUCTION

Heterogeneous federated learning addresses practical settings where clients differ in data, models, communication networks, and device capabilities. The survey organizes these challenges, reviews existing methods, and identifies future research directions.

  • Research challenges: HFL arises when participating clients have different data distributions, model structures, communication networks, and device conditions.These differences make federated collaboration challenging and define four major heterogeneity categories.
  • Statistical heterogeneity: Statistical heterogeneity includes Non-IID or unbalanced client data, which can produce inconsistent optimization directions and biased collaboration.Existing methods attempt to address this challenge from multiple perspectives.
  • Model heterogeneity: Model heterogeneity reflects clients’ different tasks and requirements, leading them to develop local models independently and creating knowledge-transfer barriers.Standard model aggregation or gradient operations may not apply across heterogeneous models.
  • System heterogeneity: Communication and device heterogeneity introduce inconsistent synchronization, limited learning efficiency, and stragglers caused by unequal storage and computation capabilities.These issues become especially consequential when the number of clients is large.
  • Survey contributions: The survey systematically summarizes challenges, reviews state-of-the-art methods with advantages and limitations, and discusses unresolved problems and future directions.Its new taxonomy classifies HFL approaches at the data, model, and server levels.
  • Future directions: The survey emphasizes efficiency, effectiveness, privacy, fairness, and robust knowledge use as continuing priorities for heterogeneous scenarios.It also notes that broadly recognized benchmark datasets and testing frameworks remain lacking.

2 PROBLEMS: RESEARCH CHALLENGES IN HETEROGENEOUS FEDERATED LEARNING

Heterogeneous federated learning faces challenges from statistical, model, communication, and device heterogeneity, along with knowledge-transfer barriers and privacy leakage. Statistical heterogeneity includes four Non-IID skew patterns that can make local objectives diverge from the global objective.

  • Research challenges: Heterogeneous federated learning involves statistical, model, communication, and device heterogeneity, plus additional knowledge-transfer and privacy challenges.These challenges arise from differing data distributions, model architectures, network environments, and device capabilities.
  • Model heterogeneity: Model heterogeneity is divided into partial and complete heterogeneity based on the architectural differences among participating clients.Complete heterogeneity is described as a special case of partial heterogeneity in which all participant network structures differ.
  • Communication and device heterogeneity: Communication heterogeneity reflects network differences that create high communication costs and low efficiency, while device heterogeneity produces stragglers and computation constraints.Device hardware differences in CPU, memory, and battery life lead to unequal storage and computation capabilities.
  • Statistical heterogeneity: Statistical heterogeneity is categorized as label, feature, quality, or quantity skew according to differences in clients’ data distributions.The survey distinguishes these four Non-IID patterns from a distribution perspective.
  • Statistical heterogeneity: Different local data distributions make client optimization objectives inconsistent with the global objective and can drive local models toward different optima.The survey notes that federated performance may become worse than local learning without federated communication.
  • Statistical heterogeneity: Label skew changes clients’ label distributions, whereas feature skew changes their feature distributions, with both reflecting distinct Non-IID conditions.Label distribution skew can occur even when conditional feature distributions are shared; feature skew concerns differing feature distributions across clients.

3 METHODS: STATE-OF-THE-ART

The survey organizes heterogeneous federated learning methods into data-level, model-level, and server-level approaches. These levels respectively address data heterogeneity or privacy, model design and optimization, and server-supported participation or clustering.

  • Taxonomy: Existing heterogeneous federated learning approaches are divided into data-level, model-level, and server-level methods.The taxonomy follows operations performed at different stages of the federated learning procedure.
  • Data-level methods: Data-level methods smooth statistical heterogeneity across clients or improve privacy through operations such as data augmentation and anonymization.Their operations target local data before or alongside federated model training.
  • Model-level methods: Model-level methods address heterogeneous architectures through techniques including partial-structure sharing and model optimization.These methods operate directly on model representations or training procedures.
  • Server-level methods: Server-level methods require server engagement through mechanisms such as participating-client selection and client clustering.These approaches use server-side coordination to manage heterogeneous participants.

3.1 Data-Level Methods

Data-level methods improve federated learning by processing private data or introducing external data to enhance quality, privacy, security, and collaboration under heterogeneous distributions.

  • Overview: Data-level methods comprise private data processing and external data utilization, targeting data quality, security, statistical heterogeneity, and communication difficulties.Private processing includes preparation and privacy protection; external data supports knowledge distillation or constraints on model updates.
  • Private Data Processing: Data preparation includes collection, filtering, cleaning, and augmentation, which can improve client data and directly alleviate statistical heterogeneity.Collection quality depends on data quantity, quality, and diversity; filtering can remove irrelevant, noisy, or malicious samples.
  • External Data Utilization: External data utilization uses knowledge distillation or update constraints, commonly addressing model heterogeneity while alleviating data and communication heterogeneity.Unsupervised representation learning can also alleviate statistical heterogeneity between clients’ local data.
  • Private Data Processing: Private data augmentation smooths distributions across clients, while GAN-based augmentation generates missing samples to improve statistical homogeneity.MAFL exchanges mean local data for global information and approximates global MixUp without accessing other clients’ raw data.
  • Private Data Processing: Federated data augmentation can require uploading local samples, increasing privacy-breach risk; proxy datasets are one approach to circumvent this risk.The survey identifies data augmentation in federated settings as highly challenging because of these privacy requirements.
  • Private Data Processing: Data privacy protection uses encryption, perturbation, and anonymization; homomorphic encryption enables computation on encrypted data, while DP clips and noises updates.Personalized local DP allows clients to choose privacy budgets according to private-data sensitivity.

3.2 Model-Level Methods

Model-level methods adapt federated models to heterogeneous client distributions through optimization, cross-model knowledge transfer, and architecture sharing.

  • Overview: Model-level methods include federated optimization, knowledge transfer across models, and architecture sharing.These categories address statistical, model, communication, and device heterogeneity through different model-side designs.
  • Federated Optimization: Federated optimization adapts models to local distributions while learning global information, enabling local-model personalization under statistical heterogeneity.FedProx constrains local and global models with a proximal term, improving training stability and accelerating convergence.
  • Knowledge Transfer across Models: Knowledge transfer enables model-agnostic collaboration and is typically used to address model and communication heterogeneity.Clients can exchange logits or knowledge over public data while keeping local models and private data decentralized.
  • Architecture Sharing: Architecture sharing personalizes federated learning by sharing part of the model structure, simultaneously addressing statistical, model, and device heterogeneity to some extent.Sharing subsets of global parameters reduces local-client computation and can avoid catastrophic forgetting to some extent.
  • Federated Optimization: Regularization methods address client drift, forgetting, and personalization by penalizing parameter changes or coupling global, current-local, and previous-local models.Examples include FedCurv, FedCL, pFedME, FedFomo, FedAMP, FedBN, SCAFFOLD, and MOON.
  • Meta-learning: Meta-learning frames global training and local personalization as meta-training and meta-testing, allowing satisfactory adaptation to new tasks with limited data.MAML-based approaches align with personalized objectives in heterogeneous federated learning.

3.3 Server-Level Methods

Server-level methods use server-side selection, clustering, and decentralized communication to manage heterogeneous clients and improve participation, efficiency, and collaboration.

  • Overview: Server-level methods include client selection, client clustering, and decentralized communication.Client selection chooses participants per iteration, clustering aggregates similar clients, and decentralized communication supports peer-to-peer collaboration.
  • Client Clustering: Client clustering aggregates similar clients to alleviate communication and device heterogeneity, while weakening harmful interference between clients with different data.Similarity can be measured using model, data-distribution, gradient, or optimization-direction information.
  • Client Selection: Client selection addresses data, communication, and device heterogeneity by considering client distributions, resources, computation, connectivity, reliability, or training history.Strategies include class-balancing selection, Gaussian-process loss modeling, reinforcement learning, and resource-aware selection.
  • Client Selection: FedCS alleviates communication and device heterogeneity while maximizing participants in each round and improving training efficiency.It selects clients using data resources, computing capabilities, and wireless channel conditions under a round deadline.
  • Client Selection: Regulating the proportion of selected clients mitigates straggler and dropout problems caused by communication and device heterogeneity.Adaptive layer selection groups clients by training time and selects clients from the same layer in each round.
  • Client Clustering: Clustered and multi-center approaches personalize collaboration by matching clients to cluster or global models according to model or update similarity.Examples include FeSEM, FedFMC, FedGroup, CFL, IFCA, and multi-center aggregation.

4 FUTURE DIRECTIONS

Future HFL research must improve communication efficiency while addressing fairness, privacy, security, and the absence of standardized benchmarks. Existing approaches reduce communication overhead or address privacy and fairness, but important deployment challenges remain.

  • Communication Efficiency: Communication-efficient HFL methods compress updates, transmit synthetic data, select sparse parameters, or exchange sub-models to reduce communication overhead.These approaches include synthetic-data transmission, personalized sparse masks, decentralized training, and Fed-Dropout.
  • Communication Efficiency: Communication methods still face multi-hop delay, unbalanced communication loads, asynchronous communication, bandwidth variation, and client-side computing or storage constraints.Practical systems must balance communication efficiency against model accuracy.
  • Fairness: HFL fairness research must account for unequal client contributions, biased global objectives, free-riding participants, and trade-offs between fairness and communication efficiency.Existing fairness methods are not well suited to heterogeneous federated learning, and fairness criteria may differ across decentralized layers.
  • Privacy and Security: Privacy mechanisms can reduce inference attacks and improve robustness against poisoning attacks, but heterogeneous clients require more flexible, fine-grained privacy constraints.Data sharing, model updates, distributions, and knowledge transfer can still expose private information; noise-based protection may sacrifice model performance.
  • Uniform Benchmarks: HFL lacks widely recognized benchmark datasets and testing frameworks, motivating unified benchmarks that represent varied client data distributions and model structures.Standardized evaluation is identified as a future research need.

5 CONCLUSION

This survey organizes heterogeneous federated learning around its research challenges and state-of-the-art methods. It proposes a three-level taxonomy and identifies open directions intended to support future HFL development.

  • Survey Scope: The survey provides a comprehensive and systematic understanding of heterogeneous federated learning and its research challenges.Its challenge overview is presented as a central part of the survey.
  • Challenge Analysis: It categorizes HFL challenges into statistical, model, communication, and device heterogeneity, while reviewing recent HFL research.The survey focuses specifically on heterogeneity problems in federated learning.
  • Method Taxonomy: The proposed taxonomy classifies state-of-the-art HFL methods at the data, model, and server levels.The levels correspond to different stages of the HFL procedure.
  • Future Outlook: The survey discusses promising research directions and open problems to support further development of heterogeneous federated learning.The authors present these discussions as contributing to high-quality development of the HFL field.
Loading 2307.10616v2…