PaperlayerView exact revision on arXiv

Source-linked AI summary

FedGH: Heterogeneous Federated Learning with Generalized Global Header

Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, Han Yu

arXiv:2303.13137v2cs.LGcs.DC

TL;DR

Model-heterogeneous federated learning must accommodate clients with diverse models, while existing approaches can require public datasets and substantial communication or computation. FedGH trains a shared global prediction header from heterogeneous client representations, achieving stronger performance and lower communication overhead in reported experiments.

  • Problem

    Model-heterogeneous FL approaches often require public datasets and incur high communication or computation costs.

  • Method

    FedGH trains a homogeneous global prediction header from representations extracted by heterogeneous client feature extractors and replaces clients’ local prediction headers with it.

  • Results

    FedGH outperforms seven state-of-the-art personalized FL models in model-homogeneous and model-heterogeneous settings, with gains up to 8.87% and 1.83% in average test accuracy and communication savings up to 85.53%.

  • Takeaways & Limitations

    FedGH provides a communication- and computation-efficient framework for sharing global knowledge across clients with heterogeneous models.

  • Takeaways & Limitations

    FedGH is formulated for supervised classification with shared input, representation, and output dimensions and a common task across clients.

Abstract

from arXiv · show

Federated learning (FL) is an emerging machine learning paradigm that allows multiple parties to train a shared model collaboratively in a privacy-preserving manner. Existing horizontal FL methods generally assume that the FL server and clients hold the same model structure. However, due to system heterogeneity and the need for personalization, enabling clients to hold models with diverse structures has become an important direction. Existing model-heterogeneous FL approaches often require publicly available datasets and incur high communication and/or computational costs, which limit their performances. To address these limitations, we propose a simple but effective Federated Global prediction Header (FedGH) approach. It is a communication and computation-efficient model-heterogeneous FL framework which trains a shared generalized global prediction header with representations extracted by heterogeneous extractors for clients' models at the FL server. The trained generalized global prediction header learns from different clients. The acquired global knowledge is then transferred to clients to substitute each client's local prediction header. We derive the non-convex convergence rate of FedGH. Extensive experiments on two real-world datasets demonstrate that FedGH achieves significantly more advantageous performance in both model-homogeneous and -heterogeneous FL scenarios compared to seven state-of-the-art personalized FL models, beating the best-performing baseline by up to 8.87% (for model-homogeneous FL) and 1.83% (for model-heterogeneous FL) in terms of average test accuracy, while saving up to 85.53% of communication overhead.

1 INTRODUCTION

Practical FL faces system, data, and model heterogeneity, while existing model-heterogeneous approaches can remain costly or depend on public datasets. FedGH addresses these gaps with a shared global prediction header trained from heterogeneous clients’ representations and reports strong accuracy and communication savings.

  • Mobile edge clients have heterogeneous and constrained computing, bandwidth, memory, storage, and battery resources.
  • Low-end clients may be unable to train large global models, while small homogeneous models underuse high-end clients’ resources.
  • Non-IID client data creates statistical heterogeneity alongside device and model heterogeneity.
  • Existing model-heterogeneous FL methods can deliver limited improvement while incurring high computation and communication costs or requiring suitable public datasets.
  • FedGH trains a shared generalized prediction header from representations produced by heterogeneous feature extractors, then replaces clients’ local headers.
  • Up to 8.87% and 1.83% higher average test accuracy and 85.53% communication-overhead savings are reported for model-homogeneous and model-heterogeneous FL, respectively.

2 RELATED WORK

Model-heterogeneous FL includes subnet-based and completely heterogeneous-model approaches, with public-data distillation limited by dataset availability and computation. FedGH instead uses class-labeled local representations to train a shared header that accommodates heterogeneous extractors and non-IID data.

  • Model-heterogeneous FL methods comprise heterogeneous subnets of a server model and completely heterogeneous client models.
  • Public Data-Dependent: Public-data distillation methods operate on clients, the server, or both, but require suitable public datasets and may incur substantial computation.
  • Public Data-Dependent: Public datasets may be unavailable, distributionally mismatched, or computationally expensive to process, limiting these methods’ applicability.
  • Public Data-Independent: Public-data-independent methods include model mixup, mutual learning, and data-free knowledge distillation.
  • Public Data-Independent: FedGH trains a homogeneous shared global header from local representations and labels, rather than class distributions, and uses it to replace local headers.
  • Public Data-Independent: Each client supplies one averaged embedding vector per seen class, enabling the global header to better accommodate non-IID data.

3 THE PROPOSED FEDGH APPROACH

FedGH enables heterogeneous clients to share a generalized prediction header by exchanging class-wise averaged representations instead of complete models. The method combines heterogeneous feature extractors with a shared header while reducing communication and computation costs.

  • Model formulation: FedGH splits each client model into a heterogeneous feature extractor and a homogeneous prediction header.All clients share the representation and prediction dimensions, while extractor structures may differ.
  • Federated Global Header: Clients compute class-wise averaged representations and labels, which the server uses to train a global prediction header.The server updates the header using received local averaged representations and hard classification loss.
  • Federated Global Header: The updated global header is broadcast to selected clients, replacing their local prediction headers before the next training round.Local heterogeneous models retain their extractors while receiving shared global header knowledge.
  • Workflow: The workflow repeats local training, representation aggregation, server header training, and header replacement until local heterogeneous models converge.After training, the heterogeneous local models are used for inference.
  • Cost reduction: Representation extraction and class averaging impose acceptable or negligible costs, making FedGH suitable for resource-constrained mobile devices and more powerful participants.The representation-extraction cost is half the forward-and-backward cost of local training in one epoch, and class averaging has O(n) complexity.
  • Cost reduction: FedGH communicates only class labels, class-wise representations, and global-header parameters rather than complete local or global models.For S classes and representation dimension r, the uplink cost is (S+S×r) × 32 bits, while the downlink is lower than broadcasting a complete model.

4 CONVERGENCE ANALYSIS

The convergence analysis introduces round and local-iteration notation and assumes smoothness and bounded gradient and header variance. Under these assumptions, FedGH admits a non-convex convergence-rate guarantee for arbitrary clients.

  • Notation: The analysis indexes communication rounds by t and local iterations by e, with up to E local iterations per round.The first iteration of round t+1 is indexed by tE+1, and the final iteration by tE+E.
  • Round structure: FedGH replaces each client’s local prediction header with the latest global header at the beginning of a new communication round.The clients then continue local model updates using the received header.
  • Assumptions: The convergence proof assumes Lipschitz-smooth local gradients, unbiased stochastic gradients with bounded variance, and bounded variance for local and global prediction headers.These assumptions support the stated loss bounds and convergence theorems.
  • Intermediate results: The analysis derives loss bounds for arbitrary clients during local iterations and after replacing local headers with the latest global header.These results are presented as Lemmas 4.1 and 4.2.
  • Main guarantee: Theorem 2 establishes a non-convex convergence rate for FedGH for any arbitrary client and any ϵ> 0 under the stated assumptions.The paper expresses the resulting rate as ϵ∼O(...).

5 EXPERIMENTAL EVALUATION

FedGH is evaluated against seven existing approaches on CIFAR-10 and CIFAR-100 using non-IID client data, with average test accuracy and communication overhead as metrics.

  • Experimental setup: Experiments compare FedGH with seven existing approaches on CIFAR-10 and CIFAR-100.The FL processes are implemented with PyTorch and simulated on NVIDIA GeForce RTX 3090 GPUs.
  • Evaluation metrics: Average test accuracy reports the mean accuracy of clients’ local models, while communication overhead measures traffic until target accuracy is reached.Communication overhead includes rounds, participating clients, transmitted floating-point data, and 32-bit representation.
  • FedGH training: FedGH clients upload local averaged representations and their labels to the server after extracting representations with heterogeneous local feature extractors.This training strategy uses class-level representation summaries rather than complete local models.

5.2 Results and Discussion

FedGH is tested in homogeneous and heterogeneous FL settings under varying client configurations and model structures, where it achieves the strongest reported accuracy and fast convergence.

  • Model-homogeneous FL: 0.54% and 8.87% are FedGH’s average accuracy improvements over FedProto on CIFAR-10 and CIFAR-100, respectively.The larger CIFAR-100 gain is reported as evidence of effectiveness under statistical heterogeneity.
  • Model-heterogeneous FL: Five heterogeneous CNN models, CNN-{1, 2, ..., 5}, are distributed evenly among clients by varying convolutional filters and fully connected dimensions.The detailed structures and sizes are reported in Table 2.
  • Model-heterogeneous FL: 1.17% and 1.83% are FedGH’s accuracy improvements over FedProto on CIFAR-10 and CIFAR-100, respectively, in heterogeneous FL.FedGH also requires the fewest communication rounds to reach target accuracy and reduces CIFAR-100 communication overhead by 85.53% versus FedProto.
  • Model-heterogeneous FL: FML fails to converge, while FedKD converges with lower accuracy than FedGH in the heterogeneous setting.The authors associate these outcomes with limited information interaction and the immature shared small model during initial rounds.

5.3 Case Studies

Case studies examine robustness to Non-IIDness and client participation rates, along with sensitivity to the global header learning rate.

  • Sensitivity to hyperparameter: The global prediction header learning rate is FedGH’s only hyperparameter, and the authors report no influence from its value on training.The header-training process is described as easier than training complete local large models.
  • Robustness to Non-IIDness: FedGH achieves the highest model accuracy across tested Non-IID degrees on both CIFAR-10 and CIFAR-100.The study varies the number of classes assigned to each client to control the degree of Non-IIDness.
  • Robustness to partial participation: FedGH achieves the highest model accuracy under different client participation rates on both CIFAR-10 and CIFAR-100.Participation rates range from 0.1 to 1 with N=100.
  • Robustness to partial participation: Model accuracy decreases as client participation increases, while generalization is enhanced and personalization becomes more challenging.This pattern is reported for the tested participation-rate experiments.

6 CONCLUSIONS AND FUTURE WORK

FedGH is presented as a model-heterogeneous FL framework that combines heterogeneous feature extractors with a shared global prediction header. The authors report theoretical convergence and empirical advantages, while identifying two future directions concerning representation distortion and header personalization.

  • FedGH uses same-dimension representations from heterogeneous local feature extractors to train a homogeneous global prediction header.
  • The global header transfers all-class knowledge to clients by replacing their local prediction headers.
  • Theoretical derivations establish a non-convex convergence rate for FedGH.
  • Future work includes reducing information distortion from local class-averaged representations and combining generalized global and personalized local headers.

A PROOF FOR LEMMA 4.2

The proof tracks the evolution of a client's local model and the global prediction header across communication rounds and local iterations. It invokes an assumption and gradient-descent updates with a shared learning rate.

  • At the start of the (t + 2)-th round, a client's local model combines the locally trained model from the previous round with the updated global header.
  • In the last local iteration of the (t + 1)-th round, the client's local model contains the locally trained component and the global header from that round.
  • The proof applies Assumption 4.1 to one inequality step.
  • Model training uses gradient descent, with the local-model and global-header learning rates both set to η.

B PROOF FOR THEOREM 1

The proof of Theorem 1 substitutes Lemma 4.1 into the second term on the right-hand side of Lemma 4.2.

  • The proof derives the target inequality by substituting Lemma 4.1 into Lemma 4.2's second right-hand-side term.
  • This step connects the bound established by Lemma 4.1 to the corresponding term in Lemma 4.2.
  • The passage provides the proof operation but not the resulting expanded inequality.

C PROOF FOR THEOREM 2

The proof of Theorem 2 rewrites the theorem, takes expectations, sums the resulting inequality over communication rounds, and applies convergence conditions. It concludes that FedGH has a non-convex convergence rate under a bounded learning rate.

  • Theorem 1 is re-expressed as an intermediate form for proving Theorem 2.
  • The proof takes expectations of the model on both sides of the relevant equation.
  • Summing the inequality over T rounds yields the cumulative bound used in the convergence argument.
  • The derivation assumes the local model can converge and uses L_t=0 − L* > 0.
  • The learning rate η is upper bounded, after which the proof identifies the non-convex convergence behavior from the first term of Eq. (21).
Loading 2303.13137v2…