PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Federated Learning using Hypernetworks

Aviv Shamsian, Aviv Navon, Ethan Fetaya, Gal Chechik

arXiv:2103.04628v1cs.LG

TL;DR

Personalized federated learning seeks collaborative training for clients with differing data distributions while preserving personalized models and controlling communication costs. pFedHN trains a central hypernetwork that generates each client’s model from its embedding. The paper reports improved accuracy across datasets and learning setups, with better generalization to novel clients and support for models of different sizes.

  • Problem

    Personalized federated learning must benefit from joint training while allowing clients to retain unique models under heterogeneous data and limited communication.

  • Method

    pFedHN trains a central hypernetwork that maps each client’s descriptor to personalized model weights and optimizes it using client update directions.

  • Results

    pFedHN achieves significant accuracy improvements across all datasets and learning setups, with better generalization to novel clients and models of different sizes.

  • Takeaways & Limitations

    Central hypernetwork sharing supports personalized models, generalization to novel clients, different client compute capacities, and compact transmitted models.

  • Takeaways & Limitations

    Generalization is affected by the hypernetwork’s Lipschitz constant, reflecting a trade-off between flexibility and generalization.

Abstract

from arXiv · show

Personalized federated learning is tasked with training machine learning models for multiple clients, each with its own data distribution. The goal is to train personalized models in a collaborative way while accounting for data disparities across clients and reducing communication costs. We propose a novel approach to this problem using hypernetworks, termed pFedHN for personalized Federated HyperNetworks. In this approach, a central hypernetwork model is trained to generate a set of models, one model for each client. This architecture provides effective parameter sharing across clients, while maintaining the capacity to generate unique and diverse personal models. Furthermore, since hypernetwork parameters are never transmitted, this approach decouples the communication cost from the trainable model size. We test pFedHN empirically in several personalized federated learning challenges and find that it outperforms previous methods. Finally, since hypernetworks share information across clients we show that pFedHN can generalize better to new clients whose distributions differ from any client observed during training.

1. Introduction

Personalized federated learning addresses heterogeneous client data by combining collaborative training with unique client models. pFedHN uses a central hypernetwork to generate personalized models, share parameters, reduce communication costs, and support different client capacities.

  • 1. Introduction: Personalized federated learning lets clients use unique models when a single global model fails under heterogeneous data distributions.The challenge is preserving joint training while limiting communication costs.
  • 1. Introduction: pFedHN uses one joint hypernetwork and a unique client embedding to generate personalized model weights for each client.This provides parameter sharing while retaining distinct client models.
  • 1. Introduction: The hypernetwork is not transmitted, so communication costs remain decoupled from the trainable model size.Clients receive only their own network parameters and return gradients or update directions.
  • 1. Introduction: Clients with different computational resources can select generated networks with different computational capacities.The hypernetwork can produce several networks per input, each with a different computational capacity.
  • 1. Introduction: The paper reports a new hypernetwork-based approach, better generalization to novel and resource-diverse clients, and state-of-the-art results on CIFAR10, CIFAR100, and Omniglot.The paper also reports experimental evaluation across several datasets and learning setups.

2. Related Work

Federated learning enables collaborative learning without sharing local data, but faces data, device, and communication heterogeneity. Related approaches include global aggregation, personalized training strategies, and hypernetworks that dynamically generate target models.

  • 2. Related Work: Federated learning coordinates multiple clients while maintaining privacy and communication efficiency across disjoint local datasets.Clients collaborate without directly sharing their local data.
  • 2. Related Work: FedAvg aggregates local models into a global model, whereas personalized methods seek per-client solutions.FedAvg is described as learning a global model by aggregating local models trained on IID data.
  • 2. Related Work: Personalized federated learning addresses data heterogeneity by personalizing the global model for each client.Existing approaches include MAML-based methods, model mixing, and architectures with base and personalized layers.
  • 2. Related Work: Hypernetworks output the weights of another target network, with output weights varying according to the input.This mechanism supports dynamically generating target networks conditioned on client-specific inputs.
  • 2. Related Work: Hypernetworks are suitable for diverse personalized models because they dynamically generate target networks conditioned on input.Figure 1 applies this idea by mapping client embeddings to personal model weights and receiving update directions.

3. Method

pFedHN uses a server-side hypernetwork and client descriptors to generate personalized models while sharing parameters across clients. It decouples hypernetwork size from communication cost and supports local optimization and personalized classifiers.

  • Federated Hypernetworks: pFedHN trains a server-side hypernetwork that maps each client descriptor to personalized model weights.The hypernetwork learns a family of client-specific models while sharing its parameters across clients.
  • Communication: Client communication depends on target-network parameters rather than hypernetwork size, allowing large hypernetworks without increasing communication costs.Clients receive model parameters and return gradients or local update directions, not the hypernetwork parameters.
  • Optimization: The training algorithm samples clients, generates their models, performs local updates, and uses the resulting model changes to update hypernetwork parameters.After several local steps, the client returns Δθi = ˜θi − θi, which contributes to the hypernetwork update.
  • Personal Classifier: pFedHN can generate feature extractors while clients learn local classifiers when end-to-end hypernetwork generation is undesirable.This design addresses clients with different label orderings or separate tasks and class counts.

4. Analysis

The analysis interprets linear pFedHN as task-aware dimensionality reduction and denoising, then derives generalization bounds reflecting embedding dimension, shared hypernetwork size, and Lipschitz flexibility.

  • A Linear Model: In the linear setting, minimizing the pFedHN objective yields PCA on the clients’ ERM solutions.W contains the top k principal components, while each client embedding supplies the corresponding coefficients.
  • A Linear Model: pFedHN interpolates between a shared model when k = 1 and local training when k ≥ n, while preserving task-relevant information.The analysis states that reconstructing every local model with k ≥ n is generally suboptimal because no information is shared.
  • Denoising: Under Gaussian-noise assumptions, local ERM solutions are noisy versions of optimal models, so linear hypernetworks perform PCA-based denoising.The nonlinear hypernetwork is viewed as an extension that constrains models to a low-dimensional manifold.
  • Generalization: The generalization bound separates an embedding-dimension term from a shared-hypernetwork term that is reduced by a factor n.The first term is independent of the number of clients, whereas shared hypernetwork weights reduce the second term.
  • Generalization: Generalization also depends on the hypernetwork’s Lipschitz constant, which limits the effective embedding-reachable space.An initial attempt to improve this through spectral normalization showed no significant improvement.

5. Experiments

pFedHN is evaluated across heterogeneous data, resource budgets, novel clients, personalization, and learned client representations. Across these settings, it achieves strong accuracy, adapts to different model sizes, generalizes to novel clients, and captures client structure.

  • 5.1. Heterogeneous Data: pFedHN achieves 2%-10% improvements over competing approaches across heterogeneous CIFAR10, CIFAR100, and Omniglot tasks.The experiments compare personalized and non-personalized federated baselines across 10, 50, and 100 clients.
  • 5.2. Computational Budget: pFedHN trains one hypernetwork to generate target models with different sizes while sharing parameters across clients.The computational-budget evaluation uses 75 clients divided into small, medium, and large architecture groups.
  • 5.2. Computational Budget: 4% −8% improvement over competing methods is reported for CIFAR10/100 clients with varying computational capacities.The results are presented for 75 clients divided into small, medium, and large groups.
  • 5.3. Generalization to Novel Clients: pFedHN achieves the best generalization performance for all tested total variation distances between novel and training clients.The accuracy generalization gap is measured as accnovel − acctrain, with 90 clients used for training and 10 held out as novel clients.
  • 5.4. Heterogeneity of Personalized Classifiers: Mix-and-match accuracy varies significantly across clients, indicating that pFedHN-PC produces personalized feature extractors while some compatible pairs retain accuracy.The experiment replaces each client’s feature extractor with another client’s while keeping its personalized classifier unchanged.
  • 5.5. Learned Client Representation: t-SNE shows that learned client embeddings cluster clients assigned to the same CIFAR100 coarse-label group.Clients with the same coarse labels are marked with the same color in the visualization.

6. Conclusion

pFedHN trains a central hypernetwork that generates unique personal models for clients, improving accuracy across datasets and learning setups. The framework also supports novel-client generalization and compact transmitted models, while leaving several extensions open.

  • pFedHN trains a central hypernetwork to output a unique personal model for each client.
  • Extensive experiments show significant accuracy improvement on all evaluated datasets and learning setups.
  • Central hypernetwork sharing improves generalization to novel clients without retraining the central model.
  • Transmitted local models can be more compact than the central model, decoupling training complexity from communication complexity.
  • Future work includes allocating learning capacity between central and locally trained components and analyzing generalization to new distributions.

A. Proof of Results

The proofs analyze pFedHN’s parameterization and optimization through orthonormal representations, client embeddings, covering arguments, and a PCA equivalence. They also identify a missing log(1/ϵ) factor in Theorem 1’s stated sample bound.

  • The optimization can be restricted to matrices with orthonormal columns because equivalent transformations preserve the loss.
  • For fixed W, the optimal client embedding is v_i = W^T θ̄_i when W has orthonormal columns.
  • The resulting optimization problem is equivalent to PCA on the client-optimal parameters {θ̄_i}_i.
  • Theorem 1’s proof notes that the paper statement omits a log(1/ϵ) factor from the correct sample bound.
  • Covering the hypernetwork and client embeddings yields a sample-complexity expression involving embedding dimension, hypernetwork size, and client count.

B. Experimental Details

The experiments use a three-layer fully connected hypernetwork and aggregate five clients per round. They evaluate heterogeneous data and computational-budget settings with validation-based hyperparameter selection.

  • Main-text experiments use a fully connected hypernetwork with 3 hidden layers of 100 units each.
  • Relevant baselines aggregate over 5 clients at each communication round.
  • Heterogeneous Data: CIFAR experiments reserve 10,000 training examples for validation, while Omniglot uses a 70%/15%/15% train/validation/test split.
  • Computational Budget: The computational-budget setup uses LeNet-based target networks with S/M/L variants containing 8/16/32 filters in the first convolution layer.
  • Computational Budget: pFedHN’s hypernetwork outputs the combined weight vector and sends only weights relevant to each client’s selected target-network size.

C.1. MNIST

The experiments examine pFedHN on MNIST and CIFAR10, including local optimization, architecture capacity, embeddings, spectral normalization, and novel-client generalization. Results show strong or comparable performance, robustness to several choices, and declining accuracy as client distributions diverge.

  • C.1. MNIST: On MNIST, pFedHN achieves similar results to pFedMe, while comparisons are difficult because all methods attain high accuracy.
  • Effect of Local Optimization: Using multiple local updates produces large improvements in convergence speed and final accuracy over the one-step chain-rule update.
  • Effect of Local Optimization: pFedHN is relatively robust to the number of local optimization steps, with experiments comparing k = 1, 25, 50, 100, and 200.
  • Embedding and Hypernetwork Capacity: Embedding-dimension ablations show robustness across dimensions 5, 15, 25, and 35.
  • Embedding and Hypernetwork Capacity: pFedHN reaches 88.38 and 88.42 accuracy with 3 and 4 hidden layers, respectively, and uses 3 layers thereafter.
  • Spectral Normalization: Spectral normalization produces no significant improvement over the unnormalized results.
  • Novel Clients: For novel clients, test accuracy decreases on average as total variation distance from the nearest training client increases.
  • Embedding and Hypernetwork Capacity: Trainable and fixed client embeddings achieve similar performance: 84.12 ± 0.42 and 83.92 ± 0.36, respectively.
Loading 2103.04628v1…