Source-linked AI summary
FedALA: Adaptive Local Aggregation for Personalized Federated Learning
Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Haibing Guan
TL;DR
Statistical heterogeneity makes global-model generalization difficult across federated clients, motivating personalized local models. FedALA uses adaptive local aggregation to combine global and local models for client-specific initialization, outperforming eleven state-of-the-art methods and improving other methods as well.
Problem
Statistical heterogeneity from Non-IID and unbalanced client data makes it difficult for a global model to generalize to each client.
Method
FedALA adaptively aggregates the downloaded global model and old local model toward each client’s local objective to initialize local training.
Results
FedALA outperforms eleven state-of-the-art methods by up to 3.27% in test accuracy and applying ALA to other methods improves test accuracy by up to 24.19% on Cifar100.
Takeaways & Limitations
Element-wise ALA captures desired global-model information while retaining the communication pattern of downloading one global model and uploading one local model.
Takeaways & Limitations
Some personalized methods benefit relatively little from ALA when their learning processes already emphasize or aggregate relevant information during training.
Abstract
from arXiv · showhide
A key challenge in federated learning (FL) is the statistical heterogeneity that impairs the generalization of the global model on each client. To address this, we propose a method Federated learning with Adaptive Local Aggregation (FedALA) by capturing the desired information in the global model for client models in personalized FL. The key component of FedALA is an Adaptive Local Aggregation (ALA) module, which can adaptively aggregate the downloaded global model and local model towards the local objective on each client to initialize the local model before training in each iteration. To evaluate the effectiveness of FedALA, we conduct extensive experiments with five benchmark datasets in computer vision and natural language processing domains. FedALA outperforms eleven state-of-the-art baselines by up to 3.27% in test accuracy. Furthermore, we also apply ALA module to other federated learning methods and achieve up to 24.19% improvement in test accuracy.
Introduction
FedALA addresses statistical heterogeneity in personalized federated learning by adaptively selecting useful information from the global model for each client. Its element-wise local aggregation preserves FedAvg-level communication while improving benchmark performance and applicability to existing methods.
- Motivation: Statistical heterogeneity from Non-IID and unbalanced client data makes it difficult for a global model to generalize across clients.
- Motivation: Existing personalized methods may initialize local models with all global-model information, including information that is undesirable for an individual client.Personalized aggregation methods aim to capture client-relevant information from the global model.
- FedALA: FedALA adaptively aggregates the downloaded global model with the old local model toward each client’s local objective before local training.The ALA module performs local initialization within the client learning process.
- FedALA: FedALA learns real-valued, element-wise aggregation weights on the full local dataset, offering finer information selection than PartialFed’s binary, layer-wise strategy.
- Evaluation: FedALA outperforms eleven state-of-the-art methods by up to 3.27% in test accuracy without additional communication overhead per iteration.The method downloads one global model and uploads one local model, matching FedAvg’s communication pattern.
- Evaluation: The ALA module can be applied to traditional and personalized federated learning methods, improving test accuracy by up to 24.19% on Cifar100.
Method
FedALA formulates personalized federated learning as collaborative optimization of client-specific models, then initializes each local model by element-wise aggregation of the downloaded global model and the previous local model. Its ALA module learns aggregation weights toward the local objective, applies them mainly to higher layers, and is followed by standard local training.
- FedALA seeks individual local models across heterogeneous private client datasets while using a shared global model for external information.
- ALA element-wise aggregates the downloaded global model and local model instead of overwriting the local model.The aggregation weights are learned with gradient-based optimization and clipped element-wise to [0, 1].
- ALA is applied to p higher layers, while lower-layer parameters are overwritten as in FedAvg for local initialization.The design reflects the stated use of more general information in lower DNN layers and reduces computation overhead.
- ALA initializes its weights to one, learns them from a sampled fraction s% of local data, and freezes the global and local models during weight learning.After initialization, each client performs local model training as in FedAvg.
- The dynamic global–local term supplies information across iterations, while the learned weights converge early and change little afterward.The paper reports convergence in the second iteration and little subsequent change, with one epoch used for later updates.
- FedALA preserves the standard federated process of client sampling, global-model downloading, local processing, and model uploading.Algorithm 1 describes ALA at initialization followed by client uploads to the server.
Experiments
Experiments across heterogeneous vision and language settings show that FedALA improves personalization while retaining FedAvg-like communication and modest computation costs. Its adaptive element-wise aggregation remains effective across client counts and heterogeneity levels, although gains vary across other personalized methods.
- Experimental setup: FedALA was compared with eleven state-of-the-art federated learning baselines across computer vision and natural language processing experiments.The evaluation included pathological and practical heterogeneous settings.
- Pathological heterogeneous setting: FedALA outperformed all baselines when clients were separated into groups, while FedAvg and FedProx performed poorly because of poor global-model generalization.The results support adaptive capture of useful global information under pathological heterogeneity.
- Aggregation comparison: FedALA’s element-wise adaptive weights precisely capture useful global information, unlike model-level, layer-level, or binary aggregation schemes.FedFomo and APPLE also incur additional communication costs by downloading multiple client models.
- Benchmark results: 3.27% higher test accuracy than FedRep was achieved by FedALA on TINY in the practical heterogeneous setting.FedALA remained superior to the other baselines across the additional Tiny-ImageNet and AG News tasks.
- Efficiency: FedALA costs 1.93 minutes per iteration after a 7-minute initialization, adding 0.34 minutes while using one-model communication like FedAvg.FedFomo and APPLE require the highest communication overhead because they download M client models each iteration.
- Scalability and transfer: FedALA’s accuracy drops less than 1% as the client count increases to 100, while ALA also improves several traditional and personalized methods.Performance remains robust across different heterogeneity levels, whereas some methods become worse than FedAvg at moderate heterogeneity.
Conclusion
FedALA is an adaptive, fine-grained personalized federated learning method that facilitates local training with the received global model. Experiments demonstrate its effectiveness against eleven state-of-the-art methods and when applying ALA to other federated learning methods.
- FedALA adaptively facilitates local model training with the received global model.
- FedALA outperforms eleven state-of-the-art methods in the reported experiments.
- Applying the ALA module to other federated learning methods improves their accuracy.
Additional Details of weight learning in ALA
ALA learns higher-layer aggregation weights while freezing the remaining learnable weights, including the entire global and local models. The required gradients are obtained through backpropagation.
- ALA updates the higher-layer aggregation weights while freezing the other learnable weights.The frozen parameters include the entire global model and entire local model.
- The higher-layer components of the initialized model and model difference are represented separately in the weight-learning formulation.
- ALA obtains the required gradients through backpropagation and calculates gradients only for higher-layer model parameters.
Convergence of FedALA
FedALA optimizes a global objective based on client losses, with the global model providing external information to each client. The objective loss decreases for both trained and initialized local models and converges after more than 1000 global iterations.
- The global objective minimizes an aggregate of client losses across the participating clients.The objective is expressed as an optimization over the set of initialized local models.
- FedALA defines each client loss using its initialized local model, local dataset, and the global model.The global model supplies external information to client i.
- After more than 1000 global iterations, losses for trained and initialized local models become almost identical, indicating FedALA convergence.Both loss curves keep decreasing before reaching this behavior.
The Range of ALA
The range of ALA depends on the backbone and the parameter p. Increasing p expands the range so that ALA can cover more layers.
- ALA has backbone-specific ranges for different values of p.
- As p increases, the range of ALA increases.
- A larger p allows ALA to cover more layers.
Effect of p on CNN and fastText
Increasing p increases ALA's number of learnable weights, but has negligible effect on accuracy; the best accuracy for both four-layer CNN and fastText occurs at p = 1.
- p = 1 gives the best accuracy for both the four-layer CNN and fastText.
- As p increases, the number of learnable weights in ALA increases.
- Accuracy changes negligibly as p varies.
Additional Results on MNIST
In the practical heterogeneous MNIST setting, FedALA continues to outperform all evaluated baselines.
- FedALA still outperforms all baselines on MNIST in the practical heterogeneous setting.
Additional Results on Cifar100 with ρ = 0.5
With only half of the clients participating in each iteration, FedALA maintains its superiority on Cifar100, while the comparison excludes methods requiring all clients to join.
- FedALA maintains its superiority on Cifar100 with 100 clients when only half participate in each iteration.The experiment uses ρ = 0.5 and reports averaged results from joining clients.
- FedAMP and APPLE are excluded from the ρ = 0.5 comparison because they require all clients to join each iteration.
- The default practical setting tunes hyperparameters by grid search.
- FedALA uses η = 1.0, s = 80, and p = 1 among the tested settings.
Dataset URLs
The supplementary material identifies benchmark datasets and illustrates their client data distributions under several heterogeneous settings.
- The listed benchmarks are MNIST, Cifar10, Cifar100, Tiny-ImageNet, and AG News.
- Client data distributions are illustrated in Figures 6–9.
- Figure 6 shows Tiny-ImageNet client distributions across three heterogeneous settings, with circle size representing sample count.
- Figures 7 and 8 show client distributions for MNIST, Cifar10, and Cifar100 in pathological and default heterogeneous settings.
- Figure 9 shows Cifar100 client distributions in the default setting with 50 and 100 clients.