PaperlayerView exact revision on arXiv

Source-linked AI summary

Similarity-Aware Personalized Federated Learning in Heterogeneous Environments

Arun Kumar A, Sunil Gupta, Dang Ngyuen, Bao Duong, Dat Phan Trong

arXiv:2609.02241v1cs.LG

TL;DR

Federated learning can suffer when non-IID clients are poorly served by a single global model or rigid personalization schemes. SAPE-FL uses dual global-and-peer anchoring with adaptive similarity-based regularization and reports superior performance under heterogeneous settings, while requiring additional computation and communication overhead.

  • Problem

    Non-IID client distributions can degrade global generalization and client-level performance, while existing personalization methods may rely on static regularization or fixed aggregation.

  • Method

    SAPE-FL anchors each client model to the global model and a similarity-weighted peer average using adaptive regularization based on model-output and weight similarity.

  • Results

    SAPE-FL achieves superior performance compared with state-of-the-art methods under heterogeneous settings, especially high statistical heterogeneity and limited client data.

  • Takeaways & Limitations

    Similarity-aware dual anchoring supports personalized and robust federated learning by combining global knowledge with selectively aligned peer information.

  • Takeaways & Limitations

    SAPE-FL introduces additional computation and communication overhead from pairwise similarity estimation and peer maintenance.

Abstract

from arXiv · show

Federated Learning (FL) allows decentralized clients to train models collaboratively while preserving data privacy. However, distribution mismatch across clients often leads to poor global generalization and degraded local client-level performance. In such scenarios, some of the clients with their local models trained solely on local data may perform better than the globally learnt model, thus nullifying the benefits of collaborative federated learning. To address this, we propose SAPE-FL (Similarity-Aware Personalized Federated Learning), a novel personalization framework that anchors each client's model to both the global model and a similarity-weighted peer averaged model. By incorporating dynamic, client-specific regularization based on both model similarity and output similarity, SAPE-FL adaptively balances global knowledge transfer and peer collaboration while filtering out dissimilar clients. This dual anchoring mitigates negative transfer and enhances robustness in heterogeneous settings. We theoretically analyze our algorithm establishing its convergence guarantees and empirically show that SAPE-FL outperforms state-of-the-art methods under high statistical heterogeneity and low client data regimes.

1 Introduction

SAPE-FL addresses non-IID client distributions and rigid personalization by combining global collaboration with adaptive, similarity-aware client customization. It provides convergence analysis and empirical validation under high statistical heterogeneity.

  • Motivation: Statistical heterogeneity can degrade global model performance, motivating personalized federated learning for diverse client populations.FedAvg was designed for IID data, while non-IID distributions create generalization challenges across clients.
  • Motivation: Existing personalized methods use static regularization or fixed similarity aggregation, which may cause sub-optimal personalization and negative transfer under highly non-IID data.These approaches do not adapt to changing client drift, data distributions, or model reliability across rounds.
  • Proposed framework: SAPE-FL anchors each client model to both the global model and a similarity-weighted peer average using adaptive model-output and weight-based similarities.Dissimilar peers are filtered so that only relevant peer information influences personalization.
  • Proposed framework: Similarity-aware aggregation scales client updates by peer coherence, helping prevent negative transfer while supporting global knowledge sharing and client-specific adaptation.The framework jointly enables global collaboration and personalized optimization.
  • Validation: SAPE-FL is theoretically analyzed for convergence and empirically validated across diverse datasets under high statistical heterogeneity.Experiments include synthetic classification, human activity recognition, and image classification, with especially strong performance reported when client data are limited.

2 Related Work

Related federated-learning research addresses heterogeneity through adaptive participation, multi-task learning, clustering, and personalization. SAPE-FL combines similarity-aware peer matching with dual anchoring to address limitations of static or inefficient approaches.

  • Federated learning under heterogeneity: FedAvg struggles with statistical heterogeneity because non-IID data can produce conflicting gradient updates and degraded performance.This motivates methods that adapt training or aggregation to client differences.
  • Adaptive FL: Adaptive federated-learning methods modify update frequency, training continuation, or client participation to improve efficiency and handle heterogeneous conditions.Examples include resource-aware updates, selective continued training, and informational client prioritization.
  • Multi-task and meta-learning: Multi-task and meta-learning methods model client-specific objectives, task similarity, or inter-task dependencies to accommodate heterogeneous data distributions.MOCHA and VIRTUAL are cited as representative approaches.
  • Personalized FL: Personalized FL spans local fine-tuning, decomposition, clustering, and hypernetwork-based approaches for adapting models to non-IID clients.Recent clustering methods form compact client groups using cosine-based distances.
  • SAPE-FL positioning: SAPE-FL addresses reported limitations of static regularization, inefficient aggregation, and clustering through hybrid weight-and-output similarity for peer matching and aggregation.The framework anchors client models to both global and peer-averaged models.

3 Mathematical Preliminaries

The preliminaries formalize classical and personalized federated learning under heterogeneous client distributions. They motivate client-specific models because a single weighted global objective may be inappropriate for non-IID data.

  • Classical FL: Classical FL learns a single global model by aggregating client updates without centralizing sensitive client data.The server iteratively updates θ_G from client contributions.
  • Classical FL: FedAvg weights clients by participation probabilities, typically setting p_c proportional to client data volume.The probabilities satisfy Σ_c p_c = 1 and p_c ≥ 0.
  • Statistical heterogeneity: A single global objective may be unsuitable in heterogeneous settings because clients contribute different non-IID data distributions.Statistical heterogeneity includes differing label or feature distributions across clients.
  • Personalized FL: Non-IID distributions can degrade global performance, motivating personalized models that combine global knowledge with adaptation to local data.PFL targets client-specific models θ_c rather than only one shared model.
  • Personalized FL: Proximal-based PFL adds regularization that constrains local-model drift from a shared global reference.This extends the personalized optimization objective for heterogeneous clients.

4 Framework

SAPE-FL personalizes federated learning through dual anchoring, hybrid similarity-based regularization, selective peer averaging, and similarity-weighted server aggregation. The workflow alternates local personalization with collaborative global updates.

  • Framework objective: SAPE-FL targets personalized models for clients with non-IID data distributions through adaptive balancing of local personalization and global collaboration.Its framework combines dual anchoring with similarity-aware server aggregation.
  • Dual anchoring: Each client optimizes against both the global model and a peer-averaged model using client-specific similarity-aware regularization coefficients.The coefficients enforce alignment with the two anchors at each training round.
  • Similarity-aware regularization: The regularization coefficients combine model-output similarity and model-weight similarity, with δ controlling their relative influence.Output similarity captures predictive behavior, while weight similarity captures structural alignment.
  • Peer filtering and averaging: Clients filter peers using a similarity threshold and construct peer averages only from sufficiently aligned models.The threshold balances inclusiveness with robustness by excluding dissimilar peers.
  • Server aggregation: Similarity-weighted aggregation gives more weight to peer-aligned clients and down-weights lower-scoring updates, suppressing noisy updates.This end-to-end design is presented as robust to data heterogeneity and misaligned signals.
  • End-to-end workflow: At each round, selected clients receive the global model and peer models, train locally, compute similarities, personalize, and return updates with similarity scores.The server then performs similarity-weighted aggregation, increasing the influence of peer-aligned updates and down-weighting lower-scoring updates.

5 Convergence Analysis of SAPE-FL Method

SAPE-FL is analyzed through global and personalized objectives, similarity-weighted aggregation, and peer averaging. Under stated smoothness, convexity, and bounded-variance assumptions, the analysis establishes descent, convergence, and variance-reduction properties.

  • Objective and setup: The analysis studies SAPE-FL with clients initialized from the global model and trained using local and peer-regularized objectives.The server refines the global model from similarity-weighted client updates.
  • Global convergence: Similarity scores act as attention weights that emphasize aligned updates and reduce noisy or misaligned client contributions.Expected global loss decreases each communication round when updates are sufficiently aligned and similarity weights are effective.
  • Peer averaging: Peer averaging preserves the mean of aligned peer models while reducing estimation-error variance by a factor of 1 |Ck|.This result is framed as an ensemble-like robustness benefit for personalized anchors.
  • Convergence guarantees: The global objective decreases monotonically and converges to a first-order stationary point under the stated similarity filtering and step-size conditions.An ϵ-approximate solution is reached in O communication rounds, with |Ck| denoting the number of similar peers.
  • Convergence guarantees: Theoretical results give linear convergence in the strongly convex case and the standard first-order rate in the non-convex case.The analysis relies on smoothness, strong convexity where applicable, and bounded gradient variance.

6 Experiments

Experiments compare SAPE-FL with federated and personalized baselines across synthetic, human-activity-recognition, and image-classification tasks. SAPE-FL generally achieves the strongest final performance, particularly under high heterogeneity and limited client data, while adding similarity-estimation overhead.

  • Baselines: Experiments compare SAPE-FL with FedAvg, PerFedAvg, FedProx, Ditto FL, FedACS, LCFed, FedAFK, FedAS, and a local-only baseline.The comparisons target methods conceptually aligned with federated and personalized learning.
  • Experimental setup: The experiments use κ = 0.3 Dirichlet partitions, 200 communication rounds, 100 clients, 70% participation, and Smin = 0.65.Clients use an 80/20 train-test split, and results are repeated with 5 random initializations.
  • Caveat: Pairwise similarity estimation and peer maintenance add computation and communication overhead.The paper states that low-rank factorization can mitigate this cost in large-scale deployments.
  • Results: SAPE-FL consistently outperforms baselines and achieves the highest mean test accuracy in most classification tasks.Ditto FL and LCFed remain competitive in a few cases, while SAPE-FL is particularly strong on HAR tasks.

7 Conclusion

The paper concludes that SAPE-FL addresses statistical heterogeneity through dual anchoring, adaptive similarity-based regularization, and similarity-aware server aggregation. Reported results show superior performance against state-of-the-art methods in heterogeneous settings.

  • Conclusion: SAPE-FL addresses statistical heterogeneity by aligning each client with both the global model and a similarity-weighted peer average.Adaptive regularization uses model-parameter and output similarity.
  • Conclusion: Similarity-aware server aggregation prioritizes coherent client updates to enhance robustness and personalization.The conclusion reports superior performance compared with state-of-the-art methods under heterogeneous settings.
  • Conclusion: The theoretical analysis covers convergence of the similarity-weighted global objective under convex and non-convex settings.The proofs are provided in the appendix.

A.1.1 Proof of Theorem 1

Theorem 1's proof interprets similarity-weighted client aggregation as an approximate stochastic-gradient step. Under smoothness and alignment assumptions, this update is a descent direction for the global objective.

  • Proof strategy: Each client performs approximate local SGD from the current global model, producing an update used in the server's global-model update.The proof relates the client update to stochastic-gradient descent on the local objective.
  • Similarity weighting: Unlike full-participation FedAvg, SAPE-FL assigns similarity-based weights to client gradients in the aggregated update.The weights can reflect data importance or gradient reliability when appropriately aligned.
  • Similarity weighting: Similarity weighting emphasizes mutually consistent gradients, reducing conflicting directions and making the aggregate more representative of aligned clients.The proof describes this as reducing gradient variance and biasing the update toward the true global gradient.
  • Descent condition: Under L-Lipschitz smoothness and sufficient update alignment, the aggregated update has a negative inner product with the true global gradient and yields descent.This condition is the key step in establishing one-round improvement.
  • Peer regularization: Peer regularization anchors clients to similar-model neighborhoods, limiting extreme updates and bounding the heterogeneity of server-aggregated updates.Peers are selected using similarity above Smin.

A.1.2 Proof of Lemma 1

The lemma shows that averaging similarly distributed peer models reduces estimation variance, including under non-convexity, while filtering dissimilar peers preserves personalization benefits.

  • With uniform similarity weights, the weighted procedure reduces to standard averaging and retains the 1/|Ck| variance-reduction factor.This holds for both equal unnormalized and normalized weights.
  • Peer averaging reduces estimation-error variance by a factor of 1/|Ck| when combining independent, similarly distributed peer estimates.This is the classical variance-reduction effect of averaging.
  • The peer-averaged model provides no worse, and often better, loss than the average loss of individual peers when client losses are convex.Jensen’s inequality supplies the stated comparison.
  • Similarity-weighted peer averaging improves local-optimum estimates by leveraging similar clients while filtering dissimilar clients’ drift.The method sets Scj = 0 for dissimilar peers.
  • The variance-reduction result remains applicable in non-convex settings when peer models are independent, unbiased estimates of a shared latent central tendency.The argument does not require a unique optimizer or convergence to a true minimum.

A.1.3 Proof of Theorem 2

The theorem analysis treats SAPE-FL updates as optimization of a similarity-weighted objective with peer regularization. Under stated smoothness, boundedness, participation, and convexity assumptions, the method has descent and convergence guarantees, including stationary-point convergence in non-convex settings.

  • Each client performs a proximal step on a composite objective anchored to global and similarity-weighted peer models.The joint objective includes peer regularization and similarity-weighted aggregation.
  • Similarity weighting favors highly similar clients, while peer regularization promotes consensus within similarity-based neighborhoods.The resulting updates couple clients toward shared local optima within similarity groups.
  • With a sufficiently small learning rate, the joint objective decreases monotonically under the bounded-regularization analysis.The descent argument uses the stated learning-rate condition η ≤ 1/(2L).
  • Peer averaging tightens the convergence bound by reducing update variance, yielding fewer rounds to reach an ϵ-stationary solution.The analysis attributes the faster convergence to statistical averaging across similar peers.
  • Under strong convexity and smoothness, SAPE-FL converges linearly to the unique minimizer, with each round shrinking the optimality gap by (1 −ηµ).The result assumes the similarity-weighted aggregation and sufficiently small constant step size.
  • Under non-convexity, SAPE-FL converges in expectation to an ϵ-stationary point within O(1/ϵ^2) communication rounds.Filtering dissimilar updates is described as an additional safeguard against high-variance updates.

A.2.2 Datasets

The evaluation uses synthetic and real-world classification datasets partitioned across heterogeneous clients, with performance measured through client-level test accuracy and AUC. Experiments also include stability, scalability, and client-trajectory analyses.

  • Datasets: Experiments cover synthetic classification, human activity recognition, and image classification datasets under simulated statistical heterogeneity.Synthetic data use multivariate Gaussian clusters; image tasks include MNIST, EMNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100.
  • Experimental settings: The experiments use 100 clients, an 80–20% train-test split per client, and 70% client participation per communication round.The influence factor is set to δ = 0.5.
  • Metrics: Performance is evaluated using mean test accuracy and average AUC across participating clients, each reported with standard deviation.AUC is included for assessing discrimination under imbalanced data distributions.
  • Stability and scalability: When no peer satisfies the similarity threshold, SAPE-FL disables peer regularization and relies solely on the global anchor.This fallback avoids arbitrary parameter averaging and preserves a well-defined objective.
  • Stability and scalability: When aggregate similarity mass is zero, the server skips the global model update to avoid unreliable updates under extreme heterogeneity.The fallback is intended to maintain stable training dynamics, particularly in early rounds.
  • Stability and scalability: In practice, each client compares with a small sampled peer subset, keeping added peer-processing complexity O(C) where C ≪ N.The additional work consists of a few forward passes on local mini-batches.

B.2 Uplifting Clients with Poor Performance

The weak-client experiment compares SAPE-FL with FedAvg using local accuracy gains for the lowest-performing quarter of 20 clients. SAPE-FL produces larger gains for the selected weak clients, while output similarity contributes strongly to performance.

  • Evaluation setup: The experiment compares local-only training, FedAvg, and full SAPE-FL from a common initialization.The local-only baseline trains each client exclusively on its own data.
  • Weak-client selection: Accuracy gains are computed relative to each client’s local-only accuracy, then evaluated for the bottom 25% of clients by that baseline.With C = 20, the selected group contains five clients.
  • Weak-client results: SAPE-FL yields larger accuracy gains than FedAvg for the five selected clients most affected by data heterogeneity.Figure 2 compares the per-client gains using orange SAPE-FL bars and blue FedAvg bars.
  • Similarity ablation: Higher δ values consistently improve performance across datasets in the ablation study.δ = 1 uses only output similarity, whereas δ = 0 relies solely on weight similarity.
  • Similarity ablation: A shared influence factor balances output-space and weight-space similarities for both global and peer anchors without additional tuning.The unified formulation is reported to produce stable results across datasets.

B.4 Impact of Data Heterogeneity κ

SAPE-FL remains robust across varying statistical heterogeneity, with strongest reported performance at κ = 0.3. Threshold ablations indicate that Smin values from 0.6 to 0.8 best balance peer-set size and peer alignment.

  • Impact of Data Heterogeneity κ: 90.66% on Synthetic, 81.22% on HAR, 81.24% on AccGyro, and 80.98% on MNIST are reported at κ = 0.3.Smaller κ indicates more severe data skew across clients.
  • Impact of Data Heterogeneity κ: SAPE-FL remains stable and competitive across all evaluated κ values.The evaluation varies κ from 0.2 to 1.0 to represent different levels of statistical heterogeneity.
  • Similarity Threshold Ablation: Smin values between 0.6 and 0.8 consistently yield better performance across the evaluated datasets.This range balances peer-set size with the quality and alignment of included peers.
  • Practical Considerations: Pairwise similarity computation and peer-set maintenance add computational and communication burden, particularly in large-scale deployments.The paper suggests low-rank factorization as one possible mitigation.
  • Practical Considerations: Highly disjoint or adversarial client distributions may require adaptive peer filtering or fallback to global-model regularization.These strategies are proposed to maintain personalization quality and stability when meaningful peer alignment is unavailable.
Loading 2609.02241v1…