Source-linked AI summary
FedSoft: Soft Clustered Federated Learning with Proximal Local Updating
Yichen Ruan, Carlee Joe-Wong
TL;DR
Hard clustered federated learning does not represent clients whose data mix multiple source distributions and may fail to exploit similarities between clusters. FedSoft uses proximal local updating to train cluster and personalized models with one optimization task per selected client, and it achieves strong performance under the paper’s analyzed and experimental settings. Its guarantees rely on assumptions about distribution mixtures and similarity between proximal sub-problems.
Problem
Hard clustered FL assumes one distribution per client and independently trains clusters, whereas client data may mix distributions and related clusters may share useful structure.
Method
FedSoft uses proximal local objectives that combine local data with all cluster models, allowing one client optimization to support cluster training and personalization.
Results
FedSoft learns high-quality cluster and personalized models, with cluster training converging linearly to an error determined by cluster heterogeneity and outperforming existing FL implementations.
Takeaways & Limitations
FedSoft provides soft clustered FL with nearly conventional client workload while retaining centrally deployable cluster models and personalized models for participating clients.
Takeaways & Limitations
The analysis assumes that coexisting distributions produce sufficiently similar proximal sub-problems, so highly divergent sub-problems are outside the stated guarantee.
Abstract
from arXiv · showhide
Traditionally, clustered federated learning groups clients with the same data distribution into a cluster, so that every client is uniquely associated with one data distribution and helps train a model for this distribution. We relax this hard association assumption to soft clustered federated learning, which allows every local dataset to follow a mixture of multiple source distributions. We propose FedSoft, which trains both locally personalized models and high-quality cluster models in this setting. FedSoft limits client workload by using proximal updates to require the completion of only one optimization task from a subset of clients in every communication round. We show, analytically and empirically, that FedSoft effectively exploits similarities between the source distributions to learn personalized and cluster models that perform well.
1 Introduction
FedSoft addresses two practical weaknesses of hard clustered federated learning: clients may mix multiple source distributions, and independently trained clusters cannot exploit similarities. It uses proximal local updating to learn personalized and cluster models while limiting each participating client to one optimization task.
- FedSoft: FedSoft produces centrally available cluster models for future users and personalized local models for participating clients.The method therefore supports both deployment without local adaptation and personalization for current clients.
- Motivation: Hard clustered FL assumes each client belongs to one distribution, but real client datasets may contain mixtures whose ratios vary across clients.This can produce different overall client distributions even when the underlying source distributions are identical.
- FedSoft: FedSoft uses soft clustered FL, allowing each client’s data to follow a mixture of multiple distributions rather than enforcing a hard association.This directly targets the first practical challenge of hard clustered FL.
- Motivation: Hard clustered FL trains different cluster models independently, making it difficult to transfer shared structure between similar source distributions.The paper contrasts this with shared knowledge such as common grammar across user groups with different slang usage.
- FedSoft: Proximal local updating combines local information with all cluster models in one objective, so each client solves one optimization problem while supporting multiple clusters.The resulting workload is almost the same as conventional FL.
2 Related Works
Prior clustered federated learning methods generally alternate between identifying client partitions and training cluster models, using model distances, loss-based assignments, or exogenous data information. FedEM handles mixed client data but requires a local update for every cluster each round, increasing training time.
- Hard clustered FL: Existing hard clustered FL methods generally use EM-like alternation between cluster identification and model training.They are categorized by how the partition structure is discovered.
- Hard clustered FL: Prior methods infer associations from model-parameter distances, hierarchical clustering, or related model-based criteria.The supplied related-work passage identifies distance-based approaches and IFCA-like methods.
- Hard clustered FL: Some methods use exogenous patient or imaging information to form clusters, but this typically requires direct access to user data.That requirement limits applicability in the general federated setting.
- Soft clustered FL: FedEM allows client data to follow mixtures of distributions but requires every client to update for every cluster in each round.The resulting training time is significantly greater than conventional FedAvg.
- Proximal personalization: Other proximal personalization methods include FedAMP, which adds attention-inducing regularization, and pFedMe, which uses Moreau envelopes.These methods provide related proximal-regularization precedents outside clustered settings.
3 Formulation and Algorithm
FedSoft models each client as a mixture of source distributions and optimizes a proximal objective that combines local risk with cluster-model references. It estimates mixture importance, selects client subsets for each cluster, and aggregates personalized local models into cluster centers.
- Mixture of distributions: Each data point is drawn from one of S source distributions, and data following the same distribution forms a cluster; S is fixed beforehand.A client may contain data from multiple clusters.
- Mixture of distributions: A client’s importance weight for cluster s is its fraction of local data assigned to distribution P_s, u_ks = n_ks/n_k.The weights are unknown initially and estimated during learning.
- Proximal local updating: FedSoft replaces direct local-risk minimization with a proximal objective combining local information and regularized distances to the cluster models.The regularization is weighted by each client’s estimated cluster importance.
- Proximal local updating: The joint objective lets one personalized local model support multiple centers, bonding cluster training through clients whose component distributions share knowledge.This is the mechanism used to exploit similarities between source distributions.
- Algorithm: The algorithm outputs trained cluster models and a personalized local model for every participating client.The personalized model is obtained as a byproduct of the same local optimization process.
- Algorithm: Each client solves only one proximal optimization problem, while the server uses selected client subsets and aggregates their models to update cluster centers.Client selection uses K < N clients per cluster, with overlap reducing the union of participating clients relative to SK.
4 Convergence Analysis
FedSoft’s convergence analysis shows how proximal subproblem optimization, similarity among mixed-distribution tasks, and bounded cluster divergence support convergence of both personalized client models and cluster models. The guarantees also characterize residual error from misclassification and conditions under which soft clustering performs best.
- Assumptions: FedSoft’s analysis assumes inexact proximal solutions, similar client subproblems, strongly convex and smooth cluster risks, and bounded initial center error.The similarity assumption makes solving one subproblem informative for others, while the proximal formulation preserves strong convexity and smoothness.
- Assignment estimation: Theorem 1 bounds estimation errors in the inferred client-cluster assignments under the stated assumptions.The bound uses an error parameter combining misclassification and sampling effects.
- Cluster models: Theorem 3 bounds the divergence of a client’s local model from a cluster center, while Theorem 4 establishes convergence of cluster centers in terms of cluster risks.For zero mixture weight, approaching a center introduces an error of at most O(∆).
- Local models: Theorem 2 shows that accurately solving one proximal subproblem can approximately solve other sufficiently similar subproblems with positive mixture weights.The result supports using mixture-related quantities as aggregation weights in FedSoft.
- Error behavior: FedSoft performs best when cluster divergences are similar, because misclassification error decreases as ∆ approaches δ; highly divergent clusters increase local-solution difficulty.When divergences are homogeneous, misclassification effects are amortized across clusters, reducing remaining error.
- Joint convergence: Theorem 5 states that, for fixed importance weights, FedSoft’s client and cluster models converge linearly to the joint minimizer.The joint objective combines the proximal client subproblems with the cluster centers.
5 Experiments
FedSoft is evaluated across synthetic and letter datasets with varied mixture patterns, comparing cluster and personalized models against baselines. The experiments show accurate centers, correct center–distribution associations, and useful client importance estimates across settings.
- Experimental setup: Experiments use 100 clients and evaluate mixtures of two, four, and eight distributions across synthetic, EMNIST, and CIFAR-10 datasets.The two-distribution experiments use 10:90, 30:70, linear, and random partitions; larger mixtures use random partitions.
- Two-distribution mixtures: FedSoft automatically distinguishes two synthetic distributions, with centers developing the correct associations after roughly five global epochs.The importance estimates converge from approximately 50:50 toward the real 10:90 mixture ratio.
- Two-distribution mixtures: FedSoft produces high-quality cluster models under all tested two-distribution mixture patterns, with each center strongest on one distribution.The performance gap between distributions is larger for synthetic data than for letter data because the synthetic distributions diverge more.
- Baseline comparison: On EMNIST letters, FedSoft outperforms IFCA and FedEM on cluster and local models while achieving a better balance between its two centers.FedSoft and IFCA recover the correct client association, whereas FedEM assigns lower-than-0.5 weights to the first cluster.
- More distributions: For mixtures of eight synthetic and four letter distributions, FedSoft retains high-quality cluster outcomes and good associations between centers and distributions.These results are reported for randomly generated mixture weights and for original and 90°-rotated letter images.
6 Conclusion
FedSoft generalizes clustered federated learning to clients whose data come from mixtures of distributions. Proximal local updating supports efficient training of personalized and cluster models in this setting.
- Conclusion: FedSoft generalizes traditional clustered federated learning by allowing each client to sample data from a mixture of distributions.The method is designed for soft rather than hard client–distribution associations.
- Conclusion: Proximal local updating makes FedSoft efficient for training in the soft clustered federated learning setting.
A Proof of Theorems
The proof analyzes FedSoft through its proximal objectives and block updates. Under joint convexity, the training process is characterized as cyclic block coordinate descent with at least linear convergence to a stationary point.
- Optimization view: Under Assumption 3, FedSoft’s summed proximal objective is jointly convex in the center and local-model blocks for fixed importance weights.
- Optimization view: FedSoft can be viewed as cyclic block coordinate descent that sequentially updates local models and centers while holding other blocks fixed.
- Convergence argument: This block-coordinate process converges at least linearly to a stationary point under the stated convexity characterization.
- Center updates: The center-averaging update follows by setting the objective gradients to zero, yielding FedSoft’s center update rule.
B The impact of τ on the convergence
The estimation interval τ affects importance-weight accuracy rather than the asymptotic convergence rate. Larger intervals increase estimation error because the relevant separation measure is expected to grow over time.
- Effect of τ: τ affects the accuracy of importance-weight estimates, which determines the estimation error pϵ.
- Effect of τ: The analysis assumes 0 < αt ≤ 0.5 and expects αt to increase as center distances decrease over time.
- Estimation error: Larger estimation intervals τ increase the estimation error because the interval uses an earlier, smaller α value.
- Convergence rate: The resulting convergence behavior has the same asymptotic rate as Corollary 1, differing only in small constant terms.
Experiment parameters
The experiments vary FedSoft’s regularization and distribution divergence while holding other parameters unchanged in the reported comparisons. Synthetic training uses fixed optimizer and schedule settings, while λ is selected by grid search for letters.
- Synthetic experiments use linear regression without an intercept, Adam, 10 local epochs, batch size 10, learning rate 5e-3, λ = 1.0, and 50 global epochs.
- For the letters dataset, λ = 0.1 is selected through grid search while other parameters remain unchanged.
- At λ = 0, clients train separately without cooperation, producing poorly trained models.
- As divergence between distributions increases, FedSoft’s MSE significantly increases, validating Theorem 4.
CIFAR-10 Results
The CIFAR-10 evaluation uses original and 90°-rotated image distributions and compares FedSoft’s cluster and client models with baseline results. FedSoft produces high-quality, separated cluster models and the best cluster-model performance in the reported comparison.
- The CIFAR-10 experiment uses original images and 90° counterclockwise rotations as two data distributions.
- The experiment uses 20 clients, selects K = 15 clients per round, sets τ = 2 and λ = 0.01, and trains for 200 global epochs.
- Table 7 reports cluster-model accuracy with each row representing a test distribution and the highest-accuracy center underlined per row.
- FedSoft yields high-quality cluster models with clear separation between the different CIFAR-10 distributions.
- FedSoft has the best cluster-model performance against IFCA and FedEM while producing fairly accurate local models.