Source-linked AI summary
A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization
Yong Zhao, Chunlin You, Minh N. Dao, Zai-Yun Peng
TL;DR
Federated multiobjective optimization addresses the need to train models on multiple conflicting objectives without centralizing private data. The paper proposes FSMGDA-M-VR, which adds momentum-based variance reduction to local federated multi-gradient updates. For a uniformly random output iterate, the expected Pareto stationarity measure decays at O(T^-2/3), improving on O(T^-1/2) rates reported for FSMGDA and FedCMOO.
Problem
Existing federated learning studies mostly optimize one global loss, although practical models may require simultaneous optimization of conflicting objectives.
Method
FSMGDA-M-VR integrates objective-wise momentum variance-reduced estimators into federated local updates and server-side Pareto direction construction.
Results
O(T^-2/3) expected Pareto stationarity for a uniformly random output iterate improves upon the O(T^-1/2) rates established for FSMGDA and FedCMOO.
Takeaways & Limitations
The method provides a federated multiobjective training approach with improved stationarity and communication-round complexity relative to FSMGDA and FedCMOO.
Takeaways & Limitations
The analysis assumes synchronous full participation with equal client weights and excludes client dropout, communication compression, and quantization.
Abstract
from arXiv · showhide
Federated learning has traditionally been formulated as a single-objective optimization problem, primarily focused on maximizing model utility. In real-world applications, however, machine learning models often need to optimize multiple and potentially conflicting objectives simultaneously. This motivates federated multiobjective optimization (FMOO), which provides a natural framework for jointly handling multiple task-specific objectives in federated learning. In this paper, we propose a momentum-based variance-reduced algorithm for federated multiobjective optimization. The method incorporates a momentum-driven gradient estimator into the local updates to reduce the variance of stochastic updates, leading to an improved convergence rate. We establish theoretical guarantees showing that the expected Pareto stationarity measure of a randomly selected output iterate decays at a rate of $\mathcal{O}(T^{-2/3})$, improving upon the $\mathcal{O}(T^{-1/2})$ rates established for existing methods such as FSMGDA and FedCMOO. Numerical experiments on federated multiobjective optimization benchmarks demonstrate the effectiveness and competitive performance of the proposed algorithm.
1. Introduction
Federated multiobjective optimization combines federated learning’s privacy-preserving collaboration with multiobjective optimization for conflicting objectives. The proposed FSMGDA-M-VR method uses momentum-based variance reduction to improve convergence, with a stated communication–convergence trade-off.
- MOO jointly optimizes multiple often-conflicting objectives, so algorithms target Pareto-optimal or Pareto-stationary solutions rather than one universally minimizing solution.
- Federated learning enables collaborative model training without sharing raw data, addressing privacy and data-silo challenges.
- Existing federated learning research has largely focused on single-objective optimization, despite applications requiring simultaneous optimization of conflicting goals such as accuracy and training cost.
- FSMGDA-M-VR combines federated momentum-based variance reduction with stochastic multiobjective optimization to address conflicting objectives, heterogeneous clients, local stochastic updates, and communication.
- FSMGDA-M-VR differs from FSMGDA and FedCMOO by constructing objective-wise momentum variance-reduced directions rather than using accumulated stochastic gradients or server-side Gram-matrix weight updates.
- O(T^-2/3) convergence improves upon the O(T^-1/2) rates stated for FSMGDA and FedCMOO, while communication cost scales linearly with the number of objectives.
2. Preliminaries
The preliminaries define federated multiobjective objectives, Pareto notions, stochastic multi-gradient stationarity, and the assumptions used for analysis. The framework models client-specific expected objectives under synchronous full participation.
- FMOO optimizes multiple objectives simultaneously through client-specific expected objectives fs,i(x), with M clients and local data samples drawn from client distributions.
- The federated setting uses synchronous full participation with equal client weights and excludes client dropout, communication compression, and quantization.
- Pareto optimality excludes solutions dominated across all objectives, while weak Pareto optimality excludes solutions strictly worse across every objective.
- Pareto stationarity means no direction simultaneously decreases every objective, making it a practical first-order criterion in non-convex multiobjective optimization.
- Multi-gradient methods form G(x), choose simplex-constrained weights minimizing ||G(x)λ||^2, and use the resulting vector as a common descent direction when nonzero.
- The stochastic stationarity measure ||G(x)λ*||^2 assesses convergence, and expected ε-Pareto stationarity accounts for randomness generated through the iteration.
- The analysis assumes smooth objective and stochastic functions, bounded stochastic-gradient variance, and additional conditions introduced for convergence analysis.
3. FSMGDA-M-VR algorithm
FSMGDA-M-VR performs objective-wise momentum variance-reduced local updates, aggregates the resulting directions across clients, and constructs a Pareto direction on the server. Its key modification is local variance reduction, which supports the improved stationarity rate analyzed later.
- FSMGDA-M-VR is a momentum-based variance-reduced federated stochastic multi-gradient descent algorithm for solving federated multiobjective optimization.
- At each communication round, clients independently sample data and perform K local updates for every objective using momentum-corrected variance-reduced descent directions.
- The same sampled data point is used across objectives for each client, local step, and communication round, while client data distributions may differ.
- Clients average objective-specific gradient estimates and send directions to the server, which aggregates them across clients.
- The server computes Pareto-balancing weights through a convex quadratic optimization problem and combines objective directions into the global direction.
- The global model update is xt+1 = xt − γgt, after which the procedure repeats over communication rounds.
- Compared with FSMGDA, the algorithm retains server-side Pareto aggregation but adds an objective-wise momentum variance-reduced estimator to local updates.
4. Convergence analysis
The convergence analysis establishes assumptions, filtrations, and technical lemmas for controlling stochastic and momentum-related errors. Under these conditions, Algorithm 1 achieves an O(T^-2/3) random-iterate Pareto stationarity rate and improved communication complexity.
- Variance control: Independent initialization and subsequent stochastic samples support the conditional variance and martingale arguments used in the technical lemmas.Samples are independent across clients, local steps, and rounds, while client data distributions need not be identical.
- Proof framework: The analysis uses filtrations to condition expectations on randomness revealed before each communication round and local update.This organizes the dependence among stochastic samples, historical gradients, local variables, and global quantities.
- Assumptions: Assumption 4.1 requires client-averaged squared norms of local expected gradients to remain uniformly bounded along the generated trajectory.The condition is trajectory-specific rather than a global bounded-gradient requirement over all model parameters.
- Technical bounds: The technical lemmas control local-update error, conditional variance, and momentum variance under restrictions involving η, L, K, and β.These bounds are combined across local steps and communication rounds to prove the final convergence result.
- Complexity: O(ϵ^-3/2) communication rounds are obtained in the high-accuracy regime, improving on the O(ϵ^-2) complexity of FSMGDA and FedCMOO.The communication complexity is identified with the number of global rounds, while total stochastic gradient computations are O(SMKT).
5. Numerical experiments
Experiments evaluate FSMGDA-M-VR across three federated multiobjective benchmarks under controlled client, data, model, and optimization settings. Across comparisons, it generally converges faster and more stably than FSMGDA, while matching or outperforming FedCMOO on several accuracy and loss measures.
- Experimental setup: Experiments use MultiMNIST, MNIST+FMNIST, and CIFAR10+MNIST as three federated multiobjective benchmarks.The tasks use two objectives constructed from paired or superimposed image datasets.
- Experimental setup: The main experiments use 100 heterogeneous clients, with a Dirichlet concentration parameter α = 0.3 and matched partitions, initializations, seeds, and minibatch sequences across methods.These controls support a fair comparison under non-identical client data distributions.
- Comparative performance: FSMGDA-M-VR converges faster and more stably than FSMGDA and FedCMOO across the three datasets, with comparable or better final accuracy and lower test loss.The reported curves show particularly clear early-round advantages on MultiMNIST and MNIST+FMNIST.
- Comparative performance: FSMGDA-M-VR obtains the highest mean accuracy on both CIFAR10+MNIST objectives and leads the first objective on MultiMNIST and MNIST+FMNIST.On MultiMNIST it ties FedCMOO on the second objective, while FedCMOO leads the second objective on MNIST+FMNIST; results are means over 10 random seeds.
- Sample-access comparison: Against cumulative sample access per client, FSMGDA-M-VR generally has higher accuracy and lower loss, although FedCMOO improves faster initially on CIFAR10+MNIST before FSMGDA-M-VR catches up.On CIFAR10+MNIST, FSMGDA-M-VR reaches comparable final accuracy and near-lowest final test loss.
- Local training performance: FSMGDA-M-VR shows steeper local loss reduction and sharper early accuracy gains than FSMGDA, while remaining comparable to FedCMOO.The comparison measures average local training progress across clients.
- Hyperparameter effects: Larger batch sizes generally accelerate convergence, improve final accuracy, reduce test loss, and smooth training curves across the benchmark datasets.Batch sizes 128 and 256 perform especially strongly, with smaller late-stage differences between them on MNIST+FMNIST.