Source-linked AI summary
Personalized and Private Peer-to-Peer Machine Learning
Aurélien Bellet, Rachid Guerraoui, Mahsa Taziki, Marc Tommasi
TL;DR
The paper asks how agents can collaboratively learn personalized models from local data while satisfying strong decentralization and privacy requirements. It introduces asynchronous peer-to-peer coordinate descent, adds differential privacy through noisy updates, and analyzes convergence and the utility–privacy trade-off. Experiments report improved performance over isolated models under privacy constraints and improvement over prior work in the non-private setting.
Problem
Connected devices generate sensitive personal data, but existing privacy-preserving distributed methods often use central servers or learn a single global model rather than decentralized personalized models.
Method
The paper combines asynchronous peer-to-peer block coordinate descent with update perturbation by Laplace noise to provide differential privacy and analyzes its convergence and utility loss.
Results
The approach has provable convergence behavior, improves over previous work in the non-private case, and improves over isolated models under privacy constraints.
Takeaways & Limitations
Collaborative personalized learning can be performed without a central entity while keeping data local and trading utility against privacy through noisy updates.
Takeaways & Limitations
The proposed noise allocation policy assumes advance knowledge of the schedule and global iteration counter, which is unrealistic without a global clock.
Abstract
from arXiv · showhide
The rise of connected personal devices together with privacy concerns call for machine learning algorithms capable of leveraging the data of a large number of agents to learn personalized models under strong privacy requirements. In this paper, we introduce an efficient algorithm to address the above problem in a fully decentralized (peer-to-peer) and asynchronous fashion, with provable convergence rate. We show how to make the algorithm differentially private to protect against the disclosure of information about the personal datasets, and formally analyze the trade-off between utility and privacy. Our experiments show that our approach dramatically outperforms previous work in the non-private case, and that under privacy constraints, we can significantly improve over models learned in isolation.
1 Introduction
The paper targets personalized collaborative learning that keeps data local and avoids sensitive-information leakage through a fully decentralized, asynchronous peer-to-peer network. It addresses gaps in prior work by combining personalization, decentralization, and differential privacy.
- Agents collaborate asynchronously over a peer-to-peer network while keeping their data on local devices and avoiding a central coordinator.
- Prior privacy-preserving distributed methods commonly rely on a central server, assume similar data distributions, or learn one global model instead of personalized models.
- The paper proposes an asynchronous decentralized block coordinate descent algorithm with simple updates and provable convergence rates for non-private learning.
- A differentially private scheme perturbs each update and formally analyzes the resulting utility loss and privacy trade-off.
- Experiments evaluate the approach on synthetic and real-world data, including its utility–privacy trade-off and performance under strong privacy constraints.
2 Peer-to-Peer Personalized Learning with Coordinate Descent
The method formulates personalized learning as graph-regularized multi-task optimization and solves it with asynchronous decentralized coordinate descent. Local updates use neighboring models and achieve fast linear convergence under strong convexity, while general convex objectives admit an O(1/t) rate.
- Problem setting: Each agent learns a model with small expected loss under its own local data distribution, rather than relying solely on a shared global model.
- Decentralized operation: The fully decentralized design restricts communication to local neighborhoods and requires neither global network knowledge nor synchronization.
- Objective: Graph regularization favors similar models for strongly connected agents while preserving accuracy on their respective local datasets.Agents with less data can benefit from better-endowed neighbors without imposing their inaccurate models on others.
- Algorithm: At each asynchronous wake-up, an agent updates its local model from previously received neighbor models and broadcasts the updated model.
- Convergence: Under strong convexity, randomized block coordinate descent has a fast linear convergence rate, while general convex objectives achieve an O(1/t) rate.
- Convergence: Convergence is faster when agents’ degree-weighted confidence is similar, whereas high-confidence high-degree and low-confidence poorly connected agents can bottleneck progress.
3 Differentially Private Algorithm
The paper protects asynchronous peer-to-peer learning against eavesdropping by adding calibrated noise to local gradient updates and formally analyzing privacy composition and utility loss. The private algorithm trades iteration-dependent optimization error against an additive noise error, while practical noise allocation has decentralized implementation limits.
- Privacy model: The privacy model protects each agent’s dataset against network eavesdroppers and colluding agents that cannot access internal memory.Agents are modeled as honest-but-curious: they follow the protocol while attempting to learn from received information.
- Privacy model: Each agent’s transmitted iterates must be (ε, δ)-differentially private with respect to its local dataset.The mechanism maps an agent’s dataset to all information it broadcasts during algorithm execution.
- Privacy-preserving scheme: The private update adds scaled Laplace noise to each agent’s local-loss gradient before broadcasting the resulting iterate.Noise may depend on the global iteration number, and Gaussian noise is an alternative under L2-Lipschitzness.
- Privacy-preserving scheme: Theorem 1 composes per-update privacy parameters: with δ̄_i = 0, an agent’s total privacy parameter is ε̄_i = Σ_ti ε_i(t_i).The stated per-update scale is s_i(t_i) = 2L_0/(ε_i(t_i)m_i), under the paper’s Lipschitz loss assumptions.
- Utility–privacy trade-off: After T iterations, private optimization error combines the non-private term, which decreases with T, and an additive noise term.With constant noise across iterations, the additive error converges to a finite limit as T →∞, so T controls the trade-off.
- Utility–privacy trade-off: Optimal noise allocation can improve utility but requires advance knowledge of agents’ schedules and a global iteration counter, unavailable in the fully decentralized setting.The paper leaves practical strategies based on expected wake-up times for future work; a differentially private warm start is also required.
4 Related Work
The paper extends decentralized personalized learning beyond prior work by combining block coordinate descent, differential privacy, and real-data evaluation. It contrasts this approach with centralized, local, gossip-based, and global-model privacy settings.
- Unlike consensus-oriented decentralized learning, the approach handles agents with distinct objectives and learns personalized models regularized over a network graph.
- Prior decentralized personalized learning did not address privacy constraints, while distributed-data privacy-preserving learning remained less studied, especially without central coordination.
- The paper proposes a simpler decentralized block coordinate descent method with linear convergence, improving practically over the earlier gossip ADMM approach.
- The paper also evaluates its approach on real data and argues that decentralized architecture and graph regularization may limit a documented collaborative-learning attack, leaving careful study for future work.
- Existing differential privacy work commonly assumes a trusted curator, whereas local data perturbation can yield poor accuracy, particularly for small local datasets.
5 Numerical Experiments
Experiments compare the proposed method with ADMM in synthetic linear classification and evaluate private learning on classification and MovieLens recommendation tasks. The results report faster non-private optimization and improvements over isolated models under privacy constraints.
- 5.1 Linear Classification: The comparison tracks objective value and test accuracy against iterations and transmitted p-dimensional vectors for two asynchronous decentralized algorithms.
- 5.1 Linear Classification: On linear classification with p = 100, coordinate descent significantly outperforms ADMM despite ADMM taking 10 local gradient steps per iteration.
- 5.1 Linear Classification: More private iterations reduce one bound term but increase another through added noise; a private warm start achieves lower objectives with fewer iterations and significantly improves test accuracy.
- 5.1 Linear Classification: Under a privacy budget of ϵ̄ = 0.15, private models significantly outperform purely local models across dimensions, with especially large gains for users with small datasets.
- 5.2 Recommendation Task: On MovieLens-100K, the approach largely outperforms purely local models in both non-private and private settings using per-user test RMSE.
6 Conclusion
The paper introduces and analyzes an efficient algorithm for personalized peer-to-peer machine learning under privacy constraints. It identifies dynamic scenarios and alternative privacy technologies as directions for further research.
- The paper introduces and analyzes an efficient algorithm for personalized and peer-to-peer machine learning under privacy constraints.
- Further research is needed for dynamic settings in which agents join or leave and data are collected online.
- Secure multiparty computation and homomorphic encryption are proposed as alternative or complementary approaches to differential privacy, trading higher accuracy for more computation.
SUPPLEMENTARY MATERIAL
The supplementary material contains proofs, additional privacy analysis, a private warm-start case, and further experimental details.
- Section A contains proofs of the main-text results.
- Section B analyzes Theorem 2 further when noise scales are uniform across iterations.
- Section C studies model propagation as a private warm-start strategy, while Section D provides additional experiments and details.
A.1 Proof of Theorem 1
The proof establishes differential privacy for noisy asynchronous updates by bounding dataset sensitivity and composing per-iteration guarantees. It uses neighboring datasets, Lipschitz assumptions, and noise calibrated to the resulting sensitivity.
- Privacy setup: Neighboring datasets differ in a single data point, defining the privacy comparison used throughout the proof.
- Sensitivity bound: L0-Lipschitzness with respect to the L1-norm bounds each local loss gradient's L1-norm by L0.
- Sensitivity bound: The local gradient is the only update quantity that depends on agent i's local dataset, enabling sensitivity analysis of the next iterate.
- Noise mechanism: Noise is calibrated from update sensitivity, with Laplace noise for the L1 setting and Gaussian noise under an L2-Lipschitz assumption.
- Privacy guarantee: Composing the per-iteration guarantees makes publishing all noisy iterates (¯ϵi, ¯δi)-differentially private as stated in Theorem 1.
A.3 Proof of Proposition 2
The proposition derives privacy parameters that minimize expected utility loss under per-agent privacy budgets. The allocation matches each budget in expectation, while a known wake-up schedule permits renormalization for exact budget use.
- Optimal allocation: The privacy parameters in Lemma 3 minimize expected utility loss over the total number of iterations T and agents' overall budgets.
- Optimization: The privacy-parameter optimization minimizes the utility-loss bound separately for each agent under a total privacy-budget constraint.
- Optimal allocation: ϵ∗i(t) allocates the amount of noise an agent should add in expectation at global iteration t.
- Budget usage: The allocation satisfies PT −1 t=0 ϵi(t) = ¯ϵi in expectation, but a particular asynchronous run may leave part of an agent's privacy budget unused.
- Budget usage: Conditioning on a known wake-up schedule allows privacy parameters to be renormalized so agents use their entire budgets during their active iterations.
B Further Analysis of Theorem 2
The privacy-induced utility loss separates into optimization and noise terms. Increasing iterations reduces optimization error but increases accumulated noise, so iteration count should be tuned to minimize total error.
- The optimization error decomposes into an optimization term matching the non-private setting and a privacy-induced noise term.
- At convergence, the remaining additive utility loss is due to privacy noise.
- The iteration count T should be tuned because the total error can be lower at finite T than after convergence.
- Increasing iterations drives optimization error from b to 0 while increasing noise error from 0 to aρ.
C Propagation of (Private) Local Models
The propagation framework generalizes decentralized model propagation to broader local losses and private pretrained models. Experiments show collaborative private learning improves accuracy for agents of all dataset sizes, with larger gains for agents having less data, while local DP on data performs poorly.
- Propagation of Local Models: The quadratic propagation objective yields an exact block-coordinate minimizer depending only on solitary and neighboring models.
- Propagation of Local Models: The resulting algorithm recovers Vanhaesebrouck et al.’s model-propagation update while extending it to general local losses with convergence rates.
- Private Local Models: A differentially private local model can be used as a warm start without compromising the algorithm’s overall privacy guarantee.
- Experimental Results: Agents with smaller local datasets generally receive larger accuracy gains and can almost catch up with better-endowed agents.
- Experimental Results: Local differential privacy causes a huge accuracy loss, and collaboration on perturbed data does not significantly improve those models.
D.2 Recommendation Task
The recommendation experiment uses normalized user ratings, learned movie features, gradient clipping for privacy calibration, and validation-based tuning of private iterations.
- Experimental Setup: Ratings are normalized user-wise by subtracting each user’s average rating.
- Experimental Setup: Movie features are generated with alternating least squares on a random 80% training split, using feature dimensionality p = 20.
- Experimental Setup: Point-wise gradients are clipped when their norms exceed C to avoid overestimating the Lipschitz constant and adding excessive privacy noise.
- Experimental Setup: Each agent uses λ_i = 1/m_i and μ = 0.04, while private iterations are tuned for each ε̄ on a validation split.