Source-linked AI summary
Peer-to-peer Federated Learning on Graphs
Anusha Lalitha, Osman Cihan Kilinc, Tara Javidi, Farinaz Koushanfar
TL;DR
The paper addresses decentralized model learning when nodes hold private, potentially insufficient local data and no central server coordinates training. It formulates learning as peer-to-peer social learning with Bayesian-like beliefs, analyzes error guarantees, and uses variational inference for DNNs. Experiments report negligible accuracy loss for decentralized DNN training and encouraging decentralized-learning results, while extensive studies across DNN architectures remain future work.
Problem
The paper studies how to learn a shared global model from private, localized data without collecting all data or maintaining a central controller.
Method
Nodes iteratively update Bayesian-like beliefs and aggregate information from one-hop neighbors, using variational inference to make the approach applicable to DNNs.
Results
Experiments report negligible accuracy drop for decentralized DNN training relative to a central node with all training data, alongside encouraging decentralized-learning results.
Takeaways & Limitations
The proposed learning rule supports collaborative model estimation over networks without a central server, including an approximation suitable for DNN training.
Takeaways & Limitations
The paper identifies extensive empirical evaluation across varied DNN architectures as future work.
Abstract
from arXiv · showhide
We consider the problem of training a machine learning model over a network of nodes in a fully decentralized framework. The nodes take a Bayesian-like approach via the introduction of a belief over the model parameter space. We propose a distributed learning algorithm in which nodes update their belief by aggregate information from their one-hop neighbors to learn a model that best fits the observations over the entire network. In addition, we also obtain sufficient conditions to ensure that the probability of error is small for every node in the network. We discuss approximations required for applying this algorithm to train Deep Neural Networks (DNNs). Experiments on training linear regression model and on training a DNN show that the proposed learning rule algorithm provides a significant improvement in the accuracy compared to the case where nodes learn without cooperation.
1. Introduction
The paper extends federated learning from a server-centered setting to peer-to-peer learning over graphs with localized data. It introduces a Bayesian-like social-learning formulation, analyzes its guarantees, and approximates it for DNN training.
- Conceptual contributions: Fully decentralized federated learning lets nodes communicate only with one-hop neighbors, without a central controller or globally collected training data.The framework is designed for settings where privacy concerns prevent nodes from sharing raw data.
- Motivation: Localized observations may be insufficient for an individual node to identify the shared global model, motivating collaboration across neighboring nodes.The toy distributed linear-regression example illustrates this deficiency when node types observe different input regions.
- Conceptual contributions: The paper formulates federated machine learning as social learning on a graph and uses a Bayesian-like belief over the unknown global parameter.This connection draws on tools related to consensus, belief propagation, and distributed hypothesis testing.
- DNN approximation: Variational inference replaces computationally intractable Bayesian marginalization when adapting the learning framework to DNNs.The approximation converts the Bayesian modeling step into an optimization problem.
- Empirical validation: Experiments with two nodes training a shared DNN show little to no accuracy drop relative to a central node with access to all training data.The paper also reports theoretical learning of the true parameter in the specialized regression example.
2. Problem Setup
The problem setup distributes local datasets and likelihood models across nodes connected by a directed graph. Global learning requires a commonly learnable parameter and network connectivity sufficient to disseminate information.
- The Model: Each node observes a local dataset drawn from its local instance space, while the global instance space is covered by the union of these local spaces.The label space supports examples such as real-valued regression and binary classification.
- The Model: The model uses a finite parameter set and node-specific likelihood functions to evaluate how well each parameter explains local observations.The setup defines local expected KL divergence between the true label distribution and candidate likelihoods.
- The Model: A parameter is globally learnable when it belongs to the intersection of the nodes' locally optimal parameter sets.The framework assumes that at least one globally learnable parameter exists.
- The Model: If a node's local input space leaves multiple locally compatible parameters, its local dataset cannot identify the true parameter alone, whereas collaboration can enable learning it.This condition motivates information exchange across the network.
- The Communication Network: The communication network is a directed graph represented by a stochastic matrix whose weights encode each node's confidence in information received from neighbors.The analysis assumes a strongly connected, aperiodic graph so information can spread throughout the network.
- The Learning Criterion: The learning criterion requires every node to agree on a parameter that best fits the data distributed across the entire network.The estimate at each node is evaluated after observing its local instance-label pairs.
3. Peer-to-peer Federated Learning Algorithm
The peer-to-peer algorithm has each node maintain a private belief over the parameter space and repeatedly combine local learning with information from neighboring nodes.
- Algorithm: Each node maintains a private belief vector over the parameter space at every algorithmic instant.The beliefs are updated over successive rounds indexed by k.
- Algorithm: The algorithm distributes learning by having nodes execute the same iterative procedure using information exchanged over the network.This procedure is based on distributed hypothesis testing methods for cooperative model learning.
4. Analysis of Learning Algorithm
The paper analyzes peer-to-peer Bayesian learning under technical assumptions and derives finite-sample guarantees for identifying globally learnable parameters and controlling network-wide risk.
- Learning algorithm: Algorithm 1 alternates local Bayesian updates with consensus exchanges among neighboring nodes.Each node updates its belief using local observations, then averages log beliefs received from neighbors.
- Assumptions: The analysis assumes positive prior beliefs, bounded likelihoods, and a globally realizable parameter represented through a compact parameter set and finite covering.The continuous model parameter θ belongs to compact Φ, while Θ provides an r-covering of Φ.
- Guarantees: Theorem 1 provides a high-probability finite-sample bound for distributed learning over a finite parameter set.The bound depends on the confidence level, network size, parameter count, stationary distribution, and convergence properties of W.
- Guarantees: The required sample count grows logarithmically with network size and the number of parameters to distinguish.It also worsens when parameter distinguishability is weak or when W converges slowly to its stationary distribution.
- Risk bound: Under realizability and bounded-risk assumptions, the algorithm yields a high-probability upper bound on average expected risk across the network.The risk result follows from Theorem 1 together with Pinsker’s and Jensen’s inequalities.
5. Experiments
Experiments evaluate peer-to-peer Bayesian learning on distributed linear regression and Fashion-MNIST DNN classification. Cooperation improves performance with incomplete or heterogeneous local data and can match centralized performance in the reported settings.
- Distributed Bayesian Linear Regression: In linear regression, Algorithm 1 matches the central node’s test-set MSE, whereas noncooperating nodes perform worse because local information is insufficient.The experiment uses two nodes with complementary local feature observations and compares cooperation, no cooperation, and centralized training.
- Training DNN models: Variational inference replaces intractable Bayesian marginalization with optimization for practical DNN training.The method uses approximating distributions and can update them from batches of observations; consensus can use unnormalized beliefs.
- Training DNN models: With IID data, the central node reaches 88.28% accuracy, while cooperation produces 87.43% and 87.84% at the two nodes without accuracy loss from cooperation.The corresponding noncooperating accuracies are 87.07% and 87.43%.
- Training DNN models: In non-IID balanced case (a), cooperation raises node accuracies from 44.89% and 48.22% to 83% and 67%.The local label distributions leave each node weak on classes emphasized in the other node’s data.
- Training DNN models: In non-IID balanced case (b), cooperation raises accuracies from 40.4% and 47.8% to 85.78% and 85.86%.The paper reports that distributed training turns nodes expert on local tasks into experts on the network-wide task.
- Training DNN models: In the non-IID unbalanced case, cooperation raises accuracies from 69.44% and 19.95% to 85.8% and 85.2%.The paper attributes the improvement to one network-wide expert pulling up the other node’s accuracy.
6. Discussion and Future Work
The paper concludes that peer-to-peer learning can achieve encouraging accuracy with negligible decentralized-training loss, while identifying broader empirical and application directions for future work.
- The peer-to-peer algorithm lets nodes aggregate one-hop neighbors’ beliefs to estimate the globally optimal parameter without a central server.
- Experiments report encouraging results with negligible accuracy loss when training occurs in a decentralized manner.
- Extensive empirical studies across various DNN architectures remain an important area of future work.
- Applying the learning algorithm to random graphs and decentralized reinforcement learning is identified as an additional future direction.
A. Proof of Theorem 1
The proof establishes error guarantees for decentralized learning by combining graph-mixing properties, likelihood-ratio concentration, and a finite-sample condition.
- The proof begins from a recursion for each node and parameter outside the optimal set, using graph-weight properties and bounded log-likelihood ratios.
- McDiarmid’s inequality provides concentration for every nonoptimal parameter, yielding a sample-size condition dependent on C, N, ϵ, and 1−λmax(W).
- The resulting bound controls the worst-case error across all parameters and nodes in the network.
- n ≥ 16C log NM δ K(Θ)^2(1 −λmax(W)) is a sufficient lower bound on training samples for the stated confidence guarantee.