Source-linked AI summary
Decentralized Federated Learning: A Segmented Gossip Approach
Chenghao Hu, Jingyan Jiang, Zhi Wang
TL;DR
Federated learning avoids sharing original data, but conventional centralized designs are poorly matched to limited and distributed network capacity. The paper introduces model-segment-level decentralized federated learning with segmented gossip, reporting substantially shorter training time while retaining convergence and only slight accuracy degradation.
Problem
Conventional parameter-server federated learning faces limited inter-node bandwidth and network congestion in geo-distributed settings.
Method
Combo uses segmented gossip aggregation, dynamically exchanging model segments among peers and handling workers that become unreachable.
Results
Combo significantly reduces training time while maintaining good convergence performance and only slight accuracy degradation.
Takeaways & Limitations
Decentralized segmented aggregation can use peer-to-peer bandwidth more fully in practical federated-learning network settings.
Takeaways & Limitations
The convergence proof is deferred to an extended version, and the bound approaches a noise ball rather than the exact optimum.
Abstract
from arXiv · showhide
The emerging concern about data privacy and security has motivated the proposal of federated learning, which allows nodes to only synchronize the locally-trained models instead their own original data. Conventional federated learning architecture, inherited from the parameter server design, relies on highly centralized topologies and the assumption of large nodes-to-server bandwidths. However, in real-world federated learning scenarios the network capacities between nodes are highly uniformly distributed and smaller than that in a datacenter. It is of great challenges for conventional federated learning approaches to efficiently utilize network capacities between nodes. In this paper, we propose a model segment level decentralized federated learning to tackle this problem. In particular, we propose a segmented gossip approach, which not only makes full utilization of node-to-node bandwidth, but also has good training convergence. The experimental results show that even the training time can be highly reduced as compared to centralized federated learning.
1 Introduction
Federated learning protects data by synchronizing locally trained models, but centralized coordination and limited peer bandwidth make efficient synchronization difficult. The paper proposes segmented gossip, which exchanges model segments through dynamically selected peers to use node-to-node bandwidth while preserving convergence.
- Federated learning lets nodes synchronize locally trained models instead of sharing original data.
- Centralized federated learning relies on a parameter server that periodically receives model updates from participating workers.
- All-reduce avoids central coordination but sends updates to every worker, consuming substantial inter-worker bandwidth.
- Model updates can reach 1360MB in BERTLARGE, while real-world inter-node capacities are smaller and relatively uniform than datacenter links.
- Segmented gossip transmits model segments peer-to-peer and forms dynamic synchronization groups to improve bandwidth utilization and training convergence.
- Workers split models into non-overlapping equal-parameter segments and aggregate each local segment with the corresponding segment from k other workers.The analysis indicates that k can be much smaller than the total worker count while still achieving good convergence.
- Workers stochastically select a few peers per iteration to transfer model segments, while Model Replica supplies additional information during aggregation.
- Combo experiments significantly reduce training time in practical network settings with only slight accuracy degradation.
2 Related work
Prior distributed and federated learning systems reduce coordination bottlenecks or communication volume through scaling, topologies, partitions, FedAvg, and gossip. However, frequent aggregation, multi-hop propagation, parameter-server congestion, and WAN bottlenecks remain problematic for geo-distributed federated learning.
- Centralized distributed ML systems periodically send local updates to parameter servers, but poor networks can still produce long convergence times.
- All-reduce has communication cost O(n^2) for n workers, while ring, tree, and graph topologies reduce communication complexity but may require multiple hops.
- Ako partitions gradients to reduce synchronization time, with communication overhead determined by partition size and worker number.
- Existing decentralized approaches aggregate gradients every epoch, which remains costly for federated learning over slow internet connections.
- Structured and sketched updates reduce exchanged data size at the cost of accuracy loss, while FedAvg aggregates parameters after several epochs.
- Parameter-server methods face network congestion when updates arrive concurrently at the server.
- Gossip-based ML methods exchange messages among selected workers and can provide consensus convergence or reduce communication complexity to O(1).
- Traditional gossip schemes do not fully use geo-distributed workers’ bandwidth because transmissions are limited to one or a few WAN links.
3 Segmented Gossip Aggregation
Segmented gossip addresses geo-distributed federated learning’s limited inter-worker bandwidth by splitting model transfers across links while preserving decentralized aggregation. Workers pull model segments from multiple peers, build mixed models, and aggregate each segment separately, with Model Replica R trading transmission overhead for faster propagation and model quality.
- Motivation and design: Segmented Gossip Aggregation splits model transmission across multiple peer-to-peer links to use geo-distributed workers’ bandwidth more fully.Traditional gossip limits transmissions to one or a few links, whereas segmented gossip feeds the transmission task into more links.
- Segmented pulling: Each worker pulls different model segments from different peers in parallel, then rebuilds a mixed model for aggregation.This keeps total transmission equal to one model while distributing traffic across S links rather than one.
- Model Replica: Model Replica R controls propagation: rebuilding R mixed models requires pulling S × R segments, increasing bandwidth overhead but helping preserve model quality.The design uses extra transmission as a compromise for faster update propagation in the absence of a centralized server bottleneck.
- Segmented aggregation: Segment-wise aggregation weights each segment using the dataset sizes of workers that provide that segment, rather than weighting each patched mixed model as a whole.The approach aggregates the received segments and then combines the aggregated segments into the final model.
- Training cycle: After rebuilding the final aggregation result, the worker resumes local training until the next aggregation phase.The final model is formed by combining all aggregated segments.
4 Combo Design
Combo is a decentralized federated learning system whose workers train locally, exchange model segments through gossip, and aggregate received segments iteratively. Its design also addresses dynamic participation and analyzes convergence through gradient and aggregation divergence.
- 4.1 Implementation Details: Before training, a server initializes every worker with identical model parameters and broadcasts the worker list.The paper states that differing initial parameters may prevent convergence.
- 4.1 Implementation Details: Combo workers repeatedly perform local SGD, exchange segment requests and responses, aggregate the received segments, and begin the next iteration.The worker uses the previous aggregation result as input; communication may follow multiple local SGD rounds.
- 4.1 Implementation Details: Combo distributes S × R pulling requests across workers to engage more links and balance transmission workload.Targets are randomly selected without replacement when enough workers are available.
- 4.2 Dynamic Workers: Offline workers trigger request cancellation or reassignment, while newcomers obtain segments from other workers before starting local training.New workers register with the server, request the worker list, and are disseminated through their pulling requests.
- 4.3 Convergence Analysis: The convergence gap consists of gradient divergence δ and aggregation divergence ρ, with larger communication intervals τ exacerbating ρ's influence.The bound approaches a noise ball around the optimum rather than the optimum itself; increasing replicas can reduce aggregation divergence.
5 Performance Evaluation
The evaluation simulates Combo and compares it with FedAvg and naive gossip under a constrained peer-to-peer network. Combo converges faster, reduces synchronization time through model segmentation, and exposes a tradeoff between replica count, convergence, and communication cost.
- 5.1 Setup: Combo is evaluated through sequential training-trace simulation followed by network-topology simulation to estimate training time.The traces record validation accuracy, iterations, and synchronization partners; communication time uses simulated bandwidth limits.
- 5.1 Setup: The experiments compare Combo with FedAvg and naive gossip using a fully connected topology with 100Mbps worker limits and 10Mbps peer links.Combo uses S = 10 model segments and R = 2 replicas by default.
- Convergence Speed: Combo reaches the target accuracy faster without affecting final validation accuracy, requiring the least training time across 20, 30, and 40 workers.The target is 80% validation accuracy, while convergence occurs around 82%.
- Convergence Speed: 2.25× to 3.01× speedup over FedAvg is observed as the worker scale expands.The reported speedup increases across the evaluated scales of 20, 30, and 40 workers.
- Benefit of Model Segments: Ten segments preserve the convergence trend, while two segments reduce synchronization time by half and additional segmentation beyond six provides no further time improvement.The benefit arises from dividing transmission across more links until bandwidth is fully exploited.
- Impact of Model Replicas: Increasing replicas improves per-iteration accuracy but creates a tradeoff because transmission workload grows proportionally with replica count.Increasing R from 1 to 2 rapidly reduces total training time, whereas larger values slow convergence in time.
6 Conclusion
The paper addresses poor WAN connectivity in geo-distributed federated learning with Combo, a decentralized segmented-gossip design. Experiments report reduced training time while retaining good convergence performance.
- 6 Conclusion: Combo uses segmented gossip aggregation to saturate peer-to-peer bandwidth in decentralized federated learning.The design targets geo-distributed workers connected through slow WAN links.
- 6 Conclusion: Experiments show that Combo significantly reduces training time while retaining good convergence performance.