Source-linked AI summary
Detailed comparison of communication efficiency of split learning and federated learning
Abhishek Singh, Praneeth Vepakomma, Otkrist Gupta, Ramesh Raskar
TL;DR
The paper asks how to train deep networks across distributed, data-sensitive sources without excessive communication or local computational burden. It compares split and federated learning analytically and across practical scenarios with varying client counts, dataset sizes, and model sizes. Split learning becomes more communication efficient as client count or model size increases, while federated learning becomes more efficient as data samples increase when client count or model size is small.
Problem
The paper examines how distributed deep learning can avoid excessive communication bandwidth and computational burden while training from data sources that cannot share raw data.
Method
The paper compares split and federated learning using communication calculations and practical scenarios that vary client counts, dataset sizes, and model sizes.
Results
Split learning becomes more communication efficient with more clients or model parameters, whereas federated learning becomes more efficient with more data samples when client count or model size is small.
Takeaways & Limitations
The analysis identifies practical use cases in which either split learning or federated learning is more effective in communication efficiency.
Abstract
from arXiv · showhide
We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.
1 Introduction
The paper compares split learning and federated learning for privacy-preserving distributed training, focusing on communication bandwidth and computational burden across distributed data sources.
- Both methods train models across distributed data sources without sharing raw data, but their communication efficiencies may differ as devices and model complexity increase.
- The study asks how distributed deep learning can use data from smart watches, hospitals, word corpora, and biobanks without excessive communication or local computation.
- Split learning: Split learning partitions the neural network between a client and server, with the client handling beginning layers and the server handling remaining layers.
- Split learning: Split learning transmits client-side output tensors and server-side gradients while keeping raw data and the other party’s model details private.
- Federated learning: Federated learning keeps a complete model copy on each client, uploads client weight updates for server averaging, and downloads the updated weights.
2 Communication efficiency
The paper derives communication costs for split and federated learning by tracking client data transfer and weight synchronization, then uses a hyperbola to separate their efficiency regions.
- Communication efficiency is calculated from data transferred by each client during training and client weight synchronization.
- With equal client datasets, split learning communicates (p/K)q activations forward and (p/K)q gradients backward per client.
- Federated learning uploads and downloads weights or gradients of size N each, while split learning may add ηN communication for client weight sharing.
- Split learning has weight-sharing and alternating-epoch variants, with weight sharing improving synchronization but adding communication overhead and information leakage.
- The derived hyperbola divides regions where split learning or federated learning has higher communication efficiency.
3 Analysis
The analysis tests communication efficiency across practical scenarios that vary client counts, model sizes, and dataset sizes.
- Smart watches: In smart-watch settings with millions of clients, split learning uses less data transfer for large models, while federated learning is more efficient as models and client counts decrease.
- Healthcare: In healthcare settings with large models and few clients, the methods perform roughly similarly, while federated learning performs better with larger datasets and fewer clients.
- The analysis includes practical efficiency curves whose three lines represent different positions of the split layer and use activation sizes from early CNN layers.
4 Conclusion and future work
The analysis identifies conditions under which split learning or federated learning is more communication efficient, while noting future extensions to resource use and convergence. Split learning favors larger client populations and models, whereas federated learning favors more data when clients or models are small.
- Split learning becomes more communication efficient as the number of clients increases and remains highly scalable with model parameter count.
- Federated learning becomes more efficient as the number of data samples increases, particularly when the number of clients or model size is small.
- Split learning is reported to converge drastically quicker than federated learning.
- Future work could analyze resource utilization and the number of epochs required for convergence under different practical scenarios.