Source-linked AI summary
SplitFed: When Federated Learning Meets Split Learning
Chandra Thapa, M. A. P. Chamikara, Seyit Camtepe, Lichao Sun
TL;DR
FL burdens clients with full-model computation and exposes model information, while SL improves privacy and resource suitability but suffers relay-based training overhead. The paper proposes SFL, combining parallel client processing with network splitting and privacy measures. SFL achieves similar accuracy to SL while training faster than SL, with privacy and robustness further evaluated using differential privacy and PixelDP.
Problem
FL requires clients to run full models and exposes local and global models, while SL incurs substantial training overhead from relay-based training.
Method
SFL combines FL’s parallel client processing with SL’s network splitting, adding differential privacy and PixelDP measures.
Results
SFL provides similar model accuracy to SL and is faster than SL through parallel processing across clients.
Takeaways & Limitations
SFL supports machine learning with resource-constrained devices while enabling fast training for periodically updated global models.
Takeaways & Limitations
The observation window was limited to 100 or 200 global epochs because some architectures required extensive training time on shared computing resources.
Abstract
from arXiv · showhide
Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. Both follow a model-to-data scenario; clients train and test machine learning models without sharing raw data. SL provides better model privacy than FL due to the machine learning model architecture split between clients and the server. Moreover, the split model makes SL a better option for resource-constrained environments. However, SL performs slower than FL due to the relay-based training across multiple clients. In this regard, this paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches eliminating their inherent drawbacks, along with a refined architectural configuration incorporating differential privacy and PixelDP to enhance data privacy and model robustness. Our analysis and empirical results demonstrate that (pure) SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients. Furthermore, as in SL, its communication efficiency over FL improves with the number of clients. Besides, the performance of SFL with privacy and robustness measures is further evaluated under extended experimental settings.
1 Introduction
FL trains complete models in parallel but burdens clients and exposes full models, while SL reduces client computation and improves model privacy yet incurs relay-based overhead. SFL combines these strengths for resource-constrained, fast-training settings.
- FL trains a complete model at each client in parallel, enabling efficient training across many clients.
- FL requires resource-constrained clients to run the full model and gives servers and clients access to local and global models.
- SL splits the model between client and server, reducing client processing load and preventing either side from accessing the other’s model.
- SL’s relay-based training leaves clients idle because only one client engages with the server at a time, increasing training overhead with many clients.
- SFL combines FL and SL to improve model privacy and robustness while providing performance similar to SL and faster training than SL.
- SFL targets environments where full-model training is infeasible and rapid global-model updates are required, including healthcare and finance.
2 Background and Related Works
FL and SL distribute model training differently: FL aggregates complete client models, whereas SL partitions a network across clients and servers. Differential privacy formalizes privacy with budgets that trade off against added noise.
- Federated learning: FL trains complete networks locally, sends client updates for server aggregation, and returns the global model for subsequent rounds.
- Split learning: SL partitions a deep learning network into client-side and server-side portions, communicating split-layer activations between them.
- Differential privacy: Differential privacy defines privacy through a stochastic framework.
- Differential privacy: A mechanism is (ϵ, δ)-differentially private when its outputs satisfy the stated bound for adjacent datasets and all result subsets.
- Differential privacy: Smaller ϵ and δ increase privacy but require more noise in the differential-privacy algorithm.
3 The Proposed Framework
SFL combines SL’s split client/server network with FL’s parallel client processing, while adding differential privacy and PixelDP protections. Its workflow separates client-side and server-side computation, aggregates client-side updates through a fed server, and analyzes communication and training costs.
- Overall Structure: SFL combines network splitting between client-side and server-side sub-networks with parallel processing across distributed clients.Clients process their client-side models in parallel, while the main server handles client-specific smashed data and server-side computation.
- Overall Structure: SFL uses a main server for server-side forward and backward propagation and a fed server for aggregating and synchronizing client-side model updates.The main server returns smashed-data gradients to clients, after which client-side updates are aggregated through FedAvg.
- Variants: SFL includes SFLV1 and SFLV2 variants that differ in whether server-side models are aggregated during server computation.SFLV2 removes the server-side model aggregation step to potentially increase model accuracy.
- Privacy Protection: Differential privacy clips client-side gradient norms and adds calibrated noise to limit the influence of individual training examples.Gradients above C′ are scaled down to C′ before calibrated noise is applied.
- Privacy Protection: A client-side noise layer based on PixelDP privatizes smashed data sent to the main server and can improve robustness against adversarial examples.The private smashed representation is produced with calibrated noise while maintaining utility; detailed mathematical guarantees are deferred.
- Total Cost Analysis: As the number of clients increases, total training-time cost is ordered SFLV2<SFLV1<SL under the paper’s cost analysis.The analysis attributes SL’s overhead to client-side model uploads and downloads repeated across clients, whereas SFL processes clients in parallel on the server.
4 Experiments
Experiments compare FL, SL, SFLV1, and SFLV2 across image datasets, architectures, client counts, convergence, client-level variation, and privacy settings. Results generally show similar accuracy across methods in several settings, while client count, convergence limits, and privacy budgets affect performance.
- Experimental setup: Experiments use four public image datasets with uniformly distributed, horizontally partitioned data across clients.Implementations use Python and PyTorch on an HPC platform, with clients and servers running on separate computing nodes.
- Experimental setup: Comparisons split LeNet, AlexNet, VGG16, and ResNet18 at specified intermediate layers for SL, SFLV1, and SFLV2.Differential privacy and PixelDP are excluded from the comparative evaluations of SFLV1 and SFLV2 with FL and SL.
- Performance comparison: Around 76% test accuracy was reached by FL, SL, SFLV1, and SFLV2 for ResNet18 on HAM10000 within 200 global epochs.The convergence was reported as almost the same across these methods under similar settings with five clients.
- Client-level variation: CV values for SL, FL, SFLV1, and SFLV2 ranged from 0.06 to 2.63 during training and from 0.54 to 6.72 during testing on ResNet18 with HAM10000.The reported CV values were below 10, a range characterized as good in the cited literature and interpreted as uniform individual client-level performance.
- Effect of client count: Around 74% (FL), 77% (SL), 75% (SFLV1), and 77% (SFLV2) average test accuracy was observed at 100 global epochs for ResNet18 on HAM10000 with 100 clients.Increasing clients from five to 100 had a negligible performance effect in this experiment, although other dataset-architecture combinations degraded or failed to converge.
- Privacy evaluation: 40%, 64%, 73%, 77%, and 78% testing accuracy was observed at global epoch 50 for ε′ values of 0.5, 1, 2, 5, and no PixelDP, respectively.The SFLV1 AlexNet-on-MNIST experiment maintained ε = 0.5; convergence was gradual and slow with differential privacy measures, and utility decreased with a lower privacy budget.
5 Conclusion
SFL combines FL and SL to provide model privacy, parallel training, and support for resource-constrained devices. Its accuracy is similar to SL, while privacy and robustness extensions are evaluated separately.
- SFL combines network splitting from SL with parallel client processing from FL.Network splitting supports resource-constrained devices, while parallel handling enables faster training.
- SFL offers model privacy through network splitting and differential-private client-side model updates.
- SFL provides similar model accuracy to SL while training faster through parallel processing across clients.
- Differential privacy and PixelDP extensions were analyzed for data privacy and model robustness.
- Detailed privacy–utility trade-offs and homomorphic-encryption integration remain future work.
Appendix
The security analysis assumes honest-but-curious, passive servers and a standard client-server trust relationship during training.
- Servers are modeled as honest-but-curious adversaries that follow assigned tasks while seeking private client data and the full model.
- The server attack model is passive: servers observe updates and may perform calculations to obtain information.
- SFLV1 and SFLV2 assume clients and servers establish a level of trust before network model training begins.
- In the stated health-domain example, hospitals may opt out when the platform has malicious clients or servers.
B Total Cost Measurement
The total-cost evaluation combines theoretical and empirical analyses of communication and training time across FL, SL, SFLV1, and SFLV2.
- The study analyzes communication cost and model training time theoretically for FL, SL, SFLV1, and SFLV2.
- Empirical tests measure communication and training time for the same four learning approaches.
- The empirical results complement the total-cost analysis.
B.1 Communication Measurement
Communication measurement evaluates how much data clients upload and download, because communication affects the practicality of distributed learning in resource-constrained environments.
- Communication volume indicates whether a distributed learning approach can operate in resource-constrained environments.High communication slows training and testing and requires clients to handle greater communication costs.
- High data communication can slow machine-learning training and testing.
- SL is more communication-efficient than FL as the number of clients or model size increases.
- FL is preferred when data samples increase while client count and model size remain low.
- Figure 6 measures average upload and download data per client per global epoch for ResNet18 on HAM10000 and AlexNet on MNIST.
- Figure 7 reports training time per global epoch.
B.2 Time Measurement
SFLV1 and SFLV2 reduce training time per global epoch relative to SL when multiple clients participate, with performance approaching or exceeding FL.
- SFL is faster than SL for multiple clients because its server processes clients in parallel, avoiding SL’s client-count product terms.In SL, client-side model uploads and downloads repeat across the total number of clients for each global epoch.
- Four to six times faster, SFLV1 and SFLV2 significantly reduced training time versus SL with multiple clients.This result was observed for ResNet18 on HAM10000 and AlexNet on MNIST.
- SFLV1 and SFLV2 achieved similar or better speed than FL in multiple-client cases.For a single client, SL and FL had similar times, while both SFL variants were slightly slower.
C.1 Performance of FL, SL, SFLV1, and SFLV2
Across five-client convergence experiments, FL, SL, SFLV1, and SFLV2 generally show similar training and testing behavior, although outcomes vary by dataset and architecture.
- FL, SL, SFLV1, and SFLV2 exhibit similar training and testing characteristics in most five-client experiments.The experiments span multiple datasets and model architectures, including HAM10000, MNIST, FMNIST, and CIFAR10.
- The best-performing approach depends on the dataset, with CIFAR10 often favoring FL in similar settings.SL and both SFL variants retain lower client-side computation and model-privacy advantages from split architectures.
- SL, SFLV1, and SFLV2 can show no learning in some naive implementations, including VGG16 on CIFAR10.The paper states that this case requires further investigation.
- FedAvg in the server-side network is not always efficient among SFLV1 and SFLV2, although it contributes to promising results in many cases.AlexNet on HAM10000 is given as an example where FedAvg contributes to promising results in FL and SFLV1.
C.2 Effects of Number of Users on the Performance
As the number of clients increases from 5 to 100, convergence usually slows and performance degrades within the observed global-epoch window, with similar behavior across distributed approaches.
- Convergence usually slows and performance degrades as the number of users increases from 5 to 100.The comparison uses LeNet5 on FMNIST and AlexNet on HAM10000.
- All evaluated distributed collaborative machine learning approaches show similar behavior across different numbers of clients.The observation concerns the varied-client experiments reported in Fig. 9.
- SL can show a sharp testing-performance fall despite learning well during training, as with AlexNet on HAM10000 at 100 users.The paper identifies this case as requiring further investigation.
- SFLV2 shows a similar testing-performance fall at 50 users in the corresponding setting.The paper relates this behavior to SFLV2’s similarity to SL in server-side operations.