PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things

Yansong Gao, Minki Kim, Sharif Abuadbba, Yeonjae Kim, Chandra Thapa, Kyuyeon Kim, Seyit A. Camtepe, Hyoungshick Kim, Surya Nepal

arXiv:2003.13376v2cs.CRcs.DCcs.LG

TL;DR

The paper asks how FL and SplitNN compare for learning performance and device overhead in real-world IoT settings without centralizing sensitive data. It evaluates both methods across data distributions and Raspberry Pi deployments, finding distribution-dependent learning differences and lower communication overhead for FL. It also identifies practical constraints and limitations affecting SplitNN and resource-constrained IoT deployment.

  • Problem

    Existing evidence did not comprehensively compare FL and SplitNN on learning performance under imbalanced or non-IID data and implementation overhead in real-world IoT settings.

  • Method

    The paper empirically evaluates FL and SplitNN across distributed data conditions, datasets, model settings, and Raspberry Pi deployments using learning and device-overhead metrics.

  • Results

    SplitNN performs better than FL with imbalanced data but worse under extreme non-IID data, while FL has relatively lower communication overhead in the IoT evaluation.

  • Takeaways & Limitations

    FL is preferred when communication cost is critical, whereas SplitNN is recommended when communication is less important except under extreme non-IID data.

  • Takeaways & Limitations

    Complicated models with several million parameters are challenging to run on Raspberry Pi devices, motivating evaluation with a smaller 1D CNN.

Abstract

from arXiv · show

This work is the first attempt to evaluate and compare felderated learning (FL) and split neural networks (SplitNN) in real-world IoT settings in terms of learning performance and device implementation overhead. We consider a variety of datasets, different model architectures, multiple clients, and various performance metrics. For learning performance, which is specified by the model accuracy and convergence speed metrics, we empirically evaluate both FL and SplitNN under different types of data distributions such as imbalanced and non-independent and identically distributed (non-IID) data. We show that the learning performance of SplitNN is better than FL under an imbalanced data distribution, but worse than FL under an extreme non-IID data distribution. For implementation overhead, we end-to-end mount both FL and SplitNN on Raspberry Pis, and comprehensively evaluate overheads including training time, communication overhead under the real LAN setting, power consumption and memory usage. Our key observations are that under IoT scenario where the communication traffic is the main concern, the FL appears to perform better over SplitNN because FL has the significantly lower communication overhead compared with SplitNN, which empirically corroborate previous statistical analysis. In addition, we reveal several unrecognized limitations about SplitNN, forming the basis for future research.

I. INTRODUCTION

The paper addresses the lack of broad empirical comparisons between FL and SplitNN for privacy-sensitive, distributed IoT data. It evaluates learning performance and implementation overhead across realistic data distributions and resource-constrained devices.

  • Motivation: Sensitive client-side data creates demand for training deep learning models without aggregating or accessing raw data.Examples include medical-sensor and microphone data, which users may resist sharing with service providers.
  • Motivation: Distributed learning keeps data local while training a joint model across clients, reducing privacy leakage risks.The paper frames this approach as especially relevant to data generated by distributed IoT devices.
  • Research gap: Prior work had not evaluated SplitNN learning performance under imbalanced or non-IID data, nor compared it with FL under those conditions.Existing comparisons considered communication efficiency but not model accuracy and convergence speed.
  • Research gap: The paper also addresses the absence of end-to-end overhead studies for FL and SplitNN on real-world IoT hardware such as Raspberry Pi.The overhead scope includes communication cost, power consumption, and training time.
  • Contributions: SplitNN outperforms FL with imbalanced data but underperforms FL with extreme non-IID data, while its accuracy remains sensitive to distributed-data characteristics.The comparison covers model accuracy and convergence with up to simulated 100 clients.
  • Study scope: The study evaluates learning performance and Raspberry Pi implementation overhead for both distributed learning methods across experiments and datasets.The paper organizes its evaluation around distributed-learning models, datasets, learning performance, and device overhead.

A. Federated Learning

FL trains complete models locally and aggregates client updates, whereas SplitNN partitions a network between clients and a server. The paper also describes FedAvg, SplitNN privacy and computation properties, and ensemble-learning extensions.

  • Federated Learning: FL sends a global model to clients, has each client train on local data, and aggregates updated client models into the next global model.This repeated process is organized into communication rounds until convergence.
  • Federated Learning: FedAvg lets each client train for several local epochs before sending its model to the server in one communication round.It is described as a commonly used FL optimization method.
  • Federated Learning: FedAvg usually works well for nonconvex problems but has no convergence guarantees and may diverge with non-IID or imbalanced client data.The stated limitation concerns practical FL settings with heterogeneous data distributions.
  • Split Learning: SplitNN divides a neural network into client and server subnetworks, with a cut layer separating the portions trained by each party.The server lacks access to the clients’ subnetworks and data, while clients train only a few layers.
  • Evaluation scope: SplitNN learning performance under non-IID and imbalanced data was an evaluation target of this work.The passage identifies this setting as previously uninvestigated for SplitNN.
  • Ensemble Learning: SplitNN training can use idle clients to obtain multiple models through ensemble learning rather than training each model sequentially.The described arrangement uses a resourceful server and round-robin client interactions.

D. Datasets

The paper evaluates privacy-sensitive speech-command and ECG datasets collected by IoT devices, focusing on how client data distribution affects SplitNN and FL learning performance.

  • Dataset motivation: Both datasets represent privacy-sensitive sequential or time-series data commonly collected and processed by IoT devices.SC is personalized, while ECG is medical data.
  • Speech Command (SC): Speech Command (SC) recognition uses 20,827 one-second audio samples across 10 spoken-digit classes, with 11,360 samples for training.The samples are spoken by a variety of speakers.
  • Electrocardiogram (ECG): The MIT-BIH ECG dataset contains 26,490 samples representing five heartbeat types, split evenly between training and testing.The targets are N, L, R, A, and V heartbeat types.
  • Research questions: The experiments address which factors affect SplitNN performance and when SplitNN outperforms FL under distributed client data settings.The questions include client count, non-IID data, and imbalanced data.

A. IID and Balanced Dataset

Under ideal IID and balanced data, SplitNN converges faster than FL with one local epoch, but its accuracy can become unstable and remain below centralized-model baselines. Increasing client counts particularly expose these limitations.

  • SplitNN converges relatively faster than FL with one local epoch across experiments using 2, 5, 50, and 100 clients.FL has greater difficulty converging as the number of clients increases.
  • SplitNN testing accuracy drops after reaching an optimum, so additional rounds may waste training time rather than improve accuracy.Its learning curve also shows many spikes, indicating unstable training.
  • 85.29% for SC and 97.78% for ECG are the centralized-model baseline accuracies that SplitNN cannot reach.The limitation is especially clear with 50 or 100 clients.
  • SplitNN’s accuracy and convergence performance do not always match centralized training under the evaluated IID and balanced setting.The study reports a more generalized finding by not assuming preserved arrival order or identical weight initialization.
  • The experiments use SC and ECG datasets with a 4conv+2dense 1D CNN, learning rate 0.001, and batch size 32 unless otherwise stated.Figure 4 identifies the imbalanced data setting, while this section begins with IID and balanced data.

B. Imbalanced Data Distribution

Under imbalanced data, SplitNN is less sensitive and converges faster than FL, although its accuracy can deteriorate as the client count increases. The experiments simulate normally distributed client data, applying the same distribution to both methods.

  • Experimental setting: Client sample counts are drawn from a normal distribution to simulate realistic imbalance, with variance controlling the degree of imbalance.With 10 clients and 11,360 samples, one client may hold 48 samples while another holds 3,855.
  • Results: SplitNN is less sensitive to imbalanced data distribution because it can converge quickly.In one case, SplitNN does not learn during the first 50 rounds but converges quickly after learning begins.
  • Results: FL struggles to reach centralized-model baseline accuracy with many clients, even when using multiple local epochs per round.More local epochs can accelerate convergence and reduce communication overhead, but proportionally increase client-side training time.
  • Results: SplitNN accuracy deteriorates when the number of clients becomes large under the imbalanced distribution.The experiments simulate up to 100 clients.
  • Results: SplitNN converges faster than FL under the imbalanced-data experiments.The paper reports faster SplitNN convergence for RQ2, while FL converges more slowly as client count increases.

IV. IMPLEMENTATION OVERHEAD EVALUATION ON RASPBERRY PI

The Raspberry Pi evaluation measures implementation overhead for FL and SplitNN in a representative IoT gateway setting. It uses ECG data and evaluates time, power, communication, memory, and temperature with fixed experimental settings.

  • Evaluation scope: The benchmark evaluates training time, power consumption, communication overhead, memory usage, and temperature on Raspberry Pi devices.The IoT scenario models a gateway aggregating sensor data and interacting with a server for distributed learning.
  • Experimental settings: The experiments compare FL and SplitNN across two to five clients with shared and varying model architectures.They also test concurrent ensemble learning across two clients and train M1 and M2 individually via SplitNN.
  • Measurement methods: Training time includes communication time and is measured as Tend − Tstart.Tstart is recorded when training begins and Tend when training finishes.
  • Experimental settings: The implementation comparison fixes FL at one local epoch per round and both methods at 100 rounds.The learning rate is fixed at 0.001.
  • Measurement methods: Memory usage records used memory on Raspberry Pi, whose total memory is 926 MB.The Linux free -h command reports total, used, free, cached, and available memory.
  • Measurement methods: Communication overhead is measured in transmitted data size in both directions between each client and the server.The measurements use Python pickle and a wireless router connecting Raspberry Pi devices to the server.

C. Implementation Considerations

Resource constraints make full end-to-end evaluation of larger CNNs difficult on Raspberry Pi devices. The paper therefore uses a simpler 1D CNN for the main IoT tests after benchmarking MobileNet and SplitNN or FL timing.

  • Performance constraints: FL requires 8 hours 41 minutes per round for MobileNetV1 on Raspberry Pi, whereas SplitNN requires about 2.5 hours per epoch across five devices.The MobileNetV1 model has 20 convolutional layers and 3,228,170 parameters.
  • Performance constraints: SplitNN takes about 1 hour per round across five Raspberry Pis, compared with 37 minutes for FL using one local epoch.For SplitNN, only the first two layers run on the Pi devices in this comparison.
  • Performance constraints: Running ResNet20 or MobileNet V1 on Raspberry Pi devices is challenging because their several-million-parameter models are computationally heavy there.The paper distinguishes this training limitation from inference after a model has been trained and optimized.
  • Design choice: The full end-to-end tests use a relatively simple 1D CNN on sequential time-series data instead of the larger 2D CNN models.The choice reflects the prevalence of sequential sensor data in IoT settings.

2) Install Pytorch on Raspberry Pi:

The paper documents Raspberry Pi deployment constraints and evaluates SplitNN ensemble learning as a way to obtain multiple models concurrently. Ensemble learning reduces time, memory, and power overhead in the reported experiment but not communication overhead.

  • Installation: The authors provide a unified manual for installing PyTorch v1.0.0 on Raspberry Pi and resolving installation errors.They state that the guide addresses errors for which online solutions were unavailable.
  • Thermal considerations: Raspberry Pi training can exceed 80°C, while a cooling fan reduces temperature from 83°C to 54°C.The paper reports this issue especially for FL, which runs the entire model on the device.
  • Ensemble learning: SplitNN ensemble learning obtains two models in 11,704 seconds versus 15,175 seconds when trained individually, reducing time overhead by 22.87%.Individual training takes 8,267 seconds for M1 and 6,908 seconds for M2.
  • Ensemble learning: Ensemble training uses 222 MB, compared with 185 MB and 186 MB for individual M1 and M2 training.The memory comparison includes 119 MB used by the operating system by default.
  • Ensemble learning: Sequential M1 and M2 training consumes 4 Wh and 3 Wh, while ensemble training consumes 6 Wh.The paper concludes that ensemble learning can reduce power consumption in this comparison.
  • Ensemble learning: Ensemble learning does not reduce communication overhead but can reduce training time, used memory, and power consumption.The reported benefits arise from obtaining multiple models through concurrent server computation.

E. Effects of Number of Clients

Varying client counts exposes a trade-off: FL generally reduces training time and keeps communication overhead stable, while SplitNN reduces communication with more clients but usually takes longer. SplitNN uses less memory and peak power, whereas FL consumes less total energy for the same rounds.

  • Training time: FL training time decreases as devices increase, while SplitNN training time slightly increases because each device trains sequentially.Overall, SplitNN usually takes several times longer than FL for the same number of rounds.
  • Communication overhead: FL communication remains around 28,552,161 bytes, whereas SplitNN communication decreases with more clients because it depends strongly on local data size.The FL communication cost is determined by model parameters rather than local data size.
  • Resource usage: FL uses more memory and reaches higher power peaks because each Raspberry Pi trains the entire model, while SplitNN trains only a few split layers.Without cooling, Raspberry Pi temperature can reach 83◦C during FL learning.
  • Energy: FL consumes less total energy than SplitNN for the same number of rounds despite its higher power peak.Parallel local training makes FL’s accumulated computation and communication time shorter than SplitNN’s.
  • Communication overhead: SplitNN communication overhead is orders of magnitude higher than FL’s, with megabytes versus gigabytes reported as an example.This comparison explains why FL is favored when communication traffic is the main concern.
  • Split-layer effects: With more client-side split layers, SplitNN communication remains unchanged, while time and energy overhead increase.Memory usage increases only slightly, and the authors prefer running few layers on the client for overhead reduction.

G. Effects of Different Models

As model complexity increases, FL device overhead rises with the number of layers, whereas SplitNN overhead remains approximately constant when client-side split layers are fixed. This makes SplitNN more advantageous for complicated models in the reported setting.

  • Model complexity: For FL, time, communication, memory, and energy overhead increase linearly with model complexity, defined by the number of model layers.The experiments vary convolutional layers from four to eight.
  • Model complexity: For SplitNN, overhead remains more or less constant as model complexity increases because the number of layers running on each client is fixed.The split-layer count remains fixed regardless of the full model’s layer count.
  • Implication: SplitNN becomes more advantageous when the model is complicated because client-side overhead does not scale with the entire model depth.This conclusion concerns the reported time, communication, memory, and energy measures.

A. Summary of the Evaluation Results

The evaluation finds that SplitNN’s learning performance depends strongly on data distribution, while IoT deployment introduces distinct overhead and applicability constraints. FL is generally more practical for communication-sensitive IoT settings with the tested simple 1D CNN models.

  • Learning performance: SplitNN converges much faster than FL under imbalanced data, but can fail to learn under extreme non-IID cases.Examples include one client holding only two or three classes.
  • IoT deployment: When communication cost is critical, FL is preferred because it has relatively lower communication overhead, which also benefits training time and energy consumption.The paper recommends SplitNN when communication is not a significant concern, such as when Ethernet or 5G is available.
  • Limitations: The study follows a typical, non-optimized FL and SplitNN setting, and training 2D CNNs such as ResNet and MobileNet on Raspberry Pis was computationally infeasible.The authors identify optimization and lighter computation as future directions.
  • Model applicability: The experiments use 1D CNNs because they can be vertically split, while LSTM and RNN models were difficult to apply to SplitNN.Applicability of SplitNN to sequential models remains future work.
  • Overall conclusion: Across the evaluation, SplitNN performs better than FL with imbalanced data but worse under extreme non-IID data, while FL requires less overall communication, time, and power for simple 1D CNNs.The paper presents this as its overall IoT recommendation while noting that both methods are influenced by data distribution.
Loading 2003.13376v2…