Source-linked AI summary
Scalability and Performance Evaluation of Federated Learning Frameworks: A Comparative Analysis
Bassel Soudan, Sohail Abbas, Ahmed Kubba, Manar Wasif Abu Talib, Qassim Nasir
TL;DR
Existing work lacks a comprehensive comparison of prominent Federated Learning frameworks. This paper systematically examines their scalability and performance across varying client counts, finding OpenFL especially compelling for consistent loss and accuracy.
Problem
Existing work does not present a comprehensive examination of prominent Federated Learning frameworks.
Method
The study conducts a comprehensive, systematic examination of prominent Federated Learning frameworks using scalability and key performance metrics across varying client counts.
Results
OpenFL consistently achieved very good loss and accuracy measures above 96%, while Flower achieved the best average accuracy at 98%.
Takeaways & Limitations
The comparison provides guidance for determining which Federated Learning framework is most appropriate for a given application.
Takeaways & Limitations
Loss seems to increase as client counts increase.
Abstract
from arXiv · showhide
This paper presents a systematic examination and experimental comparison of the prominent Federated Learning (FL) frameworks FedML, Flower, Substra, and OpenFL. The frameworks are evaluated experimentally by implementing Federated Learning over a varying number of clients, emphasizing a thorough analysis of scalability and key performance metrics. The study assesses the impact of increasing client counts on total training time, loss and accuracy values, and CPU and RAM usage. Results indicate distinct performance characteristics among the frameworks, with Flower displaying an unusually high loss, FedML achieving a notably low accuracy range of 66% to 79%, and Substra demonstrating good resource efficiency, albeit with an exponential growth in total training time. Notably, OpenFL emerges as the most scalable platform, demonstrating consistent accuracy, loss, and training time across different client counts. OpenFL's stable CPU and RAM underscore its reliability in real-world scenarios. This comprehensive analysis provides valuable insights into the relative performance of FL frameworks, offering good understanding of their capabilities and providing guidance for their effective deployment across diverse user bases.
3 Introduction
The paper motivates a systematic comparison of prominent FL frameworks because prior work did not comprehensively benchmark scalability and framework expansion. It evaluates FedML, Flower, Substra, and OpenFL to compare capabilities, performance, and resource efficiency.
- Federated Learning keeps training data on clients while sharing model parameters with a server for aggregation and redistribution.Clients train local replicas, exchange parameters rather than raw data, and retain control over private data.
- The comparison is motivated by the need to select frameworks whose capabilities and performance align with application requirements.
- Prior studies did not comprehensively evaluate all selected FL frameworks or benchmark their scalability as client counts expand.
- The study experimentally compares FedML, Flower, Substra, and OpenFL across scalability and key performance metrics.The stated objective includes identifying framework capabilities, strengths, weaknesses, and a benchmark for diverse scenarios.
- The work also assesses computational resource utilization as part of its framework benchmark.
4 Background
Federated Learning enables collaborative model training without sharing sensitive client data, but its deployment must address heterogeneous devices, client participation, data distributions, and security. The background distinguishes FL types and reviews safeguards including secure aggregation, differential privacy, Byzantine fault tolerance, and anomaly detection.
- FL clients train local model replicas, send parameters to a server, and receive aggregated updates without sharing sensitive data.The process repeats until the desired training outcome is reached, while client data remains on its originating device.
- Device heterogeneity challenges fault tolerance and consistent progress, including when clients become temporarily inactive.
- Horizontal, vertical, and federated transfer learning address different relationships between clients, samples, features, and available data.HFL combines similar features across distinct samples, VFL combines complementary features for shared instances, and FTL transfers knowledge across domains with limited related data.
- Cross-device FL involves many resource-limited devices, whereas cross-silo FL involves roughly two to one hundred organizational participants.
- Security mechanisms include secure aggregation, differential privacy, Byzantine fault-tolerant aggregation, and anomaly detection.These approaches protect individual updates, limit inference, mitigate malicious-node influence, or identify abnormal behavior.
5 Methodology
The methodology establishes a benchmark for four prominent open-source FL frameworks using simulated multi-client training on MNIST. It measures model effectiveness, training time, and CPU/RAM utilization while excluding network communication delays.
- 5.1 Selection of FL Frameworks: The evaluation compares Flower, FedML, Substra, and OpenFL because they are frequently cited open-source frameworks with diverse capabilities.Table 1 summarizes their features and capabilities.
- 5.2 Dataset: MNIST provides the experimental benchmark because it is widely recognized, simple, scalable, and resource efficient.The dataset contains 60,000 training images and 10,000 testing images.
- 5.3 Experimental Setup: The experiments simulate FL by splitting MNIST into client-specific sub-datasets and training many clients with an aggregation algorithm.Dataset-splitting effort is excluded from framework-performance measurements.
- 5.3 Experimental Setup: All clients and the aggregation server run in parallel on one Windows 10 workstation using multithreading and built-in timing functions.The workstation has 32 GB RAM, an Nvidia RTX 4000 GPU, and a four-core Intel Xeon W-2102 CPU.
- 5.3 Experimental Setup: Network communication delays are omitted because the experiments execute on the same workstation and aim to measure intrinsic framework capabilities.
- 5.4 Evaluation Metrics: Performance is assessed using loss and accuracy, total training time, and CPU and RAM usage.These metrics capture training effectiveness, responsiveness, and resource utilization under varying resource constraints.
6 Results and Discussion
The evaluation compares FedML, Flower, Substra, and OpenFL across client counts using loss, accuracy, training time, CPU, and RAM metrics. OpenFL is the most scalable overall, while Substra is resource-efficient but scales poorly in training time.
- Loss and Accuracy: Flower achieves the highest average accuracy at 98.75% with 1% variation, while OpenFL averages 96.34%, Substra 93.367%, and FedML ranges from 66% to 79%.OpenFL and Substra vary by less than 2%, whereas FedML shows substantial variability across client counts.
- Total Training Time: Substra’s total training time grows almost exponentially with client count, whereas FedML and OpenFL grow linearly and Flower remains consistently high.All frameworks require more training time as the number of clients increases, but Flower’s high baseline overhead offsets much of the client-count effect.
- Resource Usage Efficiency: Memory usage is generally reasonable across frameworks, but OpenFL’s memory occupancy increases significantly as client count grows.Substra is followed by FedML and Flower in memory efficiency.
- Holistic comparison: OpenFL appears most scalable, maintaining consistent accuracy, loss, and training time across client counts with stable CPU and RAM usage.Its resource metrics show only minor variations as the number of clients changes.
7 Conclusion
The study compares Flower, FedML, Substra, and OpenFL as client counts increase, measuring training time, loss, accuracy, CPU, and RAM. OpenFL offers the most stable overall scalability, while the other frameworks expose distinct accuracy, efficiency, or training-time trade-offs.
- Evaluation design: The evaluation implemented the four frameworks while varying client counts and measured total training time, loss, accuracy, CPU usage, and RAM usage.These metrics were used to assess scalability, training effectiveness, and resource utilization under changing client counts.
- Flower: Flower achieved the best average accuracy at 98% but had an unusually high loss value regardless of client count.Its CPU and RAM usage remained relatively stable, with only a marginal CPU increase as the number of clients grew.
- FedML: FedML averaged 68% accuracy, with slightly better performance as client counts increased and total training time growing linearly.FedML nevertheless consumed more than 95% of CPU cycles consistently, limiting its suitability for resource-constrained applications.
- Substra: Substra provided the best resource efficiency, maintaining low CPU and RAM usage alongside consistently good loss and accuracy measures.Its main scalability weakness was exponentially increasing total training time as the number of clients increased.
- OpenFL: OpenFL maintained loss and accuracy above 96% on average, with reasonably stable resource use and training time as client counts increased.Its training time was somewhat elevated for low client counts but remained substantially stable as the number of clients increased.