Source-linked AI summary
FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
Fan Lai, Yinwei Dai, Sanjay S. Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, Mosharaf Chowdhury
TL;DR
Existing FL benchmarks provide limited evidence for realistic, heterogeneous, large-scale deployments. FedScale combines 20 realistic federated datasets with a scalable, extensible runtime, and its experiments show that benchmark scale can materially change evaluation outcomes. The paper concludes that realistic benchmarking exposes opportunities for practical FL optimization.
Problem
Existing benchmarks often omit realistic data, system behavior, connectivity, availability, and deployment scale needed to evaluate FL comprehensively.
Method
FedScale combines realistic datasets based on client behavior with an extensible runtime that simulates and evaluates FL at practical scale.
Results
For σ=0.01, final model accuracy drops 12.8% with N=30 participants but 4.6% with N=100, showing that supported participant scale changes privacy evaluation.
Takeaways & Limitations
FedScale enables standardized, reproducible evaluation across diverse tasks and realistic client statistical and system conditions.
Takeaways & Limitations
The paper focuses on simulation mode because benchmarking is its primary focus.
Abstract
from arXiv · showhide
We present FedScale, a federated learning (FL) benchmarking suite with realistic datasets and a scalable runtime to enable reproducible FL research. FedScale datasets encompass a wide range of critical FL tasks, ranging from image classification and object detection to language modeling and speech recognition. Each dataset comes with a unified evaluation protocol using real-world data splits and evaluation metrics. To reproduce realistic FL behavior, FedScale contains a scalable and extensible runtime. It provides high-level APIs to implement FL algorithms, deploy them at scale across diverse hardware and software backends, and evaluate them at scale, all with minimal developer efforts. We combine the two to perform systematic benchmarking experiments and highlight potential opportunities for heterogeneity-aware co-optimizations in FL. FedScale is open-source and actively maintained by contributors from different institutions at http://fedscale.ai. We welcome feedback and contributions from the community.
1. Introduction
FedScale addresses the limitations of existing FL benchmarks by combining realistic workloads with scalable, standardized evaluation. It targets data, device, connectivity, availability, scale, and task diversity in practical FL settings.
- Comprehensive FL evaluation must cover data and device heterogeneity, connectivity, availability, multiple scales, and diverse ML tasks.
- Existing FL benchmarks often use synthetic partitions or conventional datasets that do not represent realistic application characteristics.
- FedScale introduces an FL benchmark and accompanying runtime for comprehensive and standardized evaluations.
- Its datasets span 20 realistic workloads, multiple scales, and tasks including image classification, object detection, word prediction, speech recognition, and reinforcement learning.
- The runtime supports mobile and cluster backends, practical runtime metrics, scalable client training, and extensible APIs.
2. Background
Prior FL work addresses heterogeneous data and device resources through adaptive optimization, client selection, clustering, and related strategies. However, existing benchmarks can omit realistic client behavior and large-scale deployment conditions, misleading evaluation.
- Existing FL methods address data heterogeneity with control variates, guided client selection, mixtures of models, and client clustering.
- Existing benchmarks often lack realistic client statistical and system behavior datasets or fail to reproduce large-scale FL deployments.
- These omissions can make benchmarks insufficient for diverse FL optimizations and mislead performance evaluations.
3. FedScale Dataset: Realistic FL Workloads
FedScale curates realistic federated datasets from real client mappings and supports diverse task scales and categories. It also models heterogeneous system speeds and dynamic client availability using real-world traces.
- FedScale currently provides 20 real-world federated datasets across practical scenarios and disciplines.
- Datasets are cleaned, repartitioned into consistent FL formats, categorized by use case, and exposed through standardized Python APIs.
- Real client identifiers determine data partitions, preserving variation in sample quantities and data distributions across clients.
- FedScale covers small, medium, and large scales across tasks ranging from hundreds to millions of clients.
- System traces capture heterogeneous training, inference, and network speeds across mobile devices.
- A 136k-user behavior dataset with 180 million trace items models dynamic client availability over a week.
4. FedScale Runtime: Evaluation Platform
FedScale Runtime is an automated, extensible evaluation platform with mobile and cluster backends for realistic FL benchmarking. It supports practical metrics, large-scale simulation, and low-effort deployment of new plugins.
- Platform overview: FedScale Runtime provides mobile and cluster backends for evaluating FL under realistic deployment settings.The platform is designed to simplify and standardize evaluation while supporting large-scale simulations.
- Mobile backend: The mobile backend enables on-device evaluation using unmodified Python and PyTorch code on Android devices.It is built atop Termux, preserving the full operator set and reducing deployment effort.
- Mobile backend: FedScale benchmarks mobile training time, power, and energy across models, datasets, and smartphone devices.The reported setup trains ResNet34 and ShuffleNet on ImageNet and CIFAR-10 using Xiaomi Mi10 and Samsung S10e devices.
- Cluster backend: The cluster backend supports deployment and simulation, using realistic traces to emulate client computation, communication, latency, and wall-clock behavior.The same code can run in both modes with little modification, reducing migration overhead.
- Extensibility: Flexible APIs let developers add new FL plugins by inheriting base modules and accommodate multiple execution backends.The runtime includes example APIs and is intended to minimize developer effort when benchmarking new designs.
- Cluster backend: FedScale scales to thousands of clients per round through GPU sharing, resource management, adaptive dispatch, and virtual-clock orchestration.Its runtime reduces fleet-training overhead and supports large-scale evaluations more efficiently than platforms with limited distributed-evaluation support.
5. Experiments
FedScale evaluates FL algorithms and system choices under realistic statistical, device, connectivity, and availability conditions. Experiments show that practical scale and heterogeneity can materially change measured performance and expose opportunities for joint optimizations.
- Statistical efficiency: Non-IID data produces worse round-to-accuracy performance and final model accuracy than IID data, while algorithm rankings vary across datasets.FedYoGi performs best on OpenImage but is inferior to FedAvg on Google Speech.
- Benchmarking FL performance: 100 participants per round is the default practical scale, but FedML supports only 30 participants and can under-report attainable FL performance.FedScale and FedML use the same underlying training and FL protocols, while limited scalability changes the evaluated setting.
- System efficiency: FedScale fast-forwards practical runtime evaluation and reports communication cost and client system duration to expose accuracy-cost trade-offs.These measurements allow developers to evaluate local-step choices and other large-scale system optimizations efficiently.
- Privacy and security: For σ=0.01, final accuracy drops 12.8% with N=30 but 4.6% with N=100 in differential-privacy benchmarking.Participant scale therefore changes the measured privacy–utility outcome, while FedScale also reports runtime and communication costs.
- Privacy and security: Security optimizations that appear effective on synthesized datasets can still face large accuracy drops under more practical OpenImage backdoor settings.The experiment compares training without enhancement against clipped model updates.
- Future optimizations: Stragglers and resource-specific bottlenecks can make isolated communication or computation optimizations yield marginal round-duration improvements.The authors identify heterogeneity-aware co-optimization of communication and computation as an opportunity.
- Future optimizations: Long intensive on-device computation can drain batteries or overheat devices, motivating power- and temperature-aware FL training.The proposed direction includes varying local steps across clients or using device-aware neural architecture search.
6. Conclusion
FedScale combines realistic federated datasets with a scalable, extensible evaluation platform to support reproducible FL research and broader applications.
- FedScale introduces diverse realistic FL datasets spanning multiple scales, task categories, and client system behaviors.
- FedScale Runtime provides a scalable and extensible platform for evaluating practical FL runtime metrics.
- The platform offers ready-to-use datasets and flexible APIs for applications including NAS, model inference, and federated computation.
A. Experiment Setup
The evaluations use mobile-relevant applications and models with standardized training settings, including fixed minibatches, local steps, and learning-rate schedules.
- The scalability evaluation compares FedScale Runtime, FedML, and Flower on 10 GPU nodes using ShuffleNet-V2 and OpenImage.
- Applications and models used in the evaluations are widely used on mobile devices.
- Each participant uses minibatch size 32 and 20 local steps in the evaluation setup.
- The initial learning rate is 0.04 for CV tasks and 5e-5 for NLP tasks, decaying by 0.98 every 10 training rounds.
B. Introduction of FedScale Datasets
FedScale provides 20 realistic federated datasets covering diverse applications, modalities, and task categories, including speech, vision, language, streaming, advertising, and motion data.
- FedScale currently contains 20 realistic federated datasets across wide scales and task categories.
- Google Speech Commands contains over ten thousand one-second clips covering 35 common words spoken by thousands of people.
- OpenImage contains 16M bounding boxes spanning 600 object classes and is cleaned using client indices.
- Reddit and StackOverflow provide user-partitioned comments for language modeling, restricted to the 30k most frequent words.
- VLOG contains more than 10k lifestyle videos from more than 4k actors for scene classification, hand-state prediction, and hand detection.
- LibriTTS provides 585 hours of read English speech from 2456 speakers at a 24kHz sampling rate.
- Taobao contains 1,140,000 users’ advertising display and click logs collected over 8 days.
- Waymo Motion contains 103,354 segments of object tracks and map data, subdivided into 9-second client scenarios.
C. Comparison with Existing FL Benchmarks
FedScale addresses limitations in existing FL benchmarks by combining realistic data and client behavior traces with scalable, modular runtime support for large-scale evaluation.
- Data Heterogeneity: Existing benchmarks often rely on synthetic partitions, limited task coverage, and simulated environments rather than realistic FL data.
- System Heterogeneity: Existing benchmarks largely overlook heterogeneous client system behaviors, which can produce misleading evaluations.
- Data Heterogeneity: FedScale provides 20 realistic client datasets partitioned using real client-data mappings.
- System Heterogeneity: FedScale incorporates real-world mobile traces, client system speeds, and availability information into its evaluations.
- Scalability: Its resource scheduler monitors utilization, queues overcommitted requests, dispatches simulations across machines, and balances load.
- Scalability: Overlapping communication and computation improves FedScale scalability across machines and on single machines.
- Scalability: FedScale Runtime has been run on a 20-GPU setup and by at least one group with more than 60 GPUs.
- Modularity: FedScale Runtime supports modular deployments through APIs, a data loader, client simulator, and resource manager.
D. Examples of New Plugins
FedScale demonstrates that its flexible runtime APIs can integrate and evaluate diverse federated learning efforts with limited code changes. Examples cover client selection, gradient compression, and comparisons with Flower.
- FedScale provides flexible APIs across runtime modules, allowing developers to access and customize methods for diverse federated learning efforts.
- A random client-selection implementation samples client_ids and returns them through a configuration object.
- New plugins are automatically integrated into evaluations, which then produce practical federated learning metrics.
- Oort client selection can be evaluated by modifying a few lines in the clientManager module.
- Comparison with other work: Figure 20 compares gradient compression in Flower with FedScale’s evaluation and indicates substantially greater implementation effort in Flower.Flower requires implementation of gray components, with each function potentially spanning tens of lines of code.
- Comparison with other work: Flower lacks APIs for passing metadata such as client IDs between clients and servers, leaving servers and workers client-agnostic.