PaperlayerView exact revision on arXiv

Source-linked AI summary

FedScale: Benchmarking Model and System Performance of Federated Learning at Scale

Fan Lai, Yinwei Dai, Sanjay S. Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, Mosharaf Chowdhury

arXiv:2105.11367v5cs.LGcs.AIcs.DCcs.PF

TL;DR

Existing FL benchmarks do not adequately cover practical conditions such as task diversity, heterogeneity, availability, connectivity, and scale. FedScale combines realistic datasets with a scalable evaluation runtime, and its comparisons show task-dependent algorithm rankings and performance differences between limited and practical participant scales.

  • Problem

    Existing FL benchmarks provide insufficient coverage of practical evaluation conditions spanning heterogeneity, connectivity, availability, task breadth, and scale.

    Source: §1 · p. 1, §5.1 · p. 7

  • Method

    FedScale combines realistic client-partitioned datasets across practical tasks with a scalable, extensible runtime and APIs for implementing and evaluating FL algorithms.

    Source: §3.1 · p. 3, §6 · p. 9

  • Results

    Across tasks, algorithm rankings vary; with σ=0.01, final accuracy drops 12.8% at N=30 versus 4.6% at N=100, and 30-client limits can under-report practical-scale performance.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    The benchmark enables studies of how optimization choices perform across tasks and at practical participant scales.

    Source: §5.1 · p. 8

    The paper focuses on simulation mode because benchmarking is its primary focus.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

We present FedScale, a federated learning (FL) benchmarking suite with realistic datasets and a scalable runtime to enable reproducible FL research. FedScale datasets encompass a wide range of critical FL tasks, ranging from image classification and object detection to language modeling and speech recognition. Each dataset comes with a unified evaluation protocol using real-world data splits and evaluation metrics. To reproduce realistic FL behavior, FedScale contains a scalable and extensible runtime. It provides high-level APIs to implement FL algorithms, deploy them at scale across diverse hardware and software backends, and evaluate them at scale, all with minimal developer efforts. We combine the two to perform systematic benchmarking experiments and highlight potential opportunities for heterogeneity-aware co-optimizations in FL. FedScale is open-source and actively maintained by contributors from different institutions at http://fedscale.ai. We welcome feedback and contributions from the community.

1. Introduction

FedScale aims to make federated-learning evaluation comprehensive and standardized through realistic datasets and a scalable runtime. Its introduction argues that credible benchmarks must account for heterogeneous data, devices, connectivity, availability, scale, and task variety.

Introduction

  • Federated learning coordinates distributed clients to train or evaluate models on end-user data without gathering their raw data, while prior efforts target system efficiency, statistical efficiency, and privacy and security.FedScale’s systematic experiments highlight co-optimizing system and statistical efficiency, including stragglers, accuracy bias, and device-energy trade-offs.Source: §1 · pp. 1–2
  • A comprehensive FL benchmark must cover data and device heterogeneity, connectivity, client availability, multiple scales, and a broad variety of ML tasks.The introduction warns that overlooking any of these conditions can mislead evaluation and impair convergence.Source: §1 · p. 1
  • Existing benchmarks often use unrealistic data partitions, omit client speed, connectivity, or availability, support only tens of participants per round, and lack user-friendly integration APIs.These limitations can discourage system-efficiency work and produce overly optimistic statistical performance.Source: §1 · pp. 1–2

Contributions

  • FedScale provides 20 realistic datasets spanning small, medium, and large scales and tasks including image classification, object detection, word prediction, speech recognition, and reinforcement learning.It associates clients with measured computation and communication speeds and availability over time.Source: §1 · p. 2
  • FedScale Runtime offers mobile and cluster backends, flexible APIs, and support for training thousands of clients per round on a handful of GPUs.The cluster backend benchmarks practical metrics such as real client round duration using real FL statistical and system datasets.Source: §1 · p. 2

2. Background

Prior FL work addresses heterogeneous data, limited device resources, and privacy, but these interventions navigate accuracy-computation-privacy trade-offs. Existing benchmarks can mislead evaluations because they often omit realistic client behavior or cannot support practical participant scales.

Existing efforts toward practical FL

Existing FL benchmarks can be misleading

  • Realistic client failures and availability dynamics worsen statistical performance, while training with hundreds of participants per round performs better than training with tens.Benchmarks without system traces can therefore yield overly optimistic statistical results, and platforms unable to support large participant counts can under-report FL optimizations.Source: §2 · p. 3

3. FedScale Dataset: Realistic FL Workloads

FedScale combines real-world client partitions and task-specific datasets with client system traces to represent varied statistical and runtime conditions in federated learning.

3.1. Client Statistical Dataset

  • FedScale curates 20 realistic datasets across practical FL scenarios, cleaning and standardizing raw data and providing Python APIs for reuse.Datasets come from diverse sources and formats; the APIs support using different distributions or adding datasets in other frameworks.Source: §3.1 · p. 3
  • Real-world partitions across video, audio, image, and text tasks show wide client-to-client variation in sample counts and data distributions.The examples span hundreds to millions of clients and millions of data points; their distinct Non-IID patterns make artificial partitions impractical substitutes.Source: §3.1 · p. 3, Figure 3 · p. 3
  • FedScale offers small-, medium-, and large-scale datasets across tasks, from hundreds to millions of clients, with APIs for task derivation and customization.For example, OpenImage supports object detection and can be transformed into an image-classification dataset; developers can enforce distributions or select client subsets.Source: §3.1 · p. 4

3.2. Client System Behavior Dataset

  • Mobile clients’ compute and network capacities can differ by orders of magnitude, requiring FL orchestration to account for system heterogeneity.The system traces combine model training and inference speeds with cloud-to-edge throughput measurements, focusing on devices with more than 2GB RAM connected by WiFi.Source: §3.2 · p. 4, Figure 4 · p. 4
  • Client availability varies diurnally, and available slots can be as short as an FL training round.FedScale uses a week of device-behavior traces from 136k users containing 180 million trace items; devices charging are treated as available, highlighting the need to handle clients going offline during training.Source: §3.2 · p. 4, Figure 5 · p. 4

4. FedScale Runtime: Evaluation Platform

FedScale Runtime is an automated, extensible platform for evaluating federated learning under realistic settings, with mobile and cluster backends. It supports real-device deployments and scalable simulations while reducing developer effort.

4.1. FedScale Runtime: Mobile Backend

  • On Android phones, the mobile backend uses Termux to run unmodified Python scripts with full operator support, letting server-prototyped models and algorithms deploy on devices.This design aims to minimize developers’ engineering effort; gRPC for distributed mobile devices interfacing with the cloud server was still being implemented.Source: §4.1 · pp. 4–5
  • Mobile benchmarks show ResNet34 uses more instantaneous power but less total energy than ShuffleNet, while ImageNet takes longer per mini-batch than CIFAR-10.The tests used batch size 32 on Xiaomi Mi10 and Samsung S10e devices; the Mi10 outperformed the S10e due to its more capable processor.Source: §4.1 · p. 5, Figure 7 · p. 5

4.2. FedScale Runtime: Cluster Backend

  • The cluster backend supports real-device deployment and cost-efficient simulation, using practical FL metrics and allowing the same code to run in both modes with little change.Simulation emulates computation and communication costs, latency, and wall-clock time; the paper focuses on simulation for benchmarking.Source: §4.2 · p. 5
  • The simulation coordinates an aggregator, client simulators, and a resource manager to select available clients, run real training/testing, track communication and computation costs, and queue overcommitted tasks.A global virtual clock preserves simulated runtime and event order despite resource queuing.Source: §4.2 · pp. 5–6
  • Thousands of clients per round on 10 GPUs: FedScale Runtime also runs faster than FedML and Flower while supporting large-scale evaluations.GPU sharing and a resource manager that queues and distributes client tasks enable simulations across standalone and distributed settings.Source: Figure 10 · p. 6, §4.2 · pp. 6–7
  • Flexible APIs let developers add plugins by extending the base client module and import realistic client-behavior traces such as availability profiles.The paper illustrates adding a local-training design with a few lines of code.Source: §4.2 · p. 6, Figure 9 · p. 6

5. Experiments

FedScale uses realistic FL settings to benchmark statistical, system, privacy, and security performance, and its experiments expose opportunities for joint optimization across heterogeneous clients. Evaluations use representative datasets and aggregate updates from the first 100 completed participants among 130 per round by default.

5.1. How Does FedScale Help FL Benchmarking?

  • Non-IID data yields worse round-to-accuracy performance and final accuracy than IID data, while optimization preferences vary by task: FedYoGi leads on OpenImage but trails FedAvg on Google Speech.FedScale supports comparisons across FedAvg, FedProx, FedYoGi, and IID settings using realistic client datasets.Source: §5.1 · pp. 7–8
  • With 100 participants per round, FedScale reports higher practical FL performance than FedML can show when its scalability limits it to 30 participants.The comparison attempts to reproduce practical FL scale in both frameworks; FedML's participant limit under-reports the algorithm's achievable performance.Source: §5.1 · p. 8
  • FedScale's runtime evaluates how local-step count affects practical FL runtime in hours and uses realistic system traces to measure communication cost and individual-client duration.The experiments use 10 NVIDIA Tesla P100 GPUs; the runtime collects updates from the first 100 completed participants among 130 each round by default.Source: §5 · p. 7, §5.1 · p. 8, Figure 12 · p. 8
  • For σ=0.01, final accuracy drops 12.8% at N=30 versus 4.6% at N=100, showing that small-scale benchmarks can mislead privacy evaluations.FedScale can also measure privacy-optimization wall-clock time, communication cost, and rounds until client privacy leakage under realistic client data and Non-IID distributions.Source: §5.1 · p. 8, Figure 13 · p. 9
  • In OpenImage backdoor-attack tests, clipping model updates can mitigate attacks on synthesized datasets, yet practical FL settings can suffer large accuracy drops.The comparison uses poisoned clients that flip ground-truth labels, with and without update clipping.Source: §5.1 · p. 8, Figure 14 · p. 9

5.2. Opportunities for Future FL Optimizations

  • Stragglers can greatly slow round aggregation, and improving communication alone may not shorten rounds when another participant is computation-bound.Communication optimization greatly benefits Client 4 but only marginally improves round duration when Client 5 is computation-bound, motivating heterogeneity-aware communication-computation co-optimization.Source: §5.2 · p. 9, Figure 15 · p. 9
  • Joint statistical-system optimization could use predictable availability to prioritize offline clients before data-distribution drift, while faster workers trade latency for greater statistical benefit.The paper contrasts this with one-size-fits-all local-step or gradient-compression strategies.Source: §5.2 · p. 9, Figure 16 · p. 9
  • Intensive on-device computation can drain batteries or overheat devices, motivating power- and temperature-aware training such as client-specific local steps or device-aware NAS.Such device effects can make clients unavailable.Source: §5.2 · p. 9

6. Conclusion

FedScale combines realistic, diverse federated learning datasets with FedScale Runtime, a scalable evaluation platform, to support scalable and reproducible FL research. The runtime evaluates practical FL metrics and offers flexible APIs for broader FL applications.

6. Conclusion

  • FedScale combines realistic datasets spanning scales, task categories, and client system behaviors with a scalable, extensible evaluation platform.The platform is designed to enable scalable and reproducible federated learning research.Source: §6 · p. 9
  • FedScale Runtime fast-forwards evaluation of practical FL runtime metrics needed in current research.Source: §6 · p. 9
  • Ready-to-use datasets and flexible APIs support applications including NAS benchmarking, model inference, and broader federated computation such as multi-party computation.Source: §6 · p. 9
  • FedScale is open-source and invites the community to contribute state-of-the-art FL efforts.Source: §6 · p. 9

A. Experiment Setup

The experiments evaluate mobile-relevant applications and models with fixed training hyperparameters, and benchmark runtime scalability on a 10-GPU-node cluster.

A. Experiment Setup

  • Scalability tests compare FedScale Runtime, FedML, and Flower on 10 GPU nodes using ShuffleNet-V2 on OpenImage, with four processes per node.Each node has a P100 GPU with 12GB memory and 192GB RAM; each training process uses around 2800MB GPU memory.Source: A. Experiment Setup · p. 12
  • Evaluations use mobile-relevant applications and models, minibatches of 32, K=20, learning rates of 0.04 for CV and 5e-5 for NLP, decayed by 0.98 every 10 rounds.The hyperparameters were selected by grid search and described as consistent with the literature.Source: A. Experiment Setup · p. 12

B. Introduction of FedScale Datasets

FedScale currently includes 20 realistic federated datasets spanning a wide range of scales and task categories. The section illustrates this breadth with representative speech, vision, language, advertising, motion, and streaming datasets.

B. Introduction of FedScale Datasets

C. Comparison with Existing FL Benchmarks

FedScale contrasts with existing FL benchmarks through 20 realistic datasets, real-world client system traces, scalable runtime designs, and modular tools for broader deployments and use cases.

Data Heterogeneity

  • FedScale provides 20 realistic client datasets spanning diverse tasks and small, medium, and large scales, unlike benchmarks limited to synthetic data and few FL tasks.The datasets use real-world collection and client-data mappings, and can support data analysis to motivate FL designs.Source: C. Comparison with Existing FL… · p. 12, Table 5 · p. 13

System Heterogeneity

  • Because existing benchmarks often overlook client system behavior, FedScale uses real-world mobile-device traces, per-client speeds and availability, and an evaluation runtime that automates benchmarking.System heterogeneity includes internet bandwidth and client availability, which affect practical FL performance.Source: C. Comparison with Existing FL… · pp. 12–13

Scalability

  • FedScale addresses simulation crashes and underused resources in traditional ML architectures with a cluster scheduler, client virtual-clock orchestration, and overlapped communication and computation.The scheduler monitors fine-grained utilization, queues overcommitted requests, and dispatches client simulations across machines for load balance; overlap improves utilization for FL's smaller batches.Source: C. Comparison with Existing FL… · p. 13
  • Table 6 characterizes FedScale as more scalable and faster; experiments ran a 20-GPU setup, and one group ran FedScale Runtime with more than 60 GPUs (Lai et al., 2021).The reported 20-GPU experiments used different datasets and models.Source: Table 6 · p. 13, C. Comparison with Existing FL… · p. 13

Modularity

  • FedScale Runtime extends modularity beyond APIs with a data loader for realistic data, a client simulator for customizable system traces, and a resource manager that hides large-scale training complexity.Developers can analyze realistic data, import datasets or customize distributions, and evaluate system efficiency with custom traces.Source: C. Comparison with Existing FL… · pp. 13–14

D. Examples of New Plugins

FedScale Runtime’s flexible APIs make it straightforward to integrate and evaluate new federated-learning plugins, while requiring less developer effort than the Flower comparison described here.

D. Examples of New Plugins

  • FedScale Runtime exposes flexible APIs for customizing module methods and automatically integrates new plugins into evaluations to produce practical FL metrics.The section presents this as a way to facilitate diverse federated-learning efforts.Source: D. Examples of New Plugins · p. 14
  • Oort client selection requires only a few clientManager changes, while gradient compression and malicious-attack enhancement extend the basic Client module.The examples correspond to Oort (Lai et al., 2021), gradient compression (Rothchild et al., 2020), and malicious-attack enhancement (Sun et al., 2019).Source: D. Examples of New Plugins · p. 14
  • Flower requires more human effort for the same gradient-compression evaluation because developers implement many components and manage client metadata between server and workers.The comparison notes Flower’s lack of APIs for passing metadata such as client IDs, leading developers to override components and work through source code; Figure 20 marks developer-implemented functions in gray.Source: D. Examples of New Plugins · p. 14, Figure 20 · p. 14

Related papers

Loading 2105.11367v5…