Source-linked AI summary
Flower: A Friendly Federated Learning Research Framework
Daniel J. Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, Nicholas D. Lane
TL;DR
Federated learning needs research frameworks that can study scalable workloads on heterogeneous edge devices rather than relying mainly on limited simulations. Flower provides framework-agnostic abstractions and resource-aware execution for large-scale and heterogeneous FL experiments, reaching up to 15 million clients and supporting migration to real devices.
Problem
Existing FL research frameworks do not adequately support scalable execution of workloads across heterogeneous mobile and edge devices.
Method
Flower combines framework-agnostic FL components with resource-aware virtual clients that support large-scale experiments across varied hardware and device conditions.
Results
Flower supports FL experiments with up to 15 million clients across five machine-learning workloads while quantifying effects of client heterogeneity and fluctuating network speeds.
Takeaways & Limitations
Researchers can transition existing ML workloads from large-scale simulation to heterogeneous edge-device execution for studying scale and system heterogeneity.
Takeaways & Limitations
The secure-aggregation simulations ignore communication latency and model dropouts only after the Share Keys Stage.
Abstract
from arXiv · showhide
Federated Learning (FL) has emerged as a promising technique for edge devices to collaboratively learn a shared prediction model, while keeping their training data on the device, thereby decoupling the ability to do machine learning from the need to store the data in the cloud. However, FL is difficult to implement realistically, both in terms of scale and systems heterogeneity. Although there are a number of research frameworks available to simulate FL algorithms, they do not support the study of scalable FL workloads on heterogeneous edge devices. In this paper, we present Flower -- a comprehensive FL framework that distinguishes itself from existing platforms by offering new facilities to execute large-scale FL experiments and consider richly heterogeneous FL device scenarios. Our experiments show Flower can perform FL experiments up to 15M in client size using only a pair of high-end GPUs. Researchers can then seamlessly migrate experiments to real devices to examine other parts of the design space. We believe Flower provides the community with a critical new tool for FL study and development.
1 INTRODUCTION
Federated learning keeps user data on edge devices while enabling collaborative model training, but existing research frameworks struggle with scalable, heterogeneous execution. Flower addresses this gap with framework-agnostic abstractions, large-cohort support, and experiments spanning up to 15 million clients.
- Federated learning enables edge devices to collaboratively train a shared prediction model while keeping personal data private.
- Existing frameworks support FL algorithm experimentation but lack scalable execution on heterogeneous mobile and edge devices.
- Flower provides a language- and machine-learning-framework-agnostic implementation supporting algorithmic and systems-related FL experimentation.
- Flower supports large-cohort research with heterogeneous compute, memory, and network resources on simulated or real edge devices.
- Up to 15 million clients were used across five machine-learning workloads to quantify bottlenecks from client heterogeneity and fluctuating network speeds.
- Flower is open-sourced under the Apache 2.0 License and has been adopted by major academic and industry research organizations.
2 BACKGROUND AND RELATED WORK
Federated learning research spans optimization, privacy, robustness, secure aggregation, and heterogeneous-system performance. Existing open-source frameworks primarily support small simulations with homogeneous clients, while Flower unifies exploratory research with broader system support.
- Prior federated-learning work covers optimization, privacy, model-poisoning mitigation, differential privacy, secure aggregation, robustness, and heterogeneous settings.
- TFF, PySyft, and LEAF are open-source frameworks primarily used for simulations with small numbers of homogeneous clients.
- Flower unifies open-source exploratory research with support for broader federated-learning system scenarios.
3 FLOWER OVERVIEW
Flower is an end-to-end federated learning framework designed to support scalable, heterogeneous experiments and transitions between simulation and real edge devices. Its architecture combines flexible abstractions with resource-aware client execution to address gaps in existing FL research workflows.
- Flower supports experiments spanning simulation and large cohorts of real edge devices, allowing implementations to migrate between these settings during development.
- Use cases: Flower targets large client pools and concurrent training, heterogeneous devices, simulation-to-device transitions, and workloads combining different machine-learning frameworks.
- Design goals: Flower’s design goals are scalability, client and communication agnosticism, privacy agnosticism, and flexibility for experimental research and new approaches.
- Core architecture: The framework architecture separates global orchestration from local computation through Strategy abstractions and supports reference algorithms such as FedAvg and FedYogi.
- Core architecture: The Virtual Client Engine schedules and instantiates clients according to compute and memory budgets, enabling large-scale workloads across desktops, GPU racks, and multi-node clusters.
- Framework comparison: Flower supports heterogeneous client pools through language- and communication-agnostic interfaces, while its ML framework-agnostic design accommodates varied local training pipelines.
4 IMPLEMENTATION
Flower’s implementation uses modular communication and client abstractions to support heterogeneous deployments. Its Virtual Client Engine manages resource-constrained execution by delaying client instantiation until resources are available.
- Flower provides FL averaging algorithms, a communication stack, and examples for deploying clients in real and simulated environments.
- Communication stack: The communication protocol uses bidirectional gRPC streams with generated implementations for languages including Python, Java, and C++.
- Serialization: Clients exchange language-independent byte-array messages, deserialize instructions, execute local tasks, and serialize results for return to the server.
- Alternative communication stacks: Modular server abstractions allow user-provided RPC frameworks and workloads combining clients connected through different communication systems.
- ClientProxy: ClientProxy represents available clients to server-side logic, while separate implementations encapsulate communication details such as gRPC.
- Virtual Client Engine: The Virtual Client Engine creates client proxies but delays loading each client’s model and data until resources are available, avoiding simultaneous retention of many models and datasets.
- Virtual Client Engine: Ray schedules client-side computations and can sequence them under resource constraints, enabling larger experiments on common hardware.
5 FRAMEWORK EVALUATION
Flower is evaluated for scalability, heterogeneity, realism, and privacy across large-scale simulations and heterogeneous real devices. The experiments show how client scale, framework design, device capabilities, and network conditions affect federated training.
- Scalability: Flower supports experiments with millions of clients while sampling thousands in each training round.
- Large-scale experiment: 15M users are used to fine-tune DistilBERT with 10–1000 clients sampled per round, revealing faster convergence through 500 clients but slower convergence at 1k.The authors attribute the slowdown at 1k clients to different client data distributions challenging FedAvg.
- Single-machine experiments: Flower’s training time varies by workload and framework, outperforming all compared frameworks at 100 local epochs by finishing in about 80 minutes versus FedJax’s over 173 minutes.For three clients and one local epoch, FedJax finishes first at 05:18 while Flower takes 59:19; with 35 clients, Flower remains among the fastest at 230:18.
- Heterogeneity: Flower clients run on six types of heterogeneous mobile and embedded devices without client-side code modifications.The deployment includes Java-based Android smartphones, Nvidia Jetson devices, and Raspberry Pi systems.
- Real-device profiling: Flower’s framework overhead is less than 100ms per round, while local training remains the most time-consuming operation on the tested Android devices.
- Computational heterogeneity: CPU clients with E=10 take 1.27× more time to converge than Jetson TX2 GPU clients.
- Network heterogeneity: Network heterogeneity increases training time from 8.9 minutes for the fastest-country profile to 108 minutes for the slowest-country profile.The fastest profile corresponds to Canada at 40.1Mbps mean download speed, while the slowest corresponds to Iraq at 6.76Mbps.
- Realism: Profiling under network heterogeneity can inform client sampling, and FedFS reduces convergence time by up to 30% over FedAvg random sampling.
6 CONCLUSION
Flower is presented as a framework for heterogeneous federated learning workloads at scale, spanning large simulations and real edge-device execution. The conclusion reports evaluations up to 15M clients, framework comparisons, device measurements, and deployment across Android devices.
- Flower enables heterogeneous federated learning workloads at scale and supports research across simulation and real-world devices.
- Flower supports migrating existing machine-learning workloads from large-scale simulation to heterogeneous edge devices.
- The evaluation scales federated learning to 15M clients and includes framework comparisons, energy measurements, bandwidth-constrained convergence, and Android deployment.
- Flower is open-sourced under the Apache 2.0 License.
A.1 Survey on papers
The survey identifies a gap between federated-learning research practice and scalable execution on mobile and edge devices. Most surveyed research experiments use at most 100 total clients.
- Scalable execution of federated-learning methods on mobile and edge devices remains a major systems bottleneck.
- A vast majority of surveyed papers use up to 100 total clients in their experiments.
- Figure 9 surveys 150 federated-learning papers from the previous two years after excluding unreproducible frameworks.
A.2 FedFS Algorithm
FedFS addresses heterogeneous devices and non-IID data by combining partial work, importance sampling, and dynamic timeouts. It represents incomplete client computation and adjusts selection toward clients with lower prior contributions.
- FedFS combines partial work, importance sampling, and dynamic timeouts to address heterogeneous devices and non-IID data.
- Partial work: FedAvg clients perform E m_k/B local gradient updates before returning model updates to the server.
- Timeout handling: Clients that complete updates before timeout Δ are accepted, while unsuccessful updates are discarded and successful updates are averaged using m_k.
- Partial work: FedFS submits partially completed client updates with c_k examples and averages them by c_k rather than m_k.The actual work c_k varies with device speed, concurrent processes, timeout, and local dataset size.
- Partial work: Partial work lets FedFS progress under aggressive or adversarial timeouts by combining K partial updates instead of discarding all client work.
- Importance sampling: FedFS samples clients with probability proportional to 1−w_k+ε, selecting clients with lower prior contributions more frequently.The minimum probability ε prevents clients that previously completed their maximum work from being excluded from future rounds.
- Importance sampling: When all clients can complete the expected work equally, FedFS reduces to uniform client selection as in synchronous FedAvg.
- Alternating timeout: Alternating fast and slow timeout rounds can prioritize either wall-clock convergence or overall classification accuracy.
A.3 Scaling FedAvg to ImageNet-scale datasets
Flower supports federated training on web-scale workloads such as ImageNet, including a ResNet-18 experiment with 50 clients. The experiment ran for about 15 days and produced lower accuracies than centralized training, while indicating possible gains from longer training.
- Flower supports federated training on ImageNet, which the paper presents as the first-ever attempt of its kind.
- The experiment used 50 clients with four physical CPU cores, partitioning ImageNet’s 1.2M training images into 50 IID partitions.A ResNet-18 model was trained with random horizontal flipping and cropping.
- About 15 days were required to train with 50 clients and 3 local epochs, demonstrating support for long-term realistic experiments.
- FL achieved 59.1% top-1 and 80.4% top-5 accuracy, compared with 63% and 84% for centralized training.The authors note that longer training might have increased FL accuracies and reduced the gap.
- Flower’s server-side model-state persistence supports resuming federated training after client failures.
A.4 Datasets and Network Architectures
The experiments use CIFAR-10, Fashion-MNIST, and ImageNet with network architectures suited to each dataset. The datasets range from small image benchmarks to a 1.2M-image training corpus.
- CIFAR-10 contains 60,000 RGB images across 10 classes, with 50,000 for training and 10,000 for testing.The images are 32 x 32 pixels.
- Fashion-MNIST contains 60,000 training and 10,000 test grayscale images across 10 fashion-item classes.The model is a 2-layer CNN followed by 2 fully-connected layers.
- ImageNet contains 1.2M training images and 50K test images, using ResNet-18 for federated training.