Source-linked AI summary
Towards Federated Learning at Scale: System Design
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, Jason Roselander
TL;DR
Training machine-learning models on decentralized mobile-device data raises privacy, scale, reliability, and deployment challenges. This paper presents a production federated-learning system addressing these challenges, which supports applications across approximately 10M daily active devices and improves Gboard next-word prediction recall from 13.0% to 16.4%.
Problem
Federated learning requires production infrastructure for training on decentralized device data while addressing privacy, locality, availability, connectivity, resource, and deployment challenges.
Method
The system trains TensorFlow models on devices, combines updates with Federated Averaging and Secure Aggregation, and coordinates device participation in server-managed rounds.
Results
Approximately 10M daily active devices use the system, while Gboard next-word prediction improves top-1 recall from 13.0% to 16.4% over an n-gram baseline.
Takeaways & Limitations
The system demonstrates that federated learning can support production applications using data that remains on mobile devices.
Takeaways & Limitations
Training converges roughly 7× slower than comparable data-center training, although the comparison uses different proxy data and is not the primary goal.
Abstract
from arXiv · showhide
Federated Learning is a distributed machine learning approach which enables model training on a large corpus of decentralized data. We have built a scalable production system for Federated Learning in the domain of mobile devices, based on TensorFlow. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, and touch upon the open problems and future directions.
1 INTRODUCTION
The paper presents a scalable Android Federated Learning system that trains TensorFlow models on-device while combining updates in the cloud without exposing individual updates. It outlines the system’s algorithmic design, practical deployment challenges, and remaining open problems.
- Federated Learning trains models on decentralized data residing on devices, addressing data privacy, ownership, and locality concerns.
- The system supports large-batch SGD-style algorithms and runs Federated Averaging in production.
- The paper sketches major components and challenges but remains an early-stage account that does not solve all problems or comprehensively cover every required component.
- TensorFlow models train on phone data that never leaves the device, while cloud aggregation constructs a global model for phone inference.
- Secure Aggregation makes individual phone updates uninspectable at the global level.
- The system addresses correlated device availability, unreliable connectivity, interrupted execution, lock-step orchestration, and constrained storage and compute resources.These issues are handled at the communication protocol, device, and server levels.
2 PROTOCOL
The protocol coordinates Android devices and a cloud-based FL server through rounds that select participants, distribute computation and global state, and aggregate local updates. Pace steering regulates reconnections to support small and large populations while handling device drop-outs and time-varying availability.
- Protocol participants and rounds: Devices announce availability for an FL population, and the server selects typically a few hundred from potentially tens of thousands for each round.A round is the rendezvous during which selected devices remain connected to the server.
- Round phases: Each round has selection, configuration, and reporting phases that advance the population’s singleton global model.The server sends an FL plan and checkpoint, participants compute locally and return updates, and the server aggregates them using Federated Averaging.
- Round phases: A round succeeds only if enough devices report in time; otherwise it is abandoned, while configurable tolerance allows late or nonresponsive devices to be ignored.Selection uses a participant goal count, timeout, and minimal required percentage to determine whether the round starts or is abandoned.
- Pace steering: Pace steering suggests reconnection windows to regulate device connections, scaling the protocol down for small populations and up for very large ones.For small populations it promotes contemporaneous check-ins; for large populations it randomizes check-ins to avoid the “thundering herd” problem.
- Pace steering: Pace steering also adjusts reconnection windows for diurnal changes in active devices, avoiding excessive peak-hour activity without hurting FL performance at other times.The protocol’s network traffic is encrypted on the wire even without Secure Aggregation.
3 DEVICE
The device architecture separates application-managed example stores from the FL runtime, which executes server-provided training or evaluation plans and reports updates and metrics. It schedules work under device-friendly conditions and supports multi-tenancy, anonymous participation protected by remote attestation.
- Data Management: Applications expose locally collected training and evaluation data through example stores that the FL runtime can access.An example store may be an SQLite database recording suggestions shown and whether users accepted them.
- Task Lifecycle: The FL runtime receives server-provided plans, queries the appropriate example store, computes model updates or held-out-data metrics, reports them, and cleans up temporary resources.Plans support both training and evaluation analogous to validation in data-center training.
- Resource Constraints: The runtime schedules periodic jobs through Android’s JobScheduler and requests execution only when the phone is idle, charging, and connected to an unmetered network.These conditions are intended to avoid negative effects on user experience, data usage, and battery life.
- Multi-Tenancy: The implementation supports multiple FL populations in one app or service, coordinating training activities to avoid simultaneous-session overload.The runtime can run inside the configuring application or in a centralized service, with communication implemented through Android’s AIDL IPC mechanism.
- Attestation: Devices participate anonymously, while Android remote attestation helps verify genuine devices and applications and provides some protection against poisoning from compromised devices.The design does not address manipulation by content farms using uncompromised phones.
4 SERVER
The FL server is designed to scale across populations from tens to hundreds of millions of devices, rounds with tens of thousands of participants, and updates from kilobytes to tens of megabytes. Its actor-based architecture supports sequential message handling, dynamic workload distribution, in-memory ephemeral state, pipelined rounds, and recovery from actor failures.
- The server must support populations from tens to hundreds of millions of devices, participant counts from tens to tens of thousands, and update sizes from kilobytes to tens of megabytes.
- The server uses the Actor Programming Model, where actors communicate solely through message passing and process event streams sequentially.
- Actor instances scale across processors and machines, make local decisions, communicate with other actors, and dynamically create additional actors.
- Coordinators synchronize rounds, Selectors manage device connections, and Master Aggregators dynamically spawn Aggregators to scale task execution with device counts and update sizes.
- Ephemeral in-memory actors avoid distributed-storage latency and persistent per-device update logs, while independent Selection enables pipelining with a previous round’s Configuration and Reporting phases.
- Actor failures preserve progress: Aggregator or Selector crashes lose only their connected devices, whereas a failed Master Aggregator causes its round to fail and the Coordinator restarts it.
5 ANALYTICS
The system relies on device and server analytics to monitor federated training health, performance, and failures without harming device utility. State-transition logs and dashboards help distinguish failure types, while deployment analytics reveal and verify operational and participation issues.
- Device analytics: Device analytics log training state, duration, memory use, errors, and health parameters to avoid wasting battery or bandwidth and degrading phone performance.These logs are sent to the cloud for monitoring.
- Failure diagnosis: State-transition events are visualized and counted in dashboards, enabling rapid differentiation between issue types across devices.A failed upload after successful training indicates a network issue, whereas failure immediately after model loading indicates a model issue.
- Server analytics: Server analytics track accepted and rejected devices, round-phase timing, uploaded and downloaded throughput, and errors.These measurements complement device-side monitoring of federated training behavior.
- Operational monitoring: Since deployment, analytics have repeatedly uncovered and helped verify fixes for device-health problems and unexpectedly high participant dropout rates.Examples include training running when it should not and training participants dropping out more often than expected.
- Protecting device utility: Accurate analytics are needed because improper federated-training operation can indirectly degrade mission-critical device utility, even without immediate user-experience impacts.Such utility degradations are difficult to pinpoint and easy to misdiagnose.
6 SECURE AGGREGATION
Secure Aggregation protects individual federated-learning updates from server inspection by revealing only their sum after enough updates arrive. The system optionally runs this four-round, dropout-robust protocol at the Aggregator level because several costs grow quadratically with users, limiting practical instances to hundreds of devices.
- 6 SECURE AGGREGATION: Secure Aggregation encrypts individual device updates so the server cannot inspect them, revealing only their sum after sufficient updates are received.It can protect against additional data-center threats by keeping updates encrypted even in memory.
- 6 SECURE AGGREGATION: The platform supports Secure Aggregation and differential privacy as additional privacy-enhancing technologies, but its goal is to provide tools for building privacy-preserving applications.The cited differential-privacy techniques of McMahan et al. (2018) are currently implemented.
- 6 SECURE AGGREGATION: The optional Secure Aggregation protocol runs in four interactive rounds during an FL round’s reporting phase and tolerates substantial device dropout.Each round gathers device messages and returns an independently computed response to every device.
- 6 SECURE AGGREGATION: Quadratic costs, especially server computation, limit practical Secure Aggregation instances to hundreds of users.To avoid constraining round participation, the system runs one instance per Aggregator actor, producing an intermediate sum from that Aggregator’s devices.
7 TOOLS AND WORKFLOW
The system provides Python interfaces and tools for defining, testing, simulating, and deploying TensorFlow-based FL tasks to mobile devices. Its workflow addresses proxy-data development, compiled FL plans, and automated safety checks for resource use and runtime compatibility.
- Task definition: Model engineers define FL and evaluation tasks in Python using TensorFlow functions that map input tensors to metrics such as loss or accuracy.During development, sample test data or proxy data can stand in for on-device examples.
- Testing: FL tasks are validated against engineer-provided test data and expectations, with tests required before deployment.The infrastructure is intended to let engineers focus on their models while libraries build and test the corresponding FL tasks.
- Simulation: Simulation runs the same code as on-device execution across cloud jobs emulating devices and can scale to a large number of devices.It uses proxy data and may pre-train models before field refinement through FL.
- FL plans: Each FL task has an automatically generated plan that describes orchestration independently of Python, with separate device and server portions.The device portion includes the TensorFlow graph, data-selection and batching instructions, epochs, and computation labels; the server portion contains aggregation logic.
- Deployment safety: Deployment requires auditable peer-reviewed code, passing simulation predicates, safe tested resource consumption, and compatibility across claimed TensorFlow runtimes.Engineers rely on versioning, testing, and deployment infrastructure because tasks may consume excessive RAM or be incompatible with fleet runtimes.
8 APPLICATIONS
Federated Learning is most useful when relevant, privacy-sensitive, or otherwise impractical-to-transmit data remains on devices, with current applications focused on supervised tasks using labels inferred from user activity. Applications include on-device ranking and suggestions, as well as Gboard next-word prediction, whose federated RNN improves top-1 recall over an n-gram baseline and matches a server-trained RNN.
- Applications: Federated Learning targets on-device supervised learning when data is more relevant locally, privacy-sensitive, or undesirable or infeasible to transmit to servers.Current applications typically infer labels from user activity such as clicks or typed words.
- On-device item ranking: On-device ranking eliminates expensive server calls while keeping potentially private search queries and user selections on the device.User interactions with the ranking feature can provide labeled data points.
- Content suggestions for on-device keyboards: Federated Learning trains on-device keyboard models both to trigger content suggestions and rank contextually relevant items, as demonstrated by Google’s Gboard team.The Gboard team used the described Federated Learning system for this application.
- Next word prediction: 1.4 million parameters characterize Gboard’s federated RNN, which converges in 3000 FL rounds after processing 6e8 sentences from 1.5e6 users over 5 days.Each round takes about 2–3 minutes.
- Next word prediction: 13.0% to 16.4% is the next-word prediction RNN’s top-1 recall improvement over a baseline n-gram model, while matching a server-trained RNN requiring 1.2e8 SGD steps.The federated model was trained using Gboard’s Federated Learning platform.
9 OPERATIONAL PROFILE
The deployed FL system has operated production workloads for over a year and currently serves approximately 10M daily active devices, with up to 10k participating simultaneously. Production observations indicate that a few hundred devices per round usually suffice, while dropout rates range from 6% to 10%; convergence remains roughly 7× slower than comparable data-center training.
- Operational measurement: The reported operational metrics come from more than a year of production workloads, are examples rather than a complete characterization, and were not collected under controlled measurement conditions.The authors note that FL has not yet been applied to a diverse enough set of applications for complete characterization.
- Convergence performance: Wall-clock convergence is roughly 7× slower than comparable data-center training of the same model, motivating continued optimization for rapid model-engineering iteration.The comparison uses different proxy data for data-center training, and the authors do not consider it the primary comparison.
- Population scale: Approximately 10M daily active devices currently span several applications, while the system is designed to elastically scale potentially into the billions.The 10M figure is a cumulative FL population size.
- Concurrent participation: Up to 10k devices participate simultaneously, with observed participation varying by as much as 4× between low and high levels.Participation depends on device eligibility, pace steering, and local time of day; devices are more likely to participate when idle and charging at night.
- Round sizing: A few hundred device updates per FL round are sufficient for most models, with diminishing convergence-rate improvements from larger device counts.This observation is based on prior work and experiments conducted on production FL populations.
- Reliability: 6%–10% of devices drop out on average because of computation errors, network failures, or eligibility changes.The observed dropout portion varies across production FL populations.
10 RELATED WORK
The system is presented as the first production-level Federated Learning implementation, centered on Federated Averaging on mobile phones. Related work spans alternative privacy-preserving and resource-aware FL algorithms, distributed ML and parameter-server systems, and MapReduce, while emphasizing mobile-specific constraints and broader applicability.
- Alternative Approaches: The authors describe their system as the first production-level Federated Learning implementation, primarily using Federated Averaging on mobile phones.They note that other approaches and application settings for learning from mobile-held data exist.
- Alternative Approaches: Pihur et al. (2018) avoids server aggregation and adds formal privacy guarantees for generalized linear models, while the authors argue their synchronous design remains scalable through online update processing.Their server can process updates as received without storing them, addressing synchronization and storage concerns.
- Alternative Approaches: Federated Learning has also been proposed for vehicle-to-vehicle communication and medical applications, although this system does not directly apply to those scenarios.The authors state that many aspects of the system would likely remain relevant for production applications there.
- Alternative Approaches: Nishio and Yonetani (2018) addresses heterogeneous device availability and sequential update reception, offering a resource-aware participant-selection algorithm implementable within this system.The algorithm maximizes the number of participants in a round under cellular bandwidth limits.
- Distributed ML: Unlike general distributed ML systems and parameter servers designed for datacenter nodes, this system uses a more structured approach for mobile devices with lower bandwidth and reliability.The parameter-server comparison concerns shared global state, asynchronous worker access, and vectors sized 10^9 to 10^12.
- MapReduce: FL resembles MapReduce with the server as Reducer and devices as Mappers, but devices retain their data, control participation, and may drop out during computation.The server actively scans for available devices and selects subsets for each computation round.
11 FUTURE WORK
Future work addresses participation bias, slower convergence, operational scheduling, bandwidth costs, and broader applications beyond machine learning. The system is intended to evolve toward Federated Computation, including Federated Analytics without logging raw device data to the cloud.
- Bias: Device availability may bias Federated Averaging because training requires unmetered connectivity and charging, while deployment is limited to certain recent Android phones with at least 2 GB.The system currently detects resulting model problems through live A/B experiments with application-specific metrics; no practical issue has been observed so far.
- Convergence Time: Federated Learning currently converges more slowly than centralized training, and Federated Averaging efficiently uses only 100s of devices despite greater available parallelism.Future algorithms should exploit increased parallelism.
- Convergence Time: Training-selection and reporting windows should be dynamically adjusted to reduce dropout and increase round frequency, using online ML with contextual signals such as time of day.These windows are currently configured statically for each Federated Learning population.
- Device Scheduling: The multi-tenant on-device scheduler uses a simple worker queue and avoids parallel sessions, potentially repeating training on older data while neglecting newer app data.The scheduler is blind to users’ app-usage patterns.
- Bandwidth: For some models, especially recurrent language-modeling networks, weight updates can exceed raw-data uploads, motivating improved compression to reduce bandwidth.The paper frames this as a privacy tradeoff but identifies substantial room for improvement.
- Federated Computation: The system aims to generalize from Federated Learning to Federated Computation, supporting general MapReduce-like workloads and Federated Analytics over aggregate statistics without logging raw device data to the cloud.This architecture is not restricted to machine learning with TensorFlow.
A OPERATIONAL PROFILE DATA · B FEDERATED AVERAGING
Operational data show that device availability, dropout behavior, capped participation time, asymmetric network traffic, and session outcomes shape deployed federated-learning rounds. The section also presents Federated Averaging, which selects extra clients, aggregates K updates, and returns compressed weighted model updates.
- A OPERATIONAL PROFILE DATA: Round completion rate oscillates with device availability because the server schedules tasks only after enough devices are available and selected.Availability varies through the day, producing synchronized changes in successful round completion.
- A OPERATIONAL PROFILE DATA: The server selects more devices than needed, allowing rounds to finish despite dropouts and aborting excess devices after enough successful completions.Dropout rate also correlates with time of day.
- A OPERATIONAL PROFILE DATA: Round run time is roughly equal to most device participation time because execution stops when enough selected devices complete.The server caps device participation time to handle straggler devices.
- A OPERATIONAL PROFILE DATA: Server download traffic dominates upload because devices download both the task plan and global model, while model updates are more compressible.The task plan is comparable in size to the global model, whereas devices upload only model updates.
- A OPERATIONAL PROFILE DATA: 75% of clients complete training rounds successfully, 22% complete them but have results rejected, and 2% are interrupted before completion.Rejections occur after the reporting window closes, while interruptions can result from leaving the idle state.
- B FEDERATED AVERAGING: Federated Averaging initializes w0 and, each round, selects 1.3K eligible clients before waiting for updates from K clients.The algorithm targets updates from K clients per round.
- B FEDERATED AVERAGING: Each client performs minibatch updates from w, returns ∆ = n · (w − winit) with n, and sends the update to the server for weighted aggregation.The update ∆ is noted as more amenable to compression than w.