Source-linked AI summary
Architectural Patterns for the Design of Federated Learning Systems
Sin Kit Lo, Qinghua Lu, Liming Zhu, Hye-young Paik, Xiwei Xu, Chen Wang
TL;DR
Federated learning faces design challenges as it brings many client devices and stakeholders into a distributed system, while non-IID client data can affect global-model quality. The paper describes architectural patterns for managing federated-learning systems, with reported consequences including better fit to non-IID data for multi-task models, client motivation, and scalability.
Problem
Global models may have low accuracy and lack generality when client devices generate non-IID data.
Source: Design Challenges · p. 2Method
The paper presents architectural patterns for client management and multi-task model training in federated-learning systems.
Source: §3.3.1 · p. 11, §4 · p. 17Results
Multi-task learning is reported to fit non-IID data at each node better than a global model.
Source: §3.3.1 · p. 11Takeaways & Limitations
The reported patterns include an incentive mechanism effective in attracting client contributions and an edge layer that helps the system handle more client devices.
Source: §3.3.3 · p. 13, §3.4.3 · p. 16
Abstract
from arXiv · showhide
Federated learning has received fast-growing interests from academia and industry to tackle the challenges of data hungriness and privacy in machine learning. A federated learning system can be viewed as a large-scale distributed system with different components and stakeholders as numerous client devices participate in federated learning. Designing a federated learning system requires software system design thinking apart from machine learning knowledge. Although much effort has been put into federated learning from the machine learning technique aspects, the software architecture design concerns in building federated learning systems have been largely ignored. Therefore, in this paper, we present a collection of architectural patterns to deal with the design challenges of federated learning systems. Architectural patterns present reusable solutions to a commonly occurring problem within a given context during software architecture design. The presented patterns are based on the results of a systematic literature review and include three client management patterns, four model management patterns, three model training patterns, and four model aggregation patterns. The patterns are associated to the particular state transitions in a federated learning model lifecycle, serving as a guidance for effective use of the patterns in the design of federated learning systems.
1. Introduction
The introduction frames federated learning as a response to data scarcity and privacy constraints, while emphasizing that its distributed architecture brings design challenges beyond machine-learning techniques. It proposes lifecycle-linked architectural patterns as guidance for those design decisions.
Federated Learning Overview
- Federated learning can address data scarcity and privacy constraints, but its many stakeholders and client devices create architectural challenges that require software design thinking.The introduction highlights non-IID data, communication costs, limited client resources, and coordination, provenance, reliability, and security concerns.Source: Federated Learning Overview · p. 1, Design Challenges · p. 2
A Motivation Example
- In the keyboard example, clients train locally and send model updates to a server for aggregation, while repeated rounds, limited device resources, and security concerns complicate the process.After convergence, the coordinator deploys the model to users and must maintain different local-model versions associated with the global model.Source: A Motivation Example · pp. 1–2
Design Challenges
- The design challenges include global-model quality and generality with non-IID data, communication costs, client resource limits, and coordinating reliability, security, and provenance.The paper identifies a gap in systematic architectural guidance for selecting designs that meet differing quality requirements and constraints.Source: Design Challenges · p. 2
Research Contribution
- The paper presents architectural patterns for federated learning and associates each pattern with a state transition in the model lifecycle.The collection spans client management, model management, model training, and model aggregation, and is intended as a solution pool and design guide for practitioners.Source: Research Contribution · p. 2
2. Methodology
The authors derive patterns using a systematic literature review, supplemented by later literature and real-world applications. They describe each pattern using a structured form covering its context, problem, forces, solution, consequences, relations, and known uses.
Method choice
- The authors chose a systematic literature review to derive architectural design patterns, rather than situational method engineering, which they considered more suitable for information-system development processes.They also noted that federated-learning materials and research were still highly academic-based.Source: §2 · p. 3
Pattern collection sources
- The pattern collection draws on a systematic literature review of 231 papers, later literature, and reviews of 22 real-world applications.Twelve patterns came from the SLR or additional literature review, while two were identified through real-world applications.Source: §2 · p. 3
Pattern description form
- Each pattern is presented through a form covering context, problem, forces, solution, consequences, related patterns, and known uses.Known uses refer to empirical evidence that a solution has been used in the real world.Source: §2 · pp. 3–4
3. Federated Learning Patterns
The paper organizes federated-learning patterns around the model lifecycle, including deployment of the trained global model to client devices for inference. It groups the patterns into four areas.
3. Federated Learning Patterns
- The lifecycle includes deployment of the completely trained global model to client devices for data inference, involving communication between the server and devices.Source: §3 · p. 4
- The patterns are grouped into client management, model management, model training, and model aggregation.Source: §3 · p. 4
3.1. Client Management Patterns
Client management patterns manage device information, select participants for training, and group clients by shared characteristics. These designs weigh model and system benefits against privacy and resource costs.
3.1.1. Pattern 1: Client Registry
- A central-server client registry requests participating devices’ information, which they return with their first local model updates.Source: §3.1.1 · p. 4
3.1.2. Pattern 2: Client Selector
- The central server selects clients each round using resource status, data quality and distribution, or local-model performance criteria.Source: §3.1.2 · p. 6
- Client selection can conserve aggregation resources, reduce dropout and latency, and improve global quality, but may reduce generality, raise privacy concerns, and add information-transfer costs.Source: §3.1.2 · p. 6
3.1.3. Pattern 3: Client Cluster
- Clients grouped by resource, data, feature, or location similarity receive cluster-related global models, targeting model performance and training efficiency.Source: §3.1.3 · p. 7
- Clustered clients can improve performance on highly personalised prediction tasks and converge faster when their data distributions and IIDness are similar.Source: §3.1.3 · p. 7
- Clustering adds server computation and time; its extra information needs may risk data leakage, while raw-data access to resolve non-IID issues raises privacy concerns.Source: §3.1.3 · p. 7
3.2. Model Management Patterns
The model-management patterns address communication, model provenance and replacement, and deployment across clients with different data or application needs.
3.2.1. Pattern 4: Message Compressor
- The compressor packages and compresses model parameters and task scripts before transfer, improving exchange efficiency at the cost of per-round computation and possible information loss.The pattern applies before each round of model exchange.Source: §3.2.1 · p. 8
3.2.2. Pattern 5: Model Co-versioning Registry
- The registry links each client’s local model versions to global versions, supporting quality tracing, accountability, adversarial-client detection, and identification of harmful local updates.It also supports early stopping by monitoring aggregated-model performance; storing all models and links adds retrieval burdens.Source: §3.2.2 · pp. 8–9
3.2.3. Pattern 6: Model Replacement Trigger
- Retraining starts when below-threshold performance is confirmed as a global event across sampled clients for a fixed number of consecutive checks, helping maintain quality as personalized data changes.Periodic client evaluation and sharing results add computation and communication costs.Source: §3.2.3 · pp. 9–10
3.2.4. Pattern 7: Deployment Selector
- After convergence, the selector matches clients to global models by data characteristics or application; deploying suitable models improves performance for those client groups.The pattern adds costs for multiple models and raises privacy concerns when the server collects client information for matching.Source: §3.2.4 · pp. 10–11
3.3. Model Training Patterns
The training patterns address non-IID and heterogeneous client data through related-task training and data handling, and use rewards to encourage client participation.
3.3.1. Pattern 8: Multi-Task Model Trainer
- Trade-offs include higher per-round computation, privacy exposure from exploring local distributions, weaker performance on non-convex losses, and portability problems across devices.The pattern is described as often working only with convex loss functions.Source: §3.3.1 · p. 11
- The multi-task trainer performs related tasks on client devices, letting task-specific local models use fitting local data and outperform a single global model on non-IID data.For example, next-word prediction can use on-device messages, searches, and emails with similar keyboard usage patterns.Source: §3.3.1 · p. 11
3.3.2. Pattern 9: Heterogeneous Data Handler
- The heterogeneous data handler uses local augmentation to balance data volume and classes, and federated distillation to share knowledge without direct access to other clients’ data, improving global-model performance and generality.Handling heterogeneity alongside local model training is computationally costly.Source: §3.3.2 · p. 12
3.3.3. Pattern 10: Incentive Registry
- The incentive registry records client contributions and agreed rewards, which can motivate participation; fraudulent submissions and privacy risks remain concerns.Reward schemes may use deep reinforcement learning, blockchain or smart contracts, or a Stackelberg game model.Source: §3.3.3 · pp. 12–13
3.4. Model Aggregation Patterns
This section presents four model aggregation patterns addressing aggregation speed, system reliability and accountability, model quality and efficiency, and model security, and identifies SEAL22 as a homomorphic encryption API for computing directly on encrypted data.
3.4.1. Pattern 11: Asynchronous Aggregator
- Asynchronous aggregation updates the global model as client updates arrive, reducing per-round latency but risking model bias and requiring more iterations to convergence.Delayed updates are included in the next round with reduced weight, and fewer simultaneous transfers reduce bandwidth usage per iteration.Source: §3.4.1 · pp. 13–14
3.4.2. Pattern 12: Decentralised Aggregator
- Decentralised aggregation replaces the central server with peer-to-peer exchanges and blockchain storage, improving reliability and accountability through immutable, transparent model records.A random client can aggregate updates from nearby devices while blockchain provides model storage and is maintained by the learning coordinator.Source: §3.4.2 · pp. 14–15
- Decentralised aggregation can incur latency from blockchain consensus, high computation and storage costs, and privacy exposure because clients can access model records.The client devices’ limited resources make repeated aggregation costly, while access to stored models may expose privacy-sensitive information.Source: §3.4.2 · p. 15
3.4.3. Pattern 13: Hierarchical Aggregator
- Hierarchical aggregation adds edge servers for partial aggregation, improving communication efficiency and scalability while reducing non-IID effects on the global model.Clients send updates to nearby edge servers, which aggregate every k1 local-training rounds; the cloud aggregates edge models every k2 edge aggregations.Source: §3.4.3 · pp. 15–16
3.4.4. Pattern 14: Secure Aggregator
- A secure aggregator uses secure multiparty computation, encryption, pairwise masking, and differential privacy to protect model exchanges and aggregation, with costs to efficiency and model performance.Encryption and decryption slow training and aggregation, aggressive privacy methods can obscure model information, and compromised keys increase privacy threats.Source: §3.4.4 · p. 17
- SEAL22, introduced by Microsoft AI, is a homomorphic encryption API that allows computations directly on encrypted data.Source: §3.4.4 · p. 17
4. Discussion
The discussion maps architectural patterns to federated learning’s management, training, exchange, and security challenges. It highlights trade-offs involving non-IID data, system resources, client information, and distributed ownership.
4. Discussion
- Client registry, selector, and cluster manage devices during job creation to improve model performance and system and training efficiency.The discussion also identifies heterogeneous data handler and incentive registry for training-stage issues associated with non-IID local data.Source: §4 · p. 17
- Non-IID data can improve local personalisation while hurting global-model generalisation, prompting patterns for heterogeneity and personalised deployment.Multi-task model trainer uses multi-task or transfer learning; deployment selector chooses user clients for models suited to their local data.Source: §4 · p. 17
- Selection and scheduling patterns can improve resource consumption, but collecting client resource or performance information raises privacy and overhead concerns.Message compressor and hierarchical aggregator are proposed to address those information-collection concerns; incentive registry aims to encourage participation.Source: §4 · p. 17
- Distributed component ownership creates security risks from unauthorised clients and dishonest updates, which secure aggregator, model co-versioning registry, and client registry target.The discussion also proposes a decentralised aggregator to address trust concerns and the central server’s possible single-point-of-failure.Source: §4 · pp. 17–18
5. Related Work
Prior work examined machine-learning engineering and federated-learning system design, but the paper identifies a gap in reusable architectural patterns for federated learning. It presents its collection as guidance for system designers.
5. Related Work
- Enterprise machine-learning applications are often software components, motivating systematic software-engineering approaches to model-building challenges.The passage frames this as a way researchers view models within larger enterprise systems.Source: §5 · p. 18
- Prior machine-learning engineering work examined development practices and industrial adoption maturity, mapping challenges to machine-learning pipeline stages.The passage attributes these studies respectively to Wan et al. and Lwakatare et al..Source: §5 · p. 18
- Machine-learning design-pattern work studied software-engineering patterns and proposed an architectural pattern for operational stability.That work separates business-logic and machine-learning components and addresses pipeline management, data management, and model versioning.Source: §5 · p. 18
- Earlier federated-learning system-design work focused on a high-level basic system and protocol definition, while the authors report no prior study defining reusable architectural patterns for its design.The authors describe their work as the first comprehensive and systematic collection of such patterns, intended to guide practitioners.Source: §5 · p. 18
6. Conclusion
The conclusion presents 14 federated-learning architectural patterns associated with the model lifecycle as guidance for system design. It identifies improving trust as future work.
6. Conclusion
- The paper presents 14 architectural patterns associated with the federated-learning model lifecycle to guide system design and development.The conclusion characterizes federated learning as privacy-preserving distributed machine learning using IoT and smart-device data and resources.Source: §6 · p. 18
- Future work will explore architectural designs that help improve trust in federated learning.Source: §6 · p. 18