PaperlayerView exact revision on arXiv

Source-linked AI summary

From Distributed Machine Learning to Federated Learning: A Survey

Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, Dejing Dou

arXiv:2104.14362v4cs.DCcs.AIcs.LG

TL;DR

Federated learning addresses the challenge of training on distributed, sensitive data that cannot be directly shared because of legal, security, and privacy constraints. This paper surveys FL and its parallelization techniques, proposes a four-layer architecture and taxonomy, and reviews training, communication, security, frameworks, limitations, and future directions. It concludes that current frameworks primarily support horizontal FL while differing in supported data distributions, parallelism, interfaces, engines, and security techniques.

  • Problem

    Sensitive data distributed across devices, regions, and organizations cannot always be centralized or directly shared because of legal, security, and privacy constraints.

  • Method

    The paper provides a survey with a four-layer FL architecture, a taxonomy of related techniques, and analyses of distributed training, data communication, security, frameworks, and research directions.

  • Results

    Current FL frameworks primarily focus on horizontal FL, while differing in supported vertical and hybrid FL, parallelism, interfaces, engines, and security techniques.

  • Takeaways & Limitations

    The architecture and taxonomy provide a basis for discussing, assessing, comparing, and selecting FL systems and related techniques.

  • Takeaways & Limitations

    Open limitations include interpretability, decentralized aggregation, FL on graphs, benchmarks, imbalanced data, and applications to distributed intelligent systems.

Abstract

from arXiv · show

In recent years, data and computing resources are typically distributed in the devices of end users, various regions or organizations. Because of laws or regulations, the distributed data and computing resources cannot be directly shared among different regions or organizations for machine learning tasks. Federated learning emerges as an efficient approach to exploit distributed data and computing resources, so as to collaboratively train machine learning models, while obeying the laws and regulations and ensuring data security and data privacy. In this paper, we provide a comprehensive survey of existing works for federated learning. We propose a functional architecture of federated learning systems and a taxonomy of related techniques. Furthermore, we present the distributed training, data communication, and security of FL systems. Finally, we analyze their limitations and propose future research directions.

1 Introduction

Federated learning addresses privacy, security, ownership, and legal constraints by collaboratively training models on distributed resources without transferring raw training data. This survey organizes FL through a four-layer architecture, a taxonomy of related techniques, and analyses of training, communication, security, and open issues.

  • Motivation: Legal restrictions and sensitive information make transferring raw data to centralized servers a privacy and security concern.The paper cites laws including GDPR, CLPR, PDP, CCPA, and CPBR.
  • Motivation: Federated learning keeps raw data decentralized while allowing multiple users or organizations to collaboratively train machine learning models.Only intermediate data is transferred among distributed computing resources, bringing code to data rather than data to code.
  • Scope: The paper surveys distributed training, data communication, security, existing FL frameworks, and research issues for improving FL efficiency and security.Its organization spans system execution, frameworks, open issues, and concluding findings.
  • Contributions: The survey proposes a four-layer FL architecture and a taxonomy covering parallelization, aggregation, data communication, and security techniques.The architecture is intended as a baseline for assessing and comparing FL systems.

2 An Overview of Federated Learning

The overview defines FL as distributed machine learning that trains models across locally retained data, then describes its optimization, life cycle, and four-layer system architecture. It also distinguishes data, model, and pipeline parallelism and explains how the layers support model design, execution, and analysis.

  • Distributed Training: Distributed training uses data, model, and pipeline parallelism, with data parallelism associated with horizontal FL and model parallelism with vertical FL.The overview frames these parallelism types as mechanisms for distributing training across resources.
  • Basic Concepts: Federated learning trains a shared model across distributed resources while keeping raw data from being moved to a central server or data center.The approach is presented as a special type of distributed machine learning with legal, privacy, and security distinctions.
  • Basic Concepts: The FL training objective learns a function from hypotheses H while minimizing expected loss over distributed datasets D = D1, D2, ..., Dn.The paper states that stochastic gradient descent is generally used to minimize the loss function.
  • FL Model Life Cycle: The paper describes an FL model life cycle condensed into four phases, beginning with composition and including deployment as the final phase.Composition creates or adapts a model for a specific machine learning problem, while deployment places the final model in a real-life scenario.
  • Functional Architecture: The functional architecture has presentation, user services, FL training, and infrastructure layers, with higher layers exploiting lower layers.The FL training layer generates a Federated Learning Execution Plan and carries out distributed training.
  • Functional Architecture: The presentation layer provides textual or graphical user interaction for designing FL models and viewing distributed-training status.FATE is cited as an example of a framework providing a web-portal graphical interface.
  • Functional Architecture: User services support monitoring, steering, logging, interpretability and explainability, and graph data functionality.Monitoring helps users track long-running distributed training and verify whether execution proceeds normally.

3 Distributed Training

The paper presents distributed FL training through two components: parallelism and aggregation algorithms. These components are generally implemented in the parallelization and scheduling modules, respectively.

  • Distributed Training: Distributed FL training is organized around parallelism approaches and aggregation algorithms.The section first introduces three types of parallelism, then discusses aggregation methods for distributed training.

3.1 Parallelism & FL Types

Federated learning maps data and model organization onto distributed-computing parallelism: horizontal FL generally uses data parallelism, vertical FL model parallelism, while hybrid FL relies on transfer learning. Data, model, and pipeline parallelism differ in what is distributed and how resources communicate.

  • FL Types: FL has horizontal, vertical, and hybrid types; horizontal generally exploits data parallelism, vertical typically uses model parallelism, and hybrid relies on transfer learning.Hybrid FL is outside the paper’s parallelism scope because transfer learning is not a parallelism approach.
  • Data Parallelism: Data parallelism runs the same model on different data points at separate computing resources, transferring models or gradients rather than training data.The resources perform forward and backward propagation concurrently, then average models or gradients for subsequent training.
  • Model Parallelism: Model parallelism distributes different parts of each input data point across computing resources, with inter-resource dependencies possibly linking processing nodes.Distributed nodes may be independent or dependent, depending on whether one node’s execution relies on another’s output.
  • Pipeline Parallelism: Pipeline parallelism distributes dependent processing nodes across resources so successive stages can process data in parallel.Its many inter-computing-resource transfers make pipeline parallelism uncommon in federated learning.

3.2 Aggregation Algorithms

Federated aggregation combines locally computed models or gradients through centralized, hierarchical, or decentralized architectures. These designs trade off coordination structure, communication, topology, fairness, regularization, and other system requirements.

  • Aggregation Architectures: Aggregation algorithms combine models or gradients from computing resources using centralized, hierarchical, or decentralized coordination.Centralized schemes use one parameter server, hierarchical schemes use multiple parameter servers, and decentralized schemes communicate across neighboring resources.
  • Centralized Aggregation: Centralized aggregation sends local models or gradients to one parameter server, which computes global values and returns them to the resources.The server synchronizes or schedules distributed computation, while SGD updates the model during training.
  • Centralized Aggregation: FedAvg aggregates selected users’ models with a weighted sum, shares the resulting global model, and continues training locally.It is introduced as the aggregation method in Google’s federated-learning implementation.
  • Centralized Aggregation: Other centralized algorithms address objectives including regularization, fairness, permutation alignment, and communication-round reduction.SAFL and FedMGDA+ target fairness; FedMA aligns processing nodes; SCAFFOLD uses stateful variables to reduce communication rounds.
  • Hierarchical Aggregation: Hierarchical aggregation averages models within regional clusters, sends regional averages to a global server periodically, and broadcasts the global average back.Regional parameter servers can reduce model-transfer time between computing resources and the global parameter server.
  • Decentralized Aggregation: Decentralized aggregation uses peer-to-peer communication over a connected topology, where weights scale information flowing between neighboring resources.Topology degree and connectivity affect communication efficiency and aggregation convergence; doubly stochastic weights support asymptotic global averaging.

4 Data Manipulation

FL data manipulation addresses distributed execution, communication, data privacy, and model security. The surveyed techniques include compression, trusted environments, encryption, differential privacy, and defenses against poisoning attacks.

  • Infrastructure: FL infrastructure combines distributed data processing, intermediate-data transfer, and data security mechanisms.Data security covers privacy protection for raw data and model security against malicious attacks.
  • Data Transfer: Moderate-quality networks motivate compressing gradients or models before transfer during distributed training.Sketched updates and subsampling reduce the amount of intermediate data sent between resources.
  • Data Transfer: Sparsification transfers only intermediate-data components exceeding a threshold, after which the server decompresses them for aggregation.Quantization convergence is analyzed as supporting good convergence.
  • Data Privacy: TEE, encryption, differential privacy, and anti-GAN methods protect privacy, with combinations used in some FL systems.TEE can protect either the entire training process or a checked portion while insecure resources reduce training time.
  • Data Privacy: Differential privacy improves privacy through added noise but can slow model convergence, creating a privacy–performance trade-off.DP may also fail to ensure privacy under certain GAN attacks.
  • Model Security: Poisoning attacks modify training data or intermediate model updates to reduce targeted model accuracy.Model poisoning directly changes model weights and can be more powerful than data poisoning.
  • Model Security: Defenses identify malicious users by analyzing updates with dimensionality reduction, anomaly detection, and interpretability techniques.Byzantine failures can also cause model poisoning in distributed computing resources.

5 Federated Learning Frameworks

The survey compares four open-source FL frameworks across interfaces, training capabilities, privacy mechanisms, and supported FL types. All support horizontal FL, while vertical and hybrid support differ across frameworks.

  • Frameworks: PaddleFL, TensorFlow Federated, FATE, and PySyft are open-source frameworks used for federated learning across mobile, healthcare, and finance domains.The frameworks differ in interfaces, engines, aggregation, parallelism, and security support.
  • Comparison: All four frameworks implement centralized aggregation algorithms and employ differential privacy and homomorphic encryption for data security.Their broader capabilities vary by framework and underlying machine-learning engine.
  • Comparison: PaddleFL supports data, model, and pipeline parallelism, while FATE provides a Web portal and PySyft is compatible with PyTorch tasks.PaddleFL is compatible with Paddle and its pretrained models; FATE and TFF use TensorFlow engines.
  • FL Support: All frameworks support horizontal FL and GPU execution, whereas vertical FL excludes TFF and hybrid FL is limited to PaddleFL and FATE.PySyft can use PyVertical for vertical FL, and its execution may produce out-of-memory errors.
  • Practical Comparison: PaddleFL may take slightly longer while achieving higher accuracy than TFF and FATE, whereas PySyft may generate out-of-memory errors.These observations concern practical framework comparisons reported for the surveyed support scenarios.

6 Research Directions

The survey identifies limitations and future directions spanning benchmarks, interpretability, aggregation, graph-based FL, imbalanced data, and distributed intelligent systems.

  • Existing FL research remains limited in interpretability, decentralized aggregation, graph-based learning, benchmarking, imbalanced-data handling, and distributed intelligent-system applications.These areas are presented as open research directions for improving FL systems.
  • 6.1 Benchmarks: Public benchmarks cover horizontal FL, but datasets for vertical and transfer FL, and decentralized IID or non-IID distributions of popular datasets, remain unavailable.FEMNIST and Shakespeare support horizontal FL experiments, while the other identified settings lack public datasets.
  • 6.2 Interpretability: Interpretability remains an open problem for understanding FL models, evaluating user contributions, and informing incentives in domains such as healthcare and finance.Existing Shapley-value work focuses on vertical FL.
  • 6.3 Decentralized Aggregation: Current aggregation emphasizes fully connected or star topologies, while alternative topologies and security against poisoning attacks require further investigation.Dynamic exponential-2 graphs may accelerate distributed training, but diverse attacks remain a security concern.
  • 6.4 Federated Learning on Graphs: Graph-based FL frameworks exist, but security for decentralized graphs and efficient multimodal knowledge-graph construction remain open problems.The cited examples include GraphFL and SGNN.
  • 6.5 Imbalanced Data: Real-world decentralized data are often imbalanced, requiring attention to sampling, augmentation, and loss-function design within FL.The survey identifies optimized approaches for imbalanced data as a future research need.

7 Conclusion

The paper surveys FL systems through their architecture, distributed training, data manipulation, and security techniques. It also reviews current systems and identifies research directions for further investigation.

  • Functional architecture: The survey presents a four-layer functional architecture covering presentation, user services, FL training, and infrastructure.The architecture supports detailed discussion and comparison of FL systems.
  • Distributed training: It analyzes distributed training through data, model, and pipeline parallelism, alongside centralized, hierarchical, and decentralized aggregation.Data parallelism is associated with horizontal FL, while model parallelism relates to vertical and cross-silo FL.
  • Data manipulation and security: The paper examines distributed execution, communication, fault tolerance, intermediate-data compression, privacy, and model-security defenses.Discussed techniques include RPC, sketching, differential privacy, homomorphic encryption, trusted execution environments, and malicious-user identification.
  • Research directions: It identifies benchmarks, interpretability, decentralized aggregation, graph-based FL, imbalanced data, and distributed intelligent-system applications as research directions.The survey mainly introduces PaddleFL, TensorFlowFederated, FATE, and PySyft, while noting that current solutions primarily focus on horizontal FL.
Loading 2104.14362v4…