PaperlayerView exact revision on arXiv

Source-linked AI summary

Communication Efficiency in Federated Learning: Achievements and Challenges

Osama Shahid, Seyedamin Pouriyeh, Reza M. Parizi, Quan Z. Sheng, Gautam Srivastava, Liang Zhao

arXiv:2107.10996v1cs.LG

TL;DR

Privacy and data-sharing policies make centralized data transfer challenging, while repeated exchanges over device networks can constrain FL communication. This paper surveys communication challenges and proposed efficiency approaches. It concludes that communication remains important but less researched than privacy, security, and resources.

  • Problem

    Centralized data transfer can be challenging under privacy and data-sharing policies, and communication in FL remains a bottleneck that receives limited dedicated survey coverage.

    Source: §I · pp. 1–2, C. Publication Analysis · p. 4

  • Method

    The paper surveys FL communication challenges and research approaches, including local updating, client selection, and model-compression techniques.

    Source: §I · p. 2, 1) Local Updating · p. 6, 2) Client Selection · p. 7

  • Results

    The survey presents communication constraints and a range of proposed solutions, while concluding that communication research remains limited compared with work on privacy, security, and resources.

    Source: §V · p. 11

  • Takeaways & Limitations

    Communication is an important part of FL because model training occurs through communication between participants and the server.

    Source: §V · p. 11

Abstract

from arXiv · show

Federated Learning (FL) is known to perform Machine Learning tasks in a distributed manner. Over the years, this has become an emerging technology especially with various data protection and privacy policies being imposed FL allows performing machine learning tasks whilst adhering to these challenges. As with the emerging of any new technology, there are going to be challenges and benefits. A challenge that exists in FL is the communication costs, as FL takes place in a distributed environment where devices connected over the network have to constantly share their updates this can create a communication bottleneck. In this paper, we present a survey of the research that is performed to overcome the communication constraints in an FL setting.

I. INTRODUCTION

Federated learning offers an alternative to centralized training when privacy and data-sharing policies make transferring data difficult. This survey focuses on communication constraints in FL and research aimed at making communication more efficient.

I. INTRODUCTION

  • Privacy and data-sharing policies can make centralized data transfer challenging, motivating alternatives for building reliable machine-learning models.The introduction notes additional computational challenges with the centralized approach.Source: §I · p. 1
  • In FL, data stays on devices while participants collaborate on a global model, allowing hospitals to combine learning without sharing their private datasets.The paper gives hospitals as an example: each trains locally, then uploads a model for central averaging.Source: §I · p. 1
  • FL also distributes computation across participating parties, with training taking place directly on end devices rather than relying only on a central server.The introduction frames this as a theoretical benefit of decentralization.Source: §I · pp. 1–2
  • Communication can bottleneck FL because devices use slower or unreliable network connections and must repeatedly share updates.The survey identifies quantization and sparsification as model-compression techniques that can be integrated with FL averaging algorithms.Source: §I · p. 2
  • The survey focuses solely on FL communication challenges, aiming to summarize recent research because few surveys address the topic specifically.It identifies quantization and sparsification among the approaches discussed.Source: §I · p. 2

II. PROBLEM STATEMENT

The section introduces FL as collaborative machine learning over decentralized data, then describes its participants, central server, and communication process. Local training and central aggregation form the basic cycle.

A. Background

  • Federated learning trains machine-learning models over decentralized data, with devices retaining their data rather than transferring it to a central server.The background traces its introduction to Google's research team and motivation to preserve user privacy.Source: A. Background · p. 2

B. The components of FL systems

  • Participants keep private datasets on their devices, while the central server distributes the global model and aggregates uploaded local models into an updated global model.The section identifies data owners or participants and the central server as the two main FL entities.Source: B. The components of FL systems · p. 2

1) The processes of an FL system:

2) The different types of FL systems:

FL settings differ according to how data is distributed across devices and domains. The section distinguishes horizontal, vertical, and federated transfer learning by their overlap in features and samples.

2) The different types of FL systems:

  • Horizontal FL combines data with similar feature sets but different instances across devices or domains.The section presents this as the original sense of FL, with local models contributing to a global model; Figure 3 depicts horizontal FL.Source: 2) The different types of FL systems · p. 3, Figure 3 · p. 3
  • Vertical FL combines data from unrelated domains that may share sample IDs but differ in features, such as a bank's customer data and an e-commerce site's records.Figure 4 depicts vertical FL.Source: 2) The different types of FL systems · p. 3, Figure 4 · p. 3
  • Federated transfer learning applies when data sources share little similarity in samples or features, with transfer learning used across the partitioned data.The example describes geographically restricted institutions with a small intersection; the approach may outperform a newly built model in some cases.Source: 2) The different types of FL systems · p. 3, Figure 5 · p. 3

C. Publication Analysis

Prior FL surveys cover the technology, applications, privacy, security, and related topics, but communication is generally treated briefly. This survey distinguishes itself by focusing specifically on communication challenges and efficiency.

C. Publication Analysis

  • Existing FL surveys cover topics including applications, healthcare, edge computing, privacy, security, and threats such as poisoning and inference attacks.The section reviews surveys including,,,, and [30].Source: C. Publication Analysis · pp. 3–4
  • This survey focuses solely on FL communication, aiming to bridge the gap in dedicated coverage of communication challenges and efficiency.The paper presents its focus as distinct from surveys that address communication only alongside broader topics.Source: C. Publication Analysis · p. 4
  • Although other surveys introduce communication challenges, communication typically receives only brief, surface-level treatment.The paper cites and [32] in describing this coverage gap.Source: C. Publication Analysis · p. 4

III. RESEARCH QUESTIONS & COMMUNICATION EFFICIENT METHODS

The section frames two questions: what creates communication challenges in FL, and how communication can be made more efficient. It identifies device scale, constrained or unreliable networks, edge-device limits, and statistical heterogeneity as bottleneck sources.

III. RESEARCH QUESTIONS & COMMUNICATION EFFICIENT METHODS

  • The section asks what communication challenges FL faces and which methods could make communication more efficient.These are posed as RSQ1 and RSQ2.Source: §III · p. 4

in FL with regards to communication?

  • Millions of participating devices, repeated model exchanges, and limited bandwidth, energy, or power can make communication rounds slow.Model updates are still described as lower-cost to share than transferring large volumes of device data to a central server.Source: in FL with regards to communication? · p. 4
  • A large number of simultaneous participants can create communication bottlenecks and increase computational cost.More participants may also provide more training data and potentially improve model performance and accuracy.Source: in FL with regards to communication? · p. 4
  • Unreliable device bandwidth and differing upload and download speeds can delay model uploads and disrupt FL.The survey notes that FL reduces communication cost relative to traditional ML, but bandwidth still needs to be preserved.Source: in FL with regards to communication? · p. 4
  • Limited edge computation, power, storage, and bandwidth constrain devices compared with central servers.The passage contrasts GPU training at 56Gbps with a powerful smartphone connected over 5G averaging 50Mbps.Source: in FL with regards to communication? · p. 5
  • Non-i.i.d. data and uneven dataset sizes can delay updates because aggregation may wait for every client to train and upload.A device with more data may take longer to update than one with less data.Source: in FL with regards to communication? · p. 5

B. RSQ2 - How can we make communication more efficient

FL communication involves distributing a global model, training it locally, and returning updates for aggregation. The survey reviews local updating approaches that seek to reduce communication rounds while addressing participation, latency, and data-distribution challenges.

in an FL environment?

  • Each FL round distributes the global model, trains it on private local data, and returns client models for aggregation into an updated global model.The survey identifies reducing communication across these steps as a target for efficiency.Source: in an FL environment? · p. 5

1) Local Updating:

  • Local updating faces dropout and synchronization-latency constraints, while non-i.i.d. local data can challenge accuracy and affect global-model performance [41].The survey presents flexibility in client participation and local updates as a motivation for more efficient local-update methods.Source: 1) Local Updating · p. 6
  • Hierarchical clustering groups clients by similarity of local updates, and the authors report reduced communication rounds using a Manhattan-distance comparison [43],.The passage provides no numerical reduction value.Source: 1) Local Updating · p. 6
  • FedPAQ uses periodic averaging, partial node participation, and quantized message-passing to reduce communication overhead.Periodic averaging adjusts how many local iterations occur before synchronization with the central server.Source: 1) Local Updating · p. 6
  • SCAFFOLD addresses FedAvg client drift on heterogeneous data and is reported to provide reliable convergence rates like SGD even on non-i.i.d. data,, [48].The survey says client drift can cause slow and unstable convergence.Source: 1) Local Updating · p. 6
  • FedDANE collects gradient updates from a subset of devices each round and is described as suitable for low client participation.Source: 1) Local Updating · p. 6

2) Client Selection:

Client selection reduces communication by limiting which devices participate in a round. The surveyed approaches account for client resources and network conditions, and report differing effects on communication rounds, convergence, energy, or training time.

2) Client Selection:

  • Client selection limits participating devices so only a fraction of parameters is updated in a communication round.Source: 2) Client Selection · p. 6
  • FedMCCS considers device CPU, memory, energy, and time, and is reported to reduce communication rounds versus other approaches while achieving reasonable accuracy.Its participating-client count increases over successive rounds.Source: 2) Client Selection · p. 7
  • A client-selection algorithm using wireless-channel information provides a long-term performance guarantee and adapts its selection pattern to network environments.The authors describe learning rounds as temporally interdependent and varying in significance.Source: 2) Client Selection · p. 7
  • FedCS requests client resources and estimates distribution, scheduling, and upload time so the server can aggregate as many clients as possible within a time frame.The passage says accounting for data, energy, and computation could affect energy consumption and bandwidth cost.Source: 2) Client Selection · p. 7
  • Mobile Crowd Machine Learning uses deep Q-learning to make client-participation decisions without prior network knowledge and is described as reducing energy use and training and communication time.Source: 2) Client Selection · p. 7
  • Power-Of-Choice trades off selection bias and convergence speed; its results are reported as faster and more accurate than baseline random selection [54].The selection bias favors clients with higher local loss, which the passage associates with faster error convergence.Source: 2) Client Selection · p. 7

3) Reducing Model Updates:

The survey covers approaches that reduce the frequency or burden of model updates. These include drift-adaptive training, partitioned inference, one-shot communication, and sampling clients to optimize expected convergence speed.

3) Reducing Model Updates:

  • An approach that adapts to model-training drifts is reported to reduce communication substantially without depreciating predictive performance.Source: 3) Reducing Model Updates · p. 7
  • Partitioned Variational Inference supports synchronous or asynchronous updates for federated Bayesian neural networks and is described as communication-efficient on non-i.i.d. data [56].The passage notes that this approach is integrated with other methods.Source: 3) Reducing Model Updates · p. 7
  • A one-shot approach uses a single communication round, with clients training local models to completion and ensemble methods combining device-specific information [58].The authors suggest ensemble learning may suit global modeling better than averaging.Source: 3) Reducing Model Updates · p. 7
  • FOLB samples participating devices each round to optimize expected convergence speed and estimates device capabilities through adaptation to device aggregations.Source: 3) Reducing Model Updates · p. 7

4) Decentralized Training & Peer-to-Peer Learning:

Federated learning commonly uses a central-server star topology, while peer-to-peer approaches offer alternatives that may reduce communication costs. These alternatives bring their own convergence and communication constraints.

4) Decentralized Training & Peer-to-Peer Learning:

  • The star topology is predominant in FL, with participating devices connected to the central server holding the original model.QuanTimed-DSGD imposes iteration deadlines and has nodes exchange quantized models to address decentralized training constraints.Source: 4) Decentralized Training &… · p. 8
  • Online Push-Sum handles complex topologies in a generic social-network FL framework while achieving optimal convergence rates.In peer-to-peer learning, devices can communicate with neighbors and aggregate their local models with neighbors’ updates.Source: 4) Decentralized Training &… · p. 8
  • A server-free distributed algorithm has devices iteratively exchange and aggregate model information with one-hop neighbors to estimate global parameters.Its authors motivate the approach by the communication costs of central-server FL.Source: 4) Decentralized Training &… · p. 8
  • BrainTorrent’s peer-to-peer framework avoids a central training server, which its authors say makes the environment failure-resistant and removes the need for a trusted governing body.Source: 4) Decentralized Training &… · p. 8
  • Decentralized training could theoretically reduce communication costs compared with a central server, but the paper notes that further research is needed.The discussion also describes decentralized learning as potentially faster under low bandwidth and high latency, while noting node computation time and communication overhead can slow convergence.Source: 4) Decentralized Training &… · p. 8

5) Compression Schemes:

The survey reviews compression approaches intended to reduce FL communication costs, covering sparsification and quantization alongside changes to model exchange and training. The approaches address bandwidth and participation constraints, with reported trade-offs and settings varying across methods.

5) Compression Schemes:

  • Compression schemes such as quantization and sparsification can reduce communication rounds while providing reasonable training accuracy.Source: 5) Compression Schemes · p. 8
  • Compression targets client-to-server updates and server-to-client global models, while training-algorithm changes target computational efficiency.The paper notes that clients generally have slower upload bandwidth than download bandwidth, making reduced exchanges a potential way to lower communication costs.Source: 5) Compression Schemes · pp. 8–9

Sparsification

  • STC compresses both upstream and downstream communication, and its reported convergence is faster than FedAVG under non-IID data and fewer communicated iterations.The paper also describes STC as using sparsification, ternarization, error accumulation, and optimal Golomb encoding, and reports effectiveness under bandwidth constraints.Source: 5) Compression Schemes · p. 9
  • FetchSGD compresses client gradients with Count Sketch, while GGS combines local-gradient batch-normalization updates and gradient correction to address synchronization overhead.The GGS authors report high accuracy and reduced communication overhead; CPFed combines privacy masking with an unbiased random sparsifier for communication efficiency.Source: 5) Compression Schemes · pp. 9–10
  • Sparse Binary Compression combines communication delay, gradient sparsity, and binarization; its authors report higher compression gains with a slight accuracy decrease.Source: 5) Compression Schemes · p. 10

Quantization

  • FedPAQ quantizes local models before uplink, while Lossy FL quantizes both the global model broadcast and local updates to reduce communication load.Source: 5) Compression Schemes · p. 10
  • HSQ targets low per-iteration communication while preserving convergence accuracy; UVeQFed’s distortion reduces substantially as the number of users grows.The survey also describes quantizers designed for distributed deep-network training and rate-constrained FL channels.Source: 5) Compression Schemes · pp. 10–11
  • Heir-Local-QSGD uses client-edge-cloud hierarchy for partial edge aggregation and quantized model updates to improve communication efficiency.Source: 5) Compression Schemes · p. 11

IV. DISCUSSION

The discussion frames communication as the repeated model exchange between the server and participating devices, which can be slowed by limited bandwidth or poor client participation. It reviews efficiency techniques but notes that their resource and dataset settings vary.

IV. DISCUSSION

  • FL communication repeatedly sends the global model to devices and returns locally trained models to the server, requiring substantial upload and download bandwidth.Source: §IV · p. 11
  • Limited bandwidth or poor client participation can create delays in completing FL training.Source: §IV · p. 11
  • The reviewed methods use different computational resources and datasets, and the discussion suggests combining techniques could yield a more communication-robust FL setting.Source: §IV · p. 11

V. CONCLUSION & FUTURE EXPECTATIONS

The conclusion presents the survey as a bridge addressing limited research on FL communication relative to privacy, security, and resources. It summarizes communication challenges and solutions and calls for further research.

V. CONCLUSION & FUTURE EXPECTATIONS

  • The survey aims to bridge a gap in research on communication in FL, which the authors describe as limited compared with work on privacy, security, and resources.Source: §V · p. 11
  • The paper organizes communication challenges, including bandwidth and limited computation, into two research questions alongside a list of proposed solutions.Source: §V · p. 11
  • The authors hope future research on FL communication will grow and make communication efforts more efficient.Source: §V · p. 11

Related papers

Loading 2107.10996v1…