PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning: A Signal Processing Perspective

Tomer Gafni, Nir Shlezinger, Kobi Cohen, Yonina C. Eldar, H. Vincent Poor

arXiv:2103.17150v2eess.SPcs.LG

TL;DR

Federated learning addresses how to train models from privacy-sensitive data distributed across edge devices without explicitly exchanging that data. The paper develops a signal processing perspective and surveys methods for communication, aggregation, privacy, and heterogeneity challenges. It provides a systematic framework and design guidelines for scaling federated learning with signal processing and communication tools.

  • Problem

    Federated learning must train across distributed, privacy-sensitive edge data while handling communication bottlenecks, heterogeneous users and data, privacy, and malicious participants.

  • Method

    The paper formulates federated learning as a three-stage signal processing flow and surveys methods for model distribution, local training, global aggregation, and their communication design.

  • Results

    The paper presents a unified survey framework and design guidelines covering candidate signal processing and communication approaches for large-scale federated learning.

  • Takeaways & Limitations

    Signal processing and communication methods provide a basis for adapting federated learning to its distributed operation, communication constraints, privacy needs, and diverse users.

Abstract

from arXiv · show

The dramatic success of deep learning is largely due to the availability of data. Data samples are often acquired on edge devices, such as smart phones, vehicles and sensors, and in some cases cannot be shared due to privacy considerations. Federated learning is an emerging machine learning paradigm for training models across multiple edge devices holding local datasets, without explicitly exchanging the data. Learning in a federated manner differs from conventional centralized machine learning, and poses several core unique challenges and requirements, which are closely related to classical problems studied in the areas of signal processing and communications. Consequently, dedicated schemes derived from these areas are expected to play an important role in the success of federated learning and the transition of deep learning from the domain of centralized servers to mobile edge devices. In this article, we provide a unified systematic framework for federated learning in a manner that encapsulates and highlights the main challenges that are natural to treat using signal processing tools. We present a formulation for the federated learning paradigm from a signal processing perspective, and survey a set of candidate approaches for tackling its unique challenges. We further provide guidelines for the design and adaptation of signal processing and communication methods to facilitate federated learning at large scale.

I. INTRODUCTION

Federated learning trains a centralized model from data distributed across edge devices without sharing the data itself. The paper frames its communication, heterogeneity, privacy, and optimization challenges through signal processing and communication methods.

  • Motivation: The paradigm addresses edge-generated data that may be difficult or inappropriate to upload centrally because of privacy considerations.Relevant edge devices include mobile phones, sensors, vehicles, and medical devices.
  • Federated learning setup: Federated learning distributes training across edge users while a server coordinates a shared model without receiving their local datasets.Users train locally and exchange model parameters with the server instead of explicitly sharing private data.
  • Challenges: Federated learning introduces communication bottlenecks, statistical and system heterogeneity, privacy and security concerns, and additional optimization design choices.These include selecting devices and local steps, tuning aggregation, and configuring compression algorithms.
  • Signal processing perspective: The article surveys signal processing approaches for the three-stage flow of model distribution, local training, and global aggregation.It emphasizes adapting methods to federated learning’s distributed operation and diverse users.
  • Federated learning flow: The federated learning procedure combines edge users with local datasets, local objectives, model training, server aggregation, and broadcast synchronization.User datasets may come from different, non-i.i.d. distributions, motivating collaboration among many users.

B. Federated Averaging

Federated Averaging uses local stochastic-gradient updates followed by weighted server aggregation and redistribution of the resulting global model. The procedure can vary how often synchronization occurs between local training periods.

  • Aggregation: FedAvg implements the aggregation mapping as weighted averaging of participating users’ models.The weights are the user-specific values defined by the federated objective.
  • Local training: Each participating user performs E local SGD iterations with mini-batches before transmitting its updated model to the server.The local updates use a learning rate and stochastic gradients evaluated on randomly sampled mini-batches.
  • Global synchronization: At synchronization indices, the server aggregates received local models into a new global model and broadcasts it back to users.Users then synchronize their local models with the broadcast global parameters.
  • Synchronization schedule: The synchronization schedule ranges from aggregation after every iteration to one-shot averaging only at the end of training.These cases correspond to E = 1 and E = T, respectively.

C. Challenges

Federated learning differs from centralized training because model updates must be repeatedly exchanged across diverse devices and data distributions. These conditions create communication, statistical, system, privacy, and security challenges.

  • Communication Bottleneck: Repeated exchange of high-dimensional model updates over rate-limited channels creates a communication bottleneck that can dominate federated training time.This is especially relevant for large DNNs and shared, resource-limited wireless networks.
  • Statistical Heterogeneity: Statistical heterogeneity arises because users’ local data distributions vary, so independently trained local models may be biased before aggregation.Personalized data at different devices is a typical source of this heterogeneity.
  • System Heterogeneity: System heterogeneity reflects differences in device behavior, availability, storage, computation, energy, and network conditions.Participation may depend on whether devices are idle, charging, or connected to Wi-Fi.
  • Privacy and Security: Federated learning offers privacy advantages by exchanging model updates rather than raw data, but privacy and security remain explicit concerns.The system must account for information leakage and unreliable or malicious users.

Convergence of Local SGD

Under standard smoothness, convexity, variance, and participation assumptions, local SGD with FedAvg reaches the global optimum at the same asymptotic rate as centralized mini-batch SGD. The convergence guarantee depends on synchronization and learning-rate choices, while federated operation remains exposed to privacy and malicious-user risks.

  • Convergence guarantee: O(1/T) is the FedAvg convergence rate under the stated smoothness, strong-convexity, bounded-gradient, and variance assumptions.This matches the asymptotic rate of centralized mini-batch SGD.
  • Synchronization trade-off: The number of local SGD iterations E controls the trade-off between communication cost and divergence toward user-specific minimizers.Small E can increase communication, whereas large E can move local models toward minimizing their own local objectives.
  • Convergence conditions: FedAvg requires diminishing learning rates for convergence in the described analysis.This requirement is part of the stated convergence conditions.
  • Security boundaries: Model updates can leak information about local data, and malicious users can send tainted updates that affect the trained model.These risks remain relevant alongside the optimization convergence result.

D. Signal Processing Stages in Federated Learning

Federated learning can be organized into model distribution, local training, and global aggregation. Global aggregation comprises processing and encoding local updates, uplink transmission, and server-side combining, each presenting a signal-processing or communication task.

  • The federated learning procedure has three main steps: model distribution, local training, and global aggregation.
  • Global aggregation: Global aggregation processes and encodes local training outcomes before transmitting them over shared channels and combining received signals at the server.
  • Local updates processing and encoding: Local processing maps model updates into transmitted representations, supporting compression, reliability, and privacy over rate-limited shared links.
  • Uplink transmission: Uplink transmission must allocate bandwidth and time, select participating users, and can exploit over-the-air computation on shared wireless media.
  • Global combining: Server-side combining estimates a global model from channel outputs and can compensate for channel effects, statistical heterogeneity, and malicious-device effects.

III. LOCAL UPDATES PROCESSING AND ENCODING

Local update processing treats federated learning as learning-aware distributed source coding: updates must be compressed and jointly decoded for aggregation despite unknown statistics and changing participation. Quantization and sparsification reduce uplink costs, while dithered quantization supports accurate learning without requiring update distributions.

  • Local encoding maps each user’s model update into a representation jointly decoded for aggregation, because the server needs the models rather than encoded symbols.
  • Compression reduces uplink communication cost, addressing a central federated-learning bottleneck while introducing possible distortion in the optimization task.
  • Federated compression differs from conventional source coding because update statistics are not unified, encoding occurs within global aggregation, and decoding must recover aggregate models.
  • Sparsification preserves selected update entries, whereas quantization maps parameters to finite-bit representations such as one-bit signs.
  • Dithered quantization enables accurate federated learning without knowing the update distribution because its distortion is statistically uncorrelated with the updates.

B. Privacy Preservation

Federated learning requires privacy-preserving update encoding because model updates can reveal local data, while encryption and artificial-noise approaches impose different communication, key-management, and robustness trade-offs. Subtractive dithering with lattice quantization preserves convergence behavior and can improve accuracy in non-convex DNN training.

  • Model updates may reveal the data used to compute them, motivating privacy against servers and malicious users.
  • Privacy-preserving encoding: Encryption protects updates but requires mechanisms that support aggregation, and encrypted representations generally increase communication overhead.
  • Dithered quantization: Subtractive dithering with multivariate lattice quantization retains O(1/T) asymptotic convergence and improves learned-model accuracy for non-convex DNN losses.
  • Privacy-preserving encoding: Artificial noise obscures updates without key sharing and can provide privacy guarantees such as differential privacy.
  • Design considerations: Privacy-preserving encoders must remain robust across repeated rounds with changing participation and user dropout.

IV. UPLINK TRANSMISSION

Uplink transmission maps local model updates into channel inputs and conveys them to the server, where communication design must account for learning rather than reliability alone. Conventional orthogonal transmission is reliable but can create substantial delay as participating users increase.

  • Conventional Transmission: Conventional communication separates learning from transmission and assigns orthogonal resource blocks to users, enabling reliable recovery of model updates.OFDM-based orthogonalization gives users separate channels, with code rates no larger than individual channel capacities.
  • Conventional Transmission: Conventional transmission delay can dominate local optimization time because limited wireless bandwidth and energy reduce each user’s achievable rate as participation grows.This delay can affect convergence of the learned global model.
  • IV. UPLINK TRANSMISSION: Uplink transmission encodes updated local models into channel inputs and sends them from devices to the server.The transmission stage includes forming channel inputs through modulation and coding, alongside resource allocation.
  • Learning-Aware Transmission: Learning-aware transmission treats resource allocation as part of the learning problem because the server needs useful model updates, not merely decoded messages.The section motivates allocating channels and transmission resources with the learning procedure in mind.

A. Learning-Aware Resource Allocation

Learning-aware resource allocation jointly considers which users participate and how channel resources are divided. This differs from conventional wireless allocation because partial participation is desirable even when resources are available.

  • Learning-Aware Resource Allocation: Resource division should account for the overall learning procedure, since equal allocation can limit the convergence rate of the global model.The uplink is modeled as resource blocks representing bandwidth or temporal transmission slots.
  • Design Considerations: Federated learning typically separates user selection from resource management because limiting participation is desirable for communication, computation, and load balancing.This contrasts with conventional wireless resource allocation, where selection is more directly tied to resource availability.
  • Learning-Aware Resource Allocation: Resource allocation in federated learning combines user selection with channel-resource management to support rapid and reliable delivery of fixed-length updates.The relevant objective is maximal transmission delay rather than conventional achievable sum-rate.
  • User Selection: User-selection policies include deterministic scheduling, random selection of K ≤ N users, and decentralized opportunistic access.Deterministic scheduling can group computationally weak devices, while uniform random selection samples subsets independently across rounds.

B. Over-The-Air Federated Learning

Over-the-air federated learning reuses shared wireless resources so that channel superposition performs aggregation while users transmit model updates. The approach can improve throughput and convergence delay, but requires synchronization, channel knowledge, and compatible aggregation mappings.

  • Over-The-Air Federated Learning: Full resource reuse lets users transmit simultaneously over shared channels, exploiting interference as an over-the-air computation of model-update aggregates.All temporal and spectral resources can be reused rather than orthogonally divided among users.
  • Design Considerations: Over-the-air methods can increase throughput and reduce transmission delay, but analog signaling requires accurate synchronization and channel input-output knowledge.These requirements are needed to transform shared-channel combining into the desired aggregation mapping.
  • Design Considerations: Over-the-air aggregation is naturally compatible with nomographic mappings such as FedAvg, but this limits direct combination with non-nomographic rules such as trimmed mean.Transmission and server post-processing must also mitigate channel noise for training to continue effectively.
  • Over-The-Air Functional Computation: Over-the-air computation is attractive because federated updates are typically transmitted for averaging, allowing analog superposition to produce a weighted average without decoding discrete symbols.The server processes the channel output as an aggregation of transmitted updates.
  • Over-The-Air Federated Learning: Dedicated precoding and related compressed-sensing or multi-user MIMO methods can improve accuracy and convergence delay under fading, heterogeneous data, and multiple local steps.Dedicated precoding can guarantee convergence for heterogeneous datasets with E > 1 local iterations.

Delay Minimization with Probabilistic User Selection

The section frames wireless global aggregation as a statistical estimation problem: the server uses noisy, attenuated, compressed, or privacy-amplified observations to form an accurate global model. Conventional combining can be limited by heterogeneous data distributions and unreliable participants.

  • Global Combining: The server processes received signals to combine local updates into a global model whose accuracy is evaluated through the global objective.This is the final processing and combining stage of global aggregation.
  • Conventional Combining: FedAvg forms the global model as a weighted average of participating users’ local updates, with weights typically determined by dataset sizes.The weights may not reflect heterogeneous data distributions or the differing contributions of instantaneous updates.
  • Conventional Combining: Conventional combining can produce models that poorly infer samples from local distributions and can be arbitrarily skewed by a single Byzantine-faulty user.The first issue follows from optimizing against an averaged local-data distribution, while the second arises under attacks such as poisoning.
  • Signal Processing Perspective: Received channel outputs are statistically related to local updates through noise, attenuation, compression, and privacy amplification, making global combining a statistical estimation task.The server estimates a global model from channel observations rather than directly accessing clean updates.
  • Signal Processing Perspective: Reliable statistical estimation is difficult because the server rarely has a reliable characterization of the relationship between observations and the desired global model.This limits direct application of estimation methods that depend on prior knowledge of the joint distribution.

Time-Varying Precoding Over-the-Air Federated Learning

Over-the-air federated learning exploits shared-channel superposition to average model updates, while time-varying precoding makes channel-noise effects decay during training. COTAF therefore retains the asymptotic convergence behavior of noise-free orthogonal transmission while using shared channel resources.

  • Over-the-Air Aggregation: Shared-channel transmission inherently averages users’ analog model updates, enabling over-the-air federated learning with local SGD.The channel inputs are precoded versions of the model updates, and the shared channel performs the summation.
  • Time-Varying Precoding: COTAF uses a time-varying precoding parameter whose setting accounts for the decreasing magnitude of model updates during training.The precoder is set to α_t = P / max_i E[...], so the scaling changes across iterations.
  • Time-Varying Precoding: The resulting global model contains the desired federated averaging term plus an additive noise contribution that gradually decays with iteration t.This is the mechanism by which COTAF limits the effect of channel noise during optimization.
  • Convergence and Throughput: O(1/T) convergence is achieved asymptotically, matching noise-free orthogonal transmissions while reusing the full channel and improving throughput.The result is stated for strongly-convex smooth objectives with bounded gradients.
  • Federated Combining: Federated combining methods either produce one corrected global model or maintain multiple models whose inference rules are combined.The alternatives address channel distortion, noise, and statistical heterogeneity through different aggregation strategies.
  • Federated Combining: Mixture-of-models approaches reduce heterogeneity effects but require separate models for users or clusters, increasing the resulting global-model volume.Learning adaptive combining weights for a single model can likewise require additional communication.

B. Security-Enhanced Federated Combining

Byzantine-robust federated combining addresses unreliable or malicious updates that can bias the learned model and degrade convergence. Robust methods replace ordinary averaging with aggregation rules designed to limit abnormal contributions while accounting for heterogeneity and communication constraints.

  • Security Threats: Unreliable or malicious user updates can significantly deviate from normal values, biasing the learned model and severely degrading convergence.This motivates Byzantine-robust learning in federated systems.
  • Design Constraints: Byzantine-robust combining should preserve optimization performance without requiring significant additional communication between users and the server.The design question is therefore not only fault suppression but also the accuracy attainable under robustness constraints.
  • Design Constraints: Robust aggregation must distinguish malicious outliers from legitimate diversity caused by heterogeneous data, unequal dataset sizes, and device conditions.Diversity that signals abnormality in conventional robust optimization may be expected even when federated users are reliable.
  • Security Threats: Byzantine users may falsify dataset sizes, corrupting both their own contribution and the server’s aggregation weights.This attack exploits the sample-count weighting used by conventional federated aggregation.
  • Robust Aggregation: Median aggregation remains close to at least half of the considered updates and does not require knowing the fraction of unreliable users.Geometric, marginal, and mean-around-median variants extend the robust median idea to multidimensional updates.
  • Robust Aggregation: Krum selects one update near neighboring updates, whereas truncation discards abnormal updates before applying FedAvg to the remainder.Both replace ordinary averaging with non-affine operations that reduce sensitivity to extreme values.

VI. FUTURE RESEARCH DIRECTIONS

The paper identifies future directions for signal-processing support of federated learning, especially learning-aware encoding, distributed compression, and privacy-preserving distortion. It also considers adapting encoding across heterogeneous devices and communication rounds.

  • Local Updates Encoding: Encoding local model updates can reduce communication burden through quantization and compression while supporting privacy through encryption and differential privacy.These operations are applied before transmission from edge devices to the server.
  • Local Updates Encoding: Future compression schemes should adapt quantization resolution to device technology and communication conditions rather than imposing one encoding mechanism on every user.Different resolutions may translate into different communication delays across devices.
  • Local Updates Encoding: Compression mappings could change across communication rounds to minimize total communication burden as learning progresses.The paper also points to distributed source coding when users’ model updates are statistically correlated.
  • Privacy-Preserving Encoding: Lossy compression may itself contribute to privacy preservation because its distortion can reduce the artificial noise needed for differential privacy.Quantization also discretizes updates, which can facilitate encryption methods that assume integer quantities.

B. Uplink Transmission

The paper identifies communication, heterogeneity, and topology as central design issues in federated learning, and outlines signal-processing directions for addressing them. It highlights adaptive transmission, robust combining, joint stage optimization, hierarchical networking, and data selection as open avenues.

  • Uplink Transmission: Shared noisy channels make uplink transmission a bottleneck because repeated high-dimensional model exchanges can delay convergence and reduce accuracy.Proposed responses include orthogonal transmission, prioritizing users with greater expected learning contribution, and over-the-air computation.
  • Uplink Transmission: User selection and resource allocation should account for device delays and statistical heterogeneity when choosing diverse or homogeneous participants.The paper notes that current prioritization methods do not capture the broad range of device technologies or all relevant distributional information.
  • Uplink Transmission: Dynamic spectrum access could integrate orthogonal and over-the-air transmissions while adapting to users experiencing deep fading across time and frequency.The paper suggests modeling fading as a known or unknown Markovian process to develop rigorous channel-allocation performance measures.
  • Uplink Transmission: Federated learning over hierarchical multi-hop networks requires dedicated treatment of channel division, intermediate aggregation, model caching, and hierarchical over-the-air learning.Most existing algorithms assume a star topology in which users communicate directly with the server.
  • Global Combining: Nonlinear global combining and cluster-specific Byzantine-robust aggregation are promising responses to transmission distortions, unreliable users, malicious users, and heterogeneous distributions.Cluster-based aggregation divides users by distribution and applies robust aggregation separately within homogeneous clusters.
  • Additional Directions: Jointly optimizing encoding, transmission, and combining may reduce model-update delay and improve convergence speed beyond the paper’s stage-wise framework.Other directions include downlink privacy and overload reduction, informative-data selection, and collaborative or personalized inference using multiple learned models.
Loading 2103.17150v2…