Source-linked AI summary
Federated Learning over Wireless Fading Channels
Mohammad Mohammadi Amiri, Deniz Gunduz
TL;DR
The paper studies how federated learning can perform DSGD over bandwidth-limited fading wireless MACs with many power-constrained devices. It proposes opportunistic digital transmission and compressed analog over-the-air aggregation, finding that CA-DSGD generally converges faster and reaches higher accuracy, particularly for non-i.i.d. data, while remaining robust to imperfect CSI.
Problem
Federated learning must coordinate many devices over bandwidth-limited wireless links, while prior communication studies largely abstract away wireless physical-layer effects.
Method
The paper proposes D-DSGD with opportunistic single-device scheduling and quantized reliable transmission, plus CA-DSGD with error-accumulating sparsification and low-dimensional analog projection.
Results
CA-DSGD generally converges faster and achieves higher accuracy than D-DSGD and other considered schemes, with a larger gap for non-i.i.d. data and robustness to imperfect CSI.
Takeaways & Limitations
The results show advantages for analog over-the-air DSGD and support jointly designing learning and communication algorithms for wireless-edge machine learning.
Abstract
from arXiv · showhide
We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS). We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD). We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its gradient estimate to a finite number of bits imposed by the channel condition, and transmits these bits to the PS in a reliable manner. Next, motivated by the additive nature of the wireless MAC, we propose a novel analog communication scheme, referred to as the compressed analog DSGD (CA-DSGD), where the devices first sparsify their gradient estimates while accumulating error, and project the resultant sparse vector into a low-dimensional vector for bandwidth reduction. Numerical results show that D-DSGD outperforms other digital approaches in the literature; however, in general the proposed CA-DSGD algorithm converges faster than the D-DSGD scheme and other schemes in the literature, and reaches a higher level of accuracy. We have observed that the gap between the analog and digital schemes increases when the datasets of devices are not independent and identically distributed (i.i.d.). Furthermore, the performance of the CA-DSGD scheme is shown to be robust against imperfect channel state information (CSI) at the devices. Overall these results show clear advantages for the proposed analog over-the-air DSGD scheme, which suggests that learning and communication algorithms should be designed jointly to achieve the best end-to-end performance in machine learning applications at the wireless edge.
I. INTRODUCTION
The paper addresses federated learning over bandwidth-limited, unreliable wireless links by designing DSGD schemes that account for fading-channel constraints. It compares digital opportunistic transmission with compressed analog over-the-air aggregation, reporting stronger convergence and accuracy for CA-DSGD, especially with non-i.i.d. data.
- Motivation: Federated learning involves many devices connected to the parameter server through low-capacity and unreliable links.These wireless constraints motivate reducing communication requirements while preserving distributed training.
- Motivation: Existing FL communication studies largely assume interference-and-error-free links despite FL’s focus on mobile devices.The paper instead incorporates physical-layer properties of wireless connections.
- Digital DSGD: The paper proposes D-DSGD, which opportunistically schedules one device per iteration, quantizes its gradient, accumulates quantization error, and reliably transmits the resulting bits.The quantization level is constrained by the available channel conditions and bandwidth.
- Digital DSGD: For MNIST, D-DSGD outperforms digital schemes using QSGD or SignSGD and also outperforms orthogonal transmission by all devices.The comparison includes digital gradient-compression baselines and a scheme allocating orthogonal channel resources to participating devices.
- Analog DSGD: CA-DSGD sparsifies error-accumulated gradients, exploits shared sparsity patterns, and projects each gradient into a low-dimensional vector for bandwidth reduction.The method flexibly adjusts the transmitted gradient dimension while satisfying the average power constraint.
- Results: CA-DSGD improves upon the considered analog and digital schemes under equal average power and bandwidth resources, with larger gains for non-i.i.d. datasets.Its performance is also robust to imperfect CSI at the devices; analog gains are particularly pronounced in the low-power regime.
- Results: Over-the-air analog computation benefits from simultaneous transmission of correlated gradients and is attractive for low-power IoT sensors.The paper reports that analog performance improves only marginally with additional average power, unlike D-DSGD.
- Results: CA-DSGD can improve when each iteration uses less bandwidth and the training runs for more DSGD iterations.Increasing the number of devices improves accuracy for all schemes, although the analog diversity gain is more limited and diminishes with training duration.
II. SYSTEM MODEL
The system performs federated learning across wireless devices that compute local stochastic gradients and send them to a parameter server over a bandwidth-limited fading MAC. The PS estimates the average gradient, updates the model, and multicasts the updated parameters back over an error-free shared link.
- Each device computes a stochastic gradient from its local dataset and the current shared model parameter θt.
- The devices transmit gradient estimates to the PS over a fading MAC using s subchannels and N time slots, with s ≤ d.
- The wireless channel includes device-specific fading gains, scheduling vectors, and complex Gaussian noise.
- After receiving a noisy gradient estimate, the PS updates the model and multicasts a consistent parameter vector to all devices for the next iteration.
- The PS seeks to recover the average of local gradient estimates, treating the task as distributed lossy computation over a noisy MAC.
III. DIGITAL DSGD
The digital DSGD scheme opportunistically schedules one device per iteration, adapting gradient compression to the available channel capacity. It uses sparsification, error accumulation, and reliable coded transmission, while orthogonal multi-device scheduling performs much worse.
- D-DSGD schedules a single device opportunistically at each iteration according to channel state information.All devices compute gradients, but only the selected device transmits to the PS before the updated model is shared.
- The scheduled device transmits a quantized gradient using a bit budget determined by an upper bound on the fading-channel capacity.The capacity is modeled through parallel Gaussian channels and optimized by waterfilling.
- The capacity upper bound used for D-DSGD can be quite loose, especially when s is small.
- D-DSGD sparsifies the error-compensated gradient by retaining the highest q(t) positive and lowest q(t) negative entries.The remaining information is preserved in an accumulated error vector for later iterations.
- OD-DSGD schedules all devices with ⌊s/M⌋ subchannels each and performs much worse than D-DSGD.Dividing subchannels among devices shortens transmission blocklengths, increasing error probability or reducing rate in practice.
IV. ANALOG DSGD
Analog DSGD exploits the additive wireless MAC because the PS needs the average of local gradients. The paper develops scheduled, error-compensated, and compressed analog schemes, culminating in CA-DSGD.
- Analog DSGD is motivated by the PS’s interest in the average gradient and the MAC’s ability to sum simultaneously transmitted gradients.
- The paper generalizes over-the-air computation through entry-wise scheduled analog DSGD, or ESA-DSGD.
- Error-compensated ESA-DSGD extends ESA-DSGD by accumulating transmission errors.
- CA-DSGD is a novel analog scheme built on the paper’s earlier work.
A. ESA-DSGD
ESA-DSGD transmits gradient entries over scheduled fading links using power allocation, enabling the PS to recover an analog gradient estimate for the DSGD update. The scheme uses multiple time slots when the gradient dimension exceeds the channel bandwidth.
- ESA-DSGD transmits each device’s full gradient estimate through entry-wise scheduled analog transmission.
- The scheme uses N = ⌈d/2s⌉ time slots and exploits both real and imaginary channel components to transmit a d-dimensional gradient.The gradient is zero-padded to dimension 2sN before transmission.
- A gradient entry is transmitted when its corresponding channel gain exceeds a threshold determined by the power-allocation design.
- The PS uses the received analog signal and scaling information to form an estimated gradient, then updates θt+1 = θt −ηt ĝe(θt).
- The analysis assumes perfect CSI at the PS and relaxes comparison constraints to average power over all iterations.
B. ECESA-DSGD
ECESA-DSGD extends ESA-DSGD with error accumulation while transmitting complete gradient estimates across multiple time slots. This preserves gradient entries omitted because of unfavorable channel conditions.
- Entries not transmitted because of poor channel conditions are retained in the error accumulation vector for transmission in the next iteration.
- ECESA-DSGD incorporates error accumulation into ESA-DSGD to retain the accuracy of local gradient estimates.
- Each device transmits its entire d-dimensional gradient estimate over N = ⌈d/2s⌉ time slots after zero-padding it to dimension 2sN.
- The PS recovers the average gradient estimate and updates the parameter vector as θt+1 = θt −ηtˆgv (θt).
C. CA-DSGD
CA-DSGD compresses device gradients before analog over-the-air transmission by combining sparsification, error accumulation, and random projection. The PS uses AMP to recover the compressed average gradient for DSGD updates.
- CA-DSGD sparsifies each device’s error-compensated gradient by retaining the k entries with highest magnitudes.
- Each device projects its sparse gradient into a low-dimensional vector of length ˜s ≤ d using a shared random matrix.
- The PS applies approximate message passing with knowledge of the projection matrix and CSI to recover the compressed gradient.
- CA-DSGD updates the model with the recovered gradient when available, and otherwise retains the previous parameter vector.
- Under non-IID data, transmissions can still align on a small number of coordinates, allowing AMP to recover the average gradient with reasonable accuracy.The DSGD process converges more slowly than in the IID case.
- The projected gradients are transmitted over the wireless MAC, exploiting signal superposition to combine device contributions.
V. NUMERICAL EXPERIMENTS
Experiments evaluate wireless edge-learning schemes on MNIST under IID and non-IID data, varying transmit power, device and dataset configurations, and compression dimensions. CA-DSGD generally outperforms the alternatives, with especially large gains under non-IID data and strict bandwidth limits.
- CA-DSGD outperforms all other analog and digital schemes for IID data, with a substantially larger improvement under non-IID data.
- Under non-IID data, CA-DSGD converges more slowly than under IID data but reaches higher accuracy and converges faster than the other schemes.
- CA-DSGD benefits from wireless signal superposition and more efficient bandwidth use through linear projection, especially in the non-IID case.
- D-DSGD significantly outperforms OD-DSGD in accuracy and convergence speed, with stronger superiority at higher average transmit power.
- Increasing the CA-DSGD projection dimension ˜s degrades performance, while transmitting more sparse versions enables further iterations and higher accuracy.
- CA-DSGD and ECESA-DSGD are robust to imperfect CSI, with reported accuracy losses of 0.67% and 0.76%, respectively.
VI. CONCLUSIONS
The paper studies federated learning over a bandwidth-limited fading MAC and proposes digital and analog DSGD schemes. CA-DSGD combines gradient sparsification, error accumulation, projection, and power allocation, and numerically outperforms D-DSGD and other analog schemes, especially with non-IID data.
- Federated learning is performed by devices communicating model updates to a parameter server over a bandwidth-limited fading MAC.
- D-DSGD selects one device according to channel states, quantizes its gradient estimate, and reliably transmits the resulting bits to the parameter server.
- CA-DSGD exploits wireless-MAC superposition instead of orthogonalizing device transmissions, using sparsification with error accumulation and linear projection for bandwidth reduction.
- A power allocation scheme aligns received vectors at the parameter server while satisfying devices' average power constraints.
- CA-DSGD uses limited channel bandwidth more efficiently and benefits from a beamforming effect arising from similarity among devices' gradient patterns.
- Numerical MNIST results show that CA-DSGD consistently outperforms D-DSGD and other state-of-the-art analog schemes, with larger improvement under non-IID data.