PaperlayerView exact revision on arXiv

Source-linked AI summary

Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data

Jin-Hyun Ahn, Osvaldo Simeone, Joonhyuk Kang

arXiv:1907.02745v1cs.ITcs.LGmath.IT

TL;DR

Cooperative learning protocols are often studied with ideal channels, while wireless links introduce noise and communication constraints. This paper evaluates digital and over-the-air implementations of FL, FD, and HFD, including a hybrid method that adds per-label average inputs to FD. Under bandwidth- or SNR-limited links, digital and analog FD and HFD can outperform FL, while FL is best with ideal links.

  • Problem

    Prior cooperative-learning work assumes noiseless, orthogonal channels, while constrained wireless links can make FL communication cost and latency limiting.

    Source: §I · p. 1

  • Method

    The paper evaluates digital and over-the-air implementations of FL, FD, and HFD over Gaussian multiple-access channels, with HFD adding offline-shared per-label average inputs to FD.

    Source: §I · p. 2

  • Results

    Under bandwidth- or SNR-limited links, digital and analog FD and HFD can outperform FL; FL is best with ideal communication channels.

    Source: §VI · p. 10

  • Takeaways & Limitations

    FD and HFD have lower communication overhead than FL, and HFD can significantly improve FD accuracy.

    Source: §VI · p. 10

Abstract

from arXiv · show

Cooperative training methods for distributed machine learning typically assume noiseless and ideal communication channels. This work studies some of the opportunities and challenges arising from the presence of wireless communication links. We specifically consider wireless implementations of Federated Learning (FL) and Federated Distillation (FD), as well as of a novel Hybrid Federated Distillation (HFD) scheme. Both digital implementations based on separate source-channel coding and over-the-air computing implementations based on joint source-channel coding are proposed and evaluated over Gaussian multiple-access channels.

I. INTRODUCTION

Cooperative training can improve local models, but constrained wireless links bring communication costs, latency, noise, and opportunities from signal superposition. The paper introduces HFD and studies digital and over-the-air implementations of FL, FD, and HFD in a device-to-access-point setting.

I. INTRODUCTION

  • Capacity constraints can make FL communication cost and latency limiting, while wireless noise and signal superposition bring additional challenges and opportunities.Over-the-air computing can use superposition to estimate sums without separately decoding each signal.Source: §I · pp. 1–2
  • HFD adds offline-shared average input vectors per label to FD's average-logit exchange, aiming to bridge the performance gap between FD and FL.The proposed scheme applies FD regularization to average probability vectors and average inputs per label.Source: §I · p. 2
  • The paper studies FL, FD, and HFD over Gaussian multiple-access channels using separate source-channel coding or over-the-air computing.The over-the-air implementation uses joint source-channel coding.Source: §I · p. 2

A. System Set-Up

  • Each device holds local data and trains a classification model, with devices communicating over a shared uplink through an access point connected to a parameter server.Global iterations combine local training with model-information exchange.Source: A. System Set-Up · pp. 2–3

B. Communication Model

The communication model treats the shared uplink as a Gaussian multiple-access channel, with device transmissions and additive noise. The downlink from the access point to devices is assumed noiseless.

B. Communication Model

  • On the uplink, devices share a Gaussian multiple-access channel; each device transmits a T ×1 real signal, and the noise is a T ×1 vector.Source: B. Communication Model · p. 3
  • Each device has a transmission power constraint.The supplied passage does not specify the constraint's value.Source: B. Communication Model · p. 3
  • The access-point-to-device link is assumed noiseless so the analysis focuses on the more challenging multi-access uplink.Source: B. Communication Model · p. 3

C. Baseline Training Protocols

The baseline protocols range from local-only training to parameter averaging and logit-based distillation. HFD combines distillation over shared per-label average covariates and logits with local training, using a preliminary offline exchange.

C. Baseline Training Protocols

  • IL trains each device’s model on its own local dataset using SGD to minimize cross-entropy loss.The baseline protocol review assumes ideal, noiseless communication; the impact of the multi-access channel is studied separately.Source: C. Baseline Training Protocols · p. 3
  • FL has devices perform local SGD updates, exchange weight vectors with the parameter server, and initialize the next iteration from the server’s average weights.Exchanging local weight updates rather than full weight vectors is noted as a potential communication-efficiency option.Source: C. Baseline Training Protocols · p. 3
  • FD periodically averages devices’ per-label output logits at the parameter server and trains locally using both regular cross-entropy and a distillation cross-entropy term.The server returns global average logits, which devices use to form the distillation targets during local training.Source: C. Baseline Training Protocols · p. 4

III. HYBRID FEDERATED DISTILLATION

  • HFD uses per-label average covariate vectors shared in an offline phase alongside average logits, then combines distillation updates with local-data training.During distillation, the average covariates are associated with corresponding average logits; devices then perform SGD updates on their local datasets.Source: §III · p. 5

IV. WIRELESS COOPERATIVE TRAINING

The paper develops wireless implementations of FL, FD, and HFD using both separate source-channel coding and over-the-air computing with joint source-channel coding. It also defines sparsification, thresholding, and quantization functions used in the implementations.

IV. WIRELESS COOPERATIVE TRAINING

  • Wireless implementations of FL, FD, and HFD use either separate source-channel coding or over-the-air computing with joint source-channel coding.The section considers these cooperative training schemes over wireless links.Source: §IV · p. 6
  • The sparseq function retains the largest and smallest q elements and replaces retained elements with the mean of the larger-magnitude sign group.The opposite-sign elements are zeroed; the retained positive or negative elements are set to their respective mean value.Source: §IV · p. 6
  • The threshq function keeps only the q elements with largest absolute values, while Qb uniformly quantizes each nonzero element using b bits.Source: §IV · p. 6

A. Digital Transmission

The digital implementation constrains each device’s transmitted information to a bit budget derived from uplink capacity. For FL, devices sparsely compress local weight updates, accumulate quantization error, and transmit quantized values with their indices.

A. Digital Transmission

  • Each device compresses its information to no more than BD bits per global iteration, with the per-device budget set using the shared uplink capacity.The capacity expression is based on the conventional digital implementation in which devices share the uplink equally.Source: A. Digital Transmission · p. 6
  • For FL, each device applies sparse binary compression with error accumulation to its W × 1 local weight update.The accumulated quantization error is updated as part of the compression procedure.Source: A. Digital Transmission · p. 6
  • FL devices transmit b-bit quantized nonzero values together with their positions, choosing q as the largest integer satisfying B_FL ≤ BD.The transmitted value and index information together define the per-device bit count.Source: A. Digital Transmission · p. 6

2) FD and HFD:

For FD and HFD, each device sends its logit information for every label to the parameter server. The vectors are quantized and transmitted with nonzero-entry positions under the device bit budget.

2) FD and HFD:

  • Each FD or HFD device sends an L × 1 logit vector for every label at global iteration i to the parameter server.Source: 2) FD and HFD · p. 6
  • Devices quantize each label’s vector and send the resulting bits together with the positions of its nonzero entries.Source: 2) FD and HFD · p. 6
  • The number of retained entries q is the largest integer satisfying B_FD ≤ BD.Source: 2) FD and HFD · p. 7

B. Analog Transmission for Over-the-Air Computing

The analog over-the-air scheme has devices transmit simultaneously without coding, allowing the access point to recover a desired sum. For dimensionality reduction, devices project their vectors before transmission, and the parameter server estimates their sum using AMP.

B. Analog Transmission for Over-the-Air Computing

2) FD and HFD:

FD and HFD communicate each device's label-logit vector at every global iteration. The stated channel-use condition permits repetition coding with redundancy when transmitting these vectors.

2) FD and HFD:

  • At each global iteration, each device communicates an L-dimensional logit vector containing values for all labels.Source: 2) FD and HFD · p. 7
  • The assumed channel uses satisfy ρL^2 ≤ T, with ρ ≥ 1; if T is not greater than L^2, the earlier compression scheme could be used.Source: 2) FD and HFD · p. 7
  • The matrix R_ρ implements repetition coding with redundancy ρ for the logit transmission.Source: 2) FD and HFD · p. 7

V. NUMERICAL RESULTS

The MNIST experiments compare IL, FL, FD, and HFD under ideal links and wireless implementations. HFD performs strongly when communication is constrained, while FL is preferred when channel uses are sufficiently plentiful.

V. NUMERICAL RESULTS

  • The MNIST experiment allocates disjoint 1,000-sample training sets to devices and leaves only five samples per device-specific target label.Source: §V · p. 7
  • With ideal noiseless orthogonal links, protocol choice significantly affects accuracy on target labels, and HFD partially bridges the performance gap between FL and FD.Source: §V · p. 8, Table II · p. 8
  • At 0 dB, HFD outperforms FD and the other schemes unless T is sufficiently large, when FL is preferred; analog and digital implementations perform comparably in that regime.Source: Figure 3 · p. 9, §V · pp. 9–10
  • At T = 3000, FD and HFD implementations perform well at low SNR, as does analog FL, while D-FD can outperform all schemes at sufficiently high SNR.Source: Figure 4 · p. 9, §V · p. 10

VI. CONCLUSIONS

The conclusions compare wireless FL, FD, and HFD under ideal and communication-limited conditions. FD and HFD have lower communication overhead than FL, and HFD can improve accuracy over FD.

VI. CONCLUSIONS

  • Under ideal communication, FL is best; with bandwidth- or SNR-limited links, digital and analog FD and HFD can outperform it.Source: §VI · p. 10
  • FD and HFD have significantly lower communication overhead than FL.Source: §VI · p. 10
  • The proposed HFD can significantly improve the accuracy obtained via FD.Source: §VI · p. 10
Loading 1907.02745v1…