PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Federated Learning with Noisy Communication

Fan Ang, Li Chen, Nan Zhao, Yunfei Chen, Weidong Wang, F. Richard Yu

arXiv:1911.00251v1cs.LGstat.ML

TL;DR

Noise in wireless communication affects federated learning’s local updates and model aggregation. The paper develops robust training designs for two noise models, with simulations reporting improved prediction accuracy and loss-function performance.

  • Problem

    Imperfect wireless estimation affects federated learning’s local updates, making training difficult to fit the input data.

    Source: §II · p. 3

  • Method

    The paper formulates robust federated training under expectation-based and worst-case noise models, using regularized loss approximation and sampling-based successive convex approximation.

    Source: §I · p. 2

  • Results

    Simulations show improved prediction accuracy and loss-function performance for both proposed training processes, while the expectation-based design converges at the rate of centralized training that ignores noise.

    Source: §I · p. 2, §VII · p. 12

  • Takeaways & Limitations

    The proposed designs provide analyzed convergence properties and improve prediction accuracy and loss-function values in the reported simulations.

    Source: §VII · p. 12

Abstract

from arXiv · show

Federated learning is a communication-efficient training process that alternates between local training at the edge devices and averaging the updated local model at the central server. Nevertheless, it is impractical to achieve a perfect acquisition of the local models in wireless communication due to noise, which also brings serious effects on federated learning. To tackle this challenge, we propose a robust design for federated learning to alleviate the effects of noise in this paper. Considering noise in the two aforementioned steps, we first formulate the training problem as a parallel optimization for each node under the expectation-based model and the worst-case model. Due to the non-convexity of the problem, a regularization for the loss function approximation method is proposed to make it tractable. Regarding the worst-case model, we develop a feasible training scheme which utilizes the sampling-based successive convex approximation algorithm to tackle the unavailable maxima or minima noise condition and the non-convex issue of the objective function. Furthermore, the convergence rates of both new designs are analyzed from a theoretical point of view. Finally, the improvement of prediction accuracy and the reduction of loss function are demonstrated via simulations for the proposed designs.

I. INTRODUCTION

Wireless noise challenges federated learning, while prior work on noise reduction focused on centralized learning. The paper proposes robust training under expectation-based and worst-case noise models and describes the wireless exchange between edge nodes and the server.

I. INTRODUCTION

  • Earlier federated-learning work assumed perfect signal detection, while noise reduction had not been studied for federated learning.The paper notes that imperfect channel estimation, feedback quantization, and signal-acquisition delay make noise unavoidable in wireless communication.Source: §I · pp. 1–2
  • The paper proposes robust federated-learning designs under expectation-based noise statistics and worst-case noise uncertainty sets, with convergence analysis for both.The expectation-based model represents statistical noise properties; the worst-case model uses fixed uncertainty sets.Source: §I · p. 2
  • The expectation-based design formulates parallel optimization for each edge node and uses regularization for loss-function approximation to address noise and non-convexity.The passage reports superior prediction accuracy and loss-function performance to a scheme that ignores noise.Source: §I · p. 2
  • The worst-case design uses sampling and successive convex approximation to address unavailable worst noise conditions and a non-convex objective.Simulations report better prediction accuracy and loss-function values than the conventional design.Source: §I · p. 2

II. SYSTEM MODEL

  • Federated learning alternates local updates on each node with central averaging and broadcast; wireless links introduce noise during aggregation and broadcast.Each node uses its own dataset to update a local model, which is sent to the center for averaging before the current global model is broadcast back.Source: §II · p. 3

III. PROBLEM FORMULATION

The section formulates federated training under stochastic expectation-based and deterministic worst-case noise models. It sets up per-node optimization and motivates regularization to address the expectation-based objective’s non-convexity.

III. PROBLEM FORMULATION

A. Training Under Expectation-based Model

B. Training Under Worst-Case Robust Model

The worst-case formulation treats estimation errors as bounded uncertainty and poses each node’s training problem as a min-max optimization. Sampling and successive convex approximation address unavailable worst-case conditions and objective non-convexity.

B. Training Under Worst-Case Robust Model

IV. ROBUST DESIGN USING EXPECTATION-BASED MODEL

  • For the separate expectation-based design, the paper introduces an RLA algorithm to represent noise effects and find local optimal models through optimization.Source: §IV · p. 4

A. Proposed Training Algorithm

The proposed expectation-based training method approximates the noise-affected, non-convex loss with a tailored regularizer and uses distributed gradient descent. The resulting problem is described as deterministic and convex.

A. Proposed Training Algorithm

B. Convergence Analysis

The convergence analysis first equates the proposed federated process with centralized learning under the expectation-based model, then derives an O(1/t) rate. The rate worsens as noise increases, subject to stated loss-function assumptions.

B. Convergence Analysis

V. ROBUST DESIGN USING WORST-CASE MODEL

The worst-case design uses sampling-based successive convex approximation (SCA) to address noise uncertainty and a non-convex objective, then derives a convergence property for the resulting training process.

V. ROBUST DESIGN USING WORST-CASE MODEL

  • Sampling-based SCA represents noise and approximates the objective loss function to address noise uncertainty and non-convexity.The section also derives the proposed design’s convergence property.Source: §V · p. 6

A. Proposed Training Algorithm

The training algorithm tackles unavailable worst-case noise values and non-convex objectives by sampling noise and using SCA, with regularization to control the local approximation. It then alternates node updates and central aggregation.

A. Proposed Training Algorithm

  • Sampling-based SCA approximates each node’s objective and supplies updating rules for worst-case training.The algorithm combines sample-based approximation with SCA to address the robust objective.Source: A. Proposed Training Algorithm · pp. 6–8
  • Unavailable worst-case noise values and non-convex objectives make the robust training problem difficult to optimize.The paper identifies inability to estimate worst-condition noise and objective non-convexity as the two main issues.Source: A. Proposed Training Algorithm · p. 6
  • The SAA approach replaces an objective affected by an unknown distribution with an optimization based on available independent, identically distributed samples.The paper motivates sampling because only noise samples, rather than exact noise values, may be available.Source: A. Proposed Training Algorithm · p. 7
  • The sampling method uses noise realizations at each iteration and exploits the fact that worst-case noise occurs on the boundary.This provides a sampling-based way to handle the unavailable worst-condition noise.Source: A. Proposed Training Algorithm · p. 7
  • A regularizer shrinks the gap between w and the previous model wt because the local expansion is established only near wt.The local loss function is then proposed as P4.Source: A. Proposed Training Algorithm · p. 7
  • Each node minimizes a sample approximation of the unstable objective, while a second term controls the pace of each iteration.The accumulated gradient estimate is formed from samples collected over iterations, with estimation accuracy increasing as t increases when ρt is properly chosen.Source: A. Proposed Training Algorithm · p. 7
  • The algorithm uses conditional gradient descent for each node, aggregates local updates centrally, and broadcasts the global model for the next iteration.The process repeats until the specified stopping condition is met.Source: A. Proposed Training Algorithm · pp. 7–8
  • The proposed training process alternates conditional gradient descent and aggregation to obtain a global model, with performance evaluated by simulation.The described algorithm is presented as the sampling-based SCA approximation of each node’s training objective.Source: A. Proposed Training Algorithm · p. 8

B. Convergence Analysis

The convergence analysis establishes equivalence between distributed and centralized worst-case learning under stated conditions, then gives a convergence rate for the global loss. It also states a separate rate comparison for the expectation-based design.

B. Convergence Analysis

  • The analysis proves worst-case distributed learning converges by first establishing its equivalence to centralized learning and then proving centralized convergence.The paper also states that the global loss function satisfies Assumption 2.Source: B. Convergence Analysis · p. 8
  • Under Assumption 2, suppose τ > 0 and step sizes γt and ρt are chosen as stated; distributed learning then equals centralized learning at iteration t.Source: B. Convergence Analysis · p. 8
  • For centralized worst-case learning under Assumption 2 and the stated step-size condition, the global loss function converges at O(γt).The analysis assumes convex, Lipschitz, and smooth local loss functions.Source: B. Convergence Analysis · pp. 8–9
  • The expectation-based robust design is stated to converge at O(γt), compared with O(1/t) for centralized gradient descent under perfect estimation.The paper says the comparison is simulated in Section VI.Source: B. Convergence Analysis · p. 9

VI. SIMULATION RESULTS AND DISCUSSIONS

The simulations evaluate image classification on MNIST using i.i.d. data partitions across distributed nodes, measuring prediction accuracy and training loss over iterations. The expectation-based figure presents performance against iteration time.

VI. SIMULATION RESULTS AND DISCUSSIONS

  • The evaluation uses MNIST image classification with 70,000 digit examples and i.i.d. random assignment of samples to nodes.The dataset has 60,000 training and 10,000 testing examples, and SVM classification is used as the loss function.Source: §VI · p. 9
  • Prediction accuracy and training loss are measured against iteration count for comparison of the learning approaches.The baseline set includes centralized training with perfect estimation and conventional federated training with imperfect estimation.Source: §VI · p. 9
  • Figure 3 presents performance versus iteration time under the expectation-based model.The listed approaches are centralized learning, conventional federated learning, and proposed federated learning.Source: Figure 3 · p. 10, §VI · p. 10

A. Simulations Under Expectation-based Model

Under the expectation-based model, the proposed design outperforms the conventional scheme on accuracy and loss as node counts grow. Its accuracy approaches centralized learning with few nodes, while performance generally declines as data are divided among more nodes.

A. Simulations Under Expectation-based Model

B. Simulations Under Worst-case Model

Under the worst-case model, the proposed design improves accuracy and loss relative to conventional training across the reported comparisons. Its advantage grows with node count, while performance declines as nodes increase.

B. Simulations Under Worst-case Model

VII. CONCLUSIONS

The paper proposes robust federated learning designs for noise in aggregation and broadcast under expectation-based and worst-case models. It reports acceptable convergence rates and simulations with improved accuracy and loss for both designs.

VII. CONCLUSIONS

  • Noise in aggregation and broadcast is formulated as parallel optimization under expectation-based and worst-case models.The expectation-based problem is handled with SLA, and the worst-case problem with sampling-based SCA.Source: §VII · pp. 11–12
  • Both proposed designs have acceptable convergence rates and improve prediction accuracy and loss values in simulations.The conclusion attributes the simulation results to appropriate punishment during training.Source: §VII · p. 12
Loading 1911.00251v1…