PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast-Convergent Federated Learning with Adaptive Weighting

Hongda Wu, Ping Wang

arXiv:2012.00661v2cs.LGcs.AI

TL;DR

Federated learning faces slower convergence and extra communication rounds when participating nodes hold non-IID data. FedAdp adaptively weights nodes using their gradient-based contributions at each round. Compared with FedAvg, it reduces communication rounds by up to 54.1% on MNIST and 45.4% on FashionMNIST.

  • Problem

    Non-IID data across participating federated-learning nodes slows convergence, adds communication rounds, and introduces training instability.

  • Method

    FedAdp measures node contribution from the angle between local and global gradients and adaptively assigns aggregation weights each communication round.

  • Results

    FedAdp reduces communication rounds versus FedAvg by up to 54.1% on MNIST and 45.4% on FashionMNIST.

  • Takeaways & Limitations

    Adaptive weighting can reinforce positive and suppress negative node contributions, producing faster federated-learning convergence under non-IID data.

Abstract

from arXiv · show

Federated learning (FL) enables resource-constrained edge nodes to collaboratively learn a global model under the orchestration of a central server while keeping privacy-sensitive data locally. The non-independent-and-identically-distributed (non-IID) data samples across participating nodes slow model training and impose additional communication rounds for FL to converge. In this paper, we propose Federated Adaptive Weighting (FedAdp) algorithm that aims to accelerate model convergence under the presence of nodes with non-IID dataset. We observe the implicit connection between the node contribution to the global model aggregation and data distribution on the local node through theoretical and empirical analysis. We then propose to assign different weights for updating the global model based on node contribution adaptively through each training round. The contribution of participating nodes is first measured by the angle between the local gradient vector and the global gradient vector, and then, weight is quantified by a designed non-linear mapping function subsequently. The simple yet effective strategy can reinforce positive (suppress negative) node contribution dynamically, resulting in communication round reduction drastically. Its superiority over the commonly adopted Federated Averaging (FedAvg) is verified both theoretically and experimentally. With extensive experiments performed in Pytorch and PySyft, we show that FL training with FedAdp can reduce the number of communication rounds by up to 54.1% on MNIST dataset and up to 45.4% on FashionMNIST dataset, as compared to FedAvg algorithm.

I. INTRODUCTION

Federated learning preserves local data privacy but faces communication bottlenecks and slower convergence under heterogeneous non-IID data. FedAdp addresses this by adaptively weighting nodes according to their inferred contributions to global aggregation.

  • Motivation: Non-IID data across local nodes slows federated-learning convergence, increases communication rounds, and introduces training instability.These effects make communication cost a critical bottleneck in wireless FL settings.
  • FedAdp: FedAdp assigns distinct aggregation weights adaptively at each communication round using gradient information to estimate node contribution.The strategy is designed to reinforce useful contributions and reduce expected training loss per round.
  • Analysis: Theoretical analysis derives a federated-learning convergence bound incorporating both non-IID data distributions and the weighting strategy.This analysis links node heterogeneity and aggregation weighting to convergence behavior.
  • FedAdp: FedAdp measures node contribution from gradient information and lowers the convergence bound through discriminative, round-wise aggregation weights.The contribution measure is inferred at the server side from participating nodes.
  • Relation to prior work: FedAdp differs from prior approaches by avoiding additional communication and computation burden on local nodes while operating orthogonally to some adaptive methods.The cited comparison concerns methods based on node selection, local updates, asynchronous weighting, and tier-based weighting.

III. PRELIMINARIES

Standard federated learning coordinates locally trained models through a central server, while FedAvg becomes less effective when local data distributions differ. Non-IID data and local updates can pull models toward local rather than global objectives, slowing convergence.

  • Standard Federated Learning: Federated learning minimizes a global objective by coordinating multiple devices that train local models on locally collected data.The framework uses a central server to coordinate the consensus-learning task.
  • Standard Federated Learning: Each communication round selects K≪N nodes, sends them the previous global model, and has them perform local stochastic-gradient training.Local training uses a shared configuration such as optimizer, learning rate, and batch size.
  • Standard Federated Learning: The participating nodes upload local model updates, which the central server aggregates to update the global model.The update can be represented as a model difference or model parameters.
  • Non-IID Data: Non-IID local data violates the shared-distribution condition underlying unbiased stochastic-gradient estimates and makes local objectives differ across nodes.Each node’s local empirical risk is tied to its own data distribution.
  • Non-IID Data: Multiple local updates can move models toward local optima instead of the global objective, accumulating inconsistency and requiring more communication rounds.Under non-IID data, this process can hurt convergence and even lead to divergence.
  • Empirical impact: The presence and skewness of non-IID nodes slow convergence, with stronger skewness producing slower convergence in the reported examples.The comparisons include 5 IID versus 5 IID + 5 non-IID (1), and 3 IID + 7 non-IID (2) versus 3 IID + 7 non-IID (1).

IV. FEDERATED ADAPTIVE WEIGHTING

The proposed analysis connects federated-learning loss reduction to gradient alignment, local-gradient dissimilarity, and aggregation weights. This motivates weighting nodes according to their contribution while smoothing noisy round-specific angle measurements.

  • Methodology: The methodology analyzes how gradient information and data distribution affect expected federated-learning loss decrease and convergence.Experiments examine diversity in node contributions to reducing loss each round.
  • Convergence Analysis: The analysis assumes β-Lipschitz smoothness and bounded local dissimilarity between local and global gradients.The bounded-dissimilarity condition constrains local gradient norms relative to the global gradient norm.
  • Convergence Analysis: Theorem 1 bounds the expected decrease in global loss between consecutive communication rounds when the global model is not stationary.The theorem is established under the stated smoothness and dissimilarity assumptions.
  • Convergence Analysis: The loss-decrease bound depends on learning rate, local dissimilarity, gradient correlation, and the node-weighting strategy.The gradient correlation is expressed through the normalized inner product of local and global gradients.
  • Convergence Analysis: Larger dissimilarity weakens the bound and worsens convergence, while suitable weighting can regularize divergence among local gradients.The stated intuition attributes dissimilarity to heterogeneous datasets.
  • Measurement of Node Contribution: FedAdp measures contribution through the angle between local and global gradients, assigns larger weights to higher-contribution nodes, and smooths instantaneous angles across prior rounds.Smoothing is introduced to restrain instability from round-specific angle randomness.

B. Measurement of Node Contribution

FedAdp measures each participating node’s contribution through the angle between its local and global gradients, using a smoothed angle to reduce instability. This angle reflects data heterogeneity and distinguishes positive, weak, or negative contributions to aggregation.

  • A node’s local gradient can diverge from the global gradient under highly skewed data, producing negative effects during aggregation.
  • FedAdp quantifies each node’s contribution at every global round using the angle between its local and global gradients.
  • Small angles indicate aligned gradients and positive contribution, whereas angles larger than π/2 indicate opposite directions and negative contribution.
  • The smoothed angle averages angles across previous rounds to restrain the instability of instantaneous measurements.
  • Smoothed-angle differences across nodes depend on data distribution and can quantify local data dissimilarity.
  • In the experiment, highly skewed one-class nodes became nearly orthogonal to the global gradient after 15 communication rounds, contributing little to the global model.

C. Federated Adaptive Weighting (FedAdp)

FedAdp chooses adaptive aggregation weights to lower the expected loss bound by favoring nodes with higher measured contribution. Its weighting design first maps smoothed angles through a nonlinear function, then assigns normalized weights.

  • FedAdp seeks to lower the expected loss upper bound in each communication round by assigning different node weights.
  • The strategy assigns higher aggregation weights to nodes with higher measured contribution while maintaining nonnegative weights that sum to one.
  • Adaptive weighting consists of two steps: nonlinear contribution quantification from angle information, followed by weight assignment.
  • FedAdp uses a variant of the Gompertz function as a nonlinear decreasing mapping from smoothed angle to node contribution.

1) Non-linear mapping function:

The nonlinear mapping converts smoothed gradient angles into contribution values before aggregation weights are calculated. Its parameter controls how sharply contributions differ across angles, while Softmax produces normalized weights.

  • Non-linear mapping function:: A smaller α makes the mapped contribution decrease more slowly as the smoothed angle increases.
  • Non-linear mapping function:: Increasing α amplifies the contribution gap between small- and large-angle nodes, but eventually narrows the angle range considered distinguishable.
  • Weighting:: FedAdp applies Softmax after contribution mapping to calculate normalized aggregation weights.
  • Weighting:: With equal data sizes, weights depend only on mapped contribution; otherwise, they depend on both contribution and data size.
  • Weighting:: Unlike FedAvg’s data-size weighting, FedAdp incorporates node contribution into global-model aggregation.
  • Weighting:: Softmax gives larger angles smaller weights and can reinforce or suppress contributions according to gradient alignment.
  • Weighting:: Theorem 2 states that FedAdp achieves a tighter FL-loss-decrease bound than FedAvg under its designed weights.

V. EVALUATION AND ANALYSIS

The evaluation tests FedAdp on convex and non-convex image-classification models using MNIST and FashionMNIST, measuring communication rounds needed to reach target accuracy. The experiments report reduced rounds relative to FedAvg under non-IID settings.

  • FedAdp is implemented with PyTorch and PySyft and evaluated on MNIST and FashionMNIST image-classification tasks.
  • FedAdp computes smoothed angles and aggregation weights during the global update procedure.
  • FedAdp alleviates the impact of gradient divergence from data heterogeneity, reducing FL loss per round and accelerating convergence compared with FedAvg.
  • The study uses Multinomial Logistic Regression for convex learning and CNN for non-convex learning objectives.
  • Communication rounds required to reach target testing accuracy are used as the performance metric, with targets of 95% for MNIST and 80% for FashionMNIST.

A. Data Heterogeneity

FedAdp is evaluated under extreme and general non-IID data heterogeneity using MLR and CNN models. Across these settings, it generally outperforms FedAvg, with the largest communication-round reductions reported for CNN tasks.

  • MLR Model: FedAdp always outperforms FedAvg with non-IID nodes in the MLR experiments and converges rapidly during early training.Its advantage is more prominent as the proportion of non-IID nodes increases, although the gap narrows after the simple MLR model approaches its learning capability.
  • CNN Model: FedAdp always outperforms FedAvg with non-IID nodes in the CNN experiments, particularly during the early training stage.The paper attributes the stronger early effect to more pronounced gradient divergence in initial rounds.
  • CNN Model: 54.1% fewer communication rounds is the maximum reduction for MNIST at the 1-class non-IID setting, compared with FedAvg.The corresponding MNIST reduction is up to 43.2% at the 2-class non-IID setting.
  • CNN Model: 45.4% fewer communication rounds is the maximum reduction for FashionMNIST at the 2-class non-IID setting, compared with FedAvg.The corresponding FashionMNIST reduction is up to 43.7% at the 1-class non-IID setting.
  • General Heterogeneity: FedAdp outperforms FedAvg in both general heterogeneity cases and reduces federated-learning loss more rapidly in each global round.These cases have worse convergence performance than the earlier CNN experiment because fewer nodes are IID and local dissimilarity is greater.

B. Choosing 훼

The paper selects α heuristically because larger values can accelerate early convergence by amplifying contribution differences, but excessive α can become ineffective later.

  • B. Choosing 훼: Large α can hasten initial convergence by amplifying differences between small and large node contributions.The paper links this amplification to faster convergence during the initial training stage.
  • B. Choosing 훼: Because ν is proportional to α, increasing α narrows the interval [ν, π/2] in which node contributions are distinguished.Contributions from nodes with angles in [0, ν] become indistinguishable as this boundary narrows.
  • B. Choosing 훼: Increasing α empirically leads to faster convergence under the evaluated 5 IID + 5 non-IID (1) setting with a CNN model.Figure 6 evaluates the effect of α under this data heterogeneity and model configuration.
  • B. Choosing 훼: A larger α is not always effective, especially after the initial training stage.The reported heuristic search found α = 5 to be best for the experimental setting.
  • B. Choosing 훼: FedAdp is reported to reduce communication rounds by up to 54.1% on MNIST and 45.4% on FashionMNIST compared with FedAvg.The conclusion attributes the reduction to dynamically reinforcing positive and suppressing negative node contributions.

A. Proof of Theorem 1

Theorem 1 is established by bounding the global-update norm and gradient-update inner product, then substituting the resulting bounds into the preceding inequality.

  • A. Proof of Theorem 1: The proof bounds the squared norm of the global model update using the definition of global aggregation and SGD optimization.The derivation invokes Cauchy–Schwarz for one intermediate inequality.
  • A. Proof of Theorem 1: The proof rewrites the gradient-update inner product as −ηE_i|t[⟨∇F(w(t)), ∇F_i(w(t))⟩].This expression follows from the global aggregation definition and is then further transformed.
  • A. Proof of Theorem 1: The derivation uses local-dissimilarity bounds, including an upper bound B and a lower bound A, to control intermediate terms.These bounds are tied to Assumptions 2 in the proof.
  • A. Proof of Theorem 1: Substituting the derived expression and bounds into the preceding inequality completes the proof of Theorem 1.The paper explicitly states that Theorem 1 follows after substituting A7 into A1.

B. Proof of Theorem 2

Theorem 2 compares FedAdp and FedAvg by ranking virtual-node contributions and showing that contribution-proportional weighting yields a greater expected decrease in federated-learning loss.

  • B. Proof of Theorem 2: Each participating node with D_i samples is represented by D_i virtual nodes sharing its gradient information and weight.This construction reduces unequal data sizes to unit-sample virtual nodes while preserving node-specific quantities.
  • B. Proof of Theorem 2: All virtual nodes split from one participating node share the same weight, so that node’s total weight scales with D_i.The proof defines the node weight as the sum of its D_i virtual-node weights.
  • B. Proof of Theorem 2: The contribution measure decreases monotonically with the gradient-angle-related quantity, and the generic FedAdp weight function is decreasing.The proof uses this ordering to compare weighted aggregation with FedAvg.
  • B. Proof of Theorem 2: FedAdp assigns larger aggregation weights to virtual nodes with higher contribution, while weights remain normalized and nonnegative.Contribution is ordered by the gradient-based quantity u, and the corresponding FedAdp weights follow the same order.
  • B. Proof of Theorem 2: FedAdp produces a greater decrease in federated-learning loss per global round than FedAvg under the theorem’s comparison.The proof concludes this from the greater expectation term in equation (5).
Loading 2012.00661v2…