PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning with Fair Averaging

Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Chenglu Wen, Cheng Wang, Rongshan Yu

arXiv:2104.14937v5cs.LG

TL;DR

Federated learning can be unfair when conflicting client gradients have substantially different magnitudes, motivating a method that mitigates conflicts before averaging. FedFV detects conflicts with cosine similarity and iteratively modifies gradient directions and magnitudes; the paper reports favorable fairness, accuracy, and efficiency comparisons, with convergence guarantees under stated assumptions.

  • Problem

    Federated learning may produce imbalanced client accuracy because conflicting gradients with large magnitude differences can make updates unfair.

  • Method

    FedFV detects gradient conflicts using cosine similarity and iteratively modifies gradient directions and magnitudes before averaging.

  • Results

    FedFV compares favorably against state-of-the-art methods in fairness, accuracy, and efficiency across a suite of federated datasets.

  • Takeaways & Limitations

    FedFV provides a conflict-mitigation approach for fair federated averaging and is supported by theoretical convergence results in the analyzed settings.

  • Takeaways & Limitations

    The convergence analysis assumes differentiable, L-smooth, convex objectives, and the paper identifies complete analysis of complex external conflicts as future work.

Abstract

from arXiv · show

Fairness has emerged as a critical problem in federated learning (FL). In this work, we identify a cause of unfairness in FL -- conflicting gradients with large differences in the magnitudes. To address this issue, we propose the federated fair averaging (FedFV) algorithm to mitigate potential conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient conflicts, and then iteratively eliminate such conflicts by modifying both the direction and the magnitude of the gradients. We further show the theoretical foundation of FedFV to mitigate the issue conflicting gradients and converge to Pareto stationary solutions. Extensive experiments on a suite of federated datasets confirm that FedFV compares favorably against state-of-the-art methods in terms of fairness, accuracy and efficiency. The source code is available at https://github.com/WwZzz/easyFL.

1 Introduction

The paper links unfairness in federated learning to conflicting gradients whose magnitudes differ substantially, and proposes FedFV to mitigate these conflicts before averaging. It identifies internal and external conflict sources and reports favorable fairness, accuracy, and efficiency results against state-of-the-art methods.

  • Motivation: Conflicting gradients with large magnitude differences can make averaging favor some clients and produce unfair updates for others.The paper illustrates this with gradients whose average is dominated by the largest gradient and moves away from the global optimum.
  • Sources of unfairness: Non-IID, imbalanced client data can create internal conflicts through divergent gradients and unequal numbers of local training steps.Clients with more data may take more local steps, producing larger gradient magnitudes and potentially unfair updates among selected clients.
  • Sources of unfairness: Party selection and client dropout create external conflicts because selected clients may dominate updates while absent clients’ gradients are unavailable or effectively zero.Repeatedly selecting particular client types can make the round’s gradient distribution unrepresentative, while dropout can produce unfair global updates.
  • Contribution: FedFV mitigates client conflicts before gradient averaging, complementing existing fair federated learning systems.Its reported evaluation covers fairness, accuracy, and efficiency across a suite of federated datasets.
  • Contribution: FedFV identifies internal and external gradient conflicts as major causes of unfairness in federated learning.The paper presents these two conflict types as central to its fairness analysis.

2 Preliminaries

The preliminaries formulate federated learning as minimizing a weighted average of client objectives while seeking a fair accuracy distribution. FedAvg aggregates sampled clients’ local updates efficiently, but can produce imbalanced accuracy across clients.

  • Federated learning objective: Federated learning minimizes a weighted average of client objectives, with each local objective defined from empirical risk on that client’s dataset.Client weights are nonnegative and sum to one.
  • FedAvg: FedAvg samples a subset of clients, receives their local updates, and aggregates them into an average update each communication round.The method is described as preserving privacy while reducing communication costs.
  • Fairness objective: FedAvg can minimize the global objective efficiently, yet its client accuracy distribution may remain imbalanced.The paper therefore distinguishes global optimization from achieving fairness across clients.
  • Fairness objective: The paper aims to minimize the federated objective while achieving a fair accuracy distribution among clients.

3 Proposed Approach

FedFV detects and mitigates internal and external gradient conflicts before averaging, using loss- and history-based projection procedures. Its analysis bounds conflicts and establishes convergence to Pareto stationary points or optima under stated convexity and smoothness assumptions.

  • Conflict definition: FedFV defines a gradient conflict when two client gradients have a negative inner product.
  • FedFV pipeline: In each round, FedFV records recent gradients, orders selected updates by client loss, and sequentially mitigates internal and external conflicts.The ordering determines which gradients serve as projection targets.
  • Mitigating Internal Conflicts: For internal conflicts, FedFV projects gradients onto normal planes in a loss-based order and lets αm high-loss clients retain their original gradients.The ordering places larger-loss clients later as projection targets to improve performance on less well-trained data; α controls conflict mitigation and α = 1 recovers FedAvg.
  • Mitigate External Conflicts: For external conflicts, FedFV estimates absent clients’ gradients from recent history and projects the combined update against conflicting historical gradients.It processes prior rounds in time order and rescales the update after projection because projection can enlarge its length.
  • Theoretical analysis: FedFV can bound the maximum conflict by selecting projection parameters, with Theorem 2 giving an upper bound for the projected average.The method applies detected-conflict mitigation in practice and uses α to control how strongly conflicts are addressed.
  • Theoretical analysis: Under differentiable, L-smooth, convex objectives and stated stepsize or cosine conditions, FedFV converges to a Pareto stationary point or the optimum.The convergence claims are presented for two-user and m-objective settings.

4 Related Work

Prior federated-learning work addresses fairness through robustness objectives, resource allocation, personalization, and gradient projection. The paper situates its contribution alongside these approaches and reports convergence to Pareto stationary solutions.

  • Fairness in federated learning: Fairness methods in FL include worst-case protection, fair resource allocation, client-specific personalization, and uniform accuracy objectives.AFL targets good-intent fairness but is limited to small networks; other methods balance performance and fairness or personalize client models.
  • Gradient projection: Gradient projection methods from continual learning mitigate harmful updates by projecting current gradients relative to previously learned tasks.Prior approaches use quadratic programming or projections onto planes defined by earlier-task gradients.
  • FedFV: FedFV differs from these studies by connecting fairness to conflicting gradients with large magnitude differences and proving convergence to Pareto stationary solutions.The paper presents this connection as addressing both internal and external gradient conflicts.

5 Experiments

The experiments evaluate FedFV on three federated image datasets under non-IID client constructions and compare it with FedAvg and fairness-oriented baselines. Training settings use one local epoch, selected batch sizes and learning rates, and each method’s best hyper-parameter performance.

  • Datasets: FedFV is evaluated on CIFAR-10, Fashion MNIST, and MNIST using non-IID federated datasets.CIFAR-10 uses 100 clients with two randomly selected shards each; MNIST uses 200 clients and analogous preprocessing.
  • Baselines: Comparisons include FedAvg, AFL, q-FedAvg, FedFa, and FedMGDA+ across the federated datasets.AFL is compared only on Fashion MNIST because it is suitable for small networks with dozens of clients.
  • Training and implementation: All experiments use local epoch E = 1, dataset-specific batch sizes, and learning rate η ∈ {0.01, 0.1}, with each method tuned to its best performance.The implementation runs on a 64g-MEM Ubuntu server with 40 CPUs and four 2080Ti GPUs.
  • Evaluation protocol: CIFAR-10 experiments report average, variance, worst 5%, and best 5% client test accuracy over 2000 rounds and 5 random seeds.The table uses full-batch training, learning rate η = 0.1, and local epochs E = 1.

5.2 Experimental Results

FedFV improves the fairness–accuracy trade-off across client evaluations and converges faster than competing methods. Its loss-based projection order further reduces variance, while mitigating both internal and external conflicts yields the strongest accuracy and efficiency gains.

  • CIFAR-10 fairness: FedFV achieves a 50.42% mean accuracy at variance 9.59, compared with variance 9.72 for q-FedAvg|q=5.0.FedFV also has higher worst-5% performance than the other experiments, whereas q-FedAvg sacrifices best-client accuracy from 69.28% to 63.60%.
  • Fashion-MNIST fairness: FedFV has variance as low as 1.77 while outperforming AFL|ηλ=0.5 on all evaluated clothing classes and mean accuracy.The comparison excludes AFL from some broader settings because AFL is designed for small client populations.
  • Overall finding: Across Tables 2 and 3, FedFV eases the negative impact of conflicting gradients with different magnitudes, producing fairer and more accurate models with lower variance.The result is observed under both sampled-client and no-sample-bias settings.
  • Accuracy and efficiency: FedFV outperforms state-of-the-art methods by up to 7.2% on CIFAR-10 and 78% on MNIST while using fewer communication rounds and retaining low variance.Mitigating internal conflicts alone converges faster with lower variance; mitigating both internal and external conflicts provides further accuracy and efficiency improvement.
  • Effects of projecting order: Projecting gradients in FedFV’s loss-based order produces the lowest variance, while random order is fairer than the reverse order.The order experiment uses α = 0 and τ = 0, with 20% of CIFAR-10 clients sampled per round and all Fashion-MNIST clients selected.

6 Conclusions and Future Work

The paper identifies conflicting gradients with large magnitude differences as a source of unfairness and proposes FedFV to mitigate them before gradient averaging. Theory and experiments support favorable fairness, accuracy, and efficiency, while future work targets a complete analysis of complex external conflicts.

  • Conclusions: FedFV mitigates potential client conflicts before averaging gradients and is supported by convergence results toward Pareto stationary solutions or convex optima.The algorithm addresses conflicting gradients with large magnitude differences, identified as bringing unfairness in FL.
  • Future work: Experiments across federated datasets find that FedFV compares favorably with state-of-the-art methods in fairness, accuracy, and efficiency.The stated future direction is a complete theoretical analysis of mitigating complex external conflicts.

A.1 Proof of Theorem 1

The proof projects each gradient onto other gradients’ normal planes in increasing order, eliminating potential conflicts before averaging. Later projection targets yield tighter bounds on the final average gradient’s conflicts.

  • Projection procedure: FedFV projects each gradient onto g_k’s normal plane in increasing k order, skipping the gradient currently being projected.The update rules remove conflicts sequentially across the projection order.
  • Conflict elimination: After updating, each projected gradient no longer conflicts with the gradients used as projection targets.The proof then analyzes the final average gradient against gradients in reverse order.
  • Conflict analysis: The angle φ_i,j denotes the angle between the relevant projected gradients used in the conflict analysis.The proof uses these angles to express inner products and conflict bounds.
  • General case: The proof applies the same calculation to any gradient g_k by removing its corresponding projected component from the average.This generalizes the argument beyond the last and penultimate projection targets.
  • Bound: Later projection targets produce smaller upper bounds on conflicts between the final average gradient and the target gradient.Theorem 1 therefore links projection order to the residual conflict bound.

A.2 Proof of Theorem 2

The proof bounds the conflict between any client gradient and FedFV’s final average after iterative projection updates. The resulting maximum conflict is bounded under the theorem’s projection conditions.

  • Conflict bound: The proof expresses the conflict between any client gradient g_k and the final average gradient ḡ′ using the bound established in Equation (5).This transfers the earlier projection analysis into a general maximum-conflict bound.
  • Update analysis: FedFV’s update rules provide the inequalities needed to bound the remaining conflict for each client gradient.The argument combines the projection updates with the preceding theorem’s bound.
  • Conclusion: The maximum gradient conflict is bounded by the theorem’s resulting expression.The supplied passage states the conclusion of the bound but does not preserve the complete expression.

A.3 Proof of the Convergence

The convergence proof analyzes nonconflicting and conflicting two-gradient updates under smoothness assumptions. It shows objective decrease under the step-size condition and identifies optimal or Pareto-stationary outcomes.

  • Nonconflicting updates: When two gradients do not conflict, FedFV reduces to FedAvg’s average update, which strictly decreases the objective for η ≤ 1/L.This case applies when g_1 · g_2 ≥ 0.
  • Conflicting updates: When gradients conflict, FedFV modifies the update using a projection-based correction before applying it to the parameters.The proof substitutes this corrected update into the smoothness inequality.
  • Decrease condition: Using g_1 · g_2 = ||g_1||||g_2|| cos φ_12, the proof rewrites the decrease condition in terms of gradient magnitudes and their angle.This reformulation isolates the effect of gradient conflict on objective decrease.
  • Step-size condition: For η ≤ 1/L, the smoothness terms preserve objective decrease under the derived inequality.The proof uses η − Lη^2 ≥ 0 to establish the required sign.
  • Convergence outcome: The objective reaches an optimum or a Pareto-stationary point when the relevant gradient or corrected average vanishes.The two-gradient proof handles exact opposition through a convex combination whose weighted gradients sum to zero.
  • Multiple objectives: For multiple objectives, repeated FedFV updates decrease the average objective unless the average gradient vanishes or conflicting gradients admit a zero convex combination.The multi-gradient argument assumes differentiable, L-smooth, convex objectives and uses the smoothness inequality.
Loading 2104.14937v5…