PaperlayerView exact revision on arXiv

Source-linked AI summary

FedImp: Enhancing Federated Learning Convergence with Impurity-Based Weighting

Hai Anh Tran, Cuong Ta, Truong X. Tran

arXiv:2608.14654v1cs.LGcs.AI

TL;DR

Non-IID device data makes federated learning aggregation difficult and slows convergence. FedImp weights devices by the entropy of their local data, reducing EMNIST communication rounds by up to 64.4% versus FedAvg.

  • Problem

    Non-IID data across devices makes local-model aggregation difficult, slowing convergence and harming generalization.

  • Method

    FedImp measures each device’s data impurity with entropy and normalizes these contributions into aggregation weights.

  • Results

    Across EMNIST and CIFAR-10, FedImp converged faster than FedAvg, FedProx, and FedAdp, reducing EMNIST communication rounds by up to 64.4% versus FedAvg.

  • Takeaways & Limitations

    FedImp can reach target accuracy with fewer communication rounds, improving federated-learning efficiency under non-IID distributions.

  • Takeaways & Limitations

    FedAdp performs poorly when seven imbalanced nodes have identical data distributions, limiting its reliability in that scenario.

Abstract

from arXiv · show

Federated Learning (FL) is a collaborative paradigm that enables multiple devices to train a global model while preserving local data privacy. A major challenge in FL is the non-Independent and Identically Distributed (non-IID) nature of data across devices, which hinders training efficiency and slows convergence. To tackle this, we propose Federated Impurity Weighting (FedImp), a novel algorithm that quantifies each device contribution based on the informational content of its local data. These contributions are normalized to compute distinct aggregation weights for the global model update. Extensive experiments on EMNIST and CIFAR-10 datasets show that FedImp significantly improves convergence speed, reducing communication rounds by up to 64.4%, 27.8%, and 66.7% on EMNIST, and 44.2%, 44%, and 25.6% on CIFAR-10 compared to FedAvg, FedProx, and FedAdp, respectively. Under highly imbalanced data distributions, FedImp outperforms all baselines and achieves the highest accuracy. Overall, FedImp offers an effective solution to enhance FL efficiency in non-IID settings.

I. INTRODUCTION · II. RELATED WORK

Federated learning preserves privacy by training across decentralized devices, but non-IID data makes aggregation difficult and slows convergence. FedImp addresses this challenge by using entropy-based impurity to derive aggregation weights, alongside prior adaptive weighting, personalization, and regularization methods.

  • I. INTRODUCTION: Federated learning trains models across devices without sharing raw data, preserving privacy and reducing data transfer.Clients compute local updates from unique data, which are aggregated into a global model.
  • I. INTRODUCTION: Non-IID data across devices makes local-model aggregation harder, causing slow convergence and poor generalization.Differences can arise from factors such as location or user behavior.
  • I. INTRODUCTION: FedImp quantifies each client’s informational impurity with entropy and normalizes these contributions into distinct global-model aggregation weights.The method is introduced to address limitations associated with FedAvg and FedAdp.
  • I. INTRODUCTION: FedImp achieves faster convergence than FedAvg, FedProx, and FedAdp in non-IID FL experiments on EMNIST and CIFAR-10.The experiments evaluate convergence and accuracy across multiple FL scenarios.
  • II. RELATED WORK: FedAvg averages client models after multiple local SGD epochs, but non-IID client data can reduce its performance.FedAvg forms each new global model by averaging models sent to the server.
  • II. RELATED WORK: 76.9% classification accuracy was achieved by FedAvgM versus 30.1% for FedAvg in the most skewed CIFAR-10 settings.FedAvgM applies server momentum by accumulating gradient history to dampen oscillations.
  • II. RELATED WORK: Prior methods address heterogeneity through adaptive weighting, parameter-distance weighting, regularization, or personalized subnetworks.Examples include IDA, FedAdp, FedDyn, and Sub-FedAvg, which respectively target outliers, unequal contributions, global-loss alignment, and client-specific architectures.
  • II. RELATED WORK: FedAdp can suffer from global gradient misalignment when nodes lack label diversity, while FedImp instead quantifies informational richness.The related-work summary contrasts FedImp with methods relying on gradient similarity, parameter distance, or client-specific architectures.

III. PRELIMINARIES OF FEDERATED LEARNING ALGORITHMS · A. Overview of Federated Learning Algorithms

This section introduces federated learning and summarizes FedAvg, FedProx, and FedAdp, emphasizing their distinct approaches to aggregation and heterogeneity. It also identifies algorithmic limitations motivating a more comprehensive method.

  • A. Overview of Federated Learning Algorithms: Federated learning lets clients train a shared model without sharing raw data while optimizing an aggregated global loss across communication rounds.Each round selects K nodes, transmits the previous global model, and has participating clients perform SGD locally.
  • A. Overview of Federated Learning Algorithms: Global model parameters are updated by aggregating local model parameters or gradients, with node-specific weighting factors determining each contribution.The weighting factor is denoted ψ_i.
  • A. Overview of Federated Learning Algorithms: FedAvg weights participating clients according to their numbers of local training samples rather than assigning universally equal aggregation weights.When all clients have identical sample counts, FedAvg reduces to equal weighting.
  • III. PRELIMINARIES OF FEDERATED LEARNING ALGORITHMS: Preliminary tests compare FedAvg and FedAdp across identical, imbalanced, and highly skewed non-IID data scenarios to demonstrate their limitations.These limitations provide motivation for proposing a more comprehensive algorithm.
  • A. Overview of Federated Learning Algorithms: FedProx addresses statistical heterogeneity by adding a proximal term that keeps local updates closer to the previous global model.When μ = 0, FedProx is equivalent to FedAvg; large μ may slow convergence by constraining local parameters.
  • A. Overview of Federated Learning Algorithms: FedProx’s proximal coefficient creates a convergence trade-off: small μ may resemble FedAvg, whereas large μ may slow convergence.The coefficient controls how strongly local parameters are forced toward the previous global parameters.
  • A. Overview of Federated Learning Algorithms: FedAdp dynamically weights clients using the angle between local and global gradient vectors followed by a nonlinear mapping.This gradient-correlation strategy is intended to produce more adaptive global model updates.

B. Limitation of FedAvg and FedAdp · IV. METHODOLOGY: FEDERATED IMPURITY WEIGHTING · A. Impurity Weighting Updating Factor

FedAvg degrades on non-IID data, while FedAdp can fail when similarly weighted clients have severely skewed, non-representative distributions. FedImp addresses these limitations by weighting clients according to entropy-based impurity, with temperature τ controlling aggregation sharpness and requiring a balance that preserves unique information.

  • B. Limitation of FedAvg and FedAdp: FedAvg has low accuracy and slow convergence on non-IID data, while FedAdp can encounter challenges in complex distribution scenarios.FedAdp remains stable in many settings but does not consistently resolve non-IID limitations.
  • B. Limitation of FedAvg and FedAdp: FedAvg deteriorates as imbalanced nodes increase, whereas FedAdp suddenly declines with 7 imbalanced nodes having identical data distributions.FedAvg performs well with ten balanced local nodes, while FedAdp is otherwise described as relatively stable.
  • B. Limitation of FedAvg and FedAdp: FedAdp’s similar weighting of heavily skewed clients can let biased nodes dominate aggregation, causing overfitting to limited class patterns, poorer generalization, and slower convergence.The issue arises when clients share highly imbalanced, non-representative data, such as samples from only a small subset of classes.
  • IV. METHODOLOGY: FEDERATED IMPURITY WEIGHTING: The methodology quantifies each node’s data contribution and uses those contributions to weight global aggregation in FedImp.The approach is introduced to improve federated-learning convergence and performance.
  • A. Impurity Weighting Updating Factor: Entropy Si represents each node’s informational richness, with higher entropy indicating greater data diversity or uncertainty and lower entropy indicating more homogeneous data.Si is normalized between 0 and 1 using a logarithm base C and is computed consistently across varying local class distributions.
  • A. Impurity Weighting Updating Factor: FedImp applies a Softmax function with temperature τ > 0 to entropy-based impurity values to assign aggregation weights ψi(t) to participating clients.The weighting also accounts for each client’s local training-data count Di and the number of participating clients K.
  • A. Impurity Weighting Updating Factor: A lower τ increases contrast between high- and low-entropy clients, while a high τ flattens weights toward uniform averaging.Excessively low τ can exaggerate small entropy differences and disproportionately favor a few nodes.
  • A. Impurity Weighting Updating Factor: Weighting must preserve influence from less-informative nodes when they contain classes absent from more-informative nodes, preventing loss of unique information.The paper emphasizes balancing impurity-based emphasis with integration of all relevant client information.

B. FedImp Algorithm

FedImp performs iterative federated training across communication rounds, updating local models with stochastic gradient descent and aggregating them using impurity-based weights. Its weighting incorporates data quantity when participating nodes hold unequal sample sizes.

  • Round structure: FedImp iterates for T communication rounds, randomly selecting K participating nodes in each round.The algorithm uses local minibatch size B, local epochs E, learning rate η, and control parameter τ.
  • Local updates: Each selected node updates its local model through E local epochs and minibatch stochastic gradient descent with step size η.The LocalUpdate function processes local minibatches and returns updated model parameters.
  • Global aggregation: The server aggregates local model parameters using calculated weights ψi to update the global model w(t).FedImp aggregates parameters rather than raw gradients, supporting multiple local epochs without coordinating client learning rates or batch statistics.
  • Impurity weighting: When participating nodes have equal sample quantities, FedImp weights depend exclusively on their impurity levels.This weighting is derived from equation (7).
  • Impurity weighting: When node sample sizes differ, FedImp assigns weights using both local-data impurity and data quantity.Thus, the weighting adapts to both informational content and dataset size across participating nodes.

C. Complexity and Convergence Analysis … V. EXPERIMENTS AND RESULTS

FedImp adds entropy-based impurity weighting while retaining the same asymptotic computational and communication complexity as FedAvg and FedProx. Its convergence benefits arise from prioritizing clients with diverse data, and experiments compare it against FedAvg, FedProx, and FedAdp.

  • 1) Computational Cost Per Client: O(ED_iP + D_i) is FedImp’s per-client complexity, including entropy-based impurity computation over local samples.The additional O(D_i) entropy term accounts for computing class proportions and impurity weights.
  • 1) Computational Cost Per Client: O(ED_iP) remains FedImp’s overall complexity because O(D_i) is negligible when model size P greatly exceeds the number of classes C.This matches the per-client local-training complexity reported for FedAvg and FedProx.
  • 1) Computational Cost Per Client: A single label pass and histogram-based class counter make FedImp’s impurity computation require negligible memory and minimal additional computation.The passage specifically notes suitability for resource-constrained IoT sensors and mobile phones.
  • 2) Communication Overhead Per Round: O(P) communication cost is retained because FedImp’s additional impurity weight ψ_i is only one scalar relative to the transmitted model parameters.FedAvg and FedProx also communicate model vectors of size P per client per round.
  • 3) Convergence of FedImp: FedImp’s weighting gives greater influence to clients with higher entropy and more diverse, representative data than dataset-size weighting in FedAvg.The mechanism is intended to promote balanced and informative aggregation.
  • 3) Convergence of FedImp: FedImp is expected to converge better with moderate class counts, mixed client balance, and SGD variants using bounded gradients and smooth losses.These conditions correspond to typical non-IID settings with skewed or low-diversity client data.
  • 3) Convergence of FedImp: By emphasizing representative-client updates, FedImp reduces bias and accelerates convergence toward a global optimum, although no formal theoretical convergence bound is provided.The study focuses on empirical validation of convergence improvements.
  • V. EXPERIMENTS AND RESULTS: The experiments evaluate FedImp against FedAvg, FedProx, and FedAdp.This comparison defines the performance evaluation in the experiments and results section.

A. Dataset, Federated Learning Models, and Testing Scenario · 1) Creation of Non-IID Data

Experiments use EMNIST and CIFAR-10 with Dirichlet-based client partitions to model balanced and imbalanced non-IID data. FedImp is evaluated after each communication round on each dataset’s fixed centralized test set.

  • 1) Creation of Non-IID Data: EMNIST Balanced contains 112,800 training and 18,800 testing samples across 47 handwritten-character classes, with 28x28-pixel images.
  • 1) Creation of Non-IID Data: CIFAR-10 contains 60,000 32x32 color images across 10 classes, including 50,000 training and 10,000 testing images.
  • 1) Creation of Non-IID Data: The datasets serve as standardized, computationally efficient benchmarks for direct comparisons with existing federated-learning methods.
  • 1) Creation of Non-IID Data: Each client’s class probabilities are sampled from a symmetric Dirichlet distribution q ∼Dir(θ), where larger θ promotes balance and smaller θ produces imbalance.
  • 1) Creation of Non-IID Data: The partitioning algorithm assigns M samples to each of Nb balanced and Nimb imbalanced nodes using concentration parameters θb and θimb.
  • 1) Creation of Non-IID Data: Each node samples labels according to q, selects unused examples from the chosen class, and adds them to its local dataset Di.
  • 1) Creation of Non-IID Data: The procedure removes exhausted classes and renormalizes q, guaranteeing that original samples are used without duplication across nodes.
  • 1) Creation of Non-IID Data: θb = 100 and θimb = 0.01 define the experimental partitions, while the entire original EMNIST and CIFAR-10 test sets provide fixed centralized evaluation.

2) Structures of the Federated Learning Models · 3) Testing scenarios

The study evaluates four ANN architectures across EMNIST and CIFAR10, using four federated testing scenarios that vary node balance and distribution skew. The scenarios compare FedImp with FedAvg, FedProx, and FedAdp under increasingly non-IID conditions.

  • 2) Structures of the Federated Learning Models: Experiments use four ANN models: MLP and 2-layer CNN for EMNIST, plus 2-layer and 4-layer CNNs for CIFAR10.These architectures are summarized in Figure 3.
  • 2) Structures of the Federated Learning Models: The EMNIST MLP contains two fully connected 200-unit ReLU hidden layers, a 28x28 input, and 47 Softmax output neurons.
  • 2) Structures of the Federated Learning Models: The EMNIST 2-layer CNN uses 5x5 convolutions with 32 and 64 channels, 2x2 max pooling, and a 512-unit ReLU fully connected layer.
  • 2) Structures of the Federated Learning Models: The CIFAR10 2-layer CNN follows the EMNIST CNN design, adapting its input size and using 10 output neurons for image classification.
  • 2) Structures of the Federated Learning Models: The CIFAR10 4-layer CNN has 32, 32, 64, and 64 channels, 3x3 kernels, pooling after layers two and four, and 512- and 128-unit dense layers with 40% dropout.
  • 3) Testing scenarios: Testing compares FedImp with FedAvg, FedProx, and FedAdp while varying the number of balanced and imbalanced nodes across EMNIST and CIFAR10.
  • 3) Testing scenarios: The first three non-IID scenarios use 7 balanced plus 3 imbalanced nodes, 5 balanced plus 5 imbalanced nodes, or 3 balanced plus 7 imbalanced nodes.Their data is partitioned using Algorithm 2 with θb = 100 and θimb = 0.01.
  • 3) Testing scenarios: The extreme-skew scenario uses 3 balanced nodes and 7 identically distributed, imbalanced nodes whose training data are randomly selected from only a few classes.This creates highly skewed non-IID data, while balanced nodes sample M examples from the original training set.

4) Pre-processing and Hyperparameter Configuration · B. Results and Discussion · 1) Results on EMNIST data

The experiments normalize EMNIST inputs and use fixed federated training settings, then compare convergence across heterogeneous EMNIST distributions. FedImp consistently converges quickly and often achieves the strongest accuracy or largest communication-round reductions.

  • 4) Pre-processing and Hyperparameter Configuration: EMNIST pixels are normalized to mean 0.5 and standard deviation 0.5 before federated training.
  • 4) Pre-processing and Hyperparameter Configuration: All nodes participate each round; clients train one epoch with batch size 100 using SGD, cross-entropy loss, learning rate 0.1, and decay rate 0.995.
  • B. Results and Discussion: Convergence is compared by measuring communication rounds required for FedAvg, FedProx, FedAdp, and FedImp to reach convergence.
  • 1) Results on EMNIST data: FedImp consistently displays a fast convergence rate across heterogeneous EMNIST scenarios with the MLP model.
  • 1) Results on EMNIST data: 35 and 33 rounds let FedImp and FedProx surpass 83% accuracy with 7 balanced and 3 imbalanced nodes, compared with 79 for FedAvg and 71 for FedAdp.
  • 1) Results on EMNIST data: 47 rounds let FedImp achieve at least 83% accuracy with 5 balanced and 5 imbalanced nodes, requiring 16.1%, 50%, and 64.4% fewer rounds than FedProx, FedAdp, and FedAvg.
  • 1) Results on EMNIST data: 78 rounds let FedImp reach 83% accuracy with 3 balanced and 7 imbalanced nodes, requiring 27.8% and 66.7% fewer rounds than FedProx and FedAdp, while FedAvg cannot reach the target.
  • 1) Results on EMNIST data: With the 2-layer CNN, FedImp reaches 87% accuracy in 19 rounds for 5 balanced and 5 imbalanced nodes, versus 21 for FedProx, 41 for FedAvg, and 35 for FedAdp.

2) Results on CIFAR-10 Data

On CIFAR-10, FedImp generally converges in fewer communication rounds than competing methods across heterogeneous node distributions and can attain higher accuracy when data are highly imbalanced. Its faster convergence also reduces total communication cost despite a minor per-round weighting-factor overhead.

  • 2-Layer CNN for CIFAR-10: 44.2%, 30.6%, and 1.2% fewer rounds: FedImp converges at 79% in 252 rounds with 5 balanced and 5 imbalanced nodes, compared with FedAvg, FedProx, and FedAdp.The corresponding baselines require 452, 363, and 255 rounds, respectively.
  • 2-Layer CNN for CIFAR-10: 21.9%, 15.6%, and 4.1% fewer rounds: FedImp reaches 76% accuracy in 211 rounds with 3 balanced and 7 imbalanced nodes, ahead of FedAvg, FedProx, and FedAdp.The baselines require 270, 250, and 220 rounds, respectively.
  • 4-Layer CNN for CIFAR-10: 44%, 43.7%, and 25.6% fewer rounds: FedImp surpasses 80% accuracy in 296 rounds with 5 balanced and 5 imbalanced nodes, compared with FedAvg, FedProx, and FedAdp.FedAdp requires 398 rounds, while FedAvg and FedProx require 526 and 529 rounds, respectively.
  • Communication overhead: 43.1% and 29.2% lower bandwidth consumption: FedImp reduces total communication cost versus FedAvg and FedProx through fewer rounds, despite a minor weighting-factor overhead.FedImp’s per-round overhead is 25.5 MB, including transmission of the scalar weighting factor ψ_i; the base model size is 2.5 MB.

3) Discussion

FedImp achieves comparable or better accuracy with fewer communication rounds in non-IID federated learning, especially with more imbalanced nodes, by weighting updates according to local data entropy. The discussion also identifies temperature selection, privacy implications, and real-world validation as important considerations.

  • Empirical performance: FedImp reaches comparable or better test accuracy than FedAvg, FedProx, and FedAdp using considerably fewer communication rounds, particularly with more imbalanced data nodes.This advantage demonstrates improved efficiency in non-IID federated learning scenarios.
  • Why FedImp outperforms baselines: FedImp accelerates convergence by assigning aggregation weights from local dataset entropy, prioritizing diverse information and down-weighting highly imbalanced contributions.FedAvg uses dataset-size weighting, FedProx adds a proximal constraint without distinguishing data quality, and FedAdp relies on gradients that can mislead in extreme non-IID settings.
  • Impact of temperature parameter τ: FedImp’s temperature parameter τ controls local-model weighting, but inappropriate or overly small values may produce suboptimal performance.Figure 8 evaluates different τ values with EMNIST, an MLP model, and 5 balanced plus 5 imbalanced nodes; automated selection remains future work.
  • Privacy and security considerations: FedImp preserves raw-data locality, while client-side computation of impurity and weighting factors supports compatibility with Differential Privacy and Secure Aggregation.The discussion nevertheless notes potential privacy implications because aggregation weights derive from local entropy statistics.
  • Limitations and future work: FedImp still requires validation in real-world deployments and may struggle when clients have low-entropy, mutually exclusive class distributions.Real deployments add device availability, communication delays, hardware heterogeneity, and natural data drift; formal privacy integration is also proposed as future work.

VI. CONCLUSION

FedImp addresses non-IID data in federated learning by weighting clients according to the informational entropy of their local data, improving convergence over established baselines. Future work will examine deployment constraints, privacy-preserving integration, robust defenses, and alternative impurity measures.

  • Conclusion: FedImp assesses each participating node’s informational entropy and standardizes these contributions into distinct weights for global-model aggregation.The method is designed for federated learning under non-IID data.
  • Conclusion: FedImp achieves a superior convergence rate to FedAvg, FedProx, and FedAdp across varied non-IID scenarios.This conclusion is based on extensive experimental evaluations.
  • Future Work: Practical deployment must account for device computational constraints and privacy implications.Planned privacy-preserving integrations include Differential Privacy and Secure Aggregation.
  • Future Work: Robust aggregation and adversarial defenses could strengthen FedImp for real-world applications requiring both efficiency and privacy.These directions are identified as ways to improve practical suitability.
  • Future Work: Alternative measures beyond Shannon entropy, including Gini impurity, Simpsons index, and kernel-based diversity metrics, may improve weighting sensitivity or robustness.These measures could capture different aspects of client data heterogeneity under specific distributions.
Loading 2608.14654v1…