PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning on Non-IID Data Silos: An Experimental Study

Qinbin Li, Yiqun Diao, Quan Chen, Bingsheng He

arXiv:2102.02079v4cs.LGcs.DC

TL;DR

Federated learning must support machine learning across privacy-constrained data silos whose distributions are non-IID, while prior evaluations lacked systematic coverage. The paper proposes comprehensive partitioning strategies and benchmark experiments, finding that non-IID data substantially challenges accuracy and that no existing state-of-the-art algorithm consistently outperforms the others.

  • Problem

    Prior federated-learning studies lacked a standard, systematic benchmark because they evaluated algorithms on only one or two rigid non-IID data partitions.

  • Method

    The paper proposes NIID-Bench with six non-IID partitioning strategies and evaluates four state-of-the-art algorithms across nine datasets.

  • Results

    Non-IID data significantly challenge FL accuracy, and no existing state-of-the-art algorithm consistently outperforms the others across all cases.

  • Takeaways & Limitations

    Future FL algorithm design and evaluation should consider comprehensive non-IID partitioning strategies and tasks because performance depends on the type and combination of data skew.

Abstract

from arXiv · show

Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple "data silos" (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in "data silos".

I. INTRODUCTION

Federated learning addresses privacy-preserving collaboration across distributed data silos, where non-IID data distributions create a central challenge. The paper introduces NIID-Bench and evaluates existing algorithms across comprehensive non-IID settings.

  • Distributed databases require machine learning over local data without exchanging raw records.
  • Non-IID distributions arise when parties differ in label or feature distributions, such as disease patterns or writing styles.
  • Prior studies evaluated non-IID algorithms with only one or two rigid partitioning strategies, leaving cases insufficiently covered.
  • NIID-Bench introduces six partitioning strategies covering label, feature, and quantity skew, and evaluates four state-of-the-art algorithms on nine datasets.
  • Non-IID data significantly challenge accuracy, no evaluated algorithm dominates all cases, and label skew is more difficult than quantity skew.

B. FedAvg

FedAvg sends a global model to selected parties, aggregates their locally updated models, and can suffer from update drift under non-IID data. FedProx modifies the local objective to limit this drift but requires tuning.

  • FedAvg repeatedly broadcasts the global model, performs local multi-epoch updates, and averages the returned models.Multiple local epochs reduce communication rounds relative to traditional distributed SGD.
  • Non-IID local objectives can direct updates toward local optima far from the global optimum, degrading the averaged model’s accuracy.
  • The reviewed methods all build on FedAvg while targeting effective global-model learning under non-IID data.
  • FedProx adds an L2 regularization term that limits the distance between each local model and the global model.
  • FedProx introduces computation overhead without additional communication overhead, but its regularization weight requires careful tuning.A small weight has little effect, whereas a large weight can make updates very small and slow convergence.

B. FedNova

FedNova modifies FedAvg’s aggregation to account for unequal numbers of local training steps across parties. It normalizes and scales local updates before updating the global model.

  • FedNova addresses parties performing different numbers of local steps because of computation power or dataset-size differences.
  • FedNova normalizes and scales each party’s local update by its number of local steps to avoid biased global updates.

C. SCAFFOLD

SCAFFOLD treats non-IID effects as variance among parties and uses server and client control variates to estimate and correct local-training drift. The paper situates it within a benchmark comparing existing FL algorithms.

  • SCAFFOLD models non-IID data as variance among parties and applies variance reduction to address it.
  • Server and client control variates estimate update directions, allowing local drift to be approximated from their difference.
  • The study compares SCAFFOLD with FedAvg, FedProx, and FedNova under more comprehensive non-IID partitioning strategies.

IV. SIMULATING NON-IID DATA SETTING

NIID-Bench addresses the limited coverage of prior non-IID experiments by synthesizing partitioned datasets and organizing comprehensive non-IID scenarios. The approach supports controlled evaluation of imbalance properties across federated settings.

  • NIID-Bench was developed because existing studies used limited partitioning strategies that could not represent a comprehensive view of non-IID cases.
  • The benchmark synthesizes distributed non-IID datasets by partitioning real-world datasets into multiple smaller subsets.
  • Partitioning strategies allow researchers to quantify and control local-data imbalance while varying factors such as the number of parties and data size.
  • The non-IID cases are defined from differences in P(x_i, y_i), including label distribution skew, feature distribution skew, and quantity skew in horizontal FL.
  • Criteo demonstrates both label distribution skew and quantity skew across parties, while Digits is used to demonstrate non-IID properties.

B. Label Distribution Skew

The benchmark models label distribution skew through two complementary partitioning strategies, while also describing feature-skew constructions for controlled and real-world variation. These strategies vary local features or labels while preserving specified balance properties where designed.

  • Label Distribution Skew: Label distribution skew varies P(y_i) across parties and is simulated with quantity-based and distribution-based label imbalance.
  • Label Distribution Skew: Quantity-based label imbalance assigns each party a fixed number k of labels, randomly allocates label IDs, and divides same-label samples among owning parties.
  • Label Distribution Skew: Distribution-based label imbalance allocates each class across parties according to proportions sampled from a Dirichlet distribution with concentration parameter β > 0.
  • Feature Distribution Skew: Feature distribution skew varies P(x_i) across parties while keeping P(y_i|x_i) the same, and includes noise-based, synthetic, and real-world feature imbalance.
  • Feature Distribution Skew: Noise-based feature imbalance adds party-specific Gaussian noise levels to otherwise randomly and equally divided local datasets.
  • Feature Distribution Skew: FCUBE creates feature imbalance by assigning symmetric cube regions to parties, so feature distributions differ while labels remain balanced.
  • Feature Distribution Skew: FEMNIST creates real-world feature imbalance by assigning handwritten-character writers to parties, producing differences in writing features across parties.

D. Quantity Skew

The experiments compare four FL algorithms across comprehensive partitioning strategies and nine datasets. Label distribution skew is substantially more damaging than feature or quantity skew, and algorithm performance depends on the non-IID setting.

  • NIID-Bench contains six partitioning strategies, making it more comprehensive and representative than the partial non-IID coverage in prior studies.
  • The evaluation uses FedAvg, FedProx, SCAFFOLD, and FedNova on nine public datasets, including six image and three tabular datasets.
  • Top-1 accuracy is compared after the same number of communication rounds, with 50 rounds used by default.
  • Comparison among different non-IID settings: Label distribution skew is the most challenging setting, especially when each party has samples from only one class.
  • Comparison among different non-IID settings: Feature distribution skew usually has accuracy close to IID, while FedAvg has almost no accuracy loss under quantity skew.
  • Algorithm selection: The decision tree recommends algorithms according to the observed non-IID distribution and dataset, but identifying the distribution without prior knowledge remains challenging.
  • Comparison among algorithms: FedProx usually performs best under label and quantity skew, whereas SCAFFOLD usually performs best under feature distribution skew.
  • Comparison among algorithms: SCAFFOLD is unstable across cases, and FedNova shows little superiority over the other FL algorithms.

3) Comparison among different tasks:

Task difficulty and training behavior vary substantially across non-IID settings. CIFAR-10 and tabular datasets are especially challenging, while algorithm stability, local-epoch sensitivity, and party participation affect outcomes.

  • CIFAR-10 and tabular datasets are challenging under non-IID settings, whereas MNIST is simple and the studied algorithms perform similarly well.
  • FedProx has convergence speed close to FedAvg, while SCAFFOLD and FedNova exhibit more unstable training.On CIFAR-10, FedProx is often close to FedAvg in both convergence speed and final accuracy.
  • The number of local epochs strongly affects accuracy, and its optimal value is sensitive to the non-IID distribution.For example, with #C = 2, accuracy generally degrades significantly at 80 local epochs; FedAvg’s optimum is 20 for #C = 1 and #C = 2, but 10 for pk ∼Dir(0.5) and #C = 3.
  • With 100 parties and a sample fraction of 0.1, SCAFFOLD cannot work effectively under partial participation, while other algorithms show unstable training accuracy.The corresponding CIFAR-10 training curves are reported for the party-sampling setting.

E. Scalability

Scalability and mixed data skew expose additional weaknesses in federated learning. More parties reduce accuracy, mixed skews degrade model quality, and algorithm choices trade computation, communication, and stability.

  • Scalability: Accuracy decreases significantly for all approaches as the number of parties increases.With more parties, each client has less local data and local training is more prone to overfitting.
  • Scalability: FedProx has larger computation overhead than FedAvg, while SCAFFOLD requires twice FedAvg’s communication cost.FedAvg, SCAFFOLD, and FedNova have close computation costs, but FedProx is much more computationally expensive.
  • Mixed skew: Mixed types of skew degrade the accuracies of all approaches compared with a single skew, making mixed non-IID settings more challenging.The paper identifies mixed skew as an important direction for future algorithm design because such settings are common in reality.
  • Mixed skew: Under feature and quantity skew, FedAvg and FedProx retain accuracy close to feature-skew performance, whereas SCAFFOLD and FedNova perform poorly.Quantity skew significantly degrades SCAFFOLD and FedNova but does not affect FedAvg and FedProx in the same setting.
  • Implications: Future evaluations should cover diverse partitioning strategies and tasks because no studied algorithm consistently outperforms the others across settings.Accuracy, communication efficiency, training stability, and factors such as local epochs and party sampling all warrant evaluation.

VI. FUTURE DIRECTIONS

The paper proposes future directions connecting federated learning with data management and learned database systems. These directions emphasize profiling non-IID distributions, supporting federated queries and learning, and improving FL under practical constraints.

  • Opportunities for data management: Learned database systems should be revisited for distributed databases, including federated search and learned index structures that avoid exchanging local data.Existing learned systems are mostly designed for centralized databases.
  • Opportunities for data management: Lightweight profiling techniques could identify non-IID distributions before federated learning, using data sampling, sketching, or metadata representations.Extending current statistics estimation methods to non-IID distributions remains an open problem.
  • Opportunities for data management: Federated databases need methods that preserve data privacy while supporting both querying and learning.
  • Opportunities for better FL design: Future FL design should address single-label parties, faster training, and communication efficiency because these settings remain difficult or costly.The paper discusses speaker recognition as an application with single-label devices and suggests fewer communication rounds or faster initialization as possible directions.
  • Benchmarking: NIID-bench contributes six partitioning strategies and comprehensive algorithm comparisons to analyze non-IID federated learning beyond prior limited benchmarks.The paper positions this benchmark as a basis for studying algorithm strengths and weaknesses across distributed data silos.

APPENDIX

The appendix extends the experiments across datasets and training factors. It reports additional training curves, local-epoch results, party-sampling results, and batch-size analyses.

  • Figures 12–16 show training curves for the studied approaches on datasets other than CIFAR-10.
  • Figures 12 and 18–21 report accuracy under different numbers of local epochs for datasets other than CIFAR-10.
  • Figure 22 shows training curves on CIFAR-10 under the party-sampling setting.
  • Heterogeneity of local data does not appear to influence the behavior of different batch-size choices.For batch sizes from 16 to 256, the studied algorithms behave similarly; larger batches slow learning as in centralized training.

E. Model Architectures

The experiments examine model architecture, local training choices, and training behavior across datasets and partitions. Batch normalization is identified as a source of instability under non-IID data, particularly for ResNet-50.

  • Averaging batch normalization layers introduces instability in non-IID settings.
  • VGG-9 and ResNet-50 achieve usually close final accuracies on CIFAR-10, but ResNet-50 training appears more unstable.The passage attributes this challenge to aggregating batch normalization layers in ResNet-50.
  • The study compares training curves across FMNIST, SVHN, FCUBE, FEMNIST, and CIFAR-10 under different approaches and experimental settings.Additional comparisons vary local epochs, party counts, sample fractions, batch sizes, architectures, and data partitions.
  • CIFAR-10 experiments examine approaches with 100 parties and sample fraction 0.1, different batch sizes, and VGG-9 or ResNet-50 under different partitions.
Loading 2102.02079v4…