PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Learning on Non-IID Data Silos: An Experimental Study

Qinbin Li, Yiqun Diao, Quan Chen, Bingsheng He

arXiv:2102.02079v4cs.LGcs.DC

TL;DR

Prior studies of federated learning on non-IID data tested only a few rigidly partitioned distributions, leaving no standard benchmark or systematic evaluation. This paper introduces six partitioning strategies and evaluates existing algorithms, finding that none consistently outperforms the others.

  • Problem

    Prior studies evaluated only one or two rigidly partitioned non-IID distributions, and no standard benchmark or systematic evaluation existed.

    Source: E. Motivation of this study · p. 4

  • Method

    The paper introduces six non-IID partitioning strategies and evaluates four state-of-the-art federated learning algorithms on nine datasets.

    Source: §I · p. 2

  • Results

    No state-of-the-art federated learning algorithm outperforms the others in all non-IID cases.

    Source: §I · p. 2

  • Takeaways & Limitations

    Future federated learning algorithm design and evaluation should consider more comprehensive non-IID partitioning strategies and tasks.

    Source: H. Insights on the Experimental Results · p. 11

Abstract

from arXiv · show

Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple "data silos" (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in "data silos".

I. INTRODUCTION · II. PRELIMINARIES · A. Notations

The paper introduces NIID-Bench to study federated learning (FL) across non-IID data silos, where regulations restrict raw-data exchange. It finds that FL accuracy and stability depend on the type of data skew, with no existing state-of-the-art algorithm outperforming others in every case.

I. INTRODUCTION

  • Privacy concerns and regulations fragment training data into silos and restrict raw-data transfer, while non-IID label or feature distributions across parties can degrade machine learning effectiveness.The paper motivates learning from distributed databases without exchanging local data.Source: §I · p. 1
  • The study focuses on horizontal FL, where parties share a feature space but have different sample spaces and iteratively exchange models with a server rather than raw data.The server updates a global model and sends it back to parties in each round.Source: §I · p. 1
  • Prior FL studies often tested only one or two rigid partitioning strategies, leaving non-IID cases insufficiently covered and algorithm advantages and disadvantages systematically unexamined.Examples include two-class partitions in FedAvg [56] and Dirichlet class counts in FedNova [72], both covering label skew.Source: §I · pp. 1–2
  • NIID-Bench introduces six partitioning strategies covering label, feature, and quantity skew; four are designed in this study and two are adopted from prior work.The authors describe the benchmark as more comprehensive than previous studies [28],,, [72].Source: §I · p. 2
  • The experiments evaluate FedAvg [56], FedProx, SCAFFOLD, and FedNova [72] on nine datasets, and the authors make code publicly available with a leaderboard for non-IID settings.The benchmark is intended to support evaluation of existing or new algorithms across different partitioning strategies.Source: §I · p. 2

A. Notations

  • The notation defines D as the global dataset, D_i as party P_i’s local dataset, and w_t and w_t^i as the global and local models at round t.The global model is the output of the federated learning process.Source: A. Notations · p. 2

B. FedAvg … A. FedProx

The paper introduces FedAvg and explains how non-IID data cause local-update drift that can undermine its accuracy. It then surveys FedAvg-based approaches and describes FedProx’s regularization mechanism and tuning tradeoffs.

B. FedAvg

  • FedAvg [56] selects parties each round, sends them the global model, aggregates their locally trained models, and uses multiple local epochs to reduce communication rounds.The passage notes that this communication efficiency may come at the cost of accuracy, as prior studies have shown [28],, [47].Source: B. FedAvg · p. 2

C. Effect of Non-IID Data

  • Under non-IID data, divergent local distributions create drift toward local optima, so averaging can move the global model far from its optimum and reduce accuracy versus IID.The risk is greater when local updates are large, such as with more local epochs,, [71], [72].Source: C. Effect of Non-IID Data · p. 3
  • FedAvg’s non-IID drift is illustrated as local optima diverging from the global optimum, leaving the averaged model far from that optimum.The figure caption identifies the visual as an example of drift under non-IID data.Source: C. Effect of Non-IID Data · p. 3, Figure 2 · p. 3

III. FL ALGORITHMS ON NON-IID DATA

  • The paper summarizes FedAvg [56], FedProx, FedNova [72], and SCAFFOLD as approaches based on FedAvg that target effective global learning under non-IID data.Personalization [13],, [22] and robustness to combinations of local distributions [10], [57], [63] are outside its scope.Source: §III · p. 3

A. FedProx

  • FedProx adds an L2 regularization term to the local objective to limit the distance between local and global models, constraining updates so their average stays closer to the global optimum.This is a lightweight change to FedAvg that adds computation overhead but no communication overhead.Source: A. FedProx · p. 3
  • FedProx’s regularization weight µ requires careful tuning: too little has almost no effect, while too much makes local updates small and slows convergence.The algorithm summary specifies local datasets, parties, communication rounds, local epochs, and learning rate as inputs, with the final model as output.Source: A. FedProx · p. 3

B. FedNova

FedNova addresses unequal local-step counts by normalizing and scaling each party’s update before aggregation, reducing bias in global updates with negligible computation overhead.

B. FedNova

  • Different parties may perform different numbers of local steps because of computation-power or dataset-size differences.These differences can arise under the same time constraint or the same local epochs and batch size.Source: B. FedNova · pp. 3–4
  • FedNova normalizes and scales each party’s local update according to its number of local steps before updating the global model.This counters the greater influence that larger updates would have under simple averaging.Source: B. FedNova · p. 4
  • FedNova makes lightweight modifications to FedAvg and adds negligible computation overhead when updating the global model.Source: B. FedNova · p. 4

C. SCAFFOLD · D. Other Studies · E. Motivation of this study

The study motivates a comprehensive benchmark because prior non-IID evaluations used only one or two distributions and rigid partitioning strategies. It also describes SCAFFOLD’s control-variate correction and notes contemporary related approaches whose comparison it leaves for future work.

C. SCAFFOLD

  • SCAFFOLD uses server and client control variates to estimate update directions, approximate local-training drift from their difference, and correct local updates by adding that drift.It models non-IID as introducing variance among parties and applies variance reduction.Source: C. SCAFFOLD · p. 4
  • SCAFFOLD updates local control variates using gradients at the global model or previously computed gradients; the latter costs less computation, while the former may be more stable.Compared with FedAvg, SCAFFOLD intuitively doubles communication size per round because of the additional control variates.Source: C. SCAFFOLD · p. 4

D. Other Studies

  • FedDyn [2] adds local-training regularization based on global and previous-round models, while FedBN keeps client batch-normalization layers local for feature shift.The paper leaves comparison with these contemporary studies for future work.Source: D. Other Studies · p. 4
  • A class-imbalance monitor and new loss and model-contrastive learning [40] are other contemporary non-IID approaches; the paper leaves their comparison for future work.The contrastive approach compares representations from current local, previous-round local, and global models to correct local training.Source: D. Other Studies · p. 4

E. Motivation of this study

  • Prior studies,, [72] showed preliminary gains over FedAvg but evaluated only one or two non-IID distributions using rigid partitioning strategies.The paper identifies the lack of a standard benchmark or systematic study as the resulting gap.Source: E. Motivation of this study · p. 4
  • The study proposes a benchmark with more comprehensive data distributions and partitioning strategies to evaluate existing algorithms’ pros and cons and identify future challenges and opportunities.The authors say this is needed because there is no standard benchmark or systematic study of FL algorithms on non-IID data.Source: E. Motivation of this study · p. 4

IV. SIMULATING NON-IID DATA SETTING · A. Research Problems

NIID-Bench addresses the limited partitioning strategies in prior work by framing two design questions: whether to use real or synthesized federated datasets, and how to cover non-IID scenarios comprehensively. The study motivates partitioning real-world datasets for controlled experiments, focuses on three non-IID skews, and illustrates them with Criteo and Digits.

IV. SIMULATING NON-IID DATA SETTING

  • NIID-Bench is proposed to address the limited partitioning strategies of prior studies and provide a more comprehensive view of non-IID cases.Source: §IV · p. 4

A. Research Problems

  • The study frames two design problems: choosing real or synthesized non-IID datasets, and designing comprehensive non-IID scenarios.Source: A. Research Problems · p. 4
  • Partitioning public real-world datasets enables controlled imbalance and flexible party or data sizes, while real federated datasets are difficult to obtain [29].Prior studies use partitioning strategies, [56], [72]; compared with real federated datasets [6], [29], generated data may not capture complex real distributions, so the approaches are complementary.Source: A. Research Problems · p. 5
  • For horizontal FL, the paper considers label, feature, and quantity skew, but excludes feature-conditional label skew and the vertical-FL case [33].The excluded label-conditional feature case is mainly associated with vertical FL; the excluded feature-conditional label case is outside scope under a shared P(y|x) assumption, though domain adaptation [61] or personalization [13], may address it.Source: A. Research Problems · p. 5
  • The paper notes that mixed skew types can occur and reserves their discussion for Section V-G.Source: A. Research Problems · p. 5
  • Criteo illustrates label and quantity skew across users, while Digits illustrates feature skew between MNIST and SVHN despite shared labels.The Criteo figure cells count samples of each class per party; these examples support the practical relevance of the considered cases.Source: A. Research Problems · p. 5, Figure 3 · p. 5

B. Label Distribution Skew · C. Feature Distribution Skew

The paper models label skew by varying label distributions across parties and feature skew by varying feature distributions while keeping P(y_i|x_i) the same. It proposes partitioning strategies for both types of heterogeneity.

B. Label Distribution Skew

  • Label skew is simulated with quantity-based and distribution-based imbalance strategies.The underlying label distributions P(y_i) vary across parties.Source: B. Label Distribution Skew · p. 5
  • In quantity-based imbalance, each party receives samples from a fixed k labels, selected randomly and split equally across parties that own each label, with no sample overlap.The paper denotes this strategy #C = k.Source: B. Label Distribution Skew · p. 6
  • In distribution-based imbalance, each class’s samples are allocated across parties using Dirichlet proportions, and smaller β produces a more unbalanced partition.For class k, the proportions are sampled from Dir(β); β controls imbalance level.Source: B. Label Distribution Skew · p. 6

C. Feature Distribution Skew

  • Feature skew varies P(x_i) across parties while keeping P(y_i|x_i) the same, and noise-based imbalance adds party-specific Gaussian noise to randomly and equally divided data.For party P_i, noise is sampled as Gau(σ · i/N); users can increase σ to increase feature dissimilarity.Source: C. Feature Distribution Skew · p. 6
  • The synthetic FCUBE dataset splits a three-dimensional cube by x1 = 0, x2 = 0, and x3 = 0, assigning symmetric parts to parties so features vary while labels remain balanced.Labels are classified by the plane x1 = 0.Source: C. Feature Distribution Skew · p. 6
  • FEMNIST models real-world feature skew by randomly and equally assigning EMNIST writers and their digit images to parties, whose writer-specific features differ.The passage identifies stroke width and slant as examples of differences among writers.Source: C. Feature Distribution Skew · pp. 6–7

D. Quantity Skew … 1) Comparison among different non-IID settings:

The study compares federated-learning algorithms across varied non-IID settings, finding label-distribution skew especially challenging while feature and quantity skew are generally less harmful. It frames the experiments with six partitioning strategies and offers guidance for choosing algorithms by data distribution.

D. Quantity Skew

  • Quantity skew varies local dataset sizes by allocating sample proportions from a Dirichlet distribution, with β controlling the imbalance level.The paper denotes this partitioning strategy q ∼Dir(β).Source: D. Quantity Skew · p. 7

E. Experiments in Existing Studies

  • NIID-bench uses six partitioning strategies, while each compared prior study covers only partial non-IID cases, making results across papers impossible to compare directly.The paper describes the six strategies as more comprehensive and representative of different non-IID data cases.Source: E. Experiments in Existing Studies · p. 7

V. EXPERIMENTS

  • The study uses nine public datasets: six image datasets and three tabular datasets.Image datasets use a CNN and tabular datasets use a three-hidden-layer MLP; the default is 10 parties, except four for FCUBE, with all parties participating each round.Source: §V · p. 7
  • Algorithms are compared using test top-1 accuracy after the same number of rounds, set to 50 by default.This controls for differences in training duration across the studied algorithms.Source: §V · p. 7

A. Overall Accuracy Comparison

  • The experiments evaluate FedAvg, FedProx, SCAFFOLD, and FedNova across non-IID settings and homogeneous IID partitions.The accuracy comparison is presented in Table III.Source: A. Overall Accuracy Comparison · p. 7

1) Comparison among different non-IID settings:

  • Across Table III, label-distribution skew most reduces accuracy, especially for single-class partitions; feature skew is near IID except on CIFAR-10, and quantity skew barely affects FedAvg.All studied algorithms perform worse on label skew, and accuracy improves as each party has data from more classes. FedAvg’s weighted averaging helps it handle quantity imbalance.Source: 1) Comparison among different non-IID… · pp. 7–8, Table III · p. 8
  • The decision tree recommends SCAFFOLD for feature skew and FedProx for similar distributions with different dataset sizes; choosing without prior knowledge of the distribution remains challenging.The examples are digits from different writers for feature skew and databases with different capacities for quantity skew.Source: 1) Comparison among different non-IID… · p. 8, Figure 7 · p. 9

2) Comparison among different algorithms: … D. Party Sampling

Across non-IID data-silo experiments, algorithm performance varies by partition and task, while training stability and accuracy are sensitive to local updates and party sampling. No single algorithm is best across the reported settings.

2) Comparison among different algorithms:

  • FedProx usually achieves the best accuracy under label and quantity skew, while SCAFFOLD usually leads under feature skew; improvements over FedAvg are sometimes under 1%.When #C = 1, FedProx can significantly outperform FedAvg, SCAFFOLD, and FedNova.Source: 2) Comparison among different algorithms · p. 9
  • SCAFFOLD's accuracy is highly variable across settings, and FedNova shows little superiority over other algorithms.FedNova also leaves substantial room for improvement relative to FedAvg on homogeneous partitions.Source: 2) Comparison among different algorithms · p. 9

3) Comparison among different tasks:

  • CIFAR-10 and tabular tasks are challenging under non-IID settings, whereas MNIST is simple enough that studied algorithms perform similarly well in most settings.Heterogeneity particularly degrades CIFAR-10 and tabular accuracy, with large losses on tabular data under label skew.Source: 3) Comparison among different tasks · p. 9

B. Communication Efficiency

  • FedProx converges at nearly the same speed as FedAvg, while SCAFFOLD and FedNova are less stable during training.On CIFAR-10, FedAvg and FedProx are very unstable at #C = 1, and SCAFFOLD and FedNova fail to improve over rounds; FedNova is also unstable under q ∼Dir(0.5). Small best µ values make FedProx's regularization have little influence.Source: B. Communication Efficiency · p. 9, Figure 8 · p. 9

C. Robustness to Local Updates

  • Local-epoch choices strongly affect CIFAR-10 accuracy, and the optimal number of epochs varies with the non-IID setting.With #C = 2, accuracy generally degrades significantly at 80 epochs; FedAvg's optimum is 20 epochs for #C = 1 and #C = 2, but 10 for pk ∼Dir(0.5) and #C = 3.Source: C. Robustness to Local Updates · pp. 9–10, Figure 9 · p. 9

D. Party Sampling

  • With 100 parties and a 0.1 sample fraction, training curves are unstable in most non-IID settings, and SCAFFOLD performs poorly across all settings.Sampling changes the local distributions across rounds, causing averaged-gradient directions to vary; infrequent control-variate updates make SCAFFOLD's direction estimates inaccurate.Source: D. Party Sampling · p. 10, Figure 10 · p. 10

E. Scalability … H. Insights on the Experimental Results

The experiments show that more parties and mixed non-IID skew challenge accuracy, while algorithm efficiency involves computation and communication trade-offs. The authors conclude that no studied algorithm performs well across all settings and call for broader evaluation and methods designed for realistic mixed skew.

E. Scalability

  • Accuracy decreases significantly as the number of parties increases, with less local data per party making local training prone to overfitting.The study identifies effective, communication-efficient learning with small amounts of client data at large scale as an open problem.Source: Figure 11 · p. 10, E. Scalability · p. 10

F. Efficiency

  • FedProx has much higher computation cost than other algorithms, while SCAFFOLD’s communication cost is twice FedAvg’s.The computation costs of FedAvg, SCAFFOLD, and FedNova are close; FedProx’s objective modification adds per-batch gradient-descent overhead, whereas FedNova and SCAFFOLD add few operations each round.Source: Table IV · p. 10, F. Efficiency · p. 10

G. Mixed Types of Skew

  • Mixed skew lowers all approaches’ accuracy in the label-plus-feature setting, while quantity skew significantly degrades SCAFFOLD and FedNova but does not affect FedAvg or FedProx.In the feature-plus-quantity setting, FedAvg and FedProx are close to their feature-skew accuracy; mixed-skew cases are identified as important for future algorithm design because they are common in practice.Source: G. Mixed Types of Skew · pp. 10–11, Table V · p. 11

H. Insights on the Experimental Results

  • No studied algorithm consistently outperforms the others or performs well in all settings, motivating evaluation across more non-IID partitioning strategies and tasks.The authors identify addressing distributed data-silo challenges with federated learning as a promising research direction.Source: H. Insights on the Experimental Results · p. 11
  • Accuracy and communication efficiency are important evaluation metrics, and the study finds trade-offs between them as well as stability in both during training.Source: H. Insights on the Experimental Results · p. 11
  • Federated learning adds evaluation factors such as local epochs, batch normalization, party sampling, and number of parties, while batch size behaves similarly to centralized training.The authors say these challenging factors deserve more attention in future FL evaluations.Source: H. Insights on the Experimental Results · p. 11
  • Mixed types of skew cause greater model-quality degradation than single skew types, so effective algorithms for multiple skews deserve investigation because such settings are more practical.The paper illustrates this with images from different areas that vary in label distributions and camera-related features such as contrast.Source: H. Insights on the Experimental Results · p. 11, G. Mixed Types of Skew · p. 11

VI. FUTURE DIRECTIONS … VIII. CONCLUSION

The paper introduces NIID-bench with six data-partitioning strategies and experiments comparing federated-learning algorithms. It outlines future directions for data management and robust FL on non-IID distributed databases.

VI. FUTURE DIRECTIONS

  • The paper frames future work around data-management techniques and federated learning for non-IID distributed databases.The proposed directions address both the distributed-data setting and the effects of non-IID data on federated learning.Source: §VI · p. 11

A. Opportunities for data management

  • Data-management directions include adapting learned systems to data silos, profiling non-IID distributions, sampling parties selectively, protecting privacy, and supporting queries alongside learning.Potential approaches include learned indexes and federated search; sampling and sketching,; skew-resistant and stratified sampling,,; and differential privacy. Extending statistics estimation to non-IID data, balancing privacy with accuracy, and combining SQL queries with learning while preserving privacy remain open problems.Source: A. Opportunities for data management · pp. 11–12

B. Opportunities for better FL design

  • Single-label parties can make FL accuracy very poor, yet this setting has practical applications such as speaker recognition on devices with one user’s voice data.The paper notes that the setting may seem unrealistic, but it can occur in practice.Source: B. Opportunities for better FL design · p. 12
  • Because training speeds are usually similar across algorithms, faster FL may require fewer-round communication or initialization and more efficient tuning of local epochs.Single-round methods have been studied [21], [41], though reliance on public data may limit applications; fast initialization may also help [41]. Bayesian optimization has been studied for federated hyperparameter search [9].Source: B. Opportunities for better FL design · p. 12
  • Robust FL algorithms should address varying non-IID settings, while batch-normalization aggregation may need to average learned parameters but retain local statistics.The paper links these directions to inconsistent algorithm performance across settings and to heterogeneity in parties’ batch-normalization statistics; averaging parameters while leaving mean and variance local is a possible approach [4].Source: B. Opportunities for better FL design · p. 12

VII. RELATED WORK

  • Prior benchmarks cover datasets, metrics, implementations, or domains, but do not analyze algorithms across different non-IID settings; NIID-bench addresses this gap with broader partitioning strategies and experiments.Earlier work also summarized non-IID cases without providing strategies to generate their distributions; this study summarizes six strategies for three cases, four designed in the study and two adopted from prior FL work.Source: §VII · p. 12

VIII. CONCLUSION

  • NIID-bench introduces six data-partitioning strategies and comprehensive experiments that reveal strengths and weaknesses of existing federated-learning algorithms.The study aims to inform future work on effective machine-learning services over distributed databases.Source: §VIII · p. 12

APPENDIX … D. Batch Size

The appendix reports training-curve and local-epoch accuracy analyses, plus CIFAR-10 party-sampling curves. For batch size, local-data heterogeneity did not appear to alter batch-size behavior in federated learning.

A. Training Curves

  • The appendix presents training curves for different approaches on the studied datasets except CIFAR-10.Source: A. Training Curves · p. 14

B. Number of Local Epochs

C. Party Sampling

D. Batch Size

  • Local-data heterogeneity did not appear to influence behavior across batch-size choices, and batch-size knowledge from centralized training still applied in non-IID federated learning.The study tested batch sizes from 16 to 256 using FedAvg and FedProx.Source: D. Batch Size · p. 14
  • A large batch size slows the learning process, as in centralized training.Batch size was varied from 16 to 256.Source: D. Batch Size · p. 14
  • The four studied algorithms showed similar behaviors across different batch sizes.Source: D. Batch Size · p. 14

E. Model Architectures

The architecture experiments compare VGG-9 and ResNet-50 on CIFAR-10; their final accuracies are usually close, but ResNet-50 training is more unstable. Simple averaging of batch-normalization layers adds instability in non-IID settings.

E. Model Architectures

  • Simple averaging of ResNet-50 batch-normalization layers introduces instability in non-IID settings by failing to capture global distribution statistics.Local batch-normalization layers handle local distributions well, while their simple average may not capture the global distribution statistics.Source: E. Model Architectures · p. 15
  • VGG-9 and ResNet-50 usually reach close final CIFAR-10 accuracies, but ResNet-50 training appears more unstable.The experiments compare the models under different data partitions.Source: E. Model Architectures · p. 15, Figure 24 · p. 20

Related papers

Loading 2102.02079v4…