Source-linked AI summary
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
Tzu-Ming Harry Hsu, Hang Qi, Matthew Brown
TL;DR
The paper asks how non-identical client data distributions affect visual classification in Federated Learning. It synthesizes clients across a continuous identicalness range, benchmarks FedAvg, and evaluates server momentum as a mitigation. Performance declines with greater distributional non-identicalness, while server momentum improves FedAvg across skewed settings, including a rise from roughly 35% to above 75% in one example.
Problem
The paper investigates how differing data distributions across Federated Learning clients affect visual classification performance.
Method
The paper samples client class distributions from Dirichlet distributions across concentration parameters, benchmarks FedAvg on CIFAR-10, and tests server momentum.
Results
FedAvg performance changes substantially with concentration parameter α, while FedAvgM consistently improves test accuracy and often approaches the 86.0% centralized baseline.
Takeaways & Limitations
Server momentum improves FedAvg performance across non-identical data settings, including accuracy above 75% where FedAvg falls to around 35% at E = 1 and C = 0.05.
Abstract
from arXiv · showhide
Federated Learning enables visual models to be trained in a privacy-preserving way using real-world data from mobile devices. Given their distributed nature, the statistics of the data across these devices is likely to differ significantly. In this work, we look at the effect such non-identical data distributions has on visual classification via Federated Learning. We propose a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm. We show that performance degrades as distributions differ more, and propose a mitigation strategy via server momentum. Experiments on CIFAR-10 demonstrate improved classification performance over a range of non-identicalness, with classification accuracy improved from 30.1% to 76.9% in the most skewed settings.
1 Introduction
Federated Learning trains models from decentralized edge-device data while preserving privacy. This work studies how differing client data distributions affect visual classification and evaluates FedAvg across a continuous range of non-identicalness.
- Federated Learning trains models from decentralized user data on edge devices in a privacy-preserving framework.
- The study examines client data drawn independently from differing local distributions rather than assuming identical data statistics.
- The evaluation covers a continuous range of non-identical distributions and empirical results across hyperparameters and optimization strategies.
2 Related Work
Prior work studied FedAvg on both pathological and more realistic non-identical client partitions, while theoretical work analyzed convergence under modified objectives and sampling assumptions. This paper extends Dirichlet-based synthesis across a continuous range of concentration parameters and explores optimization settings.
- Prior MNIST work used sorted class-label shards, assigning two shards per client, and reported 99% accuracy with FedAvg despite slower convergence.
- Other studies generated more realistic client distributions by partitioning Extended MNIST over writers or using a Dirichlet concentration parameter of 0.5.
- This work extends Dirichlet synthesis by exploring a continuous range of concentrations α and detailed hyperparameter and optimization settings.
- Theoretical studies introduced proximal client objectives or analyzed FedAvg under proper sampling and averaging in strongly convex problems.
3 Synthetic Non-Identical Client Data
The paper synthesizes non-identical clients by sampling class-probability vectors from a Dirichlet distribution whose concentration parameter α controls similarity to a prior distribution. CIFAR-10 experiments use balanced populations of 100 clients with 500 images each, spanning eight α values.
- Each client’s class labels follow a categorical distribution parameterized by q, with nonnegative components summing to one.
- Client distributions are generated by sampling q ∼ Dir(αp), where p is the prior class distribution and α controls client identicalness.
- As α →∞, clients match the prior distribution, whereas as α →0, each client contains examples from only one randomly chosen class.
- The CIFAR-10 setup uses 100 clients with 500 images each, a uniform prior over 10 classes, and client image assignments determined by sampled q values.
- Figure 1 depicts a two-class sort-and-partition population alongside Dirichlet-generated populations with different concentration parameters and 30 random clients each.
4 Experiments and Results
Experiments benchmark FedAvg across client-distribution heterogeneity and training settings, then evaluate server momentum as a mitigation. Greater non-identicalness reduces accuracy and increases training instability, while FedAvgM improves performance across heterogeneous settings.
- Classification Performance with Non-Identical Distributions: Lower α produces larger accuracy changes, while increasing the reporting fraction yields diminishing returns, especially for identical client datasets.With a fixed optimization-round budget, synchronizing every round (E = 1) does not always improve accuracy on non-identical data.
- Classification Performance with Non-Identical Distributions: More non-identical data causes reduced end-of-training accuracy and more volatile training error; small reporting fractions may not converge within 10,000 rounds.These learning-curve effects are evaluated with fixed learning rates.
- Classification Performance with Non-Identical Distributions: Learning-rate tuning becomes more important as the reporting fraction and α decrease, whereas a broad range of learning rates performs well for large α.Under smaller C and α, careful tuning is required to reach good test accuracy.
- Accumulating Model Updates with Momentum: FedAvgM adds server-side momentum by accumulating aggregated client updates before updating the model.The server computes v ← βv + Δw and updates w ← w − v; experiments use Nesterov momentum across several β values.
- Accumulating Model Updates with Momentum: Above 75% accuracy is maintained by FedAvgM with E = 1 and C = 0.05, while FedAvg falls rapidly to around 35%.FedAvgM performance is close to the 86.0% centralized-learning baseline in many cases.
- Accumulating Model Updates with Momentum: FedAvgM requires tuning its effective learning rate ηeff = η/(1 − β), with narrower viable ranges when few clients report and lower ηeff after extensive local optimization.Extensive local optimization increases client-update variance, requiring a lower effective learning rate to counteract noise.