Source-linked AI summary
Federated Visual Classification with Real-World Data Distribution
Tzu-Ming Harry Hsu, Hang Qi, Matthew Brown
TL;DR
Federated learning preserves privacy by training visual models on decentralized devices, but client data are heterogeneous and imbalanced rather than IID. This paper characterizes those effects using realistic species and landmark datasets, and introduces algorithms that resample and reweight clients to improve training accuracy and stability. It also releases the datasets as benchmarks for future federated visual-learning research.
Problem
Federated visual learning must handle client data whose class distributions and available quantities differ, contrary to the IID assumption used in centralized training.
Method
The paper studies these shifts with realistic per-user datasets, controlled distributions, and algorithms for client resampling and importance reweighting.
Results
The work reports large-scale real-world visual classification in a federated setting and substantial improvements in classification accuracy and training stability from its algorithms.
Takeaways & Limitations
The released species and landmark datasets provide benchmarks for further research on privacy-preserving federated visual learning.
Takeaways & Limitations
The paper notes that performance remains challenging when the participating-client pool and round budget are small, alongside other practical federated-learning challenges.
Abstract
from arXiv · showhide
Federated Learning enables visual models to be trained on-device, bringing advantages for user privacy (data need never leave the device), but challenges in terms of data diversity and quality. Whilst typical models in the datacenter are trained using data that are independent and identically distributed (IID), data at source are typically far from IID. Furthermore, differing quantities of data are typically available at each device (imbalance). In this work, we characterize the effect these real-world data distributions have on distributed learning, using as a benchmark the standard Federated Averaging (FedAvg) algorithm. To do so, we introduce two new large-scale datasets for species and landmark classification, with realistic per-user data splits that simulate real-world edge learning scenarios. We also develop two new algorithms (FedVC, FedIR) that intelligently resample and reweight over the client pool, bringing large improvements in accuracy and stability in training. The datasets are made available online.
1 Introduction
Federated learning trains models from decentralized device data while keeping raw training data on-device, but real-world client distributions are heterogeneous and imbalanced rather than IID. This work studies those effects for visual models, proposes mitigation algorithms, analyzes realistic per-user data, and releases large-scale datasets for species and landmark classification.
- Privacy and federated learning: Federated learning trains models across devices while raw training data remain on-device, with optional differential privacy for bounded protection against individual inference.Each round sends a model to participating clients, aggregates their returned gradients, and updates the server model.
- Real-world challenges: Client data are non-IID because user habits, preferences, and geographic locations vary, while participating devices also have different amounts of training data.These correspond to non-identical class distributions and imbalanced client sizes.
- Study focus: The paper studies how heterogeneous client distributions affect federated visual learning and proposes techniques for more effective and efficient training.The focus is specifically on non-identical class distributions and imbalanced client sizes.
- Contributions: The authors analyze real-world per-user data alongside controlled Dirichlet and geographic distributions, propose two algorithms, and provide large-scale datasets for two classification problems.The datasets cover natural-world species and landmark recognition, with per-user data splits.
- Contribution: The work reports itself as the first attempt to train large-scale visual classification models for real-world problems in a federated setting.The authors make the datasets available to enable future research.
2 Related Work
Prior federated-learning studies include synthetic non-identical partitions, more realistic but limited per-user datasets, and variance-reduction methods for heterogeneous clients. This work addresses dataset limitations while applying importance reweighting in a setting with multiple client proposal distributions and partially overlapping support.
- Synthetic client partitions: Synthetic MNIST and CIFAR-10 partitions show that FedAvg can converge on non-identical clients, although it may require more rounds than IID training.The MNIST result reaches 99% accuracy under a shard-based pathological split.
- Realistic data distributions: Existing realistic client datasets use writers, celebrities, online users, or street-level cameras, but are limited in size and not representative of federated device data.The paper positions its datasets as addressing these limitations.
- Variance reduction: Variance-reduction approaches address distribution shift through proximal objectives, control variates, or importance sampling and reweighting.The paper adopts importance reweighting for a federated setting with heterogeneous clients.
- Variance reduction: Unlike classic importance sampling with one proposal distribution, heterogeneous federated clients form multiple proposal distributions with partially common support.This distinction motivates the paper's augmented client objectives.
3 Federated Visual Classification Problems
The paper builds federated visual-classification problems around geographically varying species and landmark images contributed by users. Both datasets preserve per-user splits, enabling realistic on-device learning experiments in a federated simulation environment.
- Motivation: Visual data can vary substantially by geography, as illustrated by regional Douglas-Fir and Red Maple observations in iNaturalist.User interests and photography habits can further influence device-level data.
- Natural species classification: The iNaturalist problem uses images and annotations contributed by citizen scientists around the globe.The setting could support training fine-grained species classifiers without transferring images.
- Dataset construction: The iNaturalist benchmark includes five partitionings that vary class-distribution shift and client-size imbalance, with different client counts.Figure 2 shows re-balancing plus class and example counts across clients.
- Landmark recognition: The landmark problem uses images uploaded by Wikipedia contributors and models a smartphone scenario involving photographs of natural and architectural landmarks.The proposed federated setting avoids uploading or centrally storing private user photos.
- Dataset construction: Both datasets partition labeled images by user, supporting realistic federated learning in which data remain on-device.Experiments use a federated-learning simulation engine similar to TensorFlow Federated.
4 Datasets
The paper constructs federated benchmarks from species and landmark datasets, using balanced or filtered data and realistic user, authorship, and geographic partitions. These datasets support controlled study of client-level distribution variation and imbalance.
- iNaturalist datasets: The benchmark uses iNaturalist-2017, a species dataset with 579,184 training examples, 95,986 test examples, and over 5,000 classes.Each image includes a species label, geographic coordinates, and authorship information.
- iNaturalist datasets: The iNaturalist federated benchmark balances classes to isolate client-level variation, producing 1,203 classes with 100 training examples each.Classes with fewer than 100 training examples are removed, and remaining classes are subsampled to 100 examples.
- iNaturalist datasets: The balanced iNaturalist training set is partitioned by user into 9,275 clients, creating the iNaturalist-User-120k population.The contributor partitioning is intended to resemble images collected per user.
- iNaturalist datasets: Additional iNaturalist populations vary client deviation from the global distribution by partitioning examples at different geographic granularities.An adaptive S2-grid partition recursively subdivides cells until each contains at most Nmax examples, discarding cells below Nmin.
- Landmarks-User-160k: The landmark benchmark uses the cleaned GLD-v2 subset, whose images are partitioned by authorship after filtering to reduce the training long tail.GLD-v2-clean removes images whose local geometric features cannot be matched to at least two same-label images.
- Landmarks-User-160k: Landmarks-User-160k contains 164,172 training images from 2,028 landmarks and 1,262 users, plus a 19,526-image test split with non-overlapping authors.The test split is relatively balanced among classes, with 1,835 landmarks having exactly 10 test images.
5 Methods
The paper uses FedAvg as a baseline and introduces FedIR and FedVC to address non-identical class distributions and imbalanced client sizes. These methods reweight local objectives or standardize per-client training workload while preserving the federated training framework.
- FedAvg baseline: FedAvg randomly selects K clients uniformly, runs local SGD for E epochs, and averages their updates at the server.The server applies the aggregated update using learning rate γ.
- FedAvg baseline: FedAvgM adds server-side momentum by updating the model with an exponentially weighted moving average of client updates.Its update uses vt ← βvt−1 + ḡt before applying the server learning rate.
- Importance reweighting: FedIR applies importance weights to each client’s local objective to correct bias from client-specific data distributions relative to a target distribution.The scheme assumes clients share approximately the target’s conditional image distribution given each class.
- Importance reweighting: FedIR can preserve privacy because each client retains its class distribution while the server transmits only the target class probabilities.Communication requires C scalars for C classes, and the method can be combined with FedAvgM.
- Motivation and evaluation setup: The methods target practical effects of client imbalance, including increased local-update variance, slow convergence or divergence, and clients missing reporting deadlines.The dataset statistics table compares partitionings while retaining shared centralized baselines when the underlying data pool is unchanged.
- Virtual clients: FedVC fixes the number of training examples per round, conceptually splitting large clients and repeating small clients so virtual clients have similar sizes.With mini-batch size B, each client performs exactly S = NVC/B optimization steps.
6 Experiments
The experiments characterize how heterogeneous class distributions and imbalanced client sizes affect federated visual classification, then evaluate FedIR and FedVC on synthetic and realistic user partitions. The proposed modifications improve convergence, accuracy, or stability, with performance approaching centralized learning at favorable operating points.
- Distribution non-identicalness: CIFAR-10/100 experiments vary client class distributions using Dirichlet partitions and measure non-identicalness with weighted average EMD.EMD uses the L1 distance between client and population class distributions, bounded between 0 and 2.
- Distribution non-identicalness: Accuracy approaches centralized performance for identical data but drops substantially around EMD 1.7–2.0 across both CIFAR datasets.FedAvgM significantly improves convergence under heterogeneous conditions, while larger report goals help with diminishing returns.
- Importance reweighting: FedIR consistently improves Landmarks-User-160k over FedAvg, with especially larger convergence and accuracy gains when few clients report per round.FedAvgM provides the strongest improvements overall, while FedIR further helps at small report goals.
- Importance reweighting: Landmarks-User-160k has higher non-identicalness than iNaturalist-User-120k, with EMD 1.94 versus 1.83, and benefits more from FedIR.The comparison links the stronger FedIR benefit to the more skewed client data distribution.
- Federated virtual clients: FedVC improves accuracy and efficiency for imbalanced clients while reducing learning volatility and stabilizing training.It is evaluated under both fixed communication-round and fixed batch budgets, where batch count proxies a fixed time budget.
- Natural user benchmarks: 57.2% accuracy from FedAvgM with FedVC and K = 100 nearly matches the 57.9% centralized accuracy on iNaturalist-User-120k.The benchmark contains 9,275 clients, 120k examples, and 1,203 species classes.
- Natural user benchmarks: Small reporting-client counts and limited round budgets remain important unresolved settings for practical visual federated learning.The authors identify fast learning with a limited client pool as a critical research area.
- Natural user benchmarks: FedAvgM reaches 65.9% on Landmarks-User-160k at 5k rounds, 1.2% below centralized learning, while last-layer fine-tuning reaches 39.8% versus 40.3%.The dataset contains 164,172 images across 2,028 landmarks and 1,262 clients.
7 Conclusions
The paper demonstrates that large-scale visual classifiers can be trained through a privacy-preserving federated approach despite challenges from per-user data distributions. It contributes two datasets and benchmarks while identifying several practical challenges beyond its scope.
- Large-scale visual classifiers can be trained using a privacy-preserving federated approach.
- The paper provides two datasets and benchmarks as a platform for further research on real-world federated visual classification.
- Domain shift, label noise, poor data quality, duplication, model size, bandwidth, and unreliable connections remain beyond the paper’s scope.
A.1 Hyperparameter Sensitivity
The appendix examines how hyperparameter sensitivity changes with client-distribution heterogeneity. Increasing non-identicalness reduces relative accuracy and narrows the high-performing hyperparameter region, while larger report goals broaden it.
- Figure 8 evaluates relative accuracy across effective learning rates, with each grid point selecting the best result over equivalent (β, η) combinations.The effective learning rate is defined as ηeff = η/(1 − β).
- Larger report goals K support good performance across a wider range of effective learning rates.More reporting clients provide gradients with less variance because the server observes more data.
- CIFAR-10 and CIFAR-100 show similar relative-accuracy characteristics despite differing class counts and centralized accuracies.
A.2 The Effect of Pretraining
The appendix compares federated training from ImageNet-pretrained weights with training from scratch on large visual classification models. Pretraining converges in far fewer communication rounds and reaches higher final relative accuracy, while efficient scratch training remains open.
- The study evaluates FedAvg and FedAvgM with different report goals on iNaturalist-Geo-3k, using relative accuracy against matching centralized upper bounds.Relative accuracy accounts for different final centralized accuracies from scratch and pretrained initialization.
- Pretraining requires orders of magnitude fewer communication rounds for convergence and yields higher final relative accuracy than training from scratch.
- 94% fewer rounds, 977 → 60, are needed for FedAvgM with K = 100 to reach 10% relative accuracy from pretrained weights versus FedAvg.FedAvgM also reaches 90% relative accuracy in 6.9× the rounds needed for 10%, compared with 20× for FedAvg.
- Efficient and effective federated training from scratch remains an open challenge despite evidence that it is possible.
B CIFAR-10/100 Dataset Details
The CIFAR-10 and CIFAR-100 datasets are partitioned into 100 clients with 500 training examples each, using Dirichlet-sampled class distributions to control client heterogeneity. The synthesis procedure also handles classes exhausted during sampling, and Figure 10 summarizes the resulting class-count distributions.
- Each CIFAR partition contains 100 clients with 500 training examples per client.
- Client class distributions q_k are sampled from Dir(αp), where α controls concentration and client identicalness.As α approaches infinity, clients resemble the prior p; as α approaches zero, clients tend toward sparse labels.
- The synthesis procedure removes classes exhausted during sampling and restricts subsequent clients to the remaining classes.
- Figure 10 shows client class counts for CIFAR-10 and CIFAR-100 partitions synthesized with different Dirichlet concentration parameters α.
C Experiment Run Time
Federated learning experiments are simulated in parallel on NVIDIA Tesla P100 GPUs. Communication rounds take roughly 0.5–2.0 seconds per reporting client, and a 5000-round Landmarks-User-160k run with K = 100 requires about 9 GPU-days.
- Experiments run in parallel on a cluster of NVIDIA Tesla P100 GPUs.
- 0.5–2.0 seconds per communication round per reporting client is the approximate runtime range.Runtime varies with model complexity, dataset, local steps E, and reporting clients per round K.
- About 9 GPU-days are required for 5000 rounds of the Landmarks-User-160k experiment with K = 100.