Source-linked AI summary
LEAF: A Benchmark for Federated Settings
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, Ameet Talwalkar
TL;DR
Federated learning research needs benchmarks that reflect the scale and heterogeneity of real device-generated data. LEAF provides a modular framework of realistic open-source datasets, evaluation metrics, and reference implementations, with examples showing reproducibility and granular performance analysis. The framework is intended to support reasoning about learning methods under more realistic assumptions than previous benchmarks.
Problem
Existing benchmarks and evaluations often fail to capture the realistic statistical and systems characteristics of federated networks.
Method
LEAF builds a modular benchmark from open-source federated datasets, statistical and systems metrics, and reference implementations.
Results
LEAF reproduces prior FedAvg convergence behavior and provides granular metrics revealing distributional and communication-versus-computation differences across federated methods.
Takeaways & Limitations
LEAF supports research and practice in federated, meta-learning, and multi-task settings under more realistic assumptions than previous benchmarks.
Abstract
from arXiv · showhide
Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose LEAF, a modular benchmarking framework for learning in federated settings. LEAF includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.
1 Introduction
Federated data creates statistical, systems, privacy, and security challenges that existing benchmarks often fail to represent realistically. LEAF addresses this gap with a modular framework for realistic federated evaluation.
- Challenges: Federated devices generate heterogeneous data distributions and unequal numbers of data points across users.
- Challenges: Federated networks involve many more devices than typical distributed systems, with heterogeneous storage, computation, communication, and power constraints.
- Challenges: Privacy and security requirements must be balanced with statistical accuracy, scalability, and efficiency when learning from personally generated data.
- Benchmark gap: Existing evaluations often rely on artificial datasets, proprietary federated data, or benchmarks that do not capture realistic federated conditions.
- LEAF's aim: LEAF bridges this gap through realistic datasets, a clear evaluation methodology, and a modular framework for massively distributed federated networks.
2 LEAF
LEAF is an open-source benchmark organized around datasets, metrics, and reference implementations for federated experimentation. Its datasets are designed to reflect large-scale, naturally partitioned, and statistically skewed device data.
- Framework: LEAF combines open-source datasets, statistical and systems metrics, and reference implementations in a modular benchmark.
- Datasets: Its datasets use natural device or user keys, span thousands to millions of devices, and contain skewed numbers of data points per device.
- Datasets: The six datasets are FEMNIST, Sentiment140, Shakespeare, CelebA, Reddit, and a more challenging Synthetic dataset.
- Datasets: LEAF supplies preprocessing scripts and small or full dataset versions for prototyping and final testing.
- Metrics: Its metrics include device-level performance percentiles, natural-hierarchy stratification, and measures of computing resources.
- Reference implementations: Reference implementations currently cover minibatch SGD, FedAvg, and Mocha, with logs supporting statistical and systems-metric analysis.
3 LEAF in action
LEAF demonstrates reproducible experiments, granular evaluation, and modular reuse across multiple learning pipelines. Its examples expose both convergence behavior and differences across users and system-resource dimensions.
- Reproducibility: FedAvg on Shakespeare reproduces divergence in training loss as the number of local epochs increases.
- Granular metrics: With Sentiment140 users having k = 3 samples, median performance declines slightly while the 25th percentile declines dramatically.
- Granular metrics: For FEMNIST, LEAF measures the FLOPS and uploaded bytes required for minibatch SGD and FedAvg to reach per-sample accuracy 0.75.
- Modularity: LEAF's dataset module supports local-model, pooled-data, and Reptile pipelines across CelebA, Synthetic, Reddit, and FEMNIST.
4 Conclusions
LEAF is a modular framework for learning in federated settings, designed to support more realistic research across distributed-device learning paradigms. Its datasets, metrics, and reference implementations are intended to enable grounded evaluation and continued progress.
- LEAF targets federated settings marked by massively distributed networks of devices.
- LEAF supports federated learning, meta-learning, multi-task learning, and on-device learning.
- Researchers can use LEAF to assess proposed solutions under more realistic assumptions than previous benchmarks.
- LEAF’s modularity is demonstrated through new experimental pipelines and sample test accuracies reported beyond FedAvg.
A Synthetic Dataset
The synthetic dataset creates tasks with task-dependent models clustered around multiple centers, introducing heterogeneity intended to challenge current meta-learning methods. Generation begins from user-specified device and cluster-probability inputs, then samples cluster structure, task parameters, and bounded task sizes.
- The dataset is designed so true task models are task-dependent and clustered around more than one center.Its additional heterogeneity is intended to make current meta-learning methods such as Reptile fail.
- Dataset inputs: Users specify the desired number of devices T ≥1 and positive cluster probabilities.
- Dataset construction: Cluster means are sampled hierarchically, task parameters are projected through a random matrix Q, and feature covariance uses diagonal entries Σ_i,i = i^-1.2.The construction samples cluster means from Gaussian distributions, draws Q from a standard Gaussian, and sets the diagonal covariance entries by index.
- Task generation: Each task samples a cluster center according to the input probabilities, draws its parameter vector around that center, and maps it through Q.
- Task generation: Task sample counts are generated from a log-normal distribution and clipped to the range implied by nt = min(mt + 5, 1000).This imposes lower and upper bounds on the number of samples per task.
- Task generation: The generator then samples task-specific latent variables and feature vectors, adding Gaussian noise and padding features to account for the intercept.
B Experiment Details
The experiments use dataset-specific models, training procedures, evaluation metrics, and data splits across Shakespeare, Sent140, FEMNIST, CelebA, and additional pipelines. Reported configurations include subsampling, client participation schedules, learning rates, training rounds, and repeated runs.
- The section documents implementation details for experiments presented in Section 3.
- Shakespeare convergence: The Shakespeare convergence experiment subsamples 118 devices and uses an 8-dimensional character embedding followed by a two-layer LSTM with 256 units per layer.The model uses sequence length 80, dot-product vocabulary scoring, softmax, and AccuracyTop1 evaluation.
- Statistical and systems analyses: Sent140 uses bag-of-words logistic regression with learning rate 3 · 10^-4, while FEMNIST uses two convolutional layers, pooling, and a 2048-unit dense layer.For FEMNIST, FedAvg uses learning rate 4 · 10^-3 and minibatch SGD uses 6 · 10^-2.
- Additional pipelines: Table 2 experiments use 60% training, 20% validation, and 20% test data per user.The reported results are evaluated on the test set, with hyperparameters varying by experiment.
- Additional pipelines: CelebA uses 10% of clients, FedAvg trains 10 clients per round for 100 rounds, and results are averaged over 5 runs.Local models explore four learning rates, while the FedAvg model uses a best learning rate of 0.001.
- Additional pipelines: The additional sequence-based pipeline uses sequence length 10, AccuracyTop1 evaluation, and treats predictions of unknown and padding tokens as incorrect.
- Additional pipelines: FEMNIST runs each algorithm for 1,000 rounds with 5 clients per round, local learning rate 10^-3, and five mini-batches of size 10.Evaluation uses an unseen set of test devices.