PaperlayerView exact revision on arXiv

Source-linked AI summary

FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings

Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, Boris Muzellec, Constantin Philippenko, Santiago Silva, Maria Teleńczuk, Shadi Albarqouni, Salman Avestimehr, Aurélien Bellet, Aymeric Dieuleveut, Martin Jaggi, Sai Praneeth Karimireddy, Marco Lorenzi, Giovanni Neglia, Marc Tommasi, Mathieu Andreux

arXiv:2210.04620v3cs.LGcs.CV

TL;DR

Realistic healthcare cross-silo FL datasets are scarce, despite privacy constraints that prevent pooling data across hospitals and other sites. The paper introduces FLamby, a modular suite of seven naturally partitioned healthcare datasets with baseline code and benchmarking guidance, and illustrates its use by evaluating standard FL strategies. Results vary across datasets: some strategies outperform local training, but pooled training is generally unmatched except for two tabular datasets; the suite’s current scope remains limited in domains, backends, and FL constraints.

  • Problem

    Realistic healthcare cross-silo datasets are scarce, while privacy and strategic concerns prevent many site-held datasets from being centralized.

  • Method

    FLamby provides seven naturally partitioned healthcare datasets spanning tasks and modalities, with baseline training code, reproducible guidelines, and benchmarked FL strategies.

  • Results

    Across datasets, no local or FL strategy matches pooled training except Fed-TCGA-BRCA and Fed-Heart-Disease; some strategies beat local training on Fed-KITS2019 and Fed-ISIC2019.

  • Takeaways & Limitations

    FLamby supplies a modular benchmark reflecting heterogeneous healthcare cross-silo use cases to advance understanding of how data collection affects FL performance.

  • Takeaways & Limitations

    FLamby currently covers only healthcare, provides PyTorch code, and does not integrate all cross-silo constraints, especially privacy aspects.

Abstract

from arXiv · show

Federated Learning (FL) is a novel approach enabling several clients holding sensitive data to collaboratively train machine learning models, without centralizing data. The cross-silo FL setting corresponds to the case of few ($2$--$50$) reliable clients, each holding medium to large datasets, and is typically found in applications such as healthcare, finance, or industry. While previous works have proposed representative datasets for cross-device FL, few realistic healthcare cross-silo FL datasets exist, thereby slowing algorithmic research in this critical application. In this work, we propose a novel cross-silo dataset suite focused on healthcare, FLamby (Federated Learning AMple Benchmark of Your cross-silo strategies), to bridge the gap between theory and practice of cross-silo FL. FLamby encompasses 7 healthcare datasets with natural splits, covering multiple tasks, modalities, and data volumes, each accompanied with baseline training code. As an illustration, we additionally benchmark standard FL algorithms on all datasets. Our flexible and modular suite allows researchers to easily download datasets, reproduce results and re-use the different components for their research. FLamby is available at~\url{www.github.com/owkin/flamby}.

1 Introduction

Cross-silo FL addresses privacy-constrained collaboration among a few large clients, but realistic healthcare datasets with natural heterogeneity remain scarce. FLamby responds with an open-source suite of seven naturally partitioned healthcare datasets, reproducible benchmarking guidance, and accessible code.

  • Cross-silo FL enables collaboration among fewer, larger clients such as hospitals when local data cannot be centralized because of privacy or strategic concerns.
  • Publicly available cross-silo datasets are scarce, and synthetic partitions may not reproduce the complex heterogeneity of real-world healthcare data.
  • FLamby is an open-source healthcare cross-silo dataset suite with natural partitions, code examples, and benchmarking guidelines.
  • 7 datasets span classification, segmentation, and survival tasks across multiple application domains, modalities, scales, and natural client splits.
  • The suite provides fair-comparison guidelines, illustrative benchmark results, reproducible code, and integration with different FL frameworks.

2 Related Work

Existing FL resources provide natural partitions mainly for cross-device settings, while cross-silo research has relied on scarce standalone datasets or artificial partitions. FLamby consolidates seven medical datasets with real partitions into one reproducible benchmark suite.

  • Synthetic partition methods assign data using class-based, Dirichlet, hierarchical, or feature-clustering heuristics, but may miss complex statistical heterogeneity.
  • LEAF and TensorFlow Federated provide naturally partitioned cross-device benchmarks spanning language, sentiment, image, character, and related tasks.
  • Before FLamby, extensive natural-split cross-silo benchmarks were unavailable, although isolated works used geolocation, sensors, individuals, cameras, or other real partitions.
  • Medical studies separately created natural hospital-based splits from datasets including Camelyon, CheXpert, LIDC, chest X-ray, IXI, diabetic retinopathy, and TCGA.
  • FLamby reunites seven medical datasets and adds reproducible code and precise benchmarking guidelines to connect prior and future cross-silo FL work.

3 The FLamby Dataset Suite

FLamby is a modular suite combining naturally split healthcare datasets, baseline models, standardized APIs, and benchmarkable FL strategies. Its datasets span diverse modalities and exhibit client heterogeneity that can be obvious, subtle, modality-specific, or absent in particular measurements.

  • The suite combines datasets with baseline models and FL strategies with benchmarking code through standardized APIs compatible with existing FL libraries.
  • Each dataset defines a natural client split, task, metric, reproducible per-client train/test sets, and a pooled-training baseline model.
  • FL strategies are supplied as plain Python with a standardized API compatible with the dataset API, enabling benchmarking across datasets.
  • Fed-Camelyon16: Fed-Camelyon16 uses two hospital clients for slide classification measured by AUC, with weakly supervised DeepMIL applied to tile-level ImageNet-pretrained ResNet50 features.
  • Fed-LIDC-IDRI: Fed-LIDC-IDRI uses four scanner-manufacturer clients for lung-nodule segmentation measured by DICE, with a VNet baseline trained on 3D patches.
  • Dataset heterogeneity: Client heterogeneity varies by modality: Camelyon16 RGB distributions differ strongly, MRI intensities differ visibly, some ISIC clients form isolated clusters, and heart-disease features vary.
  • Dataset heterogeneity: Voxel-intensity distributions show no significant client differences for Fed-LIDC-IDRI and Fed-KITS2019, while TCGA-BRCA survival curves differ significantly for some but not all clients.

4 FL Benchmark Example with FLamby

The benchmark evaluates FL strategies under standardized, reproducible conditions across FLamby datasets, revealing substantially different outcomes across datasets. Federated methods generally improve over local training but often remain below pooled performance, with strategy rankings also varying.

  • Benchmark design: The benchmark evaluates one global model on per-client test sets, averaging performance across clients without personalization.All experiments use the same model and ML hyperparameters as pooled and local baselines while tuning FL-specific parameters separately.
  • Benchmark design: The benchmark fixes local updates at E = 100 and sets communication rounds to approximate the pooled training effort.The round calculation uses pooled epochs, total distributed samples, clients, batch size, and local updates; restricting rounds may affect convergence.
  • Interpretation: The datasets exhibit strikingly different benchmark behavior, underscoring the difficulty of tuning FL strategies on heterogeneous cross-silo data.The authors call for more robust FL strategies in this setting.
  • Results: Federated strategies generally improve over local baselines but fall short of pooled performance, except for Fed-TCGA-BRCA and Fed-Heart-Disease.For Fed-Camelyon16, Fed-LIDC-IDRI, and Fed-IXI, the results show no benefit from collaboration.
  • Results: FedOpt variants achieve the best performance where an FL strategy outperforms pooled baselines, while Cyclic systematically underperforms other strategies.FedAvg is not usually top-performing but remains competitive, except on Fed-Camelyon16 and Fed-IXI.

5 Conclusion

FLamby provides a modular healthcare cross-silo FL benchmark spanning heterogeneous datasets, tasks, and modalities. Its current scope centers on natural-split heterogeneity, while privacy and broader application coverage remain future directions.

  • Contribution: FLamby is a modular benchmark covering multiple healthcare tasks, data modalities, and real-world cross-silo heterogeneity.The benchmark is intended to advance understanding of how healthcare data collection affects FL performance.
  • Scope: The current suite is limited to healthcare datasets and provides only PyTorch machine-learning backends.The authors identify datasets from other domains and support for TensorFlow or JAX as possible future extensions.
  • Scope: The benchmark mainly studies heterogeneity induced by natural splits and does not yet integrate all cross-silo constraints, especially privacy aspects.Future directions include Differential Privacy, Secure Aggregation, Personalized FL, and communication constraints.

Checklist

The checklist records affirmative disclosures on claims, reproducibility, asset use, ethics, licensing, and maintenance, while noting dataset-bias risks and absent privacy metrics.

  • Disclosure: The authors state that the paper reports limitations, societal-impact considerations, ethics compliance, and that theoretical-results requirements are not applicable.They identify the work as repurposing existing assets rather than collecting new data or conducting human-subject research.
  • Reproducibility: The authors report providing code, data, training details, and reproducibility materials, including error bars and compute information.They point readers to the abstract, supplementary material, code, and Appendix J.1.
  • Assets and licensing: Existing assets are cited and licensed, and users must agree to dataset terms before downloading them.The repository links each dataset to its data terms or license and prompts users for explicit agreement.
  • Broader impact: The study reports limited risks from repurposing open-source materials, but five image datasets may contain biases that are difficult to trace to ethnicity or gender.Pseudonymisation and the nature of image-based medical records limit checking these potential biases.
  • Maintenance: The repository describes bug fixes, security updates, monthly dataset reviews, and community-contribution review as part of its maintenance plan.Dataset review may include changes prompted by patient opt-outs or ethical concerns, including full revocation when necessary.

C.5 Baseline, loss function and evaluation

The appendix describes dataset-specific baselines, losses, evaluation metrics, preprocessing, client construction, and hyperparameter procedures for healthcare benchmarks.

  • Fed-Camelyon16: The Camelyon16 baseline uses DeepMIL with attention-weighted patch features, binary cross entropy loss, and AUC evaluation.The model is trained end to end on extracted patch representations.
  • Optimization: FL-specific hyperparameters are tuned separately from ML hyperparameters, using client learning rates and strategy-specific server rates or FedProx µ values.The pooled benchmark uses the stated configuration without further tuning, whereas FL strategies search predefined grids.
  • Fed-LIDC-IDRI: LIDC-IDRI is split across four scanner-manufacturer clients, with 661 GE, 205 Siemens, 69 Toshiba, and 74 Philips samples.Each client is further divided into stratified 80% training and 20% testing sets.
  • Fed-LIDC-IDRI: LIDC-IDRI scans are resized to 384 × 384 × 384, while preprocessing clips Hounsfield units to [−1024, 600] and normalizes them to [0, 1].Cropping and reflection-padding standardize dimensions, and the same spatial operation is applied to ground-truth masks.

D.4 Task

The section describes the Fed-LIDC-IDRI and Fed-IXI segmentation tasks, their data sources, preprocessing, evaluation, and training constraints.

  • LIDC task: The LIDC task segments lung nodules from CT scans using sampled patches to reduce GPU memory requirements.Images are sampled as 128 × 128 × 128 patches, with class-imbalance handling during sampling.
  • LIDC task: The LIDC objective combines DICE maximization with a balanced cross-entropy term because DICE alone performed poorly on non-nodule regions.The added term addresses the strong imbalance between rare nodules and background.
  • LIDC baseline: The LIDC baseline is a VNet with dropout, sigmoid voxel probabilities, RMSprop, an initial learning rate of 10^-2, and 100 epochs.Training takes approximately 48 hours on a P100.
  • LIDC baseline: LIDC federated training takes approximately 70 hours on a P100, limiting hyperparameter search compared with other datasets.The final parameters are reported in the paper’s appendix.
  • IXI task: Fed-IXI provides T1-weighted brain MR images for 566 subjects with corresponding binary brain segmentation masks.The masks were obtained through automatic whole-brain extraction using ROBEX.
  • IXI task: Fed-IXI contains images from Guys, HH, and IOP hospitals, with stratified 80%/20% train-test splits and one test set per hospital.Guys and HH use Philips 1.5T systems, while IOP uses a GE 1.5T system.
  • IXI task: Fed-IXI preprocessing aligns scans to the MNI template, reorients them, normalizes intensities, and resizes volumes to 48 × 60 × 48 voxels.The segmentation task is evaluated with the DICE score.

E.5 Baseline, loss function and evaluation

This section presents the Fed-TCGA-BRCA survival task and its linear Cox baseline, evaluation metric, optimization setup, and data provenance.

  • Baseline: The UNet baseline is optimized with AdamW at learning rate 10^-3 and batch size 2, using a specified convolutional architecture.The architecture includes batch normalization, max-pooling, linear upsampling, zero-padding, PReLU activations, and three encoding blocks.
  • Task: Fed-TCGA-BRCA uses clinical tabular data from 1,066 breast-cancer patients, represented by 39 input features and split by geographic extraction-site regions.Patients are labeled by overall survival time and death-event status.
  • Data partition: Fed-TCGA-BRCA clients correspond to six geographic regions, with patient counts ranging from 51 to 311.The regions are four areas in the USA, Canada, and Europe.
  • Task: The dataset terms prohibit attempts to identify the individual human research participants represented in the data.TCGA policies also emphasize privacy protection and HIPAA compliance, while genetic data are excluded because access is restricted.
  • Preprocessing: The pooled TCGA-BRCA benchmark uses preprocessed data already included in the FLamby repository.No additional downloading or preprocessing is required for the pooled dataset.
  • Evaluation: The survival task predicts outcomes from clinical covariates and is evaluated as a ranking problem using the concordance index, which accounts for right-censoring.The study’s censoring ratio is 86%.
  • Baseline: The linear Cox model assigns each patient a risk score η_i = β^T x_i, with β learned by minimizing the negative Cox partial log-likelihood.The pooled baseline uses Adam with learning rate 0.1, batch size 8, and 30 epochs.

F.5 Baseline, loss function and evaluation

The section introduces survival-analysis quantities, defines the Fed-TCGA-BRCA evaluation and baseline, and identifies a limitation in federating the Cox objective.

  • Background: Survival analysis models time-to-death using a survival function and a hazard function that describes event occurrence among patients who have survived to time t.Patients are characterized by covariates x, an observed time t, and censoring indicator δ.
  • Baseline: The baseline uses a linear Cox proportional-hazard model whose parameters are learned by gradient descent on the negative Cox partial log-likelihood.The model trains in a matter of seconds on modern CPUs.
  • Limitation: The Cox partial log-likelihood is not separable across samples or clients, but the baseline independently computes each client’s objective and federates the resulting local gradients.The authors refer to prior work for a more rigorous federated survival-analysis treatment.
  • Fed-KiTS19: Fed-KiTS19 contains CT scans and segmentation masks from 210 patients across 77 hospitals, reduced to six clients by removing hospitals with fewer than 10 training samples.The hospital distribution was obtained separately from one challenge organizer because it was absent from the original dataset.
  • Fed-KiTS19: Fed-KiTS19 segments kidneys and tumors, scoring the average of their two DICE scores after intensity clipping and z-score normalization.The background is labeled 0, kidney 1, and tumor 2.

G.5 Baseline, loss function and evaluation

This section details the KiTS19 training setup, including memory-efficient patch sampling, the combined loss, nnU-Net baseline, optimization, and hyperparameter search.

  • Sampling: KiTS19 training extracts [64, 192, 192] patches because image sizes are heterogeneous and foreground voxels are sparse.The median resampled patient volume is [116, 282, 282].
  • Loss function: The KiTS19 loss sums DICE and cross-entropy losses with equal weight for the two foreground classes.The classes are kidney and tumor, with ε = 1e-5 used in the formulation.
  • Baseline: The pooled baseline uses nnU-Net with the architecture specified for KiTS19 and trains in under 24 hours on a P100.The model uses five convolutional and pooling-kernel configurations.
  • Optimization: The model is trained with Adam at learning rate 0.0003 for 500 epochs and evaluated using the average kidney-and-tumor DICE score.The authors note that 8,000 epochs can yield higher performance at greater computational cost.
  • Hyperparameter search: For Cyclic and FedAvg, hyperparameter search selected a learning rate of 0.3 as the best value among the tested candidates.Other federated strategies used separate learning-rate and server-learning-rate search spaces.

H.1 Dataset description

Fed-ISIC2019 uses naturally partitioned dermoscopy data from six hospital or imaging-site clients, with substantial class imbalance and standardized preprocessing. The broader dataset descriptions also include Fed-Heart-Disease, a binary heart-disease classification task based on records from four hospitals.

  • Fed-ISIC2019: 23,247 public-train dermoscopy images are used from ISIC2019 after restricting the original 25,331-image dataset because of metadata availability.The train-test split is static.
  • Fed-ISIC2019: Six Fed-ISIC2019 clients represent distinct sites or imaging technologies, ranging from 12,413 images in the largest client to 439 in the smallest.Three clients come from imaging systems used by the ViDIR Group, and three from clinical sites in Australia, Spain, and New York.
  • Fed-ISIC2019: The ISIC2019 baseline fine-tunes an ImageNet-pretrained EfficientNet using weighted focal loss and image augmentations after resizing and color-consistency preprocessing.The weighted focal-loss class weights are inverse class proportions calculated over the pooled dataset, with γ chosen as 2.
  • Fed-ISIC2019: The ISIC2019 task is eight-class skin-lesion classification, with class prevalence ranging from 49% to less than 1%.Ground truth is established through histopathology, follow-up examination, expert consensus, or microscopy.
  • Fed-Heart-Disease: Fed-Heart-Disease contains 920 patient records from four hospitals and defines a binary classification task predicting disease presence.Preprocessing removes features and records with missing values, dummy-encodes two categorical features, and normalizes features per center.

I.5 Baseline, Loss Function, and Evaluation

This section establishes baseline models, federated optimization, evaluation procedures, and extension examples for FLamby. It also reports illustrative effects of synthetic client splitting, differential privacy, personalization, and heterogeneity measurement.

  • Baseline, Loss Function, and Evaluation: The logistic-regression baseline predicts labels with a sigmoid and evaluates accuracy after thresholding predictions at 0.5.The pooled benchmark uses Adam with learning rate 0.001, batch size 4, and 50 epochs.
  • Baseline, Loss Function, and Evaluation: Federated strategies use SGD with grid-searched hyperparameters, including learning rates for FedAvg, Cyclic, FedYogi, FedAdam, FedAdagrad, and Scaffold.FedProx additionally searches its μ parameter, and the selected hyperparameters are reported in Section J.
  • Implementation: FLamby’s standardized API supports sequential client simulation, GPU acceleration, and integration with dedicated federated-learning libraries for more realistic communication experiments.The suite is designed as a lightweight codebase, while dedicated libraries are encouraged for communication-constraint studies.
  • Benchmarking: Figure 2 uses E = 100 local updates for every strategy and dataset, with five independent random-seed runs except for Fed-LIDC-IDRI.Computational resources prevented repeated training for the largest dataset.
  • Synthetic Dataset Splits: Synthetic Dirichlet splitting redistributes samples from original clients to a desired number of clients, with α ≥ 1/2 recommended to avoid empty clients.The procedure applies to both train and test sets, and lower α produces sharper probability distributions.
  • Differential Privacy: Differentially private FedAvg performance diminishes as ϵ approaches 0, especially for small δ, relative to regular FedAvg without privacy.The example uses Fed-Heart-Disease and compares DP-FedAvg with a same-initialization baseline without differential privacy.
  • Personalized Federated Learning: FedAvg followed by local fine-tuning improves performance on Fed-Heart-Disease and Fed-ISIC2019 but slightly degrades it on Fed-Camelyon16.The authors hypothesize that fine-tuning overfits local training datasets in the Fed-Camelyon16 case.
  • Heterogeneity Analysis: Heterogeneity is analyzed with client-level statistical distributions, pairwise distances, and comparisons between natural and uniformly distributed synthetic splits.Rescaled distances indicate which clients are closest or most divergent and the degree of within-dataset heterogeneity.

M.3 Datasets analysis

FLamby measures client heterogeneity through pairwise distances for features and labels, while accounting for client-size imbalance. The datasets exhibit varied relationships between feature and label similarity, with several distinctive client configurations.

  • Pairwise-distance matrices compare clients’ feature and label heterogeneity, with column widths proportional to client sample size.The matrices include an i.i.d. baseline, and Table 17 additionally reports mean, maximum, and sample-distribution entropy measures.
  • Camelyon16 has highly heterogeneous features but i.i.d. labels, indicating that different features led to close labels.
  • IXI contains two close clients and a smallest-client outsider in both feature and label distances.
  • TCGA-BRCA has relatively close features among clients 0–3, while client 1 differs in labels and clients 4–5 combine distinct features with nearly identical labels.
  • Kits19 is homogeneous for both features and labels, whereas Isic2019 combines a feature-outlying client with label differences among clients having close features.
  • Heart disease shows limited, similarly scaled feature and label heterogeneity overall, but the smallest specialized hospital is an outsider and clients 1 and 2 differ most in labels.
  • The reported heterogeneity measurements are indicative rather than exhaustive because alternative measures and PCA representations may yield different similarity conclusions.
Loading 2210.04620v3…