PaperlayerView exact revision on arXiv

Source-linked AI summary

Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment

Yan Gao, Mohammad Naseri, Javier Fernandez-Marques, Dimitris Stripelis, Lorenzo Sani, Davide Eynard, Fan Zhang, Hong Jia, Ting Dang, D. B. Emerson, Fatemeh Tavakoli, Ole Werger, Lars Wulfert, Petros Demetrakopoulos, Sofia Tsekeridou, InSeo Song, KangYoon Lee, Honghao Li, Lingjuan Lyu, John P Dickerson, Daniel Janes Beutel, Nicholas D. Lane

arXiv:2608.25114v1cs.LGcs.AI

TL;DR

FL benchmarking remains difficult to reproduce, compare, and extend because existing evaluations are often coupled to custom infrastructure and dominated by simulation. Flower Hub addresses this gap by packaging benchmarks as executable, versioned applications with standardized metadata and workflows, then demonstrates the approach across five domains and both simulation and deployment. The platform supports unified evaluation of task performance and system behavior, while results show task-dependent variation across aggregation strategies.

  • Problem

    FL benchmarking lacks reproducible, reusable packaging and portable evaluation across infrastructure and runtime settings.

  • Method

    Flower Hub decouples application logic from infrastructure and publishes executable, versioned benchmarks with standardized configurations, evaluation workflows, and system monitoring.

  • Results

    The platform demonstrates five realistic benchmarks across cross-silo and cross-device settings, with task-dependent performance variation across six aggregation strategies and execution in simulation and deployment.

  • Takeaways & Limitations

    Flower Hub provides a portable and reusable foundation for publishing, running, comparing, and extending federated benchmarks.

  • Takeaways & Limitations

    The study emphasizes baseline evaluation, does not cover all FL settings, and uses deployment experiments limited in scale.

Abstract

from arXiv · show

Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend. Existing evaluations are often tied to custom infrastructure, released as incomplete research code, and conducted primarily in simulation, which limits portability and practical relevance. We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications. We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation workflows. We instantiate this approach with a multi-domain benchmark suite spanning cross-silo and cross-device settings, and including tasks in medical imaging, financial tabular learning, legal instruction tuning, phishing URL detection, and audio tagging. We further demonstrate that the same benchmarking application can run across both simulation and deployment runtimes without changing the application code, enabling unified evaluation across varying learning environments. Beyond model quality, our benchmark design supports system-aware reporting, including runtime and communication metrics. This work advances benchmarking in FL settings from ad hoc code artifacts towards portable, executable, and reusable benchmark applications.

1 Introduction

Federated learning benchmarking is difficult to reproduce and reuse because infrastructure, application code, packaging, and evaluation workflows remain fragmented. Flower Hub addresses these challenges with executable benchmark applications that support standardized, portable evaluation across simulation and deployment.

  • Motivation: Distributed-system requirements force FL researchers to manage infrastructure alongside algorithms, increasing the engineering burden of benchmarking.This differs from centralized ML experiments, where collaborative distributed infrastructure is generally not required.
  • Motivation: Tightly coupled infrastructure and application code make existing FL benchmarks difficult to share, extend, and reproduce across teams.Reproduction can require sharing an entire infrastructure stack and substantial system expertise.
  • Motivation: Simulation dominates FL evaluation, while moving benchmarks to real deployment often requires substantial code changes and engineering overhead.The field also lacks a standardized format covering training code, evaluation protocols, distributed configurations, and runtime environments.
  • Flower Hub: Flower Hub decouples infrastructure from application logic and packages benchmarks with standardized training, evaluation, system configuration, and versioning.It also provides tools for measuring communication cost and latency.
  • Demonstration: Five benchmarks span healthcare, finance, security, automotive, and legal services across text, tabular, imaging, and audio modalities.The study evaluates six FL optimization algorithms in both simulation and deployment settings.

2 Flower Hub

Flower Hub provides an executable application model for publishing, versioning, and running federated benchmarks through shared infrastructure. Its standardized packages, dual-runtime execution, and monitoring tools support reproducible evaluation of both ML and system behavior.

  • Platform: Flower Hub represents each benchmark as an executable application that users can publish, download, and run through a unified interface.The platform is designed to shift FL benchmarking toward a collaborative ecosystem supporting reproducibility and reuse.
  • Application and infrastructure: Shared Flower infrastructure lets benchmark developers focus on ServerApp and ClientApp application logic instead of maintaining distributed-system code.ServerApp defines the FL workflow, while ClientApp defines local training, evaluation, and data handling.
  • Runtime portability: The same benchmark app runs in simulation or deployment with the unified flwr run command and without code modifications.Simulation launches the workflow locally, whereas deployment connects independently launched SuperNodes to a SuperLink.
  • Standardized benchmark package: Standardized packages specify schemas, configurations, pinned dependencies, and machine-readable metadata for task and system settings.Configurations cover datasets, models, hyperparameters, aggregation, partitioning, participation, rounds, and simulation resource requirements.
  • Versioning: Version control lets users run specific benchmark releases or receive an automatically selected version compatible with their local environment.This mechanism is intended to improve usability and reproducibility across systems.
  • Monitoring: Flower Hub monitors round latency, client training time, server aggregation time, communication cost, and client CPU/GPU memory in real time.Benchmark applications separately expose task-specific losses, accuracies, convergence statistics, and other ML observables.

3 Benchmark Development

The benchmark suite covers realistic cross-silo and cross-device FL tasks with heterogeneous data, modalities, and client structures. Lightweight pipelines evaluate multiple aggregation strategies while recording both task performance and system costs, including deployment consistency.

  • Benchmark Development: Five tasks cover medical segmentation, financial fraud detection, legal instruction tuning, phishing detection, and on-device audio tagging.Together they span healthcare, finance, legal services, security, and automotive-related settings across imaging, tabular, text, and audio data.
  • Medical image segmentation: Medical imaging uses site-partitioned multimodal glioma MRI scans, with each acquisition site representing a federated client.Training and held-out evaluation subsets are formed within each site.
  • Financial fraud detection: Financial transactions are assigned to five simulated banks by originating account, preventing account-level leakage across clients.Bank heterogeneity varies size, fraud prevalence, and active non-fraud account composition.
  • Legal instruction tuning: The legal benchmark assigns five legal NLP source tasks to separate client silos, creating differences in samples, task formulations, label spaces, and subdomains.
  • Cross-device benchmarks: Phishing URLs are canonicalized, deduplicated, and distributed across 100 clients, while audio clips preserve fsID groups across 50 clients.These constructions retain client imbalance and label-distribution heterogeneity for decentralized evaluation.
  • Training and evaluation: Task pipelines use Dice score, PR-AUC, token-level F1 score, ROC-AUC, and accuracy for the five benchmarks, respectively.These metrics align evaluation with segmentation, severe class imbalance, language modeling, phishing detection, and audio classification tasks.
  • Training and evaluation: Six aggregation methods are compared, including FedAvg as baseline, FedProx for client drift, FedAvgM with server momentum, and three adaptive FedOpt methods.
  • System evaluation: System evaluation records computational cost, communication overhead, memory usage, and runtime in addition to model performance.Client memory, training time, and end-to-end federated-round duration are measured.

4 Experimental Evaluation of Benchmarks

The benchmarks reveal strong task- and setting-dependent performance, while Flower Hub evaluates both predictive quality and system behavior across simulation and deployment. Results provide baseline references and expose communication, training, preprocessing, and deployment costs.

  • Evaluation Design: The evaluation measures model performance, system efficiency, and deployment behavior, including predictive quality, computational and communication costs, and reliable execution beyond simulation.Deployment uses independently deployed Flower SuperNodes without code modifications.
  • Model Performance Evaluation: No aggregation strategy consistently dominates across tasks, with FedProx strongest in most cases while FedOpt methods are often unstable or suboptimal.FedAvg and FedAvgM show similar convergence; legal instruction tuning is an exception for FedOpt methods, while severe class imbalance complicates financial fraud detection.
  • Model Performance Evaluation: Cross-silo medical segmentation and legal instruction tuning generally exceed 70%, whereas financial fraud detection performs worse under severe non-IID data.Full client participation and sufficient aggregate data benefit the stronger cross-silo tasks.
  • Model Performance Evaluation: Phishing URL detection exceeds 90% across methods, while audio tagging reaches only slightly above 50% accuracy.The contrast reflects the relative simplicity of phishing detection and the heterogeneous distributions and representation-learning difficulty of audio tagging.
  • Overall Findings: The reported results establish baseline performance and provide reference points for future research.Flower Hub also supports system monitoring during benchmark execution.
  • System Performance Evaluation: Communication cost primarily follows trainable parameter size, with the Medical task highest and Finance requiring approximately 0.01 GB.The Medical task uses a large 3D U-Net, whereas Finance uses a lightweight MLP.
  • System Performance Evaluation: Legal LLM fine-tuning has the longest client training time, but preprocessing and communication contribute substantially to end-to-end latency.Training time does not dominate total latency across tasks.
  • Deployment Evaluation: All benchmark applications execute in deployment without code modifications, with wall-clock time and cumulative communication tracked over federated rounds.Communication cost increases approximately linearly with the number of rounds.

5 Related Work

Prior FL research spans benchmark suites, execution frameworks, and reproducibility studies, but existing resources often remain tightly coupled repositories or execution substrates. Flower Hub addresses this missing systems layer as a platform for portable, executable, and reusable benchmark exchange.

  • Research Strands: Federated benchmarking research includes benchmark suites, frameworks and tooling, and evaluation or reproducibility studies.These strands address datasets and metrics, execution infrastructure, and methodological comparability or deployment.
  • Benchmark Suites: Benchmark suites broaden realism and domain coverage, but are typically released as standalone repositories with tightly coupled infrastructure.This limits reuse and standardization.
  • Frameworks and Tooling: Federated frameworks reduce engineering overhead but primarily function as execution substrates rather than standardized, portable benchmark definitions.Examples include TensorFlow Federated, FedLab, FedML, OpenFL, NVFlare, and APPFL.
  • Evaluation and Reproducibility: Evaluation and reproducibility efforts improve standardization and deployment, yet prior approaches do not fully address benchmark exchange and portability.Examples include FedEval, UniFed, MedPerf, and studies of the simulation-to-deployment gap.
  • Flower Hub: Flower Hub is a benchmark platform built on Flower that decouples application logic from infrastructure and packages benchmarks as executable, versioned units.It complements existing work by targeting the systems layer for exchange, portability, and long-term reuse.

6 Conclusion

Flower Hub packages federated benchmarks as reproducible applications that can run across simulation and deployment without code changes. Its Flower-based infrastructure separates coordination from task logic and supports system-aware evaluation, while the study remains limited in scope and deployment scale.

  • Conclusion: Flower Hub packages benchmarks as executable, versioned applications with standardized metadata and evaluation workflows.The platform supports reproducible benchmarking across federated and decentralized learning settings.
  • Conclusion: The platform demonstrates five cross-silo and cross-device benchmarks across multiple domains and reports task-dependent performance variation across six aggregation strategies.Built-in monitoring covers communication, memory, runtime, and deployment behavior.
  • Flower Infrastructure: SuperLink coordinates communication between ServerApp and connected SuperNodes without implementing learning logic.ServerApp defines client selection, configuration, aggregation, and federated optimization strategies such as FedAvg.
  • Flower Infrastructure: SuperNodes represent participating devices, machines, or institutional silos and initiate outbound connections, supporting restricted or firewall-protected environments.They retrieve tasks, execute them, and return results through the SuperLink.
  • Flower Infrastructure: ClientApp performs local training, evaluation, preprocessing, and local-data access, while SuperExec schedules and manages application processes.ClientApp is short-lived and launched on demand when a SuperNode participates.
  • Execution Workflow: Deployed runs establish a federation through SuperLink and SuperNodes, then launch ServerApp and selected ClientApps for coordinated training.Communication occurs indirectly through the infrastructure components.
  • Execution Modes: Simulation uses a managed local SuperLink and simulated clients, whereas deployment uses independent SuperLink and SuperNode processes across distributed environments.The two runtimes support different execution environments for the same federated workflow.
  • Flower Infrastructure: Flower infrastructure separates persistent networking and coordination from short-lived application processes implementing federated learning logic.SuperLink and SuperNodes provide communication and coordination, while SuperExec manages lifecycles and ServerApp and ClientApp implement learning tasks.

B Flower Datasets

Flower Datasets provides a reproducible data-preparation layer for federated experiments, while Flower Hub packages applications for fetching, execution, publication, and versioned reuse.

  • Flower Datasets: Flower Datasets transforms centralized or local data into client-specific partitions for federated learning, analytics, and evaluation.It supports public datasets, local CSV, JSON, image, audio, and in-memory sources.
  • Flower Datasets: FederatedDataset integrates dataset loading, preprocessing, and configurable IID or non-IID partitioning.Supported strategies include IidPartitioner, DirichletPartitioner, and PathologicalPartitioner.
  • Flower Datasets: Training data can be partitioned across clients while the test split remains centralized for evaluation.Resulting partitions use Hugging Face Dataset objects compatible with common machine-learning frameworks.
  • Using Flower Hub: Users fetch applications through the Flower CLI and run them in Simulation or Deployment Runtime environments.Deployment requires configured SuperNodes and may require local dataset paths and application-specific parameters.
  • Publishing applications: Published applications retain version history, with publication requiring project metadata, source validation, authentication, and version increments for updates.Applications publish source files rather than prebuilt artifacts, and users can access specific releases.

C.4 Signing an Application from Flower Hub

Flower Hub lets authenticated reviewers sign published applications using cryptographic keys, creating decentralized, inspectable verification metadata tied to specific versions.

  • Signing an application: Application signing attaches reviewer-generated verification metadata to applications already published on Flower Hub.The signing and verification features are currently preview features and may evolve.
  • Signing an application: Reviewers use a cryptographic public–private key pair and can sign either the latest application version or a specified release.The Flower CLI supports inspection and signing through the flwr app review command.
  • Signing an application: The generated signature covers the Flower App Bundle digest and timestamp, tying verification to a specific application artifact.The reviewer submits the signature with the application identifier and version.
  • Decentralized verification: Any authenticated user with a registered signing key can review applications published by other accounts.Application pages display signatures so users can inspect verifying entities and decide which signers to trust.
  • Decentralized verification: The decentralized model supports layered trust across organizations, devices, and data-owning sites without guaranteeing application trustworthiness.Verification functions as a transparency mechanism rather than a guarantee.
  • Application packaging: Benchmark applications use standardized packages with pinned dependencies and configurable metadata and hyperparameters.The configuration separates reproducibility-oriented metadata from federated training and evaluation settings.

E Application Detail Page on Flower Hub

Flower Hub application detail pages expose benchmark identifiers and comprehensive project information, supporting open access, reuse, and extension.

  • Application listings: Five benchmark applications are publicly listed on Flower Hub: medical segmentation, financial fraud, legal language modeling, phishing detection, and audio tagging.The applications are identified by their Flower Hub account paths.
  • Application detail pages: Each application detail page provides metadata, source code, documentation, and usage statistics.The pages are designed to make benchmark applications transparent and accessible.
  • Application detail pages: The applications are fully open source, facilitating community engagement, reuse, and extension.The paper connects this openness with the robustness and sustainability of the Flower Hub ecosystem.
  • Benchmark evaluation: Figure 11 compares evaluation metrics across federated learning rounds for financial fraud detection and audio tagging.Its caption reports suboptimal or unstable performance for FedAdam, FedAdagrad, and FedYogi on these tasks.

F.1 Client-level Label Distribution of FL Datasets

The benchmark datasets exhibit pronounced client-level heterogeneity, creating challenging evaluation settings that are intended to reflect realistic federated conditions.

  • Audio tagging: No audio-tagging client contains the full class set, making globally consistent model learning more difficult.The missing classes are distributed across clients rather than available within each individual client.
  • Financial fraud detection: All financial-fraud clients have fraud rates below 0.2%, producing an extremely imbalanced benchmark condition.The reported rates are intended to reflect realistic real-world conditions.
  • Phishing URL detection: Phishing rates vary substantially across clients, introducing additional heterogeneity into phishing URL detection.This client-level variation contributes to the benchmark’s challenging data distribution.
  • Evaluation conditions: The benchmark uses commonly adopted hyperparameter settings without extensive tuning, so reported results should be interpreted as baseline performance.The configurations provide a reference for future optimization and exploration.
  • Deployment setting: Deployment experiments use geographically distributed clients and heterogeneous GPU resources to approximate realistic cross-region federated learning.The server is hosted in Canada, while clients span the United States, Poland, Spain, and Japan.

G Additional Experimental Results

Additional experiments compare optimizer behavior on financial fraud detection and audio tagging, while deployment results report client-side memory usage and its task-dependent drivers.

  • FedProx achieves the best final performance on both financial fraud detection and audio tagging tasks.
  • FedAvg, FedProx, and FedAvgM show similar convergence, whereas FedAdam, FedAdagrad, and FedYogi perform poorly and unstably.The tasks have highly non-IID client distributions, and FedOpt methods are sensitive to hyperparameter choices.
  • Deployment experiments report both peak and average client-side memory usage.CPU memory is mainly driven by preprocessing and loading, while GPU memory is dominated by model training.
  • Financial fraud detection uses significantly less GPU memory than the other tasks because it employs a lightweight MLP model.
Loading 2608.25114v1…