Source-linked AI summary
MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
Alexandros Karargyris, Renato Umeton, Micah J. Sheller, Alejandro Aristizabal, Johnu George, Srini Bala, Daniel J. Beutel, Victor Bittorf, Akshay Chaudhari, Alexander Chowdhury, Cody Coleman, Bala Desinghu, Gregory Diamos, Debo Dutta, Diane Feddema, Grigori Fursin, Junyi Guo, Xinyuan Huang, David Kanter, Satyananda Kashyap, Nicholas Lane, Indranil Mallick, Pietro Mascagni, Virendra Mehta, Vivek Natarajan, Nikola Nikolov, Nicolas Padoy, Gennady Pekhimenko, Vijay Janapa Reddi, G Anthony Reina, Pablo Ribalta, Jacob Rosenthal, Abhishek Singh, Jayaraman J. Thiagarajan, Anna Wuest, Maria Xenochristou, Daguang Xu, Poonam Yadav, Michael Rosenthal, Massimo Loda, Jason M. Johnson, Peter Mattson
TL;DR
Medical AI lacks systematic performance measurement on large-scale heterogeneous data, while existing evaluations may focus on narrow datasets and tasks. MedPerf addresses this gap with an open federated-evaluation framework that securely evaluates models at multiple facilities and aggregates metrics without sharing patient data. The paper presents this approach as a privacy-prioritizing, human-supervised platform for reproducible and broader medical AI benchmarking.
Problem
Medical AI requires systematic evaluation on large-scale heterogeneous data, but existing challenges can focus on single datasets and tasks that do not reflect diverse real-world use cases.
Method
MedPerf coordinates federated evaluation by securely distributing models to data owners, evaluating them in controlled settings, and aggregating evaluation metrics without sharing patient data.
Results
The platform is designed to reduce privacy and data-misuse risks while enabling evaluation across multiple facilities and diverse patient populations.
Takeaways & Limitations
MedPerf aims to provide reproducible benchmarks that broaden performance assessment and support more reliable medical AI deployment decisions.
Takeaways & Limitations
Federated learning can leak data through model parameters, and mitigation research remains ongoing, slowing adoption.
Abstract
from arXiv · showhide
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.
2 Challenges: Risk, Cost and Uncertain Return
Medical AI data sharing is constrained by privacy and regulatory risks, substantial preparation costs, and uncertain returns from evaluations that may not reflect real-world diversity.
- Risk: Data sharing exposes providers to liability, regulatory, and public-perception risks, including misuse, penalties, and privacy concerns.The paper cites risks from data theft or misuse, complex regulations such as HIPAA and GDPR, and sharing without explicit consent.
- Cost: Preparing data for AI requires costly, human-dependent, error-prone steps spanning collection, transformation, anonymization, labeling, review, licensing, and sharing.Additional costs include annotation correction, versioning, dataset maintenance, secure transmission, and data enclaves.
- Cost: These preparation steps require medical, AI, and software-engineering expertise, with errors potentially emerging only through downstream consequences.The paper emphasizes careful validation at each step, sometimes requiring multiple iterations.
- Uncertain Return: Existing medical AI challenges often focus on a single dataset and task, limiting their representation of diverse, complex clinical workflows.The paper contrasts these settings with multi-modal, multi-institutional real-world use cases involving domain shifts.
- Uncertain Return: Non-diverse training and evaluation datasets increase overfitting risk and may fail to predict model generalization across real-world institutions.Data owners may also bear greater risks while model builders receive a greater share of eventual benefits.
3 Proposed Solution: An Open Benchmarking
MedPerf proposes an open, federated-evaluation platform that assesses models across data owners’ controlled settings without sharing patient data. It combines aggregated metrics, human approval, and open infrastructure to reduce evaluation and data-sharing risks while broadening performance assessment.
- Federated Learning: Federated evaluation is presented as a lower-risk on-ramp to federated learning, whose model-parameter leakage risks still require ongoing mitigation research.The authors state that mitigation research is ongoing and slows federated-learning adoption.
- Federated Evaluation: Federated evaluation securely sends models to multiple data owners, runs controlled evaluations, and aggregates metrics without sharing patient data.Limiting the process to model evaluation and evaluation metrics reduces patient-privacy, intellectual-property, and data-misuse risks relative to collaborative training.
- Open Platform: MedPerf provides common open-source infrastructure for defining medical AI benchmarks and coordinating model evaluation across participating facilities.The platform is intended to align model developers and data owners through a neutral organization and shared benchmarking practices.
- Risk Mitigation: Human evaluators approve model evaluations and result uploads, while auditing and implementation restrictions address malicious or erroneous code.Novel code lacks network access and restricted local file-system access, and data owners explicitly approve uploaded statistics.
- Infrastructure: The platform keeps proprietary models and data with their owners while using registries, hashes, and stored results to support integrity checking and comparison.Benchmark groups, model developers, and data owners are coordinated through a central hub that does not directly handle proprietary assets.
- Evaluation Benefits: Evaluating models against multiple providers’ datasets creates a larger virtual test set and provides multi-site feedback about performance across facilities and patient populations.The paper states that this can reduce evaluation uncertainty and increase the odds of successful deployment.
4 MedPerf Technical Approach
MedPerf structures benchmarks as coordinated assets and processes for specifying clinical tasks, preparing and registering datasets, evaluating models, and documenting reproducible use. Containerized components support consistent interaction across models, frameworks, and hardware.
- Benchmark Structure: A MedPerf benchmark defines a clinical problem, population, labeling methodology, evaluation metrics, and approved assets for quantitative model-performance measurement.Its datasets may come from multiple facilities representing a diverse patient population.
- Benchmark Components: Dataset Preparation readies evaluation datasets and can test their quality and compatibility before registration.Registered datasets are prepared according to the benchmark criteria.
- Benchmark Components: Evaluation provides consistent testing pipelines and metrics, while reference implementations include model code, evaluation components, and de-identified or synthetic sample data.Registered Models and Documentation support running and understanding the benchmark.
- Implementation: MLCube containers package dataset preparation, evaluation, and registered models with standard metadata and a consistent file-system interface.This enables infrastructure to interact with models using different approaches, frameworks, and hardware platforms.
1. In many cases, a single organization may participate in multiple roles, and multiple
The platform workflow registers a benchmark, recruits data and model owners, obtains approved evaluations at data-owner sites, and distributes results according to benchmark policy. Long-term success also depends on community standards and coordinated roles.
- Roles and Standards: Organizations may participate in multiple operational roles, while sustained benchmarking requires community interoperability standards such as VNA, DICOM, and OMOP.
- 4.3 Benchmarking Workflow: Benchmark groups register benchmarks and recruit data owners and model owners through the open benchmarking platform.
- 4.3 Benchmarking Workflow: Data owners approve and execute evaluation requests, vet the process, and push results to the benchmarking platform.
- 4.3 Benchmarking Workflow: The platform shares evaluation results with participants according to a policy specified by the benchmark group.
5 MedPerf Roadmap
MedPerf’s roadmap describes an open platform for federated evaluation that aims to provide high-confidence evidence of generalized medical AI performance to regulators, providers, and patients.
- MedPerf aims to provide high-confidence evidence of generalized model performance through federated evaluation.
- The roadmap reviews the steps, scope, and current progress required to develop the open platform.
6 Related Work
MedPerf combines federated evaluation, standardized measurement, open community-driven benchmark development, and shared AI best practices for medical AI.
- MedPerf adopts a federated approach focused first on evaluation to lower barriers to adoption.
- The platform generalizes standardized medical AI measurement beyond problem-specific benchmarks.
- MedPerf uses an open, community-driven approach to benchmark development modeled on successful hardware efforts.
- The effort promotes shared best practices for AI by extending approaches associated with MLflow, Kubeflow, MONAI, and GaNDLF.
7 Discussion and Conclusion
MedPerf seeks to make medical AI evaluation reproducible, standardized, open, and useful across diverse institutions and patient populations. The authors call for broad stakeholder participation to support adoption and clinical impact while preserving data privacy.
- MedPerf’s initial goal is to provide medical AI researchers with reproducible benchmarks based on diverse patient populations.
- The authors envision benchmarks increasing development interest and solution quality, leading to patient benefit.
- The platform is intended to support research on data utility, noisy annotations, and model failures, alongside broader adoption.
- Healthcare stakeholders are asked to define benchmark specifications and oversee result analyses.
- AI researchers are asked to test the end-to-end platform and validate models across multiple institutions.
- Data owners can register data without sharing the data itself, while standardization efforts support institutional collaboration.
- The authors call for regulatory requirements that include technically robust and standardized benchmarking.
- MedPerf is presented as an open effort that can bridge AI research and real-world clinical impact through broad collaboration and open computation.
Contribution
The contribution section records author roles and states that the authors declare no competing interests.
- The listed authors contributed to implementation, concept design, supervision, revision, and editorial work.
- The authors declare that there are no competing interests.