Source-linked AI summary
MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis
Jiancheng Yang, Rui Shi, Bingbing Ni
TL;DR
Medical image analysis lacks accessible, standardized benchmarks for comparing AutoML across modalities and dataset scales. The paper introduces MedMNIST and its 10-dataset Classification Decathlon, then compares baseline, open-source, and commercial methods; no algorithm generalizes well enough across all 10 datasets.
Problem
Medical image analysis requires substantial domain knowledge and model-tuning effort, while few benchmarks compare AutoML methods across medical-image classification tasks.
Method
The paper constructs 10 standardized 28 × 28 medical-image datasets and evaluates AutoML methods and ResNet baselines across all datasets without manual tuning.
Results
No algorithm generalizes well enough across all 10 datasets; Google AutoML Vision performs well generally but does not always outperform the ResNet baselines.
Takeaways & Limitations
MedMNIST supports education, rapid prototyping, multi-modal learning, and further research on AutoML methods that generalize across diverse medical-image tasks.
Takeaways & Limitations
The study uses retrospectively collected human-subject open-source data, and ethical approval was not required under the attached licenses.
Abstract
from arXiv · showhide
We present MedMNIST, a collection of 10 pre-processed medical open datasets. MedMNIST is standardized to perform classification tasks on lightweight 28x28 images, which requires no background knowledge. Covering the primary data modalities in medical image analysis, it is diverse on data scale (from 100 to 100,000) and tasks (binary/multi-class, ordinal regression and multi-label). MedMNIST could be used for educational purpose, rapid prototyping, multi-modal machine learning or AutoML in medical image analysis. Moreover, MedMNIST Classification Decathlon is designed to benchmark AutoML algorithms on all 10 datasets; We have compared several baseline methods, including open-source or commercial AutoML tools. The datasets, evaluation code and baseline methods for MedMNIST are publicly available at https://medmnist.github.io/.
1. INTRODUCTION
MedMNIST addresses the lack of accessible, standardized medical-image benchmarks for AutoML by providing lightweight, diverse datasets and a decathlon evaluation.
- Motivation: Medical image analysis is difficult for novices and offers few benchmarks for comparing AutoML methods.Different modalities require specialized background knowledge, while deep-learning models require substantial tuning effort.
- MedMNIST: MedMNIST provides 10 pre-processed open medical-image datasets standardized as lightweight 28 × 28 classification tasks.The collection covers primary medical modalities and is designed for users without background knowledge.
- Uses: MedMNIST is intended for education, rapid prototyping, multi-modal learning, and AutoML experimentation.Its 28 × 28 images reduce the computational and engineering burden of experimentation.
- MedMNIST: The datasets span scales from 100 to 100,000 and include binary, multi-class, ordinal-regression, and multi-label tasks.This diversity supports evaluation across different medical-image settings.
- Benchmark: The MedMNIST Classification Decathlon evaluates AutoML algorithms on all 10 datasets without manual tuning.It compares early-stopping ResNets, open-source AutoML tools, and a commercial AutoML tool.
2. MEDMNIST DATASET
MedMNIST standardizes 10 open datasets from multiple medical modalities into small image-classification benchmarks while preserving varied scales and task types.
- Dataset overview: The collection covers X-ray, OCT, ultrasound, CT, and other modalities, with dataset scales ranging from 100 to 100,000.Source datasets use Creative Commons or free licenses, enabling derivative dataset construction.
- Dataset examples: PathMNIST contains 100,000 histology patches, 7,180 external-center test patches, and 9 tissue classes.Source images are resized from 3 × 224 × 224 to 3 × 28 × 28.
- Dataset examples: ChestMNIST formulates 112,120 frontal X-ray images with 14 disease labels as a multi-label binary-classification task.Images are resized from 1 × 1024 × 1024 to 1 × 28 × 28.
- Dataset examples: DermaMNIST provides 10,015 dermatoscopic images across 7 lesion categories for multi-class classification.The data are split into training, validation, and test sets at a 7:1:2 ratio.
- Dataset examples: OCTMNIST and PneumoniaMNIST use single-channel 28 × 28 images for four-class retinal-disease and binary pneumonia classification, respectively.Both datasets are center-cropped and resized from larger source images.
- Dataset examples: RetinaMNIST is an ordinal-regression task grading diabetic-retinopathy severity across 5 levels, while BreastMNIST simplifies three breast-ultrasound categories into binary classification.OrganMNIST performs multi-class classification of 11 body organs from CT scans.
3. BENCHMARK
The Classification Decathlon benchmarks AutoML across all 10 datasets without manual tuning, using baseline deep networks, open-source tools, and commercial AutoML.
- Benchmark design: The benchmark evaluates algorithm performance across all 10 datasets without manual tuning.It is designed as a lightweight AutoML benchmark for medical image analysis.
- Baseline: ResNet-18 and ResNet-50 with validation-based early stopping serve as baseline methods.Models use 28 or resized 224 input resolutions, cross-entropy loss, SGD, and 100 training epochs.
- AutoML methods: The compared AutoML systems include auto-sklearn, AutoKeras, and Google AutoML Vision.They represent statistical machine learning, deep neural networks, and commercial black-box AutoML, respectively.
- AutoML methods: auto-sklearn searches scikit-learn algorithms and hyper-parameters, while AutoKeras searches neural architectures and hyper-parameters.AutoKeras runs up to 20 trials for 20 epochs each and selects the highest validation AUC model.
- AutoML methods: Google AutoML Vision trains Edge-exportable quantized models, with node-hour budgets scaled to dataset size.Budgets are 1, 2, or 3 node hours for datasets around 1,000, 10,000, or 100,000 samples.
- Evaluation: AUC and ACC are the standardized evaluation metrics reported for each dataset.AUC is less sensitive to class imbalance, while ACC evaluates thresholded or arg-max labels.
4. RESULTS
Across the MedMNIST benchmark, Google AutoML Vision generally performs well but does not consistently win, while no algorithm generalizes well across all 10 datasets. Performance also varies with dataset scale, with small datasets tending to produce overfitting.
- Google AutoML Vision performs well in general but does not always outperform ResNet-18 and ResNet-50.
- AUC and ACC are used to evaluate performance on training, validation, and test splits.
- auto-sklearn performs poorly on most datasets compared with the evaluated baselines.The results indicate that typical statistical machine learning algorithms do not work well on these medical image datasets.
- AutoKeras performs well on large-scale datasets but relatively worse on small-scale datasets.
- No evaluated algorithm generalizes well across all 10 datasets, motivating AutoML methods that handle diverse modalities, tasks, and scales.
- Algorithms tend to overfit small-scale datasets, with severe overfitting observed for auto-sklearn.Google AutoML Vision controls overfitting issues well.
5. CONCLUSION
The paper presents MedMNIST as a 10-dataset, pre-processed benchmark for medical image classification and introduces the MedMNIST Classification Decathlon for AutoML evaluation. Experiments show that no algorithm generalizes well across all 10 datasets.
- MedMNIST consists of 10 pre-processed datasets designed for educational use, rapid prototyping, and multimodal machine learning and AutoML.
- The MedMNIST Classification Decathlon evaluates AutoML algorithms on all 10 datasets without manual tuning.
- No evaluated algorithm generalizes well enough on all 10 datasets.
6. COMPLIANCE WITH ETHICAL STANDARDS
The study used retrospective human-subject open-source data, and ethical approval was not required under the attached open-access data licenses.
- The retrospective study used human-subject open-source data, and ethical approval was not required according to the attached licenses.