PaperlayerView exact revision on arXiv

Source-linked AI summary

MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification

Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni

arXiv:2110.14795v2cs.CVcs.AIcs.LGeess.IV

TL;DR

Biomedical image classification spans diverse modalities, dataset scales, and task types, making standardized evaluation difficult. MedMNIST v2 addresses this with a standardized 2D/3D benchmark and baseline evaluations, supporting machine-learning and computer-vision research, including 3D classification.

  • Problem

    Biomedical image analysis involves diverse modalities, dataset scales, and classification tasks, creating a need for efficient comparative evaluation.

  • Method

    MedMNIST v2 constructs standardized 2D and 3D biomedical image classification datasets spanning diverse modalities, scales, tasks, and fixed data splits, then benchmarks neural networks and AutoML tools.

  • Results

    Across benchmarks, performance varied by modality, scale, dimensionality, resolution, and model type, with 3D and ACS models generally outperforming 2.5D models.

  • Takeaways & Limitations

    MedMNIST v2 provides a lightweight benchmark for comparing machine-learning methods across diverse biomedical image-classification settings, including relatively scarce 3D tasks.

  • Takeaways & Limitations

    The dataset is not intended for clinical use because substantially reduced image resolution may inadequately represent different disease pathologies.

Abstract

from arXiv · show

We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D. All images are pre-processed into a small size of 28x28 (2D) or 28x28x28 (3D) with the corresponding classification labels so that no background knowledge is required for users. Covering primary data modalities in biomedical images, MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label). The resulting dataset, consisting of 708,069 2D images and 10,214 3D images in total, could support numerous research / educational purposes in biomedical image analysis, computer vision, and machine learning. We benchmark several baseline methods on MedMNIST v2, including 2D / 3D neural networks and open-source / commercial AutoML tools. The data and code are publicly available at https://medmnist.com/.

Background & Summary

MedMNIST v2 is a large-scale benchmark for 2D and 3D biomedical image classification that addresses diverse modalities, dataset scales, and tasks. It provides standardized, lightweight datasets with fixed splits and labels for accessible algorithm comparison and research or education.

  • Contribution: MedMNIST v2 comprises 12 2D datasets with 708,069 images and 6 3D datasets with 9,998 images.It is introduced as a new “decathlon” dataset for biomedical image analysis.
  • Design: It spans diverse data modalities, dataset scales from 100 to 100,000, and binary/multi-class, multi-label, and ordinal regression tasks.Both 2D and 3D biomedical images are provided to evaluate generalizable performance across settings.
  • Design: Each sub-dataset uses the same pre-processing format, requires no background knowledge, and includes standard train-validation-test splits for direct algorithm comparison.The collection focuses on machine learning classification with small images rather than end-to-end systems.
  • Contribution: The benchmark extends MedMNIST v1 with 2 additional 2D bioimage datasets and 6 newly developed 3D datasets, alongside benchmarks of standard deep learning methods and AutoML tools.Its 3D datasets address the popularity of 3D imaging in biomedicine and the scarcity of 3D image classification datasets.

Methods · Design Principles

MedMNIST v2 is designed as a standardized collection spanning 2D and 3D biomedical datasets, primary imaging modalities, varied classification tasks, and dataset scales from 100 to 100,000. The collection also provides official data splits and groups datasets into MedMNIST2D and MedMNIST3D.

  • Design Principles: 12 2D and 6 3D standardized datasets cover primary biomedical imaging modalities, including X-ray, OCT, ultrasound, CT, and electron microscopy.The collection uses imaging resolution to represent modality in its landscape visualization.
  • Design Principles: Dataset scales range from 100 to 100,000, supporting variation in collection size.
  • Design Principles: The collection includes binary-class, multi-class, ordinal-regression, and multi-label classification tasks.
  • Design Principles: Official train-validation-test splits are provided for each subset to avoid repeatedly comparing the impact of train-validation splitting.Official source-dataset splits are used when available.
  • Design Principles: When a source provides only training and validation data, its official validation set is used as test data and the official training set is split 9:1 into training and validation.
  • Design Principles: The 2D dataset collection is called MedMNIST2D, while the 3D collection is called MedMNIST3D.
  • Design Principles: Table 2 summarizes each dataset’s source, modality, task type, class or label count, total samples, and training, validation, and test sizes.It covers 12 MedMNIST2D datasets and 6 MedMNIST3D datasets.
  • Design Principles: Figure 2 maps dataset scale and imaging resolution, using triangles for 2D versus 3D datasets and colors for different tasks.

Details for MedMNIST2D … DermaMNIST

MedMNIST2D includes PathMNIST, ChestMNIST, and DermaMNIST, each derived from established biomedical imaging datasets and standardized to 28×28 inputs. The datasets cover multi-class and multi-label binary-class classification tasks across histology, X-ray, and dermatoscopic images.

  • PathMNIST: PathMNIST uses 100,000 non-overlapping hematoxylin & eosin-stained colorectal histology patches from NCT-CRC-HE-100K.Its test dataset, CRC-VAL-HE-7K, contains 7,180 patches from a different clinical center.
  • PathMNIST: PathMNIST covers 9 tissue types in a multi-class classification task.Source images are resized from 3×224×224 to 3×28×28.
  • ChestMNIST: ChestMNIST is based on NIH-ChestXray14, comprising 112,120 frontal-view X-ray images from 30,805 unique patients.The images carry text-mined labels for 14 diseases.
  • ChestMNIST: ChestMNIST formulates the 14 disease labels as a multi-label binary-class classification task.The official data split is used, and source images are resized from 1×1,024×1,024 to 1×28×28.
  • DermaMNIST: DermaMNIST is based on HAM10000, a multi-source collection of dermatoscopic images of common pigmented skin lesions.The dataset contains 10,015 images categorized into 7 diseases.
  • DermaMNIST: DermaMNIST formulates the 7 disease categories as a multi-class classification task.Images are split into training, validation, and test sets at a 7 : 1 : 2 ratio.
  • DermaMNIST: DermaMNIST resizes source images from 3×600×450 to 3×28×28.This standardizes the dermatoscopic inputs for the MedMNIST2D format.

OCTMNIST … BreastMNIST

OCTMNIST, PneumoniaMNIST, RetinaMNIST, and BreastMNIST standardize biomedical image datasets into distinct classification tasks, using source-specific splits and resizing procedures. The tasks span multi-class, binary, ordinal regression, and simplified binary classification.

  • OCTMNIST: OCTMNIST uses 109,309 valid optical coherence tomography images covering four retinal-disease diagnosis categories in a multi-class classification task.The source training set is split 9 : 1 for training and validation, with the source validation set used for testing.
  • OCTMNIST: OCTMNIST source images are gray-scale, range from (384 −1,536) × (277 −512), and are center-cropped before standardization.
  • PneumoniaMNIST: PneumoniaMNIST contains 5,856 pediatric chest X-Ray images for binary classification of pneumonia against normal.The source training set is split 9 : 1 for training and validation, with the source validation set used for testing.
  • PneumoniaMNIST: PneumoniaMNIST center-crops variable-sized gray-scale images using the short-edge length and resizes them into 1×28×28.
  • RetinaMNIST: RetinaMNIST uses 1,600 retina fundus images for ordinal regression across five diabetic-retinopathy severity grades.The source training set is split 9 : 1 for training and validation, with the source validation set used as the test set.
  • RetinaMNIST: RetinaMNIST center-crops 3×1,736×1,824 source images by the short-edge length and resizes them into 3×28×28.
  • BreastMNIST: BreastMNIST contains 780 breast ultrasound images originally categorized as normal, benign, and malignant.The source dataset is split 7 : 1 : 2 into training, validation, and test sets.
  • BreastMNIST: BreastMNIST simplifies the original three-class task into binary classification by combining normal and benign as positive against malignant as negative, resizing 1×500×500 images into 1×28×28.

BloodMNIST · TissueMNIST · Organ{A,C,S}MNIST

BloodMNIST, TissueMNIST, and Organ{A,C,S}MNIST standardize biomedical image data from blood cells, kidney cortex cells, and organ CT images for classification. Their construction combines labeled source datasets with dataset-specific splitting, projection, cropping, and intensity-processing procedures.

  • BloodMNIST: 17,092 images across 8 classes comprise BloodMNIST, derived from individual normal blood cells collected from disease-free, untreated individuals.The source data are split 7:1:2 into training, validation, and test sets.
  • TissueMNIST: 236,386 human kidney cortex cells from 3 reference tissue specimens form TissueMNIST, organized into 8 categories.The cells come from the BBBC051 dataset in the Broad Bioimage Benchmark Collection.
  • TissueMNIST: TissueMNIST uses a 7:1:2 split for training, validation, and testing, with each grayscale source image measuring 32×32×7 pixels.The seven slices are projected into 2D by taking the maximum pixel value along the axial axis.
  • Organ{A,C,S}MNIST: Organ{A,C,S}MNIST derives from 3D CT images in the Liver Tumor Segmentation Benchmark and renames the earlier axial, coronal, and sagittal OrganMNIST datasets.Organ labels are obtained using bounding-box annotations for 11 body organs from another study.
  • Organ{A,C,S}MNIST: Organ{A,C,S}MNIST converts Hounsfield-Unit values to grayscale using an abdominal window and crops 2D images from center slices of 3D organ bounding boxes.The center slices are taken in axial, coronal, and sagittal orientations.

Details for MedMNIST3D

MedMNIST3D comprises diverse lightweight 3D biomedical classification datasets derived from CT, vascular meshes, and electron-microscopy volumes. The datasets cover organ, nodule, adrenal, fracture, vessel, and synapse classification tasks with standardized 28×28×28 representations where specified.

  • OrganMNIST3D: OrganMNIST3D processes 3D bounding boxes into 28×28×28 images for 11-body-organ multi-class classification, using 115, 16, and 70 CT scans for training, validation, and testing.The scans correspond to the Organ{A,C,S}MNIST source training and test sets.
  • NoduleMNIST3D: NoduleMNIST3D uses thoracic CT images from LIDC-IDRI for lung-nodule segmentation and 5-level malignancy classification, with malignancy levels 1/2 negative and 4/5 positive in binary classification.Level 3 cases are ignored, and the source dataset is split 7 : 1 : 2 into training, validation, and test sets.
  • AdrenalMNIST3D: AdrenalMNIST3D contains shape masks from 1,584 adrenal glands in 792 patients, annotated by an expert endocrinologist for binary classification of normal glands versus adrenal masses.The dataset was collected at Zhongshan Hospital Affiliated to Fudan University, and source CT scans are withheld for privacy.
  • FractureMNIST3D: FractureMNIST3D derives from around 5,000 rib fractures in 660 CT scans and classifies buckle, nondisplaced, and displaced fractures after excluding segmental fractures.The four original clinical categories include buckle, nondisplaced, displaced, and segmental fractures; segmental fractures are disregarded because of low image resolution.
  • VesselMNIST3D: VesselMNIST3D uses 103 intracranial aneurysm vessel models to generate 1,694 healthy and 215 aneurysm segments, repairing meshes and voxelizing them into 28 × 28 × 28 volumes.The source models were reconstructed from MRA images, and the dataset is split 7 : 1 : 2 into training, validation, and test sets.
  • SynapseMNIST3D: SynapseMNIST3D classifies synapses as excitatory or inhibitory using a 3D adult-rat volume acquired by multi-beam scanning electron microscopy and expert-segmented neuronal data.The original volume measures 100×100×100um3 at 8×8×30nm3 resolution, with three neuroscience experts segmenting and proofreading a pyramidal neuron.

Data Records

MedMNIST v2 provides 12 pre-processed 2D and 6 pre-processed 3D datasets through Zenodo, with each subset distributed as a uint8 NumPy npz file containing standardized train, validation, and test images and labels.

  • Dataset files: 12 pre-processed 2D datasets and 6 pre-processed 3D datasets are available through Zenodo.Each subset is saved in NumPy npz format with six keys: train_images, train_labels, val_images, val_labels, test_images, and test_labels.
  • Dataset files: Each subset uses uint8 data and is named <data>mnist.npz for MedMNIST2D or <data>mnist3d.npz for MedMNIST3D.The files contain separate training, validation, and test arrays.
  • Image arrays: Image arrays have shape N × 28 × 28 for 2D grayscale, N × 28 × 28 × 3 for 2D RGB, and N × 28 × 28 × 28 for 3D datasets.N denotes the number of samples in the corresponding training, validation, or test set.
  • Label arrays: Label arrays have shape N × 1 for multi-class, binary-class, and ordinal regression datasets, or N × L for multi-label binary-class datasets.For ChestMNIST, L equals 14 task labels.

Technical Validation

The technical validation benchmarks ResNet baselines and AutoML methods on MedMNIST2D and MedMNIST3D using AUC and ACC, with performance averaged across repeated trials. Results show distinct method and architecture patterns across 2D and 3D datasets, including strong Google AutoML Vision performance in 2D and standard 3D ResNet-50 performance in 3D.

  • AutoML methods: Google AutoML Vision, auto-sklearn, and AutoKeras represent commercial black-box, open-source statistical, and open-source deep-learning AutoML methods, respectively.auto-sklearn search limits vary by dataset scale, while AutoKeras tests 20 models for 20 epochs and selects the highest validation AUC.
  • Evaluation: Evaluation uses AUC and ACC, with AUC measuring threshold-free scores and ACC measuring thresholded or argmax labels.Because the datasets lack severe class imbalance, ACC is also considered a suitable metric.
  • MedMNIST2D results: On 2D datasets, Google AutoML Vision generally performs well but does not always beat ResNet baselines, auto-sklearn performs poorly, and AutoKeras is stronger on large-scale datasets.At the aggregate level, Google AutoML Vision has the highest average AUC, while ResNets surpass auto-sklearn and AutoKeras in average ACC.
  • MedMNIST3D results: On 3D datasets, standard 3D ResNet-50 has the best average performance, while 2.5D models underperform 3D and ACS models and auto-sklearn outperforms 2.5D baselines in general.AutoKeras performs poorly overall, whereas 3D and ACS models are comparable; auto-sklearn achieves the highest ACC on AdrenalMNIST3D.
  • MedMNIST2D results: For 2D models, resolution 224 yields higher AUC and ACC than resolution 28 under the same backbone, while ResNet-18 exceeds ResNet-50 at the same resolution.These comparisons concern the reported average performance over all 2D datasets.

Usage Notes

MedMNIST is freely available online, but it is not intended for clinical use because substantially reduced image resolution may fail to represent and capture different disease pathologies.

  • Usage and citation: MedMNIST is freely available at medmnist.com, and users are requested to cite MedMNIST v1, v2, and the corresponding source dataset in publications.The passage refers to MedMNIST v19 and v2; the v19 notation is preserved from the supplied text.
  • Clinical-use limitation: MedMNIST is not intended for clinical use because substantially reduced medical-image resolution may inadequately represent and capture different disease pathologies.The warning applies to clinical use of the dataset after resolution reduction.

Author contributions statement

The authors describe contributions spanning experiment conception, software and benchmarking, data preparation, manuscript writing, and final review.

  • JY conceived the experiments and, with RS, developed the code and benchmark.
  • JY, RS, DW, ZL, LZ, BK, and HP contributed to data collection, cleaning, and annotations.
  • JY, RS, DW, and BN wrote the manuscript, and all authors reviewed it.
Loading 2110.14795v2…