Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development

Motivation

• Fragmented medical datasets limit pre-training scale and balanced foundation-model development because most collections are small, narrowly scoped, and difficult to construct under clinical, ethical, and privacy constraints.Public datasets often contain only thousands of images, while natural-image resources contain billions; fragmentation also obscures opportunities to combine related datasets.Source: §1 · p. 5

Related work

• Existing dataset-integration efforts commonly target specific imaging types or organ systems, risking reinforcement of modality, task, and anatomy biases without comprehensive dataset guidance.Integration can bridge data and models, but incomplete overviews may prevent balanced, general-purpose foundation-model development.Source: §1 · pp. 5–6, Figure 2 · p. 6

Motivation

• Prior surveys lack subject- and image-level statistics, omit newer large-scale datasets, and do not systematically connect dataset characteristics with foundation-model requirements.A comprehensive survey is needed to expose coverage gaps, identify integration opportunities, and standardize dataset selection and evaluation.Source: §1 · p. 6

Contributions

• The survey covers over 1,000 open-access datasets published from 2000 to 2025, organizing them by modality, anatomy, task, and label availability to support gap analysis.The taxonomy establishes priorities for future dataset creation by identifying underrepresented modalities, tasks, and anatomical regions.Source: §1 · p. 6
• The proposed metadata-driven fusion paradigm and interactive portal enable fine-grained dataset search, statistical analysis, automated integration, and public access to surveyed information, tools, and a merged dataset.The released resources include the portal, unified dataset information, a related Python toolkit, and a merged large-scale dataset.Source: §1 · p. 6

2.1 Total Growth

• Image releases show clear growth after 2012 and another surge after 2023, yet public medical datasets remain orders of magnitude smaller than natural-image and language corpora.AbdomenAtlas aggregates 1.5 million 2D CT images and 5,195 3D CT volumes, while CT-RATE provides 25,692 non-contrast 3D chest CT scans from 21,304 patients.Source: Figure 6 · p. 9, §2.1 · p. 10

2.2 Imaging Dimensionalities

• 2D images dominate dataset scale, whereas 3D volumes and videos grow more slowly despite offering clinically informative volumetric context and temporal dynamics.Storage, acquisition, curation, and annotation burdens constrain 3D and video availability, making higher-quality resources in these formats a priority.Source: Figure 6 · p. 9, §2.2 · p. 11

2.3 Imaging Modalities

• Pathology contains substantially more images than other modalities because whole-slide images generate thousands of multi-scale patches; MRI represents about 10.4% of all images, while PET, mammography, and endoscopy remain comparatively scarce.Pathology, X-ray, fundus photography, and microscopy grew especially after 2023, whereas MRI growth is slowed by cost, acquisition time, and complex multi-sequence labeling.Source: Figure 8 · p. 11, §2.3 · p. 12

2.4 Tasks

• Classification and segmentation account for the largest shares of released images, while generation rises after 2023 and registration, detection, and tracking remain comparatively underrepresented.These imbalances reflect label economics, acquisition and annotation burdens, temporal-label requirements for tracking, and limited ground truth for registration.Source: Figure 7 · p. 10, §2.4 · p. 13

2.5 Anatomical Regions

• Brain and lung contributed the largest image volumes before 2023, followed by surges in brain, liver, lung, and breast data, while retina and several smaller anatomical regions remain less represented.The anatomical taxonomy maps native dataset labels to standard vocabularies and compact, consistently reported regions to improve comparability across sources.Source: §2.5 · pp. 14–15, Figure 10 · p. 14

2.6 Summary

• Open medical imaging is long-tailed, with strong skew toward 2D data, pathology, X-ray, CT, and MRI, and brain, lung, liver, and breast organs; metadata-driven fusion is therefore needed.The survey recommends normalized counting, balanced sampling, and task-aware objectives to avoid amplifying existing biases, while using heterogeneity as a design signal for scalable training distributions.Source: §2.6 · p. 15

3 2D Medical Image Datasets

• The survey collected 502 2D medical image datasets, including 475 labeled and 45 unlabeled datasets, with labeled data analyzed by modality, task, and anatomy.2D images substantially outnumber 3D volumes and video frames; unlabeled datasets are summarized by modality and anatomy because they lack explicit task definitions.Source: §3 · p. 15

3.1 Overview

• 2D labeled datasets show clear long-tail distributions: pathology and X-ray dominate modalities, several organs dominate anatomy, and generation, classification, segmentation, regression, and detection dominate tasks.Endoscopy and less-represented tasks such as registration, tracking, localization, reconstruction, and visual question answering have fewer images, while uterus, heart, esophagus, limb joints, and small substructures remain underrepresented.Source: §3.1 · p. 15, Figure 11 · p. 16

3.2 CT Modality

• CT comprises 39 labeled datasets totaling approximately 1.4 million images, ranging from 2 images in The Visible Human Project (#14) to 874,000 in RSNA Intracranial Hemorrhage Detection (#6).CT slices are typically axial, sagittal, or coronal, and segmentation commonly targets organs or pathologies for clinical applications.Source: §3.2 · p. 16
• Brain CT datasets contain ∼874,400 images across 5 datasets, whereas lung datasets are more numerous at 11 datasets and ∼60,400 images; abdominal and other structures remain relatively scarce.The brain image count is dominated by RSNA Intracranial Hemorrhage Detection (#6), while lung collections emphasize COVID-19 and cancer screening.Source: §3.2 · p. 16
• Abdomen/pelvis CT includes 7 datasets and ∼1,500 images, making large-scale training difficult despite challenging multi-organ segmentation involving subtle and variable boundaries.Full-body/multistructure datasets provide broader anatomical diversity for generalized representation learning, while other datasets include specialized limb-fracture and head-CT collections.Source: §3.2 · pp. 16–17
• CT task coverage is dominated by classification, with 12 datasets and ∼513,900 images, while segmentation has 9 datasets and ∼2,100 images and detection/localization has 2 datasets and ∼874,100 images.Detection/localization is driven mainly by RSNA Intracranial Hemorrhage Detection (#6); segmentation supports quantitative analysis, whereas classification is prominent in pulmonary and oncology applications.Source: §3.2 · p. 17
• CT reconstruction appears in only 1 dataset with 28 images, while multi-task datasets comprise 3 datasets and ∼500 images and other or unspecified tasks comprise 12 datasets and ∼11,200 images.LoDoPaB-CT (#1) addresses sparse-view reconstruction; examples of multi-task resources include CMB-CRC (#18) and CMB-PCA (#22), while LDCTIQAC2023 (#26) supports registration.Source: §3.2 · p. 17

Dataset overview

• MRI datasets total approximately 722,400 images, ranging from 2 images in The Visible Human Project (#42) to 673,000 in RadImageNet (Subset: MR) (#56).MRI provides superior soft-tissue contrast without ionizing radiation, while multiple sequences such as T1-weighted, T2-weighted, and FLAIR create both rich information and fusion challenges.Source: §3.3 · p. 17

MRI Datasets by Anatomical Regions/Structures

• Brain datasets comprise 2 datasets and 220 images, while abdomen/pelvis datasets comprise 4 datasets and ∼560 images, reflecting generally small region-specific collections.Brain collections target tumor and disease segmentation, whereas abdominal and pelvic datasets address colorectal and prostate cancer segmentation or classification.Source: §3.3 · p. 18
• Full-body and multistructure datasets comprise 8 datasets and ∼704,900 images, dominated by RadImageNet (Subset: MR) (#56) with 673,000 images.ImageCLEF 2016 (#43) contributes 31,000 images, while QUBIQ series datasets provide multi-organ challenge data.Source: §3.3 · p. 18
• Other anatomical datasets comprise 10 datasets and ∼16,700 images, including cardiac, spine, knee, and unspecified-structure collections.Cardiac Atrial Images (#49) contains 8,000 images for heart segmentation, while SpinalDisease2020 (#41) and KNOAP2020 (#38) address spine and knee osteoarthritis.Source: §3.3 · p. 18

MRI Datasets by Tasks

• Classification accounts for 3 datasets and 704,000 images, whereas segmentation spans 6 datasets and ∼8,900 images, making it more numerous but much smaller in volume.Classification volume is dominated by RadImageNet (Subset: MR) (#56) and ImageCLEF 2016 (#43), while segmentation covers cardiac, brain, and abdominal organs.Source: §3.3 · pp. 18–19

3.4 PET Slices

• PET comprises 13 datasets and approximately 41,942 images, typically combining PET with CT or MRI and focusing mainly on brain and abdominal segmentation or classification.Dataset sizes range from 14 images in CMB-GEC (#62) to 31,000 in ImageCLEF 2016 (#60).Source: §3.4 · p. 19
• PET anatomy coverage is concentrated in full-body or multistructure collections, which contribute approximately 31,200 images, while brain datasets contain only approximately 269 images.Eight datasets lack explicit anatomical regions and collectively contribute approximately 10,000 images.Source: §3.4 · p. 19
• PET task coverage is highly uneven: one classification dataset contains 31,000 images, while segmentation and multi-task datasets contain only 255 and 486 images, respectively.Many remaining collections lack explicit tasks and may support unsupervised or semi-supervised learning.Source: §3.4 · pp. 19–20

3.5 Ultrasound (US) Images

• Ultrasound includes 19 datasets and approximately 457,663 images, dominated by RadImageNet-US (#76) with 390k images despite speckle noise and operator-dependent variability.Common tasks include segmentation, classification, and measurement.Source: §3.5 · p. 20
• Ultrasound spans skull, breast, heart, thyroid, liver, full-body, and multistructure imaging, but RadImageNet-US (#76) supplies 390k images across 15 abdominal structures.Its commercial license may restrict accessibility, whereas other datasets are generally much smaller.Source: §3.5 · p. 20
• Ultrasound tasks include classification, segmentation, measurement, tracking, estimation, and reconstruction, with classification contributing approximately 421,500 images and segmentation approximately 26,300 images.RadImageNet-US (#76) supports 15-category classification, while ImageCLEF 2016 (#75) provides 30 classes.Source: §3.5 · pp. 20–21
• Ultrasound segmentation datasets provide pixel-level organ and tumor annotations, but weak boundaries and acoustic shadowing artifacts remain major analysis challenges.HC18 (#70) contributes 1,300 images for fetal head-circumference measurement, and BUSI (#71), CAMUS (#89), and Ultrasound Nerve Segmentation (#80) support segmentation.Source: §3.5 · p. 21
• Three ultrasound datasets are unlabeled: APOLLO-5 (#72) contains 6,203 images, AREN0532 (#87) contains 1,021, and CMB-LCA (#608) has none available.Their multimodal nature may enable future fusion studies.Source: §3.5 · pp. 20–21

Overview

• 61 major X-ray datasets contain approximately 1,657,000 images, with CheXmask (#143) dominating dataset volume at 676,800 images.Most other datasets contain hundreds to thousands of samples, producing a long-tail distribution.Source: §3.6 · p. 21

X-Ray Datasets by Anatomical Regions/Structures

• Approximately 46% of X-ray datasets focus on the chest or lungs, while musculoskeletal, neurological, and abdominal applications remain limited.Thoracic emphasis reflects X-ray’s widespread use in pulmonary and cardiac diagnostics and respiratory disease screening.Source: §3.6 · p. 21
• Thorax/lung datasets comprise 28 collections and approximately 537,900 images, whereas breast imaging has only 3 datasets despite approximately 248,300 images.VICTRE (#139) dominates mammography volume but lacks disease annotations, while thoracic datasets commonly support multi-label pathology classification.Source: §3.6 · p. 21
• Musculoskeletal datasets comprise 8 collections and approximately 15,681 images, while brain/head datasets include only 2 collections and approximately 1,400 images.Musculoskeletal collections commonly address fracture detection and classification, whereas the limited brain/head coverage indicates a gap relative to CT/MRI.Source: §3.6 · p. 21

X-Ray Datasets by Tasks

• Classification is the largest task group with 30 datasets and approximately 670,100 images, while segmentation includes 10 datasets and approximately 708,500 images.CheXpert (#114) and NIH Chest X-ray 14 (#96) anchor classification, whereas CheXmask (#143) accounts for most segmentation images.Source: §3.6 · p. 22
• Detection/localization spans 9 datasets and approximately 59,700 images, while 31% of collections provide pixel-level annotations or detection labels.Other applications include spinal curvature regression, vascular reconstruction, and bone-age estimation, demonstrating task versatility beyond conventional diagnosis.Source: §3.6 · p. 22

3.7 Optical Coherence Tomography (OCT) Images

• OCT comprises 22 highly specialized datasets totaling approximately 221k retinal images, with OCT2017 (#150) and MedMNIST (#158) accounting for most images.OCT provides micrometer-resolution cross-sectional tissue images and primarily supports retinal-layer and retinal-disease analysis; most other datasets contain hundreds to thousands of samples.Source: §3.7 · p. 22
• OCT segmentation includes 11 datasets and ∼2,600 images, emphasizing retinal-layer delineation and lesion segmentation for anatomical analysis and thickness measurement.SinaFarsiu-009 (#163) and SinaFarsiu-018 (#167) provide the largest annotation sets, while DRAC22 (#153) and iChallenge-GOALS (#154) target lesions and three-layer segmentation.Source: §3.7 · pp. 22–23
• Classification is the largest OCT task by image count, with 5 datasets and ∼210,200 images focused mainly on diabetic retinopathy and glaucoma detection.These datasets distinguish diseases from retinal-layer morphology, including drusen or intraretinal fluid, while iChallenge-AGE19 (#152) targets glaucoma with angle-closure annotations.Source: §3.7 · pp. 22–23
• OCT prediction datasets total 3 datasets and ∼8,500 images for diabetic-retinopathy severity prediction using the International Clinical Diabetic Retinopathy scale.The APTOS series demonstrates OCT’s use in quantitative disease-progression monitoring.Source: §3.7 · p. 23

3.8 Fundus Images

• Fundus photography comprises 75 retinal datasets and approximately 412,400 images, dominated by AIROGS (#182) with 101,400 images and otherwise exhibiting a long-tail size distribution.Its RGB-like format supports transfer learning from general computer vision, although pathology detection remains challenging because abnormalities can be subtle within complex retinal anatomy.Source: §3.8 · p. 23
• All 75 fundus datasets target retinal imaging, and classification dominates with 42 datasets and ∼304,200 images focused mainly on diabetic retinopathy and glaucoma detection.About 30% of collections provide pixel-level annotations or detection labels; examples include OIA-ODIR (#214), APTOS 2019 (#194), Yangxi (#200), and JSIEC (#207), which spans 38 fundus disease categories.Source: §3.8 · p. 23
• Fundus segmentation includes 21 datasets and ∼5,300 images, covering vessels, optic disc and cup, photoreceptors, and lesions for quantitative assessment of vascular and glaucomatous changes.RIM-ONE (#193), GAMMA CFP (#240), HRF Seg (#211), and AO-SLO (#197) provide representative structure-specific annotations, while the iChallenge-GAMMA series ((#241), (#218)) supports multimodal analysis.Source: §3.8 · p. 23
• Fundus regression comprises 6 datasets and ∼2,300 images, including arteriovenous-ratio measurement in the INSPIRE series and disease-progression prediction in DeepDR-Task2 (#219).These datasets extend fundus photography toward quantitative disease monitoring.Source: §3.8 · p. 24

3.9 Dermoscopy Images

• Skin dominates dermoscopy, comprising 13 datasets and approximately 133,600 images, while foot, thyroid, and multi-structure collections provide smaller coverage.DFUC2020 (#252) targets diabetic foot ulcers, DDTI targets thyroid nodule segmentation, and ImageCLEF2016 (#254) spans skin, cell, and breast imaging.Source: §3.9 · p. 24
• Classification spans 10 dermoscopy datasets and approximately 157,500 images, whereas segmentation covers 5 datasets and approximately 9,400 images, with Vitiligo (#256) the sole unlabeled collection.These datasets primarily support supervised skin-lesion categorization and boundary delineation, while Vitiligo (#256) may support unsupervised learning.Source: §3.9 · p. 24

3.10 Histopathology

• Histopathology comprises 117 datasets and approximately 2.22 million images, including approximately 2.15 million patches and 67,000 whole-slide images, with 82% providing high-quality supervised labels.Whole-slide images often exceed 100,000×100,000 pixels, requiring patch extraction or tiling; Quilt-1M (#371) and PatchCamelyon (#345) are the largest collections.Source: §3.10 · pp. 24–25
• Multi-organ histopathology contains approximately 1.18 million images across 17 datasets, while lymph-node collections contain approximately 537,000 images and colorectal collections approximately 113,000 images.Quilt-1M (#371), CRC100K (#303), and SICAPv2 (#300) illustrate the concentration on cancer diagnosis across multiple anatomical sites.Source: §3.10 · p. 25
• Histopathology task coverage includes 38 classification datasets with approximately 709,000 images, 31 segmentation datasets with approximately 368,000 images, and emerging multi-task and whole-slide-image use.Specialized collections also support detection, registration, generation, and visual question answering, while 32% of recent datasets use whole-slide images.Source: §3.10 · p. 25

3.11 Microscopy Imaging

• Microscopy comprises 34 datasets and approximately 1.8 million images, dominated by cellular imaging with approximately 1.51 million images across 8 datasets.CellTracking2019 (#372) contributes 1.44 million frames, while microscopy emphasizes cellular and subcellular morphology, counting, behavior, and supervised labels.Source: §3.11 · p. 25
• Microscopy tasks include 12 classification datasets with approximately 81,400 images, 11 segmentation datasets with approximately 99,000 images, and one tracking dataset with approximately 1.4 million images.CellTracking2019 (#372) dominates tracking, while Kaggle-HPA (#385) supports segmentation and protein localization and CBC series combine counting with detection.Source: §3.11 · p. 26

3.12 Infrared Imaging

• Infrared imaging comprises six datasets focused exclusively on retinal applications, totaling approximately 424,532 images, with all datasets created since 2018.The MRL Eye series contributes 424,490 images across five sub-datasets, indicating strong concentration in one collection.Source: §3.12 · p. 26
• Infrared datasets primarily support classification, while RAVIR (#406) is the sole segmentation dataset, delineating retinal blood vessels into background, arteries, and veins.Five datasets address classification tasks including glasses detection, eye-state recognition, reflection analysis, image-quality assessment, and sensor identification; RAVIR contains 42 images.Source: §3.12 · p. 26

3.13 Endoscopy Imaging

• Endoscopy includes 41 major datasets totaling approximately 322,200 images and videos, predominantly endoscopic (39/41), with EndoSlam (#413) the largest at 76,837 images.Endoscopy data are commonly extracted video frames affected by camera motion, lighting changes, specularity, bubbles, and debris; 16/41 datasets contain over 1,000 images.Source: §3.13 · p. 26
• Colon and bowel imaging is the most represented endoscopic anatomical region, spanning eight datasets and approximately 109,400 images for polyp detection, segmentation, and artifact detection.Large collections include SUN_SEG (#428) with 49,136 images, SARAS-ESAD (#425) with 33,398, and Kavsir (#412) with 14,000.Source: §3.13 · p. 27
• Multi-structure gastrointestinal datasets comprise six collections and approximately 86,000 images, led by EndoSlam (#413) with 76k images covering the esophagus, stomach, and colon.Other anatomical coverage includes the esophagus, uterus, gallbladder, and prostate, generally with smaller sample sizes; AIDA-E_2 (#420) contains 157 esophagus images.Source: §3.13 · p. 27
• Endoscopy task coverage includes segmentation in 17 datasets, detection in 6, and classification in 10, spanning polyps, surgical instruments, abnormalities, tissue, and histology.Representative scales are approximately 20,000 segmentation images, 86,600 detection images, and 77,700 classification images.Source: §3.13 · p. 27
• Five endoscopy datasets provide multitask benchmarks totaling 156k images, combining annotations such as captioning, classification, localization, segmentation, and detection.Examples include HyperKvasir (#429), SUN_SEG (#428), and Endo-FM (#445).Source: §3.13 · p. 27
• Additional endoscopy benchmarks address reconstruction, depth estimation, and registration, including EndoSlam (#413) with 76,837 images and P2ILF (#441).These tasks extend endoscopy beyond recognition and delineation toward geometric and structural understanding.Source: §3.13 · p. 27

3.14 Other Modalities

• Other-modality datasets comprise approximately 858,000 images across diverse modalities, with classification representing 75% of datasets and coverage spanning all major anatomical regions unevenly.Digital Mammography (#451) is largest at 640,000 images, followed by MRL Eye Gender (#462) at 84,898 and ADDI ALZHEIMER’S DETECTION CHALLENGE (#450) at 34,614.Source: §3.14 · p. 27
• These datasets concentrate anatomically in breast and retinal imaging, while also covering thoracic/lung, brain/head, and whole-body or multi-structure collections.Breast datasets contain approximately 663,000 images, retinal datasets approximately 88,000, thoracic/lung datasets approximately 27,000, and brain/head datasets approximately 5,000.Source: §3.14 · pp. 27–28
• Task specialization is pronounced, with approximately 798,000 images in classification datasets, approximately 2,300 in segmentation datasets, and additional multi-task and reconstruction resources.CDD-CESM (#459) supports segmentation and classification, while BigNeuron (#452) represents reconstruction tasks.Source: §3.14 · p. 28

3.15 Challenge and Opportunity

• 2D imaging offers exceptional pretraining scale but remains fragmented, heterogeneous, and limited by the partial spatial information of individual images.Inconsistent protocols, resolutions, metadata, staining, preparation, projection, and exposure settings create domain shifts that complicate integration.Source: §3.15 · p. 28
• Severe long-tail imbalance concentrates data in pathology, X-ray, fundus, thoracic, and breast imaging while underrepresenting modalities and anatomical regions such as endoscopy, ultrasound, and abdominal organs.Over 80% of collected images come from thoracic and breast datasets, and X-ray datasets average approximately 12.5K images each, limiting generalization.Source: §3.15 · p. 28
• Noisy weak supervision, scarce pixel-level annotations, and the absence of a unified annotation ontology hinder label harmonization and comprehensive multi-dataset benchmarks.Radiology-report-derived labels can be imprecise, whereas pixel-level annotation is labor-intensive and difficult to scale.Source: §3.15 · p. 28
• Millions of images and standardized large collections enable self-supervised pretraining, while diverse modalities and report-paired datasets support multimodal vision-language learning.Masked auto-encoding and contrastive learning can build reusable backbones, and MIMIC-CXR and CheXpert [64] align visual features with clinical narratives.Source: §3.15 · pp. 28–29
• Strategic dataset consolidation, balanced anatomical coverage, standardized multi-task annotations, and advanced self-supervised and multimodal methods could enable scalable clinical applications and global-health tools.Common 2D data such as X-rays, fundus, and dermoscopy images could support screening for tuberculosis, diabetic retinopathy, and skin cancer in resource-constrained settings.Source: §3.15 · p. 29

4 3D Medical Image Datasets

• 591 3D datasets contain 1,242,022+ volumes, with labeled collections dominant and unlabeled data offering opportunities for self-supervised learning.These datasets provide richer spatial information for volumetric analysis and clinical decision-making, despite having fewer volumes than 2D datasets.Source: §4 · p. 29

4.1 Overview

• 3D datasets exhibit long-tailed distributions: MRI and CT dominate modalities, brain, abdomen, and lung dominate anatomies, and classification and segmentation dominate tasks.PET, ultrasound, prostate, teeth, registration, localization, and detection are comparatively less represented.Source: §4.1 · p. 29

4.2 CT Volumes

• CT contributes 252 datasets and approximately 516,087 volumes, concentrated in lung/chest and segmentation, while whole-body collections support multi-organ learning but annotation consistency remains challenging.CT datasets range from small domain-specific collections to large weakly supervised compilations and expert-verified challenge datasets, with detection, classification, reconstruction, localization, and registration also represented.Source: §4.2 · pp. 30–31

4.3 MRI Volumes

• MRI contributes 231 datasets and approximately 523,847 volumes, dominated by brain/neuro imaging, with classification largest by volume and segmentation largest by dataset count.Sequence and scanner heterogeneity, ambiguous annotations, and cross-vendor generalization gaps constrain consistency, despite broad coverage of reconstruction, registration, tracking, and detection.Source: §4.3 · pp. 32–33

4.4 Ultrasound Volumes

• Ultrasound includes 27 datasets and approximately 56,609 volumes, primarily embedded in multimodal workflows, with registration dominant and coverage limited by operator dependence and underrepresentation of point-of-care imaging.Representative applications include US-MR brain guidance, prostate fusion, cardiac motion analysis, and oncological classification.Source: §4.4 · pp. 33–34

4.5 PET Volumes

• PET includes 65 datasets and 95,456 volumes, remains scarce and usually multimodal, and is concentrated in oncology-focused lung/chest, head-and-neck, and brain applications.Its task distribution follows functional and metabolic imaging needs in diagnosis, staging, and treatment monitoring.Source: §4.5 · pp. 34–35
• PET datasets are dominated by classification, with 45 datasets and ∼60,000+ volumes, while segmentation, multi-task, detection, and registration support oncology, neurological, and longitudinal clinical workflows.Segmentation comprises 20 datasets and ∼25,000 volumes, whereas multi-task, detection, and registration collections comprise 8, 3, and 3 datasets, respectively.Source: §4.5 · pp. 35–36

4.6 Other 3D Volumes

• Specialized 3D modalities comprise 26 datasets and 5,381+ volumes, led by OCT with 14 datasets and 4,288+ volumes for retinal screening, classification, segmentation, and reconstruction.Other collections include 3D DSA for aneurysm analysis, CBCT for dental and treatment-planning applications, and small 3D microscopy datasets for cellular structural analysis.Source: §4.6 · p. 36

4.7 Challenges and Opportunities

• High acquisition, storage, computational, and expert-annotation costs constrain the scale, diversity, processing, and generalization of 3D medical imaging datasets.Volumetric scans require expensive equipment and infrastructure, while slice-by-slice annotation, high dimensionality, low signal-to-noise ratios, and small pathological regions increase resource and modeling demands.Source: §4.7 · p. 37
• Dataset overlap, modality and anatomical imbalance, and task concentration threaten external-validation reliability and limit versatile foundation-oriented systems.The collection includes identical retinal images across OCT2017 and MedMNIST OCT, while CT and MRI dominate volumes; registration, reconstruction, and multi-task datasets remain comparatively scarce.Source: §4.7 · pp. 37–38
• Unlabeled repositories totaling 219 datasets with hundreds of thousands of volumes create opportunities for self-supervised and contrastive pretraining, including large-scale CT, MRI, and multisequence MR resources.Foundation-model-driven generation could provide synthetic volumes for rare diseases and underrepresented anatomy while helping address privacy and acquisition constraints.Source: §4.7 · p. 38
• Multimodal integration and multi-task learning can combine complementary imaging, pathology, and clinical-text information while jointly supporting segmentation, classification, and detection.The proposed direction includes PET/CT and PET/MR fusion, vision-language and pathology-imaging alignment, and unified architectures that transfer knowledge across related tasks.Source: §4.7 · p. 38

5 Medical Video Datasets

• The survey catalogs 77 medical video datasets with 166,691 samples spanning diverse tasks, modalities, and anatomical structures.Video data supports modeling motion patterns, procedural workflows, and temporal consistency for dynamic clinical applications.Source: §5 · p. 39

5.1 Overview

• Endoscopy accounts for 85.9% of videos and 56 datasets, while stomach, colon, and esophagus each comprise about 30% of videos, exposing severe modality and anatomical imbalance.Retina, heart, pupil, and iris each represent less than 2% of videos, while ultrasound microscopy and RGB remain scarce.Source: §5.1 · p. 39, §5.3 · p. 42

5.2.1 Classification

• Classification is the largest surveyed task, comprising 40 datasets and 81,701 samples, whereas segmentation covers 32 datasets and 18,739 video instances.Classification commonly uses accuracy, F1-score, AUC, and segmental edit score; segmentation uses Dice, IoU, and pixel-level accuracy.Source: §5.2.1 · pp. 39–40, §5.2.2 · p. 40

5.2.3 Detection

• Detection spans 27 datasets, while tracking is substantially smaller at 8 datasets and 2,420 samples, with resources progressing toward joint spatiotemporal understanding.Detection uses precision, recall, mAP, and frame-level F1, whereas tracking commonly evaluates MOTA, MOTP, identity switches, and track purity.Source: §5.2.3 · pp. 40–41, §5.2.4 · p. 41

5.2.5 Estimation

• Estimation is represented by only two datasets covering depth, pose, motion, or physiological variables, including SimCol-to-3D and Endovis 2019-SCRE.SimCol-to-3D includes 15 sequences and a 30-video Colposcopy subset, while Endovis 2019-SCRE includes videos from 9 medical sites.Source: §5.2.5 · p. 41

5.2.6 Registration

• Registration is represented by one dataset with 167 samples: P2ILF pairs endoscopy videos with CT across 25 cases for multimodal landmark alignment.The cases comprise 10 training, 10 validation, and 5 testing cases, with evaluation commonly using TRE, DSC, and clinically acceptable-error success rates.Source: §5.2.6 · pp. 41–42

5.4 Anatomical Structures

• Abdominal anatomy dominates the collection with 40 datasets, including 13 on the gallbladder and 6 on the colon, while 7 datasets concern eye anatomy.Other datasets cover structures including the thyroid, pituitary, placenta, artificial blood vessels, and porcine cadavers.Source: §5.4 · p. 43

5.5 Challenges and Opportunities

• Medical video datasets remain constrained by expert-intensive annotation, privacy-preserving release, cross-domain variability, and computational demands for high-resolution, multi-task, real-time analysis.Multimodal datasets, foundation models, self-supervised learning, domain adaptation, federated evaluation, and standardized benchmarks are identified as opportunities to address these limitations.Source: §5.5 · pp. 43–44

6.1 Dataset Collection and Processing

• The collection pipeline catalogs public datasets by dimensions, modalities, anatomies, cases, labels, and tasks, then preserves provenance and task-linked annotations through standardized JSONL schemas.Volumetric, 2D, and video data are converted or stored in formats that preserve quantitative information, geometry, timing, and essential metadata.Source: §6.1 · pp. 45–46, Table 4 · p. 45

6.2 MDFP

• MDFP uses four sequential, metadata-centered phases for discovery, auditing, semantic alignment, fusion planning, and indexing while avoiding raw-pixel handling to strengthen privacy, reproducibility, and auditability.The phases transform heterogeneous metadata into interoperable resources and support goal-oriented dataset assembly.Source: §6.2 · pp. 46–47, §6.2.1 · p. 47, §6.2.2 · p. 48

6.2.1 Phase 1: Metadata Harmonization

• Phase 1 harmonizes modality, dimensionality, anatomy, provenance, and annotation metadata using UMLS and MeSH terminology with API searches and LLM-based refinement.This produces a uniform metadata table for cross-dataset comparison, reproducible filtering, and integration.Source: §6.2.1 · p. 47

6.2.2 Phase 2: Semantic Alignment

• Phase 2 maps machine-learning tasks to clinical applications and records label availability, secondary modalities, and dataset-specific notes, yielding a clinically grounded vocabulary for benchmarking.Examples include diagnosis, lesion delineation, screening, and biomarker quantification.Source: §6.2.2 · pp. 47–48

6.2.3 Phase 3: Fusion Blueprints

• Phase 3 groups datasets by modality, task, and anatomy, quantifies volume, validated images, storage, and diversity, and flags incompatibilities requiring harmonization or domain-adaptive strategies.The resulting blueprints balance quantity, annotation quality, and content diversity for scalable merging.Source: §6.2.3 · p. 48

6.2.4 Phase 4: Dataset Indexing and Community Sharing

• Phase 4 publishes canonical dataset metadata, including names, release dates, links, and licenses, enabling rapid discovery, responsible reuse, benchmarking, pretraining, and open collaboration.The indexed representation provides the community-facing infrastructure for large-scale dataset reuse.Source: §6.2.4 · p. 49

6.2.5 Case Study: Goal-Conditioned Fusion via MDFP

• 57 datasets contributed 2,135,301 validated 2D images across CT, MR, and Fundus, with labeled ratios of 1.000, 1.000, and 0.952, respectively.The composition covered classification, segmentation, detection, and regression while satisfying the case-study constraints.Source: §6.2.5 · pp. 49–50, Table 6 · p. 49

6.3 Interactive Discovery Portal

• The Medical Dataset Browser is a client-side static portal supporting reproducible rule-based filtering, faceted search, live distribution summaries, audit tables, and CSV/JSON export without server-side dependencies.Users can fork, reconfigure, and redeploy the retrieval environment using the standardized manifest.Source: §6.3 · p. 50

7.1 Limitations in Task Definition and Evolution of Data Engineering Paradigms

• Legacy datasets emphasize indirect tasks, while clinically relevant diagnosis and treatment tasks require costly expert re-annotation; flexible annotation frameworks should anticipate evolving AI needs.Segmentation masks, for example, do not encode whether a lung nodule is benign, malignant, or requires biopsy.Source: §7.1 · p. 51

7.2 Scarcity of Multimodal Medical Datasets and Constraints in Further Development

• Public multimodal datasets remain rare and largely unimodal, while missing alignment protocols, semantic standards, benchmarks, and infrastructure impede cross-modal reasoning and clinical integration.Heterogeneous modalities differ in scale and temporal meaning, and storage, synchronization, versioning, and privacy constraints further fragment development.Source: §7.2 · p. 51

7.3 Challenges and Opportunities in Medical Foundation Models

• Medical foundation models require greater scale and representational diversity across diseases, protocols, specialties, demographics, rare conditions, atypical presentations, and underrepresented populations.Current datasets capture narrow slices of clinical reality, limiting generalizability in specialized domains such as pediatric imaging, rare diseases, and longitudinal monitoring.Source: §7.3 · pp. 51–52
• Privacy regulations and institutional intellectual-property policies restrict sharing, fragment the field, and limit the community-wide benefits of synthetic-data augmentation [104] [81].These constraints force redundant institutional efforts and impede collaborative progress.Source: §7.3 · p. 52
• Current training paradigms inadequately support contextual and temporal intelligence, including workflow constraints, patient-specific context, treatment history, emergency protocols, and disease progression [106].These capabilities are presented as essential for real-world deployment beyond pattern recognition.Source: §7.3 · p. 52
• Federated learning infrastructures, standardized licensing models, data governance, and clinically grounded evaluation benchmarks are needed to advance medical foundation models beyond narrow applications.The proposed systemic solutions aim to balance privacy protection with research advancement and support broader clinical impact.Source: §7.3 · p. 52

Conclusion

• The survey finds that predominantly small-scale, task-specific, and modality-restricted datasets, with anatomical and modality disparities, fundamentally constrain medical foundation model development.It identifies scarcity of multimodal datasets, restrictive licensing and privacy regulations, and insufficient contextual intelligence as critical gaps; segmentation and classification dominate while visual question answering and multimodal reasoning remain underrepresented.Source: §8 · p. 52
• MDFP addresses these limitations by enabling integration into larger, more diverse training resources for foundation model development.The framework responds to the field’s incomplete transition from task-oriented to foundation-oriented data engineering paradigms.Source: §8 · p. 52

Future directions

• Future progress requires open, efficient, and inclusive data engineering through broader dataset release, synthetic data, annotation-efficient learning, and public foundation models trained on private data.These strategies target transparency, reproducibility, equitable access, privacy, data scarcity, and effective use of partially labeled or weakly supervised data.Source: §8 · p. 52
• Together, these measures provide a sustainable collaborative framework for building generalizable and clinically impactful medical foundation models.Publicly released models can democratize advanced medical AI capabilities even when raw institution-specific datasets cannot be shared.Source: §8 · p. 52

A Tables of 2D Medical Image Datasets

• The 2D catalog covers CT, MRI, PET, ultrasound, X-ray, OCT, fundus, dermoscopy, histopathology, microscopy, infrared, endoscopy, and other modalities.The tables also document multimodal datasets and task abbreviations including classification, detection, segmentation, registration, reconstruction, localization, estimation, measurement, and tracking.Source: Table 7 · p. 132, Table 8 · p. 133, Table 9 · p. 133

B Tables of 3D Medical Image Datasets

• The 3D CT tables include segmentation datasets for brain, head and neck, chest, lung, spine, pelvis, ribs, abdomen, kidney, lymphatic structures, and other regions.Examples include ISLES 2024 with 250 cases, SegRap2023 with 400, CTSpine1K with 1005, CTPelvic1K with 1184, and RibFrac2020 with 660.Source: Table 21 · p. 143, B Tables of 3D Medical Image Datasets · p. 144
• The 3D collection also supports registration, classification, detection, and combined segmentation or registration tasks across respiratory motion, cancer, organ, and phantom studies.Listed resources include Learn2Reg datasets, Continuous Registration benchmarks, HECKTOR, ACRIN studies, CT-Lymph-Nodes, and radiomics phantom datasets.Source: B Tables of 3D Medical Image Datasets · p. 144
• Later 3D entries include multimodal CT, MR, PET, X-ray, ultrasound, microscopy, and pathology resources for kidney, lymphoma, head and neck, lung, colon, bladder, brain, ovary, prostate, and other cancers.Annotation availability varies, with entries explicitly marked Yes or No for datasets such as AREN0532-Tumor-Annotations, AHOD0831-Tumor-Annotations, TCGA-KIRC, and CPTAC-CM.Source: B Tables of 3D Medical Image Datasets · pp. 144–145
• The segment includes CT datasets with additional MR, PET, US, and microscopy or whole-slide imaging modalities, including CPTAC, CMB, and QIN resources.Examples span head-and-neck, lung, and colorectal collections; listed datasets include CPTAC-GBM, CPTAC-HNSCC, CPTAC-LUAD, CMB-CRC, QIN-HEADNECK, and LIDC-IDRI.Source: B Tables of 3D Medical Image Datasets · pp. 145–146
• Annotation availability is heterogeneous: many datasets provide annotations, whereas others are marked without annotations, including several clinical cancer, quality-assurance, and imaging resources.Annotated datasets support segmentation, detection, classification, registration, and related supervised tasks.Source: B Tables of 3D Medical Image Datasets · pp. 145–147
• Anatomical coverage spans lung, brain, head and neck, chest, abdomen, liver, spine, heart, breast, prostate, kidney, skin, pelvis, and whole body.The segment also includes multi-organ and full-body resources, alongside focused datasets for specific organs, tumors, and anatomical structures.Source: B Tables of 3D Medical Image Datasets · pp. 145–147
• The catalog covers classification, segmentation, detection, registration, reconstruction, localization, tracking, synthesis, atlas construction, and recognition tasks.Several datasets support multiple tasks, such as segmentation/classification or segmentation/detection, while others target specialized applications including radiotherapy and pulmonary disease analysis.Source: B Tables of 3D Medical Image Datasets · pp. 145–147
• Reported resources include 68 QIN-Breast cases, 26,254 National Lung Screening Trial examinations, and 52,000 QIDW quality-assurance cases across breast and lung imaging datasets.Source: B Tables of 3D Medical Image Datasets · pp. 145–146
• The 3D dataset tables cover diverse modalities, anatomies, diseases, and tasks, including CT and MR oncology datasets alongside brain, breast, prostate, lung, and cardiac resources.Examples include Head-Neck-CT-Atlas with 215 cases for segmentation, Duke-Breast-Cancer-MR with 922 cases for segmentation/classification, and EMIDEC with 150 cases for segmentation/classification.Source: B Tables of 3D Medical Image Datasets · pp. 147, 150, 151
• The tables include both large cohorts and small specialized datasets, ranging from 10 cases in QIN Breast DCE-MR to 295200 cases in Dynamic Passive Threat.Other large examples include PI-CAI with 1500 cases, TADPOLE with 1667 cases, and MUDI2019 with 1344 cases.Source: B Tables of 3D Medical Image Datasets · pp. 149–151
• Brain MRI resources are especially numerous, spanning glioma, stroke, multiple sclerosis, Alzheimer’s disease, healthy ageing, developmental disorders, and neuropsychiatric studies.The listings include BraTS 2023 with 5880 cases for glioma segmentation, ADNI with 2500 cases for Alzheimer’s disease classification, and Cam-CAN with 3000 cases for healthy-ageing classification.Source: B Tables of 3D Medical Image Datasets · p. 149
• Annotation availability varies across the catalog, with datasets marked as annotated or unannotated and supporting tasks from segmentation and classification to registration, reconstruction, detection, tracking, localization, and regression.Examples include Lausanne TOF-MRA Aneurysm Cohort for detection, IXI Dataset for registration/reconstruction, MASiVar for tracking, and Pragmatic Language for localization.Source: B Tables of 3D Medical Image Datasets · p. 150
• Dataset scale and annotation availability are highly variable, ranging from tiny challenge resources such as IronTract Challenge with 2 subjects to large annotated collections such as OpenMind with 114570 subjects.The tables mark whether annotations are available and identify task-specific resources spanning challenges, healthy volunteers, clinical diseases, and quality-assurance studies.Source: B Tables of 3D Medical Image Datasets · pp. 151, 152, 154
• Several entries combine modalities or dimensionalities, including 2D/3D MR, PET/CT, microscopy, ultrasound, X-ray, and whole-slide imaging for multimodal disease analysis.Examples include CMRxRecon for 2D/3D MR heart reconstruction and segmentation, CSI15 for 2D/3D MR/X-ray spine analysis, and CMB collections combining PET/CT with MR, ultrasound, microscopy, or WSI.Source: B Tables of 3D Medical Image Datasets · pp. 152, 154
• The catalog covers 3D MR datasets across brain, heart, breast, liver, abdomen, whole-body, and other anatomies, with segmentation, classification, registration, reconstruction, tracking, and detection tasks.Examples include OpenMind with 114570 subjects for segmentation/reconstruction, UPENN-GBM with 3680 for glioblastoma segmentation, and MUDI2019 with 6720 for registration/reconstruction.Source: B Tables of 3D Medical Image Datasets · pp. 151–152
• The PET/CT and PET/MR entries emphasize oncologic and neurodegenerative applications, including segmentation, classification, detection, and regression across lung, head-and-neck, breast, prostate, and brain diseases.Examples include OASIS-3 with 5699 subjects, TADPOLE with 1667, and QIDW with 52000 quality-assurance cases.Source: B Tables of 3D Medical Image Datasets · p. 154

• SAGES-CVS lists 18,000 samples, whereas NurVid lists 1,538, illustrating substantial variation in video-dataset scale.Source: C Tables of Medical Video Datasets · p. 156
• Retinal video and microscopy datasets address surgical phase, instrument, anatomy, and full-scene understanding, including EndoVis20-CATARACTS, Cataract-1K, LensID, and CaDIS.These datasets provide segmentation, detection, and classification annotations for retina, iris, pupil, and surgical instruments.Source: C Tables of Medical Video Datasets · p. 156
• Surgical endoscopy datasets cover instruments, actions, phases, anatomy, and landmarks through classification, detection, segmentation, tracking, and retrieval tasks.Examples include AdaptOR2021 for 2D landmarks, EndoCV 2021 for surgical actions, and m2cai16-tool-locations for instrument tracking.Source: C Tables of Medical Video Datasets · p. 156
• Beyond surgical instruments, the datasets cover polyps, angiodysplasia, vessels, kidneys, placenta, pituitary anatomy, and abdominal organs through detection, segmentation, and classification.GIANA provides segmentation and detection for colonic angiodysplasia, while PolypGen provides segmentation for polyps and the Dresden dataset segments surgical anatomy.Source: C Tables of Medical Video Datasets · p. 156
• The catalog includes multimodal and geometric tasks such as EndoVis22-P2ILF’s video, 3D endoscopy, and CT liver registration, plus depth and camera-pose estimation.Source: C Tables of Medical Video Datasets · p. 156