Source-linked AI summary
AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, Ping Luo
TL;DR
Abdominal multi-organ segmentation lacks a large, diverse clinical benchmark because 3D data collection and labeling are costly, limiting samples and organ coverage. AMOS provides a 600-scan, 15-organ dataset spanning diverse clinical settings, benchmarks existing methods, and supports extended learning-task evaluations. The benchmark shows that current state-of-the-art methods do not achieve satisfactory performance, while AMOS serves as a general benchmark and resource for OOD generalization, cross-modality learning, and transfer learning.
Problem
Existing abdominal segmentation datasets provide limited samples or organ coverage and insufficient clinical diversity, hindering comprehensive and fair evaluation.
Method
AMOS constructs a dataset of 500 CT and 100 MRI scans with annotations for 15 organs and benchmarks representative segmentation models using mDice and mNSD.
Results
Current state-of-the-art algorithms fail to achieve satisfactory performance on AMOS, while UNet achieves 88.87% mDice on AMOS-CT and 85.59% mDice on AMOS-MRI.
Takeaways & Limitations
AMOS can serve as a general multi-organ segmentation benchmark and support OOD generalization, cross-modality learning, and transfer learning studies.
Abstract
from arXiv · showhide
Despite the considerable progress in automatic abdominal multi-organ segmentation from CT/MRI scans in recent years, a comprehensive evaluation of the models' capabilities is hampered by the lack of a large-scale benchmark from diverse clinical scenarios. Constraint by the high cost of collecting and labeling 3D medical data, most of the deep learning models to date are driven by datasets with a limited number of organs of interest or samples, which still limits the power of modern deep models and makes it difficult to provide a fully comprehensive and fair estimate of various methods. To mitigate the limitations, we present AMOS, a large-scale, diverse, clinical dataset for abdominal organ segmentation. AMOS provides 500 CT and 100 MRI scans collected from multi-center, multi-vendor, multi-modality, multi-phase, multi-disease patients, each with voxel-level annotations of 15 abdominal organs, providing challenging examples and test-bed for studying robust segmentation algorithms under diverse targets and scenarios. We further benchmark several state-of-the-art medical segmentation models to evaluate the status of the existing methods on this new challenging dataset. We have made our datasets, benchmark servers, and baselines publicly available, and hope to inspire future research. Information can be found at https://amos22.grand-challenge.org.
1 Introduction
AMOS addresses the small-scale and low-diversity limitations of abdominal segmentation benchmarks by providing a large, diverse clinical dataset and benchmarking existing methods. It is designed to support comprehensive evaluation and broader learning applications.
- Accurate abdominal organ segmentation supports clinical applications by representing organ interrelations, positions, and shapes for decision-making.
- Existing benchmarks are limited by costly 3D data acquisition and annotation, resulting in few samples or organs of interest.
- Homogeneous datasets from limited centers, scanners, phases, or diseases can perform catastrophically when clinical test distributions shift.
- AMOS contains 600 CT/MRI scans, over 74K annotated slices, and 15 organs, with data from two centers and eight scanners across diverse clinical conditions.The dataset is reported as 20× larger than BTCV’s 3.6K slices.
- The authors benchmark existing segmentation baselines and conduct extended experiments for multiple learning tasks.
2 Related Work
Prior abdominal segmentation research includes single-organ methods, multi-organ architectures, and training paradigms addressing limited annotations. AMOS’s annotation workflow combines automated coarse labeling with repeated expert refinement.
- Single-organ segmentation has been dominant, with methods targeting organs such as the liver, tumor, spleen, kidney, and pancreas.
- Multi-organ methods use architectures such as multiview fusion and dense 3D networks to discriminate pixel-wise features across organs.
- AMOS uses pretrained segmentors to generate coarse labels, followed by repeated refinement by junior radiologists and senior specialists.The workflow is illustrated in Figure 2.
3 AMOS
AMOS is a large, diverse clinical abdominal segmentation dataset with 600 CT/MRI scans, annotations for 15 organs, and evaluation-oriented data partitions. Its statistics and transfer experiments support its breadth, annotation quality, and utility for robust segmentation research.
- Dataset construction: AMOS collects clinical data from 600 patients across two institutes and eight machines, with de-identified DICOM data and diagnosis reports stored securely.
- Dataset construction: AMOS provides in-distribution and out-of-distribution evaluation settings to study scanner-vendor domain shifts in medical imaging.The split design treats image acquisition scanners as domains and includes unseen test data for OOD evaluation.
- Dataset statistics: AMOS spans a broad range of diseases and affected organs, while its annotations follow a natural long-tail distribution that makes small-organ segmentation more difficult.Liver annotation scale is about 200× larger than adrenal-gland annotation.
- Dataset statistics: 74K annotated slices make AMOS 2.2× larger than AbdomenCT-1K and 20× larger than BTCV, while also offering more organs, annotations per scan, and modalities than AbdomenCT-1K.
- Dataset statistics: Models pretrained on AMOS outperform BTCV-pretrained models across several target datasets and achieve 84.47% mDice on the BTCV validation set.On MSD-Spleen, MSD-Liver, MSD-Pancreas, and KiTS, AMOS-pretrained models achieve 94.73%, 95.40%, 71.76%, and 88.33% mDice, respectively.
- Evaluation: AMOS evaluates segmentation with category-wise DSC and NSD scores averaged into mDice and mNSD, while also reporting model parameters and computational cost.
- Potential usages: AMOS contains five CT domains and three MRI domains, supporting extensions to OOD generalization, cross-modality learning, transfer learning, domain adaptation, and federated learning.
4 Experiments
The experiments benchmark CNN, Transformer, and hybrid segmentors on AMOS and examine generalization, cross-modality learning, and transfer. Results reveal strong baseline performance but substantial MRI OOD degradation and benefits from joint modalities and AMOS pretraining.
- Benchmark results: UNet achieves 88.87% and 85.59% mDice on AMOS-CT and AMOS-MRI, respectively, while outperforming newer methods by a margin.It also reaches 79.87% and 80.56% mNSD on AMOS-CT and AMOS-MRI.
- Benchmark results: Transformer-based models provide no obvious performance advantage over CNN-based models while incurring greater computational overhead.On AMOS-CT, UNet surpasses nnFormer and Swin-UNetr by over 2–3% mDice and 4–5% mNSD.
- Benchmark results: Large organs are segmented well across model families, but small organs such as the duodenum and adrenal gland remain difficult.
- Generalization results: On AMOS-MRI, UNet performance falls from 86.05% mDice on testing ID to 64.07% on testing OOD, whereas AMOS-CT shows nearly unchanged ID and OOD performance.The MRI gap is attributed to scanner-related distribution shift; UNet records 88.87% versus 89.04% mDice on CT ID and OOD tests.
- Extended explorations: Joint CT/MRI training improves individually trained models by +0.55% mDice and +0.81% mNSD on CT, and +2.14% mDice and +2.16% mNSD on MRI.The improvement remains after controlling for additional training data through random sample selection.
- Extended explorations: AMOS representations transfer beneficially to related target datasets without deleterious effects on substantially different domains such as MSD-Cardiac.
5 Discussion & Conclusion
AMOS is presented as a large-scale, diverse clinical benchmark for fair abdominal multi-organ segmentation evaluation and extended studies. Its scope is tempered by limitations in population and institutional coverage.
- AMOS supports general multi-organ segmentation benchmarking and studies of OOD generalization, cross-modality learning, and transfer learning.
- AMOS data were acquired at two hospitals from a subgroup of the Asian population, limiting population representativeness.
- Although multiple scanners were used, all scanners came from those two hospitals, which may affect performance on data from other medical centers.
A.1 Single-organ Datasets
Existing abdominal segmentation datasets vary substantially in organ coverage, sample size, modality, annotation scope, and acquisition setting. The reviewed benchmarks often focus on single organs or limited multi-organ targets.
- Single-organ datasets: MSD-Liver, MSD-Spleen, MSD-Prostate, MSD-Pancreas, and KiTS focus on individual organs or organ-specific structures across CT or MRI scans.
- Multi-organ datasets: BTCV contains 50 abdominal CT scans and targets 13 organs acquired at Vanderbilt University Medical Center.
- Multi-organ datasets: Chaos includes 20 CT liver cases and 20 MRI cases annotated for four organs, collected at Dokuz Eylul University Hospital from a healthy population.
- Multi-organ datasets: DenseVNet comprises 90 abdominal CT images with segmentation masks for eight organs assembled from TCIA Pancreas-CT and BTCV data.
- Multi-organ datasets: AbdomentCT-1K contains 1,132 cases covering liver, kidney, and pancreas annotations, including 1,112 three-dimensional CT scans from existing datasets.
- Multi-organ datasets: Word provides 150 cases with annotations for 16 organ types from patients with prostatic, cervical, or rectal cancer.
B.1 Data Acquisition
AMOS data were acquired across scanners and clinical protocols, while annotations were produced through a staged workflow combining automated initialization with radiologist refinement and supervision.
- Data acquisition: AMOS data were collected from eight scanners with different brands and institution-specific clinical acquisition protocols.
- Data acquisition: One CT acquisition example used 120 kVP, a 500 mm collection diameter, 500–800 ms exposure, 50–400 mA current, and 2.5–5 mm reconstructed section thickness.
- Annotation workflow: AMOS annotations began with coarse masks from a pretrained segmentation model, followed by refinement by five junior radiologists and supervision by three senior radiologists.
B.3 Data Distribution, Hosting, and Maintenance
AMOS is distributed under a noncommercial share-alike license, hosted on AWS Open Data, and maintained by its authors. The project also plans a repository for reporting annotation errors.
- Distribution and hosting: AMOS data are distributed under the CC BY-NC-SA license and hosted on the AWS open data platform.
- Distribution and hosting: The authors maintain the dataset and provide download and usage instructions through the AMOS challenge website.
- Maintenance: The authors plan to establish a GitHub repository to solicit possible annotation errors from data users.
B.4 Data Statistics
AMOS-CT and AMOS-MRI include patients with abdominal tumors or other abnormalities, with reported demographic distributions and controlled split composition. Scanner intensity and spatial properties are also analyzed.
- 500 CT scans included 314 males and 186 females, with patient ages spanning 14–94 years and a median of 54 years.
- 100 MRI scans included 55 males and 45 females, with patient ages spanning 22–85 years and a median of 50 years.
- The training, validation, and test splits were manually balanced for the distributions of the collected factors.
- The study analyzes intensity and spatial properties across data collected from different scanners and summarizes them in Table 9.
C.1 Implementation Details
The implementation follows nnUNet-based training and evaluation, with modality-specific preprocessing, crop-based training, augmentation, sliding-window testing, and DSC and NSD evaluation.
- Data Prepossessing: CT scans are clipped to [-991, 362] HU and normalized using subtraction of 50 and division by 141, while MRI scans use Z-score normalization.
- Baselines: The benchmark evaluates UNet, VNet, CoTr, nnFormer, UNetr, and Swin-UNetr using default configurations from their released codebases unless otherwise specified.
- Training Schedule: Training randomly crops modality-specific sub-volumes and uses one NVIDIA V100 GPU with batch size 2, alongside extensive geometric, intensity, noise, blur, and resolution augmentations.
- Training Schedule: Testing uses sliding-window inference with windows matching the training patch size, and applies augmentation such as flipping.
- Transfer learning: The transfer-learning experiments use AMOS-pretrained representations, standard fine-tuning protocols, and the same training and testing schedule across ten medical segmentation datasets.
- Segmentation quality is measured with Dice Similarity Coefficient and Normalized Surface Dice scores, where higher scores indicate better performance.
C.2 Additional Results
Additional experiments report organ-specific validation scores and examine transfer learning from AMOS-pretrained models across related and unrelated medical segmentation datasets.
- Class-wise results: Class-wise validation scores are reported for the AMOS-CT benchmark, covering 15 abdominal organ categories and their listed abbreviations.
- Transfer learning: Transfer learning is evaluated on ten medical segmentation datasets, including six related datasets with AMOS organ annotations and four unrelated datasets.