Source-linked AI summary
An application of cascaded 3D fully convolutional networks for medical image segmentation
Holger R. Roth, Hirohisa Oda, Xiangrong Zhou, Natsuki Shimizu, Ying Yang, Yuichiro Hayashi, Masahiro Oda, Michitaka Fujiwara, Kazunari Misawa, Kensaku Mori
TL;DR
Medical CT segmentation must handle severe background–foreground imbalance and structures ranging from large organs to thin vessels. The paper uses cascaded 3D FCNs in a coarse-to-fine pipeline, where a first network generates candidate regions for finer second-stage prediction. The method produces competitive results on unseen clinical CT data, with 3D features particularly advantageous for smaller organs and vessels.
Problem
Foreground–background imbalance in whole-volume FCN training leads to inaccurate boundaries, especially for smaller organs and vessels.
Method
A 3D U-Net-based cascade uses a first FCN to generate candidate regions and a second FCN to perform finer prediction within them.
Results
The method compares favorably with recent state-of-the-art work on an unseen clinical CT dataset, and 3D convolutional features are advantageous for smaller organs and vessels.
Takeaways & Limitations
The cascaded approach provides an efficient way to deploy volumetric FCNs for clinical CT segmentation, while a single model can cover large organs and vessels.
Takeaways & Limitations
Overlapping sub-volume predictions remain part of testing, although increasing GPU memory could reduce the need for them.
Abstract
from arXiv · showhide
Recent advances in 3D fully convolutional networks (FCN) have made it feasible to produce dense voxel-wise predictions of volumetric images. In this work, we show that a multi-class 3D FCN trained on manually labeled CT scans of several anatomical structures (ranging from the large organs to thin vessels) can achieve competitive segmentation results, while avoiding the need for handcrafting features or training class-specific models. To this end, we propose a two-stage, coarse-to-fine approach that will first use a 3D FCN to roughly define a candidate region, which will then be used as input to a second 3D FCN. This reduces the number of voxels the second FCN has to classify to ~10% and allows it to focus on more detailed segmentation of the organs and vessels. We utilize training and validation sets consisting of 331 clinical CT images and test our models on a completely unseen data collection acquired at a different hospital that includes 150 CT scans, targeting three anatomical organs (liver, spleen, and pancreas). In challenging organs such as the pancreas, our cascaded approach improves the mean Dice score from 68.5 to 82.2%, achieving the highest reported average score on this dataset. We compare with a 2D FCN method on a separate dataset of 240 CT scans with 18 classes and achieve a significantly higher performance in small organs and vessels. Furthermore, we explore fine-tuning our models to different datasets. Our experiments illustrate the promise and robustness of current 3D FCN based semantic segmentation of medical images, achieving state-of-the-art results. Our code and trained models are available for download: https://github.com/holgerroth/3Dunet_abdomen_cascade.
1. Introduction
The paper applies 3D fully convolutional networks to multi-organ and vessel segmentation in clinical CT, addressing boundary inaccuracies caused by foreground–background imbalance. Its cascaded coarse-to-fine design narrows the search space so a second network can focus on finer prediction.
- Motivation: 3D FCNs enable end-to-end voxel-wise segmentation of volumetric medical images and can handle several anatomical structures in clinical CT.The approach is compared with 2D FCNs on the same images.
- Motivation: Foreground–background imbalance causes whole-volume FCNs to prioritize background classification, producing boundary inaccuracies in smaller organs and vessels.The paper identifies the pancreas, gallbladder, and vessels as particularly affected.
- Proposed approach: The second-stage FCN focuses on boundary regions after the first stage generates a candidate region in a coarse-to-fine cascade.The first stage uses about 40% of voxels, while the second uses about 10%, narrowing the classification search space.
- Related work: Earlier multi-organ segmentation methods include statistical shape models, image registration, and multi-atlas label fusion.These approaches have been applied to clinical and abdominal imaging.
- Related work: Deep-learning approaches have addressed single-organ and multi-organ segmentation using 2D, 2.5D, and 3D convolutional architectures.The cited applications include prostate, liver, pancreas, and abdominal multi-organ segmentation.
- Contribution: The proposed method uses one automatically learned model to segment both large organs and vessels, unlike approaches requiring separate models for each organ.It also differs from cascaded methods constrained to single-organ bounding boxes or slice-wise 2D processing.
2. Methods
The method uses a multi-class 3D U-Net with weighted loss and a two-stage coarse-to-fine cascade for multi-organ and vessel segmentation. Candidate-region reduction, augmentation, and tiled inference support detailed voxel-wise predictions while controlling computation.
- Network architecture: The network is a 3D extension of U-Net, a fully convolutional architecture that produces dense voxel-to-voxel predictions using up-convolutions.The architecture uses skip connections to remap lower-resolution feature maps into denser prediction space.
- Loss function: Weighted voxel-wise cross-entropy gives greater weight to smaller organs and vessels to counter background and class-size imbalance.The weights balance common background voxels against less frequent structures such as the pancreas and vessels.
- Coarse-to-fine prediction: The first stage uses thresholding and morphological operations to create a patient-body candidate region containing about 40% of voxels.The mask reduces the voxels used for loss computation and the input regions shown during training.
- Coarse-to-fine prediction: Using dilation radius r = 3 gives recall above 99% and false-positive rates of approximately 10% for each organ on the training and validation sets.This radius provides the reported trade-off between retaining foreground anatomy and limiting false positives.
- Training and inference: Training uses batch normalization, random 3D elastic deformations, rotations, translations, and tiled inference with non-overlapping or overlapping tiles.Overlapping-tile probabilities are averaged, while non-overlapping tiles are used in the first stage for greater speed.
3. Experiments & Results
Experiments evaluate cascaded 3D FCNs on clinical abdominal CT data, including an unseen cross-hospital test set and comparisons with other segmentation methods. The results support improved segmentation of smaller structures and strong pancreas performance, while direct comparisons remain difficult across datasets and evaluation schemes.
- Training and validation: 331 contrast-enhanced abdominal CT images were split into 281 training and 50 validation cases for network development.The scans were acquired in the portal venous phase and contained 460–1177 slices of 512 × 512 pixels.
- Training and validation: r = 3 achieved a good trade-off, with recall above 99% and false-positive rates around 10% for each organ on training and validation sets.The comparison varied the dilation radius of first-stage prediction maps.
- Training and validation: 20.6 percentage points improved the artery mean Dice score, from 59.0 to 79.6%, while pancreas Dice improved from 54.8 to 63.1%.The effect was less pronounced for large organs such as the liver, spleen, and stomach.
- Testing: 82.2% ±10.2% was the second-stage pancreas Dice score on the unseen 150-scan test set, versus 75.1% ±15.4% for the previous state of the art.The test data came from a different hospital, scanners, and research study; its constrained field of view likely contributed to improved liver and pancreas performance.
- Comparison to other methods: The proposed method achieved the highest reported testing performance on the in-house pancreas dataset, while results on TCIA Pancreas-CT were comparable with recent pancreas-specific methods.The comparison used an in-house dataset of 150 CT images and a public dataset of 82 patient images with the same 4-fold cross-validation split as prior work.
- Comparison to other methods: The study also compared the cascaded 3D FCN against a 2D FCN majority-voting method on 240 CT scans with 18 manually annotated organs.The cited comparison specifically evaluates performance across multiple anatomical structures.
4. Fine-tuning to other datasets
The authors fine-tune a general 3D U-Net model across datasets to segment more anatomical structures in a whole-torso setting. Transfer learning yields strong scores for several organs and slight improvements over training from scratch, especially for kidneys.
- Fine-tuning approach: A general 3D U-Net trained on abdominal organ labels was fine-tuned for smaller datasets, more detailed classification tasks, and different fields of view.The approach uses learned features from a broader CT dataset before adapting to a target task.
- Fine-tuning approach: 20 contrast-enhanced CT images with 20 labeled anatomical structures were used for fine-tuning from the broader abdominal model.The fine-tuning process is illustrated as transfer from 8 abdominal structures to 20 whole-torso structures.
- Fine-tuning results: Dice scores on the unseen whole-torso test set were 0.96 for right lung, 0.97 for left lung, 0.95 for liver, 0.90 for spleen, 0.90 for right kidney, 0.88 for left kidney, 0.77 for gall bladder, and 0.36 for pancreas.The relatively lower pancreas score was attributed to cross-dataset contrast variation and greater pancreas shape and intensity variability.
- Fine-tuning results: Transfer learning produced slight improvements over learning from scratch, especially for the kidneys.The authors also note that data augmentation already provided good performance when training from scratch for this application.
5. Discussion
The cascaded approach improves segmentation particularly for smaller and thinner organs and vessels, while comparisons suggest different strengths for 2D and 3D FCNs. Tests across hospitals and scanners indicate robustness, although alternative losses and optimizers remain unexplored.
- The cascaded approach improves performance especially for smaller, thinner organs such as arteries and veins, with no post-processing applied.Its advantage is particularly visible compared with recent 2D FCN approaches.
- 2D FCNs may suffice for larger and midsized organs, whereas 3D convolutional kernels are important for distinguishing thin and small organs.The combination of orthogonally applied 2D FCNs slightly outperforms the 3D approach for some organs.
- Using different hospitals and scanners for training/validation and testing supports the method’s generalizability and robustness to differences in image quality and populations.
- Alternative loss functions such as the Dice score and optimizers beyond stochastic gradient descent could further improve performance.All experiments used Caffe’s stochastic gradient descent solver.
- Future directions include combining predictions from different models, adding anatomical constraints, and reducing the need for overlapping sub-volume predictions as GPU memory grows.
6. Conclusion
The paper concludes that cascaded 3D U-Net deployment produces competitive clinical CT segmentation results efficiently on a single GPU. It also identifies advantages for smaller organs and vessels and supports further application through released models and code.
- Cascaded volumetric 3D U-Nets achieve competitive medical-image segmentation on clinical CT data while being efficiently deployed on a single GPU.
- Overlapping tiles during testing improve results with only moderate additional computational cost.
- The results indicate that 3D convolutional features are advantageous for detecting smaller organs and vessels.
- The authors identify hybrid 2D and 3D FCN architectures at multiple scales as a promising future direction.
- Code, pre-trained models, and fine-tuned models are available for further applications and fine-tuning to different datasets.
- The paper reports support from MEXT KAKENHI and the JPSP International Bilateral Collaboration Grant, alongside no declared conflicts of interest.