Source-linked AI summary
A 3D Coarse-to-Fine Framework for Volumetric Medical Image Segmentation
Zhuotun Zhu, Yingda Xia, Wei Shen, Elliot K. Fishman, Alan L. Yuille
TL;DR
Pancreas segmentation from CT is difficult because the organ is small, variable, and affected by abnormalities, while 3D models face limited annotated data and computational resources. The paper proposes a 3D coarse-to-fine framework that combines whole-volume localization with localized refinement. It achieves state-of-the-art DSC results on healthy and pathological pancreas datasets, including over 2% average improvement on NIH and a worst-case result near 70%.
Problem
Pancreas segmentation is challenging because the organ is small, anatomically and texturally variable, and altered by abnormalities, while 3D processing is constrained by data and computational resources.
Method
The method uses 3D ResDSN Coarse and ResDSN Fine models in a coarse-to-fine framework, applying whole-volume localization before localized refinement with different overlap sizes.
Results
The framework achieves state-of-the-art DSC performance on normal and abnormal pancreas datasets, improving NIH average DSC by nearly 2.2% and the minimum DSC from 62.53% to 69.62%.
Takeaways & Limitations
The framework combines rich 3D spatial information with relatively low time cost and improves difficult pancreas cases across normal and abnormal datasets.
Takeaways & Limitations
The work mainly focuses on pancreas segmentation and leaves cyst and tumor segmentation, along with investigating error causes, for future work.
Abstract
from arXiv · showhide
In this paper, we adopt 3D Convolutional Neural Networks to segment volumetric medical images. Although deep neural networks have been proven to be very effective on many 2D vision tasks, it is still challenging to apply them to 3D tasks due to the limited amount of annotated 3D data and limited computational resources. We propose a novel 3D-based coarse-to-fine framework to effectively and efficiently tackle these challenges. The proposed 3D-based framework outperforms the 2D counterpart to a large margin since it can leverage the rich spatial infor- mation along all three axes. We conduct experiments on two datasets which include healthy and pathological pancreases respectively, and achieve the current state-of-the-art in terms of Dice-Sørensen Coefficient (DSC). On the NIH pancreas segmentation dataset, we outperform the previous best by an average of over 2%, and the worst case is improved by 7% to reach almost 70%, which indicates the reliability of our framework in clinical applications.
1. Introduction
Pancreas segmentation is difficult because the organ is small, highly variable, and affected by abnormalities, while 3D processing faces memory and efficiency constraints. The paper proposes a 3D coarse-to-fine framework that uses whole-volume context for localization and localized context for refinement.
- The pancreas is challenging to segment because it occupies a small volume, varies in appearance and anatomy, and may contain cystic abnormalities.
- 3D networks are difficult to train on whole CT volumes because limited GPU memory requires sub-volume training and sliding-window inference.Overlapping windows improve robustness but create an accuracy–testing-time trade-off.
- The framework trains ResDSN Coarse on whole-volume sub-volumes and ResDSN Fine on pancreas bounding-box sub-volumes to combine global localization with local refinement.
- Coarse-to-fine processing uses small overlap for rough localization and larger overlap for refinement, targeting both low inference cost and accurate segmentation.The coarse stage removes unrelated background so the fine stage can learn pancreas–background distinctions more easily.
- The framework is presented as an early 3D approach for both normal and abnormal pancreas segmentation, achieving state-of-the-art results with relatively low time cost.The authors also describe it as generic enough to apply to other medical organs.
2. Related Work
Volumetric segmentation methods are organized around 2D slice-based and 3D volume-aware CNNs. The related work highlights the spatial-context limitations of 2D approaches and the computational constraints motivating the paper’s 3D coarse-to-fine design.
- 2D CNNs for Volumetric Segmentation: 2D CNN methods segment volumes slice by slice from multiple views and fuse the resulting predictions into a 3D segmentation.
- 2D CNNs for Volumetric Segmentation: 2D methods cannot fully exploit the 3D context encoded in volumetric medical images.
- 3D CNNs for Volumetric Segmentation: 3D segmentation architectures extend CNN operations to volumetric data and include designs such as 3D U-Net, V-Net, residual networks, and auxiliary supervision.
- 3D CNNs for Volumetric Segmentation: The proposed framework addresses volumetric inference with a coarse stage followed by a fine stage using different sliding-window settings.
- 3D CNNs for Volumetric Segmentation: Few methods had leveraged 3D spatial information for challenging pancreas segmentation, while some 3D architectures remained constrained by computationally demanding designs.
3. Method
The method uses a 3D coarse-to-fine segmentation framework that first localizes the pancreas from the whole CT volume, then refines segmentation within a cropped region using residual and deeply supervised networks.
- Segmentation Model: The segmentation model predicts a binary per-voxel volume with negative log-likelihood training, auxiliary side outputs, and l2 regularization in the overall objective.Thresholding voxel probabilities yields the binary mask, while auxiliary branches share early mainstream-network weights and support deep supervision.
- Coarse-to-Fine Framework: The framework trains a coarse 3D FCN on the whole CT volume, crops a pancreas-centered region, and applies a fine 3D FCN before restoring the result to the original volume.The coarse prediction defines the crop, while DeCrop maps the fine prediction back into the original spatial dimensions.
- Coarse-to-Fine Framework: The coarse stage removes unrelated background so the fine stage operates on a smaller, less variable region and can focus on pancreas-discriminative cues.The crop uses a rectangular bounding cube around predicted pancreas voxels with an empirically selected padding margin.
- Network Architecture: The 3D network uses encoder-decoder paths with convolution, batch normalization, ReLU, pooling, deconvolution, and residual connections for dense volumetric prediction.The encoder learns increasingly high-level representations, while the decoder restores resolution; residual connections transfer low-level cues to higher-level layers.
- Network Architecture: ResDSN favors long element-wise residual sums and deep supervision over short residual connections, reducing channels for efficient computation while supporting convergence.The design differs from concatenation-based 3D U-Net and V-Net connections by summing lower- and higher-level features element-wise.
4. Experiments
Experiments evaluate the 3D coarse-to-fine framework on healthy and pathological pancreas CT datasets using DSC, comparing it with prior methods and its coarse stage.
- 4.1. Training and Testing: The coarse stage samples 64×64×64 sub-volumes across the whole CT volume, while the fine stage samples them within the pancreas-region estimate.Sliding-window overlap parameters are set to n = 6 for coarse localization and n = 12 for fine estimation.
- 4.2. NIH Pancreas Dataset: The NIH dataset contains 82 contrast-enhanced abdominal CT volumes with varying depth and in-plane resolution.The experiments retain spatial-resolution variation rather than normalizing all volumes to one resolution.
- 4.2. NIH Pancreas Dataset: Nearly 2.2% higher average DSC than previous state-of-the-art methods on the NIH dataset, with Table 2 reporting the overall comparison.The framework also has lower DSC standard deviation, indicating more stable performance across CT cases.
- 4.2. NIH Pancreas Dataset: Figure 4 shows ResDSN C2F recovering pancreas regions missed by ResDSN Coarse, with improvements of up to 10% DSC when coarse segmentation is poor.The method provides little improvement when the coarse segmentation is already very good.
5. Discussion
The discussion evaluates residual connections, testing-time trade-offs, and deep supervision within the proposed 3D coarse-to-fine framework. ResDSN C2F combines strong segmentation performance with reduced testing time and improved stability.
- 5.1. Residual Connection: ResDSN outperforms FResDSN, SResDSN, and DSN in average DSC with a small standard deviation, motivating its use in the coarse stage.The comparison concerns four residual-connection configurations under otherwise matched settings.
- 5.1. Residual Connection: Figure 4 compares same-slice axial predictions from ResDSN Coarse and ResDSN C2F using DSC values and color-coded ground truth, predictions, and overlaps.Red denotes ground truth, green prediction, and yellow overlap.
- 5.2. Time Efficiency: ResDSN C2F achieves the best segmentation results while reducing average testing time by 36%, from 382s to 245s per case versus ResDSN Fine.The framework uses smaller overlap in the coarse stage and larger overlap in the fine stage to balance efficiency and accuracy.
- 5.2. Time Efficiency: ResDSN Coarse is the fastest but least accurate option, whereas ResDSN Fine is the most time-consuming.This contrast motivates combining the two stages rather than using either stage alone.
- 5.3. Deep Supervision: 17.79% higher minimum DSC and 0.53% higher average DSC are reported for ResDSN C2F than Res C2F with deep supervision.ResDSN C2F is slightly worse on maximum DSC but is described as more stable overall.
6. Conclusion
The paper presents ResDSN integrated with a 3D coarse-to-fine framework for accurate and efficient pancreas segmentation. It reports strong performance on normal and abnormal pancreases while identifying extension to other organs and lesion types as future work.
- 6. Conclusion: The proposed ResDSN coarse-to-fine framework targets high segmentation accuracy and low time cost by leveraging 3D spatial information.The conclusion describes it as an early effort to apply 3D networks to challenging pancreas segmentation.
- 6. Conclusion: The framework achieves good segmentation accuracy on both normal and abnormal pancreases.The conclusion explicitly extends the reported scope beyond healthy anatomy.
- 6. Conclusion: Future work will investigate causes of inaccurate segmentation and extend the framework to cyst and tumor segmentation.The paper also notes possible application to other small organs, including spleen, duodenum, and gallbladder.