Source-linked AI summary
DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation
Guotai Wang, Maria A. Zuluaga, Wenqi Li, Rosalind Pratt, Premal A. Patel, Michael Aertsen, Tom Doel, Anna L. David, Jan Deprest, Sebastien Ourselin, Tom Vercauteren
TL;DR
Medical image segmentation remains difficult for fully automatic CNNs to make sufficiently accurate and robust for clinical use, while interactive methods can require burdensome user input. The paper introduces a CNN-based interactive framework that refines automatic segmentations using geodesic interaction maps, resolution-preserving prediction, and a trainable CRF with hard constraints. Across 2D placenta and 3D brain-tumor segmentation, DeepIGeoS improves automatic CNN results and achieves comparable or higher accuracy with fewer interactions and less time than traditional interactive methods.
Problem
Fully automatic CNN segmentations may still require refinement for clinical accuracy and robustness, while interactive methods can demand many user interventions.
Method
The framework uses one CNN for initial segmentation and another for interaction-based refinement, encoding interactions with geodesic distance maps and constrained trainable CRFs.
Results
DeepIGeoS improves automatic CNN segmentation and achieves comparable or higher accuracy with fewer user interventions and less time than traditional interactive methods.
Takeaways & Limitations
The framework supports efficient interactive refinement for 2D placenta segmentation from fetal MRI and 3D brain-tumor segmentation from FLAIR images.
Abstract
from arXiv · showhide
Accurate medical image segmentation is essential for diagnosis, surgical planning and many other applications. Convolutional Neural Networks (CNNs) have become the state-of-the-art automatic segmentation methods. However, fully automatic results may still need to be refined to become accurate and robust enough for clinical use. We propose a deep learning-based interactive segmentation method to improve the results obtained by an automatic CNN and to reduce user interactions during refinement for higher accuracy. We use one CNN to obtain an initial automatic segmentation, on which user interactions are added to indicate mis-segmentations. Another CNN takes as input the user interactions with the initial segmentation and gives a refined result. We propose to combine user interactions with CNNs through geodesic distance transforms, and propose a resolution-preserving network that gives a better dense prediction. In addition, we integrate user interactions as hard constraints into a back-propagatable Conditional Random Field. We validated the proposed framework in the context of 2D placenta segmentation from fetal MRI and 3D brain tumor segmentation from FLAIR images. Experimental results show our method achieves a large improvement from automatic CNNs, and obtains comparable and even higher accuracy with fewer user interventions and less time compared with traditional interactive methods.
1 INTRODUCTION
Medical image segmentation must be accurate and robust despite poor image quality, anatomical variation, pathology, and inconsistent boundary definitions. Interactive methods address these challenges but can burden users with many interactions, motivating CNN-based refinement with geodesic distances, resolution preservation, and trainable constrained CRFs.
- Automatic segmentation often lacks clinical accuracy and robustness because medical images contain noise, artifacts, low contrast, patient variation, pathology, and inconsistent boundary definitions.
- Interactive segmentation uses users’ anatomical knowledge to improve accuracy and robustness, but many required interactions increase user burden.
- CNNs achieve state-of-the-art automatic segmentation by learning features from large training datasets, with FCNs producing segmentation through a single forward pass.
- Repeated pooling and downsampling reduce feature-map resolution, causing blob-like outputs and poor accuracy for tiny structures; dilated convolution preserves resolution while expanding receptive fields.
- The paper targets efficient interactive segmentation by combining an initial CNN, a refinement CNN using user interactions and geodesic distance maps, a resolution-preserving architecture, and a trainable CRF with hard constraints.
2 RELATED WORKS
Prior work adapted CNNs and graphical models to biomedical and interactive segmentation, addressing feature learning, resolution, spatial consistency, and sparse user supervision. These approaches nevertheless include efficiency, interaction, or model-flexibility limitations that motivate the paper’s framework.
- Early patch- or region-based CNN adaptations improved accuracy over hand-crafted-feature methods but were inefficient at test time.
- U-Net, its 3D version, and V-Net adapted encoder–decoder CNN designs for biomedical segmentation, including 3D MRI volumes.
- Dilated convolution addresses resolution loss from pooling and downsampling while enlarging receptive fields for contextual information.
- Multi-scale CNN features improve segmentation accuracy through scaled inputs, concentric patches, or multi-stage image processing.
- Traditional interactive methods use clicks, contours, boxes, or scribbles, but low-level features can require many interactions on low-contrast images.
- Deep interactive methods use sparse annotations, scribbles, or bounding boxes to train CNNs for semi-automatic segmentation, including fetal MRI.
- CRFs improve segmentation through spatial consistency, commonly encouraging agreement between adjacent pixels with similar appearance.
- CRF-RNN enables end-to-end CNN–CRF training, but its weighted-Gaussian pairwise potentials are restrictive and not all parameters are trainable.
3 METHOD
DeepIGeoS combines an automatic proposal network with an interaction-driven refinement network, encoding user corrections through geodesic maps and constrained CRF inference. Its resolution-preserving CNNs and learnable CRF components support 2D and 3D medical-image segmentation.
- Interactive framework: P-Net produces an initial automatic segmentation, while R-Net refines it using the original image, initial segmentation, and user interactions.The framework is designed for interactive refinement of 2D and 3D medical images.
- User interaction encoding: User clicks or scribbles are converted into foreground and background geodesic distance maps and concatenated with the image and initial segmentation as R-Net input.The resulting input has CI+3 channels, preserving the spatial size of the original image.
- User interaction encoding: Geodesic distance accounts for image context, unlike Euclidean distance, helping distinguish neighboring pixels with different appearances and improve consistency in homogeneous regions.The method applies geodesic distance transforms specifically to encode user interactions for CNN-based segmentation.
- Back-propagatable CRF: CRF-Net(f) connects to P-Net and uses a learnable freeform Pairwise-Net for pairwise potentials based on feature differences and Euclidean pixel distance.The freeform function replaces Gaussian pairwise potentials and is represented by a fully connected neural network learned during training.
- Back-propagatable CRF: CRF-Net(fu) connects to R-Net and incorporates user scribbles as hard constraints, forcing scribbled pixels to take their specified labels during iterative updates.This extends the back-propagatable CRF framework to interactive refinement while retaining jointly trainable inference components.
4.1 Comparison Methods and Evaluation Metrics
The evaluation compares CNN and CRF variants, user-interaction strategies, and established interactive segmentation methods using Dice and ASSD.
- Comparison Methods: The study compares P-Net with FCN, DeepLab, DeepMedic, and HighRes3DNet for 2D and 3D segmentation.FCN and DeepLab are evaluated for 2D placenta segmentation, while DeepMedic and HighRes3DNet are evaluated for 3D segmentation.
- Comparison Methods: CRF-Net(f) is compared with Dense CRF and CRF-Net(g), which uses Gaussian pairwise potentials.Dense CRF parameters are manually tuned, whereas CRF-Net(f) learns freeform pairwise potentials.
- Comparison Methods: User-interaction handling is compared through min-cut editing, Euclidean-distance inputs to R-Net, and the proposed geodesic-distance R-Net.These alternatives evaluate both traditional editing and CNN-based refinement strategies.
- Comparison Methods: DeepIGeoS is also compared with Geodesic Framework, Graph Cuts, and Random Walks for 2D slices.These methods represent probability-based, min-cut, and random-walk interactive segmentation approaches.
- Evaluation Metrics: Dice score and average symmetric surface distance (ASSD) are used for quantitative evaluation.ASSD uses shortest Euclidean distances between segmented and ground-truth surface points, with Student’s t-tests used for significance comparisons.
4.2 2D Placenta Segmentation from Fetal MRI
In fetal MRI placenta segmentation, 2D P-Net provides detailed automatic results, while CRF-Net(f) and geodesic interactive refinement improve accuracy with fewer user interactions and less time than traditional methods.
- Automatic segmentation: 2D P-Net produces more detailed placenta segmentations than FCN, DeepLab, and 2D P-Net(b5), although some mis-segmented regions remain.The comparison attributes blob-like results from FCN and DeepLab mainly to downsampling and upsampling.
- Automatic segmentation: 84.78±11.74% Dice and 2.09±1.53 pixels ASSD are achieved by 2D P-Net on the testing data.
- CRF refinement: CRF-Net(f) improves placenta segmentation more than Dense CRF and CRF-Net(g), with freeform pairwise potentials producing the best CRF performance.Dense CRF remains close to 2D P-Net, while CRF-Net(g) and CRF-Net(f) significantly improve segmentation; CRF-Net(f) is better than CRF-Net(g).
- Interactive refinement: Using the same first-iteration interactions, 2D R-Net with geodesic distance and CRF-Net(fu) achieves 89.31±5.33% Dice and 1.22±0.55 pixels ASSD.The combined method is more accurate than the other refinement methods under identical initial segmentation and interactions.
- Comparison with other interactive methods: DeepIGeoS reaches similar accuracy to Geodesic Framework, Graph Cuts, Random Walks, and SlicSeg while requiring far fewer user interactions and less time.In the visual comparison, DeepIGeoS uses two short strokes for refinement, whereas the other methods require substantially more scribbles.
4.3 3D Brain Tumor Segmentation from FLAIR Images
For 3D brain tumor segmentation from FLAIR images, 3D P-Net and CRF-Net(f) improve automatic results, and DeepIGeoS achieves higher accuracy with substantially less user time than competing interactive methods.
- Automatic segmentation: 3D P-Net produces more accurate brain tumor segmentations than DeepMedic, HighRes3DNet, and 3D P-Net(b5).
- Automatic segmentation: 86.68±7.67% Dice and 2.14±2.17 pixels ASSD are achieved by 3D P-Net.The network has far fewer parameters than HighRes3DNet and can infer a 3D volume in interactive time.
- CRF refinement: CRF-Net(f) provides a more noticeable improvement than Dense CRF, and only CRF-Net(f) significantly improves segmentation over 3D P-Net.Dense CRF uses manually tuned parameters and is applied as post-processing without end-to-end learning.
- Interactive refinement: With the same user interactions, 3D R-Net refinement compares geodesic and Euclidean distance transforms, with or without CRF-Net(fu), against min-cut editing.The supplied passage describes the comparison setup but does not report its final quantitative outcome.
- Comparison with other interactive methods: DeepIGeoS outperforms GeoS and ITK-SNAP on a lower-contrast tumor example, achieves higher quantitative accuracy, and requires about one third of their user time.
5 CONCLUSION
The framework combines automatic CNN segmentation with interaction-driven refinement for 2D and 3D medical images, improving accuracy while reducing user time relative to traditional interactive methods.
- P-Net produces an initial segmentation, while R-Net refines it using user interactions represented as geodesic distance maps.
- The framework uses dilated convolutions for resolution-preserving dense prediction and incorporates interactions as hard constraints in a learnable CRF.
- Experiments on 2D fetal-MRI placenta and 3D FLAIR brain-tumor segmentation outperform automatic CNNs.
- The method requires far less user time than traditional interactive methods and achieves higher accuracy for 3D brain-tumor segmentation.
- The framework is presented as extensible to multiple organs in future work.