Source-linked AI summary
Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalities
Pim Moeskops, Jelmer M. Wolterink, Bas H. M. van der Velden, Kenneth G. A. Gilhuijs, Tim Leiner, Max A. Viergever, Ivana Išgum
TL;DR
Medical image segmentation spans multiple modalities and anatomical structures, but prior CNN applications were typically task-specific. This paper trains one CNN across brain MRI, breast MRI, and cardiac CTA tasks and finds equivalent segmentation performance to task-specific training, with very low cross-task confusion.
Problem
The paper investigates whether a single CNN can perform different medical-image segmentation tasks across modalities and anatomical structures.
Method
The same CNN architecture was trained on one, two, or all three tasks using triplanar input patches and task-dependent output classes.
Results
Multi-task training produced segmentation performance equivalent to task-specific training, while alien-task labels remained below 0.0005% per scan median in the three-task network.
Takeaways & Limitations
A single trained CNN instance can segment multiple tissue classes across multiple modalities and anatomical structures without task-specific network tailoring.
Abstract
from arXiv · showhide
Automatic segmentation of medical images is an important task for many clinical applications. In practice, a wide range of anatomical structures are visualised using different imaging modalities. In this paper, we investigate whether a single convolutional neural network (CNN) can be trained to perform different segmentation tasks. A single CNN is trained to segment six tissues in MR brain images, the pectoral muscle in MR breast images, and the coronary arteries in cardiac CTA. The CNN therefore learns to identify the imaging modality, the visualised anatomical structures, and the tissue classes. For each of the three tasks (brain MRI, breast MRI and cardiac CTA), this combined training procedure resulted in a segmentation performance equivalent to that of a CNN trained specifically for that task, demonstrating the high capacity of CNN architectures. Hence, a single system could be used in clinical practice to automatically perform diverse segmentation tasks without task-specific training.
1 Introduction
The paper asks whether one CNN can support segmentation across different modalities and anatomical tasks. This extends CNN-based medical segmentation beyond networks tailored to a single task.
- CNNs automatically extract features optimized for the classification task, unlike classical classifiers based on hand-crafted features.
- Prior medical-image studies used CNNs for individual segmentation problems while retaining problem-specific architectural optimizations.
- CNN architectures have shown generalisability across divergent natural-image and medical-image tasks, motivating their use with limited task-specific modification.
- The study tests one architecture and one trained instance across brain MRI, breast MRI, and cardiac CTA segmentation tasks.
2 Data
The study uses manually labelled brain MRI, breast MRI, and cardiac CTA datasets spanning different tissues, anatomical structures, and imaging protocols.
- 34 brain MRI scans were manually grouped into six tissue classes from 134 original labels.The classes were white matter, cortical grey matter, basal ganglia and thalami, ventricular cerebrospinal fluid, cerebellum, and brain stem.
- 34 breast MRI scans provided manually contoured pectoral-muscle segmentations.The images were acquired with a dedicated double breast array coil and resampled to isotropic voxels.
- Ten cardiac CTA scans were acquired with ECG triggering and contrast enhancement and resampled to 0.45×0.45×0.45 mm3 voxels.A human observer manually painted voxels to establish the reference standard.
3 Method
The method classifies every voxel from triplanar context using a shared CNN, while varying the number of tasks and output classes across training experiments.
- All voxels were labelled by the CNN across the seven training experiments.
- Each voxel was represented by three orthogonal 51×51 patches processed through 25 convolutional layers without subsampling.The three planes were axial, sagittal, and coronal; the architecture used small 3×3 convolution kernels.
- The convolutional features were combined into two 192-node fully connected layers followed by a softmax classifier.The output layer contained 2, 3, 7, 8, or 9 nodes depending on the tasks.
- Seven experiments trained the same model on one, two, or three task combinations, merging task-specific backgrounds into one class.
- Mini-batches were balanced across tasks and task-specific classes, and parameters were optimized with Adam using categorical cross-entropy.
4 Experiments and Results
Across the three segmentation tasks, multi-task networks produced accurate results comparable in learning behaviour to task-specific networks, with very little cross-task confusion.
- The data were split into 14/20, 14/20, and 6/4 training/test images for brain MRI, breast MRI, and cardiac CTA, respectively.Each network was trained with 25000 mini-batches per task.
- Evaluation used full test sets without post-processing beyond probability thresholding, with Dice-based operating points for breast MRI and cardiac CTA.
- Dice coefficients increased and results stabilized during training, while learning curves remained similar across experiments for each task.Repeating the brain-only experiment showed similar inter-experiment variation, indicating that slight differences were consistent with stochastic training variation.
- All four networks evaluated for each task accurately segmented the target tissues.The four networks comprised task-specific, two-task, and all-task training configurations.
- < 0.0005% was the median percentage of voxels per scan assigned to a class alien to the target for every task in the three-task network.
5 Discussion and Conclusions
A fixed CNN architecture achieved accurate segmentation across multiple modalities and anatomies, while multi-task training matched task-specific performance. The architecture also supports fast whole-image processing without post-processing beyond evaluation thresholding.
- Discussion and Conclusions: A single CNN instance segmented multiple tissue classes across multiple modalities and anatomical structures.The same architecture handled brain tissues, pectoral muscle, and coronary arteries.
- Discussion and Conclusions: The fixed CNN used triplanar orthogonal patches and modern deep-learning components, while omitting downsampling to enable fast whole-image processing.The implementation used batch normalisation, Adam optimisation, exponential linear units, very deep networks, and 1×1 convolution layers.
- Discussion and Conclusions: Multi-task training achieved segmentation performance equivalent to task-specific networks across the included tasks.This result was reported for brain MRI, breast MRI, and cardiac CTA.
- Discussion and Conclusions: Similarities between tasks, including shared pectoral-muscle presence and tissue appearance in T1-weighted MRI, led to very limited confusion.The authors propose investigating larger datasets, more tasks, and how representations are shared across tasks.
- Discussion and Conclusions: Example segmentations compare task-specific, additional-task, and all-task networks for brain MRI, breast MRI, and cardiac CTA.The figure presents input images, reference standards, and outputs from the different training experiments.