Source-linked AI summary
Adversarial Synthesis Learning Enables Segmentation Without Target Modality Ground Truth
Yuankai Huo, Zhoubing Xu, Shunxing Bao, Albert Assad, Richard G. Abramson, Bennett A. Landman
TL;DR
Deep-learning segmentation requires new manual labels across modalities or disease cohorts, motivating methods that reuse labels from another modality. EssNet jointly synthesizes CT from unpaired MRI and segments splenomegaly without CT training labels, achieving higher median DSC than the compared strategies. The study’s main architectural limitation is its 2-D networks evaluated in 3-D.
Problem
Deep-learning segmentation commonly requires manually labeled training images for new modalities or abnormal-organ cohorts, creating a need to reuse labels across modalities.
Method
EssNet performs unpaired MRI-to-CT synthesis and CT splenomegaly segmentation simultaneously using MRI labels without CT ground-truth labels.
Results
EssNet achieved significantly higher median Dice similarity coefficient than the two-stage CycleGAN strategy and the compared canonical methods.
Takeaways & Limitations
The approach supports CT splenomegaly segmentation without using CT labels and can estimate labels for multiple organs when CT labels are unavailable.
Takeaways & Limitations
The networks are 2-D but were assessed in 3-D because of time and memory concerns.
Abstract
from arXiv · showhide
A lack of generalizability is one key limitation of deep learning based segmentation. Typically, one manually labels new training images when segmenting organs in different imaging modalities or segmenting abnormal organs from distinct disease cohorts. The manual efforts can be alleviated if one is able to reuse manual labels from one modality (e.g., MRI) to train a segmentation network for a new modality (e.g., CT). Previously, two stage methods have been proposed to use cycle generative adversarial networks (CycleGAN) to synthesize training images for a target modality. Then, these efforts trained a segmentation network independently using synthetic images. However, these two independent stages did not use the complementary information between synthesis and segmentation. Herein, we proposed a novel end-to-end synthesis and segmentation network (EssNet) to achieve the unpaired MRI to CT image synthesis and CT splenomegaly segmentation simultaneously without using manual labels on CT. The end-to-end EssNet achieved significantly higher median Dice similarity coefficient (0.9188) than the two stages strategy (0.8801), and even higher than canonical multi-atlas segmentation (0.9125) and ResNet method (0.9107), which used the CT manual labels.
1. INTRODUCTION
Deep-learning segmentation often requires new manual labels for different modalities or abnormal-organ cohorts. The paper proposes reusing MRI labels through unpaired synthesis and end-to-end segmentation to avoid CT labels.
- Deep-learning segmentation typically requires manually tracing new training data for new imaging modalities or disease cohorts.
- Normal-spleen models fail to capture the spatial variation of splenomegaly, making direct reuse across disease settings inadequate.
- CycleGAN enables unpaired inter-modality synthesis, allowing labels from one modality to support training for another.
- EssNet jointly performs MRI-to-CT synthesis and CT splenomegaly segmentation using unpaired scans and MRI labels without CT ground-truth labels.
2. DATA
The experiments used unpaired whole-abdomen MRI and CT data, with extensive multi-organ labels on MRI and spleen labels on CT.
- The dataset comprised 60 whole-abdomen MRI T2w scans and 19 whole-abdomen CT scans containing splenomegaly.
- Each MRI had six manually delineated labels, whereas each CT scan had one manually traced spleen label.
- An additional 75 whole-abdomen CT scans with normal spleens trained a baseline DCNN method.
3. METHOD
EssNet combines CycleGAN-based bidirectional synthesis with a segmentation branch in an end-to-end network. Its training uses adversarial, cycle-consistency, and segmentation losses, while evaluation uses DSC and statistical testing.
- EssNet uses two ResNet generators and two PatchGAN discriminators for bidirectional MRI–CT synthesis with unpaired data.
- The segmentation network is concatenated after the MRI-to-CT generator, producing segmentation estimates from generated CT images during forward Path A.
- Five losses jointly optimize the network: two adversarial losses, two cycle-consistency losses, and one segmentation loss.
- The total objective weights adversarial terms by λ_1=1 and λ_2=1, cycle terms by λ_3=10 and λ_4=10, and segmentation by λ_5=1.
- Testing used only the trained segmentation network, with Dice similarity coefficient as the metric and Wilcoxon signed-rank tests at p<0.05.
4. EXPERIMENTS
The experiments compared canonical supervised and baseline approaches with CycleGAN+Seg. and EssNet for CT splenomegaly segmentation. The latter methods were trained without CT labels, while CT labels were reserved for evaluation.
- The evaluation compared SSNet trained on normal-spleen CT scans, AGMM multi-atlas segmentation, a leave-one-subject-out ResNet, CycleGAN+Seg., and EssNet.
- CycleGAN+Seg. and EssNet used 60 splenomegaly MRI scans and 19 splenomegaly CT scans, with MRI labels for training and CT labels withheld from training.
- The experiments used identical training, testing, and network hyperparameter settings for CycleGAN+Seg. and EssNet to compare independent versus end-to-end frameworks.
- Qualitative comparisons included three canonical CT-label methods and two methods trained without CT labels, with Fig. 4 showing the corresponding segmentation outputs.
- Quantitative performance across all CT testing images was summarized with boxplots, marking significant differences with “*” and nonsignificant differences with “N.S.”
5. RESULTS
EssNet produced spleen segmentations and additional abdominal-organ estimates, while quantitative evaluation compared segmentation strategies across all CT scans.
- EssNet estimated spleen, liver, left-kidney, right-kidney, and stomach segmentations in qualitative results from three subjects.The subjects represented the lowest, median, and highest EssNet Dice similarity coefficients.
- Quantitative results across all CT scans were summarized as a boxplot, with significance marked by “*” and nonsignificance by “N.S.”
6. CONCLUSION AND DISCUSSION
The study presents EssNet as an end-to-end approach for unpaired synthesis and CT splenomegaly segmentation without CT ground-truth labels. Its 2-D architecture, assessed in 3-D, remains a stated limitation and motivates future 3-D development.
- EssNet simultaneously performed MRI-to-CT synthesis, CT-to-MRI synthesis, and CT segmentation without CT ground-truth labels.It also estimated liver, kidney, and stomach labels, whereas canonical CT-only methods could not do so without those labels.
- EssNet significantly outperformed two-stage CycleGAN+Seg. and performed comparably to AGMM MAS and ResNet, which used CT labels.CycleGAN+Seg. performed significantly worse than ResNet, while SSNet trained on normal-spleen CT images was significantly worse than the other methods.
- The networks were 2-D but assessed in 3-D because of time and memory concerns, motivating 3-D consistency post-processing or a 3-D EssNet.