PaperlayerView exact revision on arXiv

Source-linked AI summary

SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth

Yuankai Huo, Zhoubing Xu, Hyeonsoo Moon, Shunxing Bao, Albert Assad, Tamara K. Moyo, Michael R. Savona, Richard G. Abramson, Bennett A. Landman

arXiv:1810.06498v2cs.CV

TL;DR

Limited generalization and the need for target-modality manual labels motivate SynSeg-Net. The paper trains an end-to-end synthesis-and-segmentation network from unpaired source and target images with source-only labels, achieving better performance than two-stage methods and comparable performance to target-label training in certain scenarios.

  • Problem

    DCNN segmentation methods have limited generalizability, and applying them to new modalities or disease cohorts typically requires manually traced target-modality training images.

  • Method

    SynSeg-Net jointly uses CycleGAN-based cross-modality synthesis and DCNN segmentation, trained with unpaired source and target images and manual labels only from the source modality.

  • Results

    Across MRI-to-CT splenomegaly and CT-to-MRI TICV experiments, SynSeg-Net outperformed two-stage methods and achieved comparable performance to target-label training in certain scenarios.

  • Takeaways & Limitations

    The results support synthetic segmentation as a way to train target-modality segmentation networks without target-modality ground-truth labels.

  • Takeaways & Limitations

    The study evaluates segmentation rather than synthesized-image quality, and its use of unpaired images without same-patient multimodal data limits synthesis validation.

Abstract

from arXiv · show

A key limitation of deep convolutional neural networks (DCNN) based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality without having manual labels. SynSeg-Net is trained by using (1) unpaired intensity images from source and target modalities, and (2) manual labels only from source modality. SynSeg-Net is enabled by the recent advances of cycle generative adversarial networks (CycleGAN) and DCNN. We evaluate the performance of the SynSeg-Net on two experiments: (1) MRI to CT splenomegaly synthetic segmentation for abdominal images, and (2) CT to MRI total intracranial volume synthetic segmentation (TICV) for brain images. The proposed end-to-end approach achieved superior performance to two stage methods. Moreover, the SynSeg-Net achieved comparable performance to the traditional segmentation network using target modality labels in certain scenarios. The source code of SynSeg-Net is publicly available (https://github.com/MASILab/SynSeg-Net).

I. INTRODUCTION

SynSeg-Net addresses limited segmentation generalizability by training target-modality networks from unpaired source and target images with manual labels only in the source modality. It is evaluated on MRI-to-CT splenomegaly and CT-to-MRI TICV segmentation, outperforming two-stage methods and matching target-label training in certain scenarios.

  • I. INTRODUCTION: CycleGAN advances enable high-quality cross-modality synthesis from unpaired training images.Earlier synthesis methods typically required paired training data across modalities.
  • I. INTRODUCTION: SynSeg-Net trains a target-modality DCNN using unpaired source and target images plus manual segmentations only from the source modality.The approach uses cross-modality synthesis learning to reduce manual segmentation effort.
  • I. INTRODUCTION: The experiments cover MRI-to-CT splenomegaly segmentation and CT-to-MRI total intracranial volume segmentation.The first experiment uses CT splenomegaly without spleen labels on CT images.
  • I. INTRODUCTION: The proposed end-to-end approach outperformed two-stage methods and achieved comparable performance to target-label training in certain scenarios.Here, comparable performance means no statistically significant difference in segmentation performance.
  • I. INTRODUCTION: The work extends an earlier conference paper with greater methodological detail, external MRI-to-CT validation, a CT-to-MRI TICV experiment, and public source code.The source code is available at the cited GitHub repository.

II. RELATED WORKS

The paper situates its approach within cross-modality medical image synthesis, emphasizing deep learning methods and the emergence of GAN- and CycleGAN-based synthesis from unpaired images.

  • A. Cross-modality Image Synthesis: Medical image synthesis generates realistic images through learned models, including cross-modality adaptation from a source image such as MRI to a target modality such as CT.The paper focuses primarily on cross-modality synthesis rather than generation from noise.
  • A. Cross-modality Image Synthesis: Cross-modality synthesis methods are grouped into registration-based, intensity-based, and deep learning-based approaches.Registration methods use aligned image collections, whereas intensity-based methods learn mappings between modality intensities.
  • A. Cross-modality Image Synthesis: Deep learning synthesis methods use intensity, spatial, contextual, or intermediate feature information to map between imaging modalities.Examples include location-sensitive, encoder-decoder, embedding, and context-aware adversarial networks.
  • A. Cross-modality Image Synthesis: GANs support image synthesis and domain adaptation with paired or unpaired training images, while cycle consistency produces more representative synthetic images than earlier adversarial methods.CycleGAN is presented as a tool for unpaired cross-modality synthesis.

B. Synthetic Segmentation

Synthetic segmentation uses generated target-modality images to train segmentation networks, progressing from independent two-stage pipelines toward end-to-end synthesis and segmentation.

  • B. Synthetic Segmentation: Synthetic segmentation uses synthesized target-modality images as training data for a segmentation network in the target modality.The paper distinguishes this use from adversarial supervision added directly to segmentation networks.
  • B. Synthetic Segmentation: Existing strategies include two-stage synthesis and segmentation, end-to-end domain adaptation, and CycleGAN followed by independent segmentation.Figure 2 compares these alternatives with the proposed integrated framework.
  • B. Synthetic Segmentation: Prior cross-modality domain adaptation used source and target sequences within MRI and overlapping modalities to support performance.The paper identifies cross-modality synthesis between independent mechanisms such as MRI and CT as an appealing direction.
  • B. Synthetic Segmentation: SynSeg-Net integrates cycle-adversarial synthesis and segmentation into an end-to-end framework without requiring target-modality manual labels.The related end-to-end work is extended here with more detailed descriptions, external validation, and new experiments.

III. METHOD

The method section describes preprocessing for MRI and CT inputs, including intensity normalization, clipping, and resampling before network processing.

  • III. METHOD: MRI intensities are normalized to 0–1 after excluding the highest and lowest 2.5% of intensities to reduce outlier effects.CT intensities are clipped to −1000 through 1000 HU and then normalized to 0–1.
  • III. METHOD: Normalized axial MRI and CT slices are resampled before being used by the network.The passage specifies resampling as the next preprocessing step after modality-specific normalization.

B. SynSeg-Net

SynSeg-Net combines cycle synthesis and segmentation in an end-to-end network for learning target-modality segmentations from unpaired images and source-modality labels.

  • SynSeg-Net uses two generators and two PatchGAN discriminators for bidirectional cross-modality image synthesis.G₁ maps source images to synthetic target images, while G₂ maps target images to synthetic source images.
  • Two forward training paths support learning from unpaired source and target modality images.
  • The segmentation subnet directly follows G₁ to integrate synthesis and segmentation training end to end.
  • Cycle-consistent losses minimize differences between true images and cycle-reconstructed images.
  • A weighted cross-entropy segmentation loss is combined with the other four losses using different weights.

C. Training and Testing

Training uses empirically fixed loss weights and randomly sampled two-dimensional slices, while testing applies preprocessing and uses only the segmentation network. The experiments include qualitative and quantitative evaluation of CT splenomegaly results.

  • λ₁=1, λ₂=1, λ₃=10, λ₄=10, and λ₅=1 were used in all experiments.The first four values followed the original CycleGAN settings, while λ₅ was assigned 1 without tuning.
  • During testing, SynSeg-Net used only Seg after normalizing target scans and resampling axial slices to 256 × 256.Final segmentation slices were resampled to the original resolution and concatenated.
  • Figure 5 qualitatively compares three CT-label-trained canonical methods with CycleGAN+Seg. and SynSeg-Net trained without CT manual labels.Splenomegaly CT labels were used only for validation in the latter comparison.
  • Table 1 reports DSC and ASD for CT splenomegaly testing images.
  • Training randomly sampled two-dimensional slices across all scans without restricting batches to consecutive slices or single subjects.

D. Evaluation Metrics

Segmentation approaches are evaluated against voxel-level ground truth using DSC, with between-method differences tested by the Wilcoxon signed rank test.

  • DSC compares segmentation results with ground truth voxel by voxel.
  • The Wilcoxon signed rank test evaluates differences between methods using a significance threshold of p<0.05.

IV. EXPERIMENTAL DESIGN AND RESULTS

The experiments evaluated MRI-to-CT splenomegaly synthetic segmentation using unpaired modality images and source-modality labels, comparing end-to-end SynSeg-Net with two-stage and supervised baselines. SynSeg-Net also underwent a half-cycle ablation and external validation.

  • MRI-to-CT Splenomegaly: MRI-to-CT splenomegaly segmentation used 60 MRI scans and 19 CT scans, with an independent CT cohort reserved for validation.The MRI and CT scans contributed 3262 and 1874 slices, respectively.
  • CT Segmentation without CT Manual Labels: The CT-label-free comparison evaluated two-stage CycleGAN+Seg. against end-to-end SynSeg-Net using matched network structures.SynSeg-Net integrated synthesis and segmentation into a single end-to-end training framework.
  • Ablation and Evaluation: SynSeg-Net-HC isolated the source-to-target path by using only G₁ and T in the half cycle, testing performance with and without the complete cycle.All networks were trained and evaluated across 100 epochs, with reporting based on selected validation performance.
  • Results: SynSeg-Net estimated spleen, liver, left-kidney, right-kidney, and stomach segmentations, while DSC and ASD measurements were reported.Only spleen performance was quantitatively evaluated because CT labels for the other organs were unavailable.
  • Results: Without CT labels, SynSeg-Net significantly outperformed CycleGAN+Seg. and SynSeg-Net-HC while achieving comparable performance to supervised ResNet using CT labels.The comparison was based on CT splenomegaly testing images.

B. External Validation for MRI-to-CT Splenomegaly Synthetic Segmentation

External validation tested pretrained MRI-to-CT synthetic segmentation directly on 66 independent CT scans acquired in axial, coronal, and sagittal views. SynSeg-Net significantly outperformed CycleGAN+Seg. and matched supervised ResNet performance for sagittal scans.

  • Data: The external validation cohort contained 66 independent CT scans: 23 axial, 21 coronal, and 22 sagittal acquisitions.The qualitative figure separates corresponding-view results from remaining-view results.
  • Experimental Design: The pretrained ResNet, CycleGAN+Seg., and SynSeg-Net models were applied directly without additional training or fine-tuning.The models and presented epochs were unchanged from the previous experiment.
  • Quantitative Results: Without CT labels, SynSeg-Net significantly outperformed CycleGAN+Seg. and achieved comparable performance to supervised ResNet for sagittal acquisition scans.The quantitative comparison used boxplots, with corresponding ASD and DSC measurements reported in Table 2.

C. CT-to-MRI TICV Synthetic Segmentation for Brain

The CT-to-MRI TICV experiment addressed synthetic segmentation in brain MRI, where the target modality provides richer tissue context than source CT. SynSeg-Net was trained with unpaired scans and CT labels only, outperforming CycleGAN+Seg. but underperforming supervised MRI ResNet.

  • Motivation: Brain CT-to-MRI synthetic segmentation was designed as a more challenging direction because brain MRI contains richer tissue context than brain CT.The experiment therefore synthesized the richer-context target modality from the less-context source modality.
  • Data: The study used 10 unpaired CT scans with TICV labels for training and 10 MRI scans whose labels were reserved for validation.The MRI labels were excluded from training.
  • Experimental Design: SynSeg-Net trained an MRI TICV segmentation network using unpaired CT and MRI images plus TICV labels only on CT.The validation compared supervised ResNet, CycleGAN+Seg., and SynSeg-Net.
  • Results: Without MRI TICV labels, SynSeg-Net significantly outperformed CycleGAN+Seg. but performed worse than supervised ResNet using MRI TICV labels.DSC and ASD measurements were reported for the MRI TICV testing images.

V. CONCLUSION AND DISCUSSION

SynSeg-Net demonstrates strong cross-modality synthetic segmentation performance while highlighting limitations in architecture choice, dimensionality, sampling, and synthesis evaluation.

  • SynSeg-Net outperformed CycleGAN+Seg. and matched labeled-target baselines in several evaluated scenarios.This included splenomegaly segmentation without CT labels, sagittal CT scans without target intensity images, and CT-to-MRI TICV segmentation.
  • The authors do not claim that the selected ResNet generators or PatchGAN discriminators are optimal.Alternative image-to-image generators, segmentation networks, or discriminators might yield better synthetic segmentation performance.
  • The study used 2D synthesis and segmentation because its training-image counts were insufficient for a reasonable 3D network.The framework could be extended to 3D when larger numbers of training scans are available.
  • Random slice sampling may bias training toward scans with more slices.The authors suggest balanced sampling strategies to reduce this imbalance.
  • The paper evaluates segmentation rather than synthesized-image quality, leaving quantitative synthesis assessment open.Using unpaired images without multimodal scans from the same patients also limited validation of synthesis quality.
Loading 1810.06498v2…