PaperlayerView exact revision on arXiv

Source-linked AI summary

SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth

Yuankai Huo, Zhoubing Xu, Hyeonsoo Moon, Shunxing Bao, Albert Assad, Tamara K. Moyo, Michael R. Savona, Richard G. Abramson, Bennett A. Landman

arXiv:1810.06498v2cs.CV

TL;DR

Limited generalization and the need for target-modality manual labels motivate SynSeg-Net. The paper trains an end-to-end synthesis-and-segmentation network from unpaired source and target images with source-only labels, achieving better performance than two-stage methods and comparable performance to target-label training in certain scenarios.

  • Problem

    DCNN segmentation methods have limited generalizability, and applying them to new modalities or disease cohorts typically requires manually traced target-modality training images.

    Source: §V · p. 8

  • Method

    SynSeg-Net jointly uses CycleGAN-based cross-modality synthesis and DCNN segmentation, trained with unpaired source and target images and manual labels only from the source modality.

    Source: §I · p. 1, B. SynSeg-Net · p. 3

  • Results

    Across MRI-to-CT splenomegaly and CT-to-MRI TICV experiments, SynSeg-Net outperformed two-stage methods and achieved comparable performance to target-label training in certain scenarios.

    Source: §I · p. 2, CT Segmentation without CT Manual Labels · p. 7, §V · p. 8

  • Takeaways & Limitations

    The results support synthetic segmentation as a way to train target-modality segmentation networks without target-modality ground-truth labels.

    Source: §I · pp. 1–2

    The study evaluates segmentation rather than synthesized-image quality, and its use of unpaired images without same-patient multimodal data limits synthesis validation.

    Source: §V · p. 9

Abstract

from arXiv · show

A key limitation of deep convolutional neural networks (DCNN) based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality without having manual labels. SynSeg-Net is trained by using (1) unpaired intensity images from source and target modalities, and (2) manual labels only from source modality. SynSeg-Net is enabled by the recent advances of cycle generative adversarial networks (CycleGAN) and DCNN. We evaluate the performance of the SynSeg-Net on two experiments: (1) MRI to CT splenomegaly synthetic segmentation for abdominal images, and (2) CT to MRI total intracranial volume synthetic segmentation (TICV) for brain images. The proposed end-to-end approach achieved superior performance to two stage methods. Moreover, the SynSeg-Net achieved comparable performance to the traditional segmentation network using target modality labels in certain scenarios. The source code of SynSeg-Net is publicly available (https://github.com/MASILab/SynSeg-Net).

I. INTRODUCTION

SynSeg-Net addresses limited segmentation generalizability by training target-modality networks from unpaired source and target images with manual labels only in the source modality. It is evaluated on MRI-to-CT splenomegaly and CT-to-MRI TICV segmentation, outperforming two-stage methods and matching target-label training in certain scenarios.

I. INTRODUCTION

  • CycleGAN advances enable high-quality cross-modality synthesis from unpaired training images.Earlier synthesis methods typically required paired training data across modalities.Source: §I · p. 1
  • SynSeg-Net trains a target-modality DCNN using unpaired source and target images plus manual segmentations only from the source modality.The approach uses cross-modality synthesis learning to reduce manual segmentation effort.Source: §I · p. 1
  • The experiments cover MRI-to-CT splenomegaly segmentation and CT-to-MRI total intracranial volume segmentation.The first experiment uses CT splenomegaly without spleen labels on CT images.Source: §I · p. 2
  • The proposed end-to-end approach outperformed two-stage methods and achieved comparable performance to target-label training in certain scenarios.Here, comparable performance means no statistically significant difference in segmentation performance.Source: §I · p. 2
  • The work extends an earlier conference paper with greater methodological detail, external MRI-to-CT validation, a CT-to-MRI TICV experiment, and public source code.The source code is available at the cited GitHub repository.Source: §I · p. 2

II. RELATED WORKS

The paper situates its approach within cross-modality medical image synthesis, emphasizing deep learning methods and the emergence of GAN- and CycleGAN-based synthesis from unpaired images.

A. Cross-modality Image Synthesis

  • Medical image synthesis generates realistic images through learned models, including cross-modality adaptation from a source image such as MRI to a target modality such as CT.The paper focuses primarily on cross-modality synthesis rather than generation from noise.Source: A. Cross-modality Image Synthesis · p. 2
  • Cross-modality synthesis methods are grouped into registration-based, intensity-based, and deep learning-based approaches.Registration methods use aligned image collections, whereas intensity-based methods learn mappings between modality intensities.Source: A. Cross-modality Image Synthesis · p. 2
  • Deep learning synthesis methods use intensity, spatial, contextual, or intermediate feature information to map between imaging modalities.Examples include location-sensitive, encoder-decoder, embedding, and context-aware adversarial networks.Source: A. Cross-modality Image Synthesis · p. 2
  • GANs support image synthesis and domain adaptation with paired or unpaired training images, while cycle consistency produces more representative synthetic images than earlier adversarial methods.CycleGAN is presented as a tool for unpaired cross-modality synthesis.Source: A. Cross-modality Image Synthesis · p. 2

B. Synthetic Segmentation

Synthetic segmentation uses generated target-modality images to train segmentation networks, progressing from independent two-stage pipelines toward end-to-end synthesis and segmentation.

B. Synthetic Segmentation

  • Synthetic segmentation uses synthesized target-modality images as training data for a segmentation network in the target modality.The paper distinguishes this use from adversarial supervision added directly to segmentation networks.Source: B. Synthetic Segmentation · p. 2
  • Existing strategies include two-stage synthesis and segmentation, end-to-end domain adaptation, and CycleGAN followed by independent segmentation.Figure 2 compares these alternatives with the proposed integrated framework.Source: B. Synthetic Segmentation · pp. 2–3
  • Prior cross-modality domain adaptation used source and target sequences within MRI and overlapping modalities to support performance.The paper identifies cross-modality synthesis between independent mechanisms such as MRI and CT as an appealing direction.Source: B. Synthetic Segmentation · p. 3
  • SynSeg-Net integrates cycle-adversarial synthesis and segmentation into an end-to-end framework without requiring target-modality manual labels.The related end-to-end work is extended here with more detailed descriptions, external validation, and new experiments.Source: B. Synthetic Segmentation · p. 3

III. METHOD

The method section describes preprocessing for MRI and CT inputs, including intensity normalization, clipping, and resampling before network processing.

III. METHOD

  • MRI intensities are normalized to 0–1 after excluding the highest and lowest 2.5% of intensities to reduce outlier effects.CT intensities are clipped to −1000 through 1000 HU and then normalized to 0–1.Source: A. Preprocessing · p. 3
  • Normalized axial MRI and CT slices are resampled before being used by the network.The passage specifies resampling as the next preprocessing step after modality-specific normalization.Source: A. Preprocessing · p. 3

B. SynSeg-Net

SynSeg-Net combines cycle synthesis and segmentation in an end-to-end network for learning target-modality segmentations from unpaired images and source-modality labels.

  • SynSeg-Net uses two generators and two PatchGAN discriminators for bidirectional cross-modality image synthesis.G₁ maps source images to synthetic target images, while G₂ maps target images to synthetic source images.Source: B. SynSeg-Net · p. 3
  • Two forward training paths support learning from unpaired source and target modality images.Source: B. SynSeg-Net · p. 3
  • The segmentation subnet directly follows G₁ to integrate synthesis and segmentation training end to end.Source: B. SynSeg-Net · p. 3
  • Cycle-consistent losses minimize differences between true images and cycle-reconstructed images.Source: B. SynSeg-Net · p. 4
  • A weighted cross-entropy segmentation loss is combined with the other four losses using different weights.Source: B. SynSeg-Net · p. 4

C. Training and Testing

Training uses empirically fixed loss weights and randomly sampled two-dimensional slices, while testing applies preprocessing and uses only the segmentation network. The experiments include qualitative and quantitative evaluation of CT splenomegaly results.

  • λ₁=1, λ₂=1, λ₃=10, λ₄=10, and λ₅=1 were used in all experiments.The first four values followed the original CycleGAN settings, while λ₅ was assigned 1 without tuning.Source: C. Training and Testing · p. 4
  • During testing, SynSeg-Net used only Seg after normalizing target scans and resampling axial slices to 256 × 256.Final segmentation slices were resampled to the original resolution and concatenated.Source: C. Training and Testing · pp. 4–5
  • Figure 5 qualitatively compares three CT-label-trained canonical methods with CycleGAN+Seg. and SynSeg-Net trained without CT manual labels.Splenomegaly CT labels were used only for validation in the latter comparison.Source: Figure 5 · p. 4
  • Table 1 reports DSC and ASD for CT splenomegaly testing images.Source: Table 1 · p. 4
  • Training randomly sampled two-dimensional slices across all scans without restricting batches to consecutive slices or single subjects.Source: C. Training and Testing · p. 5

D. Evaluation Metrics

Segmentation approaches are evaluated against voxel-level ground truth using DSC, with between-method differences tested by the Wilcoxon signed rank test.

IV. EXPERIMENTAL DESIGN AND RESULTS

The experiments evaluated MRI-to-CT splenomegaly synthetic segmentation using unpaired modality images and source-modality labels, comparing end-to-end SynSeg-Net with two-stage and supervised baselines. SynSeg-Net also underwent a half-cycle ablation and external validation.

MRI-to-CT Splenomegaly

  • MRI-to-CT splenomegaly segmentation used 60 MRI scans and 19 CT scans, with an independent CT cohort reserved for validation.The MRI and CT scans contributed 3262 and 1874 slices, respectively.Source: §IV · p. 5

CT Segmentation without CT Manual Labels

  • The CT-label-free comparison evaluated two-stage CycleGAN+Seg. against end-to-end SynSeg-Net using matched network structures.SynSeg-Net integrated synthesis and segmentation into a single end-to-end training framework.Source: CT Segmentation without CT Manual Labels · p. 6

Ablation and Evaluation

  • SynSeg-Net-HC isolated the source-to-target path by using only G₁ and T in the half cycle, testing performance with and without the complete cycle.All networks were trained and evaluated across 100 epochs, with reporting based on selected validation performance.Source: CT Segmentation without CT Manual Labels · p. 6

Results

  • SynSeg-Net estimated spleen, liver, left-kidney, right-kidney, and stomach segmentations, while DSC and ASD measurements were reported.Only spleen performance was quantitatively evaluated because CT labels for the other organs were unavailable.Source: CT Segmentation without CT Manual Labels · pp. 6–7
  • Without CT labels, SynSeg-Net significantly outperformed CycleGAN+Seg. and SynSeg-Net-HC while achieving comparable performance to supervised ResNet using CT labels.The comparison was based on CT splenomegaly testing images.Source: Figure 8 · p. 6, CT Segmentation without CT Manual Labels · p. 7

B. External Validation for MRI-to-CT Splenomegaly Synthetic Segmentation

External validation tested pretrained MRI-to-CT synthetic segmentation directly on 66 independent CT scans acquired in axial, coronal, and sagittal views. SynSeg-Net significantly outperformed CycleGAN+Seg. and matched supervised ResNet performance for sagittal scans.

Data

  • The external validation cohort contained 66 independent CT scans: 23 axial, 21 coronal, and 22 sagittal acquisitions.The qualitative figure separates corresponding-view results from remaining-view results.Source: B. External Validation for MRI-to-CT… · p. 7

Experimental Design

  • The pretrained ResNet, CycleGAN+Seg., and SynSeg-Net models were applied directly without additional training or fine-tuning.The models and presented epochs were unchanged from the previous experiment.Source: B. External Validation for MRI-to-CT… · p. 7

Quantitative Results

  • Without CT labels, SynSeg-Net significantly outperformed CycleGAN+Seg. and achieved comparable performance to supervised ResNet for sagittal acquisition scans.The quantitative comparison used boxplots, with corresponding ASD and DSC measurements reported in Table 2.Source: B. External Validation for MRI-to-CT… · p. 7

C. CT-to-MRI TICV Synthetic Segmentation for Brain

The CT-to-MRI TICV experiment addressed synthetic segmentation in brain MRI, where the target modality provides richer tissue context than source CT. SynSeg-Net was trained with unpaired scans and CT labels only, outperforming CycleGAN+Seg. but underperforming supervised MRI ResNet.

Motivation

  • Brain CT-to-MRI synthetic segmentation was designed as a more challenging direction because brain MRI contains richer tissue context than brain CT.The experiment therefore synthesized the richer-context target modality from the less-context source modality.Source: C. CT-to-MRI TICV Synthetic… · p. 7

Data

  • The study used 10 unpaired CT scans with TICV labels for training and 10 MRI scans whose labels were reserved for validation.The MRI labels were excluded from training.Source: C. CT-to-MRI TICV Synthetic… · pp. 7–8

Experimental Design

  • SynSeg-Net trained an MRI TICV segmentation network using unpaired CT and MRI images plus TICV labels only on CT.The validation compared supervised ResNet, CycleGAN+Seg., and SynSeg-Net.Source: C. CT-to-MRI TICV Synthetic… · p. 8

Results

V. CONCLUSION AND DISCUSSION

SynSeg-Net demonstrates strong cross-modality synthetic segmentation performance while highlighting limitations in architecture choice, dimensionality, sampling, and synthesis evaluation.

  • SynSeg-Net outperformed CycleGAN+Seg. and matched labeled-target baselines in several evaluated scenarios.This included splenomegaly segmentation without CT labels, sagittal CT scans without target intensity images, and CT-to-MRI TICV segmentation.Source: §V · p. 8
  • The authors do not claim that the selected ResNet generators or PatchGAN discriminators are optimal.Alternative image-to-image generators, segmentation networks, or discriminators might yield better synthetic segmentation performance.Source: §V · pp. 8–9
  • The study used 2D synthesis and segmentation because its training-image counts were insufficient for a reasonable 3D network.The framework could be extended to 3D when larger numbers of training scans are available.Source: §V · p. 9
  • Random slice sampling may bias training toward scans with more slices.The authors suggest balanced sampling strategies to reduce this imbalance.Source: §V · p. 9
  • The paper evaluates segmentation rather than synthesized-image quality, leaving quantitative synthesis assessment open.Using unpaired images without multimodal scans from the same patients also limited validation of synthesis quality.Source: §V · p. 9

Related papers

Loading 1810.06498v2…