Source-linked AI summary
Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network
Zizhao Zhang, Lin Yang, Yefeng Zheng
TL;DR
Medical volume synthesis must work with unpaired data, preserve anatomy, and support segmentation when labeled data are limited. The paper couples 3D generators and segmentors with cycle- and shape-consistency, and reports better performance than exclusive or offline alternatives. Its scope includes difficult 3D end-to-end training and a remaining gap between synthetic and real data as real-data volume increases.
Problem
Cross-modality medical synthesis needs realistic 3D images from unpaired data, anatomical consistency, and useful synthetic data for modalities with limited training samples.
Method
The method jointly trains mutually beneficial 3D generators and segmentors using adversarial, cycle-consistency, and segmentor-supervised shape-consistency losses with online synthetic-data augmentation.
Results
Coupling translation and segmentation performs better than solving them exclusively, while online training consistently improves over isolated offline synthetic-data augmentation on 4,496 cardiovascular CT and MRI volumes.
Takeaways & Limitations
A unified end-to-end approach can address unpaired 3D translation, anatomical shape preservation, and segmentation improvement together.
Takeaways & Limitations
The gap between synthetic and real data becomes harder to reduce as the amount of real data increases, and end-to-end 3D training is more difficult.
Abstract
from arXiv · showhide
Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide X-ray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. With extensive experiments on a dataset including a total of 4,496 CT and magnetic resonance imaging (MRI) cardiovascular volumes, we show both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.
1. Introduction
The paper targets cross-modality transfer for multimodal medical imaging, where paired data are often unavailable and anatomical structure can be distorted. It jointly learns translation and segmentation with mutually beneficial 3D CNNs, using synthetic data online to improve segmentation.
- Multiple imaging modalities support diagnosis and surgical planning, motivating cross-modality image transfer systems for clinical treatment.
- Medical image synthesis can supplement limited training data, but robust models require representative images that are difficult to collect for newer modalities.
- Paired pixel-to-pixel correspondence is required by direct cross-modality reconstruction methods, limiting their applicability to unpaired multimodal data.
- Unpaired GAN translation must preserve diagnostic anatomical structures because discriminator supervision alone does not guarantee shape invariance.
- The proposed end-to-end system jointly learns cross-domain volume generators and segmentors, combining adversarial, cycle-consistency, and shape-consistency supervision.Segmentors supervise shape consistency, while generators provide synthetic data to segmentors online.
- 4,496 cardiovascular CT and MRI volumes support experiments showing that online coupled training consistently improves over isolated offline synthetic-data augmentation.Offline augmentation can underperform and may distract segmentation optimization when real data are scarce.
2. Related work
Prior work uses image translation and synthetic data to address medical-image analysis, but commonly assumes paired domains or focuses on augmentation. Adversarial supervision for segmentation refines predictions without directly resolving data insufficiency.
- Image-to-image translation is commonly formulated as pixel-to-pixel encoder–decoder mapping, with medical applications using sparse coding, GANs, and CNNs.
- Many medical cross-modality translation methods presume paired cross-domain data, leaving unpaired translation comparatively underexplored.
- Synthetic medical data are studied as a way to address insufficient labels, including unsupervised modality translation and augmentation for lesion segmentation.
- Adversarial learning can provide extra supervision for segmentation or detection by constraining predictions toward ground-truth distributions.The cited strategy is described as refinement and therefore less likely to remedy data insufficiency.
3. Proposed Method
The proposed method jointly translates unpaired 3D volumes across modalities and segments them using mutually beneficial generators and segmentors. Cycle-, adversarial-, and shape-consistency objectives address unpaired translation and anatomical preservation, while online synthetic data strengthens segmentation.
- Problems in unpaired translation: Cycle-consistency alone permits geometric transformations that can distort anatomical structures without increasing data-fidelity penalties.The ambiguity includes translation, rotation, scaling, and nonrigid transformations that can be reversed during the cycle.
- Cycle-consistency: Cycle-consistency constrains reconstructed volumes to match their inputs when paired cross-domain data are unavailable.The loss uses voxel-wise L1 reconstruction for reconstructed synthetic samples.
- Shape-consistency: Shape-consistency uses segmentors mapping both modalities into a shared semantic label space to reduce geometric distortion during translation.The shape loss is a multi-class cross-entropy loss supervised by ground-truth shape representations.
- The method jointly learns cross-domain volume translation and segmentation from unpaired data using coupled 3D CNN generators and segmentors.Generators translate between domains, while segmentors use real and generator-produced synthetic volumes during training.
- Segmentation: Online synthetic volumes augment real segmentation data adaptively, unlike straightforward offline fusion of synthetic and real datasets.The authors state that generators and segmentors are mutually beneficial through the joint objective.
- Objective: The full objective combines adversarial, cycle-consistency, and shape-consistency losses and alternately updates generators, discriminators, and segmentors.The stated training alternates optimization of generators with fixed segmentors and discriminators, then optimization of segmentors and discriminators with fixed generators.
4. Network Architecture and Details
The architecture directly processes holistic 3D volumes with fully convolutional generators, discriminators, and segmentors. Its design uses skip connections and restrained downsampling, while training applies staged or from-scratch optimization with early stopping to avoid artifacts and overfitting.
- Architecture: The method directly processes holistic 3D volumes rather than using 2.5D representations or subvolumes.The authors identify optimization and memory as challenges for end-to-end 3D training.
- Generators: Long-range U-Net skip connections combine bottom- and top-layer representations to preserve anatomy and improve convergence.The architecture applies three stride-2 downsampling operations, for a maximum downsampling rate of 8, with symmetric upsampling.
- Translation quality: The generators achieve significantly better visual quality than 3D CycleGAN in qualitative comparisons.The comparison reports that CycleGAN outputs contain severe artifacts, checkerboard effects, and missing anatomies, whereas the proposed outputs overcome these issues.
- Discriminators and segmentors: PatchGAN discriminators classify overlapping sub-volumes rather than whole volumes, limiting decisions based on arbitrary volume locations.The segmentors use symmetric three-stage downsampling and upsampling without normalization layers.
- Training details: Training can begin with separately pretrained generators and discriminators before joint fine-tuning, although training all networks from scratch produced similar results.The reported segmentor learning rate is 2e−4.
- Training details: Early stopping is applied when segmentation loss stops decreasing for about 5 epochs because small learning rates produce artifacts and segmentor overfitting.The desired training point usually takes about 40 epochs to reach.
5. Experimental Results
Experiments on unpaired cardiac CT and MRI volumes evaluate translation, shape preservation, and segmentation enhancement. The coupled online approach preserves anatomy and improves segmentation over offline synthetic-data augmentation, while its advantage narrows as real-data availability increases.
- Dataset: The dataset contains 4,354 cardiac CT scans and 142 cardiac MRI scans, organized into paired-volume-count S1 and larger-CT S2 experiments.S1 matches CT and MRI counts; S2 adds 4,212 CT volumes for synthetic MRI augmentation.
- Cross-domain Translation Evaluation: Synthetic volumes were visually close to real images, preserved cardiac anatomy such as the aorta and spine, and showed no obvious geometric distortion.Shape invariance was also evaluated using the segmentation-based S-score, where higher values indicate better shape matching.
- Segmentation Evaluation: The method jointly fine-tunes generators, segmentors, and discriminators end-to-end, contrasting with ad-hoc offline fine-tuning that adds generated and reconstructed data to real data.Figure 5 contrasts offline augmentation with the proposed online use of generator outputs.
- Segmentation Evaluation: With 14% real data and all counter-modality synthetic data, the proposed method achieves much better segmentation on both modalities, whereas ADA deteriorates CT segmentation.The authors associate ADA’s CT degradation with optimization distraction when synthetic data are added offline to an unstable low-data baseline.
- Gap between synthetic and real data: The benefit of extra synthetic data grows more slowly as real-data volume increases, and excessive offline synthetic data can diverge network training.The paper attributes the slower gains partly to segmentor capacity and identifies further gap reduction as requiring study.
- Segmentation Evaluation: On MRI segmentation, the proposed method reaches ADA’s accuracy using 23% synthetic data, compared with ADA’s 100% synthetic-data setting.This experiment uses all real MRI data while varying the amount of synthetic data.
- Gap between synthetic and real data: The method reduces the ADA real–synthetic performance gap by 61% with 14% real data and by 20.9% with 85% real data.The gap is evaluated through the ability of real or synthetic data to improve segmentation.
6. Conclusion
The unified method simultaneously translates and segments unpaired 3D medical images while addressing anatomy consistency and limited segmentation data. Coupling these tasks is more effective than solving them exclusively.
- The method simultaneously learns to translate and segment medical 3D images.
- It addresses unpaired data, anatomy consistency, and effective use of synthetic data for volume segmentation.
- Coupling translation and segmentation is more effective than solving the two tasks exclusively.