Source-linked AI summary
Generative Adversarial Networks for Image-to-Image Translation on Multi-Contrast MR Images - A Comparison of CycleGAN and UNIT
Per Welander, Simon Karlsson, Anders Eklund
TL;DR
The best GAN model and how to evaluate and compare models for MR image-to-image translation remain unclear. This paper compares CycleGAN, UNIT, and supervised models on T1- and T2-weighted MR images; the models can produce visually realistic images, but quantitative performance does not necessarily track visual realness.
Problem
The best GAN model and how to evaluate and compare models for MR image-to-image translation remain unclear.
Source: §1 · p. 1Method
The study compares unsupervised CycleGAN and UNIT with supervised models for T1- and T2-weighted MR image translation, using quantitative evaluation and a perceptual study.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 2Results
CycleGAN and UNIT can generate visually realistic MR images, while quantitative superiority does not necessarily correspond to visual realness.
Source: §3 · p. 2, §4.3 · p. 3Takeaways & Limitations
For generating visually realistic images, unsupervised CycleGAN is a better alternative, and the models perform differently across domains.
Source: §4.3 · p. 3The used metrics alone cannot determine whether an image is visually realistic.
Source: §4.2 · p. 3
Abstract
from arXiv · showhide
In medical imaging, a general problem is that it is costly and time consuming to collect high quality data from healthy and diseased subjects. Generative adversarial networks (GANs) is a deep learning method that has been developed for synthesizing data. GANs can thereby be used to generate more realistic training data, to improve classification performance of machine learning algorithms. Another application of GANs is image-to-image translations, e.g. generating magnetic resonance (MR) images from computed tomography (CT) images, which can be used to obtain multimodal datasets from a single modality. Here, we evaluate two unsupervised GAN models (CycleGAN and UNIT) for image-to-image translation of T1- and T2-weighted MR images, by comparing generated synthetic MR images to ground truth images. We also evaluate two supervised models; a modification of CycleGAN and a pure generator model. A small perceptual study was also performed to evaluate how visually realistic the synthesized images are. It is shown that the implemented GAN models can synthesize visually realistic MR images (incorrectly labeled as real by a human). It is also shown that models producing more visually realistic synthetic images not necessarily have better quantitative error measurements, when compared to ground truth data. Code is available at https://github.com/simontomaskarlsson/GAN-MRI
1. INTRODUCTION
Collecting large annotated medical-imaging datasets is difficult, and conventional augmentation offers limited alternatives. The paper compares CycleGAN and UNIT for T1/T2 MR translation amid uncertainty about how to choose and evaluate image-translation GANs.
1. INTRODUCTION
- Large annotated medical-imaging datasets are challenging to acquire because classification examples correspond to individual subjects.Conventional rotation, cropping, and scaling offer only limited alternative data.Source: §1 · p. 1
- GANs can synthesize training data or translate between imaging modalities, including CT and MR, for potential multimodal classification.Combining image types may improve sensitivity in healthy-versus-diseased classification.Source: §1 · p. 1
- Because GAN models for image translation are still developing, the authors compare CycleGAN and UNIT for translating between T1- and T2-weighted MR images.Earlier work used conditional GANs; this comparison reports the authors’ implementations of CycleGAN and UNIT.Source: §1 · p. 1
2. METHOD
The study implements unpaired-capable CycleGAN and UNIT alongside supervised variants, then evaluates them on paired T1/T2 images using quantitative, visual-error, and perceptual assessments.
2.1. GAN model selection and implementation
- CycleGAN and UNIT were selected because they synthesize realistic high-resolution images and can train with unpaired data.The study also implements CycleGAN s with supervised MAE loss, Generators s with MAE alone, and a two-layer Simple baseline.Source: §2.1 · p. 2
2.2. Evaluation
- The evaluation uses paired T1- and T2-weighted images from 1,113 subjects, with 900 images per domain for training and 213 for testing.Images are registered to a common template; the evaluation uses axial slice 120 from segmented brains.Source: §2.2 · p. 2
- Quantitative evaluation uses MAE, mutual information, and PSNR on test images normalized by subtracting the mean and dividing by the standard deviation.The visual error is the relative absolute difference: the absolute image difference divided by the real image.Source: §2.2 · p. 2
- One author’s perceptual study asks whether 96 real and 72 synthetic T1/T2 images are real or synthetic; Generators s and Simple are excluded as obviously synthetic.The authors describe the study as an indicator of visual quality because anatomical image evaluation is best performed by a radiologist.Source: §2.2 · p. 2
3. RESULTS
Generators s leads the quantitative measurements, but that advantage does not indicate visual realism: adversarially trained GANs look more realistic. Error patterns also differ between synthetic T1 and T2 images.
3. RESULTS
- Generators s outperforms the other models on all quantitative measurements, while CycleGAN, CycleGAN s, and UNIT perform similarly; perceptually, UNIT leads for T1 and CycleGAN for T2.Quantitative performance is generally better for T1 images, with a few exceptions.Source: §3 · p. 2
- Despite Generators s’ quantitative superiority, its supervised outputs and those of Simple look unrealistically smooth, whereas adversarially trained GANs produce more realistic images.The Simple model also fails to map cerebrospinal-fluid color correctly.Source: §3 · p. 2
- Relative absolute error is greater for synthetic T2 than T1 images; T1 errors concentrate at the edges, while T2 errors appear across the brain.The authors relate edge errors to unique brain shapes and note difficulty learning intensity transitions such as cerebrospinal fluid and white matter.Source: §3 · pp. 2–3
4. DISCUSSION
The discussion finds that visual realism and quantitative similarity do not align consistently across models. It favors unsupervised CycleGAN when visual realism is the goal and identifies data augmentation and 3D GANs as future directions.
4.1. Quantitative comparison
- Generators s outperformed the other models quantitatively, while Simple performed worst on all measurements except MI for T2 images.CycleGAN s showed a slight T2 MAE improvement over CycleGAN, but its results were not significantly better; CycleGAN and UNIT were similar.Source: §4.1 · p. 3
- T1 images had higher MI than T2 images; Simple's high T2 MI was attributed to preserving information while changing pixel intensity.The authors relate the higher T2 MAE to the lower T2 MI.Source: §4.1 · p. 3
4.2. Qualitative comparison
- Perceptual testing found synthetic images visually realistic, though T2 was harder to classify; edge errors reflect individual brain shapes, bright T2 CSF, and difficult CSF/white-matter transitions.The authors suggest T2 darkness may also make noise realism harder to judge.Source: §4.2 · p. 3
- MAE supervision pushes generators toward smooth images, whereas adversarial loss supports visual realism; quantitative scores alone cannot establish perceptual realism.CycleGAN s may be preferable when similarity to ground-truth images is the aim.Source: §4.2 · p. 3
4.3. Future work
- CycleGAN and UNIT can generate visually realistic MR images, and unsupervised CycleGAN is favored when visual realism is the goal.The authors also note that models performed differently across domains.Source: §4.3 · p. 3
- Future work could test GAN-based data augmentation and 3D GANs, which may improve results at the cost of longer processing and greater memory use.Data augmentation could be evaluated for distinguishing healthy and diseased subjects.Source: §4.3 · p. 3