Source-linked AI summary
MedGAN: Medical Image Translation using GANs
Karim Armanious, Chenming Jiang, Marc Fischer, Thomas Küstner, Konstantin Nikolaou, Sergios Gatidis, Bin Yang
TL;DR
Medical image translation methods often rely on task-specific architectures or non-end-to-end refinement, limiting generality across medical applications. MedGAN addresses this with an end-to-end framework combining adversarial and non-adversarial losses with the CasNet generator, and applies it across three tasks, where qualitative and quantitative evaluations report superior performance to existing approaches.
Problem
Existing medical image translation approaches are often task-specific or require non-end-to-end refinement, limiting their reuse across imaging tasks.
Method
MedGAN combines adversarial and non-adversarial losses with CasNet, a progressively refining encoder-decoder generator, in an end-to-end translation framework.
Results
MedGAN was applied without application-specific modifications to PET-CT translation, PET denoising, and MR motion-artefact correction, outperforming existing translation approaches qualitatively and quantitatively.
Takeaways & Limitations
MedGAN is presented as a framework for enhancing technical post-processing tasks that require globally consistent image properties rather than direct diagnosis.
Takeaways & Limitations
The study operates on 2D slices, although volumetric 3D information is essential for most medical tasks and future adaptation is needed.
Abstract
from arXiv · showhide
Image-to-image translation is considered a new frontier in the field of medical image analysis, with numerous potential applications. However, a large portion of recent approaches offers individualized solutions based on specialized task-specific architectures or require refinement through non-end-to-end training. In this paper, we propose a new framework, named MedGAN, for medical image-to-image translation which operates on the image level in an end-to-end manner. MedGAN builds upon recent advances in the field of generative adversarial networks (GANs) by merging the adversarial framework with a new combination of non-adversarial losses. We utilize a discriminator network as a trainable feature extractor which penalizes the discrepancy between the translated medical images and the desired modalities. Moreover, style-transfer losses are utilized to match the textures and fine-structures of the desired target images to the translated images. Additionally, we present a new generator architecture, titled CasNet, which enhances the sharpness of the translated medical outputs through progressive refinement via encoder-decoder pairs. Without any application-specific modifications, we apply MedGAN on three different tasks: PET-CT translation, correction of MR motion artefacts and PET image denoising. Perceptual analysis by radiologists and quantitative evaluations illustrate that the MedGAN outperforms other existing translation approaches.
1 Introduction
Medical image translation can reduce redundant acquisitions and support technical post-processing, but synthetic images must avoid unrealistic information. MedGAN addresses application-specific and limited-capacity prior approaches with an end-to-end framework designed for multiple medical imaging tasks.
- Multiple modalities often provide supplementary information, making additional acquisitions necessary for complete diagnosis.
- Generating image information from already acquired data could shorten diagnostic procedures by making some additional scans unnecessary.
- Synthetic translations may introduce unrealistic information, limiting their reliability for diagnosis; some technical applications instead require globally consistent contrast properties.
- Existing medical translation approaches are often application-specific or have limited modelling capacity, hindering reuse across tasks.
- MedGAN combines adversarial and non-adversarial losses in an end-to-end framework intended for different medical tasks without application-specific modifications.
- CasNet chains fully convolutional encoder-decoder networks with skip connections to progressively refine outputs toward high-resolution, crisp images.
2 Materials and methods
MedGAN combines a CasNet generator, an adversarial discriminator, and a pre-trained feature extractor for medical image translation. Its generator progressively refines translated images through encoder-decoder blocks while the discriminator also supplies perceptual features.
- MedGAN comprises a CasNet generator, discriminator, and pre-trained feature extractor for medical image-to-image translation.
- The generator maps source-domain images to target-domain images through progressive refinement via encoder-decoder blocks.
- The discriminator distinguishes real from transformed images and co-serves as a trainable feature extractor for modified perceptual loss.
D LGAN (1)
Conditional image translation combines source images with adversarial supervision and reconstruction objectives. MedGAN further uses discriminator features, stabilization, and perceptual design choices to address structural inconsistency, blur, and detail loss.
- Conditional GAN translation maps a source image y to a corresponding target estimate G(y, z) = ˆx while conditioning adversarial discrimination on y.
- Pixel reconstruction losses are added because adversarial-only translation can produce outputs lacking consistent global structure.
- Pixel losses can preserve global structure but produce blurry or detail-distorted outputs and poorly reflect perceptual similarity.
- MedGAN uses discriminator hidden-layer feature maps to compute a perceptual loss that captures discrepancies in high-frequency components.
- The perceptual design retains pixel reconstruction because pixel-space discrepancy positively affects output quality.
- Spectral normalization stabilizes discriminator training by constraining its Lipschitz constant to 1.
2.3 Style transfer losses
MedGAN combines style-transfer and content losses to preserve target-image textures, fine structures, and global consistency during medical image translation.
- Style-transfer losses: Style-transfer losses match translated images to target images through texture and detail representations extracted from hidden CNN layers.The approach uses feature extraction to compare representations beyond raw pixels.
- Style loss: The style loss penalizes discrepancies between feature correlations of translated outputs and corresponding target images.These correlations capture style distributions across spatial feature representations.
- Generator architecture: CasNet progressively refines outputs through concatenated encoder-decoder pairs, supporting sharper translated medical images.The architecture is presented as part of the framework for producing detailed outputs.
- Content loss: Content loss penalizes differences between hidden feature representations to enhance low-frequency components and maintain global consistency.Unlike style loss, it does not target texture or style discrepancies.
2.4 MedGAN architecture
MedGAN uses a modality-agnostic encoder-decoder design and introduces CasNet, which chains U-blocks for end-to-end progressive refinement while preserving low-level information.
- U-block design: MedGAN uses encoder-decoder U-blocks as a fundamental building block to support translation across input modalities without application-specific modifications.The image-to-image task maps an input tensor to an output tensor with different appearance but the same underlying structure.
- U-block design: U-blocks are fully convolutional encoder-decoder networks adapted from U-net and pix2pix-style architectures for adversarial image translation.Their encoding path maps 256×256 images into high-level representations through convolutional layers.
- U-block design: Skip connections concatenate spatial information between mirrored encoder and decoder layers, passing low-level details that would otherwise be lost at the bottleneck.These connections are described as fundamental for preserving output quality.
- CasNet: CasNet concatenates several U-blocks end-to-end so outputs are progressively refined through successive encoder-decoder pairs.Skip connections within U-blocks mitigate vanishing gradients caused by the increased network depth.
- CasNet: CasNet uses deeper U-blocks and intermediate skip connections, whereas ResNets use shallower residual blocks and identity mappings.The cited comparison distinguishes their depth and information-flow mechanisms.
- Discriminator: A modified PatchGAN discriminator classifies local image patches, restricting attention to high-frequency regions and encouraging detailed generator outputs.The conventional patch size is described as 70 × 70.
2.5 MedGAN framework and training
MedGAN trains a CasNet generator with adversarial, perceptual, style, and content objectives in a paired-data min-max framework, using a cumulative weighted loss.
- Framework: MedGAN combines CasNet with perceptual and pixel-level adversarial penalties, while style-transfer losses target the target image’s style, texture, and content.The framework is trained using a cumulative loss function.
- Training: The training pipeline requires paired input-target data, 200 epochs, three generator iterations, and a pretrained VGG-19 feature extractor.The listed hyperparameters include λ1 = 20 and λ2 = λ3 = 0.0001.
- Training: Training updates discriminator and generator parameters through iterative optimization, with spectral normalization applied to discriminator weights.The algorithm specifies repeated generator and discriminator updates within each epoch.
- Implementation: Using six U-blocks, training averaged 36 hours and inference took 115 milliseconds per test image on a single Nvidia Titan-X GPU.These timings are reported for the described implementation.
3 Experimental evaluations
MedGAN was evaluated without task-specific architecture or hyperparameter changes across PET-to-CT translation, MR motion correction, and PET denoising, using qualitative and quantitative comparisons.
- Evaluation design: MedGAN was directly applied to three medical imaging tasks without task-specific modifications to architectures or hyperparameters.The tasks were PET-to-CT translation, MR motion-artefact correction, and PET denoising.
- Datasets and tasks: The PET-to-CT experiment used anonymized brain data from 46 patients, translating PET images into synthetic CT images containing detailed bone and soft-tissue information.The target CT modality contains more detailed information than the PET source modality.
- Datasets and tasks: The MR motion-correction experiment translated motion-corrupted images into motion-free images despite sequential acquisition and alignment challenges.Training used 1,445 images and evaluation used 556 images from separate patient groups.
- Datasets and tasks: The PET denoising dataset paired scans reconstructed from 25% of the original acquisition time with original PET scans.It contained 11,420 training slices and 4,411 validation images.
- Evaluation design: Experiments compared individual loss components and multiple translation approaches using qualitative and quantitative evaluation, including radiologist assessment and image-quality metrics.Five experienced radiologists compared MedGAN outputs with ground truth and pix2pix results.
- Baselines: The comparative study reimplemented pix2pix, PAN, and related approaches using a publicly verified pix2pix implementation as its basis.The cited comparison discusses differences in pixel, perceptual, and adversarial loss components.
4 Results
MedGAN was evaluated through loss-component analyses, comparisons with state-of-the-art translation frameworks, and a radiologist perceptual study across three medical image translation tasks. It produced sharper outputs, achieved the best scores across most selected metrics, and was sometimes judged realistic by radiologists.
- 4.1 Analysis of loss functions: MedGAN’s combined framework produced sharper translated images and improved quantitative metrics compared with individual loss components.The traditional adversarial loss performed worst, while perceptual and style-content losses improved detail and structure before CasNet further enhanced sharpness.
- 4.2 Comparison with state-of-the-art techniques: MedGAN was compared with pix2pix, PAN, ID-CGAN, and Fila-sGAN on the proposed medical image translation tasks.The comparison included qualitative image slices and quantitative evaluations.
- 4.2 Comparison with state-of-the-art techniques: MedGAN achieved the best score across the large majority of selected metrics for the different tasks.The competing methods showed task-dependent weaknesses, including blurry details, tilting artefacts, and inconsistent outputs.
- 4.3 Perceptual study and validation: 25.3% of MedGAN PET-to-CT images were classified as real by radiologists, compared with 6.7% for MR motion correction and 14.3% for PET denoising.The percentages represent images classified as real from triads presented to radiologists.
- 4.3 Perceptual study and validation: Radiologists rated MedGAN outputs 3.22 for PET-to-CT, 2.81 for MR motion correction, and 3.02 for PET denoising.The corresponding pix2pix scores were 1.70, 1.98, and 1.73, while the PET-to-CT ground-truth score was 3.81.
5 Discussion
MedGAN combines adversarial, perceptual, and style-content losses with CasNet to produce globally consistent images with detailed textures across three medical translation tasks. It outperformed competing approaches quantitatively and qualitatively, although its current 2D, single-channel design limits practical medical applicability and diagnostic use.
- Framework: MedGAN combines adversarial, perceptual, and style-content losses with CasNet’s progressive encoder-decoder refinement in an end-to-end framework.The losses target high- and low-frequency image components, while CasNet progressively refines translated outputs.
- Loss analysis: MedGAN surpassed individual loss components on PET-to-CT translation, jointly improving global homogeneity and output details.MedGAN-1G also outperformed models using individual losses with identical architectures, indicating the improvement was not solely due to CasNet capacity.
- Cross-task evaluation: MedGAN applied without task-specific modifications outperformed other approaches across PET-to-CT translation, MR motion correction, and PET denoising.The outputs were described as homogeneous, artefact-free, textured, fine-structured, or sharp depending on the task, with quantitative gains across most metrics.
- Perceptual evaluation: 5 radiologists rated MedGAN outputs 2.8−3.2 out of 4, versus 3.7−3.8 for ground truth and 1.7−2.0 for pix2pix.A subset of 6−25% of translated images was judged more realistic than the ground truth images.
- Limitations: MedGAN currently operates on 2D, single-channel inputs and is intended for technical post-processing rather than diagnostic applications.The authors identify 3D volumetric information and multi-channel phase information as important directions for future expansion.
6 Conclusion
MedGAN is an end-to-end medical image translation framework combining conditional adversarial learning, non-adversarial losses, and CasNet. Applied unchanged to three tasks, it outperformed comparable approaches quantitatively and qualitatively, with positive assessment from five radiologists.
- Conclusion: MedGAN combines a conditional adversarial framework, non-adversarial losses, and CasNet to improve global consistency and high-frequency details.The framework is designed for end-to-end medical image translation.
- Conclusion: MedGAN outperformed other translation approaches quantitatively and qualitatively across PET-CT translation, MR motion correction, and PET denoising.The framework required no task-specific modifications for these three medical imaging tasks.
- Conclusion: 5 experienced radiologists positively attested to the subjective performance and fidelity of MedGAN’s results.This assessment complements the reported quantitative and qualitative comparisons.
- Future work: Future work will extend MedGAN to 3D multi-channel volumes and investigate technical post-processing applications.Planned applications include PET attenuation correction using synthetic CT and analysis of corrected MR images for segmentation and organ-volume calculation.
Appendix A Analysis of Generator Architecture
The appendix evaluates CasNet depth within pix2pix models and compares those variants with MedGAN. Quantitative performance improves through the sixth U-block, while visual quality increases as U-blocks are added.
- Architecture analysis: Pix2pix models were implemented with CasNet depths ranging from 1 to 7 U-blocks to assess the architecture’s performance.The comparison includes MedGAN using six U-blocks.
- Quantitative analysis: Quantitative performance increased significantly through the sixth U-block as CasNet capacity grew through concatenated U-blocks.The analysis attributes the capacity increase to using a larger concatenation of U-blocks.
- Quantitative analysis: Beyond six U-blocks, performance saturated for PSNR and MSE or degraded for SSIM and LPIPS scores.The authors state that further investigation is needed to determine the optimum CasNet depth.
- Visual analysis: Visual quality of pix2pix translations improved significantly as the number of CasNet U-blocks increased relative to the MedGAN comparison.Figure 7 compares pix2pix variants with 1, 3, 5, and 6 U-blocks against MedGAN.
Appendix B Additional Results
Appendix B presents additional results comparing MedGAN with other translation approaches on the proposed medical image translation tasks.
- Additional results: Additional results compare MedGAN with other translation approaches across the proposed medical image translation tasks.The appendix figure reports further comparative outputs.