Source-linked AI summary
Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis
Shengye Hu, Baiying Lei, Yong Wang, Zhiguang Feng, Yanyan Shen, Shuqiang Wang
TL;DR
PET data are often unavailable because of cost, radiation, and access limitations, motivating MR-to-PET synthesis. BMGAN uses bidirectional latent mapping with a 3D DenseU-Net and extensive objectives, and experiments report superior quantitative, qualitative, and classification performance against competitive methods.
Problem
PET data are scarce because scans are unavailable in many medical centers and involve radioactive tracers, while prior methods struggle with latent-vector diversity and 3D brain structure.
Method
BMGAN jointly optimizes MR image context and latent vectors through bidirectional mapping, using a 3D DenseU-Net generator and extensive objective functions.
Results
BMGAN achieves the best reported performance among competitive methods across MAE, PSNR, MS-SSIM, and FID, generating plausible and diverse PET images.
Takeaways & Limitations
The method can faithfully preserve diverse brain-structure details while synthesizing perceptually realistic PET images from brain MR images.
Abstract
from arXiv · showhide
Fusing multi-modality medical images, such as MR and PET, can provide various anatomical or functional information about human body. But PET data is always unavailable due to different reasons such as cost, radiation, or other limitations. In this paper, we propose a 3D end-to-end synthesis network, called Bidirectional Mapping Generative Adversarial Networks (BMGAN), where image contexts and latent vector are effectively used and jointly optimized for brain MR-to-PET synthesis. Concretely, a bidirectional mapping mechanism is designed to embed the semantic information of PET images into the high dimensional latent space. And the 3D DenseU-Net generator architecture and the extensive objective functions are further utilized to improve the visual quality of synthetic results. The most appealing part is that the proposed method can synthesize the perceptually realistic PET images while preserving the diverse brain structures of different subjects. Experimental results demonstrate that the performance of the proposed method outperforms other competitive cross-modality synthesis methods in terms of quantitative measures, qualitative displays, and classification evaluation.
I. INTRODUCTION
Medical imaging synthesis addresses missing PET data by estimating target-modality images from available MR scans. BMGAN targets two limitations of prior methods: weak latent-vector diversity and loss of 3D spatial context.
- PET data is scarce because scans are unavailable in many medical centers and involve radioactive tracers, whereas MR is more widely available and non-invasive.
- Cross-modality synthesis estimates a subject’s target-modality image from a source-modality image and may provide low-cost imaging data for clinical applications.
- Prior GAN methods often ignore randomly sampled latent vectors, making it difficult to generate plausible brain images with diverse subject-specific structures.
- Most earlier methods synthesize 2D slices, causing discontinuities and spatial-information loss; patch-based 3D methods may still miss global and local volumetric context.
- BMGAN is a 3D end-to-end MR-to-PET network that jointly optimizes image context and latent vectors through bidirectional mapping.
- The method uses a 3D DenseU-Net generator and adversarial, KL-divergence, reconstruction, and perceptual objectives to improve synthetic-image quality.
- This journal version extends an earlier conference work with dense generator connections, analyses of 3D architecture and adversarial training, and additional objective-function evaluation.
II. METHOD
BMGAN learns a nonlinear mapping from paired brain MR images to PET images using a 3D generative adversarial architecture. Its generator, discriminator, and encoder jointly model target-image synthesis and spatial structure.
- The method learns a cross-modality mapping f: x → y from paired brain MR images x to corresponding PET images y.
- BMGAN comprises a generator network, discriminator network, and encoder network for 3D brain MR-to-PET synthesis.
- The DenseU-Net generator uses 3D convolutions to model PET spatial structure and eliminate slice discontinuities associated with 2D networks.
- Adversarial learning uses an additional patch-level discriminator, while the objective functions include losses intended to improve synthetic target-image visual quality.
III. BASIC IDEAS OF BMGAN
BMGAN combines latent-space bidirectional mapping with a DenseU-Net generator to synthesize variable, spatially coherent PET images from MR inputs. Dense connections address information flow in deep volumetric networks.
- GANs map latent vectors to synthetic images, but BMGAN provides an invertible connection between brain PET images and latent vectors to encourage output variability.
- Forward mapping encodes PET images into latent vectors constrained toward a standard normal distribution, then combines them with MR inputs to generate PET images.
- U-Net preserves shared low-level MR–PET structural features through contracting and expanding paths operating at multiple scales.
- DenseNet-style connections address vanishing and diluted input or gradient information in deeper networks.
- The 3D DenseU-Net combines DenseNet and U-Net, using dense connections within encoder and decoder blocks alongside long-range skip connections.
- The proposed architecture contains 13 dense blocks, 7 transition layers, and 7 upsampling layers.
- The discriminator classifies 32 × 32 × 32 input patches as real or fake, supporting local PET-structure reconstruction with relatively few parameters.
2) Patch-level Discriminator Network:
The supplied passages identify the encoder as a ResNet-34 network but do not provide enough text to explain the patch-level discriminator subsection’s intended comparison or layout.
- The encoder uses ResNet-34 components with convolution, normalization, ReLU, pooling, and residual mappings.
B. Objective functions
BMGAN optimizes image realism, latent-space consistency, structural fidelity, and perceptual similarity through an extensive objective function. These terms address distribution mismatch, latent-vector alignment, pixel-level structure, and high-level visual differences.
- Objective design: The objective combines adversarial, KL-divergence, pixel-wise reconstruction, and perceptual losses to optimize MR-to-PET synthesis.The aggregate generator objective uses hyperparameters to control the relative importance of individual loss terms.
- Adversarial loss: Least-square adversarial training is used to stabilize optimization and address convergence or mode-collapse problems.The adversarial loss aims to align synthetic and target data distributions.
- Latent-space constraint: KL-divergence constrains encoded PET latent vectors to follow the Gaussian distribution of sampled latent vectors.The encoded vector is trained toward a standard normal distribution.
- Reconstruction loss: L1 reconstruction loss imposes structural fidelity by minimizing absolute pixel-intensity differences between synthetic and real PET images.This complements adversarial training, which alone does not guarantee consistency with real image regions.
- Perceptual loss: Perceptual loss compares high-level feature representations, addressing perceptual differences that pixel-wise loss may miss.The feature maps are taken before the second max-pooling operation of a pretrained VGG-16 network.
IV. EXPERIMENT
The experiments use paired MR and FDG-PET data from 680 ADNI subjects and evaluate synthesis with reconstruction, signal-to-noise, structural-similarity, and GAN-specific metrics. The metrics capture pixel error, fidelity, multiscale similarity, and synthetic-image quality and diversity.
- Data: The evaluation uses 680 ADNI subjects with paired MR and PET images to learn nonlinear MR-to-FDG-PET mapping.Nonbrain tissues were removed and both modalities were spatially aligned to a standardized template.
- Metrics: Four metrics evaluate synthesis: MAE, PSNR, MS-SSIM, and FID.Together they assess reconstruction error, signal quality, multiscale structural similarity, and synthetic-image quality and diversity.
- Metrics: MAE measures reconstruction error using the absolute differences between real and synthetic PET image pixels.The metric compares corresponding image pixels across the real and synthetic PET images.
- Metrics: PSNR evaluates synthetic-image quality using the mean-squared error between real and synthetic PET images.The experiment defines MSE as the mean-squared error between the two image types.
- Metrics: MS-SSIM measures multiscale structural similarity, with higher values indicating greater similarity between real and synthetic PET.It extends SSIM beyond a single scale using structural information from multiple scales.
- Metrics: FID measures synthetic-image quality and diversity through a pretrained Inception network, with lower values indicating better performance.The metric compares empirical means and covariances of real and synthetic Gaussian variables modeled by the network.
B. Comparison with 2D-based BMGAN architectures
The 3D BMGAN is compared with a 2D variant using matched architectures and training settings. Across qualitative displays, difference maps, and aggregate metrics, the 3D model shows better image quality and diversity.
- Experimental comparison: The 2D BMGAN processes axial, coronal, and sagittal slices, whereas the 3D BMGAN uses the whole image as input.Both variants use the same network architectures and training settings for comparison.
- Qualitative comparison: The 3D BMGAN produces visually more pleasing PET images with greater similarity to ground truth across all three planes.The comparison uses synthetic samples from three subjects and includes real PET images.
- Difference maps: Difference maps show larger high-difference regions for the 2D variant than for the 3D model.The maps display absolute intensity differences between synthetic and real PET images using pseudo-color processing.
- Interpretation: The reported advantage of 3D convolution is attributed to using volumetric patches that incorporate brain-volume structural information and avoid slice-discontinuous estimation.The 2D model processes slices independently, whereas the 3D model considers volume structure.
- Quantitative comparison: 3D BMGAN achieves higher PSNR and SSIM and lower MAE and FID than the 2D variant.These results indicate better image quality and diversity for the 3D model.
C. Contribution of the Adversarial training strategy
Adversarial training improves BMGAN’s synthetic PET quality by producing clearer images and better quantitative results than a version without the discriminator.
- Experimental setup: The comparison removes the discriminator network to isolate the contribution of adversarial training.The simplified model retains the 3D DenseU-Net generator and encoder network.
- Qualitative results: Without adversarial training, synthetic PET images appear over-smoothed and fuzzy relative to ground-truth images.The voxel-wise synthesis strategy ignores global structure and contextual information.
- Qualitative results: With adversarial training, the synthetic results are clearer and have better visual quality.
- Quantitative results: Approximately 2.34 higher averaged PSNR is obtained by BMGAN than by the model without the discriminator.The paper reports improvement in all evaluated metrics.
D. Contribution of the DenseU-Net generator architecture
DenseU-Net provides the strongest generator architecture among the compared U-Net variants, while adversarial BMGAN outputs are qualitatively compared with a discriminator-removed model.
- Architecture comparison: The study compares original U-Net, ResU-Net, and DenseU-Net using MAE, PSNR, MS-SSIM, and FID.
- Architecture comparison: Original U-Net achieves the lowest performance, partly because vanishing gradients hinder parameter updates and optimization.
- Architecture comparison: ResU-Net improves quantitative results over original U-Net through residual paths that enhance gradient flow and convergence.
- Architecture comparison: DenseU-Net achieves the best performance across all four metrics through efficient feature reuse and stronger information flow.Dense connections and skip connections also help mitigate inconsistent MR and PET feature distributions.
E. Contribution of the extensive objective functions
The extensive objective combines adversarial, KL-divergence, reconstruction, and perceptual losses to improve structural fidelity and detailed feature representation in synthetic PET.
- Loss-function ablation: The ablation evaluates progressively adding L1 reconstruction and perceptual losses to adversarial loss and KL-divergence constraints.
- Loss-function ablation: Without reconstruction and perceptual losses, adversarial and KL-divergence terms preserve global information but insufficiently match pixel-level distributions.
- Loss-function ablation: Adding pixel-wise L1 loss significantly improves synthetic-image quality across all four metrics and captures overall PET-related structure.
- Loss-function ablation: Adding perceptual loss produces a modest further improvement, with a relatively significant gain in MS-SSIM.Perceptual loss helps retain more detailed feature representations.
- Combined objective: The full objective includes adversarial loss, KL-divergence constraint, pixel-wise reconstruction loss, and perceptual loss.The paper reports stable and robust MR-to-PET synthesis with this combination.
F. Comparison with the competitive synthetic methods
BMGAN is compared with FCN, GAN, U-Net, CycleGAN, and pGAN using qualitative images, difference maps, and MAE, PSNR, MS-SSIM, and FID. It achieves the strongest reported visual and quantitative performance, especially for diverse brain structures.
- Compared methods: The comparison includes FCN, GAN, U-Net, CycleGAN, and pGAN as baselines or competitive MR-to-PET synthesis methods.
- Qualitative comparison: Qualitative evaluation uses axial-plane synthetic and real PET images from four samples with diverse brain structures, alongside difference maps.
- Qualitative comparison: FCN differs substantially from ground-truth PET, while GAN is closer but still produces poor-quality images with many artifacts.
- Qualitative comparison: CycleGAN and pGAN improve visual quality over earlier models but remain coarser near boundaries than BMGAN.
- Qualitative comparison: BMGAN produces sharper boundaries and better detail and diversity across subjects with different brain structures.CycleGAN and pGAN tend to produce similar results and can fail when ground-truth structures change substantially.
- Quantitative comparison: BMGAN achieves the best performance on MAE, PSNR, MS-SSIM, and FID among the compared methods.Lower MAE and FID, and higher PSNR and MS-SSIM, correspond to better quality and diversity in the reported evaluation.
G. Classification experiment on the synthetic images
The study evaluates synthetic PET images not only with image-quality metrics but also on an AD-versus-Normal classification task. BMGAN achieved higher AUC than the compared methods, supporting its ability to generate realistic and diverse PET images with detailed brain attributes.
- Classification experiment on the synthetic images: The authors evaluated synthetic PET images on a binary AD-versus-Normal classification task because standard image metrics were not directly diagnostic.VGG-16 served as the gold-standard classifier.
- Classification experiment on the synthetic images: BMGAN achieved higher AUC than the other compared methods on the classification experiment.AUC was used to address class imbalance between the two classes.
- Classification experiment on the synthetic images: The classification results support BMGAN’s ability to generate realistic and diverse PET images that preserve detailed brain attributes.The authors further identify the method as an effective data augmentation approach.