PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Contrast Super-Resolution MRI Through a Progressive Network

Qing Lyu, Hongming Shan, Ge Wang

arXiv:1908.01612v2eess.IVcs.LGphysics.med-ph

TL;DR

MRI super-resolution must recover high-resolution images despite limited clinical acquisition conditions, while multi-contrast images offer complementary anatomical information. The paper proposes non-progressive and progressive neural networks that integrate contrast information in high-level feature space and reports improved image quality, with progressive reconstruction performing better at high up-sampling factors.

  • Problem

    Obtaining high-resolution MRI clinically is difficult, while single-contrast super-resolution has limited prior information and multi-contrast images provide complementary anatomical information.

  • Method

    The paper proposes a one-level non-progressive network for low up-sampling and a two-level progressive WGAN-GP network for high up-sampling, combining multi-contrast information in high-level feature space.

  • Results

    The proposed networks produce high-quality MRI super-resolution images, outperform other multi-contrast methods in SSIM and PSNR, and the progressive network outperforms the non-progressive network for highly down-sampled inputs.

  • Takeaways & Limitations

    High-quality MCSR can be achieved by high-level multi-contrast feature integration, and progressive reconstruction supports large up-sampling factors such as 4×.

Abstract

from arXiv · show

Magnetic resonance imaging (MRI) is widely used for screening, diagnosis, image-guided therapy, and scientific research. A significant advantage of MRI over other imaging modalities such as computed tomography (CT) and nuclear imaging is that it clearly shows soft tissues in multi-contrasts. Compared with other medical image super-resolution (SR) methods that are in a single contrast, multi-contrast super-resolution studies can synergize multiple contrast images to achieve better super-resolution results. In this paper, we propose a one-level non-progressive neural network for low up-sampling multi-contrast super-resolution and a two-level progressive network for high up-sampling multi-contrast super-resolution. Multi-contrast information is combined in high-level feature space. Our experimental results demonstrate that the proposed networks can produce MRI super-resolution images with good image quality and outperform other multi-contrast super-resolution methods in terms of structural similarity and peak signal-to-noise ratio. Also, the progressive network produces a better SR image quality than the non-progressive network, even if the original low-resolution images were highly down-sampled.

I. INTRODUCTION

MRI provides multi-contrast soft-tissue information but obtaining high-resolution images clinically is difficult. The paper develops non-progressive and progressive neural networks that use complementary contrasts and high-level feature integration for MRI super-resolution.

  • MRI offers clear tissue structure and functional information without ionizing radiation, while pulse sequences generate complementary T1, T2, and PD contrasts.
  • High-resolution MR imaging is difficult in clinical applications because MRI systems face trade-offs among cost-effectiveness and signal-to-noise ratio.
  • MCSR combines multiple contrast images to provide more image priors for recovering details than single-contrast super-resolution.
  • The paper proposes a one-level non-progressive network for low up-sampling factors and a two-level progressive network for high up-sampling factors.
  • The proposed networks use high-level feature-space integration, and their generator objective combines adversarial, MSE, perceptual, and texture-matching losses.

B. Down-sampling and Zero-filling

The method simulates low-resolution MRI through frequency-domain down-sampling and reconstructs it with a WGAN-GP network using reference features and multiple complementary losses.

  • Down-sampling and Zero-filling: Low-resolution T2 images are generated by retaining central k-space data at 25%, 11.1%, or 6.25% for 2-, 3-, or 4-fold down-sampling, then zero-filling and inverse Fourier transforming.
  • One-Level Non-progressive Network: The one-level non-progressive model uses a WGAN-GP framework whose generator contains an encoder-decoder network and a reference feature extraction network.
  • One-Level Non-progressive Network: Reference images are converted into feature maps and fed into the encoder-decoder network to guide reconstruction.
  • Objective Function: The generator objective combines adversarial, MSE, perceptual, and texture-matching losses.
  • Objective Function: Perceptual loss compares high-level VGG16 feature maps, while texture matching compares Gram matrices to preserve texture similarity and use correct style-transfer features.

E. Two-Level Progressive Network

The two-level progressive network performs 4-fold MCSR through sequential 2-fold stages, using intermediate supervision to improve high-factor reconstruction.

  • E. Two-Level Progressive Network: Progressive super-resolution divides a large up-sampling task into sequential steps, each increasing image resolution by a smaller factor.
  • E. Two-Level Progressive Network: The proposed two-level network combines two one-level networks to produce 4-fold MCSR, with each level performing 2-fold up-sampling.
  • E. Two-Level Progressive Network: The first level is supervised with 2-fold down-sampled images, while the second level is supervised with the original high-resolution images.
  • E. Two-Level Progressive Network: The progressive objective combines losses from both levels using inputs representing 4-fold LR images, 2-fold LR images, and original HR images.

F. Image Quality Evaluation Metrics

The study evaluates MCSR image quality with SSIM, PSNR, and IFC, combining metrics because no single measure is accurate in all circumstances.

  • SSIM compares luminance, contrast, and structural similarity between images.The reported SSIM scores are averaged over 7 × 7 patches.
  • PSNR is a fidelity measure related to MSE and expressed using the maximum image value and MSE.
  • IFC evaluates image quality through mutual information between a reference image and a distorted image, incorporating human visual perception.Unlike SSIM and PSNR, IFC is not best suited to evaluating structural similarity.
  • The three metrics are used together because each has limitations and no metric accurately evaluates image quality in every circumstance.

A. Datasets

Experiments use registered multi-contrast MRI datasets, with T2 images super-resolved using another contrast as reference and predefined training and testing pairs.

  • The IXI dataset contains registered T2- and PD-weighted MRI images from 578 patients.The study selected 7,000 T2–PD pairs for training and 1,955 pairs for testing, using 10-fold cross-validation.
  • IXI HR images have size 256 × 256, and T2 LR inputs were created using 2-fold, 3-fold, and 4-fold down-sampling.
  • The NAMIC dataset contains registered T1- and T2-weighted MRI images from 20 patients.The study used 1,620 pairs for training and 180 different pairs for testing.
  • For NAMIC, T2 images were super-resolved with T1 images as reference, using 2-, 3-, and 4-fold LR inputs derived from 256 × 256 HR images.

B. Experimental Details

Training uses tuned objective-function weights and monitors multiple losses during optimization; the reported curves indicate stable convergence, while the ablation designs compare alternative image and feature-space integrations.

  • Training used 112K 64 × 64 image patches, batch size 32, 50 epochs, and learning rate 2×10^-5.The discriminator was trained four times before each generator update.
  • Hyperparameters λ1, λ2, and λ3 were selected using the objective-function tuning results shown in Fig. 3.The figure uses logarithmic x-axes.
  • Perceptual loss, MSE loss, and Wasserstein distance were recorded each epoch to monitor convergence and compare feature-space, image-space, and distributional similarity.
  • After 40 epochs, all monitored curves were stable and close to zero, indicating stable convergence and strong similarity to HR ground truth.
  • The four ablation studies compare SISR, image synthesis, low-level multi-contrast integration, and high-level feature-space integration.Fig. 4 distinguishes the encoder-decoder network from the reference feature extraction network.

B. Ablation Study

The ablation studies show that using PD-weighted HR images improves MCSR over single-contrast SR, while combining contrasts in high-level feature space performs best. Directly synthesizing T2-weighted HR images from PD-weighted HR inputs produces the poorest results.

  • The four ablation models test reference-image removal, direct image synthesis, low-level fusion, and high-level feature fusion.All models were evaluated using the IXI dataset to identify the most effective network structure for MCSR.
  • Directly converting PD-weighted HR images into T2-weighted HR images produces the lowest SSIM, PSNR, and IFC scores and the largest pixel-value differences.This ablation also fails to recover fissure shapes visible in the other SR results.
  • Using PD-weighted HR images improves MCSR over the single-contrast baseline, yielding higher PSNR, SSIM, and IFC scores and clearer structures.The highlighted curve remains difficult to see with the single-contrast model but becomes clearer when PD-weighted HR images are used.
  • Combining T2-weighted LR and PD-weighted HR images in high-level feature space outperforms low-level image-space combination on PSNR, SSIM, and IFC.The high-level feature-space combination is the proposed network configuration.

C. Non-progressive Model MCSR Results with 2-fold, 3-fold and 4-fold Up-sampling

The one-level non-progressive model produces structurally similar MCSR images across 2-, 3-, and 4-fold up-sampling, including blurred, highly down-sampled inputs. However, image quality decreases as the down-sampling factor increases, motivating the progressive model for high up-sampling.

  • C. Non-progressive Model MCSR Results with 2-fold, 3-fold and 4-fold Up-sampling: The one-level model produces clearer shapes, texture details, and strong structural similarity even when LR images are highly down-sampled.Results are reported for 2-, 3-, and 4-fold up-sampling on the IXI and NAMIC datasets.
  • C. Non-progressive Model MCSR Results with 2-fold, 3-fold and 4-fold Up-sampling: Higher down-sampling factors produce more blurred LR images with fewer visible details, making the corresponding SR results harder to recover accurately.The paper reports lower image quality as the down-sampling factor increases.
  • C. Non-progressive Model MCSR Results with 2-fold, 3-fold and 4-fold Up-sampling: Smaller down-sampling factors yield higher PSNR, SSIM, and IFC scores and smaller pixel-value differences from the HR ground truth.This pattern indicates better image quality for SR results generated from less aggressively down-sampled inputs.
  • D. Progressive Model Further Improves Image Quality with 4-fold Up-sampling: The two-level progressive model is introduced to further improve the quality of 4-fold MCSR results from highly down-sampled inputs.Its outputs are compared with those of the one-level non-progressive model.
  • D. Progressive Model Further Improves Image Quality with 4-fold Up-sampling: The progressive model produces sharper edges, clearer texture details, higher PSNR, SSIM, and IFC scores, and smaller ground-truth differences than the non-progressive model.Both constrained and unconstrained progressive results outperform the non-progressive results; the constrained version scores higher than the unconstrained version.

E. Comparing with State-of-the-art Methods

The proposed progressive network outperforms the one-level non-progressive model and other MCSR methods, with high-level feature-space combination supporting stronger reconstruction quality.

  • The progressive model produces the highest SSIM and PSNR scores among the compared state-of-the-art MCSR methods.The one-level non-progressive model achieves the highest SSIM scores among all compared methods.
  • MCSR methods outperform SISR methods, indicating that multiple contrasts provide more image priors for super-resolution than a single contrast.
  • Multi-contrast information combined in a high-level feature space yields better MCSR results than low-level image-space combination.Separate contrast-specific feature extraction and T2 feature skip connections support flexible feature extraction and stronger T2 characteristics in restoration.
  • Image quality decreases as the down-sampling factor increases because more blurred low-resolution images contain fewer recoverable details.
  • The progressive model achieves significantly better 4-fold MCSR results than the one-level non-progressive network and can approach the latter’s 3-fold results.The authors attribute this improvement to strict parameter constraints and level-wise 2-fold training.

VI. CONCLUSION

The paper proposes distinct non-progressive and progressive networks for low- and high-factor MCSR, respectively, combining multiple losses and high-level contrast features. The progressive design supports high-quality reconstruction at larger up-sampling factors and may extend further when more contrast images are available.

  • The paper proposes a one-level non-progressive network for low-factor up-sampling and a two-level progressive network for large-factor up-sampling.
  • High-quality MCSR results are generated when multi-contrast information is combined in a high-level feature space.
  • The progressive network could be extended to larger up-sampling factors if more different contrast images are available.
Loading 1908.01612v2…