PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain

Senrong You, Yong Liu, Baiying Lei, Shuqiang Wang

arXiv:2011.04145v1eess.IVcs.CVcs.LG

TL;DR

High-resolution brain MR acquisition is limited by hardware, scanning time, and cost, while existing super-resolution models can lose anatomical detail. FP-GANs decomposes MR images into wavelet sub-bands, super-resolves them with dedicated GANs and sub-band attention, and reconstructs the whole image with IDWT; experiments on MultiRes_7T report better quantitative and qualitative performance than competing methods.

  • Problem

    Existing CNN-based MR super-resolution models can be insensitive to anatomical detail and produce over-smoothed results.

  • Method

    FP-GANs decomposes MR images into low-frequency and high-frequency wavelet sub-bands, processes each with a sub-band GAN, applies sub-band attention, and uses optimizable IDWT for reconstruction.

  • Results

    FP-GANs outperforms competing methods on SSIM and FID in MultiRes 7T experiments.

  • Takeaways & Limitations

    Separating global topology from high-frequency texture supports finer anatomical reconstruction while simplifying the super-resolution task for individual GANs.

Abstract

from arXiv · show

Magnetic resonance imaging plays an important role in computer-aided diagnosis and brain exploration. However, limited by hardware, scanning time and cost, it's challenging to acquire high-resolution (HR) magnetic resonance (MR) image clinically. In this paper, fine perceptive generative adversarial networks (FP-GANs) is proposed to produce HR MR images from low-resolution counterparts. It can cope with the detail insensitive problem of the existing super-resolution model in a divide-and-conquer manner. Specifically, FP-GANs firstly divides an MR image into low-frequency global approximation and high-frequency anatomical texture in wavelet domain. Then each sub-band generative adversarial network (sub-band GAN) conquers the super-resolution procedure of each single sub-band image. Meanwhile, sub-band attention is deployed to tune focus between global and texture information. It can focus on sub-band images instead of feature maps to further enhance the anatomical reconstruction ability of FP-GANs. In addition, inverse discrete wavelet transformation (IDWT) is integrated into model for taking the reconstruction of whole image into account. Experiments on MultiRes_7T dataset demonstrate that FP-GANs outperforms the competing methods quantitatively and qualitatively.

I. INTRODUCTION

MRI offers detailed soft-tissue imaging but acquiring high-resolution scans is constrained by time, cost, hardware, and motion blur. FP-GANs addresses detail-insensitive super-resolution through wavelet-domain decomposition, sub-band GANs, adaptive attention, and whole-image reconstruction.

  • Motivation: Existing CNN-based super-resolution models can constrain anatomical-detail sensitivity and produce over-smoothed MR images.The paper motivates this issue as a consequence of applying CNNs directly to feature extraction.
  • Method: FP-GANs decomposes MR images into low-frequency global topology and high-frequency textures, then super-resolves each wavelet sub-band with a dedicated GAN.This divide-and-conquer design simplifies each mapping task and separates global structure from detailed texture.
  • Results: FP-GANs outperforms interpolation-, GAN-, and wavelet-based competitors on SSIM and FID in MultiRes 7T experiments.The comparison uses scale factor ×4 and evaluates Bilinear, Bicubic, ESRGAN, and wavelet-based methods.
  • Method: Sub-band attention adaptively allocates focus across approximation and directional texture sub-bands rather than feature channels.The authors report that it helps balance different kinds of textures.
  • Method: Optimizable inverse discrete wavelet transformation incorporates the structure of the recomposed MR image into FP-GANs.The paper reports that this improves super-resolution performance.

II. RELATED WORK

Prior wavelet-domain methods reconstruct high-resolution images from sub-bands, while attention mechanisms adaptively weight informative components. FP-GANs combines sub-band processing, attention, and inverse reconstruction in a three-part architecture.

  • Related wavelet methods: Wavelet-domain CNNs target high-frequency texture reconstruction, but jointly processing sub-bands can still cause textures to be ignored.The paper attributes this risk to the relatively small influence of high-frequency textures.
  • FP-GANs architecture: FP-GANs applies DWT to obtain four sub-bands containing low-frequency global and high-frequency texture information.The sub-bands are LL, LH, HL, and HH.
  • FP-GANs architecture: Separate GANs learn mappings from low-resolution sub-band images to corresponding high-resolution sub-band images.Each GAN focuses on one sub-band while its generator models high-resolution sub-band distributions.
  • FP-GANs architecture: Sub-band attention balances approximation, horizontal, vertical, and diagonal textures in the generated super-resolution sub-bands.Its placement between the sub-band GAN generator and discriminator supports adaptive texture trade-offs.

A , ISR

Inverse wavelet transformation reconstructs the super-resolved MR image by synthesizing weighted super-resolution sub-band images.

  • IDWT synthesizes weighted SR sub-band images to reconstruct the SR MR image.

D. Discrete wavelet transformation

A one-level wavelet transform decomposes an MR image into four oriented sub-bands, and inverse transformation recomposes them into the original image. Haar-based equations illustrate the corresponding forward and inverse pixel relationships.

  • D. Discrete wavelet transformation: One-level DWT decomposes an image into LL, LH, HL, and HH sub-bands representing global approximation and directional textures.The sub-bands can be losslessly recomposed into the original MR image using IDWT.
  • D. Discrete wavelet transformation: FP-GANs uses one-level 2D DWT to divide differently oriented textures before processing them separately.The paper presents Haar-based DWT as an example transformation.
  • D. Discrete wavelet transformation: The Haar forward equations combine four original-image pixel intensities into four sub-band coefficients.The resulting coefficients are denoted a, b, c, and d, while A, B, C, and D denote original pixel intensities.
  • D. Discrete wavelet transformation: The inverse Haar equations recover each original pixel intensity from the four sub-band coefficients.Each reconstructed pixel is expressed as a signed average of a, b, c, and d.

E. Sub-band GANs

FP-GANs uses four same-architecture sub-band GAN pairs to super-resolve LL, LH, HL, and HH wavelet sub-bands separately. Each generator learns high-resolution sub-band distributions while its discriminator distinguishes real from generated images.

  • Four same-architecture GAN pairs independently transform LR LL, LH, HL, and HH sub-band images into higher-resolution counterparts.The generators learn high-resolution sub-band distributions, while discriminators estimate whether outputs are real or generated.
  • The adversarial procedure is defined over low-resolution MR images X and high-resolution ground-truth images Y.X denotes the low-resolution dataset, while Y denotes the high-resolution dataset used as ground truth.
  • Each sub-band GAN contains a generator built from convolutional, RRDB, trunk, upsampling, and final convolutional components.The architecture follows ESRGAN and RFB-ESRGAN designs.

F. Training Loss

FP-GANs combines adversarial, wavelet, and pixel losses to guide training from complementary perspectives. Its adversarial formulation uses relativistic GAN loss for the four sub-band GANs.

  • Adversarial, wavelet, and pixel losses jointly guide FP-GANs training from different perspectives.
  • Relativistic GAN loss is applied to the four sub-band GANs to stabilize training and improve generated MR image quality.The formulation uses the assumption that real and generated MR images each account for half of the supervised training examples.
  • The discriminator output D(x) and generated image x_f = G(x) enter the adversarial-loss formulation for optional sub-bands A, H, V, and D.

2) Wavelet Loss:

Wavelet loss emphasizes high-frequency components to recover anatomical detail, while pixel loss evaluates the globally reconstructed image. The total generator objective combines weighted sub-band losses.

  • 2) Wavelet Loss:: Wavelet loss emphasizes high-frequency components because textures are mainly represented there.The paper associates this loss with reducing over-smoothing and recovering more anatomical MR detail.
  • 2) Wavelet Loss:: Wavelet loss compares low- and high-resolution images for each corresponding sub-band.
  • 3) Pixel Loss:: Pixel loss uses a Charbonnier penalty on the reconstructed image to account for global structure and avoid the over-smoothing effect of L2 normalization.The Charbonnier penalty is described as a differentiable variant of L1 normalization.
  • 4) FP-GANs Loss:: The total generator loss sums weighted losses from the approximation, horizontal, vertical, and diagonal sub-bands.Each sub-band loss combines adversarial and wavelet terms, while λ1 through λ4 control their relative penalties.

III. EXPERIMENTS

Experiments use MultiRes 7T functional MR images at 0.8 mm and 3.0 mm resolutions, cropped into grayscale slices and downsampled at multiple scale factors. FP-GANs is implemented in PyTorch with specified RRDB, discriminator, optimizer, and learning-rate settings.

  • High-resolution 200 × 200 slices are downsampled with scale factors 2, 4, and 8 to obtain corresponding low-resolution inputs.
  • FP-GANs uses 16 RRDBs in the generator, a discriminator with 10 convolutional and 2 fully connected layers, and Adam optimization.
  • The generator, discriminator, and IDWT learning rates are set to 1 × 10^-4, 5 × 10^-4, and 1 × 10^-4, respectively, with batch size 16.

C. Evaluation metrics

Evaluation combines standard super-resolution metrics with visual and wavelet-domain comparisons to assess both image quality and anatomical detail. FP-GANs is reported to outperform competing interpolation, GAN-based, and wavelet-based methods, with sub-band recovery supporting its detail reconstruction.

  • Because PSNR can disagree with human visual quality, the evaluation includes FID alongside SSIM and visual comparisons.The paper notes that Bicubic and Bilinear can rank higher in PSNR while appearing visually worse.
  • FP-GANs achieves better SSIM and FID than Bicubic, Bilinear, ESRGAN, DWSR, and WaveletSRNet in the reported comparison.FP-GANssa, which adds sub-band attention, performs slightly better than FP-GANs.
  • FP-GANs and FP-GANssa produce sharper, clearer anatomical textures with fewer heatmap differences than the competing methods.The heatmap measures absolute difference from the ground truth; brighter pixels indicate larger differences.
  • Sub-band GANs reproduce more authentic LL, LH, HL, and HH images than the compared wavelet-based methods, supporting compelling reconstructed MR images.Figure 8 evaluates recovery of the four wavelet sub-bands.
  • FP-GANs outperforms competing interpolation, GAN-based, and wavelet-based methods in most quantitative cases, especially at ×4 scaling.Table I summarizes quantitative evaluation on the MultiRes 7T dataset.

IV. ABLATION STUDY

The ablation study evaluates instance normalization, sub-band attention, wavelet loss, and attention-weight evolution. These components are associated with faster convergence, higher SSIM, stabilized training, and greater emphasis on high-frequency anatomical textures.

  • Ablation study on instance normalization and sub-band attention: FP-GANs with instance normalization and sub-band attention converges fastest and achieves the highest SSIM, while removing both modules performs worst.This comparison is reported in Figure 9a.
  • Ablation study on wavelet loss: Wavelet loss stabilizes and accelerates FP-GANs convergence.The effect is shown in Figure 9b.
  • Sub-band attention weight evolution: Sub-band attention weights evolve from an initially random distribution toward a regular distribution that emphasizes high-frequency sub-bands, especially HH.The weight-update progression is shown across training epochs in Figure 10.
  • Discussion and conclusion: FP-GANs addresses texture loss by separately super-resolving low-frequency global topology and high-frequency texture sub-bands.The discussion links this divide-and-conquer design to detail sensitivity and more stable training than a single GAN approach.
Loading 2011.04145v1…