PaperlayerView exact revision on arXiv

Source-linked AI summary

An Arbitrary Scale Super-Resolution Approach for 3D MR Images via Implicit Neural Representation

Qing Wu, Yuwei Li, Yawen Sun, Yan Zhou, Hongjiang Wei, Jingyi Yu, Yuyao Zhang

arXiv:2110.14476v3eess.IVcs.CV

TL;DR

MRI acquisition faces a trade-off among resolution, SNR, and scan time, while many SISR methods require separate models for different up-sampling scales. ArSSR learns a continuous implicit voxel function from paired HR-LR examples, using one trained model for arbitrary scales; the authors report state-of-the-art 3D MR reconstruction results across three datasets.

  • Problem

    MRI acquisition trades off resolution, SNR, and scan time, while most deep-learning SISR methods handle fixed scales and require separate models for different up-sampling rates.

    Source: §I · pp. 1–2

  • Method

    ArSSR learns a continuous implicit voxel function from paired HR-LR examples, with an encoder extracting LR features and a decoder approximating the function.

    Source: §I · p. 2

  • Results

    Across three datasets, the authors report state-of-the-art 3D HR MR reconstruction while a single trained model supports arbitrary up-sampling scales.

    Source: §I · p. 2

  • Takeaways & Limitations

    One trained model supports arbitrary scales, addressing the need to train and store separate scale-specific models.

    Source: §I · p. 2

    ArSSR’s generalization is limited by its training data; a model trained on T1-weighted brain MRI may perform worse on other sequences or organs.

    Source: §VI · p. 11

Abstract

from arXiv · show

High Resolution (HR) medical images provide rich anatomical structure details to facilitate early and accurate diagnosis. In MRI, restricted by hardware capacity, scan time, and patient cooperation ability, isotropic 3D HR image acquisition typically requests long scan time and, results in small spatial coverage and low SNR. Recent studies showed that, with deep convolutional neural networks, isotropic HR MR images could be recovered from low-resolution (LR) input via single image super-resolution (SISR) algorithms. However, most existing SISR methods tend to approach a scale-specific projection between LR and HR images, thus these methods can only deal with a fixed up-sampling rate. For achieving different up-sampling rates, multiple SR networks have to be built up respectively, which is very time-consuming and resource-intensive. In this paper, we propose ArSSR, an Arbitrary Scale Super-Resolution approach for recovering 3D HR MR images. In the ArSSR model, the reconstruction of HR images with different up-scaling rates is defined as learning a continuous implicit voxel function from the observed LR images. Then the SR task is converted to represent the implicit voxel function via deep neural networks from a set of paired HR-LR training examples. The ArSSR model consists of an encoder network and a decoder network. Specifically, the convolutional encoder network is to extract feature maps from the LR input images and the fully-connected decoder network is to approximate the implicit voxel function. Due to the continuity of the learned function, a single ArSSR model can achieve arbitrary up-sampling rate reconstruction of HR images from any input LR image after training. Experimental results on three datasets show that the ArSSR model can achieve state-of-the-art SR performance for 3D HR MR image reconstruction while using a single trained model to achieve arbitrary up-sampling scales.

I. INTRODUCTION

MRI acquisition faces a resolution, SNR, and scan-time trade-off, making single-image super-resolution from one isotropic LR image a practical focus. ArSSR addresses the fixed-scale limits of existing methods with one model for arbitrary up-sampling rates.

I. INTRODUCTION

  • Increasing MRI resolution from 2×2×2 mm3 to 1×1×1 mm3 requires an average of 64 scans to maintain similar SNR, significantly increasing scan time.SISR reconstructs an HR image from one isotropic LR input without additional scanning.Source: §I · p. 1
  • SISR reconstructs an HR image from a single LR image, whereas MISR combines high-frequency information from multiple non-isotropic scans of the same subject.The MISR scans are acquired in different orientations.Source: §I · p. 1
  • Most deep-learning SISR methods target fixed integer scales, requiring separate training and storage for each scale and substantial computation and resources.They learn scale-specific projections between LR and HR images.Source: §I · p. 2
  • ArSSR represents different-resolution MR images as samples of a continuous implicit voxel function, enabling one trained model to reconstruct arbitrary integer or non-integer up-sampling rates.Its convolutional encoder extracts voxel-specific feature maps, and its fully connected decoder approximates the function.Source: §I · p. 2

A. Single Image Super-Resolution

  • Interpolation methods can produce blocking artifacts, while regularization-based methods are limited by prior assumptions that may not hold for image regions.Deep CNNs improved SISR accuracy by learning nonlinear LR-to-HR mappings.Source: §I · p. 1, A. Single Image Super-Resolution · p. 2
  • ArSSR treats SISR at arbitrary scales as one task, unlike prior 3D models that train and store separate models for each up-sampling scale.The paper presents this as improving practical applicability.Source: A. Single Image Super-Resolution · p. 3

B. Implicit Neural Representation

Implicit neural representations model signals continuously, so an MLP can estimate signal values at arbitrary positions independently of signal resolution. ArSSR applies this idea to 3D MR SISR by learning an implicit voxel function enriched with encoder features.

B. Implicit Neural Representation

  • Implicit neural representations approximate continuous functions mapping spatial coordinates to signal responses, allowing estimates at arbitrary positions independently of signal resolution.The function is approximated with a fully connected network.Source: B. Implicit Neural Representation · p. 3
  • ArSSR learns an implicit voxel function for arbitrary-scale 3D MR SISR, drawing on prior implicit-function work for images and guided super-resolution.The model learns the function from paired HR and LR examples, rather than fitting a separate function for each image.Source: B. Implicit Neural Representation · p. 3, A. Problem Modeling · p. 3

A. Problem Modeling

  • HR and LR images are modeled as differently sampled representations of one continuous function over normalized 3D voxel coordinates, allowing arbitrary up-sampling once the function is approximated.The coordinates are normalized to [-1, 1] along each dimension.Source: A. Problem Modeling · p. 3
  • ArSSR combines coordinate information with voxel-specific latent features extracted by a convolutional encoder, helping the implicit decoder recover fine details at higher resolutions.The latent features carry local semantic information; the passage specifies a 128-dimensional feature vector.Source: A. Problem Modeling · p. 3

B. Model Overview

ArSSR encodes LR images into feature maps and uses a decoder to predict HR voxel intensities from query coordinates and corresponding features. Training minimizes an L1 loss, while inference samples the learned implicit voxel function on a denser grid to produce HR images at arbitrary scales.

B. Model Overview

  • A convolutional encoder converts each LR image into a feature map, which is interpolated to obtain features at queried HR voxel coordinates.The decoder then uses each coordinate and its corresponding feature vector to predict that voxel's intensity.Source: B. Model Overview · p. 4
  • The encoder and decoder are optimized together by minimizing L1 loss between predicted and true HR voxel intensities.The training set contains paired LR-HR images, and the loss is evaluated over sampled HR coordinates.Source: B. Model Overview · pp. 3–4
  • At inference, ArSSR samples the implicit voxel function on a denser coordinate grid to generate an HR image at any scale k > 1.The HR grid dimensions are [H, W, D] = k · [h, w, d].Source: B. Model Overview · p. 4

C. Encoder Network

The encoder extracts voxel-wise latent features to preserve local image information for reconstruction. Its implementation is a modified 3D residual dense network, though the encoder can be replaced by another SR network without an upscaling layer.

C. Encoder Network

  • The encoder maps an LR image to a 128-channel feature vector for each voxel, forming a feature map for subsequent decoding.Source: C. Encoder Network · p. 4
  • Voxel-wise feature extraction integrates local image intensity information, supporting recovery of fine HR details even at large up-sampling scales.This design addresses the lack of local input information in approaches that encode signals into a single latent vector.Source: C. Encoder Network · p. 4
  • The encoder uses a modified residual dense network with eight residual dense blocks, each containing three 3D convolutional layers.It removes the original RDN upscaling layer, extends its 2D convolutions to 3D, and sets growth rate to 64; other SR networks without an upscaling layer can replace it.Source: C. Encoder Network · p. 4

D. Decoder Network

The decoder approximates the implicit voxel function by combining a query coordinate with a feature vector interpolated from the LR feature map. Its fully connected architecture predicts the HR voxel intensity at that coordinate.

D. Decoder Network

  • For a queried HR coordinate, the decoder's feature vector is obtained by trilinear interpolation of the LR feature map using the eight nearest feature vectors.The interpolation uses the cuboid volumes between the query and the diagonal coordinates of those feature vectors.Source: D. Decoder Network · p. 5
  • The decoder takes the query coordinate and its interpolated feature vector as input, then predicts the HR voxel intensity.Source: D. Decoder Network · pp. 4–5
  • The decoder has eight fully connected layers, each followed by ReLU, and a residual connection from its input to the output of the fourth ReLU layer.Source: D. Decoder Network · p. 5, Figure 3 · p. 5

IV. EXPERIMENTS

The experiments evaluate ArSSR across five study components. The HCP-1200 SISR evaluation compares ArSSR with interpolation and deep-learning baselines at five isotropic up-sampling scales, using one ArSSR model across those scales.

IV. EXPERIMENTS

  • The experimental section comprises five experiments intended to comprehensively evaluate ArSSR.Source: §IV · p. 5

1) The SISR Tasks on HCP-1200 Dataset:

  • The HCP-1200 test evaluates SISR at k = {2, 2.5, 3, 3.5, 4} against cubic interpolation and three deep-learning MRI SISR models.For each scale k, voxel size is reduced by k^3; the passage describes the 4× task as generating voxels 64 times smaller than before SR.Source: 1) The SISR Tasks on HCP-1200 Dataset · p. 5
  • One ArSSR model is trained for all five up-sampling scales, whereas each deep-learning baseline is trained separately for each scale.Source: 1) The SISR Tasks on HCP-1200 Dataset · p. 5

2) The SISR Tasks on Lesion Brain Dataset:

The experiments assess ArSSR on lesion and real-world healthy-brain data, use automatic segmentation for quality evaluation, and compare encoder choices. The datasets include simulated LR inputs and scans collected with different protocols.

3) The SISR Tasks on Healthy Brain Dataset:

4) Fully Automatic Segmentation based Evaluation:

  • Quality is also assessed by segmenting each GT and SR image into six regions and calculating Dice similarity between their segmentations.The evaluation uses fully automatic segmentation with ANTs, following earlier work [9],.Source: 4) Fully Automatic Segmentation based… · p. 5

5) Influence of Encoder Network:

  • The encoder study compares ResCNN, SRResNet, and RDN within ArSSR across five HCP-1200 up-sampling scales.Both qualitative and quantitative performance are reported.Source: 5) Influence of Encoder Network · p. 5

1) Datasets & Preprocessing:

  • HCP-1200 and Lesion Brain are simulated datasets, whereas Healthy Brain pairs scans collected from one subject under different protocols.HCP-1200 is skull-stripped; the other datasets are not, and no further skull processing is performed.Source: 1) Datasets & Preprocessing · p. 5

2) The SISR Tasks on Lesion Brain Dataset:

2) Evaluation Metrics:

The evaluation combines pixel-wise, perceptual, sharpness, and segmentation-based measures, while training samples voxel coordinates from paired LR-HR patches. ArSSR is optimized with L1 loss using Adam.

2) Evaluation Metrics:

  • PSNR and SSIM are supplemented by LPIPS, PSI, and LPC-SI because pixel-wise metrics may score over-smoothed results highly.LPIPS, PSI, and LPC-SI are designed for 2D images, so the study computes them slice by slice across three orthogonal directions and averages the scores.Source: 2) Evaluation Metrics · p. 6

3) Training Details:

  • Each ArSSR training step samples 15 LR-HR patch pairs and 8,000 voxel coordinates from each HR patch for the decoder.The model uses L1 loss and Adam, with a learning rate starting at 10^-4 and decaying by half every 200 epochs.Source: 3) Training Details · pp. 6–7
  • Figure 5 presents qualitative results for all compared models on one HCP-1200 test sample at k = {2, 2.5, 3, 3.5, 4}.Training runs for 2,500 epochs and takes about 12 hours on one NVIDIA TITAN RTX 24G GPU.Source: Figure 5 · p. 7, 3) Training Details · p. 7

4) Compared Methods:

On HCP-1200, ArSSR is compared with cubic interpolation and three deep-learning SISR models. ArSSR leads on perceptual and sharpness measures in most scale cases, while ResCNN leads on PSNR and SSIM.

4) Compared Methods:

  • The comparison uses cubic interpolation, SRCNN3D, DCSRN, and ResCNN as baselines for 3D MR SISR.The deep-learning baselines are trained separately for each scale.Source: 4) Compared Methods · p. 7

B. The SISR Tasks on the HCP-1200 Dataset

C. The SISR Tasks on the Lesion Brain Dataset

On lesion-brain images, ArSSR preserves clearer tissue contrasts than the compared deep-learning models as the up-sampling scale increases. Lesion boundaries remain distinguishable at larger scales, although lesion contrast does not fully match the GT.

C. The SISR Tasks on the Lesion Brain Dataset

  • At k = 3 and 4, lesion boundaries with CSF are difficult to distinguish in several baseline outputs, while ArSSR keeps GM-WM and CSF-GM contrasts clear.The comparison notes blurring and brighter CSF in the deep-learning baselines.Source: C. The SISR Tasks on the Lesion Brain… · pp. 8–9
  • GM-WM image contrast in this dataset is slightly different from that in the HCP-1200 training dataset, reportedly because of different MR scanners and image-scanning parameters.Source: C. The SISR Tasks on the Lesion Brain… · p. 8
  • For most scales, ArSSR has the best LPIPS, PSI, and LPC-SI results, but lower PSNR and SSIM.At k = 3 and 4, lesion contrast is less clear than in GT, though lesion, normal WM, and CSF boundaries remain distinguishable.Source: C. The SISR Tasks on the Lesion Brain… · p. 9

D. The SISR Tasks on the Healthy Brain Dataset

On the Healthy Brain dataset, ArSSR produces sharper, more scale-consistent anatomical details than the compared methods. In segmentation evaluation, its results are closest to the ground truth overall, with one reported white-matter exception at scale k = 4.

D. The SISR Tasks on the Healthy Brain Dataset

  • ArSSR generally achieves the best performance on PSI and LCP-SI across the three Healthy Brain up-sampling scales.Its intensity profiles show sharp changes in the CSF regions consistently across scales, unlike the flatter cubic-interpolation profiles and the varying ResCNN profiles.Source: D. The SISR Tasks on the Healthy Brain… · p. 9
  • On the real Healthy Brain images, ArSSR produces more consistent local patterns across scales, while its reconstructed white–gray matter contrast exceeds that of the low-SNR reference.The reference HR image is unavailable as a matched ground truth because it and the LR image came from independent acquisitions.Source: D. The SISR Tasks on the Healthy Brain… · p. 9

E. Fully Automatic Segmentation based Evaluation

  • Across axial, sagittal, and coronal views, ArSSR segmentation results are closest to the ground truth; Table IV reports Dice results across compared models.ResCNN slightly outperforms ArSSR for white matter at k = 4 (Dice: 0.9317 vs 0.9284), while ArSSR performs best on all segmentation regions at all up-sampling scales otherwise.Source: Table IV · p. 10, E. Fully Automatic Segmentation based… · p. 10, Figure 9 · p. 10

F. Influence of Encoder Network

The three tested encoder networks yield very close ArSSR performance, although their strongest metrics differ. The conclusion attributes the model’s arbitrary-scale reconstruction to its continuous implicit voxel function, while noting that generalization is limited by training data.

F. Influence of Encoder Network

  • The three encoder variants achieve very close SR performance, and the encoder choice does not significantly affect ArSSR overall.ResCNN scores highest on PSNR and SSIM, RDN on LPIPS and PSI, and SRResNet on LPC-SI across the tested scales.Source: F. Influence of Encoder Network · pp. 10–11, Table V · p. 11

V. CONCLUSION

  • ArSSR represents images at different resolutions with a continuous implicit voxel function rather than learning a scale-specific LR-to-HR mapping.The conclusion reports that experiments on two simulated datasets found a single model outperformed popular CNN-based methods across several scales; it also reports pleasing results on the lesion-brain dataset and finer details and sharper contrast than the low-SNR reference on real data.Source: §V · p. 11

VI. LIMITATION

  • ArSSR’s generalization is limited by its training data; trained on T1-weighted brain images, it may lose performance on other sequences or organs.The authors suggest that multi-model image data could improve generalization.Source: §VI · p. 11

Related papers

Loading 2110.14476v3…