Source-linked AI summary
NeRP: Implicit Neural Representation Learning with Prior Embedding for Sparsely Sampled Image Reconstruction
Liyue Shen, John Pauly, Lei Xing
TL;DR
Sparsely sampled medical image reconstruction is ill-posed, while conventional deep reconstruction methods require large population datasets and can face generalization and subtle-change challenges. NeRP embeds a prior image into an implicit neural representation and optimizes it against sparse measurements, with experiments reporting high-quality CT and MRI reconstructions and capture of tumor progression. The paper's scope boundary is that a suitable longitudinal prior image may not be available for natural-image applications.
Problem
Sparsely sampled reconstruction must recover images from limited measurements, while existing methods require large datasets and may be less robust across subjects, modalities, anatomical sites, or subtle structural changes.
Method
NeRP embeds a longitudinal prior image into network parameters and learns a patient-specific implicit neural representation constrained by the imaging measurements and forward physics.
Results
Experiments on 2D and 3D CT and MRI show high-quality reconstruction from sparsely sampled measurements, including accurate small structural changes and tumor progression.
Takeaways & Limitations
NeRP provides a general medical image reconstruction methodology requiring no external-subject training data and applicable across modalities, body sites, contrasts, and dimensionalities.
Takeaways & Limitations
For natural-image applications, the best prior may be unclear because a longitudinal prior image of the same object may not exist.
Abstract
from arXiv · showhide
Image reconstruction is an inverse problem that solves for a computational image based on sampled sensor measurement. Sparsely sampled image reconstruction poses addition challenges due to limited measurements. In this work, we propose an implicit Neural Representation learning methodology with Prior embedding (NeRP) to reconstruct a computational image from sparsely sampled measurements. The method differs fundamentally from previous deep learning-based image reconstruction approaches in that NeRP exploits the internal information in an image prior, and the physics of the sparsely sampled measurements to produce a representation of the unknown subject. No large-scale data is required to train the NeRP except for a prior image and sparsely sampled measurements. In addition, we demonstrate that NeRP is a general methodology that generalizes to different imaging modalities such as CT and MRI. We also show that NeRP can robustly capture the subtle yet significant image changes required for assessing tumor progression.
I. INTRODUCTION
Sparsely sampled medical image reconstruction is difficult because limited measurements make artifact-free recovery challenging, while existing deep learning methods depend on large datasets and may generalize poorly. NeRP instead learns an image's implicit neural representation using a prior image and sparse measurements, supporting multimodal reconstruction and sensitivity to structural change.
- Motivation: Sparse sampling is desirable for reducing CT radiation dose and accelerating MRI, but reconstruction requires recovering images from limited sensor measurements.Clinical workflows may require repeated scans for treatment planning, intervention guidance, and monitoring therapeutic response.
- Limitations of existing methods: Existing reconstruction networks learn mappings from measurements to images using large-scale external-subject training data.These approaches exploit transformation information embedded in population-based measurement–image datasets.
- Limitations of existing methods: Prior deep reconstruction methods can be unstable for subtle structural changes and difficult to generalize across modalities, contrasts, or anatomical sites.The cited limitations include unseen-subject robustness and challenges involving changes such as tumor growth.
- NeRP framework: NeRP embeds a subject's prior image into network parameters, then optimizes the representation using sparse sensor-domain measurements.The prior-embedded network initializes the search, while the measurement physics constrains reconstruction in projection or frequency space.
- NeRP framework: NeRP requires no external-subject training data and is designed to generalize across imaging modalities, contrasts, anatomical sites, and dimensionalities.The framework uses a continuous-function formulation of image reconstruction and incorporates longitudinal prior information through network parameters.
- Evidence: Experiments in 2D and 3D CT and MRI demonstrate effectiveness and generalizability, including robust capture of subtle changes associated with tumor progression.The authors present these results as supporting potential clinical applications.
II. RELATED WORK
Related medical reconstruction methods commonly integrate imaging physics or geometry into networks, but still learn measurement-to-image mappings. NeRP differs by learning an implicit representation of the image itself.
- Measurement-to-image learning: Deep learning reconstruction methods have used CNNs and transformation modules to map sampled sensor measurements to reconstructed medical images.Examples span MRI and CT reconstruction using measurement-to-image mappings.
- Physics and geometry priors: Physics- and geometry-integrated methods incorporate imaging-model components such as differentiable inversion or back-projection layers.These components connect sensor-domain and image-domain processing within unified reconstruction networks.
- NeRP distinction: Despite incorporating physics or geometry priors, these methods still train networks to learn mappings from measurements to images, unlike NeRP's image-representation formulation.The distinction is methodological rather than a claim that prior methods lack imaging knowledge.
B. Implicit Neural Representation Learning
Implicit neural representations model images as continuous functions encoded by neural networks, offering a basis for reconstruction that searches over network representations rather than directly over image pixels. NeRP extends this idea by embedding a prior image and enforcing sparse measurement consistency.
- Background: Neural representations encode images as continuous functions represented by network parameters, building on prior work in natural-image and scene representation.Earlier medical applications were limited, and one related method represented the measurement field rather than the image.
- Image priors: Neural networks can capture image priors through their learned parameters or architecture, motivating their use in inverse problems.The paper positions NeRP as extending image-prior use through prior embedding.
- NeRP formulation: NeRP embeds an earlier image of the same subject into network parameters, then learns the target representation under sparse measurement constraints.The method encodes the prior's spatial field and subsequently optimizes the network for the target image.
- Inverse-problem setting: Sparse reconstruction uses an imaging forward model to match predicted measurements with undersampled sensor data in an ill-posed inverse problem.The formulation applies to accelerated MRI acquisition and radiation-reduced CT acquisition.
- Inverse-problem setting: The reconstruction objective combines a measurement-consistency term with a regularizer representing generic image priors.The data term measures errors between forward-projected estimates and measurements, while regularization can encode smoothness or transform-domain sparsity.
B. Neural Representation for Image
The image representation maps normalized spatial coordinates to intensities with a coordinate-based network, encoding the full image in its parameters. Fourier feature embedding and an MLP provide the specified representation components.
- Coordinate-based representation: An implicit neural representation maps each normalized image-space coordinate to its corresponding intensity value.The network parameters encode the internal information of the entire image.
- Fourier feature embedding: The representation uses Fourier feature embedding to encode input coordinates before processing them with the coordinate-based network.The paper motivates Fourier features as effective for learning high-frequency functions.
- MLP architecture: The coordinate-based network is implemented as a deep fully connected multilayer perceptron whose structure and parameters define the continuous image function.Model depth and width are hyperparameters controlling representative capability, and periodic activations are used to represent fine details.
C. NeRP for Sparsely Sampled Image Reconstruction
NeRP reconstructs a target image by embedding a prior image into a coordinate-based MLP and optimizing its parameters against sparse measurements through a differentiable imaging forward model.
- Prior Embedding: The prior image is embedded by optimizing an MLP on coordinate-intensity pairs, encoding its internal information into the network parameters.The resulting prior-embedded network initializes the target-image network.
- Network Training: The target image is parametrized by a coordinate-based MLP initialized with the prior-embedded network.The network is trained using sparse measurements and an imaging-system forward model.
- Implicit Regularization: NeRP replaces an explicit image-space regularizer with implicit priors from the embedded prior image and neural-network parametrization.All coordinate-intensity pairs share an underlying continuous function.
- Network Training: The differentiable forward model adapts to CT or MRI, enabling end-to-end network training from the corresponding sparse measurements.CT uses a Radon transform, while MRI uses a Fourier transform.
- Image Inference: The final reconstruction is generated by evaluating the trained network at every spatial coordinate in the image grid.The inferred intensity values fill the complete reconstruction image.
D. Technical Details of NeRP
The implementation uses multilayer perceptrons with periodic activations and different widths for CT and MRI reconstruction.
- Network Architecture: CT reconstruction uses an 8-layer MLP with width 256 and periodic activations on every fully connected layer except the last.MRI reconstruction increases the MLP width to 512 layers, and Fourier feature embedding size is 256.
IV. EXPERIMENTS AND RESULTS
Experiments evaluate NeRP on sparsely sampled CT and MRI settings using modality-specific acquisition patterns, with an earlier scan supplied as a prior image.
- CT Sampling: 2D CT uses parallel-beam projections, while 3D CT uses cone-beam projections under sparse sampling.The CT experiments assume 20 projections equally distributed across a semi-circle.
- MRI Sampling: MRI experiments sample 40 radial spokes in k-space with golden-angle angular intervals.The acquisition pattern is illustrated in Fig. 1(B).
- Pancreas 4D CT: Pancreas 4D CT provides sequential 3D images across a respiratory cycle, with phase 1 used as the prior for later phases.Continuous structural changes occur across the 10 temporal phases.
- Prior Information: The reconstruction setup combines sparse measurements with an earlier scan from the same patient as a prior image.The prior supplies anatomical information while allowing structural and functional differences to remain.
A. Datasets
The experiments cover temporal pancreas CT, longitudinal head-and-neck and lung CT, and quantitative comparisons across sparse projection counts and reconstruction methods.
- Pancreas 4D CT: The pancreas 4D CT experiment reconstructs later respiratory phases from phase 1 using sparse projections.The 2D results report clear anatomy and precise capture of continuous changes across phases.
- Generalization: The head-and-neck and lung cases test generalization across anatomical sites and patients using longitudinal treatment scans.Head-and-neck reconstructions show sharper boundaries, higher-contrast bone, and fewer noise and artifacts than comparison methods.
- 2D CT Results: NeRP improves 2D CT reconstructions with increased sharpness and reduced noise, while prior embedding supports precise reconstruction from sparse sampling.The method also captures small structural changes in patient anatomy.
- 3D CT Results: The 3D pancreas experiment reconstructs phase 6 from phase 1 despite their largest respiratory structural difference.The reconstructed volume captures target-phase anatomy with high image quality and contrast.
- Method Comparisons: In the 3D pancreas comparison, FBP produces severe streaking artifacts with sparse measurements, whereas NeRP reconstructs artifact-free high-quality images.The comparison also includes GRFF and NeRP without prior embedding.
- Quantitative Results: NeRP achieves the best PSNR and SSIM performance across pancreas, head-and-neck, and lung cases with 10, 20, or 30 projections.It outperforms comparison methods without prior-image embedding.
D. Experiments on 2D MRI Image Reconstruction
NeRP reconstructs multi-contrast 2D MRI images from sparsely sampled radial k-space data by using a previous scan as a prior. The method captures tumor-region changes between examinations and generalizes across MRI contrasts.
- The experiments reconstruct 2D MRI images from sparsely sampled k-space data acquired with a radial sampling pattern.The 2D NUFFT serves as the forward model, and the prior is the first MRI exam for the same patient.
- The multi-contrast evaluation covers T1, T1c, T2, and FLAIR images from prior and target examinations.The target and prior images are compared for one randomly selected patient, with two cross-sectional cases per contrast.
- NeRP and its version without prior embedding are compared using reconstructed images and cropped tumor-region views.The figure places prior images, ground-truth targets, reconstructions, and tumor-region crops in separate rows.
- NeRP accurately captures fine structural details in tumor regions that differ from the prior image.The reported comparison emphasizes changed tumor shape and size between the two examinations.
- Prior embedding helps reconstruct high-quality second-exam images from sparse k-space data and supports application across different MRI contrasts.The prior image provides information from the same patient while the target scan contains progression-related changes.
E. Experiments on 3D MRI Image Reconstruction
The 3D MRI experiments reconstruct second-exam volumes from sparsely sampled radial k-space data using a first-exam volume as prior. Across contrasts and sampling densities, NeRP improves reconstruction quality and captures brain-tumor changes.
- The evaluation covers T1, T1c, T2, and FLAIR contrasts with 30, 40, and 50 sampled radial spokes.Performance is reported using PSNR and SSIM.
- NeRP achieves better PSNR and SSIM performance than methods without prior images across all evaluated image contrasts.The comparison includes different reconstruction methods and sampling levels.
- NeRP reconstructs precise changes in the brain-tumor region despite sparsely sampled k-space data.The FLAIR comparison uses cross-sectional slices and red-boxed tumor regions against the prior and ground truth.
- Compared with alternative and ablated reconstructions, NeRP produces more accurate tumor structures, sharper anatomic boundaries, and higher soft-tissue contrast.The figure compares NeRP with Adjoint NUFFT, GRFF, and NeRP without prior embedding.
F. Analysis of Network Structure
NeRP addresses sparse medical image reconstruction by using prior-embedded neural representations rather than population-trained reconstruction mappings. Its analyses cover network structure, sampling density, modality generalization, fine structural changes, and practical training considerations.
- Sampling analysis: Increasing sampled projections or radial spokes improves reconstructed CT and MRI image quality, with CT performance reaching a plateau around 20 projections.The analysis evaluates PSNR and SSIM as sampling increases across both modalities.
- Network structure and training: NeRP learns a patient-specific coordinate-intensity mapping, avoiding the large paired datasets required by conventional deep reconstruction mappings.The method trains on data from the same sample and does not require external-subject datasets.
- Structural fidelity: Prior embedding and implicit network priors help NeRP reconstruct fine structural details, including small tumor or lesion changes.These properties are presented as supporting robustness and reliability in reconstructed images.
- Generalization: NeRP is designed to generalize across imaging modalities, contrasts, anatomical sites, patients, and image dimensionalities.The paper contrasts this scope with difficulties transferring networks between MRI and CT because their measurement fields differ.
- Practical training: Patient-specific retraining takes only minutes in the reported experiments using a single Nvidia V100 GPU.This efficiency is linked to the absence of abundant training data requirements.
VI. CONCLUSION
The paper proposes NeRP, which incorporates prior knowledge into implicit neural representations for sparse CT and MRI reconstruction. Experiments report high-quality reconstructions without external training data, including detailed anatomical and tumor-progression changes across diverse settings.
- VI. CONCLUSION: NeRP reconstructs target medical images by learning implicit neural representations with prior embedding.The method efficiently incorporates prior knowledge during reconstruction.
- VI. CONCLUSION: Experiments on 2D and 3D MRI and CT show high-quality reconstruction from sparsely sampled measurements.The conclusion reports results across both modalities and dimensionalities.
- VI. CONCLUSION: NeRP requires no external-subject training data and can reconstruct small anatomical changes, including tumor progression.The conclusion lists these as principal advantages of the approach.
- VI. CONCLUSION: The method applies across different body sites, imaging modalities, and patients, while longitudinal scans provide useful prior knowledge.This scope is tied to repeated clinical imaging for diagnosis, treatment planning, and monitoring.