Source-linked AI summary
A Review on Deep Learning in Medical Image Reconstruction
Haimiao Zhang, Bin Dong
TL;DR
Medical image reconstruction seeks high-quality images from difficult inverse problems while balancing data fidelity, prior knowledge, and clinical constraints. This review surveys handcrafted, hybrid, and deep approaches through the unrolling-dynamics viewpoint, which derives trainable networks from optimization algorithms and differential-equation schemes. It highlights interpretable model design alongside reported advances and continuing challenges in data availability and reasoning reliability.
Problem
Medical image reconstruction is a large-scale, ill-posed inverse problem, while deep models face limited medical data and can hide their reasoning.
Method
The review analyzes deep medical-image-reconstruction models through unrolling dynamics, combining handcrafted modeling with trainable components derived from optimization algorithms and numerical dynamics.
Results
Unrolling-based designs provide a general balance between model interpretability and efficacy, while reviewed architectures include models reporting improved accuracy or state-of-the-art reconstruction results.
Takeaways & Limitations
Optimization algorithms and numerical differential equations offer guiding principles for designing neural architectures that retain domain structure and interpretability.
Takeaways & Limitations
Deep models remain constrained by scarce task-specific medical data, unequal rare-case importance, and reasoning procedures that are often hidden.
Abstract
from arXiv · showhide
Medical imaging is crucial in modern clinics to guide the diagnosis and treatment of diseases. Medical image reconstruction is one of the most fundamental and important components of medical imaging, whose major objective is to acquire high-quality medical images for clinical usage at minimal cost and risk to the patients. Mathematical models in medical image reconstruction or, more generally, image restoration in computer vision, have been playing a prominent role. Earlier mathematical models are mostly designed by human knowledge or hypothesis on the image to be reconstructed, and we shall call these models handcrafted models. Later, handcrafted plus data-driven modeling started to emerge which still mostly relies on human designs, while part of the model is learned from the observed data. More recently, as more data and computation resources are made available, deep learning based models (or deep models) pushed data-driven modeling to the extreme where the models are mostly based on learning with minimal human designs. Both handcrafted and data-driven modeling have their own advantages and disadvantages. One of the major research trends in medical imaging is to combine handcrafted modeling with deep modeling so that we can enjoy benefits from both approaches. The major part of this article is to provide a conceptual review of some recent works on deep modeling from the unrolling dynamics viewpoint. This viewpoint stimulates new designs of neural network architectures with inspiration from optimization algorithms and numerical differential equations. Given the popularity of deep modeling, there are still vast remaining challenges in the field, as well as opportunities which we shall discuss at the end of this article.
1 Introduction
Medical image reconstruction is an ill-posed inverse problem spanning multiple acquisition operators and requiring models that balance data consistency with image priors. The review traces handcrafted, hybrid, and deep models, emphasizing unrolling dynamics as an interpretable bridge between optimization and learning.
- 1 Introduction: Medical reconstruction models infer an unknown image u from measured data f through an acquisition operator A, with noise modeled according to the imaging modality.A may represent identity, convolution, restriction, sub-sampled Fourier, sub-sampled Radon, or dipole-kernel operations.
- 1 Introduction: The inverse problem is difficult because practical reconstruction is large-scale and ill-posed.
- 1 Introduction: Optimization formulations combine a data-fidelity term measuring consistency with measurements and a regularizer encoding prior knowledge, balanced by λ.The data-fidelity form depends on noise statistics, while regularization is often central to the model.
- 1 Introduction: Image-reconstruction modeling progressed from handcrafted designs to hybrid handcrafted-plus-data-driven models and then predominantly learned deep models.Hybrid models exploit data while retaining aspects of interpretability and theoretical structure.
- 1 Introduction: Medical reconstruction differs from general image restoration because meaningful quality metrics should be clinically relevant and task dependent, while most medical images are 3D and computationally challenging.
- 1 Introduction: Unrolling dynamics converts iterative optimization algorithms such as ISTA into trainable feed-forward networks, combining domain structure with end-to-end parameter learning.The approach can improve interpretability by inheriting structure from the original dynamics.
2 Review of Deep Neural Networks
The review introduces widely used deep neural network architectures in medical imaging, including ResNet, autoencoders, and U-Net. These architectures use distinct forms of residual, encoder-decoder, and skip-connection structure for representation learning and image-processing tasks.
- 2 Review of Deep Neural Networks: Deep neural networks use composite nonlinearities and end-to-end training to extract task-suitable features across medical imaging applications.Applications include reconstruction, segmentation, detection, super-resolution, and classification.
- 2.1 ResNet: ResNet uses nonlinear residual blocks and skip connections, which facilitate stable training of very deep networks.Related skip-connection architectures include TRD, DenseNet, and U-Net.
- 2.2 Autoencoder: Autoencoders learn an encoder-decoder representation that reconstructs data after encoding, with parameters optimized to reduce discrepancies between inputs and reconstructions.
- 2.3 U-Net: U-Net resembles an autoencoder but adds skip connections between its encoder and decoder, supporting biomedical image segmentation.The architecture was also adapted for volumetric segmentation and sparse-view CT artifact reduction.
3 Interpretations of Deep Neural Networks
The paper interprets deep neural networks through representation learning and differential-equation perspectives, linking architecture expressivity and principled design. Approximation theory highlights depth advantages, while ODE-based views motivate architectures such as LM-structure with comparable accuracy at reduced depth.
- 3.1 Representation Learning Perspective: Representation quality is characterized by how efficiently neural networks approximate functions, with depth and width determining approximation power.A good representation uses fewer atoms for a target error, and neural networks provide flexible approximations under suitable conditions.
- 3.1 Representation Learning Perspective: Depth can reduce the number of neurons needed for a given approximation accuracy, although ReLU networks still require a minimum width.Deep networks can achieve exponential or polynomial reductions in neuron count, while overly narrow networks may fail to approximate continuous functions even with infinite depth.
- 3.2 Differential Equation and Control Perspective: ODE discretizations provide a bridge from numerical differential equations to deep architectures, including ResNet, PolyNet, FractalNet, and RevNet.ResNet corresponds to forward Euler discretization, while other bypass architectures correspond to different temporal discretization schemes.
- 3.2.1 Numerical difference equations and architecture design: The LM-structure uses a trainable linear multistep scheme to design ResNet-like networks from numerical ODE methods.Setting all γk to zero recovers ResNet; the construction connects architecture design to established numerical schemes.
- 3.2.1 Numerical difference equations and architecture design: LM-structure networks reduce ResNet-like depth and parameter count by 50–90% without hampering accuracies, while also boosting classification accuracies on CIFAR and ImageNet.The authors attribute faster information propagation to weighting the second-order dynamics term more strongly than in ResNet.
4 Deep Models in Medical Image Reconstruction
Deep models for medical image reconstruction range from post-processing and raw-to-image mappings to optimization algorithms unrolled into trainable networks. The reviewed designs combine learned components with imaging-domain structure, including task-oriented reconstruction and multi-domain consistency.
- Unrolling dynamics: Unrolling optimization algorithms produces interpretable deep models with fewer parameters and a principled way to combine domain knowledge with learned components.The review identifies these as three advantages over regular deep models, including suitability for small-sample learning.
- Raw-to-Task: Task-based reconstruction links an unrolled reconstruction network with an image-analysis DNN so reconstructed images are evaluated for downstream analysis.This extends the pipeline toward joint reconstruction, analysis, and decision making rather than relying only on generic image metrics.
- Post-Processing: Post-processing CNNs remove noise and artifacts from FBP images, while U-Net learns streak artifacts for sparse-view CT and subtracts them from degraded inputs.The reviewed examples target noise from low tube currents and streaking caused by incomplete measurements.
- ADMM-Net: Generic-ADMM-Net achieved state-of-the-art MR image reconstruction with a significant margin over the BM3D-based algorithm.
- PD-Net: PD-Net approximates PDHG proximal operators with neural networks and significantly outperformed FBP and some handcrafted reconstruction models.
- JSR-Net: JSR-Net unrolls an ADMM-based alternative optimization algorithm and uses learned proximal operators to exploit multi-domain image features in sparse-view and limited-angle CT.
5 Challenge and Opportunities
Deep modeling in medical imaging remains constrained by limited labeled and task-specific data, multimodal clinical information, and hidden reasoning processes. These challenges motivate learning approaches that use unlabeled data, account for unequal sample importance, incorporate diverse sources, and expose reasoning.
- Data limitations: Limited labeled data, costly expert annotation, and small task-specific datasets constrain the development of deep models in medical imaging.Rare cases may be especially valuable despite being scarce.
- Data limitations: Deep models should make effective use of relatively abundant unlabeled data alongside limited labeled data.The paper frames this as an open learning-model design question.
- Data limitations: Learning models and data augmentation should extract knowledge from limited samples while acknowledging the unequal importance of rare and common cases.The paper identifies rare cases as potentially more valuable than common cases.
- Clinical information: Because clinical decisions use patient information and physician knowledge beyond images, incorporating multiple diverse data sources may improve system performance.
- Reasoning and interpretability: Hidden reasoning procedures can make accurate predictions unreliable when models reach conclusions through incorrect reasoning.The paper proposes combining deep modeling with causal inference or medical knowledge graphs as an open direction.