Source-linked AI summary
Computed Tomography Reconstruction Using Deep Image Prior and Learned Reconstruction Methods
Daniel Otero Baguer, Johannes Leuschner, Maximilian Schmidt
TL;DR
The paper addresses CT reconstruction when paired training data are scarce, a setting in which learned methods may lack generalization and classical guarantees. It benchmarks learned and data-free approaches, then combines Deep Image Prior with classical regularization or learned reconstructions. The learned primal-dual is highly data-efficient, while DIP-based combinations improve low-data reconstruction results under specific dataset conditions.
Problem
Medical-imaging inverse problems require difficult-to-obtain ground truths, while end-to-end learned methods may generalize poorly with insufficient data and lack classical guarantees.
Method
The paper benchmarks learned methods across data sizes and combines Deep Image Prior with classical regularization or an initial reconstruction from a learned method.
Results
The learned primal-dual achieved the best performance and data efficiency, while DIP+TV was best among data-free methods and DIP combinations improved selected low-data results.
Takeaways & Limitations
DIP-based reconstruction can improve low-data CT results while addressing data-consistency concerns in learned reconstructions.
Takeaways & Limitations
The original DIP requires early stopping to avoid fitting noise, but choosing the stopping time is unclear and was selected using validation data.
Abstract
from arXiv · showhide
In this work, we investigate the application of deep learning methods for computed tomography in the context of having a low-data regime. As motivation, we review some of the existing approaches and obtain quantitative results after training them with different amounts of data. We find that the learned primal-dual has an outstanding performance in terms of reconstruction quality and data efficiency. However, in general, end-to-end learned methods have two issues: a) lack of classical guarantees in inverse problems and b) lack of generalization when not trained with enough data. To overcome these issues, we bring in the deep image prior approach in combination with classical regularization. The proposed methods improve the state-of-the-art results in the low data-regime.
1. Introduction
Deep learning can achieve high-quality reconstructions for ill-posed inverse problems, but medical imaging’s low-data setting makes generalization difficult. This work therefore explores Deep Image Prior alongside other methods for low-data CT reconstruction.
- Motivation: Deep learning methods achieve state-of-the-art quantitative reconstruction quality but generally require large paired datasets to generalize well.Obtaining ground-truth data is especially challenging in medical imaging.
- Motivation: CT ground truths can require exposing patients to high doses of X-ray radiation.
- Deep Image Prior: Deep Image Prior uses a neural network without training its weights, distinguishing it from learned methods.Its applicability has been questioned because reconstruction can be time-consuming.
- Contributions: The paper combines DIP with other deep learning methods to study CT reconstruction in a rather low-data regime.It introduces DIP with classical regularization, adjusts learned reconstructions, and benchmarks methods across dataset sizes.
2. Computed Tomography
CT reconstructs internal body structure from X-ray measurements modeled by the Radon transform. Reducing radiation through lower intensity or fewer angles creates reconstruction challenges because the inverse problem is ill-posed.
- CT motivation: CT non-invasively acquires images inside the human body using X-rays, while current research seeks to reduce patient radiation exposure.
- CT motivation: Lower-intensity measurements and fewer projection angles reduce radiation but introduce particular reconstruction challenges.
- Forward model: The forward operator is the 2D Radon transform, which models X-ray attenuation through the body.
- Forward model: The Radon transform integrates along lines parameterized by distance s and angle ϕ, producing a sinogram from all parameter pairs.
- Inverse problem: Because the Radon transform is linear and compact, reconstructing an image from the sinogram is an ill-posed inverse problem.
3. Related approaches and motivation
Inverse-problem methods combine data fidelity with regularization, while data-driven approaches learn reconstruction operators from examples. Their success motivates comparison with classical regularizers in low-data settings.
- Inverse problems: Inverse problems recover an unknown quantity from indirect, often noisy measurements using an operator between spaces X and Y.
- Classical methods: Classical approaches include filter-based linear pseudoinverses and nonlinear variationally regularized inverses.
- Classical methods: Variational methods use a data discrepancy and regularizer, with examples including the ℓ2 norm, ℓ1 norm, and Total Variation.The regularization parameter α requires careful selection, potentially using validation data and PSNR or SSIM.
- Data-driven methods: Data-driven regularized inverses have achieved strong reconstruction quality and include end-to-end learned methods, learned regularizers, and generative networks.This study focuses on end-to-end learned methods.
3.1. End-to-end learned methods
End-to-end learned CT methods range from post-processing and direct inversion to unrolled iterative schemes. They can improve reconstruction quality, but learned operators may fail to enforce data consistency, motivating null-space and hybrid approaches.
- Method classes: The benchmark reviews end-to-end learned methods, including learned post-processing, direct inversion, and learned iterative schemes.
- Learned post-processing: FBP+CNN methods improve filtered back-projection reconstructions from noisy or sparse measurements through learned post-processing.The network parameters are learned from data.
- Direct inversion: Direct inversion methods map sinograms to reconstructions, but fully connected layers can require over 67 · 10^9 parameters for the LoDoPaB-CT dimensions.iRadonMap reduces this layer’s parameter count to approximately 130 · 10^6 and adds a U-Net post-processing network.
- Learned iterative schemes: Learned iterative schemes unroll a small number of optimization iterations and replace selected operators with CNNs trained on ground-truth and observation pairs.The reconstruction uses a fixed small iteration count, such as L = 10.
- Limitation: After training, learned reconstructions do not enforce data consistency, so observations may be ignored and medically important image features may be lost.
- Null Space Networks: Null Space Networks preserve the forward operator’s output and can provide data consistency when combined with a pseudoinverse.The paper notes that theoretical results exist for this approach but excludes it from the comparison.
3.2. Deep Image Prior
Deep Image Prior reconstructs CT images by optimizing a randomly initialized deep network driven by fixed noise, using the network reparameterization as an implicit prior. Its behavior depends on architectural choices and early stopping, which together shape the reconstruction and prevent noise fitting.
- Deep Image Prior: DIP reparameterizes the image as x = ϕ(θ, z), replacing an explicit regularization term with a deep generative network driven by fixed input z.The network weights θ are optimized while the architecture provides the implicit regularization.
- Deep Image Prior: The original DIP uses gradient descent with early stopping to avoid reproducing noise, combining stopping with architectural regularization.Because the network is overparameterized, it can eventually reproduce noise; stopping is selected using validation performance in PSNR.
- Deep Image Prior: Convolutional image generators fit natural images faster than noise and construct them from low to high frequencies during optimization.This implicit structural bias motivates the observed global-to-local reconstruction behavior.
- Related work: DIP has been combined with explicit approaches such as Regularization by Denoising and applied to denoising, inpainting, super-resolution, compressed sensing, PET, and MRI.These extensions position DIP as a component in broader unsupervised recovery and imaging methods.
- Deep Image Prior: Figure 2 depicts a randomly initialized U-Net-like network fed fixed Gaussian noise while gradient descent minimizes the output's data discrepancy.The experiments use 128 channels per layer, with either 4 or 0 concatenated skip channels.
- Related work: Architecture and hyper-parameter selection are crucial: depth, channel count, and skip connections affect representation power and regularization strength.For the Ellipses dataset, the U-Net without skip connections and depth 5 appeared to perform best; hyper-parameter optimization was the most time-consuming part.
4. Deep Image Prior and classical regularization
The paper combines the Deep Image Prior with classical regularization to recover inverse-problem guarantees while controlling optimization through regularization rather than early stopping.
- The DIP is analyzed with a classical regularization term J, such as total variation, to obtain existence, convergence, and convergence-rate guarantees.
- Regularization removes the need to select an optimal stopping iteration, but introduces the classical problem of choosing α according to the noise level.Higher noise calls for a higher α or fewer iterations for optimal results.
- A closed network range, together with a J-minimizing solution, supports classical well-posedness, stability, convergence, and convergence-rate theorems.The analysis restricts the operator domain to the network range and the regularizer domain.
- The network parameterization uses a fixed input and composes affine mappings with activation functions across multiple layers.The mapping is written as a composition of layer operators and activations.
- The network range is closed when the parameter set is compact and the activation functions are continuous, monotone, and bounded.The proof establishes compactness layer by layer and then uses induction through the network.
- The implementation bounds network weights and empirically checks that a J-minimizing solution exists; sigmoid in the last layer performed better than ReLU in experiments.Weight bounds can be enforced by clipping after gradient updates, while closed-range activations offer an alternative condition.
5. Deep Image Prior with initial reconstruction
The proposed method initializes a deep-neural parameterization from an end-to-end reconstruction and optimizes it to enforce data consistency while retaining the network’s natural-image bias.
- The method takes any end-to-end learned reconstruction as its initialization and further enforces data consistency through optimization over the network parameters.
- The deep-neural parameterization restricts reconstructions to a network range that has similarities to wavelet and Fourier representations.For image processing, these representation domains are described as more convenient than the classical pixel representation.
- The initial reconstruction is represented by network parameters θ0, which then serve as the starting point for gradient-descent optimization.
- Unlike direct projection onto the data-consistency set, the proposed procedure is intended to avoid the artifacts that ill-posedness can introduce.It instead optimizes from the parameterization of the initial reconstruction while preserving a structural bias toward natural images.
- Optimization stops when the data discrepancy reaches δ or after a fixed iteration count K selected using validation data.
6. Benchmark setup and results
The benchmark evaluates classical, learned, and DIP-based CT reconstruction methods across training-data sizes and datasets. The learned primal-dual is highly data-efficient, while DIP combinations improve reconstruction quality in low-data settings with dataset-dependent gains.
- Benchmark setup: The benchmark trains end-to-end methods at different dataset sizes and compares them with FBP, TV regularization, and proposed methods.Testing uses 100 samples; the Ellipses and LoDoPaB datasets represent sparse-angle and low-dose CT settings, respectively.
- End-to-end learned methods: The learned primal-dual achieved the best performance, outperforming TV with 0.1 % (32 data-pairs) on Ellipses and 0.01 % (4 data-pairs from the same patient) on LoDoPaB.iRadonMap required much data, while FBP+UNet outperformed TV with 10 % of Ellipses data and 0.1 % of LoDoPaB data.
- Data-free methods: DIP+TV achieved the best results among data-free methods, outperforming TV by 1 dB on Ellipses and 2 dB on LoDoPaB on average.It also performed better than all end-to-end learned methods at the first two smaller data-sizes.
- Data-free methods: DIP+TV produced more realistic edges than TV, which tended to create flat regions and strong staircase effects.The comparison is illustrated through FBP, TV, and DIP+TV reconstructions.
- Proposed combinations: Combining DIP with the learned primal-dual improved Ellipses reconstruction quality by up to 1 dB on average, but produced no notable LoDoPaB improvement.For dataset sizes above 2 %, it yielded no significant change; on smaller LoDoPaB sizes, it was as good as DIP+TV.
- Proposed combinations: Using an initial reconstruction improved data consistency and PSNR/SSIM while reducing runtime from 6 minutes to 3 minutes per reconstruction.On Ellipses, the method used 4000 identity-operator iterations and 1000 Radon-transform iterations, compared with 8000 DIP+TV iterations.
7. Conclusions
The conclusions synthesize classical regularization, deep-neural parameterization, and learned reconstruction for CT in low-data regimes. The proposed combinations address data-consistency and theoretical-guarantee concerns, while the results provide baselines for future work.
- Conclusions: The study combines classical regularization, deep-neural parameterization, and deep learning approaches for CT reconstruction in low-data regimes.It benchmarks their behavior across the investigated methods and data amounts.
- Conclusions: DIP+TV performed best among data-free approaches, but it was considerably slow and did not benefit from having a small dataset.The learned primal-dual was highly data-efficient but lacked data consistency when trained with insufficient data.
- Conclusions: The proposed combination adjusted learned primal-dual reconstructions to match observed data without introducing artifacts through classical regularization and DIP.The paper also derived conditions for theoretical guarantees and showed how to obtain them.
- Conclusions: The reported results provide baselines for future comparisons, and the proposed methods could be applied to other imaging modalities.These implications are stated as potential uses of the paper's results and methods.
Appendix A. More results
Appendix A presents reconstructions from all analyzed methods on test samples from the LoDoPaB dataset.
- Appendix A. More results: Figure A1 compares reconstructions produced by all analyzed methods for LoDoPaB test samples.The passage identifies the dataset and the scope of the method comparison.
Appendix B. Training details
Appendix B documents the training and validation-pair counts used for the Ellipses and LoDoPaB benchmark datasets.
- Appendix B. Training details: Table B1 lists the training and validation pairs from the Ellipses dataset used in the Section 6 benchmark.The passage identifies the table's dataset and purpose but does not provide the counts themselves.
- Appendix B. Training details: Table B2 lists the training and validation pairs from LoDoPaB and gives patient counts for the final two rows.The table is used to document benchmark training details.