PaperlayerView exact revision on arXiv

Source-linked AI summary

On hallucinations in tomographic image reconstruction

Sayantan Bhadra, Varun A. Kelkar, Frank J. Brooks, Mark A. Anastasio

arXiv:2012.00646v3eess.IVcs.LG

TL;DR

Tomographic reconstruction relies on priors because the inverse problem is ill-posed, but inaccurate learned priors can produce hallucinated structures, especially outside the training distribution. The paper decomposes estimates into generalized measurement and null components, defines hallucination maps for linear imaging systems, and uses numerical studies to examine their utility. The framework isolates prior-related errors, while its usefulness is greatest when the generalized null component is substantial.

  • Problem

    Formal definitions for hallucinations in tomographic reconstruction were lacking despite concerns that inaccurate or poorly generalized priors can introduce false structures.

  • Method

    The paper decomposes reconstructed estimates into generalized measurement and null components and introduces measurement- and null-space hallucination maps for linear imaging systems.

  • Results

    Numerical studies with simulated undersampled measurements from a stylized single-coil MRI system showed that null-space hallucination maps can be particularly useful for assessing data-driven regularization with out-of-distribution data.

  • Takeaways & Limitations

    The framework isolates hallucinations attributable to reconstruction priors and can reveal structured artifacts caused by distribution shifts in data-driven methods.

  • Takeaways & Limitations

    The framework is most useful when the generalized null component is significant; when it is small, ordinary error maps may be sufficient.

Abstract

from arXiv · show

Tomographic image reconstruction is generally an ill-posed linear inverse problem. Such ill-posed inverse problems are typically regularized using prior knowledge of the sought-after object property. Recently, deep neural networks have been actively investigated for regularizing image reconstruction problems by learning a prior for the object properties from training images. However, an analysis of the prior information learned by these deep networks and their ability to generalize to data that may lie outside the training distribution is still being explored. An inaccurate prior might lead to false structures being hallucinated in the reconstructed image and that is a cause for serious concern in medical imaging. In this work, we propose to illustrate the effect of the prior imposed by a reconstruction method by decomposing the image estimate into generalized measurement and null components. The concept of a hallucination map is introduced for the general purpose of understanding the effect of the prior in regularized reconstruction methods. Numerical studies are conducted corresponding to a stylized tomographic imaging modality. The behavior of different reconstruction methods under the proposed formalism is discussed with the help of the numerical studies.

I. INTRODUCTION

Tomographic reconstruction estimates finite-dimensional objects from discrete measurements in an ill-posed inverse problem, motivating regularization with hand-crafted or learned priors. The paper focuses on formalizing hallucinations caused by inaccurate priors and introducing hallucination maps to analyze reconstruction behavior.

  • Tomographic reconstruction estimates a finite-dimensional object from discrete measurements, often using few measurements while preserving diagnostic quality.
  • Ill-posedness and measurement incompleteness require regularization, including sparsity-promoting penalties and learned priors.
  • Poor generalization or instability in deep learning reconstruction can introduce false structures absent from the imaged object and potentially affect medical diagnosis.
  • Hallucinations in general inverse problems are not necessarily high-frequency, and formal definitions for tomographic reconstruction have been lacking.
  • The study formalizes hallucinations for general linear imaging systems and introduces task-informed hallucination maps to examine reconstruction methods.
  • The paper reviews linear operator models in which continuous objects, measurement data, noise, and discrete approximations define the reconstruction setting.

B. Generalized measurement and null components

The paper decomposes coefficient estimates into generalized measurement and null components using a truncated pseudoinverse and the singular structure of the imaging operator. The measurement component is stably recoverable from data, whereas the null component requires prior information or regularization.

  • Singular-value decomposition characterizes stability, with small trailing singular values making pseudoinverse estimates sensitive to noise.
  • A truncated pseudoinverse retains stably estimable components while excluding directions whose singular values fall below the chosen tolerance.
  • The coefficient vector is uniquely decomposed into generalized measurement and generalized null components using orthogonal projection operators.
  • The generalized null space reduces to the true null space when the retained rank reaches R and remaining singular values are zero.
  • True-null-space vectors produce zero measurement data and are therefore invisible to the imaging system.
  • The measurement component can be stably estimated from data, while the null component cannot and requires priors or regularization.

C. Regularization in tomographic image reconstruction

Regularization incorporates prior knowledge by constraining reconstructions toward plausible objects while maintaining agreement with measurements. Deep learning extends this approach by learning regularizers or priors from training data, but generalization and stability remain concerns.

  • Bayesian reconstruction treats the object, measurements, and noise probabilistically and estimates the object from a prior and noise model.
  • Penalized reconstruction balances measurement fidelity against consistency with an assumed prior through a regularization weight.
  • Regularization can restrict solutions to a parameterized subset and represent reconstruction as a mapping from measurements into that subset.
  • Deep learning methods learn regularizers or priors from training data, with the learned prior shaped by the training distribution and network topology.
  • Learning-based reconstruction methods may generalize poorly outside the training distribution and may be unstable to imperceptible measurement perturbations.

III. DEFINITION OF HALLUCINATION MAPS

The paper defines measurement- and null-space hallucination maps to isolate reconstruction artifacts that cannot be stably recovered from measurements and may arise from the imposed prior. The measurement-space map compares a reconstruction’s measurement component with a stable truncated-pseudoinverse estimate.

  • Error maps combine broad reconstruction deviations, whereas hallucination maps target false structures that cannot be stably reconstructed from measurements.
  • Measurement- and null-space hallucination maps are defined for reconstruction methods that invert a linear imaging model.
  • The maps isolate artifacts that cannot be stably reconstructed from measurement data and are attributable to the reconstruction prior.
  • The estimated measurement component should be close to the truncated-pseudoinverse solution, but regularization can create discrepancies in that component.
  • The measurement-space hallucination map requires no true-object knowledge and reveals errors relative to the stably computed estimate.
  • The definitions can be translated from coefficient space to object space using a generic basis and the corresponding approximate object representation.

B. Hallucination map in the generalized null space

The generalized null-space hallucination map measures errors in the reconstructed null component relative to the true null component, isolating effects attributable to the imposed prior. Its interpretation requires the true object’s generalized null component, unlike the measurement-space map.

  • The definition compares the estimated null component ˆθnull = Pnull ˆθ with the true generalized null vector θnull.
  • The hallucination definition is constructed so the truncated pseudoinverse solution has no null-space hallucinations because it imposes no prior.
  • Computing a generalized null-space hallucination map requires full knowledge of the true object’s generalized null component.
  • Measurement-space hallucinations can be influenced by differing noise propagation and therefore may not solely quantify prior-induced errors.
  • Null-space analysis is critical for constrained reconstruction methods that suppress measurement-space hallucinations through data consistency or null-space procedures.
  • The truncated pseudoinverse has zero hallucination by definition but may still contain artifacts because it ignores the true null component.

C. Specific hallucination maps

Specific hallucination maps process generalized hallucination maps to retain structures or textures relevant to a defined task. Their transformation and observer design are application-dependent.

  • The proposed hallucination maps do not incorporate task-specific information before this transformation is applied.
  • Algorithm 1 computes the truncated pseudoinverse, generalized components, measurement- and null-space hallucination maps, and the specific map.
  • The complete procedure obtains the specific hallucination map by applying T to the null-space hallucination map.
  • Specific hallucination maps apply a transformation T to localize potentially task-relevant features or textures while suppressing others.
  • The transformation T should reflect the specified task and the observer who will perform it.

IV. NUMERICAL STUDIES

Numerical studies used hallucination maps to compare data-driven and model-based reconstruction methods under different conditions. The preliminary analyses focused on null-space hallucination maps but could also use measurement-space maps.

  • The studies compared data-driven and model-based image reconstruction methods under different conditions.
  • The preliminary analyses focused on null-space hallucination maps, although the same analyses could be repeated with measurement-space maps.

A. Stylized imaging system

The numerical setup used a stylized two-dimensional single-coil MR system and compared U-Net, PLS-TV, and DIP reconstructions on in-distribution and out-of-distribution images. The hallucination-map computation was explicitly preliminary and illustrative.

  • A. Stylized imaging system: The simulation used a stylized 2D single-coil MR system not intended to accurately model a real-world MR imager.
  • B. Reconstruction methods: The U-Net learned a mapping from an undersampling-artifact initial estimate to an estimate of the true object using pseudoinverse-based inputs.
  • B. Reconstruction methods: PLS-TV solved a least-squares problem with a total-variation penalty, whereas DIP constrained estimates to an untrained network range while fitting the measurements.
  • C. Training, validation and test data: U-Net training used 2500 adult brain MRI images with 500 validation images, while testing used 69 in-distribution and 69 out-of-distribution images.
  • C. Training, validation and test data: Out-of-distribution images came from pediatric epilepsy-resection MRI data and differed from in-distribution images in object type and MR system.
  • D. Computation of hallucination maps: Specific null-space hallucination maps were designed to localize coherent structures rather than random errors, with small regions removed during processing.
  • D. Computation of hallucination maps: The specific-map procedure was a simplistic preliminary example and was not presented as optimal.
  • D. Computation of hallucination maps: Conventional error maps were computed as reconstructed-estimate minus true-object differences, then transformed to form specific error maps.

V. RESULTS

The results distinguish reconstruction errors caused by inaccurate priors from other error sources using null space hallucination maps. Across IND and OOD studies, these maps expose method-dependent false structures and show that distribution shift can weaken U-Net performance relative to model-based alternatives.

  • A. Differences between error and hallucination maps: Null space hallucination maps isolate errors caused by the imposed prior, whereas error maps also include measurement noise and model error.The two map types can therefore identify different regions in reconstructed images.
  • A. Differences between error and hallucination maps: For IND data, U-Net produced lower hallucinations than PLS-TV and DIP in regions where it recovered fine structures more faithfully.PLS-TV and DIP oversmoothed those structures, producing higher hallucinations.
  • A. Differences between error and hallucination maps: Under OOD shift, U-Net produced false structures attributable to inaccurate null components and limited generalization from its training distribution.Its null space hallucination map became comparable to those of PLS-TV and DIP.
  • A. Differences between error and hallucination maps: OOD error-map regions differed from hallucination-map regions, so error maps alone could not localize hallucinations caused by the imposed prior.The corresponding centroid distributions showed higher variance for error maps than for hallucination maps.
  • C. Bias maps and hallucinations: Bias maps can retain artifacts from measurement noise while hallucination maps isolate prior-related effects, so the two map types provide different information.The paper illustrates this distinction using phase noise and additive Gaussian noise in simulated MRI data.
  • B. Investigation of structured hallucinations: For IND data, U-Net had higher median SSIM in structured hallucination regions, whereas for OOD data DIP had the highest median SSIM.The OOD reversal accompanied increased U-Net null space hallucinations and lower SSIM than DIP in those regions.

VI. SUMMARY AND CONCLUSION

The study uses hallucination maps to analyze how reconstruction priors affect image estimates, especially for data-driven methods and out-of-distribution data. Numerical studies show that null space hallucination maps isolate prior-induced errors, while practical use depends on feasible projection computations and meaningful null components.

  • Deep learning reconstruction methods raise concerns about learned-prior generalization and inaccurate priors introducing hallucinated structures.The paper frames these concerns in the context of data-driven regularization and out-of-distribution data.
  • The framework defines measurement-space and null-space hallucination maps to isolate errors associated with reconstructable and prior-determined components.The null space map compares the reconstructed null component with the true object null component, while the measurement map analyzes the measurement-space component.
  • Numerical studies found null space hallucination maps particularly useful for assessing data-driven regularization with out-of-distribution data.The studies used simulated undersampled measurements from a stylized single-coil MRI system and included data-driven and non-data-driven methods.
  • Structured hallucinations caused by distribution shifts in data-driven methods may lead to significant reconstruction artifacts.This result concerns the behavior observed for out-of-distribution data in the numerical studies.
  • The framework applies to linear imaging systems and reconstruction methods when projection operators can be computed, but its usefulness depends on a significant generalized null component.Large-scale projection computation may require iterative or randomized SVD alternatives, and weak null components reduce the need for strong regularization.
  • Future work includes deriving objective figures-of-merit and estimating hallucination probabilities from ensembles of hallucination maps.

S. I. SAMPLING MASK

The supplementary methods describe the simulated imaging model and three reconstruction approaches: PLS-TV, image-domain U-Net learning, and DIP-TV. These methods use different forms of regularization, including total variation, learned image priors, and a deep image prior.

  • The reconstruction methods are based on an imaging model with observed measurements, a system matrix, and iid Gaussian noise.
  • PLS-TV reconstructs images by solving a penalized least-squares problem with a total-variation penalty.The regularization parameter was selected using reconstruction experiments on a subset of the dataset.
  • Image-domain U-Net learning feeds pseudoinverse image estimates into a CNN trained to produce artifact-free images resembling the ground-truth distribution.The trained network was later applied to previously unseen test measurements.
  • DIP-TV uses a randomly initialized convolutional network with total-variation regularization to reconstruct images from incomplete measurements.The added regularization addresses the tendency of DIP to overfit measurement noise upon convergence.

S. III. EXAMPLES OF MEASUREMENT SPACE

Measurement-space hallucination maps assess consistency between a reconstructed measurement component and the stably recoverable pseudoinverse component. They differ from measurement-component error maps when noise or modeling errors affect the imaging system.

  • The measurement-space hallucination map compares the reconstructed measurement component with the truncated pseudoinverse solution stably obtained from the data.
  • Unlike the corresponding error map, computing the measurement-space hallucination map does not require knowledge of the true object.
  • Measurement-component error maps and measurement-space hallucination maps can differ because the former requires the true object and the latter uses a truncated pseudoinverse reference.
  • Noise and modeling error can make the reconstructed measurement-space hallucination map and the true measurement component represent different information.The discrepancy is expected when measurement noise or mismatch between the true and assumed imaging operators is substantial.
  • Examples for IND and OOD U-Net reconstructions show appreciable differences between measurement-component error maps and measurement-space hallucination maps.The reported differences are attributed to non-trivial measurement noise and additional phase-noise disturbance in the simulation.
Loading 2012.00646v3…