On instabilities of deep learning in image reconstruction - Does AI come at a cost?

Vegard Antun; Francesco Renna; Clarice Poon; Ben Adcock; Anders C. Hansen

Source-linked AI summary

On instabilities of deep learning in image reconstruction - Does AI come at a cost?

Vegard Antun, Francesco Renna, Clarice Poon, Ben Adcock, Anders C. Hansen

arXiv:1902.05300v1Feb 14, 2019cs.CV

TL;DR

Deep learning reconstruction raises the question of whether methods can be safely used when small perturbations, structural changes, or sampling changes affect outputs. The paper introduces a stability test covering these cases and finds that deep-learning methods typically exhibit unstable behavior, requiring network-specific validation and potentially retraining across sampling configurations.

Problem
The paper asks whether deep-learning reconstruction methods can always be safely used in physical and life sciences when instabilities may affect medically important image details.
Method
The paper tests tiny perturbations, visible structural changes, and changes in sampling amount, using comparisons with state-of-the-art methods and network-specific empirical evaluation.
Results
Deep-learning reconstruction methods typically yield unstable behavior, including diverse artefacts, failures to recover structural changes, and reconstruction quality that may deteriorate or stagnate with more samples.
Takeaways & Limitations
Safe use requires testing each network for instability, while changing sampling configurations may require retraining networks for specific patterns, ratios, and dimensions.
Takeaways & Limitations
PSNR is unsuitable for comparing image quality between different image types, so the study uses it only to assess each network’s change with subsampling percentage.

Abstract

from arXiv · show

Deep learning, due to its unprecedented success in tasks such as image classification, has emerged as a new tool in image reconstruction with potential to change the field. In this paper we demonstrate a crucial phenomenon: deep learning typically yields unstablemethods for image reconstruction. The instabilities usually occur in several forms: (1) tiny, almost undetectable perturbations, both in the image and sampling domain, may result in severe artefacts in the reconstruction, (2) a small structural change, for example a tumour, may not be captured in the reconstructed image and (3) (a counterintuitive type of instability) more samples may yield poorer performance. Our new stability test with algorithms and easy to use software detects the instability phenomena. The test is aimed at researchers to test their networks for instabilities and for government agencies, such as the Food and Drug Administration (FDA), to secure safe use of deep learning methods.

The instability test

The paper’s instability test probes deep-learning reconstructions under tiny perturbations, structural changes, and altered sampling, revealing diverse failure modes and unstable responses across networks. It also identifies practical consequences for validation, retraining, and safe deployment.

Tiny perturbations: The test searches for small image or measurement perturbations that cause severe changes in a reconstruction, while comparing neural networks with stable state-of-the-art methods.The comparison separates network-induced instability from ill-conditioning in the inverse problem.
Small structural changes: The structural-change test adds visible details such as symbols or letters and checks whether they remain visible after reconstruction.Networks vary from removing or blurring details to preserving them, including cases where the details were absent from training data.
More samples: Changing the number of samples tests whether reconstruction quality improves, stagnates, or deteriorates away from the training sampling pattern.Ell 50 and DAGAN deteriorate, MRI-VN stagnates, and Deep MRI improves with more samples.
Tiny perturbations: Nearly invisible perturbations can produce misleading, non-physical artefacts, although the affected network may remain stable to structured changes.This contrast is reported for Deep MRI and related experiments.
Conclusion: The paper concludes that deep-learning reconstruction methods typically yield unstable behavior, with different networks exhibiting different artefacts and structural-change failures.The reported evidence spans varied architectures and sampling models, but the sample is too small to establish architecture-independent universality.
Conclusion: Because quality can deteriorate or stagnate with additional samples, networks may require retraining for each sampling pattern, ratio, and dimension, alongside network-specific empirical testing.The authors describe this as potentially requiring hundreds of networks and extensive testing to assess safe use.

Methods

The paper constructs reconstructions from under-sampled measurements and tests neural networks for instability using perturbation-search algorithms, structural-detail checks, and stable compressed-sensing benchmarks.

Reconstruction setup: Under-sampled measurements y = Ax are first mapped to a poor linear reconstruction Hy, which a neural network f is trained to improve.For MRI, H is often A∗; for CT, filtered back projection is typically used because A∗ performs poorly.
Reconstruction setup: The network is trained on paired training images and poor reconstructions by minimizing a cost function over a chosen neural-network class.The training pairs are {HAx_j, x_j}, with measurements Ax_j supplied as targets and inputs through HAx_j.
Test design: Perturbations can be generated through alternative objective choices, including p(x) = f(Ax) and p(x) = x, which may produce substantially different artefacts.The authors deliberately formulate the presented test in the image domain, while noting that sampling-domain perturbations are also possible.
Tiny perturbations: The instability test searches for a small image perturbation that maximizes the network output change, using gradient ascent with momentum.Algorithm 1 initializes a scaled random perturbation and iteratively updates it for M iterations using λ, γ, η, and τ.
Structural changes: The test also examines structural changes by adding a clearly visible detail and checking whether it remains visible after reconstruction.The paper uses small symbols as illustrative details; the tested networks recovered them even when they were absent from training images.

1 Overview

The overview catalogs the neural networks evaluated across MRI and CT inverse problems, including their sampling settings, training sources, and implementation availability.

Networks evaluated: The study evaluates AUTOMAP, DAGAN, Deep MRI, Ell 50, Med 50, and MRI-VN across MRI or CT reconstruction tasks.The networks differ in resolution, sampling percentage, transform modality, architecture, and training dataset.
MRI settings: AUTOMAP uses 60% subsampling for low-resolution single-coil MRI, while DAGAN and Deep MRI use 20% and 33% subsampling, respectively.These settings describe the networks tested in the experiments.
Reproducibility: The evaluated networks combine private weights, author-provided code or weights, online implementations, and retraining from published reproduction instructions.DAGAN was retrained from instructions, while AUTOMAP and Med 50 weights were obtained through private communication.
CT settings: Ell 50 uses 50 sinogram lines for CT or Radon-transform problems, and Med 50 shares its architecture but uses a different training dataset.The number 50 refers to the sampling lines used in the sinogram.
MRI settings: MRI-VN targets medium- to high-resolution parallel MRI with 15 coil elements and 15% subsampling in these experiments.The authors used a lower subsampling percentage than the original 25% and 33% settings while retaining other suggested parameters except batch size.

2 Deep learning and neural networks for inverse problems

Deep learning for inverse problems learns a neural network that maps measurement data to an image approximation, sometimes after applying a transformation based on the forward operator.

Core formulation: The learned network takes measurements y = Ax as input and produces an approximation to the unknown image x.A represents the matrix corresponding to the sampling modality.
Core formulation: Some networks first transform the measurements into ˜x = Hy to exploit structural knowledge of the forward operator A.This preprocessing is used instead of applying the network directly to y.
Operator-dependent preprocessing: H represents the adjoint operator for Fourier-domain measurements and a discretised filtered back projection operator for Radon measurements.The transformation therefore depends on the sampling modality.

2.1 AUTOMAP

AUTOMAP directly processes undersampled Cartesian Fourier measurements through fully connected and convolutional layers to reconstruct image magnitudes.

Scope: The experiments test AUTOMAP for image reconstruction from undersampled Cartesian Fourier samples.The network was proposed for several measurement settings, but this study evaluates only the undersampled Cartesian Fourier version.
Input representation: AUTOMAP receives complex measurement data with n = 128 and m = 9855, representing 60% of n^2 samples, reshaped into 2m real-valued entries.The measurements are represented as a zero-padded n × n image conceptually, then vectorized for the network.
Network architecture: Two fully connected layers are followed by 5 × 5 convolutional layers and a 7 × 7 deconvolutional output layer producing an n × n image-magnitude matrix.The convolutional layers use 64 feature maps and stride 1 × 1.
Training: Training uses an ℓ2 image-difference loss combined with an ℓ1 activation penalty, with λ = 0.0001, RMSProp, minibatches of 100, and 100 epochs.The training set contains 50,000 images from 131 subjects and uses 60% Poisson-disk Cartesian k-space sampling.

2.2 DAGAN

DAGAN reconstructs MRI images from undersampled Fourier data using a conditional GAN with a U-net generator, discriminator, residual connection, and multiple losses.

Formulation: DAGAN targets MRI reconstruction from measurements y = Ax, where A is a subsampled discrete Fourier transform.Its network input is the noisy magnitude image ˜x = |Hy| with H = A∗.
Generator architecture: The generator has eight convolutional and eight deconvolutional layers with batch normalization, leaky ReLU or ReLU activations, and skip connections.The leaky ReLU slope is 0.2 for negative inputs.
GAN design: The conditional GAN combines a generator that reconstructs the image with a discriminator that assesses reconstructed-image quality.The generator uses a U-net structure and is trained jointly with the discriminator.
Generator architecture: A global skip connection adds the network input to the final activated output, so the generator approximates the residual error.A ramp function clips the resulting image values to [−1, 1].
Training objective: Training minimizes image-domain MSE, frequency-domain MSE, perceptual, and generator losses combined as LTOTAL = αLiMSE + βLfMSE + γLVGG + τLGEN.The experiments use α = 15, β = 0.1, γ = 0.0025, and τ = 1.
Training data: The network is trained on 15,912 brain MR images with 4,977 validation images and 20% Fourier subsampling using 1D Gaussian masks.Training uses image flipping, rotation, shifting, brightness adjustment, zooming, and elastic distortion for augmentation.

2.3 DeepMRINet

DeepMRINet reconstructs images through cascaded CNNs interleaved with data-consistency layers that use the measured Fourier data. The network was pretrained with a complex-image MSE plus weight regularization using cardiac MRI data.

Architecture: The network maps transformed measurements ˜x = Hy into an image reconstruction while repeatedly incorporating the measurements y and sampling indices Ω.The complete cascade is expressed as f(y, Ω) = DCλ(CNNn(···DCλ((CNN1(Hy)), y, Ω)···), y, Ω).
Data consistency: Each data-consistency layer performs a weighted average between CNN output Fourier coefficients and the true samples y.The weighting parameter λ may be trained or fixed; in the cited implementation, λ is effectively chosen as ∞.
Architecture: DeepMRINet cascades five CNNs with five data-consistency layers to reconstruct images from subsampled Fourier measurements.Each CNN uses convolutional layers, ReLU activations, and a skip connection; the data-consistency layers promote agreement with measured Fourier coefficients.
Training: DeepMRINet was trained with complex-image MSE and ℓ2 weight regularization, using Adam with learning rate α = 10^-4 and batch size 10.The network used β1 = 0.9 and β2 = 0.999 for the Adam optimizer.
Training: Training used cardiac cine MRI data from five subjects, with retrospectively sampled Cartesian k-space measurements and rigid-transform data augmentation.The source dataset contains 10 fully sampled short-axis cardiac cine scans, each preprocessed into 30 complex-valued 256 × 256 frames.

2.4 FBPConvNet – The Ell 50 and Med 50 networks

The Ell 50 and Med 50 FBPConvNet networks reconstruct images from 50 uniformly spaced Radon lines after a filtered-back-projection preprocessing step. They share architecture and training settings but differ in their training datasets.

Architecture: The networks share the same architecture, training algorithm, and hyperparameters; their training datasets are the only stated difference.The architecture includes convolutional and deconvolutional layers with batch normalization and ReLU activations, except at the final layer.
Setup: Ell 50 and Med 50 reconstruct images from Radon measurements acquired along 50 uniformly spaced radial lines.Both networks use a filtered back projection H to obtain a noisy approximation ˜x = Hy before learned processing.
Training: Training uses an ℓ2 output-to-ground-truth loss with stochastic gradient descent and momentum, learning rates from 0.01 to 0.001, and momentum 0.99.The minibatch size is 1, and gradients are clipped to [−Imax, Imax] with Imax = 10^-2.
Datasets: Ell 50 is trained on 475 synthetic sinograms generated from ellipses with random intensity, size, and location.The ellipse sinograms use 1000 uniformly spaced views as the ground truth before training from 50 views.
Datasets: Med 50 is trained on synthetic sinograms generated from 475 real in-vivo CT images from the Mayo Clinic Low-dose Grand Challenge database.The sinograms use 1000 views as ground truth, while the network is trained from 50 views.

2.5 MRI Variational Network (MRI-VN)

MRI-VN unfolds a variational minimization procedure into a ten-layer network combining learned convolutional regularization with a data-fidelity term. The evaluated implementation was trained for one undersampled knee MRI protocol using 200 images from 10 patients.

Input representation: The network initializes with u0 = Hy, where H = A∗pf produces a complex image from the parallel MRI measurements.The sampling operator uses 15 coil elements.
Architecture: MRI-VN consists of T = 10 layers corresponding to iterations of an unfolded Landweber-type variational method.Each layer combines a learned nonlinear regularization operation with the measurement-fidelity term λtA∗(Aut − y).
Learned components: At each layer, learned convolutional operators Kt and nonlinear activations Ψt process the real and imaginary components of the complex image.The convolutional operators use Nk filters, while the activation is applied separately for each filter or channel.
Training: The MRI-VN parameters are trained with an ℓ2 loss evaluated on smoothed magnitude images, using IIPG optimization for 1000 epochs with step size 10^-3.Training minibatches contain five examples because the implementation was adjusted for memory limitations.
Data: The evaluation trains one network for a Coronal Spin Density weighted with Fat Suppression protocol using knee images from 10 patients, totaling 200 training images.The data were retrospectively undersampled by zeroing 85% of k-space, using a regular sampling scheme.