Learned D-AMP: Principled Neural Network based Compressive Image Recovery

1 Introduction

• When m < n, recovering x from measurements y and matrix A is underdetermined, so compressive sampling requires prior knowledge about x [1; 2].Source: §1 · p. 1

1.1 Hand-designed recovery methods

• Hand-designed methods use expert priors such as sparsity, total variation, or nonlocal self-similarity, but accurate-prior algorithms are too slow for many real-time applications and do not exploit available training data.For natural images, advanced priors outperform simple wavelet sparsity by a large margin.Source: §1.1 · p. 2

1.2 Data-driven recovery methods

• Data-driven networks learn image structure from training data [16] and can compete in accuracy while running thousands of times faster, but lack performance theory and require training for specific matrices and noise levels.Source: §1.2 · p. 2

1.3 Mixing hand-designed and data-driven methods for recovery

• Hybrid methods learn priors within expert-designed recovery algorithms, seeking realistic learned priors while retaining interpretability and guarantees; LDAMP draws on both black-box and unrolling approaches.Source: §1.3 · p. 2
• Unrolling turns iterative algorithms such as ISTA and AMP into trainable networks, with learned parameters optimized by stochastic gradient descent; unrolling and training have improved reconstruction quality and speed.Examples include LISTA, LAMP, and ADMM-Net [26].Source: §1.3 · p. 3

2.1 D-IT and D-AMP

• D-IT alternates a gradient step toward the measurement-consistent affine subspace with denoising toward natural images, repeating until convergence.The recovery problem seeks an image x from measurements y = Ax using prior information such as x ∈ C, the set of natural images.Source: §2.1 · pp. 3–4
• D-AMP adds an Onsager correction that removes bias from intermediate solutions, keeping effective noise consistent with the AWGN expected by typical denoisers.D-IT’s denoising iterations become less effective as its effective noise deviates from AWGN, slowing convergence [26].Source: §2.1 · p. 4

2.2 Denoising convolutional neural network

• DnCNN is a 16-to-20-layer convolutional image denoiser that uses residual learning and is described as more accurate and faster than competing techniques.Its layers use 3 × 3 filters, ReLUs, and batch normalization in the intermediate layers.Source: §2.2 · p. 4

2.3 Unrolling D-IT and D-AMP into networks

• LDAMP unrolls D-AMP into a 10-layer network with two tied-weight denoisers per AMP layer, while LDIT omits the Onsager correction and uses one denoiser per layer.The second LDAMP denoiser estimates divergence using a Monte-Carlo approximation; trainable denoisers must propagate gradients, so black-box BM3D is unsuitable and DnCNN is used.Source: §2.3 · pp. 4–5
• LDAMP training learns only denoiser weights, unlike LISTA and LAMP, which decouple and learn the A and AH matrices used in the network.Denoiser weights may depend on the estimated noise standard deviation.Source: §2.3 · p. 5

• End-to-end training optimizes all network weights simultaneously, the standard method for training a neural network.Source: §3 · p. 5
• Layer-by-layer training sequentially trains and fixes each AMP layer, growing the network to 10 layers.The first AMP layer contains a 16–20-layer denoiser; each added layer is trained to recover the signal before its weights are fixed.Source: §3 · p. 5
• Denoiser-by-denoiser training fits separate denoisers to AWGN problems at different noise levels, then selects weights using the estimated noise standard deviation.Selection requires discretizing noise levels; for example, an estimated standard deviation of 35 selects a denoiser for standard deviations between 20 and 40.Source: §3 · p. 5

Comparing Training Methods

• Denoiser-by-denoiser trained networks generalize well from a sampling rate of 0.20 to 0.05, whereas end-to-end and layer-by-layer trained networks perform poorly under this change.The networks are trained at one sampling rate and tested at another.Source: Figure 4 · p. 6, §3 · p. 6
• Layer-by-layer training is optimal for LDAMP, and additional end-to-end training does not improve it; for LDIT, layer-by-layer training is suboptimal and end-to-end training dramatically improves performance.The networks tested each had 10 unrolled DAMP/DIT layers with a 16-layer DnCNN denoiser.Source: §3 · p. 6
• Denoiser-by-denoiser training is theoretically optimal for LDAMP but performs slightly worse in Table 4(a), likely because its discretized noise levels are not modeled by the theory.The gap can be reduced with finer noise-level discretization or deeper denoisers that handle a range of noise levels.Source: §3 · p. 6

4.1 State-evolution

• For large m and n, state evolution predicts LDAMP’s layerwise MSE when x0=0, with i.i.d. Gaussian or subgaussian measurements, i.i.d. Gaussian noise, and Lipschitz denoisers.Figure 5 illustrates the prediction for Gaussian measurements at rate m/n=0.1; the paper notes that this D-AMP/LDAMP state-evolution result is observational, not rigorous.Source: §4.1 · pp. 6–7, Figure 5 · p. 7

4.2 Layer-by-layer and denoiser-by-denoiser training is optimal

• Under Lemma 1’s monotonicity conditions, both layer-by-layer and denoiser-by-denoiser LDAMP training are MMSE optimal.The lemma requires each denoiser’s minimum expected squared error to be non-decreasing in noise standard deviation.Source: §4.2 · p. 7
• The optimality result follows from sequentially minimizing each layer’s expected state-evolution error and fixing its weights before training the next layer.The procedure minimizes the final layer’s expected error across the full network.Source: §4.2 · p. 7

Training parameters

• Section results used denoiser-by-denoiser trained networks with 10 unrolled DAMP/DIT layers, each containing a 20-layer DnCNN denoiser.Training used 382,464 50 × 50 patches, with seven standard test images rescaled to various resolutions.Source: §5 · pp. 7–8

Testing setup

• Tests used i.i.d. Gaussian and randomly sampled coded-diffraction measurements; except for Figure 7, images were 128 × 128 (n = 1282), and recovery accuracy and runtime were reported as PSNR and seconds.The coded diffraction operator randomly changes phase, takes a 2D FFT, and randomly subsamples; results broken down by image are provided in the supplement.Source: §5 · p. 8

Gaussian measurements

• With noise-free Gaussian measurements, LDAMP achieved the best reconstructions at every sampling rate on every test image except Fingerprints.For noisy Gaussian measurements, LDAMP uniformly outperformed the other methods; those results are reported in the supplement.Source: §5 · p. 8, Table 1 · p. 9

Coded diffraction measurements

• With noise-free coded diffraction measurements, LDAMP achieved the best reconstructions on every image except Fingerprints, while LDAMP and LDIT were significantly faster than competing methods.Because the forward and backward operators take O(n log n) operations, denoising dominates computation and LDAMP takes roughly 2× as long as LDIT.Source: §5 · p. 8, Table 2 · p. 9
• At 512 × 512, LDAMP produced noticeably cleaner reconstructions and ran over 60× faster than BM3D-AMP on the Boat image.The comparison used noise-free coded diffraction measurements at a sampling rate of 0.05.Source: §5 · p. 9, Figure 7 · p. 9

6 Conclusions

• LDAMP outperforms state-of-the-art BM3D-AMP and NLR-CS in both accuracy and run time.The conclusion identifies this as the most important result.Source: §6 · p. 9
• LDAMP is easy to train, applies to a variety of measurement matrices, and includes a state-evolution heuristic that accurately predicts its performance.Source: §6 · p. 9
• Rather than train an arbitrary black box, LDAMP unrolls a conventional iterative algorithm into a neural network with well-understood behavior, performance guarantees, and predictable shortcomings.The paper frames this as part of a trend using training data and extensive offline computation to improve iterative algorithms, and hopes it encourages further research in this direction.Source: §6 · p. 9

• Table 3 reports PSNR for 128 × 128 reconstructions using i.i.d. Gaussian measurements without measurement noise.Source: Table 3 · p. 12
• Table 4 reports PSNR for 128 × 128 reconstructions using coded diffraction measurements without measurement noise.Source: Table 4 · p. 13
• Table 5 reports PSNR for the 128 × 128 Boat test image with AWGN measurement noise at various standard deviations.Source: Table 5 · p. 14