PaperlayerView exact revision on arXiv

Source-linked AI summary

Unrolled Optimization with Deep Priors

Steven Diamond, Vincent Sitzmann, Felix Heide, Gordon Wetzstein

arXiv:1705.08041v2cs.CV

TL;DR

Inverse-imaging methods need to combine physical image-formation knowledge with expressive learned image priors. The paper proposes unrolled optimization with deep priors, integrating CNN priors into truncated iterative algorithms, and reports large-margin improvements across several imaging problems while identifying useful design choices. The work is limited to a small fraction of potential ODP applications and leaves blind and nonlinear image-formation models for future study.

  • Problem

    Inverse-imaging methods lack a general framework for incorporating image-formation knowledge into deep networks and for clarifying when prior information is useful.

  • Method

    ODP unrolls classical optimization algorithms into deep networks and trains CNN priors within their optimization architectures.

  • Results

    ODP instances outperform state-of-the-art methods by a large margin across denoising, deblurring, and compressed-sensing MRI, with simple primal algorithms performing best in the studied linear setting.

  • Takeaways & Limitations

    The experiments provide guidance on when prior information is advantageous and which optimization algorithms are most suitable for unrolling.

  • Takeaways & Limitations

    The study covers only a small fraction of potential ODP applications and does not yet address blind or nonlinear image-formation models.

Abstract

from arXiv · show

A broad class of problems at the core of computational imaging, sensing, and low-level computer vision reduces to the inverse problem of extracting latent images that follow a prior distribution, from measurements taken under a known physical image formation model. Traditionally, hand-crafted priors along with iterative optimization methods have been used to solve such problems. In this paper we present unrolled optimization with deep priors, a principled framework for infusing knowledge of the image formation into deep networks that solve inverse problems in imaging, inspired by classical iterative methods. We show that instances of the framework outperform the state-of-the-art by a substantial margin for a wide variety of imaging problems, such as denoising, deblurring, and compressed sensing magnetic resonance imaging (MRI). Moreover, we conduct experiments that explain how the framework is best used and why it outperforms previous methods.

1 Introduction

The paper introduces ODP, a general framework that combines unrolled optimization with deep CNN priors to incorporate image-formation knowledge into inverse-imaging networks. Across denoising, deblurring, and compressed-sensing MRI, ODP instances outperform prior methods and provide empirical guidance on algorithm and prior design.

  • ODP integrates prior knowledge of the image formation into deep networks for inverse problems in imaging.
  • ODP unrolls optimization methods while embedding deep convolutional prior architectures within the resulting network.
  • A single ODP model trained on many image formation operators outperforms specialized state-of-the-art models for deblurring and compressed-sensing MRI.
  • Simple primal algorithms that approximately invert the image formation operator at each iteration perform best for unrolled optimization with a linear image formation model.
  • ODP instances outperform state-of-the-art results by a large margin on denoising, deblurring, and compressed-sensing MRI.
  • The experiments clarify when prior information is advantageous and which optimization algorithms are suitable for unrolling.

2 Motivation

The motivation begins with Bayesian inverse imaging, where a latent image and a known physical measurement model define the reconstruction objective. ODP unrolls iterative optimization into a network whose learned prior and data steps are parameterized for efficient reconstruction.

  • The Bayesian model represents the latent image with a prior distribution and measurements with a noise distribution after applying a linear imaging operator.
  • The MAP estimate minimizes a data term and prior term formed from the negative log-likelihoods of the measurement and image distributions.
  • Unrolled methods replace convergence-based stopping with a predetermined number of iterations, which is suitable when very high optimization accuracy is unnecessary.
  • Unrolled model capacity is primarily determined by the prior representation because each iteration uses only a small number of algorithm hyperparameters.
  • ODP directly parameterizes the gradient or proximal operator of the prior and defines the prior term implicitly.

3 Unrolled optimization with deep priors

ODP integrates image-formation-aware data steps with learned CNN priors by unrolling classical optimization algorithms into trainable networks. Its design varies the optimization algorithm, iteration count, initialization, and CNN architecture according to the inverse problem.

  • 3 Unrolled optimization with deep priors: ODP factors each network into data steps encoding the image-formation model and CNN steps representing statistical image priors.This factorization combines classical optimization structure with deep learned priors.
  • 3 Unrolled optimization with deep priors: The framework specifies an optimization algorithm, unrolled iteration count N, initialization function φ, and CNN prior for each network.The CNN output represents either a prior gradient or proximal operator, depending on the optimization algorithm.
  • 3 Unrolled optimization with deep priors: ODP networks can be viewed either as optimization algorithms with learned CNN priors or as CNNs whose layers are tailored to the image-formation model.These are complementary interpretations of the same architecture.
  • 3 Unrolled optimization with deep priors: The networks are trained directly on expected reconstruction loss over image-measurement pairs rather than only pretraining priors on natural-image datasets.The training objective uses a reconstruction metric such as PSNR, while prior parameters are learned within the optimization algorithm.
  • 3.2 Design choices: Proximal gradient is proposed as a default unrolling choice when the data-term proximal operator and Jacobian can be computed efficiently.Its CNN prior is interpreted as a scaled gradient and naturally forms a residual connection.
  • 3.2 Design choices: Algorithms that approximately invert the image-formation operator at each iteration perform on par, while ADMM and LADMM are at best slightly better than simpler primal methods.The comparison covers proximal gradient, ADMM, LADMM, and gradient descent for deblurring and compressed sensing MRI.
  • 3.2 Design choices: CNN depth and iteration count trade off differently by task: larger priors with fewer iterations suit denoising, whereas smaller priors with more iterations suit deconvolution and MRI.The distinction reflects whether the data step is trivial or a complicated global operation.

4 Related work

Prior unrolled methods use sparse filterbank or shallow field-of-experts priors, while direct-inversion methods target specific imaging problems. ODP extends these approaches with deeper CNN priors and a general optimization-based framework.

  • Deep models for direct inversion: ODP generalizes and improves upon previous unrolled-optimization and direct-inversion approaches for inverse imaging problems.The paper presents ODP results for denoising, deblurring, and compressed sensing MRI.
  • Unrolled optimization networks: Learned sparsity-prior networks use an ℓ1 norm on filterbank responses, but their representational power is limited by this simple prior.The filterbank applies convolution kernels to the image.
  • Field-of-experts: Field-of-experts methods parameterize a prior gradient or proximal operator with a filterbank and separable nonlinearity.ODP is described as improving on these approaches empirically and theoretically because deeper CNN priors are more expressive than essentially 2-layer CNNs.

5 Experiments

Experiments evaluate ODP on denoising, deblurring, and compressed sensing MRI, showing strong performance and generalization across image formation models. Ablations and algorithm comparisons indicate that model-based data steps and approximate inversion help explain these results.

  • Denoising: ODP outperforms all state-of-the-art methods on the evaluated Gaussian denoising test set.Table 1 reports average PSNR across five noise and blur settings.
  • Deblurring: ODP models outperform prior work slightly for the low-pass disk kernel and are competitive with specialized models across blur kernels.The low-pass disk kernel completely removes high-frequency content; a single out-of-focus model matches Schuler et al. across four kernels and noise levels.
  • Compressed sensing MRI: ODP outperforms BM3D-MRI for all tested sampling patterns, with larger gains for sparser Fourier sampling.A single model performs across four sampling patterns, unlike ADMM-Net models trained separately per pattern.
  • Contribution of prior information: Ablations interpret ODP as alternating local CNN corrections with a global data-step operator B satisfying BA ≈ I.The CNN denoises and corrects approximate-inversion errors, while pure residual networks must denoise and invert A directly.
  • Comparing algorithms: For deblurring, proximal-gradient and ADMM models outperform LADMM and gradient descent, while CS MRI algorithms show similar performance.The stronger deblurring models use regularized pseudoinverse data steps; Lagrange multipliers provide minimal benefit at N = 8.

6 Conclusion

The ODP framework achieves state-of-the-art results across several inverse imaging problems and provides insights into prior information and suitable unrolled algorithms. The study also identifies a broader application boundary, since the evaluated imaging problems cover only a small fraction of ODP’s potential uses.

  • ODP yields state-of-the-art results for denoising, deblurring, and compressed sensing MRI.
  • Ablation studies provide general insights into the benefits of prior information and which algorithms suit unrolled optimization.
  • The evaluated imaging problems represent only a small fraction of ODP’s potential applications, motivating future work on blind and nonlinear models.
Loading 1705.08041v2…