PaperlayerView exact revision on arXiv

Source-linked AI summary

The Little Engine that Could: Regularization by Denoising (RED)

Yaniv Romano, Michael Elad, Peyman Milanfar

arXiv:1611.02862v3cs.CVmath.NA

TL;DR

Image denoising has become highly effective, raising the question of how to reuse denoisers systematically across inverse problems while avoiding limitations of P^3. The paper proposes RED, which defines an explicit denoiser-based regularizer and supports flexible optimization. RED incorporates broad denoiser choices, targets general inverse problems, and is associated with convexity and global-optimum convergence under stated conditions.

  • Problem

    Existing denoising advances invite broader reuse, but P^3 can lack an explicit objective, require delicate ADMM tuning, and constrain iterative-procedure choice.

  • Method

    RED defines an explicit image-adaptive Laplacian regularizer from a denoising engine and its residual, rather than using the denoiser only implicitly through ADMM.

  • Results

    RED incorporates any image denoising algorithm, treats general inverse problems effectively, and yields state-of-the-art results in image deblurring and single-image super-resolution.

  • Takeaways & Limitations

    Under mild denoiser conditions and convex log-likelihoods, RED's convex regularization guarantees convergence to a global optimum while allowing flexible optimization methods.

  • Takeaways & Limitations

    The gradient approach may require many iterations because each denoiser activation updates the likelihood only mildly.

Abstract

from arXiv · show

Removal of noise from an image is an extensively studied problem in image processing. Indeed, the recent advent of sophisticated and highly effective denoising algorithms lead some to believe that existing methods are touching the ceiling in terms of noise removal performance. Can we leverage this impressive achievement to treat other tasks in image processing? Recent work has answered this question positively, in the form of the Plug-and-Play Prior ($P^3$) method, showing that any inverse problem can be handled by sequentially applying image denoising steps. This relies heavily on the ADMM optimization technique in order to obtain this chained denoising interpretation. Is this the only way in which tasks in image processing can exploit the image denoising engine? In this paper we provide an alternative, more powerful and more flexible framework for achieving the same goal. As opposed to the $P^3$ method, we offer Regularization by Denoising (RED): using the denoising engine in defining the regularization of the inverse problem. We propose an explicit image-adaptive Laplacian-based regularization functional, making the overall objective functional clearer and better defined. With a complete flexibility to choose the iterative optimization procedure for minimizing the above functional, RED is capable of incorporating any image denoising algorithm, treat general inverse problems very effectively, and is guaranteed to converge to the globally optimal result. We test this approach and demonstrate state-of-the-art results in the image deblurring and super-resolution problems.

1 Introduction

The paper asks how strong denoising algorithms can be systematically reused for broader inverse problems. RED answers with an explicit denoiser-defined regularizer that is more flexible than P^3 and supports effective general inverse-problem solving.

  • Denoising algorithms have approached near-optimal performance, motivating efforts to leverage them for other image-processing tasks.
  • P^3 uses ADMM to decompose inverse problems into denoising steps and simpler L2-regularized subproblems.
  • P^3 may lack an explicit objective, require delicate parameter tuning, and restrict the choice of iterative optimization procedure.
  • RED defines an explicit image-adaptive Laplacian-based regularization functional from a chosen denoiser and its residual.
  • RED is flexible in optimization-method choice, can incorporate any denoising algorithm, and addresses general inverse problems with a simple algorithmic structure.
  • Under mild denoiser conditions, RED's regularization is convex, yielding global-optimum convergence when the log-likelihood is also convex.

2 Preliminaries

The preliminaries formulate inverse problems as MAP optimization and review how P^3 uses ADMM to replace an explicit prior with denoising operations. They also identify objective-function, tuning, and convergence limitations of that approach.

  • MAP estimation converts inference from measurements into minimizing a log-likelihood term plus a prior-derived regularizer.
  • For linear degradation with white Gaussian noise, the log-likelihood is an L2 data-fidelity term, while other noise models yield different forms.
  • P^3 applies ADMM variable splitting, producing sequential x, v, and u updates.
  • The x-update is a simpler inverse problem with L2 proximity regularization, while the v-update is equivalent to denoising x + u under Gaussian noise power 1/β.
  • P^3 parameter tuning is difficult because the denoiser noise level is λ/β and β is commonly increased across iterations.
  • P^3 convergence results generally establish a steady state rather than the global minimizer, and an arbitrary denoiser need not correspond to a specific prior.

3 The Image Denoising Engine

The paper defines a denoising engine broadly as a same-size image-to-image map and imposes differentiability, local homogeneity, and passivity conditions. These properties support RED's derivations and are empirically examined for several established denoisers.

  • A denoising engine maps a noisy image y to an estimate f(y) of the same size, regardless of whether it is MAP-, MMSE-, or otherwise based.
  • Denoising is distinguished from blur or downscaling because additive noise typically moves natural images away from their image manifold.
  • The framework assumes differentiability and imposes local homogeneity, requiring f(cx) = cf(x) near c = 1.
  • Strong passivity constrains the denoiser Jacobian's spectral radius, preventing amplification of an input image's norm.
  • Many denoisers admit a pseudo-linear form in which nonlinear decisions adapt a linear filter applied to the noisy image.
  • Homogeneity implies that the image-dependent filter is insensitive to small perturbations and behaves as an essentially constant linear operator locally.
  • Experiments on five denoisers show approximate homogeneity by comparing f((1 + ϵ)x) with (1 + ϵ)f(x) for Peppers.
  • K-SVD, BM3D, NLM, EPLL, and TNRD experimentally satisfy passivity with spectral-radius values no greater than 1.

4 Regularization by Denoising (RED)

RED defines an explicit image-adaptive Laplacian regularizer from a denoising engine, then minimizes the resulting inverse-problem objective using flexible iterative algorithms.

  • 4.1 The Image-Adaptive Laplacian: RED constructs an image-adaptive Laplacian prior from the denoising engine and uses it to regularize general inverse problems.The framework generalizes beyond pseudo-linear filters by defining the residual through an arbitrary denoiser f(x).
  • 4.1 The Image-Adaptive Laplacian: The regularization value is low when the image is nearly a denoiser fixed point or when its denoising residual has weak cross-correlation with the image.These conditions correspond respectively to a small residual and residual behavior resembling white noise.
  • 4.1 The Image-Adaptive Laplacian: Under the stated directional-derivative assumption, the objective gradient reduces to the likelihood gradient plus λ(x − f(x)).This form requires only one denoiser activation for gradient evaluation; a finite-difference alternative requires two.
  • 4.2 Deploying the Denoising Engine for Solving Inverse Problems: RED can minimize its objective through steepest descent, ADMM, or fixed-point iterations, with each iteration applying the denoising engine alongside simpler calculations.The paper provides pseudocode for these alternatives and notes that line search can dynamically set the steepest-descent step size.
  • 4.2 Deploying the Denoising Engine for Solving Inverse Problems: Gradient-based methods update the likelihood mildly per denoiser activation, which may require many iterations; ADMM offers a more aggressive alternative.The paper describes early termination of the ADMM v-update to reduce the cost of repeated denoiser applications.

5 Analysis

The analysis studies RED’s convexity, alternative priors, relation to Plug-and-Play Prior, and computational cost, emphasizing conditions for global convergence and differences between the frameworks.

  • 5.1 Convexity: If the denoiser Jacobian is stable and the likelihood is convex, RED’s objective is convex and the algorithm converges to the global optimum.The stability condition is η(∇xf(x)) ≤1.
  • 5.1 Convexity: For NLM, K-SVD, and other denoisers, the associated Laplacian is positive semidefinite when the relevant eigenvalues lie in [0, 1].This supports convexity for these denoising engines under the paper’s stated conditions.
  • 5.2 An Alternative Prior: The alternative prior regularizes with the square of the Laplacian rather than the Laplacian itself, and the paper leaves this fourth-order option for future work.The paper characterizes this possibility as more delicate.
  • 5.3 When is Plug-and-Play-Prior = RED ?: RED and Plug-and-Play Prior are exactly equivalent under ADMM only for denoiser eigenvalues restricted to 1 or β/λ, a very limiting case.Outside this case, the two methods differ.
  • 5.3 When is Plug-and-Play-Prior = RED ?: RED and Plug-and-Play Prior have roughly the same computational cost per iteration when RED uses ADMM with one v-update iteration or uses the fixed-point option.The paper presents the comparison as secondary to differences in objective definition, convergence guarantees, and parameter tuning.

6 Results

RED is evaluated on image deblurring and super-resolution using multiple denoising engines and numerical schemes. The experiments show competitive restoration quality, convergence across schemes, and greater parameter robustness than P^3.

  • 6.1 Image Deblurring: RED deblurring experiments compare median-filter and TNRD denoisers against P^3, NCSR, IDD-BM3D, and other baselines under uniform and Gaussian blur.The evaluation uses PSNR on luminance-channel reconstructions, with parameters tuned for the dataset.
  • 6.1 Image Deblurring: TNRD-based RED achieves results competitive with NCSR and IDD-BM3D, while median-filter RED substantially trails state-of-the-art methods.The median filter nevertheless provides surprisingly strong restoration for Gaussian blur, comparable to total variation deblurring.
  • 6.1 Image Deblurring: All three RED schemes converge with both denoisers, but ADMM is fastest, fixed-point is faster than equal-cost ADMM, and steepest descent is slowest.Fixed-point uses one denoiser application per iteration, whereas ADMM with m2 = 3 uses three.
  • 6.2 Image Super-Resolution: RED super-resolution improves bicubic interpolation by 2.19dB on average with a median filter, while TNRD yields state-of-the-art results.P^3 and all three RED variants achieve similar restoration performance, with sharp and clear edges in the visual comparisons.
  • 6.3 Robustness to the Choice of Parameters: P^3 exhibits unstable behavior under parameter changes, whereas RED reaches the same final PSNR across tested β values and uses fixed parameters.Changing RED’s denoiser noise level changes the objective and resulting PSNR, while β primarily affects convergence rate.

7 Conclusions

RED places a denoising engine inside an explicit regularization functional, yielding an alternative framework with tractable differentiation and competitive restoration performance. The authors leave parameter selection and broader generalization as open questions.

  • 7 Conclusions: RED uses a denoising engine within the regularization term to handle general inverse problems.The resulting regularization terms remain differentiable using the denoiser rather than its derivative.
  • 7 Conclusions: The framework has appealing properties including convexity and a relation to advanced Laplacian smoothing.
  • 7 Conclusions: Experiments demonstrate the validity of RED's regularization strategy and competitive performance against the Plug-and-Play Prior method.
  • 7 Conclusions: Further work is required to investigate generalization of RED to other regularization functionals.
  • 7 Conclusions: The influence of the fixed parameter σf and the potential benefit of varying it remain unaddressed.The authors identify parameter setting as an open question requiring further work.

A Can We Mimic any Prior?

The paper asks which priors can be represented by RED denoisers and derives an inverse relationship from a prior to its denoising function. This construction requires a 2-homogeneous prior, although many useful regularizers satisfy or can be transformed to that condition.

  • A Can We Mimic any Prior?: Given a prior ρ(x), RED seeks a denoiser f(x) whose gradient represents the prior-induced denoising operation.The denoiser is explicitly defined in terms of the prior's gradient.
  • A Can We Mimic any Prior?: The prior ρ(x) must be 2-homogeneous for this prior-to-denoiser construction.This requirement follows from the assumed degree-1 homogeneity of the denoiser.
  • A Can We Mimic any Prior?: Squared total variation and squared norms provide examples of priors that meet the required homogeneity after transformation.Total variation and norms are 1-homogeneous, so squaring them yields 2-homogeneous functions.
  • A Can We Mimic any Prior?: Other k-homogeneous functions can be made 2-homogeneous by raising them to the power 2/k.The paper also notes that squared order-statistics such as the median fit this pattern.
  • A Can We Mimic any Prior?: The resulting relation resembles a steepest-descent step with step size 1 and initial condition x0 = y.Under the stated denoising formulation, one steepest-descent step has the same form as denoising y.
  • A Can We Mimic any Prior?: The sparsity-inspired L0 prior is identified as an exception that cannot be treated through this construction.

B Kernelizing Priors

Kernelizing the prior expresses the pseudo-linear denoiser through the prior's Hessian, producing an image-dependent Laplacian and filter matrix. Convexity and gradient-based priors then yield structural properties such as positive semidefiniteness and stochasticity.

  • B Kernelizing Priors: For pseudo-linear filters f(x) = W(x)x, the gradient of the filter equals its weight matrix W(x).
  • B Kernelizing Priors: The implied weight matrix is W(x) = I − H(ρ(x)), so the Laplacian filter is L(x) = H(ρ(x)).The prior's Hessian directly supplies the image-adaptive Laplacian weights.
  • B Kernelizing Priors: If ρ(x) is convex, its Hessian is symmetric positive semidefinite, making L(x) positive semidefinite under the stated spectral condition.When η(L(x)) ≤ 1, the paper also derives passivity and positive semidefiniteness of W(x).
  • B Kernelizing Priors: Because the Hessian and implied filter matrix are invariant to image scaling, W(x)x is 1-homogeneous.
  • B Kernelizing Priors: Defining the prior on image gradients makes the Hessian annihilate the constant vector, yielding a row-stochastic filter.The same construction also gives column-stochasticity because 1^T D^T = 0.

C More on Homogeneity

The paper emphasizes homogeneity as a key condition underlying its filter and prior results, motivating examination of whether familiar denoisers satisfy it.

  • C More on Homogeneity: Homogeneity is a key condition in the development of RED's results and is examined for popular denoising filters.

C.1 Non-Local Means and Bilateral Filter

The analysis examines when NLM and bilateral filters are homogeneous under image scaling. It concludes that, despite Gaussian kernels being non-homogeneous, these filters are nearly 1-homogeneous for small scaling perturbations.

  • Kernel choice: Homogeneity depends on the kernel choice, although homogeneous kernels make the normalized filter matrix invariant to image scaling.The invariance follows through cancellation in the normalized construction.
  • Kernel choice: Gaussian kernels are not homogeneous, but the resulting NLM and bilateral filters can nevertheless be nearly 1-homogeneous.The bilateral case is treated similarly because spatial weights are unchanged by scaling image values.
  • Weight analysis: NLM weights are constructed from patch similarities and normalized kernel rows, with image scaling absorbed into the parameter σ.The analysis considers both the near-zero-σ regime and scaling-induced changes in σ.
  • Weight analysis: For ϵ = 1/n and sufficiently large σ, the perturbation term becomes negligible, so NLM and bilateral weights change very little under (1+ϵ)x.The resulting filters are therefore almost exactly 1-homogeneous.

C.2 Tikhonov Regularizer and Wiener Filtering

This section analyzes homogeneity for linear and adaptive denoisers, then extends the argument to patch-based EPLL and K-SVD methods. Tikhonov is exactly homogeneous, while Wiener, EPLL, and K-SVD satisfy the condition under stated conditions or with high probability.

  • Tikhonov regularizer: Tikhonov regularization is homogeneous because applying its linear denoiser to (1+ϵ)y produces (1+ϵ)f(y).The result follows directly from the linear denoiser expression.
  • Wiener filtering: Image-adaptive Wiener filtering satisfies homogeneity under mild conditions when the adaptive filter changes linearly with ϵ and its second term is negligible for λ ≪1.The analysis uses the bound ∥σ2WBTB∥ < 1.
  • Patch-based denoising: Patch-based denoisers split images into overlapping patches, denoise them locally, and average the reconstructed patches.Their behavior depends on the selected local nonlinear prior.
  • EPLL: EPLL’s nonlinear GMM model selection is stable to small input scaling, with ties handled by choosing the model having the smaller log |Σk| term.The patch estimate then uses the corresponding Wiener filter.
  • K-SVD: K-SVD’s OMP selection is unchanged by infinitesimal scaling except near its stopping threshold, so K-SVD fulfills homogeneity with high probability.Once atoms are selected, the cleaned patch is obtained by a linear projection.

D The Differentiability Requirement

Differentiability of the denoiser supports several theoretical properties of RED, including convergence and possible convexity. Existing methods vary in differentiability, and replacing it with directional derivatives leaves key guarantees unresolved.

  • Benefits: Differentiability enables the directional derivative property ∇xf(x)x = f(x), supports passivity, and makes convexity of the regularization term possible.These benefits are presented as consequences of the homogeneity and derivative conditions.
  • Benefits: Under the differentiability requirement, convergence of the fixed-point algorithm is guaranteed.This is listed as a distinct benefit of requiring a differentiable denoiser.
  • Scope boundary: If a denoiser has directional derivatives but is not differentiable, convergence of the fixed-point algorithm and convexity of the regularization remain theoretically unresolved.The paper states that the behavior of the proposed algorithm is then unclear.
  • Existing denoisers: NLM, bilateral filtering, and TNRD are differentiable, whereas EPLL, BM3D, and K-SVD use sharp decisions that may require fuzzy modifications.The proposed modifications would render those methods differentiable.
Loading 1611.02862v3…