PaperlayerView exact revision on arXiv

Source-linked AI summary

The Little Engine that Could: Regularization by Denoising (RED)

Yaniv Romano, Michael Elad, Peyman Milanfar

arXiv:1611.02862v3cs.CVmath.NA

TL;DR

The paper asks whether denoisers can regularize inverse problems through an approach more general, simpler, and more stable than Plug-and-Play. RED uses a denoiser in an explicit regularization functional, whose convexity supports global-optimum guarantees when the log-likelihood is also convex.

  • Problem

    The paper asks whether image denoisers can regularize inverse problems through an approach more general, simpler, and more stable than Plug-and-Play.

    Source: §1 · p. 3

  • Method

    RED defines inverse-problem regularization with a structured smoothness penalty, specifically an explicit adaptive Laplacian-based functional built from a denoising engine.

    Source: §1 · p. 3

  • Results

    RED's regularization term is convex, so when the log-likelihood is convex, the proposed algorithms are guaranteed to converge to a global optimum.

    Source: §1 · p. 4

  • Takeaways & Limitations

    RED can incorporate any image denoising algorithm and address general inverse problems.

    Source: §1 · p. 4

    The gradient approach may require many iterations because each denoiser activation updates the likelihood only mildly toward its gradient.

    Source: §4.2 · p. 16

Abstract

from arXiv · show

Removal of noise from an image is an extensively studied problem in image processing. Indeed, the recent advent of sophisticated and highly effective denoising algorithms lead some to believe that existing methods are touching the ceiling in terms of noise removal performance. Can we leverage this impressive achievement to treat other tasks in image processing? Recent work has answered this question positively, in the form of the Plug-and-Play Prior ($P^3$) method, showing that any inverse problem can be handled by sequentially applying image denoising steps. This relies heavily on the ADMM optimization technique in order to obtain this chained denoising interpretation. Is this the only way in which tasks in image processing can exploit the image denoising engine? In this paper we provide an alternative, more powerful and more flexible framework for achieving the same goal. As opposed to the $P^3$ method, we offer Regularization by Denoising (RED): using the denoising engine in defining the regularization of the inverse problem. We propose an explicit image-adaptive Laplacian-based regularization functional, making the overall objective functional clearer and better defined. With a complete flexibility to choose the iterative optimization procedure for minimizing the above functional, RED is capable of incorporating any image denoising algorithm, treat general inverse problems very effectively, and is guaranteed to converge to the globally optimal result. We test this approach and demonstrate state-of-the-art results in the image deblurring and super-resolution problems.

1 Introduction

RED proposes an explicit, image-adaptive denoising-based regularizer for general inverse problems, with flexible optimization and conditions supporting convergence to a global optimum.

1 Introduction

  • A denoising engine maps a noisy image to a same-sized estimate intended to remove noise while preserving underlying edges and textures.The introduction describes the input as an image corrupted by zero-mean white additive Gaussian noise.Source: §1 · p. 2
  • Algorithms from the past decade treat denoising extremely effectively and stably, leading researchers to guess that performance is near optimal, while further improvements seem to yield diminishing returns.The paper identifies leveraging denoising progress to improve other image-processing tasks as a new opportunity.Source: §1 · p. 2
  • P3 uses ADMM to alternate denoising with simpler L2-regularized inverse problems, but its objective may be undefined for arbitrary denoisers and convergence and tuning are problematic.Its dependence on ADMM also limits flexibility in choosing the iterative procedure.Source: §1 · p. 3
  • RED offers an explicit adaptive-Laplacian regularizer for inverse problems and is not tied to a single optimization method, unlike the ADMM-based P3 scheme.Its explicit functional makes the overall objective clearer and better defined, and the framework can incorporate any image denoising algorithm.Source: §1 · pp. 3–4
  • RED penalizes the inner product of an image with its denoising residual; under mild assumptions on f, the regularizer’s gradient is x − f(x).The penalty uses an image-adaptive Laplacian defined by the chosen denoising engine.Source: §1 · p. 3
  • RED permits almost any denoiser, has a convex regularizer, and demonstrates state-of-the-art results for deblurring and single-image super-resolution.When the log-likelihood is also convex, the proposed algorithms are guaranteed to converge to a global optimum; the median filter is one example denoiser.Source: §1 · p. 4

2 Preliminaries

The preliminaries frame inverse problems as MAP optimization with likelihood and prior terms, then explain how P3 uses ADMM splitting to incorporate denoising while exposing tuning and objective-function limitations.

2.1 Inverse Problems as Optimization Tasks

  • MAP estimation converts inference into minimizing a log-likelihood term plus a prior, or regularizer, which can stabilize solutions when the likelihood alone is insufficient.The prior term is weighted by λ, which expresses confidence in it.Source: §2.1 · pp. 4–5
  • For y = Hx + e with white Gaussian noise of variance σ², the likelihood is a squared-error term; other noise distributions change that form across a broad family of inverse problems.Examples include denoising, deblurring, super-resolution, demosaicing, tomography, optical-flow estimation, and segmentation.Source: §2.1 · pp. 5–6

2.2 The Plug-and-Play Prior (P 3) Approach

  • ADMM splits the optimization into sequential updates, with the v-step equivalent to denoising x + u at Gaussian noise power σ² = 1/β; P3 substitutes a chosen denoiser for an explicit prior.The x-step is a simpler inverse problem with L2 proximity regularization, and u is updated as the Lagrange multiplier.Source: §2.2 · pp. 6–7
  • P3 is difficult to tune because the denoiser noise level depends on λ/β and β is often increased iteratively; an arbitrary denoiser may also lack an underlying prior or cost function.Some convergence analyses establish only convergence to a steady state, while global guarantees require additional denoiser conditions.Source: §2.2 · p. 7

3 The Image Denoising Engine

The paper defines a denoising engine broadly as an image-to-image function intended to remove noise, then states differentiability, local homogeneity, and strong passivity conditions for its analysis. It tests whether several established denoisers satisfy homogeneity and passivity.

3 The Image Denoising Engine

  • A denoising engine is any function mapping an image in [0, 255]^n to a same-sized image, intended to denoise it, whether MAP-based, MMSE-based, or otherwise.Image denoising is framed as the inverse problem y = x + e, with white Gaussian noise of variance σ2.Source: §3 · p. 8

3.1 Conditions and Properties of f(x)

  • The analysis requires differentiability, near-unit local homogeneity, and strong passivity of the denoiser’s Jacobian, interpreted as restricting input-norm magnification.Local homogeneity is relaxed to scales with |c − 1| ≤ ϵ for small ϵ; row-stochasticity is described as desirable but unnecessary.Source: §3.1 · pp. 8–10
  • For algorithms with a two-phase structure, the denoiser can be written as an image-dependent pseudo-linear filter, a formulation suggested in.Examples include NLM, kernel regression, and K-SVD; the paper notes that this form is not universal, while its framework applies to any denoiser meeting the stated conditions.Source: §3.1 · pp. 9–10
  • The pseudo-linear operator W(y) may depend on y, but its sensitivity to perturbations is negligible, making it essentially constant on a perturbed image.The authors derive this stability consequence from homogeneity.Source: §3.1 · p. 10

3.2 Denoisers Obeying the Above Conditions

  • 2.95e-4 (K-SVD), 3.38e-4 (BM3D), 1.38e-4 (NLM), 1.46e-4 (EPLL), and 9.51e-5 (TNRD) are reported homogeneity-deviation standard deviations.The tests used Peppers, ϵ = 0.01, and σ = 5; the authors report an approximate equality f((1+ϵ)x) ≈ (1+ϵ)f(x).Source: §3.2 · p. 11, Figure 1 · p. 12
  • Values ≤1 for K-SVD, BM3D, NLM, EPLL, and TNRD verify the passivity of these tested filters.The tests used the Cameraman image, σ = 5, and iterations set to an accuracy of 1e-5.Source: §3.2 · p. 13

4 Regularization by Denoising (RED)

RED defines an image-adaptive regularizer from a denoiser’s residual and uses its gradient to formulate iterative methods for inverse problems. The section presents steepest-descent, ADMM, and fixed-point approaches, and fixes the denoiser strength at a small value.

4.1 The Image-Adaptive Laplacian

  • The regularizer is small when the image is nearly a fixed point of the denoiser or when its residual has little cross-correlation with the image.The latter condition corresponds to a residual behaving like white noise or containing no image elements.Source: §4.1 · pp. 13–14
  • RED generalizes an image-adaptive Laplacian into a regularizer based on the residual x − f(x), where f may be any denoising filter.The framework is motivated by denoisers expressible as W(x)x, but the generalized residual does not require that pseudo-linear form.Source: §4.1 · pp. 13–14
  • Under the stated directional-derivative assumption, RED’s gradient is ∇xℓ(y, x) + λ(x − f(x)), requiring one denoiser activation per evaluation.The paper notes that a finite-difference approximation can instead require two activations.Source: §4.1 · pp. 14–15

4.2 Deploying the Denoising Engine for Solving Inverse Problems

  • The proposed iterative solvers include steepest descent, ADMM, and a fixed-point strategy, with each iteration applying the denoiser once or more.Gradient methods may require many iterations because each activation updates the likelihood only mildly; the next two methods treat it more aggressively.Source: §4.2 · pp. 15–17
  • In RED’s ADMM, the explicit prior makes the v-update an optimization rather than a denoiser substitution; early termination can reduce its multiple denoiser activations.The passage says RED is guaranteed to converge to the global optimum of the overall penalty function, unlike the P^3 method.Source: §4.2 · pp. 16–17
  • The method fixes the denoiser’s noise parameter σf at a small value to define a clear regularizer and favor a final image stable under weak denoising.The paper notes that the parameter’s influence and automatic setting require further study.Source: §4.2 · p. 19

5 Analysis

The analysis establishes conditions for RED’s convexity and global convergence, examines a more delicate alternative prior, and shows that exact RED–P3 equivalence is restrictive.

5.1 Convexity

  • A stable denoiser Jacobian makes ρL convex; with a convex log-likelihood, RED converges to the global optimum, whereas P3 is known at best to reach a stable point.The Hessian is I − ∇xf(x); for NLM and K-SVD denoisers, the associated Laplacian is positive semidefinite when W’s eigenvalues lie in [0, 1].Source: §5.1 · p. 21

5.2 An Alternative Prior

  • An alternative residual-small prior is related to RED’s choice; for pseudo-linear denoisers it regularizes with the squared Laplacian, a delicate fourth-order option left for future work.The paper notes that fourth-order regularization has been considered in the literature.Source: §5.2 · pp. 21–22

5.3 When is Plug-and-Play-Prior = RED ?

  • For pseudo-linear W, exact RED–P3 equivalence is restrictive: residuals must match up to a constant across y and f(y), and W’s eigenvalues must be 1 or β/λ.The condition compares the denoising residual on the input with the residual on its filtered image.Source: §5.3 · p. 23
  • RED and P3 have roughly equal per-iteration cost under the stated implementations, while RED supplies a defined penalty, convexity-based convergence claims, and simpler tuning.This cost comparison assumes one iteration in ADMM’s v update; fixed-point RED has the same cost as P3 per iteration.Source: §5.3 · pp. 23–24

6 Results

With TNRD, RED achieves competitive deblurring and state-of-the-art super-resolution results, while its robustness tests show more stable behavior than P3 under parameter changes.

6 Results

  • The experiments plug either a median filter or the published TNRD model into RED; the TNRD model was pretrained for white Gaussian noise at level 5, and parameters were manually set for best results on the tested images.The setup demonstrates RED with an arbitrary denoiser while avoiding retraining TNRD for each restoration task.Source: §6 · p. 24

6.1 Image Deblurring

  • With TNRD, RED matches P3 in deblurring quality and is competitive with NCSR and IDD-BM3D; its three variants obtain similar PSNR.The steepest-descent variant requires more steps and denoiser applications than ADMM or fixed-point, while RED parameters remain fixed and are simpler to tune than P3 parameters.Source: Table 1 · p. 25, §6.1 · p. 30
  • Across the two denoisers and three RED optimization schemes, all methods converge at different rates: steepest-descent is slowest, while fixed-point beats ADMM when both use one denoiser application per iteration.ADMM is fastest when using m2 = 3 denoiser applications per iteration; the fixed-point method then requires only one application per iteration.Source: Figure 5 · p. 27, §6.1 · p. 30

6.2 Image Super-Resolution

  • With TNRD, RED's three variants and P3 deliver similar super-resolution performance, with state-of-the-art restoration and sharp, clear edges.The reported comparison is evaluated by PSNR on the luminance channel; the P3 runs for a manually tuned fixed number of iterations because it does not converge.Source: Table 4 · p. 31, §6.2 · p. 31
  • With a median-filter denoising engine, the method improves over bicubic interpolation by 2.19 dB on average; for Gaussian blur, the integrated median filter performs similarly to total-variation deblurring.The gap between this simplistic approach and state-of-the-art methods is substantial.Source: §6.1 · pp. 25, 30, §6.2 · p. 31

6.3 Robustness to the Choice of Parameters

  • RED's final PSNR remains the same across ADMM β settings, although β and m2 affect convergence; more accurate Part II solutions improve the convergence rate.By contrast, P3 shows unstable PSNR curves, erratic behavior under α and β0 changes, and apparent convergence caused by sufficiently large βk.Source: §6.3 · p. 34
  • Changing the denoiser noise level σf changes RED's regularizer and final PSNR, as expected when the objective changes.The experiments show this behavior for the fixed-point method; changing the regularizer weight λ is expected to have a similar effect but was omitted.Source: §6.3 · p. 34

7 Conclusions

RED uses a denoising engine within the regularization term for general inverse problems while keeping differentiation tractable using the denoiser itself. The conclusions highlight RED’s properties and competitive experimental performance, while identifying generalization and parameter selection as open questions.

7 Conclusions

  • RED places a denoising engine in the regularizer, whose differentiation remains tractable using the denoiser itself rather than its derivative.The paper discusses appealing properties including convexity and a relation to advanced Laplacian smoothing.Source: §7 · pp. 34, 36
  • Experiments demonstrate the validity of RED’s regularization strategy and competitive performance, in contrast with plug-and-play-prior.Figure 10 shows average super-resolution PSNR over iterations for the test images in Table 4.Source: Figure 10 · p. 35, §7 · p. 36
  • RED can use functions with available directional derivatives, including a median filter unrelated to Gaussian denoising, but broader generalization remains future work.The paper leaves investigation of more general and more daring regularization functionals open.Source: §7 · p. 36
  • The influence of fixed σf and whether a varying σf strategy improves performance remain open; tuning it with SURE is suggested for future research.The paper does not address σf’s influence on overall performance.Source: §7 · pp. 34, 36

Appendices · A Can We Mimic any Prior?

The appendix asks whether a given prior can be represented by a denoiser and shows how to construct one for 2-homogeneous priors. It also identifies examples and limitations of this approach.

A Can We Mimic any Prior?

  • For a given prior ρ(x), the appendix seeks an effective denoiser f(x) that reproduces it through the denoiser-defined regularizer.Differentiating the defining relation yields a denoiser explicitly defined in terms of the prior.Source: A Can We Mimic any Prior? · pp. 36–37
  • Because the denoiser is assumed homogeneous of degree 1, a prior it mimics must be 2-homogeneous.The construction proceeds under this condition.Source: A Can We Mimic any Prior? · pp. 36–37
  • The eligible class includes squared total variation, squared norms of the form ρ(x) = ∥Ax∥², and other 2-homogeneous priors.More generally, k-homogeneous functions can be raised to power 2/k; 1-homogeneous order statistics such as maximum, minimum, and median can be squared.Source: A Can We Mimic any Prior? · pp. 36–37
  • The resulting denoiser resembles one steepest-descent step for denoising regularized by ρ(x), using step size 1 and initial condition x0 = y.The appendix relates this interpretation to the denoising expression obtained above.Source: A Can We Mimic any Prior? · p. 37
  • Not every prior is covered: the sparsity-inspired L0 regularizer is an exception to this construction.The appendix also notes that the denoisers satisfying its conditions 1 and 2 form a more general class than proximal mappings, whose related expression is discussed in.Source: A Can We Mimic any Prior? · p. 37

B Kernelizing Priors

The section shows how a prior ρ(x) can imply a pseudo-linear denoiser f(x) = W(x)x, with the prior’s Hessian giving the filter’s Laplacian. It then characterizes convexity, homogeneity, and stochasticity properties of the resulting filter.

B Kernelizing Priors

  • The gradient of a pseudo-linear filter is its weight matrix, and the prior implies W(x) = I − H(ρ(x)), so L(x) = H(ρ(x)).This explicitly expresses the implied filter weights and Laplacian in terms of the prior.Source: B Kernelizing Priors · p. 38
  • If ρ(x) is convex and η(L(x)) ≤ 1, then L(x) is PSD, W(x) has spectral radius at most 1, and W(x) is PSD.These properties include the strong passivity condition and a desirable PSD weight matrix.Source: B Kernelizing Priors · pp. 38–39
  • For a 2-homogeneous prior, H(ρ(x)) and W(x) are invariant to image scaling, making the applied filter W(x)x 1-homogeneous.The section notes that this behavior is consistent with earlier conditions.Source: B Kernelizing Priors · p. 39
  • The filter is not generally row-stochastic, but defining the prior on Dx makes the resulting filter row- and column-stochastic because D1 = 0.The gradient-based prior is a change of variables from x to Dx; its Hessian vanishes on the constant vector.Source: B Kernelizing Priors · p. 39

C More on Homogeneity

Homogeneity of a filter was important to the paper’s key results, and this section discusses whether popular, familiar filters satisfy the condition.

C More on Homogeneity

C.1 Non-Local Means and Bilateral Filter

The section explains how kernel choice determines denoiser homogeneity and argues that Gaussian-kernel NLM and bilateral filters are nearly 1-homogeneous under small image rescalings.

C.1 Non-Local Means and Bilateral Filter

  • A general denoising filter can be built by normalizing a symmetric positive-semidefinite kernel K(x_i, x_j) that is nonnegative.Source: §C.1 · p. 40
  • For a homogeneous kernel of any degree p, normalization makes the filter matrix invariant to image scaling; denoiser homogeneity therefore depends on the kernel.Homogeneous polynomials and other less commonly used image-processing kernels are examples.Source: §C.1 · p. 40
  • For NLM, near-zero sigma makes all weights approximately 1/n and scale-invariant, while away from zero image scaling can be absorbed into sigma.The bilateral filter’s spatial-distance weights are also invariant to scaling image values.Source: §C.1 · pp. 40–41
  • With epsilon = 1/n, sufficiently large sigma makes epsilon phi(sigma) negligible, supporting negligible weight changes under scaling by 1 + epsilon.The passage estimates phi(sigma) and the derivative contribution as on the order of n/sigma^2.Source: §C.1 · pp. 41–42
  • Gaussian kernels are not homogeneous, yet NLM and bilateral filters are almost exactly 1-homogeneous because small image rescalings change their weights very little.The claim concerns rescalings by 1 + epsilon for sufficiently small epsilon.Source: §C.1 · pp. 40, 42

C.2 Tikhonov Regularizer and Wiener Filtering … Sparsity-Inspired Denoisers: K-SVD

The appendix examines how input scaling affects Tikhonov and Wiener filters and patch-based denoisers, finding homogeneity for linear filtering and stability of nonlinear selections under stated conditions.

C.2 Tikhonov Regularizer and Wiener Filtering

  • The linear Tikhonov denoiser satisfies homogeneity: scaling the input by (1 + ϵ) scales the denoised output by the same factor.Source: §C.2 · p. 42
  • Image-adaptive Wiener filtering satisfies homogeneity under mild conditions: when λ ≪ 1 and ∥σ^2W B^T B∥ < 1, its second-order term is negligible.The filter W(ϵ) changes linearly with ϵ.Source: §C.2 · pp. 42–43

C.3 Patch-Based Denoising

  • Patch-based denoisers process overlapping patches with a nonlinear prior, then linearly filter and average them; scale-stable nonlinear decisions yield overall homogeneity.Source: §C.3 · p. 43

Gaussian Mixture Model

  • For tied GMM model scores, choosing the model with smaller log |Σk| preserves the selected model under infinitesimal input scaling.Source: Gaussian Mixture Model · p. 44
  • In EPLL, GMM denoising selects a Gaussian model before Wiener filtering, and its nonlinear model selection is stable to small input scaling when the model scores are distinct.For a strict score gap, sufficiently small ϵ preserves the selected model.Source: Gaussian Mixture Model · pp. 43–44

Sparsity-Inspired Denoisers: K-SVD

  • With high probability, OMP (and thus K-SVD) is homogeneous: scaling preserves its selected atoms except at a rare stopping-boundary case, and the cleaned patch is a linear projection.The exceptional case occurs when ∥yi − Dαi∥^2 = nσ^2.Source: Sparsity-Inspired Denoisers: K-SVD · p. 45

D The Differentiability Requirement

Differentiability of the denoiser enables key properties of RED, including a directional-derivative identity, passivity, fixed-point convergence guarantees, and possible convexity of the regularizer. The section considers weaker directional-derivative conditions and whether differentiability is practical for existing denoisers.

D The Differentiability Requirement

  • Differentiability enables the identity ∇xf(x)x = f(x), grounds passivity, guarantees fixed-point algorithm convergence, and makes convexity of ρL(x) possible.The convexity claim emerges from Condition 2.Source: D The Differentiability Requirement · pp. 45–46
  • The identity and passivity requirements could use directional derivatives instead, with the passivity alternative assuming a symmetric ∇xf(x).The section presents this as a weaker alternative to differentiability.Source: D The Differentiability Requirement · p. 46
  • If differentiability is replaced by existence of all directional derivatives, convergence and regularizer convexity may be lost, leaving the algorithm’s theoretical behavior unclear.Whether those properties can instead rely on directional derivatives remains an open question.Source: D The Differentiability Requirement · p. 46
  • NLM, the Bilateral filter, and TNRD are differentiable, while EPLL, BM3D, and K-SVD could be made differentiable by fuzzy-decision ε-modifications with little behavioral change.The difficulty for EPLL, BM3D, and K-SVD comes from sharp decisions in their nonlinear parts.Source: D The Differentiability Requirement · p. 46

Related papers

Loading 1611.02862v3…