PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

Michael Elad, Bahjat Kawar, Gregory Vaksman

arXiv:2301.03362v1eess.IVcs.CV

TL;DR

Image denoising is a fundamental, ill-posed imaging problem whose scope has expanded beyond noise removal. This survey traces classical and deep-learning approaches, emphasizing denoisers’ broader roles and the continuing prospects opened by deep learning.

  • Problem

    Image denoising is needed across imaging sensors and remains ill-posed, requiring additional information beyond the noisy observation.

  • Method

    The paper surveys the history of image denoising and related image-processing topics, highlighting classical methods, deep-learning advances, and denoisers’ connections to other imaging tasks.

  • Results

    The survey concludes that highly effective AWGN denoisers are within reach, with later deep neural-network solutions producing especially impressive results, while denoisers have also gained broader imaging roles.

  • Takeaways & Limitations

    Image denoising remains an active research playground with substantial challenges and prospects, including applications beyond denoising itself.

  • Takeaways & Limitations

    The paper leaves open which denoiser and noise level should be used within RED and whether the required mathematical properties will follow.

Abstract

from arXiv · show

Image denoising (removal of additive white Gaussian noise from an image) is one of the oldest and most studied problems in image processing. An extensive work over several decades has led to thousands of papers on this subject, and to many well-performing algorithms for this task. Indeed, 10 years ago, these achievements have led some researchers to suspect that "Denoising is Dead", in the sense that all that can be achieved in this domain has already been obtained. However, this turned out to be far from the truth, with the penetration of deep learning (DL) into image processing. The era of DL brought a revolution to image denoising, both by taking the lead in today's ability for noise removal in images, and by broadening the scope of denoising problems being treated. Our paper starts by describing this evolution, highlighting in particular the tension and synergy that exist between classical approaches and modern DL-based alternatives in design of image denoisers. The recent transitions in the field of image denoising go far beyond the ability to design better denoisers. In the 2nd part of this paper we focus on recently discovered abilities and prospects of image denoisers. We expose the possibility of using denoisers to serve other problems, such as regularizing general inverse problems and serving as the prime engine in diffusion-based image synthesis. We also unveil the idea that denoising and other inverse problems might not have a unique solution as common algorithms would have us believe. Instead, we describe constructive ways to produce randomized and diverse high quality results for inverse problems, all fueled by the progress that DL brought to image denoising. This survey paper aims to provide a broad view of the history of image denoising and closely related topics. Our aim is to give a better context to recent discoveries, and to the influence of DL in our domain.

1. Introduction.

Image denoising evolved from a heavily studied classical problem into a broader field reshaped by deep learning. The survey traces this shift from improved denoisers to denoisers used for inverse problems, image synthesis, and diverse solutions.

  • Image denoising has received exceptional attention for decades because noise removal is both practically necessary and a simple inverse-problem test bed.
  • Classical denoising progressed through Bayesian priors, regularization, robust statistics, wavelets, partial differential equations, sparse modeling, patch methods, and low-rank assumptions.
  • Deep learning introduced better-performing denoisers and broadened the field toward new denoising capabilities and applications.
  • Denoiser engines can regularize inverse problems and support randomized image synthesis through practical sampling from image priors.
  • Posterior-sampling approaches challenge the assumption of unique inverse-problem solutions by producing randomized, diverse, high-perceptual-quality results.
  • The survey presents this history and these recent developments while acknowledging that it cannot cover all published denoising literature.

2. Image Denoising – Background.

Image denoising estimates a clean image from noisy measurements, but the task is ill-posed because suppressing noise can also damage image content. Its broad importance comes from practical imaging needs, extensive research, and denoisers’ roles beyond denoising.

  • A denoiser estimates x from noisy measurements y under a known noise level σ, commonly evaluating performance with mean-squared error.
  • The task is ill-posed: filtering can suppress noise while losing small details, edges, and fine textures.
  • The survey focuses primarily on zero-mean i.i.d. Gaussian noise, while denoising research also addresses Poisson, structured, colored, and unknown noise.
  • Denoising extensions cover bursts, video, specialized imaging modalities, blind noise-level estimation, and completely unknown noise statistics.
  • Nearly 30,000 denoising papers were identified in Clarivate WoS, mostly from the past 25 years, reflecting sustained growth and intensive study.
  • Denoising matters both because imaging sensors routinely produce noise and because denoisers can serve other imaging needs through connections to image priors.

3. Image Denoising – The Classic Era.

Classical denoising formulates recovery through likelihoods, priors, and direct image structures, but the field’s perceived performance ceiling was overturned by deep learning. Modern denoisers now also support broader imaging tasks.

  • For Gaussian noise, maximum-likelihood estimation yields x̂_MLE = y, exposing the ill-posedness of denoising and the need for additional information.
  • Bayesian denoisers use image priors: MAP balances measurement fidelity with a well-behaved image, while MMSE estimates through the posterior distribution.
  • Classical prior design evolved from simple Gaussian and L2 models toward heavy-tailed, sparsity-promoting, nonlinear, and learned representations.
  • BM3D groups similar patches, jointly transforms them, and enforces sparsity; it is among the strongest classical methods for MSE results.
  • After classical progress slowed and prompted claims that denoising was nearly solved, deep learning produced highly effective denoisers and took the lead in noise suppression.
  • Denoisers now serve as priors for inverse problems and as engines for diffusion-based image synthesis, making the field “definitely not” dead.

4. Image Denoising – The Deep Learning Revolution.

Deep learning transformed image denoising by improving performance, broadening training and noise settings, and enabling image-specific adaptation. The field now extends beyond conventional denoising toward richer capabilities, while remaining dependent on data, architecture, and optimization choices.

  • Supervised deep learning: Supervised deep-learning denoising trains a parametric network on clean images and synthetically corrupted versions with known Gaussian noise.Training minimizes a loss between ideal and denoised images, commonly using stochastic gradient descent and back-propagation.
  • Scope and limitations: Modern denoisers can perform strongly, but deployment quality depends on training data, image–training similarity, architecture, and optimization settings.The survey presents deep learning as overturning the belief that denoising performance had reached its limit, while noting the field’s continuing breadth.
  • Better denoisers: Denoising architectures evolved from feed-forward CNNs through UNets to Transformers, with progress illustrated using PSNR on BSD68.Figure 4.1 compares classical and deep-learning methods at σ = 25.
  • Different training schemes: Unsupervised schemes such as Noise2Noise, Noise2Void, Noise2Self, and SURE-based methods train directly from noisy images when clean references are unavailable.These methods are especially relevant when the noise lacks a known analytic structure.
  • Image adaptation: Image adaptation fine-tunes a pretrained denoiser on an individual image, improving performance on content that deviates from the training distribution.The procedure applies the original denoiser, uses its output for a few gradient steps, and then denoises with the updated parameters.

5. Synergy between Classics and Deep Learning.

Classical denoising ideas remain valuable because they can supply structure for compact, interpretable deep networks. Unfolding converts iterative algorithms such as K-SVD and BM3D-inspired processing into learned architectures that retain algorithmic motivation while improving performance.

  • Shared principles: Locality, transform sparsity, and self-similarity remain shared principles linking classical denoisers with convolutional, ReLU-based, and attention-based networks.These properties motivate local processing, sparse representations, and aggregation of related patches.
  • Architecture design: Copying or guessing architectures can achieve strong results but often produces heavy networks with millions of parameters, costly training, and limited explainability.The survey identifies architecture selection as a central unresolved design problem for deep denoisers.
  • Unfolding: Unfolding constructs a feed-forward network by mimicking the computational stages of a well-motivated iterative denoising algorithm.This approach is described as producing concise and explainable networks with learned parameters and activations.
  • Deep K-SVD: Deep K-SVD replaces K-SVD’s sparse-coding step with LISTA, then combines patch decomposition, learned patch denoising, and patch averaging in an end-to-end network.Its trainable components include the dictionary, threshold-related parameters, and a subnetwork that evaluates patch-specific values.
  • Deep K-SVD: The unfolded Deep K-SVD performs much better than the original algorithm, surpassing classical methods and aligning with deep-learning techniques.Its learned dictionary also differs markedly from the offline dictionary used by original K-SVD.
  • LIDIA: LIDIA groups each patch with nearest neighbors and applies learned separable Transform-ReLU-Transform blocks, achieving better performance than BM3D with a small model.The separable design applies transforms within patches and across grouped patches, reducing matrix sizes for training.
  • Summary: Together, unfolding methods provide a constructive route to low-complexity, explainable neural architectures that combine classical algorithms with deep learning.LIDIA adaptation further demonstrates that these learned denoisers can be improved for images departing from general training content.

6. Image Denoising – Migration towards Recent Discoveries.

The paper frames three recent discoveries that extend denoisers beyond noise removal: regularizing inverse problems, synthesizing images, and targeting perceptually perfect inverse-problem solutions.

  • Denoisers can be leveraged constructively to solve general linear inverse problems and open new directions for recovery algorithms and regularization.
  • Denoisers can synthesize high-quality images fairly drawn from a prior probability density function.
  • The paper asks whether denoisers can solve inverse problems while targeting perfect perceptual quality, and gives a positive answer.
  • These discoveries mark a transition in how image processing is perceived and practiced.

7. Discovery 1: Solving Inverse Problems via Image Denoisers.

Denoisers can serve as priors or regularizers for general linear inverse problems, through PnP and RED formulations that replace explicit energy modeling with denoiser operations. The resulting methods apply to tasks including deblurring and super-resolution, while their guarantees depend on algorithmic or denoiser conditions.

  • General linear inverse problems include deblurring, inpainting, demosaicing, super-resolution, tomography, and compressed sensing, with denoising as the special case H = I.
  • PnP: PnP uses variable splitting and ADMM to alternate linear least-squares updates with denoising, avoiding explicit selection or learning of ρ(x).
  • RED: Under differentiability, passivity, symmetric-Jacobian, and local-homogeneity conditions, RED avoids differentiating the denoiser and supports convergence to the global MAP minimizer.
  • Open questions: The appropriate RED denoiser and noise parameter remain open questions concerning MMSE behavior and required structural properties.
  • Examples: PnP and RED produced recovery effects in deblurring and single-image super-resolution despite using denoisers unaware of the underlying task.

8. Discovery 2: Image Synthesis via Image Denoisers.

Diffusion-based image synthesis uses deep-learning denoisers to approximate score functions and generate samples through Langevin dynamics. Annealed Langevin Dynamics progressively moves from high-noise distributions to low-noise ones, although iterative sampling remains slower than GAN generation.

  • Image synthesis seeks a random generator whose outputs follow the prior distribution p(x), traditionally using learned generators such as GANs.
  • Diffusion models use deep-learning denoisers to approximate the score function for image synthesis, unlike GANs and other generative models detached from denoising.
  • Langevin dynamics: Langevin dynamics adds stochastic noise while following the gradient of log-probability, enabling sampling from p(x) under suitable conditions.
  • Limitations: Using the true data-distribution score is hindered by low-dimensional image manifolds and can require many thousands of iterations for practical sampling.
  • Annealed Langevin Dynamics: Annealed Langevin Dynamics applies Langevin iterations across Gaussian-noise distributions with decreasing standard deviations, beginning at large noise and ending at small noise.
  • Annealed Langevin Dynamics: A step size proportional to σ_i^2 matches the smoother, easier-to-sample distributions associated with larger noise levels.
  • Consequences: Diffusion models eventually surpassed GAN performance, but their iterative generation algorithms remained considerably slower than GANs.

9. Discovery 3: High Perceptual Quality Image Recovery.

The section explains why MMSE denoising can sacrifice perceptual quality and presents posterior sampling as a stochastic alternative that produces diverse, high-quality reconstructions. It also connects denoisers to perceptual-quality recovery in general inverse problems and text-to-image generation.

  • The perception-distortion trade-off: MMSE minimizes expected squared error through conditional averaging, which can produce blurry estimates outside the desired image manifold.This creates a tension between distortion minimization and perceptual quality.
  • Posterior sampling: Posterior sampling instead draws denoised images from p(x|y), preserving the image distribution while allowing multiple plausible solutions.The denoiser becomes stochastic rather than a deterministic function of the noisy input.
  • Posterior sampling: Annealed Langevin dynamics can condition score-based sampling on a noisy input, turning an MSE-trained denoiser into a posterior-sampling image denoiser without special model training.The construction uses the score function and the noisy measurement to sample from the posterior.
  • Posterior sampling: Repeated annealed Langevin runs yield valid but diverse high-perceptual-quality reconstructions, whereas direct denoising improves MMSE but produces poorer perceptual quality.The standard-deviation image exposes uncertainty across the sampled solutions.
  • Beyond denoising: Denoisers can also support high-perceptual-quality solutions for general linear inverse problems and enable text-conditioned diffusion models for image synthesis.Text-to-image generation conditions the diffusion process on a textual description of image content.

10. Conclusion.

The conclusion frames image denoising as both an engineering challenge and a source of broader opportunities in image processing and machine learning. It emphasizes denoisers' roles in inverse problems and image synthesis while identifying substantial directions for future work.

  • Conclusion: The paper highlights denoiser design and the use of denoisers for high-perceptual-quality solutions as two connected branches of research.Figure 10.1 summarizes this broader role of denoisers.
  • Conclusion: The authors identify open work in designing appropriate MMSE denoisers and applying denoisers to tasks such as compression and segmentation.They also point to broader opportunities for understanding and creatively using image denoisers.
  • Conclusion: The paper highlights proximity between PnP and RED algorithms and diffusion-based techniques for high-perceptual-quality inverse-problem solutions.This relationship is presented as an unexplored connection in the paper's summary figure.

Appendix A. Derivation of the MMSE Estimation.

The appendix derives the MMSE estimator by formulating expected squared error as a function of the estimator and setting its derivative to zero. The resulting solution is the conditional expectation of the unknown image given the measurement.

  • MMSE derivation: The derivation starts with an ideal image drawn from p(x), a measurement governed by p(y|x), and an estimator ˆx=f(y) minimizing expected mean-squared error.The measurement y is treated as known while the expectation is taken over the unknown image x.
  • MMSE derivation: Setting the derivative of the expected error with respect to f(y) to zero yields the closed-form MMSE estimator.The derivation uses the normalization of the posterior density in the final simplification.
  • MMSE derivation: Bayes' rule rewrites the posterior in terms of the likelihood p(y|x) and prior p(x), the ingredients specified at the start of the model.This connects the estimator to the assumed measurement and image distributions.

Appendix B. A Closer Look at the Evolution of Priors.

The appendix traces image priors from quadratic and smoothness-based regularization toward robust, sparse, self-similar, and learned models. This evolution reflects attempts to avoid oversmoothing and better represent natural-image structure.

  • Classical priors: Classical inverse-problem formulations use priors such as energy regularization, spatial smoothness, and learned Gaussian transforms to regularize otherwise ill-posed least-squares problems.Quadratic priors provide closed-form MAP solutions, but simple choices can yield unsatisfactory visual results.
  • Classical priors: Weighted smoothness adapts the strength of regularization spatially to address the overly smoothed results produced by uniform smoothness penalties.High weights are assigned where smoothness is appropriate, while lower weights preserve less-smooth regions.
  • Non-Gaussian priors: Robust statistics replaces quadratic penalties with L1 or heavy-tailed functions because image derivatives often depart from Gaussianity, requiring iterative minimization.This treats edges and textures as deviations from a Gaussian model rather than uniformly penalizing them.
  • Non-Gaussian priors: Total variation and related PDE-based priors promote edge-preserving, spatially adaptive smoothness through gradient-based regularization.These methods extend beyond a basic L1 penalty on spatial gradients.
  • Learned and sparse priors: Field-of-Experts learns mixtures of functions and weights from image data, while wavelet sparsity promotes concise representations with few dominant coefficients.Wavelet-based denoising can have a closed-form soft-shrinkage solution when the transform is unitary.
  • Self-similarity: Self-similarity regularizes images by enforcing proximity between matching patches, supporting methods such as Non-Local-Means and BM3D.Repeated local structures provide a third source of image simplicity alongside smoothness and sparsity.

Appendix C. Landmark Denoisers over the Years.

The appendix surveys landmark denoisers from classical sparse, probabilistic, and patch-based methods to learned neural architectures. It highlights the transition from model-based priors and iterative optimization toward end-to-end deep networks.

  • Evaluation scope: The appendix compares landmark denoisers using reported PSNR values on BSD68 at σ = 25.The list includes only methods reporting BSD68 results.
  • Classical methods: Classical methods exploit patch sparsity, non-local similarity, learned priors, or low-rank structure through explicit optimization procedures.BM3D groups similar patches into 3D blocks and enforces transform sparsity, while EPLL uses a learned Gaussian-mixture patch prior in MAP estimation.
  • Early learned methods: MPL introduced an effective deep-learning denoiser by filtering overlapping patches with a multilayer perceptron and averaging reconstructed patches.The method applies a fully connected network independently to extracted patches before reconstruction.
  • Learned architectures: TNRD unfolded prior minimization into a trainable network, whereas later CNN methods learned denoising mappings end-to-end from noisy/clean image pairs.TNRD uses a supervised trainable architecture; DnCNN applies convolution, batch normalization, and ReLU layers.
  • Performance transition: DnCNN was the first listed deep-learning method reported to outperform classical algorithms by a considerable gap.Its architecture is trained end-to-end on paired noisy and clean images.
  • Architectural extensions: Later designs incorporate non-local recurrence, wavelet transforms, nearest-neighbor operations, and fractional-order dynamical-system interpretations.These architectures extend CNN denoising with explicit image-structure or dynamical-system mechanisms.

Appendix D. Approximation of the Score Function by an MMSE Denoiser.

The appendix derives a connection between the score of a Gaussian-smoothed image distribution and the optimal MMSE denoiser. The resulting identity enables score approximation by evaluating a denoiser at a small noise level.

  • Approximation: Using a small σ0 and evaluating the denoiser provides an approximation to the score function ∇x log p(x).This connects denoising directly to score estimation for the image distribution.
  • Derivation: Marginalizing the latent image expresses p(y) as the convolution of p(x) with an isotropic Gaussian.Differentiating and dividing by p(y) yields the score-function relation.
  • Denoiser connection: The final relationship identifies the conditional expectation E(x|y), or optimal MMSE denoiser D(y, σ0), within the score expression.The denoiser supplies the conditional mean required by the derived identity.
Loading 2301.03362v1…