PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Image Restoration Using Partially Linear Denoisers

Rihuan Ke, Carola-Bibiane Schönlieb

arXiv:2008.06164v2eess.IVcs.CVcs.LG

TL;DR

Clean ground-truth images are often unavailable or expensive, although deep restoration methods commonly rely on them for training. The paper trains partially linear denoisers from noisy images under a conditional zero-mean noise assumption, and reports denoising quality close to supervised counterparts while outperforming some ground-truth-free approaches.

  • Problem

    Conventional deep denoiser training uses a loss such as MSE that requires clean ground-truth images, which may be unavailable or expensive to acquire.

    Source: §I · p. 1

  • Method

    The paper trains structured partially linear denoisers from noisy images using an auxiliary random vector to approximate MSE, assuming conditionally zero-mean noise with known variance.

    Source: §I · p. 2

  • Results

    The approach outperforms some ground-truth-free denoising methods, including SURE-based learning, and achieves quality close to fully supervised baselines.

    Source: §I · p. 2, §V · p. 15

  • Takeaways & Limitations

    The denoisers can be trained end-to-end and applied to restoration tasks without clean images or multiple noisy observations per image.

    Source: §I · p. 2

    The method assumes noise has zero mean conditioned on the image and that its variance is known.

    Source: §I · p. 2

Abstract

from arXiv · show

Deep neural network based methods are the state of the art in various image restoration problems. Standard supervised learning frameworks require a set of noisy measurement and clean image pairs for which a distance between the output of the restoration model and the ground truth, clean images is minimized. The ground truth images, however, are often unavailable or very expensive to acquire in real-world applications. We circumvent this problem by proposing a class of structured denoisers that can be decomposed as the sum of a nonlinear image-dependent mapping, a linear noise-dependent term and a small residual term. We show that these denoisers can be trained with only noisy images under the condition that the noise has zero mean and known variance. The exact distribution of the noise, however, is not assumed to be known. We show the superiority of our approach for image denoising, and demonstrate its extension to solving other restoration problems such as blind deblurring where the ground truth is not available. Our method outperforms some recent unsupervised and self-supervised deep denoising models that do not require clean images for their training. For blind deblurring problems, the method, using only one noisy and blurry observation per image, reaches a quality not far away from its fully supervised counterparts on a benchmark dataset.

I. INTRODUCTION

Deep denoisers can deliver strong restoration but typically need clean training images, which may be unavailable. The paper motivates partially linear denoisers as a noisy-only alternative and situates them among nonlinear denoisers and prior unsupervised approaches.

I. INTRODUCTION

  • CNN denoisers achieve state-of-the-art quality but conventional training relies on clean ground-truth images, limiting use where those images are inaccessible.Training commonly minimizes a distance between reconstructed and clean images, such as MSE.Source: §I · p. 1

B. Unsupervised deep learning for denoising

  • Prior unsupervised approaches rely on multiple noisy observations, independent noise regions, or Gaussian-noise assumptions; J-invariant methods also restrict reconstruction and may require model-based post-processing [20],,, [16], [32].Noise2Noise uses noisy pairs, while SURE provides an unbiased MSE estimate under i.i.d. Gaussian noise.Source: §I · p. 1, B. Unsupervised deep learning for… · p. 3

I. INTRODUCTION

  • Partially linear denoisers decompose output into an image-dependent term, a linear noise-dependent term, and a small residual, enabling learning from noisy images alone.The image-dependent term may be nonlinear, while the noise term is linear.Source: §I · pp. 1–2, Figure 1 · p. 1

III. THE PROPOSED METHOD

The method represents restoration as a partially linear denoiser and trains it using an auxiliary random vector to approximate the clean-image MSE. Its noise assumption is conditional zero mean, and the auxiliary vector need not share the noise distribution.

III. THE PROPOSED METHOD

  • The restoration model is y = Ax + n, where A represents the acquisition process; the method first treats denoising as A = I.It assumes E(n | x) = 0, without requiring noise to be independent of the image or pixel-wise independent.Source: §III · pp. 3–4
  • A partially linear denoiser writes R(y) as g(x) + Ln + e, with possibly nonlinear g, linear L, and a residual e of small variance.The decomposition is not unique; the paper characterizes it with E(e | x) = 0 and a choice of L that minimizes the stated term.Source: §III · p. 4

A. Auxiliary random vectors

  • The auxiliary vector z has conditional mean zero, is independent of n, and matches n's conditional covariance, but need not have the same distribution.Once its variance is known, z can be sampled, for example, from a Gaussian distribution.Source: A. Auxiliary random vectors · p. 4
  • The objective J uses the auxiliary vector to approximate the MSE up to an additive constant for partially linear denoisers, without knowing the image or noise distributions.For α close to zero, the paper argues that the augmented and original noise distributions are close, supporting similar optimal denoising quality.Source: A. Auxiliary random vectors · pp. 4–5

B. Linear minimum mean square error estimator

For linear denoisers, the proposed objective and MSE differ only by a constant independent of the denoiser. Therefore, minimizing the objective yields the linear minimum mean square error estimator under the stated assumptions.

B. Linear minimum mean square error estimator

  • For linear R and noise vectors satisfying assumptions A1–A4, J(R) differs from MSE by a constant independent of R, so minimizing J yields the LMMSE estimator.The objective does not depend on the clean image or require the noise distribution to be known.Source: B. Linear minimum mean square error… · p. 5
  • The proof cancels noise-related terms using the matched conditional covariances of n and z, together with their conditional zero means.The auxiliary vector's scaled contribution compensates for the noise term in the objective.Source: B. Linear minimum mean square error… · p. 5

C. Optimal partially linear denoiser

Partially linear denoisers retain nonlinear image processing while modeling noise response linearly and bounding a residual term. When that residual variance is small, the proposed objective approximates MSE-based denoiser selection, including for constant patches under Poisson noise.

C. Optimal partially linear denoiser

  • Nonlinearity can preserve image structures such as edges, while a small residual term allows denoiser behavior to remain partially linear in the noise.The exact-clean-image denoiser is a special case with zero linear and residual terms, though such denoisers generally do not exist.Source: C. Optimal partially linear denoiser · p. 5
  • With small residual variance, the proposed error provides a good estimate of MSE, and a Lipschitz residual yields a bound independent of α.The general bound contains a 1/α factor; the Lipschitz-based bound does not.Source: C. Optimal partially linear denoiser · p. 6
  • Rε is a convex class of denoisers whose decomposition has a linear noise term and residual variance bounded by ε².The class allows nonlinear image-dependent mappings and includes denoisers satisfying the stated residual-variance bound.Source: C. Optimal partially linear denoiser · p. 6
  • As ε and δ approach zero, the approximation error between the proposed objective’s minimizer and the MSE-based minimizer converges to zero.For images drawn from a single fixed clean image, the error is at most of order ε², regardless of noise distribution or variance.Source: C. Optimal partially linear denoiser · p. 7
  • For constant patches corrupted by pixelwise-independent Poisson noise, plots suggest small residual variance and good approximation of the optimal denoiser for small ε.The example uses 21×21 patches with pixel values uniformly distributed in [0, λ].Source: C. Optimal partially linear denoiser · p. 7, Figure 2 · p. 7

D. Learning a denoiser from noisy samples.

The learning procedure trains a parameterized denoiser from noisy samples using an auxiliary random vector and an empirical loss, while constraining the model to remain partially linear. Perturbed versions of each noisy sample provide a way to penalize the residual behavior without directly evaluating the unknown decomposition.

D. Learning a denoiser from noisy samples.

  • Training uses noisy-image samples and an auxiliary vector whose first two moments, rather than full distribution, are specified.The auxiliary samples can therefore be generated from normal distributions without loss of generality.Source: D. Learning a denoiser from noisy… · p. 7
  • The empirical loss is minimized subject to the denoiser belonging to Rε, implemented through a partial-linearity constraint.Directly evaluating the residual is infeasible because its decomposition terms are unknown and only one noisy observation per image is available.Source: D. Learning a denoiser from noisy… · p. 7
  • The constraint perturbs each noisy image into two versions whose weighted combination equals the original, then penalizes violations of the corresponding output relation.This formulation removes the shared image-dependent and linear terms, leaving a residual-based penalty.Source: D. Learning a denoiser from noisy… · p. 8

E. Learning image deblurring without ground truth images

The framework extends partially linear denoiser learning to linear inverse problems, including deblurring, without clean training images. For deblurring, a model trained on noisy blurry observations and blur kernels can produce test reconstructions without knowing the operator.

E. Learning image deblurring without ground truth images

  • For a linear inverse problem, direct pseudo-inverse reconstruction can be unstable under noise, while denoising the pseudo-inverse output is possible when the forward operator has full column rank.The resulting reconstruction noise has zero mean, and its covariance can be estimated if the original noise covariance is known.Source: E. Learning image deblurring without… · p. 8
  • For deblurring, the model can map noisy blurry measurements directly to clean images, trained using noisy measurements and blur kernels without ground truth images.The blur operator is indirectly encoded in the learned model.Source: E. Learning image deblurring without… · p. 8
  • At test time, the learned model directly computes the deblurred image without knowing the blur operator.The framework requires neither noisy-clean pairs nor a predefined denoiser.Source: E. Learning image deblurring without… · p. 8

IV. EXPERIMENTS

The experiments evaluate partial linearity in standard denoisers and test the proposed approach on synthetic-noise denoising, real microscopy, and blind deblurring. In the partial-linearity comparison, DnCNN has the highest reported PSNR among the three denoisers.

A. Partial linearity of denoisers

  • DnCNN outperforms BM3D by around 0.5 dB and TV by around 1.8 dB in PSNR on the parrot image.Table I reports PSNR for the denoisers, residual variance averaged over pixels, and PSNR for modified denoisers.Source: Table I · p. 9, A. Partial linearity of denoisers · p. 9
  • All three denoisers have ε²/m below 10^-4, and the CNN denoiser’s value is about half the TV method’s.The reported ε²/m is the residual variance averaged over image pixels.Source: A. Partial linearity of denoisers · p. 9, Table I · p. 9
  • TV, BM3D, and DnCNN each show partial linearity at the examined pixel, while TV has relatively larger residual variance and outliers at large |[L n̂]i|.The plotted values use 8,000 noise realizations; Fig. 3 shows denoised images above the corresponding pixelwise plots.Source: A. Partial linearity of denoisers · p. 9, Figure 3 · p. 9

B. Denoising experiments

  • Denoising experiments compare the proposed method with SURE, Noise2Self, Noisier2Noise, and BM3D on BSD68 and Set12 using Gaussian and Poisson noise.The fully supervised DnCNN baseline uses clean images for training, unlike the other neural approaches.Source: B. Denoising experiments · p. 9

1) Gaussian noise:

On BSD68, DPLD is the strongest denoiser trained without ground-truth images, though DnCNN performs best overall. Small errors in the estimated noise variance cause only a small PSNR drop, and the learned denoiser’s structure can help assess variance error.

1) Gaussian noise:

  • DPLD is best among denoisers without ground truth on BSD68, trailing DnCNN by 0.14 dB at σ = 25 and 0.11 dB at σ = 50.It beats Noise2Self, SURE, and Noisier2Noise by 1.6 dB, 0.09 dB, and 0.12 dB at σ = 25, and by 0.98 dB, 0.25 dB, and 0.17 dB at σ = 50.Source: Table II · p. 10, 1) Gaussian noise · p. 10

2) Training with estimated noise variance:

  • Small errors in estimated noise variance produce small PSNR drops, with an estimate 4% below the true value reducing PSNR by less than 0.1 dB.Variance overestimation reduces PSNR more than underestimation, and DPLD remains significantly more accurate than Noise2Self for small errors.Source: 2) Training with estimated noise… · p. 10

1) Gaussian noise:

  • On 12 commonly used images at σ = 50, DPLD beats Noise2Self by 0.91 dB and SURE by 0.43 dB on average, trailing DnCNN by 0.14 dB.DPLD has the best PSNR among denoisers without ground truth on all but Parrot and Barbara.Source: 1) Gaussian noise · p. 10, Table III · p. 11

2) Training with estimated noise variance:

  • The mean of ⟨z, Lz⟩ correlates with variance-estimation error and can help select a denoising model when the noise variance is unknown.It is a small positive number at the true variance and increases when the variance is underestimated, decreasing when it is overestimated.Source: 2) Training with estimated noise… · pp. 10–11

3) Poisson noise:

For Poisson noise, DPLD outperforms Noise2Self and approaches supervised DnCNN, while its performance depends on the partial-linearity constraint and remains robust across noise levels. On real microscopy images, it produces smooth results with slightly more structural detail than the compared methods.

3) Poisson noise:

  • For Poisson noise, the method estimates variance using the noisy pixel value because its scaled value provides an unbiased estimate of the noise variance.The experiments cover λ=60, 30, and 15; the auxiliary-vector variance is computed differently from the Gaussian-noise setting.Source: 3) Poisson noise · pp. 11–12
  • DPLD beats Noise2Self by 1.5, 1.09, and 0.74 dB at λ=60, 30, and 15, respectively, and trails DnCNN by 0.15, 0.21, and 0.23 dB.The comparison is on BSD68 under Poisson noise.Source: Table IV · p. 12, 3) Poisson noise · p. 12
  • Noise2Self produces less-smooth images and tends to remove sharp jagged-edge points, while DPLD and DnCNN recover image details.The authors attribute Noise2Self’s edge behavior to relying on neighboring pixels to restore each pixel.Source: 3) Poisson noise · p. 12, Figure 6 · p. 13

C. Stability with respect to the partial linear constraint and noise levels

3) Poisson noise:

  • DPLD exceeds Noise2Self by more than 0.7 dB in average PSNR on 12 images and trails DnCNN by less than 0.3 dB.The reported comparison is for Poisson noise.Source: 3) Poisson noise · p. 12, Table V · p. 13

D. Denoising real microscopy images

  • On real microscopy images, DPLD outputs look smooth while showing slightly more object-structure detail than BM3D and Noise2Void.BM3D outputs are smooth but contain artifacts, while Noise2Void outputs show graininess in non-constant regions.Source: D. Denoising real microscopy images · p. 14, Figure 10 · p. 14

E. Image deblurring

The deblurring method trains from unpaired observations using a partially linear constraint and an optional proxy loss. On Helen and CelebA, it is competitive with Xia et al.’s unsupervised method and close to the fully supervised baseline, but the method does not work for non-zero-mean noise.

E. Image deblurring

  • Training combines the empirical deblurring loss, a proxy loss, and the partially linear constraint.The proxy loss treats a deblurred version of the observation as ground truth for a synthetic training pair.Source: E. Image deblurring · p. 15
  • With the proxy loss, the method is comparable to Xia et al. and within 0.15 dB PSNR of the fully supervised baseline, while outperforming the listed supervised and unsupervised alternatives.It uses unpaired observations; the reported comparison assumes the blur operator is known during training.Source: Table VI · p. 15, E. Image deblurring · p. 15
  • Adding the proxy loss improves PSNR by 0.35 dB on Helen and 0.31 dB on CelebA.Without the proxy loss, PSNR is 25.65 dB on Helen and 24.78 dB on CelebA.Source: E. Image deblurring · p. 15

V. CONCLUSION

  • The method does not work for non-zero-mean noise, including impulse noise that replaces corrupted pixels with random numbers.It also requires estimates of the noise variance.Source: §V · p. 15

APPENDIX A ADDITIONAL PROOFS FOR SECTION III

The supplied appendix passages state assumptions for propositions and outline proof steps, but do not provide enough intact results to summarize their implications.

APPENDIX A ADDITIONAL PROOFS FOR SECTION III

  • Remark 2 considers conditional covariance of z equal to (1+β) times that of n, with β>−1.The excerpt does not state the resulting expression for J(R).Source: Appendix A · p. 18
  • Proposition 4 assumes conditions (A1)–(A4) and 0<δ<1 for the two denoisers it names.The excerpt does not include the proposition’s conclusion.Source: Appendix A · p. 18
  • Corollary 5 additionally assumes that x follows a delta distribution concentrated at one image.The excerpt does not include the corollary’s conclusion.Source: Appendix A · p. 18

APPENDIX B ADDITIONAL DETAILS FOR SUBSECTION IV-D

The appendix details microscopy denoiser training and dataset-specific noise-variance estimation. It also describes using learned models to refine two datasets’ estimates and comparing whole-image denoising with Noise2Noise’s frame-based approach.

Training specification

  • DnCNN is trained on overlapping 40×40 patches with a step size of 10, random flipping, and dataset-specific γ values.For N2DH-GOWT1 and C2DL-MSC, 80% of the first 80% of patches ordered by average intensity are randomly deleted to focus training on foreground objects.Source: Appendix B · p. 19

Noise variance estimation

  • For N2DH-GOWT1 and C2DL-MSC, the initial variance estimate is based on neighboring-pixel differences in smooth image regions and assumes variance depends on pixel intensity.The smooth regions are identified using a smoothed version of the noisy image.Source: Appendix B · p. 19
  • The estimated variance V(v) follows a linear pattern and is approximated by V(v) ≈ (v − µ)/λ.For N2DH-GOWT1, µ = 0 and λ = 69; for C2DL-MSC, µ = 0.09 and λ = 76, with V(v) = 0 for v ≤ µ.Source: Appendix B · pp. 19–20, Figure 12 · p. 20
  • The method denoises the summed image y, while Noise2Noise trains on half the frames; V(i) estimates the noise variance at each pixel.For TOMO110, the raw dose-fractionated frames have independent noise.Source: Appendix B · p. 20

Noise variance refinement

  • Refinement selects λ values yielding a small positive E(⟨z, Lz⟩), giving final estimates of 69 for N2DH-GOWT1 and 88 for C2DL-MSC; TOMO110 is not refined.The refinement results are shown in Fig. 13; TOMO110’s variance is estimated at individual pixels.Source: Appendix B · p. 20

APPENDIX C ADDITIONAL RESULTS ABOUT DEBLURRING

Additional deblurring visuals show that the model, trained without ground-truth images, recovers major facial structures. Its performance is comparable to the Noise2Noise-based method of Xia et al. [39].

Additional deblurring results

  • Fig. 14 compares additional model outputs with a fully supervised baseline and Xia et al. [39].The passage identifies the comparisons but does not report their relative outcomes.Source: Appendix C · p. 20
  • The model recovers major facial structures without ground-truth images, with performance comparable to the Noise2Noise-based method of Xia et al. [39].The additional visual results are in Fig. 14.Source: Appendix C · p. 20, Figure 14 · p. 21
  • The additional face images are taken from the Helen [18] and CelebA [24] test sets.Source: Figure 14 · p. 21

Related papers

Loading 2008.06164v2…