PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise2Noise Revisited: Training Pair Distributions Dominate Loss Choice in Self-Supervised Denoising

Dingyan Shang, Zhenyu Xu, Youting Wang, Bonan Shen, Bowen Liu

arXiv:2609.16788v1cs.LGcs.CVeess.IV

TL;DR

Noise2Noise can train denoisers without clean references, but the source of its cross-noise robustness and L1’s edge over L2 remain open questions. The paper tests loss, sparsity, and training-pair distribution, finding small synthetic-noise loss differences but much larger gains from matching SIDD training pairs to the test domain.

  • Problem

    Noise2Noise avoids clean targets, but how much its cross-noise robustness comes from the loss—and whether L1’s edge reflects optimization or parameter sparsity—remains unresolved.

    Source: §I · p. 1

  • Method

    The study compares MSE, MAE, and MSE+Lasso under controlled training and evaluates N2N on synthetic noise and on SIDD noisy pairs without using ground truth.

    Source: §I · p. 1, B. Training Distributions · p. 3

  • Results

    +9.4 to +11.0 dB over noisy input comes from SIDD noisy-pair training, versus +0.8 to +3.7 dB for synthetic-Gaussian-trained N2N on SIDD; synthetic MAE’s edge over MSE remains below 1 dB.

    Source: §I · p. 1, A. What the Data Confirms and Refutes · p. 6

  • Takeaways & Limitations

    On the tested SIDD setting, the training-pair distribution matters more than loss choice, and paired noisy captures can support N2N training without clean labels.

    Source: §VII · pp. 7–8

    The study uses a single consumer GPU and depth-4 U-Net, has partial seed replication, and does not test whether fixed-σ training separates training heterogeneity from MAE’s synthetic-noise edge.

    Source: C. Limitations · p. 7

Abstract

from arXiv · show

Noise2Noise (N2N) trains denoisers on pairs of independently corrupted observations, eliminating clean references. We stress-test two natural conjectures about why the L1 loss outperforms L2 here. First, the hypothesis that the L1 loss confers robustness via parameter sparsity confuses the loss with Lasso regularization: an explicit Lasso penalty produces the predicted sparsity yet fails to reproduce L1's cross-noise behavior, while L1- and L2-trained weight distributions are indistinguishable. Second, the population optima of the two losses coincide exactly for symmetric signal posteriors and nearly so for concentrated ones. Measured differences are therefore dominated by optimization dynamics (bounded-influence gradients), which we probe with gradient statistics and contaminated-target training. On Kodak24 with five synthetic noise families, the L1 loss holds a statistically significant edge over L2, below 1 dB PSNR, holding across three seeds on 13 of the 14 noise columns. On real camera noise the loss is not the decisive variable in distribution: on official SIDD validation blocks, synthetic-Gaussian-trained N2N models gain only 0.8 to 3.7 dB over the noisy input regardless of loss, while retraining on SIDD's own noisy pairs, never reading ground truth, gains 9.4 to 11.0 dB, far ahead of BM3D. All metrics are on raw network outputs, and the study makes no leaderboard claim. The training pair distribution, not the loss, carries the inductive bias. That design rule applies wherever clean references are unobtainable, from microscopy to industrial inspection sensors.

I. INTRODUCTION

Noise2Noise trains on independent noisy observations, addressing settings where clean references are costly or unobtainable. This paper examines whether its cross-noise robustness comes from the loss or from weight sparsity, and situates the question among self-supervised denoising and restoration-loss research.

I. INTRODUCTION

  • N2N uses two independent noisy observations of the same signal, avoiding the clean/noisy pairs required by supervised denoisers.This is motivated by applications such as microscopy and industrial nondestructive testing, where a noise-free reference may be physically unobtainable.Source: §I · p. 1
  • The study asks how much cross-noise robustness comes from the loss and whether L1’s reported restoration edge reflects optimization or parameter sparsity.It distinguishes the L1 residual loss from L1 weight regularization and tests their effects separately.Source: §I · p. 1
  • The authors frame the L1–L2 population-optimum gap as absent for symmetric signal posteriors and small for concentrated ones, leaving finite-sample optimization dynamics as the measurable effect.Their controls report that Lasso produces sparsity without reproducing MAE’s cross-noise profile.Source: §I · p. 1
  • On SIDD validation blocks, synthetic-Gaussian-trained N2N gains only +0.8 to +3.7 dB over noisy input, while SIDD noisy-pair training gains +9.4 to +11.0 dB.The real-pair training uses no ground truth; the study also reports a content-matched split of the approximately 8 dB gap into scene-statistics and noise-distribution contributions.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior self-supervised approaches include blind-spot methods, methods that synthesize noisy pairs, and approaches designed to address real-noise distribution mismatch.The paper presents its real-pair N2N experiment as a controlled probe of whether loss choice or training-pair distribution carries the inductive bias, not as a competition with those systems.Source: §II · p. 2
  • Earlier restoration-loss work compared L2, L1, and SSIM-based losses and attributed L1’s edge to optimization behavior; Charbonnier and Huber losses interpolate between L1 and L2 regimes.The paper treats this attribution as a question to test in its N2N setting.Source: §I · p. 1, §II · p. 2

III. THEORY: WHAT THE LOSS CAN AND CANNOT DO

For symmetric zero-mean training noise, MSE targets the posterior mean, while MAE targets the posterior median. These optima coincide for symmetric signal posteriors and approximately coincide when the posterior is concentrated relative to the noise.

A. Likelihood View and Its Limit

  • MSE’s conditional optimum is exactly the posterior mean when target noise is independent, symmetric, and zero-mean.Under these conditions, E[x2 | x1] = E[s | x1].Source: A. Likelihood View and Its Limit · p. 2
  • MAE’s conditional optimum is the median of the noisy-target posterior, which need not equal its mean when the signal posterior is skewed.Convolving a skewed signal posterior with symmetric noise does not necessarily make it symmetric.Source: A. Likelihood View and Its Limit · p. 2
  • The optima coincide exactly for symmetric signal posteriors and approximately for concentrated posteriors, while bounded MAE-gradient influence can distinguish finite-sample optimization.MSE gradients grow with residual size, so rare large residuals can dominate updates; the measured sub-dB gaps are consistent with near-coincident optima.Source: A. Likelihood View and Its Limit · p. 2

B. The L1-Loss-versus-L1-Regularization Confusion

The L1 loss penalizes pixel residuals, whereas Lasso regularization penalizes weights and produces sparse fixed points. The paper distinguishes these mechanisms and notes that range clamping makes the zero-mean-symmetry premise approximate at saturation.

B. The L1-Loss-versus-L1-Regularization Confusion

  • L1 loss penalizes pixel-domain residuals, while Lasso adds a penalty on weights and has sparse-weight fixed points.The residual-loss gradient flows to activations, not directly to weights.Source: B. The… · p. 2
  • The paper tests whether explicit Lasso regularization reproduces the cross-noise behavior associated with the L1 loss.The passage identifies this experiment as a direct test of the dissociation between residual loss and weight sparsity.Source: B. The… · p. 2
  • Clamping training pairs to [0, 1] truncates noise asymmetrically at saturated pixels, making the zero-mean-symmetry premise approximate at the range bounds.Source: B. The… · p. 2

C. Hypotheses

The hypotheses concern MAE-class cross-noise generalization, the role of weight sparsity, and failures when test noise violates the additive zero-mean assumption. A fixed U-Net is trained on synthetic and real noisy pairs to test these questions.

C. Hypotheses

  • The hypotheses predict that MAE-class losses generalize no worse than MSE to additive heavy-tailed test noise, sparsity is neither necessary nor sufficient for their profile, and matching pair distributions helps when assumptions fail.The proposed failure settings include high-variance multiplicative speckle and real signal-dependent sRGB camera noise.Source: C. Hypotheses · p. 3

A. Architecture

  • All loss variants use the same depth-4 U-Net with base width 48, batch normalization, ReLU, and a global residual connection.Source: A. Architecture · p. 3

B. Training Distributions

  • Synthetic training uses independent Gaussian-corrupted crops from 400 CBSD400 images, while real training uses paired SIDD-Medium shots from 128 training scenes without reading ground truth.The deterministic real-data split holds out 32 scenes; training and evaluation use native resolution.Source: B. Training Distributions · p. 3

C. Losses

The study compares four pure losses and an explicit Lasso sparsity control under matched training budgets. It evaluates synthetic and real noisy-pair denoising across specified noise families, datasets, metrics, and baselines.

C. Losses

  • The comparison includes MSE, MAE, Charbonnier, Huber, and MSE+Lasso, with matched training budgets of 50 epochs × 5000 crops and approximately 15.6k steps.Huber uses δ=0.05; the Lasso control uses α=10^-5 on convolutional weights.Source: C. Losses · p. 3

D. Evaluation Protocol

  • The synthetic evaluation uses full-resolution Kodak24 with Gaussian, impulse, salt-and-pepper, Poisson, and speckle noise settings.The listed levels include Gaussian σ∈{15, 25, 50}, impulse p∈{0.1, 0.3, 0.6}, and salt-and-pepper p∈{0.05, 0.1}.Source: D. Evaluation Protocol · p. 3
  • Real-noise evaluation uses official SIDD validation blocks and 32 held-out scenes, with PSNR, SSIM, and LPIPS measured on raw outputs.Validation blocks are averaged per image for testing (n=40); held-out scenes contribute two separately scored shots each (n=64).Source: D. Evaluation Protocol · p. 3
  • The evaluation compares against color BM3D and Noise2Void, with Noise2Void trained on both synthetic images and real SIDD noisy shots.The real-shot Noise2Void training never reads ground truth.Source: D. Evaluation Protocol · p. 3

A. Cross-Noise Generalization

Across synthetic noise settings, Noise2Noise variants fare differently by noise family, while MAE's advantage over MSE is small but persists across most columns and seeds. On real camera noise, the loss ordering reverses for synthetic-trained models, and bounded-influence behavior does not guarantee out-of-distribution robustness.

A. Cross-Noise Generalization

  • +10.0 to +10.5 dB: additive-Gaussian-trained N2N improves over the noisy input at speckle variance v=0.5, contrary to the predicted collapse.The speckle noise is conditionally additive zero-mean Gaussian, with signal-proportional standard deviation inside the trained family’s spatially modulated variance.Source: A. Cross-Noise Generalization · pp. 3–4
  • 7–9.5 dB: N2N variants lead BM3D on 10% impulse and both salt-and-pepper columns, while all methods fall below 17 dB at 60% impulse.At 60% impulse, the best learned method scores 16.87 dB versus BM3D at 16.80 dB.Source: A. Cross-Noise Generalization · p. 3, Table I · p. 4

B. The Loss-Function Effect

  • On every synthetic column, MAE beats MSE by +0.09 to +0.78 dB PSNR at seed 0; the advantage survives all three seeds on 13 of 14 columns.The exception is the heaviest speckle level, where one seed reverses the sign and the deviation is about four times the typical column’s.Source: B. The Loss-Function Effect · p. 4, Table III · p. 5
  • At heaviest speckle, seed spread rises as the mean MAE–MSE gap shrinks, making that loss effect inseparable from seed noise.Across speckle levels, seed spread rises monotonically from 0.02 to 0.22 dB as the mean gap more than halves.Source: B. The Loss-Function Effect · p. 4, Table III · p. 5
  • The MAE advantage concentrates in PSNR and SSIM and largely vanishes in LPIPS, contrary to the predicted perceptual concentration.The authors report that the loss effect is much smaller than training-distribution effects.Source: B. The Loss-Function Effect · pp. 4–5
  • Replacing 5% of target pixels with salt-and-pepper extremes costs MSE 0.5–1.7 dB on Gaussian columns, versus at most 0.6 dB for MAE.At σ=25, the respective losses are 1.04 dB and 0.07 dB; the authors interpret this as consistent with bounded-influence gradients.Source: B. The Loss-Function Effect · p. 5
  • On real-noise columns, synthetic-trained MSE beats MAE by 0.62 dB on official validation blocks and 0.58 dB on held-out scenes.The authors attribute a live statistical component to signal-dependent clipped noise, where the conditional mean–median gap is nonzero and PSNR rewards the mean.Source: B. The Loss-Function Effect · p. 5

C. Weight Histograms: the Sparsity Hypothesis Fails

Explicit Lasso regularization creates strong near-zero weight concentration, but MAE and MSE have indistinguishable weight distributions, and the sparse model follows MSE rather than MAE.

C. Weight Histograms: the Sparsity Hypothesis Fails

D. The Real-Noise Domain Gap, and Closing It

On real SIDD noise, retraining N2N on the noisy-pair distribution yields much larger gains than synthetic-Gaussian training, while real-pair models can fail on unseen synthetic noise.

D. The Real-Noise Domain Gap, and Closing It

  • Synthetic-Gaussian-trained N2N variants gain only +0.8 to +3.7 dB on official SIDD validation blocks, regardless of loss.The noisy input is 23.66 dB PSNR; the five variants score 24.5–27.4 dB.Source: D. The Real-Noise Domain Gap, and… · p. 5, Figure 2 · p. 6
  • Real-pair N2N scores 33.1–34.6 dB (+9.4 to +11.0 dB over the input), 8.2–9.7 dB ahead of BM3D (24.9 dB).The five losses have a 1.5 dB seed-0 spread, small relative to the roughly 8 dB separation between training regimes.Source: D. The Real-Noise Domain Gap, and… · p. 5, Figure 2 · p. 6
  • A content-matched control assigns about 5.0 dB of the 7.98 dB MSE regime gap to scene statistics and about 3.0 dB to matching sensor noise.The split comes from one control path and does not identify the content–noise interaction term.Source: D. The Real-Noise Domain Gap, and… · p. 6
  • Real-pair models transfer poorly to heavy synthetic noise, while Charbonnier and Huber degrade more gracefully than MAE-real out of distribution.At σ=50, MAE-real yields 11.2 dB and rails 19% of output pixels; Charbonnier-real and Huber-real remain near MSE-real.Source: D. The Real-Noise Domain Gap, and… · p. 6

VI. DISCUSSION

The discussion confirms a sub-1 dB synthetic-noise MAE edge and dissociates it from weight sparsity, while finding that matching real noisy-pair distributions matters more than loss choice and that robust losses do not ensure broad robustness.

VI. DISCUSSION

  • MAE exceeds MSE on all synthetic columns, with seed-replicated PSNR significance on 13 of 14 columns, though never by more than 1 dB.The proposed LPIPS rider is refuted: its advantage survives correction only under the heaviest corruption.Source: A. What the Data Confirms and Refutes · p. 6
  • The Lasso model is sparse at 91% near-zero weights yet behaves like MSE, separating sparsity from the MAE-class cross-noise profile.The MSE and MAE weight histograms coincide, while explicit Lasso alone creates the zero-spike.Source: A. What the Data Confirms and Refutes · p. 6
  • Synthetic training forfeits about 8 dB on SIDD relative to matching the training distribution; the recovery is loss-agnostic.The discussion attributes about 5 dB to scene statistics and about 3 dB to noise matching.Source: A. What the Data Confirms and Refutes · p. 6
  • No speckle collapse occurs even at v=0.5, and MAE-real is the most brittle model out of distribution; Fig. 3 illustrates the reversal across impulse-noise Kodak and SIDD scenes.In the figure, synth-trained variants suppress 30% impulse noise that BM3D cannot, while only the real-pair model fully cleans the held-out SIDD scene.Source: A. What the Data Confirms and Refutes · p. 7, Figure 3 · p. 7

B. Removed Confounds and Reproducibility Lessons

Earlier protocol defects distorted apparent comparisons, while the study's limitations include partial seed replication and an untested explanation for MAE's synthetic edge.

B. Removed Confounds and Reproducibility Lessons

  • Earlier protocol defects inflated PSNR by +2.95 dB, capped Kodak24 learned methods near 22 dB, and made SIDD's noisy baseline 34.3 dB versus the official 23.7 dB.The evaluation post-processing was removed, and the training/evaluation channel mismatch and SIDD downsampling were corrected.Source: B. Removed Confounds and… · p. 7
  • Per-channel BM3D with a missing 1/2 factor understated Gaussian-column results by ≈2 dB; reported values use bm3d_rgb and wavelet-domain σ estimation.Source: B. Removed Confounds and… · p. 7

C. Limitations

  • Seed replication is partial: MAE and MSE span the synthetic grid and real-pair losses, but Charbonnier, Huber, and Lasso have one seed in the synthetic cell.Source: C. Limitations · p. 7
  • A single consumer GPU and depth-4 U-Net constrain the evaluation; SIDD train/eval share camera models, while device-stratified splits and DND remain future work.Source: C. Limitations · p. 7
  • Variable σ1, σ2 training makes residuals heavy-tailed; fixed-σ training remains untested as a way to separate training heterogeneity from test-noise robustness.Source: C. Limitations · p. 7

D. Reproducibility

The study reports that training-pair distribution matters more than loss choice for in-distribution denoising, while loss affects out-of-distribution robustness. Reproducibility materials and practical conditions for paired-shot training are also specified.

D. Reproducibility

  • Code, scripts, the deterministic sorted-name SIDD split, and per-image metric dumps are available; models train on one RTX 3080 in about 16 minutes.Seed replicates are included as per-image dumps, allowing Table III to be recomputed without retraining.Source: D. Reproducibility · p. 7

VII. CONCLUSION

  • Switching from synthetic Gaussian to SIDD's own noisy pairs is worth ≈8 dB for every loss, dwarfing in-distribution loss effects.The reported gap comprises ≈5 dB from matched scene statistics and ≈3 dB from the noise distribution.Source: §VII · p. 7
  • Training-pair distribution sets in-distribution performance, while loss governs out-of-distribution robustness; a small quadratic basin restores graceful degradation where pure MAE fails catastrophically.Source: §VII · p. 7
  • Pair-based retraining requires pixel-registered shots of a static signal; fixed-pattern components persist across exposures, making residual-correlation checks and upstream calibration part of the recipe.Dynamic scenes or video, where paired shots are unobtainable, are left to single-image methods.Source: §VII · p. 8
Loading 2609.16788v1…