PaperlayerView exact revision on arXiv

Source-linked AI summary

AP-BSN: Self-Supervised Denoising for Real-World Images via Asymmetric PD and Blind-Spot Network

Wooseok Lee, Sanghyun Son, Kyoung Mu Lee

arXiv:2203.11799v2cs.CVeess.IV

TL;DR

Self-supervised BSN methods struggle with spatially correlated real-world noise, and directly combining BSN with pixel-shuffle downsampling does not resolve the problem. The paper introduces asymmetric training and inference strides in AP-BSN, supplemented by parameter-free random-replacing refinement, and reports strong performance against self-supervised and unpaired methods.

  • Problem

    Self-supervised BSN relies on pixel-wise independent noise, but real-world sRGB noise is spatially correlated; direct PD-BSN integration is also not straightforward.

  • Method

    AP-BSN uses different pixel-shuffle stride factors for training and inference, with random-replacing refinement added without additional parameters.

  • Results

    AP-BSN is reported to outperform state-of-the-art self-supervised and several unsupervised or unpaired denoising methods by large margins.

  • Takeaways & Limitations

    The proposed framework enables fully self-supervised denoising of real-world sRGB noisy inputs without prior knowledge of the underlying noise.

  • Takeaways & Limitations

    Direct PD-BSN remains unable to restore clean, sharp images reliably across stride factors, with PD2-BSN and PD5-BSN both failing in the reported experiments.

Abstract

from arXiv · show

Blind-spot network (BSN) and its variants have made significant advances in self-supervised denoising. Nevertheless, they are still bound to synthetic noisy inputs due to less practical assumptions like pixel-wise independent noise. Hence, it is challenging to deal with spatially correlated real-world noise using self-supervised BSN. Recently, pixel-shuffle downsampling (PD) has been proposed to remove the spatial correlation of real-world noise. However, it is not trivial to integrate PD and BSN directly, which prevents the fully self-supervised denoising model on real-world images. We propose an Asymmetric PD (AP) to address this issue, which introduces different PD stride factors for training and inference. We systematically demonstrate that the proposed AP can resolve inherent trade-offs caused by specific PD stride factors and make BSN applicable to practical scenarios. To this end, we develop AP-BSN, a state-of-the-art self-supervised denoising method for real-world sRGB images. We further propose random-replacing refinement, which significantly improves the performance of our AP-BSN without any additional parameters. Extensive studies demonstrate that our method outperforms the other self-supervised and even unpaired denoising methods by a large margin, without using any additional knowledge, e.g., noise level, regarding the underlying unknown noise.

1. Introduction

Real-world noise differs from synthetic assumptions, limiting both supervised generalization and self-supervised blind-spot denoising. AP-BSN addresses this challenge with asymmetric pixel-shuffle strides and random-replacing refinement for fully self-supervised real-world sRGB denoising.

  • Motivation: Synthetic-noise denoisers generalize poorly to real-world noise, while paired real-world datasets require substantial labor and controlled acquisition.Noise characteristics also vary across capturing devices, limiting generalization across practical scenarios.
  • Motivation: Large PD strides improve noise independence but destroy image details, whereas small strides preserve structures but violate the independence assumption.This trade-off makes direct PD-BSN integration nontrivial for fully self-supervised real-world denoising.
  • Contributions: AP-BSN uses different pixel-shuffle stride factors during training and inference to balance noise independence with image-detail preservation.The method is designed to denoise real-world noisy inputs without prior knowledge of the underlying noise.
  • Contributions: Random-replacing refinement improves AP-BSN without additional parameters, and the method is reported to outperform self-supervised and several unpaired approaches by large margins.The paper presents AP-BSN as the first self-supervised BSN covering real-world sRGB noisy inputs.
  • Motivation: Blind-spot methods train on noisy images alone but assume pixel-wise independent, zero-mean noise, which real-world sRGB noise violates.Spatial correlation can cause self-supervised methods to learn identity mappings on real-world images.

3. BSN and PD

BSN denoising relies on pixel-wise independent, zero-mean noise, but real-world imaging pipelines create spatially correlated noise. Pixel-shuffle downsampling increases distances between noise signals, though its stride controls a trade-off between independence and preserved detail.

  • Blind-spot network: BSN predicts each output pixel without seeing the corresponding center pixel, enabling self-supervised denoising under pixel-wise independent, zero-mean noise.Under these assumptions, minimizing the self-supervised loss is equivalent to minimizing a supervised denoising loss up to a constant.
  • Blind-spot network: Real-world noise becomes spatially correlated through ISP operations such as Bayer-filter demosaicking, allowing neighboring noise to reveal the unseen center pixel.BSN can therefore operate approximately as an identity mapping on real-world sRGB images.
  • Pixel-shuffle downsampling: Pixel-shuffle downsampling creates a mosaic from subsampled noisy pixels, increasing the effective distance between noise signals to reduce spatial correlation.The prior method used stride 2 to preserve textures and details, although later analysis identifies larger strides as useful for reducing dependency.

4. Method

The method combines pixel-shuffle downsampling with blind-spot denoising through asymmetric training and inference strides, then refines outputs using randomized pixel replacement. This design addresses the trade-off between reducing real-world noise correlation and preserving image structures.

  • Method: PD_s decomposes a noisy image into s^2 sub-images, applies BSN to them, and reconstructs a full-sized output using an L1 training loss.The resulting pipeline is denoted PD_s-BSN.
  • 4.1. Trade-offs in PD-BSN: Small PD strides leave real-world noise spatially correlated, while large strides introduce stronger aliasing artifacts that can be interpreted as noise.The analysis identifies s = 5 as sufficient to suppress noise dependency, but larger strides increase aliasing.
  • 4.1. Trade-offs in PD-BSN: Training with s = 5 enables BSN to remove both real-world noise and aliasing artifacts because the artifacts approximately satisfy BSN’s independence and zero-mean requirements.The method treats rapidly changing high-frequency components as having negligible spatial correlation and notes that their statistical mean is approximately that of the noisy image.
  • 4.3. Asymmetric PD for BSN: AP_a/b uses different stride factors for training and inference: a large training stride reduces noise correlation, while b = 2 limits inference-time aliasing.AP5/2-BSN is the illustrated configuration, preserving image structures while learning from noisy real-world inputs only.
  • 4.4. Random-replacing refinement: R3 replaces denoised pixels with randomly selected noisy pixels, reducing expected residual noise correlation compared with fixed-stride PD-refinement.When combined with AP-BSN, R3 feeds the replaced image directly to BSN without another PD operation.

5. Experiment

Experiments on SIDD and DND evaluate AP-BSN’s stride-factor trade-offs, refinement, and fully self-supervised real-world denoising performance. AP-BSN + R3 achieves the best reported performance among the compared unpaired and self-supervised methods, while using noisy images only.

  • Stride-factor analysis: Training stride factor a = 5 maximizes performance, while a = 2 cannot train because real-world noise remains spatially correlated.Larger a makes the input noise follow the pixel-wise independence assumption more strictly.
  • Stride-factor analysis: Inference stride factor b = 2 maximizes performance by balancing spatial correlation against aliasing artifacts.For b > 2, more image details are removed as aliasing artifacts, reducing AP-BSN performance.
  • Stride-factor analysis: Plain regions improve with larger b, whereas textured regions perform best at b = 2 because stronger aliasing causes a larger performance drop.The visual comparison links the stride-factor trade-off to the amount of high-frequency image content.
  • R3 ablation: R3 achieves maximum performance at p ≈0.16, and T = 8 balances performance with runtime.Larger p increases expected spatial noise correlation, while R3 complexity grows proportionally with T.
  • Benchmark comparison: 35.97 dB and 38.09 dB are achieved by AP-BSN + R3 on the SIDD and DND benchmarks, respectively, outperforming the listed self-supervised and unpaired methods.The fully self-supervised AP-BSN + R3 uses noisy images only; R3 improves SIDD benchmark PSNR by more than 1 dB without additional parameters.
  • Fully self-supervised evaluation: Training and evaluating AP-BSN on the SIDD benchmark split improves denoising performance by about 1 dB without modifying the method.This experiment uses the noisy test samples directly during training and demonstrates performance on a noisy-only practical setting.

6. Conclusion

The conclusion identifies stride-factor trade-offs and introduces asymmetric PD to reconcile pixel-wise independence with image-detail preservation. AP-BSN and parameter-free R3 enable fully self-supervised real-world denoising without prior noise knowledge and outperform recent alternatives.

  • 6. Conclusion: Asymmetric PD uses different training and inference stride factors to address trade-offs in integrating PD with BSN.The design targets the pixel-wise independence assumption while preserving image details.
  • 6. Conclusion: AP-BSN performs fully self-supervised real-world denoising without requiring prior knowledge of the underlying noise.The method operates on noisy real-world inputs rather than relying on additional noise information.
  • 6. Conclusion: Random-replacing refinement removes AP-BSN visual artifacts without adding parameters.R3 is presented as a post-processing method that further improves AP-BSN.
  • 6. Conclusion: AP-BSN + R3 outperforms recent self-supervised and unsupervised denoising methods.The conclusion reports this comparison without requiring prior knowledge of real-world noise.

S1. Optimization

Optimization uses cropped and augmented noisy patches from SIDD and DND, with fixed minibatch, epoch, and learning-rate-decay settings.

  • S2. Optimization: Training uses 120×120 noisy patches randomly cropped from SIDD and DND datasets.One epoch uses 24,542 SIDD-Medium patches, 24,784 DND patches, and 24,320 SIDD benchmark patches.
  • S2. Optimization: Each sample is augmented with random 90° rotations and horizontal or vertical flips, using minibatches of 8 augmented samples.
  • S2. Optimization: AP-BSN is optimized for 20 epochs, with the learning rate reduced by a factor of 10 every 8 epochs.

S2. Network architecture

The AP-BSN architecture simplifies the baseline BSN with single-branch dilated convolution modules and uses substantially fewer parameters.

  • S2. Network architecture: AP-BSN replaces multi-branch MDC modules with a sequence of single-branch dilated convolution modules.The architecture is based on Wu et al.’s BSN, with simplifications for efficiency.
  • S2. Network architecture: The resulting network has 3.7M parameters, compared with 6.6M for the original BSN.AP-BSN without R3 shows comparable results despite using noisy images only and a smaller denoising network.

S3. Effects of aliasing artifacts

The clean-to-clean experiment shows that aliasing artifacts make PD5-BSN remove image information, whereas AP5/2 preserves high-frequency components and image structure.

  • PD5-BSN removes high-frequency information from clean inputs and fails to implement an identity mapping.The experiment assumes clean images contain zero-intensity noise satisfying BSN’s spatially uncorrelated and zero-mean assumptions.
  • PD2-BSN avoids the observed high-frequency information loss on clean images.
  • AP5/2 preserves high-frequency components and image structure by using different training and inference stride factors.
  • The results identify aliasing artifacts as preventing PD5-BSN from being a feasible denoising model.

S4. AP-BSN on the NIND dataset

On NIND, AP-BSN’s stride trade-off is dataset-dependent: AP6/2 can be slightly better for some samples, while AP5/2 generalizes well on average.

  • NIND noise correlations decrease as relative distance d increases, suggesting that larger training strides may further reduce spatial correlation.
  • AP6/2 performs slightly better on some NIND samples because NIND’s noise properties differ from SIDD.
  • AP5/2 generalizes well across various real-world datasets on average, with only marginal performance gaps.

S5. Qualitative results

AP-BSN is evaluated on real-world sRGB images produced from RAW captures without explicit denoising, and it yields sharper edges while suppressing unwanted noise.

  • The evaluation captures real-world noisy images at high ISO and uses a simulated camera pipeline without an explicit denoising stage to produce sRGB inputs.
  • AP-BSN reconstructs sharper edges and effectively suppresses unwanted noise compared with hardware-specific in-camera denoising.
  • AP-BSN outperforms DnCNN trained on SIDD noisy-clean pairs while training on a single noisy image only.

S5.3. Qualitative improvement by R3

R3 substantially improves AP-BSN’s qualitative outputs by removing blocky artifacts and producing smoother, more natural image structures without added parameters or training.

  • Without R3, AP-BSN tends to generate unpleasing blocky artifacts in denoised images.
  • R3 removes artifacts and reconstructs smooth, natural image structures without additional parameters or training.
  • The supplementary comparisons examine R3 across SIDD, DND, NIND, and self-captured noisy-image examples.
Loading 2203.11799v2…