Source-linked AI summary
Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots
Zejin Wang, Jiazheng Liu, Guoqing Li, Hua Han
TL;DR
Single-image self-supervised denoising avoids costly noisy-clean pairs, but blindspot methods lose valuable information. Blind2Unblind combines global-aware mask mapping with re-visible loss to make blind spots visible while training from raw noisy images. The method reports superior performance across synthetic and real-world settings, especially complex noise patterns.
Problem
Self-supervised blindspot denoisers avoid costly noisy-clean pairs but lose valuable contextual information through masking or network design.
Method
Blind2Unblind uses a global-aware mask mapper and re-visible loss to train denoisers from raw noisy images without information loss or identity mapping.
Results
Blind2Unblind shows superior performance against compared methods, especially on real-world datasets with complex noise patterns.
Takeaways & Limitations
The framework enables blindspot schemes to use complete noisy images and transition from blind to non-blind denoising.
Takeaways & Limitations
Noise2Void's formulation assumes noise is context-independent while signal depends on surrounding pixels.
Abstract
from arXiv · showhide
Real noisy-clean pairs on a large scale are costly and difficult to obtain. Meanwhile, supervised denoisers trained on synthetic data perform poorly in practice. Self-supervised denoisers, which learn only from single noisy images, solve the data collection problem. However, self-supervised denoising methods, especially blindspot-driven ones, suffer sizable information loss during input or network design. The absence of valuable information dramatically reduces the upper bound of denoising performance. In this paper, we propose a simple yet efficient approach called Blind2Unblind to overcome the information loss in blindspot-driven denoising methods. First, we introduce a global-aware mask mapper that enables global perception and accelerates training. The mask mapper samples all pixels at blind spots on denoised volumes and maps them to the same channel, allowing the loss function to optimize all blind spots at once. Second, we propose a re-visible loss to train the denoising network and make blind spots visible. The denoiser can learn directly from raw noise images without losing information or being trapped in identity mapping. We also theoretically analyze the convergence of the re-visible loss. Extensive experiments on synthetic and real-world datasets demonstrate the superior performance of our approach compared to previous work. Code is available at https://github.com/demonsjin/Blind2Unblind.
1. Introduction
Blind2Unblind addresses information loss and training instability in self-supervised blindspot denoising by combining global-aware mask mapping with re-visible loss. The framework targets denoising from single noisy images while retaining more image information.
- Supervised denoisers require costly noisy-clean pairs and can degrade substantially on unknown noise patterns.
- Single-image self-supervision avoids pair collection, but blindspot masking removes context and optimizing only partial pixels harms performance and convergence.
- Blind2Unblind uses a global-aware mask mapper and re-visible loss to address blindspot information loss without subsampling or identity mapping.
- The global mapper samples blind spots across denoised volumes, enabling global optimization and accelerating training.
- The method provides theoretical convergence analysis and reports superior performance, especially on real-world datasets with complex noise patterns.
2. Related Work
Related work spans iterative classical denoisers, supervised neural methods, and self-supervised approaches that reduce data requirements through noisy or masked images. Existing methods still face inference, collection, information-loss, or modeling limitations.
- Classical methods iteratively customize denoising for each image, resulting in long inference times and poor performance.
- Supervised neural denoisers improve performance but depend on difficult-to-collect noisy-clean pairs.
- Noise2Noise reduces collection difficulty with noisy-noisy pairs, while Noise2Self and Noise2Void use masked schemes on individual noisy images.
- Feature-level masking enlarges receptive fields, and some methods add noise-model priors as post-processing to improve performance.
3. Theoretical Framework
The theoretical framework explains how Blind2Unblind converts blindspot supervision into non-blind denoising. It combines masked-volume mapping, a re-visible objective, and regularization to retain information while avoiding identity learning.
- Noise2Void predicts a center pixel from its receptive-field patch, assuming context-independent noise and signal dependence on surrounding pixels.
- Blindspot methods use less information, motivating a loss that converts invisible blind spots into visible ones while preserving self-supervision.
- The masked volume Ωy contains blind spots at all positions of y, while h samples corresponding denoised pixels to create a globally mapped image.
- Directly combining blind and raw-image losses can learn the identity, so the re-visible formulation uses a detached raw-image output as an implicit training medium.
- The re-visible objective combines mapped masked predictions and raw-image predictions against y, with λ weighting the latter term.
- When the error terms have opposite signs, the raw-image prediction is optimized toward the mapped masked prediction rather than directly toward y.
- At convergence, the optimal estimate is the weighted average of mapped masked and raw-image denoiser outputs, and as λ approaches infinity inference uses the raw noisy image directly.
- A regularized constrained formulation forces the blind transition term toward zero to stabilize training.
4. Main Method
Blind2Unblind combines a global-aware mask mapper with regularized re-visible loss to preserve information while transitioning blind-spot denoising toward visible inputs.
- Main framework: The framework learns self-supervised denoising from a single noisy observation using a global-aware mask mapper and re-visible loss.These are its two main components.
- Global-aware mask mapper: The mask mapper samples denoised pixels at blind-spot locations across masked volumes and assembles them into a globally denoised image.Masked volumes are created by masking corresponding pixel positions across image cells, then stacking the results.
- Global-aware mask mapper: For a 2 × 2 cell, the global masker creates four masked cells, while the mapper samples their blind-spot outputs using locations and pixel values.The sampled outputs are mapped into the final denoised cell.
- Regularized re-visible loss: The regularized re-visible loss combines a re-visible term with a regularization term to stabilize training during the blind-to-non-blind transition.The loss is expressed as L_rev + η · L_reg, with the two squared-error terms defined in the displayed equation.
- Regularized re-visible loss: The denoising network output on the unmasked noisy image is prevented from identity mapping by disabling its gradient updates during training.The loss uses hyperparameters η and λ to control stability and the transition toward visible parts.
5. Experimental Results
Experiments evaluate Blind2Unblind on synthetic Gaussian and Poisson noise, raw-RGB SIDD data, fluorescence microscopy, and ablations of its components. The method generally outperforms competing self-supervised approaches and is competitive with supervised baselines on real-world datasets.
- Implementation details: The experiments use 100 training epochs with Adam, learning-rate halving every 20 epochs, and separate initial learning rates for synthetic and real-world denoising.The initial rates are 0.0003 for synthetic sRGB and 0.0001 for real-world raw-RGB and FM denoising.
- Synthetic denoising: Blind2Unblind significantly outperforms BM3D and five self-supervised methods for fixed and variable Gaussian noise.The compared self-supervised methods are Self2Self, N2V, Laine19-mu, DBSN, and NBR2NBR.
- Synthetic denoising: For variable Poisson noise, Blind2Unblind outperforms almost all compared methods, including Laine19-pme, and exceeds N2C by 0.13 dB on BSD300 and 0.08 dB on SET14.For fixed Poisson noise, it performs comparably to Laine19-pme.
- Real-world denoising: Blind2Unblind exceeds NBR2NBR by 0.32 dB on the SIDD benchmark and 0.20 dB on validation in grayscale denoising.The reported explanation is that the blind-to-non-blind transition preserves noisy-image information and mitigates neighboring-pixel oversmoothing.
- Real-world denoising: In raw-RGB SIDD denoising, Blind2Unblind exceeds N2C by 0.19 dB on the benchmark and 0.17 dB on validation.The paper describes its performance as competitive under complex noise patterns and reports more texture continuity than NBR2NBR.
- Fluorescence microscopy: On fluorescence microscopy datasets, Blind2Unblind greatly outperforms self-supervised methods and slightly outperforms supervised N2C and N2N on Confocal Mice and Two-Photon Mice.The method is reported as competitive across the evaluated fluorescence microscopy datasets.
- Ablation studies: Ablations show that the regularization term improves stability and performance, with η = 1 at or near the best result across four noise patterns.The paper sets η = 1 in its experiments.
6. Conclusion
Blind2Unblind achieves lossless self-supervised denoising by transitioning blindspot schemes from blind to non-blind processing without information loss.
- Blind2Unblind enables blindspot schemes to use complete noisy images without information loss.
A. Training Framework for Blind2Unblind
Algorithm 1 trains Blind2Unblind by masking noisy images, denoising masked volumes, deriving a visible denoised image, and calculating re-visible loss and regularization.
- The training loop initializes a denoising network and hyper-parameters, then iterates until convergence.
- Each iteration samples a noisy image and generates a global masker to produce the network input masked volume.
- The masked volume is passed through the denoising network to derive a denoised volume at blind spots and form a blind denoised image.
- The original noisy image produces a visible denoised image without gradients before the re-visible loss is calculated.
- The algorithm subsequently calculates a regularization term within the training loop.
B. Details of Interpolation from Neighbors
Interpolation from neighbors constructs a masked image by combining interpolated masked pixels with unmasked pixels.
- Random masking, convolution, and Hadamard products generate interpolated masked pixels and preserve unmasked pixels before summation.The workflow convolves y with stride 1 and padding 1, multiplies by the mask, and combines the results.
C. Details of Random Mask Strategy
The random mask strategy divides an image into 2x2 cells and randomly selects one pixel per cell as a blind spot.
- Each 2x2 cell has four possible random masking configurations because one specific pixel is set as a blind spot.The resulting masked image Ωy is fed into the denoising network to generate fθ(Ωy).
D. More Experimental Results
This section presents additional visualizations of Blind2Unblind, including its denoising steps, interpolation details, random masking strategy, and comparisons on SIDD.
- Figure 9 illustrates the proposed method’s denoising steps for sRGB images with σ = 25.
- Figure 7 provides details of interpolation from neighboring pixels.
- Figure 8 provides details of the random mask strategy.
- Figure 10 visually compares denoising results for raw-RGB images on the challenging SIDD benchmark.