PaperlayerView exact revision on arXiv

Source-linked AI summary

CVF-SID: Cyclic multi-Variate Function for Self-Supervised Image Denoising by Disentangling Noise from Image

Reyhaneh Neshatavar, Mohsen Yavartanoo, Sanghyun Son, Kyoung Mu Lee

arXiv:2203.13009v3cs.CV

TL;DR

Well-aligned noisy-clean pairs are costly to obtain, and existing weakly supervised approaches can depend on impractical assumptions. CVF-SID cyclically disentangles clean images and signal-dependent and signal-independent noise from real-world noisy sRGB inputs using self-supervision. It reports superior performance among several un-/self-supervised methods and comparability with other approaches, while relying on stated noise-component assumptions.

  • Problem

    Well-aligned noisy-clean training pairs are complicated and costly to obtain, while some un-/self-supervised methods rely on noise assumptions or configurations that limit practical use.

    Source: §1 · pp. 1–2

  • Method

    CVF-SID uses a cyclic multi-variate function and self-supervised objectives to decompose noisy sRGB inputs into clean images and signal-dependent and signal-independent noise.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    CVF-SID achieves superior denoising performance among several un-/self-supervised methods on real-world sRGB images and is comparable with other approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CVF-SID applies directly to sRGB images without assuming a specific noise distribution, unlike R2R, which requires Raw-RGB images for pretraining.

    Source: §4.3 · p. 7

    The framework assumes zero-mean, spatially invariant noise maps and independence between each pair of clean image and noise components.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Recently, significant progress has been made on image denoising with strong supervision from large-scale datasets. However, obtaining well-aligned noisy-clean training image pairs for each specific scenario is complicated and costly in practice. Consequently, applying a conventional supervised denoising network on in-the-wild noisy inputs is not straightforward. Although several studies have challenged this problem without strong supervision, they rely on less practical assumptions and cannot be applied to practical situations directly. To address the aforementioned challenges, we propose a novel and powerful self-supervised denoising method called CVF-SID based on a Cyclic multi-Variate Function (CVF) module and a self-supervised image disentangling (SID) framework. The CVF module can output multiple decomposed variables of the input and take a combination of the outputs back as an input in a cyclic manner. Our CVF-SID can disentangle a clean image and noise maps from the input by leveraging various self-supervised loss terms. Unlike several methods that only consider the signal-independent noise models, we also deal with signal-dependent noise components for real-world applications. Furthermore, we do not rely on any prior assumptions about the underlying noise distribution, making CVF-SID more generalizable toward realistic noise. Extensive experiments on real-world datasets show that CVF-SID achieves state-of-the-art self-supervised image denoising performance and is comparable to other existing approaches. The code is publicly available from https://github.com/Reyhanehne/CVF-SID_PyTorch .

1. Introduction

Real-world noisy-clean pairs are costly to obtain, while existing un-/self-supervised methods can rely on impractical noise assumptions or data requirements. CVF-SID uses a cyclic decomposition framework and self-supervised training to denoise real-world sRGB images, with reported performance superior to several un-/self-supervised methods and comparable to other approaches.

1. Introduction

  • Well-aligned real-world noisy-clean pairs are challenging and sometimes unavailable because they require multiple shots of the same static scene under constraints.Synthetic-AWGN-trained methods also generalize poorly to practical examples because real-world noise differs from synthetic AWGN.Source: §1 · p. 1
  • Noisy-only methods can assume a specific noise distribution or impractical configuration; R2R is not directly applicable to sRGB inputs and may require Raw-RGB noise-level prior information.Digital images are usually stored in sRGB color space.Source: §1 · p. 2
  • CVF-SID introduces self-supervised training objectives and an augmentation strategy for denoising real-world images.The method is presented for sRGB images and separates clean-image, signal-dependent-noise, and signal-independent-noise components.Source: §1 · p. 2
  • CVF-SID achieves superior denoising performance among several un-/self-supervised methods on real-world sRGB images and is comparable with other approaches.Figure 1 contrasts its direct applicability to sRGB images with R2R and shows detail loss by R2R in the yellow patch.Source: Figure 1 · p. 1, §1 · p. 2
  • CVF maps a combination of multiple variables to decomposed variables, then can take a combination of its outputs as a new input.The proposed framework uses this cyclic function to disentangle a clean image and signal-dependent and signal-independent noise from a noisy input.Source: Figure 2 · p. 2, §1 · p. 2

2. Related work

Prior denoising approaches differ in their training data: paired noisy-clean images, generated noisy-clean pairs, or paired noisy-noisy images. Their constraints include difficult data collection, clean-image availability, and assumptions or configurations that may not fit real-world sRGB inputs.

2.1. Training on paired noisy-clean images

  • Supervised approaches train on synthetic noisy-clean images, often assuming AWGN, but synthetic-to-real domain discrepancy limits generalization and collecting aligned real pairs is challenging.CBDNet, RIDNet, and DIDN train on realistic noisy-clean pairs, whose collection requires substantial labor under controlled environments.Source: §2.1 · p. 2

2.2. Training on unpaired noisy-clean images

  • Generation-based methods synthesize noisy samples from clean images and train on generated noisy-clean pairs, but clean images may not match the noisy data's scene distribution.GCBD considers additive noise only, while UIDNet and C2N address real-world noise in different ways.Source: §2.2 · pp. 2–3

2.3. Training on paired noisy-noisy images

  • N2N needs multiple noisy images of the same scene, while N2V [18] and N2S [4] use blind-spot prediction that ignores center pixels and can lose useful information.Self2Self generates paired data from one noisy image using Bernoulli dropout.Source: §2.3 · p. 3
  • When raw information is unavailable for a noisy input, R2R resorts to a Gaussian noise assumption, preventing practical application on digital noisy sRGB images; CVF-SID trains directly on noisy sRGB without pseudo noisy-noisy pairs.Source: §2.3 · p. 3

3. Method

CVF-SID decomposes a noisy sRGB image into a clean image and signal-dependent and signal-independent noise maps, then reuses its network on recombinations of those outputs. Self-supervised constraints on the resulting cycles train the decomposition without noisy-clean pairs.

3.1. Cyclic multi-Variate Function

  • CVF maps a combination of component vectors to those components and can take recombinations of its outputs as input again.Its scaling property requires decomposing scaled components into the same scaled components.Source: §3.1 · p. 3

3.2. Self-supervised image denoising using CVF

  • CVF-SID represents a noisy image using a clean image, signal-dependent noise scaled by image intensity, and signal-independent noise.The network predicts all three components rather than reconstructing only a clean image.Source: §3.2 · p. 4
  • The self-supervised cycle assumes zero-mean, spatially invariant noise maps with unknown distributions and pairwise independence among the clean image and noise components.Because noisy-clean pairs are unavailable, the predicted components are not directly supervised; the same network is reapplied to its outputs and constrained by their desired properties.Source: §3.2 · p. 4

3.3. Network architecture

  • The network uses a clean-image generator followed by two branches that estimate signal-dependent and signal-independent noise from the residual.The clean generator uses DnCNN without skip connections or batch normalization.Source: §3.3 · p. 5

3.4. Loss functions for self-supervised learning

  • The consistency loss requires the combination of predicted clean and noise components to converge to the noisy input.The loss is defined on the combined outputs of the network.Source: §3.4 · p. 5
  • Identity and zero losses impose the expected network outputs for clean inputs and inputs containing only one predicted noise component.For a denoised image, the network should return that image without noise terms; identity objectives also cover signal-dependent and signal-independent inputs.Source: §3.4 · p. 5
  • Regularization discourages zero-variance noise estimates, while augmentation loss applies constraints to virtual inputs formed by recombining predicted components.The regularizer uses local patch variance estimates; recombinations provide self-supervised augmentation without extra samples.Source: §3.2 · p. 4, §3.4 · p. 5

4. Experiments

The experiments use real-world SIDD and DND sRGB denoising benchmarks, with SIDD-Medium training pairs and evaluation splits described for both datasets. Table 1 compares methods using PSNR and SSIM.

4.1. Dataset

  • SIDD-Medium training uses 320 noisy-clean image pairs; its validation and benchmark splits each contain 32 patches from 40 images.The benchmark track has no ground-truth clean images.Source: §4.1 · p. 5
  • The DND benchmark provides 1,000 evaluation patches cropped from 50 consumer-camera images, which contain weaker noise than SIDD images.Each DND image contributes 20 patches of size 512 × 512.Source: §4.1 · p. 5
  • Table 1 compares denoising methods on SIDD and DND using PSNR and SSIM, with T, S, and S2 denoting training strategies.Source: Table 1 · p. 6

CVF-SID (Ours)

CVF-SID is evaluated on real-world sRGB denoising benchmarks with several training strategies and self-supervised objectives. The reported results show strong comparisons with other methods, while ablations examine loss terms, augmentation, and the correlation parameter.

4.2. Training details

  • Training uses 40 × 40 random crops in batches of 64, ADAM with a 10^-4 learning rate, and about 10 ms inference per 256×256 image using the clean-image generator.The experiments fix γ = 1 and use λ_aug = 0.1; dense 6 × 6 patches estimate the variance for the regression loss.Source: §4.2 · p. 6

4.3. Evaluation on real-world sRGB datasets

  • The three strategies train on SIDD-Medium (T), the evaluation dataset itself (S), or denoised test images followed by a second training stage (S2).Training and testing on the same dataset matches their noise distributions; T evaluated on DND tests cross-domain generalization.Source: §4.3 · p. 7
  • CVF-SID (S2), recursively trained on test data, outperforms several unsupervised and self-supervised methods on SIDD and DND without clean images.On SIDD validation, CVF-SID also performs much better than existing self-supervised methods and C2N, and comparably to R2R.Source: §4.3 · p. 7
  • CVF-SID preserves text details, textures, and edges while suppressing noise in the qualitative SIDD and DND comparisons.The reported comparisons note that other methods may lose characters, leave noise, fail to preserve textures, or alter colors.Source: Figure 5 · p. 6, §4.3 · p. 7, Figure 6 · p. 7

4.4. Ablation study

  • The consistency loss is necessary, identity loss improves performance and stabilizes learning, augmentation set B adds 0.2 dB, and γ = 1 performs best.Increasing γ beyond 1 changes performance little.Source: §4.4 · p. 8, Table 3 · p. 8, Table 4 · p. 8

5. Conclusion

The conclusion presents CVF-SID as a cyclic decomposition framework for self-supervised denoising of real-world noisy sRGB images. It identifies the fixed correlation parameter as a limitation and proposes learning it in future work.

5. Conclusion

  • CVF decomposes an input through a cyclic procedure and underlies the CVF-SID denoising framework.The framework aims to separate clean image content from signal-dependent and signal-independent noise in noisy sRGB inputs.Source: §5 · p. 8
  • CVF-SID uses no prior information about the noise distribution, which the authors describe as making it more generalizable than previous self-supervised denoising methods.Source: §5 · p. 8
  • A fixed γ can yield suboptimal decomposition when image-noise correlation varies across real-world images.The authors note examples with little correlation between the image and signal-dependent noise.Source: §5 · p. 8
  • Future work aims to learn γ self-supervised and extend CVF to other computer-vision tasks.Source: §5 · p. 8

Supplementary Material for CVF-SID: Cyclic multi-Variate Function for Self-Supervised Image Denoising by Disentangling Noise from Image

The supplementary material lists the paper's authors and their institutional affiliation.

Supplementary Material

S1. Training on a Single Image

The supplementary experiment tests whether CVF-SID can be trained using only one noisy image. It reports that the method can learn to denoise from that image without a large-scale dataset.

S1. Training on a Single Image

S2. Network architecture details

The network uses separate generators for the clean image and two noise components, with reflection padding and channel-wise centering to prepare its outputs.

S2. Network architecture details

  • The clean-image generator uses 16 sequential 3 × 3 convolutional layers with 64 channels, followed by a 1 × 1 convolution that produces RGB output.Each 3 × 3 layer uses padding 1 and is followed by ReLU.Source: S2. Network architecture details · p. 10
  • The noise generator shares ten 3 × 3 convolutional layers before separate three-layer branches generate signal-dependent and signal-independent noise maps.The branch outputs are ˆNd and ˆNi; each uses ReLU layers and a final 1 × 1 convolution.Source: S2. Network architecture details · p. 10
  • Reflection padding of 20 pixels is cropped after inference, and each noise map is centered by subtracting its channel-wise average.Weights use Xavier uniform initialization and biases are initialized to zero.Source: S2. Network architecture details · p. 10
  • Figure 8 depicts a restored clean image obtained from only a single noisy image.Source: Figure 8 · p. 10

S3. Training Schemes

The supplementary results compare training schemes T, S, and S2, and assess successive CVF-SID applications on SIDD validation images.

S3. Training Schemes

  • Training scheme S gives slightly better results than T for all images, while S2 does not further improve some cases.Figure 9 compares predicted clean images and reports PSNR against ground truth; the fifth row illustrates a case where S suffices.Source: S3. Training Schemes · p. 10, Figure 9 · p. 11
  • Figure 10 shows synthetic noisy images generated by the proposed self-supervised augmentation strategy on SIDD validation images.Source: Figure 10 · p. 12
  • Figure 11 presents CVF-SID decomposition results on SIDD validation images trained with scheme S.Source: Figure 11 · p. 12

S4. Cascading additional models

S5. Augmentations

The augmentation strategy generates realistic noisy examples from one noisy image without additional information; decomposition results also reveal incomplete independence between two predicted components.

S5. Augmentations

  • Table S1 compares the number of cascading models on sRGB images from the SIDD validation dataset.The supplied passage does not report numerical values.Source: Table S1 · p. 12
  • The augmentation strategy generates several real-world noisy images from a single noisy image without requiring additional information.Figure 10 visualizes examples generated by the strategy.Source: S5. Augmentations · p. 12
  • The SIDD validation experiments analyze λaug, and Table S2 compares its values on sRGB images.The supplied passage does not report the comparison's numerical results.Source: S5. Augmentations · p. 13, Table S2 · p. 13

S6. Decomposition results

  • Figure 11 shows CVF-SID decomposing a noisy input into a clean image and signal-dependent and signal-independent noise maps while satisfying output constraints.Source: S6. Decomposition results · p. 13
  • The predicted clean image and signal-dependent noise map are not fully independent, possibly because the fixed correlation parameter γ may vary across real-world images.Source: S6. Decomposition results · p. 13

Related papers

Loading 2203.13009v3…