Source-linked AI summary
Dual Adversarial Network: Toward Real-world Noise Removal and Noise Generation
Zongsheng Yue, Qian Zhao, Lei Zhang, Deyu Meng
TL;DR
Real-world denoising remains difficult, while collecting clean-noisy training pairs and modeling realistic noise are challenging. DANet learns the joint clean-noisy distribution through dual adversarial denoising and generation, achieving superior performance on both tasks and enabling further denoiser improvement through synthetic-pair augmentation.
Problem
Real-world image denoising remains challenging because camera-processing steps complicate noise, while collecting paired training data and generating realistic noise are difficult.
Method
DANet learns the joint distribution p(x, y) using two factorized forms implemented as a denoiser and noise generator trained in a dual adversarial manner.
Results
DANet achieves superior performance over state-of-the-art methods in both real-world noise removal and noise generation, while DANet+ improves denoising after synthetic-pair augmentation.
Takeaways & Limitations
The framework avoids manually designed image priors and noise assumptions, provides metrics for synthetic-noise quality, and supports denoiser training with generated clean-noisy pairs.
Takeaways & Limitations
DANet+ is less suitable when simulated noise types do not match the testing set; DANet is suggested for uncertain noise types.
Abstract
from arXiv · showhide
Real-world image noise removal is a long-standing yet very challenging task in computer vision. The success of deep neural network in denoising stimulates the research of noise generation, aiming at synthesizing more clean-noisy image pairs to facilitate the training of deep denoisers. In this work, we propose a novel unified framework to simultaneously deal with the noise removal and noise generation tasks. Instead of only inferring the posteriori distribution of the latent clean image conditioned on the observed noisy image in traditional MAP framework, our proposed method learns the joint distribution of the clean-noisy image pairs. Specifically, we approximate the joint distribution with two different factorized forms, which can be formulated as a denoiser mapping the noisy image to the clean one and a generator mapping the clean image to the noisy one. The learned joint distribution implicitly contains all the information between the noisy and clean images, avoiding the necessity of manually designing the image priors and noise assumptions as traditional. Besides, the performance of our denoiser can be further improved by augmenting the original training dataset with the learned generator. Moreover, we propose two metrics to assess the quality of the generated noisy image, for which, to the best of our knowledge, such metrics are firstly proposed along this research line. Extensive experiments have been conducted to demonstrate the superiority of our method over the state-of-the-arts both in the real noise removal and generation tasks. The training and testing code is available at https://github.com/zsyOAOA/DANet.
1 Introduction
Real-world denoising is difficult because camera processing complicates noise, while deep denoisers require costly paired data. DANet addresses both denoising and noise generation by learning clean-noisy joint distributions in a dual adversarial framework.
- Real image denoising remains challenging because camera processing includes demosaicing, Gamma correction, and compression.
- Deep denoisers require large, expensive-to-collect clean-noisy image-pair datasets, motivating noise-generation methods.
- DANet models the joint distribution p(x, y) rather than only the conditional posterior p(x|y) used in conventional MAP denoising.
- Two factorized forms approximate the joint distribution through a denoiser mapping noisy images to clean images and a generator mapping clean images to noisy images.
- The jointly trained framework handles noise removal and generation, while generated clean-noisy pairs can further improve denoising after retraining.
- The work introduces two metrics for assessing generated noisy images, described as the first metrics proposed for this purpose.
2 Related Work
Prior denoising methods use image priors or explicit noise models, while deep learning methods face costly paired-data collection. Noise-generation approaches reduce data demands but struggle with camera specificity, complex noise, and GAN instability.
- MAP-based denoising methods commonly impose priors such as total variation, non-local similarity, low-rankness, or sparsity.
- Noise-modeling methods instead specify likelihoods using distributions including mixture of Gaussians and mixture of exponential-power models.
- Deep learning has advanced denoising, but collecting paired training data remains expensive and limits supervised approaches.
- In-camera unprocessing and processing can generate training pairs but requires many camera-specific hyperparameters.
- GAN-based noise generation learns real noise distributions directly, yet realistic simulation is difficult because real noise is complex and GAN training is unstable.
3 Proposed Method
The method approximates the joint distribution of clean-noisy image pairs through denoising and generation factorizations, then trains both mappings adversarially. The learned generator can augment training data to further improve denoising.
- 3.1 Two Factorizations of Joint Distribution: The framework approximates the joint distribution p(x, y) rather than only learning a noisy-to-clean mapping.It derives pseudo joint distributions from both the denoiser and generator perspectives.
- 3.1 Two Factorizations of Joint Distribution: The denoiser R maps noisy images y to clean estimates and implicitly approximates the conditional distribution p(x|y).Its pseudo joint distribution becomes more accurate as the denoiser improves.
- 3.1 Two Factorizations of Joint Distribution: The generator G maps clean images x and latent noise z to simulated noisy images, approximating p(y|x, z) and the resulting joint distribution.The latent variable represents fundamental elements driving hardware-related random noise, with p(z) settable as an isotropic Gaussian.
- 3.2 Dual Adversarial Model: A discriminator distinguishes real clean-noisy pairs from denoiser- and generator-produced pairs, pushing both pseudo joint distributions toward p(x, y).The denoiser, generator, and discriminator are updated in alternating adversarial training.
- 3.2 Dual Adversarial Model: The objective combines adversarial training with an L1 denoising loss and an L1 loss on Gaussian-filtered noise statistics.The two regularizers separately constrain denoiser outputs and generator noise features.
- 3.3 Training Strategy and 3.4 Network Architecture: After training, generated noisy pairs can augment the original dataset, enabling denoiser retraining for improved denoising performance.The framework uses UNet backbones with residual learning, while its general formulation permits alternative architectures.
4 Evaluation Metrics
The evaluation introduces PGap and AKLD to quantify how closely generated noisy images resemble real noisy images. PGap compares downstream denoiser performance, while AKLD compares conditional noise distributions.
- PGap (PSNR Gap): PGap evaluates generated noise indirectly by comparing denoisers trained on original and generated datasets under the same conditions.It is defined as PGap = PSNR(R1(T)) − PSNR(R2(T)).
- PGap (PSNR Gap): A small PGap indicates that a denoiser trained on generated noisy images performs similarly to one trained on real noisy images.The testing set T is shared when comparing the two denoisers.
- AKLD (Average KL Divergence): AKLD measures the KL divergence between fake and real conditional noisy-image distributions for the same clean image.The conditional distributions are made tractable with a pixel-wise Gaussian assumption for real noise.
- AKLD (Average KL Divergence): AKLD averages KL divergence over L synthetic noisy images generated from each real clean image, with smaller values indicating better generators.The experiments set L = 50.
- Evaluation on SIDD: Table 1 reports PGap and AKLD for compared methods on the SIDD validation dataset, with the best results highlighted.The table evaluates both proposed noise-generation metrics together.
5 Experimental Results
Experiments evaluate DANet and its augmented variant across real-world denoising and noise-generation settings. DANet performs strongly on generation and denoising, while the benefit of augmentation depends on how well simulated noise matches the test distribution.
- Experimental setup: DANet is evaluated on real-world denoising and noise-generation benchmarks, while DANet+ retrains the denoiser with synthetic clean-noisy pairs from the learned generator.Experiments include SIDD, DND, and Nam benchmarks, with SIDD also used to assess generation quality.
- Noise generation: DANet achieves the best PGap and smallest average AKLD on the SIDD validation set, outperforming competing noise-generation methods quantitatively and visually.Figure 3 compares generated noisy images, noise, and variance maps against real SIDD validation examples.
- SIDD denoising: DANet and DANet+ outperform state-of-the-art real-world denoisers on SIDD, with DANet+ exceeding DANet by about 0.18dB PSNR.Visual results show sharper edges for DANet and closer agreement with ground truth after augmentation in DANet+.
- Parameter analysis: A moderate τ1 = 1e+3 balances adversarial and L1 objectives, preserving more realistic denoising results while sacrificing a little PSNR and SSIM.Small τ1 values increase the adversarial-loss influence and reduce PSNR and SSIM; very large values make the L1 regularizer dominant.
- Parameter analysis: For generation, τ2 = 0 fails to simulate real noise, whereas training only with the regularizer is also difficult; the selected setting balances quantitative and visual quality.Table 4 reports PGap and AKLD under different τ2 values, and Figure 5 visualizes the corresponding noise.
- Generalization: On DND and Nam, GDANet variants outperform competing methods, but GDANet is more stable than GDANet+ when test noise types differ from those simulated during training.The authors recommend GDANet for uncertain noise types and DANet+ when the deployment noise resembles the training distribution.
6 Conclusion
DANet learns the joint distribution of clean-noisy image pairs through dual adversarial factorization, addressing noise removal and generation together. It also introduces metrics for synthetic-noisy-image quality and demonstrates superiority over state-of-the-art methods.
- DANet learns the joint distribution directly from data instead of relying on subjective noise and image-prior assumptions.
- Dual adversarial estimation uses two factorized forms corresponding to noise removal and noise generation.
- The framework introduces two metrics for assessing the quality of synthetic noisy images.
- Comprehensive experiments demonstrate superiority over state-of-the-art methods on both noise removal and noise generation tasks.
- DANet provides a general methodology that may facilitate other low-level vision tasks, including super-resolution and deblurring.