Source-linked AI summary
Transfer Learning from Synthetic to Real-Noise Denoising with Adaptive Instance Normalization
Yoonsik Kim, Jae Woong Soh, Gu Yong Park, Nam Ik Cho
TL;DR
Real-noise denoising is difficult because synthetic and real noise distributions differ, while real noise varies with camera and imaging conditions. The paper combines an AIN-based denoiser with synthetic-to-real transfer learning, reporting strong real-noise generalization and effective adaptation with very few labeled real pairs.
Problem
Synthetic-noise-trained CNNs overfit Gaussian distributions and perform poorly on real-world images, while real-noise training data are difficult to acquire and limited.
Method
The paper uses an AIN denoiser for regularization and transfers synthetic-domain denoising knowledge to real-noise data by updating AIN parameters.
Results
The proposed denoising method achieves state-of-the-art performance on synthetic- and real-noise images and generalizes to real noise after synthetic training.
Takeaways & Limitations
Transfer learning can effectively adapt a synthetic-noise denoiser to real-noise denoising using very few additional real-noise pairs.
Takeaways & Limitations
Real-noise paired datasets require specialized knowledge, may be insufficient for deeper CNNs, and can overfit to specific camera devices.
Abstract
from arXiv · showhide
Real-noise denoising is a challenging task because the statistics of real-noise do not follow the normal distribution, and they are also spatially and temporally changing. In order to cope with various and complex real-noise, we propose a well-generalized denoising architecture and a transfer learning scheme. Specifically, we adopt an adaptive instance normalization to build a denoiser, which can regularize the feature map and prevent the network from overfitting to the training set. We also introduce a transfer learning scheme that transfers knowledge learned from synthetic-noise data to the real-noise denoiser. From the proposed transfer learning, the synthetic-noise denoiser can learn general features from various synthetic-noise data, and the real-noise denoiser can learn the real-noise characteristics from real data. From the experiments, we find that the proposed denoising method has great generalization ability, such that our network trained with synthetic-noise achieves the best performance for Darmstadt Noise Dataset (DND) among the methods from published papers. We can also see that the proposed transfer learning scheme robustly works for real-noise images through the learning with a very small number of labeled data.
1. Introduction
Real-noise denoising generalizes poorly from synthetic training because real degradations differ across domains and camera pipelines. The paper proposes AIN regularization and synthetic-to-real transfer learning to improve generalization with limited real data.
- Synthetic-noise-trained CNNs overfit Gaussian distributions and perform insufficiently on real-world images with different noise distributions.
- Real-noise datasets improve robustness when training and test domains coincide, but paired data are difficult to acquire, limited in size, and device-specific.
- AIN regularizes the denoiser by adaptively scaling and shifting feature maps according to pixel-wise noise characteristics.
- The transfer scheme learns domain-invariant denoising information from synthetic data and updates AIN affine parameters using real-noise data.
- The proposed method reports state-of-the-art performance on both synthetic-noise and real-noise images.
2. Related Works
Real-noise statistics arise from sensor noise and are transformed by camera pipelines, making them difficult to represent with simple synthetic models. Prior work therefore develops increasingly realistic statistical and pipeline-based approximations.
- Real-noise statistics depend on camera sensors and in-camera processes such as demosaicing, gamma correction, denoising, and color correction.
- Existing noise models include Gaussian-Poisson, heteroscedastic Gaussian, Gaussian mixture, and deep-learning-based approaches.
- CBDNet and Unprocessing approximate real noise by adding heteroscedastic Gaussian noise to pseudo-raw images before simulating the camera pipeline.
3. Proposed
The proposed system combines an AIN-based denoising architecture with transfer learning from synthetic-noise data to real-noise data. It estimates noise at multiple scales, adaptively transforms normalized features, and updates only domain-specific components during real-noise training.
- Transfer learning: Transfer learning first trains on synthetic-noise pairs, then updates domain-specific parameters with real-noise pairs while preserving the remaining synthetic-noise parameters.AIN modules, the noise level estimator, and the last convolution are updated during real-noise training; other parameters remain fixed.
- Denoising architecture: AINDNet uses a noise level estimator and U-Net-based reconstruction network with AIN Residual blocks for both synthetic- and real-noise denoising.The estimator maps a noisy image to an estimated noise-level map, which the reconstruction network uses with the noisy image to produce a denoised image.
- Noise level estimation: The noise level estimator combines down-scaled and original-sized estimates to produce a region-wise smoothed noise map.The down-scaled estimate is interpolated before weighted averaging; λms is empirically set to 0.8.
- Adaptive instance normalization: AIN normalizes feature maps channel-wise and applies pixel-wise scale and shift parameters conditioned on the estimated noise level.The affine parameters adapt feature maps to spatially varying noise characteristics while regularizing the denoiser against overfitting synthetic-noise images.
- Transfer learning: The transfer scheme converges faster and achieves better performance than training from scratch with very few real-noise training pairs.It also supports multiple noise models by switching specific parameters, reducing memory requirements.
- Training objectives: Synthetic-noise training uses multi-scale asymmetric estimation loss with L1 reconstruction loss, whereas real-noise training uses reconstruction loss only.The asymmetric loss is intended to prevent underestimation, and the estimator-loss weight λms-asymm is empirically set to 0.05.
4. Experiments
The experiments evaluate denoising on synthetic Gaussian noise and several real-noise benchmarks, including DND, SIDD, and RNI15. Results support strong generalization from the proposed architecture and improved real-noise performance from transfer learning, including with limited real-noise pairs.
- Noise Level Estimation: The proposed estimator achieves more accurate results than a prior estimator with a similar number of parameters on heteroscedastic Gaussian images.Accuracy is measured using mean absolute error and error standard deviation on Kodak24 images processed through an in-camera pipeline.
- AWGN Denoising: The proposed denoiser achieves the best Set12 performance and the second-best BSD68 performance under AWGN.Set12 is compositionally independent from training data, whereas BSD68 contains objects similar to those in BSD400.
- Real Noise Denoising: AINDNet(S) achieves the best DND performance and gains 1.5 dB over CBDNet on DND and 2.4 dB on SIDD.The comparison uses the same employed noise models for CBDNet and AINDNet(S).
- Real Noise Denoising: AINDNet(R) performs better than AINDNet(S) on SIDD but shows insufficient performance on DND, indicating possible overfitting to the SIDD benchmark.AINDNet(R) is trained solely with SIDD training images, whose test scenes and objects are similar to the training set.
- Real Noise Denoising: AINDNet+RT and AINDNet+TF perform well on both DND and SIDD, while AINDNet+TF achieves the best overall performance among compared methods.The authors attribute AINDNet+TF’s advantage to better preservation of knowledge learned from synthetic-noise data.
- Real Noise Denoising: Visual comparisons show robust noise removal while preserving edges, and transfer learning removes unusual noise not removed by AINDNet(S).The reported comparisons cover SIDD and RNI15 images and include visual enhancement after applying the proposed transfer learning scheme.
- Effect of Transfer Learning with Limited RN Pairs: Transfer learning infers strong performance from a small number of SIDD real-noise image pairs.The study varies the number of training pairs from one to all 320 and reports average PSNR on the SIDD validation set.
- Architecture of Denoiser: The AIN-ResBlock achieves the best performance on real-noise datasets compared with the IN + Concat baseline and an adaptive Gaussian denoiser.The comparison replaces AIN-ResBlock with alternative reconstruction components while keeping other settings the same.
5. Conclusion
The paper presents an AIN-based denoiser and transfer-learning scheme for real-noise denoising. The scheme adapts a synthetic-noise denoiser to real noise using very few real-noise pairs and is reported effective in experiments.
- The proposed denoiser uses AIN to regularize the network and prevent overfitting to synthetic noise.
- The transfer-learning scheme updates the AIN module with real-noise data to adjust the synthetic-noise denoiser to real noise.
- Figure 7 compares results on a real noisy image from RNI15 to show the effectiveness of the proposed transfer-learning scheme.
- Table 6 reports average PSNR in dB on the DND test and SIDD validation datasets for denoisers trained with synthetic-noise data.
- The transfer-learning scheme effectively adapts a synthetic-noise denoiser to a real-noise denoiser with very few additional real-noise pairs.
6. Transfer Learning from AWGN
This section evaluates transfer learning for an AINDNet denoiser pretrained with AWGN and adapted using real noisy images. The proposed scheme improves synthetic-noise denoisers when real training data are limited.
- AINDNet(AWGN) is trained with AWGN images as the synthetic-noise baseline.
- AINDNet(AWGN)+TF1 is transfer learned with a single real noisy image.
- AINDNet(AWGN)+TF is transfer learned with 320 real noisy images.
- The proposed transfer-learning scheme significantly improves synthetic-noise denoisers, including an AWGN denoiser, when real training data are limited.
7. More Noise Level Estimation Results
The experiments assess noise-level estimation under diverse signal-dependent and signal-independent noise conditions. The proposed estimator is reported to be accurate and generally lower-variance across tested cases.
- The noise-level estimator is evaluated with images corrupted by diverse signal-dependent noise levels σs and signal-independent noise levels σc.
- The proposed estimator achieves better accuracy with lower standard deviations of errors in most Kodak24 cases.
- The proposed estimator predicts quite accurate estimates when images are corrupted with high σs and σc.
- Table 8 reports average MAE and error STD for Kodak24 images corrupted by heteroscedastic Gaussian noise including the in-camera pipeline.
8. Ablation Study
The ablation study evaluates the noise-level estimator, estimation loss, and transfer-learning update parameters. The proposed training scheme outperforms compared alternatives, while updating the estimator and last convolution contributes measurable gains.
- Noise level estimator and estimation loss: The proposed training scheme, Ours + Lms-asymm, outperforms FCN + Lasymm + LT V and Ours + L1 + LT V.
- Noise level estimator and estimation loss: Table 9 reports average PSNR in dB on the DND test and SIDD validation datasets for noise-level estimator and estimation-loss variants.
- Update parameters: The study compares performance when AIN, the noise-level estimator, or the last convolution is frozen during transfer learning.
- Update parameters: The proposed updates to the noise-level estimator and last convolution contribute 0.1 - 0.2 dB performance gain respectively.
- Update parameters: Fixing the AIN parameters produces even worse performance than the synthetic-noise denoiser.
- Update parameters: Table 10 reports average PSNR in dB on the SIDD validation dataset for transfer-learned denoisers.