Source-linked AI summary
Syn2Real Transfer Learning for Image Deraining using Gaussian Processes
Rajeev Yasarla, Vishwanath A. Sindagi, Vishal M. Patel
TL;DR
Image deraining methods are constrained by scarce fully labeled real-world data and poor transfer from synthetic training data. The paper proposes a GP-based semi-supervised framework that generates latent-space pseudo ground truth for unlabeled real-world images. Across several datasets, it reports better generalization, on-par performance with fully labeled training using limited labels, and superior performance to existing methods.
Problem
Fully supervised deraining relies on scarce labeled data, while synthetic datasets often perform sub-optimally on real-world images because of distributional shift.
Method
The method jointly models labeled and unlabeled latent-space vectors with a Gaussian process, generating pseudo ground truth to supervise unlabeled samples.
Results
The proposed method achieves better generalization by leveraging unlabeled data and can obtain performance comparable to 100% labeled training with only a fraction of labeled data.
Takeaways & Limitations
Unlabeled real-world images can be incorporated into GP-based semi-supervised deraining to improve performance over using labeled synthetic data alone.
Takeaways & Limitations
The deraining formulation assumes the rainy image is an additive combination of clean image and rain component.
Abstract
from arXiv · showhide
Recent CNN-based methods for image deraining have achieved excellent performance in terms of reconstruction error as well as visual quality. However, these methods are limited in the sense that they can be trained only on fully labeled data. Due to various challenges in obtaining real world fully-labeled image deraining datasets, existing methods are trained only on synthetically generated data and hence, generalize poorly to real-world images. The use of real-world data in training image deraining networks is relatively less explored in the literature. We propose a Gaussian Process-based semi-supervised learning framework which enables the network in learning to derain using synthetic dataset while generalizing better using unlabeled real-world images. Through extensive experiments and ablations on several challenging datasets (such as Rain800, Rain200H and DDN-SIRR), we show that the proposed method, when trained on limited labeled data, achieves on-par performance with fully-labeled training. Additionally, we demonstrate that using unlabeled real-world images in the proposed GP-based framework results in superior performance as compared to existing methods. Code is available at: https://github.com/rajeevyasarla/Syn2Real
1. Introduction
Image deraining methods often rely on fully labeled synthetic data, limiting generalization to real-world rainy images. The paper proposes GP-based semi-supervised learning to use unlabeled real-world data and recover performance with limited labels.
- Rain artifacts degrade image quality and downstream tasks such as object detection and recognition.
- Real-world labeled training data is scarce, while synthetic datasets can perform sub-optimally on real images because of distributional shift.
- Existing deraining networks are largely fully supervised and can leverage only fully labeled training data.
- Prior SIRR semi-supervised learning uses a GMM for unlabeled rain residuals and KL divergence to align synthetic and real rain distributions.
- The proposed GP-based framework uses a non-parametric approach to generate pseudo ground truth for unlabeled data and supervise the network at the intermediate level.
2. Related work
Related work spans single-image and video-based deraining, with deep models increasingly learning mappings from rainy to clean images using paired data. This paper focuses on single-image deraining augmented with unlabeled real-world images.
- Image deraining methods are classified into single-image and video-based techniques.
- Single-image methods reconstruct rain-free images using priors, patch modeling, or deep learning trained on paired data.
- Recent single-image approaches include CNN, GAN, densely connected, recurrent, depth-based, and spatial-attention methods.
- The paper focuses on single-image deraining that specifically leverages additional unlabeled real-world data.
3. Background
The paper formulates deraining as an inverse problem under an additive rain model and introduces semi-supervised learning and Gaussian-process foundations. Synthetic paired data supplies labels, while real-world images remain unlabeled.
- 3.1. Single image de-raining: The additive model represents a rainy image as x = y + r, where y is clean content and r is the rain component.
- 3.1. Single image de-raining: Single-image deraining estimates clean image y from rainy image x, and this work estimates the rain component before subtracting it.
- Semi-supervised learning jointly uses labeled input-target pairs and unlabeled inputs to learn predictions for unseen samples.
- The labeled dataset consists of synthetically generated rain pairs, while real-world images form the unlabeled dataset and are leveraged through pseudo-labeling.
- 3.3. Gaussian processes: A Gaussian process is an infinite collection of random variables whose finite subsets are jointly Gaussian, specified by mean and covariance functions.
- 3.3. Gaussian processes: GP predictions at unlabeled points are obtained from a Gaussian posterior by conditioning on observed data.
4. Proposed method
The proposed framework alternates supervised learning on labeled synthetic data with Gaussian-process-based supervision from unlabeled real-world images. It models latent vectors to generate pseudo-ground truth, using nearby labeled vectors to support prediction and adaptation.
- Network architecture: The encoder maps each rainy image to an intermediate latent vector, while the decoder predicts rain streaks that are subtracted from the input to obtain the clean image.The predicted rain component is r = g(h(x, θenc), θdec).
- Training framework: The network alternates labeled and unlabeled training phases, minimizing supervised and unsupervised losses over synthetic and real-world data.The total objective is Ltotal = Lsup + λunsupLunsup.
- Labeled training phase: During labeled training, the supervised objective combines l1 loss and perceptual loss between predictions and ground truth.The loss is defined as Lsup = L1 + λpLp.
- Unlabeled training phase: For each unlabeled latent vector, the method jointly models labeled and unlabeled latent vectors with a Gaussian process to obtain a conditional pseudo-ground-truth vector.The unlabeled latent vector is represented using labeled latent vectors with additive Gaussian noise, then conditioned through the GP.
- Unlabeled training phase: The method uses the Nn nearest labeled latent vectors for GP prediction because unrelated vectors may differ in image content and rain-streak properties, while using all vectors is computationally prohibitive.The nearest-neighbor restriction is applied to the GP formulation for each unlabeled vector.
- Unlabeled training phase: The GP mean supplies latent-space pseudo-ground truth, whose prediction error updates the encoder; variance terms are also optimized to regulate the prediction.The unlabeled loss includes latent prediction error and a variance-related term.
- Evaluation setup: Table 1 evaluates labeled-only training against combined labeled and unlabeled training on synthetic DDN-SIRR data for Dense and Sparse rain categories using PSNR.The caption states that the proposed method achieves a better PSNR gain than SIRR in both categories.
5. Experiments and results
Experiments evaluate the framework on DDN-SIRR, Rain800, and Rain200H, comparing synthetic and real-world performance and testing how unlabeled data helps with limited labels. Across these settings, the proposed method improves over relevant baselines and can approach full-label performance using fewer labeled examples.
- Use of real-world data: The DDN-SIRR experiments compare the proposed framework with existing deraining methods on synthetic and real-world test sets.The real-world evaluation uses NIQE and BRISQUE, for which lower scores indicate better performance.
- Use of real-world data: The proposed method achieves significantly better synthetic-test results than existing approaches and outperforms SIRR with significant margins.The synthetic comparison uses PSNR and includes methods that can use only synthetic data.
- Use of real-world data: The proposed method achieves better real-world-test performance than SIRR and obtains greater gains from unlabeled data.The real-world test set is evaluated with no-reference quality metrics, where lower scores are better.
- Use of real-world data: Using unlabeled real-world data with the GP-based framework results in better generalization than training without the unlabeled data.This conclusion is drawn from experiments comparing labeled synthetic training with and without additional real-world images.
- SSL experiments: In the labeled-data ablation, training uses 10%, 20%, 40%, 60%, or 100% labeled data, with the remainder treated as unlabeled data.Performance is evaluated using PSNR and SSIM on Rain800 and Rain200H.
- SSL experiments: 23.74 dB with 100% labeled data falls to 22.6 dB with 40% labeled data, while unlabeled data improves performance consistently across label fractions.With 60% labeled data plus unlabeled data, performance is comparable to using 100% labeled data.
- SSL experiments: Qualitative results with 10% and 40% labeled data show better performance when additional unlabeled data is used.These comparisons are illustrated in Figures 6 and 7.
6. Conclusion
The GP-based semi-supervised framework leverages unlabeled data during training for image deraining. It estimates latent-space pseudo-ground truth by jointly modeling labeled and unlabeled vectors, improving generalization.
- The framework uses Gaussian Processes to leverage unlabeled data during image-deraining training.
- It trains labeled samples with supervised l1 and perceptual losses.
- It jointly models labeled and unlabeled latent-space vectors to estimate pseudo-GT for unlabeled samples.
- Experiments on Rain800, Rain200H, and DDN-SIRR demonstrate better generalization when unlabeled data are leveraged.