Source-linked AI summary
Unsupervised Real-world Image Super Resolution via Domain-distance Aware Training
Yunxuan Wei, Shuhang Gu, Yawei Li, Longcun Jin
TL;DR
Unsupervised real-world SR is limited by the domain gap between generated synthetic LR images and real LR images. DASR addresses this gap through target-domain-aware training and domain-distance-weighted use of pseudo-pairs, and experiments show superior results on synthetic and real datasets.
Problem
Generated LR images used for unsupervised SR remain separated from real LR images, limiting the effectiveness of pseudo-pair training for real-world inputs.
Method
DASR jointly uses generated pseudo-pairs and real LR images, while weighting generated samples by their distance from the real-world LR domain.
Results
Experiments on synthetic and real datasets demonstrate DASR's superiority over competing unsupervised SR approaches.
Takeaways & Limitations
DASR provides an unsupervised real-world SR framework that exploits both target-domain LR data and generated source-domain pairs.
Abstract
from arXiv · showhide
These days, unsupervised super-resolution (SR) has been soaring due to its practical and promising potential in real scenarios. The philosophy of off-the-shelf approaches lies in the augmentation of unpaired data, i.e. first generating synthetic low-resolution (LR) images $\mathcal{Y}^g$ corresponding to real-world high-resolution (HR) images $\mathcal{X}^r$ in the real-world LR domain $\mathcal{Y}^r$, and then utilizing the pseudo pairs $\{\mathcal{Y}^g, \mathcal{X}^r\}$ for training in a supervised manner. Unfortunately, since image translation itself is an extremely challenging task, the SR performance of these approaches are severely limited by the domain gap between generated synthetic LR images and real LR images. In this paper, we propose a novel domain-distance aware super-resolution (DASR) approach for unsupervised real-world image SR. The domain gap between training data (e.g. $\mathcal{Y}^g$) and testing data (e.g. $\mathcal{Y}^r$) is addressed with our \textbf{domain-gap aware training} and \textbf{domain-distance weighted supervision} strategies. Domain-gap aware training takes additional benefit from real data in the target domain while domain-distance weighted supervision brings forward the more rational use of labeled source domain data. The proposed method is validated on synthetic and real datasets and the experimental results show that DASR consistently outperforms state-of-the-art unsupervised SR approaches in generating SR outputs with more realistic and natural textures.
1 Introduction
Real-world SR is difficult because models trained on simulated or generated LR data do not fully match real LR images. DASR addresses this gap by adapting to real LR data and weighting pseudo-pairs by their domain distance.
- Motivation: Real-world SR datasets are costly to collect, and models trained on simulated data can produce artifacts or generalize poorly across capture conditions.These limitations motivate unsupervised approaches that avoid requiring paired real-world data.
- Motivation: Existing unsupervised methods generate LR images from available HR images and use the resulting pseudo-pairs for supervised SR training.They also seek to match the generated LR distribution to real LR observations.
- Motivation: A domain gap remains between generated LR images Yg and real LR images Yr, even when learned down-sampling produces more realistic images than bicubic down-sampling.The paper visualizes this gap using discriminator outputs on images with the same content.
- DASR Framework: DASR trains the SR network with generated pseudo-pairs and real LR images, imposing adversarial constraints on HR estimates from real-world data.This domain-gap aware training introduces target-domain information into SR learning.
- DASR Framework: DASR weights generated pairs according to their distance from the real-world LR domain, emphasizing closer samples and limiting unrealistic samples' contribution.The framework also improves the down-sampling architecture and uses a wavelet-domain adversarial loss.
- Results: Experiments on synthetic and real datasets demonstrate DASR's superiority over competing unsupervised SR approaches.The introduction reports detailed ablation studies and experimental superiority, without specifying a numerical result here.
2 Related Works
CNN-based SR has advanced substantially on simulated benchmarks, but real-world degradation and cross-condition generalization remain challenging. Related work therefore includes blind and test-time adaptation methods, unsupervised SR, and domain adaptation.
- Single Image Super-Resolution with CNNs: CNN-based SR methods learn LR-to-HR mappings using architectures such as residual and densely connected networks.These methods have driven strong performance on benchmark datasets.
- Single Image Super-Resolution with CNNs: Many conventional SR approaches assume simple, uniform degradation in simulated datasets, limiting their fit to general real-world SR.Blind SR estimates degradation kernels, while other methods use testing images for adaptation but still rely on training-time degradation assumptions.
- Single Image Super-Resolution with CNNs: Unsupervised SR methods avoid explicit degradation assumptions by learning from groups of LR images and generating synthetic observations from available HR images.These methods train SR networks with generated pseudo-pairs, but prior approaches ignored the remaining domain gap between generated and real LR data.
- Domain Adaptation: Domain adaptation learns from a labeled source domain to perform well on an unlabeled target domain and has been applied to computer vision tasks including dense estimation.This paper applies the paradigm to low-level image enhancement and describes its work as the first such attempt for SR.
3 DASR for Unsupervised Real-World Image SR
DASR addresses the domain gap between generated and real-world LR images by combining domain-gap aware training with domain-distance weighted supervision. Its SRN uses labeled generated pairs alongside unlabeled real LR data, while domain-distance information adjusts source supervision.
- Framework: The down-sampling network generates synthetic LR-HR pairs from unpaired HR data, avoiding the bicubic preprocessing used by earlier approaches.The proposed architecture is intended to preserve information that bicubic downsampling may remove.
- Framework: DASR trains an SR network with generated LR-HR pairs and real-world LR images under a domain adaptation setting.The framework targets unsupervised real-world SR while using both source-domain labels and target-domain data.
- Down-Sampling Network: DSN training combines content and perceptual losses for consistency with bicubic-downsampled HR images, plus adversarial losses for translation between generated and real LR domains.The adversarial loss operates on Haar-wavelet high-frequency components, using LH, HL, and HH subbands to characterize image details.
- Motivation: The method addresses the performance drop caused by applying an SR network trained on synthetic data to real-world LR images.DASR combines target-domain adaptation with adaptive use of generated source-domain supervision.
- Domain-gap aware training: Domain-gap aware training applies supervised losses to generated pairs and adversarial losses to SR outputs from unlabeled real LR images.The target-domain outputs are aligned with the real HR distribution during SRN training.
- Domain-distance weighted supervision: Domain-distance weighted supervision assigns different importance to generated samples and image areas according to their distance from the real-world LR domain.A discriminator-derived distance map provides patch-level weights for adaptively adjusting source-domain losses.
4 Experimental Results on Synthetic Datasets
Experiments on the AIM synthetic dataset evaluate down-sampling design choices and the two proposed training strategies. DASR’s components generate more target-domain-like LR images, improve SR quality, and work best together.
- Experimental setting: The AIM experiments use 2,650 noisy, compressed images with unknown degradation and 800 clean HR images, with MOS, PSNR, and SSIM as evaluation metrics.MOS is averaged over 26 candidates, while PSNR and SSIM provide additional quantitative reference.
- Down-sampling network: Wavelet-space adversarial training and the proposed DSN architecture generate LR images that are more similar to the real target domain.The ablation compares HR or bicubic inputs with GBFS, RGB, and WFS adversarial-training spaces.
- Training strategies: Domain-gap aware training consistently improves SR performance, producing higher-quality HR estimates with better MOS and LPIPS than source-domain adversarial training.The comparison uses either DSN-generated synthetic pairs or bicubic LR-HR pairs as source-domain data.
- Training strategies: Domain-distance weighted supervision improves the SR network over baseline models by adaptively exploiting paired source-domain training data.The weighting strategy accounts for different distances between generated samples and the real-world LR domain.
- Training strategies: Combining domain-gap aware training with domain-distance weighted supervision yields significant improvement over using either strategy alone.The two strategies are reported to be complementary in the AIM ablation study.
- Comparison with state-of-the-art methods: Compared with ZSSR, FSSR, CinCGAN, and pre-trained ESRGAN, DASR achieves superior quantitative and visual results on real-world SR comparisons.FSSR won the AIM Challenge, while pre-trained ESRGAN was trained on synthetic bicubic data; Fig. 5 shows AIM Challenge test-image comparisons.
5 Experimental Results on Real-World Images
DASR is evaluated on RealSR and CameraSR using unpaired real LR images and DIV2K HR images, with paired validation data enabling quantitative assessment. It outperforms competing blind or unsupervised methods on both datasets, while remaining comparable to supervised ESRGAN on RealSR LPIPS.
- Experimental setup: Experiments use RealSR and CameraSR, training with their real LR images plus 800 DIV2K HR images and evaluating on paired validation data.RealSR uses 200 Canon-camera LR training images and a 100-pair validation set; CameraSR uses 100 iPhoneX LR training images.
- RealSR dataset: 595 real LR-HR pairs were collected for RealSR by varying camera focal length, with aligned pairs produced through image registration.The experiments use the 200 LR images collected by the Canon camera.
- RealSR results: DASR significantly outperforms other blind super-resolution methods on RealSR in both LPIPS and MOS, while achieving comparable LPIPS to supervised ESRGAN.The comparison includes ZSSR, FSSR, CinCGAN, pre-trained ESRGAN, and supervised ESRGAN.
- CameraSR results: DASR outperforms FSSR by a large margin on CameraSR after removing down-sampling and up-sampling operations because the dataset’s LR and HR images have the same spatial size.The CameraSR evaluation uses the iPhoneX subset.
6 Conclusions
The paper concludes that DASR addresses unsupervised real-world SR by combining synthetic pairs with real LR images under domain adaptation. Its two training strategies improve use of target-domain data and synthetic source-domain supervision, with experiments demonstrating effectiveness on synthetic and real-world datasets.
- Framework: DASR generates synthetic LR images in the real-world LR distribution, then trains the SR network with synthetic pairs and real LR images using domain adaptation.The framework assumes only unpaired data.
- Training strategies: Domain-gap aware training introduces adversarial loss in the target domain, while domain-distance weighted supervision improves use of synthetic source-domain data.These strategies address the gap between generated LR images and real images.
- Conclusion: Experiments on synthetic and real-world datasets demonstrate the effectiveness of DASR for real-world super-resolution.