Source-linked AI summary
Unpaired Image Super-Resolution using Pseudo-Supervision
Shunta Maeda
TL;DR
Real-world SR is difficult because paired datasets rely on predetermined degradation while actual LR images lack aligned HR counterparts. The paper combines unpaired kernel/noise correction with pseudo-paired SR training, and reports superior performance across diverse unpaired SR settings, with case-specific hyperparameter tuning still required.
Problem
Paired SR commonly relies on predetermined HR downscaling, but real-world LR images lack corresponding HR images and may involve more varied degradation.
Method
The method separates CycleGAN-based LR-to-clean-LR correction from clean-LR-to-HR SR learning using pseudo-clean LR images generated from HR data.
Results
The method outperformed existing unpaired SR solutions across synthetically degraded natural, real-world face, and real-world aerial datasets.
Takeaways & Limitations
Pseudo-supervision bridges established paired SR training with real-world SR without paired datasets.
Takeaways & Limitations
Hyperparameter tuning is necessary for each dataset to maximize performance, and improving robustness to hyperparameters remains future work.
Abstract
from arXiv · showhide
In most studies on learning-based image super-resolution (SR), the paired training dataset is created by downscaling high-resolution (HR) images with a predetermined operation (e.g., bicubic). However, these methods fail to super-resolve real-world low-resolution (LR) images, for which the degradation process is much more complicated and unknown. In this paper, we propose an unpaired SR method using a generative adversarial network that does not require a paired/aligned training dataset. Our network consists of an unpaired kernel/noise correction network and a pseudo-paired SR network. The correction network removes noise and adjusts the kernel of the inputted LR image; then, the corrected clean LR image is upscaled by the SR network. In the training phase, the correction network also produces a pseudo-clean LR image from the inputted HR image, and then a mapping from the pseudo-clean LR image to the inputted HR image is learned by the SR network in a paired manner. Because our SR network is independent of the correction network, well-studied existing network architectures and pixel-wise loss functions can be integrated with the proposed framework. Experiments on diverse datasets show that the proposed method is superior to existing solutions to the unpaired SR problem.
1. Introduction
The paper targets real-world SR without aligned HR–LR pairs, where predetermined degradation assumptions are inadequate. It separates LR correction from paired SR learning to address limitations of existing GAN-based approaches.
- Motivation: Real-world SR lacks corresponding HR images, making predetermined downscaling-based paired datasets impractical.Most studies synthesize LR images from HR images using operations such as bicubic downscaling.
- Motivation: Blind SR handles limited degradation types, whereas GAN-based unpaired SR avoids assuming a specific degradation process.The cited blind methods focus on restricted forms such as blur, while GAN-based methods learn LR-to-HR mappings directly.
- Existing approaches: Direct GAN approaches cannot use pixel-wise losses for their SR generators, while indirect approaches suffer train–test discrepancy from generated LR distributions.These drawbacks motivate separating correction and SR learning.
- Existing approaches: Figure 2 contrasts direct LR upscaling with indirect HR downscaling followed by paired SR training.The proposed framework is presented as addressing drawbacks associated with both approaches.
- Proposed approach: The proposed method combines an unpaired LR-to-clean-LR correction network with a pseudo-paired clean-LR-to-HR SR network.The correction network is CycleGAN-based, while pseudo-clean LR images enable paired SR training.
2. Related Work
Prior SR research improves reconstruction under predetermined degradation, while blind and unpaired methods address missing paired data with different assumptions and trade-offs. The proposed framework uses pseudo-supervision within an end-to-end data flow.
- Paired image super-resolution: Paired SR commonly uses predetermined HR downscaling and pixel-wise objectives, while perceptual and adversarial losses improve visual texture quality.Examples include bicubic preparation, residual architectures, and perception-oriented models such as SRGAN and ESRGAN.
- Blind image super-resolution: Blind SR methods typically address blur-kernel uncertainty, leaving broader combinations of real-world degradations less covered.The related-work discussion highlights limited attention to blind SR and restricted degradation settings.
- Unpaired image super-resolution: Unpaired SR methods avoid paired datasets through degradation networks, cycle-based learning, or GAN-based domain translation.These approaches aim to upscale LR images while preserving style and local structure.
- Proposed framework: The proposed data-flow diagram shows paired SR learning from pseudo-clean LR images despite unpaired source domains.The full network is described as end-to-end trainable.
3. Proposed Method
The proposed framework separates LR-domain correction from HR upscaling, using adversarial and consistency objectives to produce pseudo-clean LR–HR pairs for SR training. An additional HR discriminator reduces the discrepancy between training inputs and test-time corrected LR images.
- Framework: The framework combines LR correction GXY↓ with upscaling UY↓Y to map unpaired LR samples into HR outputs.GXY↓ maps source LR images to clean-LR domain Y↓, while UY↓Y maps clean LR images to HR images.
- Framework: Pseudo-clean LR images train UY↓Y with pixel-wise reconstruction loss, while an HR discriminator aligns training outputs with test-time outputs.The discriminator operates on UY↓Y(◦y↓) and UY↓Y◦GXY↓(x) to reduce train–test discrepancy.
- Loss Functions: Adversarial objectives optimize both generators against LR-domain discriminators, and an additional discriminator jointly updates the two generators through HR-domain outputs.The framework uses discriminators DX, DY↓, and DX↑ for the correction and alignment objectives.
- Loss Functions: One-sided cycle consistency allows GY↓X to be one-to-many, enabling the framework to represent varied noise types and distributions in the LR source domain.Unlike standard CycleGAN, cycle consistency is imposed on only one direction.
- Loss Functions: Geometric ensemble loss enforces consistent results under image flips and rotations, with eight transformation patterns increasing total training time by approximately 3/2.The loss uses eight distinct flip-and-rotation operators.
- Network Architecture: The method uses reduced RCAN-based architectures for GXY↓ and UY↓Y, while GY↓ combines residual and fusion layers with RGB and random-noise feature heads.The random-noise input simulates distortion randomness, and the SR network can use pixel-wise or other reconstruction losses.
4. Experiments
Experiments across synthetic, real-world face, and aerial-image settings evaluate the method against baselines, ablations, and challenge results. The method generally performs strongly, while comparisons also identify practical caveats and the role of pseudo-supervision.
- Experimental settings: The model was evaluated on realistic-wild images, real-world faces, aerial images, and an additional real-world super-resolution challenge dataset.The experiments use both synthetically degraded natural images and real-world data, including datasets without paired HR–LR training images.
- Comparisons: The method outperformed combined blind restoration and super-resolution baselines by a large margin, although those methods were not trained on the same dataset.The comparison combined blind denoising, deblurring, and SR methods; the authors explicitly note that this comparison was not completely fair.
- Ablation study: The full model outperformed ablated variants, indicating that pseudo-supervision reduced the train–test discrepancy relative to training on generated degraded images.The evaluated variants removed the HR discriminator or changed the data used to train the SR network.
- Perception-oriented training: The perception-oriented model produced more visually pleasing results than the normal model trained with L1 reconstruction loss.The alternative used perceptual, relativistic adversarial, and content losses while keeping the other losses and training procedure unchanged.
- Realistic distortions: 21.09 dB PSNR was obtained by the method versus 20.28 dB for Bulat et al.’s method on the real-world face benchmark.FID was used because corresponding ground-truth images were unavailable, and the method also outperformed other GAN-based unpaired approaches in FID.
- Additional experiment: On AIM 2019 Track 2, the method achieved superior PSNR, SSIM, and LPIPS scores compared with Lugmayr et al.’s unpaired SR method.The challenge setting had no available training HR–LR pairs, and lower LPIPS indicates better perceptual performance.
5. Conclusion
The paper addresses unpaired SR by generating pseudo-clean LR images from HR images and using them to train the SR network in a paired manner. Experiments across diverse synthetic and real-world datasets support the method’s effectiveness, while performance still depends on case-specific hyperparameter tuning.
- Conclusion: Pseudo-supervision generates pseudo-clean LR images from HR images, enabling paired training of the SR network when aligned HR–LR data are unavailable.The authors describe this as bridging existing paired SR methods and real-world SR without paired datasets.
- Conclusion: The method’s effectiveness was demonstrated on synthetically degraded natural images, real-world face images, and real-world aerial images.The conclusion cites experiments across diverse datasets and settings.
- Limitation: Hyperparameter tuning is necessary for each dataset to maximize performance, and improving robustness to hyperparameters is left for future work.This is the stated scope boundary despite applicability to diverse datasets.