Source-linked AI summary
Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networks
Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, Liang Lin
TL;DR
The paper studies single-image super-resolution without paired HR-LR data or a known down-sampling process, with LR inputs further degraded by noise and blur. It proposes an unsupervised Cycle-in-Cycle GAN that restores LR images, upsamples them with a pre-trained model, and jointly fine-tunes both modules. On NTIRE2018 datasets, the method achieves comparable results to state-of-the-art supervised models, while a single network can produce unstable artifacts under varying degradations.
Problem
Super-resolution lacks a practical solution when paired HR-LR data and the down-sampling and degradation functions are unavailable, especially for noisy and blurry LR inputs.
Method
CinCGAN uses two nested CycleGANs to restore degraded LR images, applies a pre-trained deep up-sampling model, and fine-tunes the modules end to end.
Results
CinCGAN achieves comparable performance to state-of-the-art supervised CNN methods on the NTIRE2018 Super-Resolution Challenge dataset.
Takeaways & Limitations
Unsupervised Cycle-in-Cycle learning provides a competitive approach to super-resolution when paired training data and degradation operators are unavailable.
Takeaways & Limitations
A single network produces unstable results and undesired artifacts when it must denoise, deblur, and up-sample images with varying degradation kernels.
Abstract
from arXiv · showhide
We consider the single image super-resolution problem in a more general case that the low-/high-resolution pairs and the down-sampling process are unavailable. Different from traditional super-resolution formulation, the low-resolution input is further degraded by noises and blurring. This complicated setting makes supervised learning and accurate kernel estimation impossible. To solve this problem, we resort to unsupervised learning without paired data, inspired by the recent successful image-to-image translation applications. With generative adversarial networks (GAN) as the basic component, we propose a Cycle-in-Cycle network structure to tackle the problem within three steps. First, the noisy and blurry input is mapped to a noise-free low-resolution space. Then the intermediate image is up-sampled with a pre-trained deep model. Finally, we fine-tune the two modules in an end-to-end manner to get the high-resolution output. Experiments on NTIRE2018 datasets demonstrate that the proposed unsupervised method achieves comparable results as the state-of-the-art supervised models.
1. Introduction
The paper addresses super-resolution when paired HR-LR data and the down-sampling process are unavailable, while LR inputs may also be noisy and blurry. It proposes CinCGAN, an unsupervised Cycle-in-Cycle pipeline that restores and up-samples degraded inputs, achieving competitive performance with supervised methods.
- For noisy DIV2K input, CinCGAN reconstructs clean results with fine details, whereas bicubic interpolation and EDSR produce low-quality results.BM3D+EDSR first denoises with BM3D and then applies EDSR for super-resolution.
- Real-world super-resolution may lack HR datasets and a known down-sampling process, while LR inputs can be noisy and blurry.
- Unknown and complex degradations make supervised learning on synthetic pairs and accurate blind-SR kernel estimation impractical.
- CinCGAN uses two nested CycleGANs to map degraded LR images into a clean, bicubic-downsampled LR space before super-resolution.
- The pipeline stacks a pre-trained deep up-sampling model after LR restoration, then fine-tunes the modules jointly with adversarial learning.
- Experiments on the NTIRE2018 Super-Resolution Challenge dataset report stable training and competitive performance with supervised deep-learning methods.
2. Related work
Prior super-resolution work includes statistical, model-based, sparse-coding, and deep CNN approaches, while blind SR addresses unknown degradation through kernel estimation. The paper positions CinCGAN as an unsupervised alternative for unknown degradations and differing input-output resolutions.
- Early SISR methods relied on natural-image statistics, predefined models, and sparse-coding mappings between LR and HR images.
- Deep CNN methods improved SR quality through architectures such as VDSR, FSRCNN, and ESPCN.
- Blind SR methods estimate unavailable degradation or blur kernels using internal or external image-patch information and related priors.
- The paper investigates deep learning for blind SR because unknown, complex, and coupled degradations make accurate kernel estimation difficult.
- GAN-based unsupervised learning uses generators and discriminators, with DualGAN and CycleGAN translating between domains using unpaired data.
- Unlike DualGAN and CycleGAN, which preserve image size, CinCGAN targets SR's larger output resolution and achieves comparable performance with supervised CNN algorithms.
3. Proposed Method
CinCGAN addresses super-resolution without paired HR-LR data or known degradation processes by coupling two CycleGANs around restoration and super-resolution. It first maps degraded LR images to clean LR images, then super-resolves them and jointly fine-tunes the pipeline.
- Problem formulation: The problem assumes unavailable HR-LR pairs, down-sampling, and degradation functions, with LR inputs affected by complex noise, blur, and shifts.The degraded input is modeled as x = fn(fd(z)) + n, where fd and fn are unavailable.
- Motivation: Direct CycleGAN application is unsuitable because super-resolution changes image size and interpolation can amplify noisy patterns, destabilizing training.The method also notes difficulty handling image-varying degradation functions in one forward pass.
- Solution pipeline: CinCGAN uses two coupled CycleGANs: one maps degraded LR images X to clean bicubic-downsampled LR images Y, and the other maps Y to HR images Z.The two cycles use unpaired xi ∈ X, yj ∈ Y, and zj ∈ Z, with yj bicubic-downsampled from zj.
- LR Image Restoration: The first LR→clean LR cycle uses G1, G2, and D1, with adversarial, cycle-consistency, identity, and total-variation losses.G1 restores the LR image, G2 reconstructs the input through the cycle, identity loss limits color variation, and TV loss imposes spatial smoothness.
- Jointly Restoration and Super-Resolution: The second cycle stacks a pretrained EDSR super-resolution network after G1 and uses G3 and D2 for adversarial and cycle-consistency training.Its identity loss is redesigned to encourage adequate super-resolved image quality, and the LR-to-HR module is fine-tuned jointly.
- Network Architecture: Generators G1 and G2 use head and tail convolutions with six residual blocks, while G3 changes two convolution strides to perform down-sampling.D2 is a 70 × 70 PatchGAN; training uses 32 × 32 LR images and 128 × 128 HR images.
4. Experiments
Experiments evaluate CinCGAN on the NTIRE2018 track 2 setting, using unpaired degraded LR and HR data, comparisons with supervised and denoising-based baselines, and ablations of its cycle structure. CinCGAN performs comparably to paired-data SRGAN+ and its complete structure produces the best visual ablation result.
- Experimental Setup: Training uses NTIRE2018 track 2 data, with separate LR and HR image subsets, bicubic-downsampled intermediate clean LR images, and 4× scaling.The setup uses the first 400 LR training images as inputs and the other 400 HR images as target-domain data.
- Comparisons: CinCGAN is compared with FSRCNN, EDSR, SRGAN, paired-data fine-tuned EDSR+ and SRGAN+, and BM3D+EDSR.BM3D+EDSR first denoises degraded LR images with BM3D before applying EDSR.
- Results: CinCGAN performs comparably to SRGAN+ on PSNR and SSIM without paired training data, while BM3D+EDSR scores lower because BM3D over-smooths inputs.The same evaluation reports that FSRCNN and EDSR do not work well when blur and noise are unknown during training.
- Ablation Study: The single-CycleGAN structure is unstable and produces undesired artifacts because one network must denoise, deblur, and up-sample degraded images simultaneously.This structure corresponds to directly transforming LR images into HR images.
- Ablation Study: Removing D2 and G3 yields a clean-LR-to-SR pipeline, whereas removing D1 and G2 leaves reconstructed HR images noisy because G1 cannot denoise and deblur effectively.The latter model becomes similar to the direct single-CycleGAN structure.
- Ablation Study: The complete CinCGAN jointly fine-tunes LR-to-clean-LR and LR-to-HR networks, with G1 denoising and deblurring before SR further restores and up-samples the intermediate image.Its final output shows the best visual result among the four tested structures.
5. Conclusions
The proposed unsupervised Cycle-in-Cycle method addresses super-resolution without paired low-/high-resolution data or a known down-sampling process. It maps degraded inputs to clean low-resolution images, up-samples them, and jointly fine-tunes the modules to produce high-resolution outputs with results comparable to supervised models.
- 5. Conclusions: The method uses two nested CycleGANs to solve super-resolution under unavailable paired data and unknown down-sampling.The second GAN covers the first one within the Cycle-in-Cycle structure.
- 5. Conclusions: First, the input low-resolution images are mapped to a clean, bicubic-downsampled low-resolution space.
- 5. Conclusions: A pre-trained deep model then up-samples the intermediate result, after which both modules are fine-tuned end-to-end.
- 5. Conclusions: The unsupervised method achieves results comparable to state-of-the-art supervised models.