Source-linked AI summary
To learn image super-resolution, use a GAN to learn how to do image degradation first
Adrian Bulat, Jing Yang, Georgios Tzimiropoulos
TL;DR
Existing super-resolution methods mostly rely on synthetically downsampled inputs, limiting their performance on real-world low-quality images. This paper learns realistic degradation with an unpaired High-to-Low GAN, then uses its outputs to train a paired Low-to-High GAN; on real-world face images, the pipeline reports large improvement over baselines and prior work. It does not claim to solve the problem, with failures concentrated in cases such as extreme blur, occlusion, and large pose.
Problem
Most prior methods use artificially downsampled low-resolution images, while real-world degradation includes difficult-to-model blur, noise, and artefacts.
Method
The paper learns degradation with an unpaired High-to-Low GAN, then uses generated degraded images as paired data for a Low-to-High GAN.
Results
The pipeline effectively increases the quality of real-world low-resolution images and reports large improvement over baselines and prior work.
Takeaways & Limitations
The proposed pipeline is potentially applicable beyond faces and targets real-world low-resolution images without assuming artificially generated inputs.
Takeaways & Limitations
Failures occur for inputs that do not resemble faces and for produced faces affected by extreme blur, occlusion, or large pose.
Abstract
from arXiv · showhide
This paper is on image and face super-resolution. The vast majority of prior work for this problem focus on how to increase the resolution of low-resolution images which are artificially generated by simple bilinear down-sampling (or in a few cases by blurring followed by down-sampling).We show that such methods fail to produce good results when applied to real-world low-resolution, low quality images. To circumvent this problem, we propose a two-stage process which firstly trains a High-to-Low Generative Adversarial Network (GAN) to learn how to degrade and downsample high-resolution images requiring, during training, only unpaired high and low-resolution images. Once this is achieved, the output of this network is used to train a Low-to-High GAN for image super-resolution using this time paired low- and high-resolution images. Our main result is that this network can be now used to efectively increase the quality of real-world low-resolution images. We have applied the proposed pipeline for the problem of face super-resolution where we report large improvement over baselines and prior work although the proposed method is potentially applicable to other object categories.
1 Introduction
The paper targets real-world super-resolution, where common synthetic degradation models overlook unknown blur, noise, and artefacts. It learns degradation with an unpaired High-to-Low GAN, then trains a Low-to-High GAN on generated paired data.
- Real-world super-resolution is the paper’s focus, using faces as the specific object category while noting potential application to other categories.
- Most prior methods train on low-resolution images created by bilinear down-sampling or blurred down-sampling rather than real-world degradations.
- Main idea: Unknown blur, compression artefacts, colour noise, and sensor noise make real-world degradation difficult to model accurately.
- Main idea: The High-to-Low GAN learns degradation from unpaired low- and high-resolution image sets, without requiring corresponding image pairs.
- Main idea: The learned High-to-Low GAN realistically degrades and downsamples high-resolution images, creating paired data for training the Low-to-High GAN.
- The proposed pipeline reports large improvement over baselines and prior work on real-world, low-quality, low-resolution Widerface images.
2 Closely related work
Related work largely uses paired training with synthetically downsampled images, while newer methods add perceptual, adversarial, recursive, or texture-based objectives. The paper instead learns degradation from unpaired low- and high-resolution images for real-world super-resolution.
- CNN super-resolution commonly uses paired low- and corresponding high-resolution images with convolutional and upsampling layers.
- Most prior methods synthetically generate low-resolution inputs through bilinear down-sampling, sometimes preceded by blurring, which does not produce good real-world results.
- Image super-resolution: Earlier methods using pixel losses often produce blurry outputs, while perceptual losses compare feature maps from a pretrained network.
- Image super-resolution: GAN-based super-resolution adds a discriminator to pixel- or feature-based losses and is reported to produce more photo-realistic results.
- The paper differs from internal-statistics methods by learning degradation from unpaired low- and high-resolution images before learning super-resolution.
- Face super-resolution: Prior face super-resolution methods generally use paired training and evaluation on frontal face datasets, whereas the cited real-world method is face-specific and landmark-based.
3 Method
The method learns real-world degradation with a High-to-Low GAN, then trains a Low-to-High GAN on the resulting paired data for super-resolution. Its GAN-centered losses model diverse corruptions while preserving facial characteristics.
- A 16 × 16 LR facial image is super-resolved to 64 × 64 by the Low-to-High network, trained with paired LR-HR facial images.
- The High-to-Low network learns degradation and downsampling from HR images using unpaired HR faces and real-world LR faces from disjoint datasets.The HR data come from face-alignment datasets, while the LR data come from blurry, low-quality Widerface images.
- A noise vector makes the High-to-Low problem one-to-many, allowing one HR face to produce multiple LR images with different degradation types and intensities.The learned variations include illumination, blur, colour, and JPEG artefacts, with noise patterns conditioned on image type.
- The High-to-Low GAN loss uses real LR images as unpaired targets, while an L2 loss preserves content such as face identity, pose, and expression.
- The Low-to-High network uses paired training; its GAN loss performs denoising, while the L2 pixel loss enforces content preservation.
- Both networks use weighted GAN and L2 losses, with the GAN term generally receiving greater weight and driving image generation.The paper contrasts this with prior methods in which the L2 loss dominates and the GAN loss mainly sharpens outputs.
4 Results
The evaluation compares the proposed pipeline with variants and related methods on real-world and bilinearly downsampled benchmarks. The proposed method performs best on the real-world LR test set, while remaining competitive under the standard LS3D-W setting, and failure cases remain.
- Evaluation setup: The evaluation uses 3,000 real-world Widerface LR test images with FID and 1,000 LS3D-W images with PSNR.Because Widerface lacks corresponding HR ground truth, its evaluation combines visual results with FID; LS3D-W provides the standard paired benchmark.
- Real-world LR results: The proposed method clearly outperforms the evaluated variants and related methods numerically in FID and visually on the real-world LR test set.Comparisons include bilinear and blurred-bilinear training variants, SRGAN, Wavelet-SRNet, FSRNet, CycleGAN, and DeepDeblur.
- Variant comparison: Adding the GAN loss enables the Low-to-High network to produce high-quality samples and denoise images in most cases, unlike L2-only training.Bilinear and blurred-bilinear variants produce noisy and blurry results, while L2 pixel loss alone does not denoise effectively.
- Real-world LR results: FID 15.27 for High-to-Low outputs versus 23.15 for bilinearly downsampled images shows closer resemblance to original Widerface LR images.The authors use this comparison to assess how faithfully the learned degradation reproduces real-world degradations.
- LS3D-W results: PSNR ≈20 dB on LS3D-W is competitive, although methods trained on bilinearly downsampled and original HR pairs outperform the proposed method there.Those methods have an advantage because the LS3D-W input setting matches their training pairs.
- Failure cases: About 10% of test cases fail, involving non-face outputs or distorted faces under extreme blur, occlusion, or large pose.Large-pose training examples were synthetically warped, which the authors expect to negatively affect performance.
5 Conclusions
The paper concludes with a two-stage pipeline that learns real-world degradation before training super-resolution. It reports improved quality for real-world low-resolution images, alongside documented failure cases.
- 5 Conclusions: The method avoids assuming artificially generated LR inputs and targets real-world low-quality images.The conclusion frames this as the central distinction from standard artificially downsampled-image settings.
- 5 Conclusions: The pipeline first learns degradation with a High-to-Low network from unpaired HR and LR images, then trains a Low-to-High network on its outputs.This two-stage process creates paired training data for real-world super-resolution.
- 5 Conclusions: The pipeline effectively increases the quality of real-world LR images and reports large improvement over baselines and prior work.The paper illustrates qualitative results in Figs. 7 and 8 and failure cases in Fig. 9.