Source-linked AI summary
Frequency Separation for Real-World Super-Resolution
Manuel Fritsche, Shuhang Gu, Radu Timofte
TL;DR
Conventional SR training relies on paired images or bicubic downscaling, which alters real-world corruptions and leaves models poorly matched to practical inputs. The paper proposes DSGAN and frequency-separated losses to generate characteristic-matched LR data and train SR models; experiments show effectiveness on artificial and natural corruptions, and the method won both AIM 2019 challenge tracks.
Problem
Conventional SR models rely on paired HR/LR data or bicubic downscaling, which alters corruptions such as sensor noise and makes paired same-source collection difficult.
Method
DSGAN modifies bicubically downsampled images to match source characteristics, while frequency-separated training applies adversarial losses to high frequencies and pixel-wise losses to low frequencies.
Results
Experiments on artificial and natural corruptions demonstrate the approach's effectiveness, and it won first place in both AIM 2019 source-domain and target-domain tracks.
Takeaways & Limitations
The approach generates matched-characteristic training pairs and can be applied to images with characteristics beyond those used in the experiments.
Takeaways & Limitations
The setup assumes source images have similar characteristics, such as the same sensor-noise distribution; in target-domain SR, it assumes bicubic downsampling removes almost all corruptions.
Abstract
from arXiv · showhide
Most of the recent literature on image super-resolution (SR) assumes the availability of training data in the form of paired low resolution (LR) and high resolution (HR) images or the knowledge of the downgrading operator (usually bicubic downscaling). While the proposed methods perform well on standard benchmarks, they often fail to produce convincing results in real-world settings. This is because real-world images can be subject to corruptions such as sensor noise, which are severely altered by bicubic downscaling. Therefore, the models never see a real-world image during training, which limits their generalization capabilities. Moreover, it is cumbersome to collect paired LR and HR images in the same source domain. To address this problem, we propose DSGAN to introduce natural image characteristics in bicubically downscaled images. It can be trained in an unsupervised fashion on HR images, thereby generating LR images with the same characteristics as the original images. We then use the generated data to train a SR model, which greatly improves its performance on real-world images. Furthermore, we propose to separate the low and high image frequencies and treat them differently during training. Since the low frequencies are preserved by downsampling operations, we only require adversarial training to modify the high frequencies. This idea is applied to our DSGAN model as well as the SR model. We demonstrate the effectiveness of our method in several experiments through quantitative and qualitative analysis. Our solution is the winner of the AIM Challenge on Real World SR at ICCV 2019.
1. Introduction
Real-world corruptions make conventional supervised SR unreliable because bicubic-generated training pairs do not preserve source-image characteristics. The paper addresses this with DSGAN and frequency-separated training for both data generation and super-resolution.
- Real-world sensor noise can produce strange artifacts in super-resolved images.
- Bicubic downscaling is convenient for creating LR images but alters corruptions, limiting generalization to real-world images.
- DSGAN generates LR images whose characteristics match source images by first bicubically downsampling HR images and then modifying their characteristics with a GAN.
- Frequency separation applies color or pixel-wise losses to low frequencies and adversarial losses to high frequencies in both DSGAN and the SR model.The approach treats low frequencies as preserved by downsampling and uses adversarial training to model high-frequency characteristics.
- Experiments on artificial and natural corruptions use quantitative and qualitative evaluations, including sensor noise, compression artifacts, and real-world DPED images.
- The method won first place in both source-domain and target-domain tracks of the AIM 2019 Real World Super-Resolution Challenge.
2. Related Work
Related SR methods improve fidelity or perceptual quality but generally rely on bicubic-generated paired data, which does not represent real-world corruptions. Alternative approaches collect camera-specific pairs, adapt to each test image, or add cleaning stages, whereas this work focuses on generating training data with limited model changes.
- CNN-based SR methods evolved from end-to-end LR-to-HR mappings toward deeper residual networks and improved architectures.
- L1 and L2 optimization can achieve strong PSNR while producing blurry images, motivating perceptual losses and training methods for visual quality.
- Bicubic-generated HR/LR pairs cause poor real-world performance because downscaling alters image characteristics such as sensor noise.
- Paired-data collection requires complicated hardware and new data for each camera source, while test-image adaptation can be slow or depend on a known downsampling operation.
- Compared with cleaning-based or test-specific methods, this approach mainly generates training data and makes only small discriminator and loss-function modifications.
3. Proposed Method
The proposed method translates bicubically downsampled HR images into the source domain and applies frequency separation so low frequencies remain faithful while adversarial training focuses on high-frequency characteristics.
- 3.2. Downsampling with Domain Translation: DSGAN maps bicubically downsampled images into the source domain while preserving their image content.It generates LR images intended to be indistinguishable from source images, using HR images as input and source images as real GAN data.
- 3.3. Frequency Separation: Frequency separation targets high-frequency characteristics because downsampling preserves low-frequency information while removing high frequencies.The discriminator receives high-pass-filtered outputs, while low frequencies remain close to the original downsampled image.
- 3.3. Frequency Separation: The downsampling generator combines color, texture, and perceptual losses to preserve low frequencies, model texture, and keep outputs perceptually consistent.Color loss uses an L1 constraint on low frequencies, texture loss is adversarial, and LPIPS supplies the perceptual loss.
- 3.4. Frequency Separation for Super-Resolution: For SR, low frequencies are learned with pixel-wise loss, whereas adversarial loss is applied only to high frequencies with a high-pass filter before the discriminator.The high-frequency component has multiple possible ground-truth values, while the low-frequency component can be reconstructed directly from the input.
- 3.4. Frequency Separation for Super-Resolution: The frequency-separated loss simplifies discrimination and enables outputs that match the target distribution more closely.The approach is applied directly to ESRGAN and is described as adaptable to other SR models.
4. Experiments
The experiments evaluate DSGAN and frequency-separated ESRGAN variants on artificial corruptions, real-world images, and the AIM2019 challenge. Across these settings, the proposed models reduce corruptions while preserving sharpness and achieve strong challenge results.
- Experimental Setup: DSGAN generates 4× HR/LR training pairs from bicubically downscaled 512×512 patches for all experiments.The discriminator uses random 128 × 128 crops of source images, with training lasting 200 or 300 epochs depending on the setting.
- Comparison with State-of-the-Art: On corrupted images, conventional methods introduce severe artifacts, whereas the proposed models produce outputs with almost no corruptions and satisfying LPIPS values.ESRGAN has the worst LPIPS values among state-of-the-art methods, while RankSRGAN performs best among those baselines but still introduces significant corruptions.
- Comparison with State-of-the-Art: On DPED iPhone 3 images without ground truth, the proposed models produce sharp images and greatly reduce corruptions compared with the evaluated state-of-the-art methods.ESRGAN and RankSRGAN introduce strong artifacts, EDSR reduces them slightly but is blurry, and ZSSR produces visually similar blurry results.
- Comparison with State-of-the-Art: In the AIM2019 Real World SR challenge, the method won first place in both source-domain and target-domain tracks.For both SDSR and TDSR, it achieved the lowest MOS; Figure 7 also reports sharp outputs with few corruptions and removal of compression-artifact block structures.
- Ablation Study: Using DSGAN greatly improves performance over bicubic downscaling, while ESRGAN-FS better matches source characteristics and produces convincing artifacts across image regions.ESRGAN-FS can be slightly sharper than ESRGAN, sometimes yielding a slightly worse LPIPS score when its added details differ from ground truth.
5. Conclusion
The paper proposes DSGAN and frequency separation to generate and super-resolve images with real-world characteristics. Experiments show effectiveness on artificial and natural corruptions, outperforming state-of-the-art methods and winning the AIM 2019 Challenge.
- DSGAN generates paired HR and LR images with similar characteristics.
- Adversarial loss is applied only to high frequencies, where relevant image characteristics mainly appear.
- Frequency separation is also applied to the SR model to match the target distribution more closely.
- Experiments with artificial and natural corruptions demonstrate the approach's effectiveness for real-world SR.
- The approach beats state-of-the-art methods and won the AIM 2019 Challenge on Real World Super-Resolution.