PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Learning for Real-World Super-Resolution

Andreas Lugmayr, Martin Danelljan, Radu Timofte

arXiv:1909.09629v1eess.IVcs.CV

TL;DR

Real-world super-resolution lacks paired low- and high-resolution images, and bicubic substitutes distort natural image characteristics. The paper learns an unsupervised inverse mapping to restore real-world low-resolution distributions and generate realistic training pairs. Experiments show improved generalization to natural images and establish a benchmark for evaluating real-world super-resolution.

  • Problem

    Real-world super-resolution lacks paired low- and high-resolution images, while bicubic downsampling alters natural characteristics and creates a distribution mismatch.

  • Method

    The method uses cycle-consistency learning to map bicubic images to the real-world low-resolution distribution, then trains super-resolution on generated pairs with pixel-wise high-resolution supervision.

  • Results

    The approach generalizes to natural images affected by sensor noise and compression artifacts, and outperforms bicubic-trained ESRGAN in the reported real-world setting.

  • Takeaways & Limitations

    The method enables unsupervised real-world super-resolution training and provides a benchmark based on simulated natural degradations.

  • Takeaways & Limitations

    Domain distribution learning can introduce artifacts, especially when training on clean high-quality data where bicubic downsampling has little distributional effect.

Abstract

from arXiv · show

Most current super-resolution methods rely on low and high resolution image pairs to train a network in a fully supervised manner. However, such image pairs are not available in real-world applications. Instead of directly addressing this problem, most works employ the popular bicubic downsampling strategy to artificially generate a corresponding low resolution image. Unfortunately, this strategy introduces significant artifacts, removing natural sensor noise and other real-world characteristics. Super-resolution networks trained on such bicubic images therefore struggle to generalize to natural images. In this work, we propose an unsupervised approach for image super-resolution. Given only unpaired data, we learn to invert the effects of bicubic downsampling in order to restore the natural image characteristics present in the data. This allows us to generate realistic image pairs, faithfully reflecting the distribution of real-world images. Our super-resolution network can therefore be trained with direct pixel-wise supervision in the high resolution domain, while robustly generalizing to real input. We demonstrate the effectiveness of our approach in quantitative and qualitative experiments.

1. Introduction

Real-world super-resolution is difficult because natural low- and high-resolution pairs are unavailable, while bicubic training data remove important image characteristics. The proposed unsupervised approach learns to restore real-world low-resolution distributions, enabling realistic paired training data and improved generalization to natural images.

  • Natural-image super-resolution must handle sensor noise, compression artifacts, and other real-world corruptions.
  • Natural low- and high-resolution image pairs are unavailable, complicating both training and evaluation.
  • Bicubic downsampling enables abundant training data but removes sensor noise and compression artifacts from natural images.
  • Networks trained on bicubic images fail to generalize to natural inputs because bicubic smoothing creates a train–test distribution shift.
  • The method learns an inverse mapping from bicubic images to real-world low-resolution images using cycle consistency, then generates realistic paired data for pixel-wise high-resolution supervision.
  • The work introduces DSR and CSR scenarios, benchmarks real-world super-resolution, and reports qualitative results on real smartphone images.

3. Proposed Method

The method learns to correct bicubic images toward the natural input distribution, then trains super-resolution separately on generated pairs. This preserves real-world image characteristics while avoiding joint-training overfitting.

  • The Super-Resolution Problem: Bicubic downsampling removes natural characteristics such as sensor noise and compression artifacts, creating an input distribution that differs from real images.The resulting mismatch limits the generalization of bicubic-trained super-resolution networks.
  • Super-Resolution Learning: The SR network S is trained in a second stage using generated pairs (ˆX, Y), rather than jointly with G.Separate training avoids the severe overfitting and poor generalization associated with jointly minimizing the cycle-consistency objective.
  • Domain Distribution Learning: The domain distribution network G maps bicubically downsampled output images toward the natural input distribution pX.This learned mapping restores image characteristics that bicubic downsampling suppresses.
  • Domain Distribution Learning: G and its inverse F are trained with adversarial and cycle-consistency losses because paired input-output samples are unavailable.Cycle consistency constrains G and F to behave as approximate inverses while preserving image content.
  • Super-Resolution Learning: The SR network uses generated pairs with pixel-wise content loss, VGG feature loss, and a relativistic GAN discriminator for perceptual quality.The approach adopts an ESRGAN architecture but is described as agnostic to the specific SR architecture.

4. Experiments

The experiments evaluate unsupervised real-world super-resolution under simulated sensor noise and JPEG artifacts using DSR and CSR settings. Across ablations and state-of-the-art comparisons, the proposed approach achieves stronger perceptual quality and fewer artifacts than bicubic-trained or alternative baselines.

  • Experimental Setup: The evaluation constructs training and test data from original images by applying downsampling and synthetic real-world degradations, with DSR and CSR defining the output distributions.DSR uses matching input and output distributions, whereas CSR targets a separate clean-image distribution.
  • Experimental Setup: The experiments use JPEG compression artifacts and simulated sensor noise, and compare methods with PSNR, SSIM, and LPIPS.The study uses JPEG quality 30 and white Gaussian noise with standard deviation σ = 8.
  • Ablation Study: 24% improvement in LPIPS is achieved in the CSR JPEG case over the bicubic-downsampling baseline.The baseline misses appropriate noise levels because bicubic downsampling smooths the inputs, producing severe artifacts in DSR and CSR.
  • Ablation Study: The proposed approach further improves LPIPS by 50% in DSR and 19% in CSR for sensor noise compared with a version that cleans the input first.The comparison suggests that allowing the SR network to act directly on the input image is beneficial.
  • Ablation Study: Low-resolution supervision produces stronger GAN hallucinations and significantly worse LPIPS and PSNR across the evaluated cases.This ablation adds a generator mapping the super-resolved image back to the original domain and supervises reconstruction in the LR domain.
  • State-of-the-art Comparison: The approach reaches performance closer to full-supervision upper bounds and is only slightly worse than full supervision for JPEG artifacts.In the sensor-noise setting, it reduces LPIPS error by 58% for DSR and 44% for CSR, achieving the best overall perceptual quality.
  • State-of-the-art Comparison: For noisy inputs, competing ESRGAN approaches produce strong artifacts, whereas the proposed method avoids them despite using a strong perceptual loss.For JPEG compression, the method also reports lower LPIPS and visually pleasing outputs without the block artifacts seen in other approaches.
  • Real-World Evaluation: The approach generates high-quality images with very few artifacts on real-world data, addressing limitations attributed to an oversimplified problem formulation.The reported real-world evaluation indicates that these limitations are not alleviated simply by more training data or architectural changes.

5. Conclusion

The approach addresses real-world super-resolution without paired data by restoring real-world low-resolution characteristics before training. It also introduces a DIV2K-based benchmark and generalizes to images with multiple real-world degradations.

  • The method learns to restore low-resolution images to the real-world image distribution, avoiding artifacts from bicubic downsampling.
  • Restored images enable generation of realistic training pairs for the super-resolution model.
  • The proposed benchmark uses DIV2K to quantitatively evaluate real-world super-resolution approaches.
  • Experiments on the benchmark and DPED show generalization to natural images affected by sensor noise, compression artifacts, and other effects.

Supplementary Material

The supplementary material adds experiments, ablation details, benchmarking information, and visual examples of the learned domain-transfer mapping.

  • Additional experiments evaluate the approach on the AIM 2019 and NTIRE 2018 challenge datasets.
  • The supplement provides illustrations and further details for the approaches analyzed in the ablation experiments.
  • It also expands the real-world super-resolution benchmarking procedure and presents example outputs of the learned domain-transfer mapping G.

S1. AIM 2019 Real-World Super Resolution Challenge

An additional AIM 2019 experiment uses increasingly complex degradation and evaluates DSR and CSR settings against prior methods. The proposed approach achieves a superior LPIPS score.

  • The AIM 2019 benchmark applies a degradation mapping composed of several different operations to Flickr2K input images.
  • Clean DIV2K training images serve as the target-domain set, with corresponding DIV2K validation and testing splits.
  • The proposed approach achieves a superior LPIPS score compared with previous approaches.

S2. NTIRE 2018 Evaluation

The NTIRE 2018 evaluation tests unsupervised training under realistic mild ×4 degradations, including sensor noise and motion blur. Domain distribution learning achieves the best reported LPIPS among the compared approaches.

  • The evaluation uses NTIRE 2018 Track 2, where unknown degradations simulate realistic sensor noise and motion blur.
  • Although paired images are available, the approach trains using only unpaired low- and high-resolution images.
  • 0.1858 LPIPS is achieved by domain distribution learning with direct high-resolution supervision, outperforming the compared alternatives.The alternatives include ESRGAN at 0.2534, cleaned-input inference at 0.2272, and low-resolution supervision at 0.2038.
  • The comparison reports PSNR, SSIM, and LPIPS, while perceptual performance is assessed primarily with LPIPS.

S3. Details of the Baseline Approaches

The ablation compares baseline and proposed strategies for training and applying real-world super-resolution networks. The proposed approach matches training and testing inputs by learning to invert bicubic downsampling effects.

  • The baseline ESRGAN version finetunes a pretrained model using low-resolution samples generated by bicubic downsampling.
  • The ablation compares three baseline versions and the proposed approach on the AIM Real World Super-Resolution Challenge settings.
  • Bicubic-trained super-resolution produces strong artifacts because training and testing input domains mismatch.
  • A test-time cleaning strategy attempts to match the super-resolution network’s input distribution before applying the network.
  • The proposed approach applies a learned mapping G during training to invert bicubic downsampling effects and match training and testing inputs.Unlike the alternative test-time strategy, it requires no extra input-processing network during inference.

S4. Real-world SR Benchmarking Details

The benchmark constructs real-world super-resolution data with separate procedures for training and evaluation. Training exposes only low-resolution images, while high-resolution images remain unseen until evaluation.

  • The benchmark uses different procedures to construct training and test data for real-world super-resolution.
  • Training data contains only low-resolution images, while original high-resolution images are reserved for evaluation.This keeps the desired output resolution unseen during training, as in real-world applications.
  • Evaluation inputs are created by downsampling an original image and then applying a degradation transformation to simulate real-world conditions.The degradation is applied after downsampling because it is heavily affected by image size.
  • Experiments use DF2K, with 3450 training images and 100 DIV2K validation images, no train-test overlap, 4× downsampling, sensor noise, and JPEG degradation.

S5. Domain Mapping Examples

The learned generator transfers bicubically downsampled images into a natural-image domain containing real-world degradation characteristics. The reported examples include sensor noise and JPEG artifacts, while the ablation evaluates perceptual quality with LPIPS.

  • The generator G transfers downsampled images to a natural domain containing sensor noise and JPEG artifacts.
  • The ablation study compares state-of-the-art methods with three proposed-method versions on the DIV2K Track 2 dataset.
  • The proposed approach achieves superior perceptual quality according to the LPIPS distance in the DIV2K Track 2 comparison.
  • Figure S1 illustrates how four approaches train and apply the super-resolution network, including pixel-wise supervision and discriminator losses.
  • Sensor-noise examples exhibit Gaussian-noise-like image characteristics, while JPEG examples show blocky compression artifacts near sharp edges.

S6. Visual Results

The supplementary visual results include examples from the NTIRE2018 challenge evaluation and the real-world DPED iPhone dataset.

  • Figure S4 presents visual examples from the NTIRE2018 challenge evaluation.
  • Figure S5 presents additional visual examples from the real-world DPED iPhone dataset.
  • The supplementary section uses visual examples to document results across both challenge and real-device datasets.

S7. Failure Cases

The domain-transfer method can introduce artifacts when applied to clean high-quality data, where bicubic downsampling has little effect on the image distribution. On real-world data, qualitative evaluation is perceptual because super-resolution ground truth is unavailable.

  • Clean-data failure case: Clean high-quality data can show slightly more artifacts after domain transfer because bicubic downsampling barely changes its image distribution.The method was designed to match noisy or JPEG-corrupted test inputs, where domain transfer is more relevant.
  • Qualitative diagnostics: The supplementary figures visualize domain-transfer outputs for inputs corrupted with sensor noise and JPEG compression.Each comparison presents the bicubic input, the transferred output, and the real degradation applied to the input.
  • Qualitative evaluation: A qualitative comparison is also reported for the NTIRE2018 Track 2 challenge.
  • Real-world evaluation: On DPED real-world images, the approach achieves superior perceptual quality, but no super-resolution ground truth exists for this dataset.The comparison therefore relies on visual quality rather than ground-truth reconstruction metrics.
Loading 1909.09629v1…