PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Xintao Wang, Liangbin Xie, Chao Dong, Ying Shan

arXiv:2107.10833v2eess.IVcs.CV

TL;DR

Real-world blind super-resolution must handle degradations more complex than idealized synthetic assumptions. Real-ESRGAN addresses this with pure-synthetic training pairs built from high-order degradation modeling, artifact-aware filtering, and a U-Net discriminator with spectral normalization, achieving stronger visual restoration across real-world images. Its scope remains limited by artifacts and out-of-distribution degradations that it cannot reliably remove.

  • Problem

    Conventional and blind SR methods remain limited by the mismatch between simple synthetic degradations and unknown, complex real-world degradations.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    Real-ESRGAN trains ESRGAN with pure synthetic pairs using high-order degradation modeling, sinc filters for ringing and overshoot artifacts, and a U-Net discriminator with spectral normalization.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    Real-ESRGAN trained with synthetic data restores most real-world images, enhancing details while removing artifacts and achieving better visual performance than previous works.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Pure synthetic training can support practical real-world blind SR when degradation synthesis and adversarial training are adapted to complex artifacts and degradation spaces.

    Source: §1 · p. 2, §5 · p. 8

    The method cannot remove out-of-distribution complicated real-world degradations and may amplify their artifacts.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Though many attempts have been made in blind super-resolution to restore low-resolution images with unknown and complex degradations, they are still far from addressing general real-world degraded images. In this work, we extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data. Specifically, a high-order degradation modeling process is introduced to better simulate complex real-world degradations. We also consider the common ringing and overshoot artifacts in the synthesis process. In addition, we employ a U-Net discriminator with spectral normalization to increase discriminator capability and stabilize the training dynamics. Extensive comparisons have shown its superior visual performance than prior works on various real datasets. We also provide efficient implementations to synthesize training pairs on the fly.

1. Introduction

Real-ESRGAN addresses the mismatch between idealized SR degradations and complex real-world images by combining high-order synthetic degradation modeling with architectural changes for stable, detailed restoration.

  • Most SR methods assume bicubic downsampling, creating a degradation mismatch that limits their practicality on real-world images.Source: §1 · p. 1
  • Blind SR methods model unknown degradations explicitly or implicitly, but simple degradation combinations fail on complex samples and learned models generalize poorly outside their training distributions.Source: §1 · pp. 1–2
  • Real-ESRGAN extends ESRGAN by synthesizing training pairs with a high-order degradation process that repeats classical degradation operations using different hyper-parameters.Source: §1 · p. 2
  • Sinc filters are incorporated to simulate common ringing and overshoot artifacts during degradation synthesis.Source: §1 · p. 2
  • A U-Net discriminator with spectral normalization increases discriminator capability and stabilizes training, supporting local detail enhancement and artifact suppression.Source: §1 · p. 2

2. Related Work

Related blind SR work uses explicit degradation representations, real or unpaired data, and synthesized pairs, but these approaches remain constrained by degradation assumptions or data coverage.

  • Blind SR methods using explicit degradation representations commonly separate or jointly perform degradation prediction and conditional restoration.Source: §2 · p. 2
  • Explicit approaches rely on predefined degradation types and levels and usually model simple synthetic degradations.Source: §2 · p. 2
  • Other methods construct training pairs from camera captures, unpaired data, or estimated blur kernels and extracted noise patches.Source: §2 · p. 2

3. Methodology

Real-ESRGAN synthesizes training pairs with repeated classical degradation processes to better represent complex real-world degradations, while adding artifact modeling and architectural changes for practical restoration.

Classical degradation model

  • The classical degradation model applies blur, resizing, noise, and JPEG compression to synthesize low-resolution inputs.Blur is modeled with convolutional kernels; resizing includes downsampling and upsampling, while Gaussian and Poisson noise are considered.Source: §3.1 · p. 3

High-order degradation model

  • Because first-order synthesis leaves a gap from realistic degraded images, Real-ESRGAN uses repeated degradation processes with different hyper-parameters.Each repeated process adopts the classical model, and the paper empirically uses a second-order process for a balance between simplicity and effectiveness.Source: §3.2 · pp. 3–4
  • The second-order pipeline replaces fixed downsampling with random area, bilinear, or bicubic resizing to retain a reasonable image-resolution range.Nearest-neighbor interpolation is excluded because it introduces misalignment.Source: §3.1 · p. 3, §3.2 · p. 4

Artifact modeling

  • Sinc filtering is used to synthesize ringing and overshoot artifacts that occur in real images and are not sufficiently represented by simpler degradation synthesis.The paper reports that different sinc cutoff frequencies produce filtered images visually similar to real ringing and overshoot examples.Source: Figure 2 · p. 4, §3.3 · p. 5

Networks and training

  • Real-ESRGAN retains the ESRGAN generator and uses pixel-unshuffle for ×2 and ×1 super-resolution to reduce spatial size before the main architecture.The reduced-resolution computation lowers GPU memory and computational-resource consumption.Source: §3.4 · p. 5
  • An U-Net discriminator with spectral normalization increases discriminative capability for local textures while stabilizing training over the larger degradation space.The training procedure first obtains a PSNR-oriented Real-ESRNet with L1 loss, then trains Real-ESRGAN with L1, perceptual, and GAN losses.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate Real-ESRGAN on diverse real-world images, ablate degradation and discriminator choices, and document both practical strengths and limitations.

Comparisons with Prior Works

  • Real-ESRGAN is compared with ESRGAN, DAN, CDC, RealSR, and BSRGAN across diverse real-world testing datasets.The evaluation includes RealSR, DRealSR, OST300, DPED, ADE20K validation, and Internet images.Source: §4.2 · p. 6
  • Real-ESRGAN outperforms previous approaches in removing artifacts and restoring texture details on representative real-world samples.Real-ESRGAN+ trained with sharpened ground-truths further boosts visual sharpness.Source: §4.2 · p. 6, Figure 7 · p. 7

Ablation Studies

  • Second-order degradation improves removal of wall noise and wheat-field blur compared with the classical first-order degradation model.The ablation uses Real-ESRNet to make degradation effects more controllable.Source: §4.3 · p. 8
  • Sinc filters prevent restored results from amplifying ringing and overshoot artifacts, especially around text and lines.Without sinc filters, these artifacts are amplified; with them, the artifacts can be removed.Source: §4.3 · p. 8
  • A U-Net discriminator improves local texture details, while its use can also introduce unnatural textures and increase training instability.The broader discriminator ablation examines the trade-off between detail restoration and artifact or stability concerns.Source: §4.3 · p. 8, Figure 9 · p. 8

Limitations

  • Limitations include twisted lines from aliasing, unpleasant GAN artifacts, and inability to remove out-of-distribution complicated degradations.The paper states that these drawbacks substantially affect practical application and require future work.Source: §4.4 · p. 8, Figure 11 · p. 8

5. Conclusion

Real-ESRGAN is trained with pure synthetic pairs for practical real-world blind super-resolution. It combines high-order degradation modeling, sinc filters, and a spectrally normalized U-Net discriminator.

  • Real-ESRGAN is trained with pure synthetic training pairs for real-world blind super-resolution.Source: §5 · p. 8
  • High-order degradation modeling and sinc filters synthesize complex degradations, including ringing and overshoot artifacts.Source: §5 · p. 8
  • A U-Net discriminator with spectral normalization increases discriminator capability and stabilizes training dynamics.Source: §5 · p. 8

A.1. Blur

The blur model uses Gaussian kernels alongside generalized Gaussian and plateau-shaped distributions to represent diverse blur shapes. Kernel shape parameters alter the resulting blur patterns.

A.1. Blur

  • Isotropic and anisotropic Gaussian filters are used as common choices for blur kernels.Figure 12 shows Gaussian kernels and their corresponding blurry images.Source: §A.1 · p. 11, Figure 12 · p. 11
  • Generalized Gaussian and plateau-shaped distributions broaden the diversity of blur kernel shapes.Figure 13 illustrates shape changes associated with the parameter β.Source: §A.1 · p. 11, Figure 13 · p. 11
  • Including diverse blur kernels produced sharper outputs for several real samples.Source: §A.1 · p. 11

A.2. Noise

The noise model considers additive Gaussian and Poisson noise. Poisson noise depends on image intensity, producing lower noise intensity in dark areas.

A.2. Noise

A.3. Resize

The resize analysis compares nearest-neighbor, area, bilinear, and bicubic operations across downsampling and upsampling combinations. Images are downsampled by four before being restored to their original size.

A.3. Resize

  • The compared resize operations are nearest-neighbor, area, bilinear, and bicubic interpolation.Source: §A.3 · p. 11
  • Each experiment downsamples an image by a scale factor of four and then upsamples it to its original size.Source: §A.3 · p. 11, Figure 15 · p. 11
  • Different downsampling and upsampling algorithm combinations produce different visual effects.Figure 15 presents the resulting combinations for comparison.Source: §A.3 · p. 11, Figure 15 · p. 11
  • Table 1 reports NIQE scores on diverse real-world image datasets, with lower scores indicating better results.Source: Table 1 · p. 12

A.4. JPEG compression

The section highlights JPEG compression as a source of artifacts and a remaining gap between synthetic and real image degradation. It also notes that different JPEG implementations can produce different compressed images.

  • JPEG compression introduces unpleasing block artifacts into digital images.Source: §A.4 · p. 12
  • Using different JPEG implementations may create an extra gap between synthetic and real samples.Source: §A.4 · p. 12
  • DiffJPEG and cv2 produce slightly different JPEG-compressed images at quality factor q = 50.Source: Figure 16 · p. 12, §A.4 · p. 12

B. Quantitative Comparisons

The quantitative comparison evaluates Real-ESRGAN against several state-of-the-art methods across diverse real-world datasets. Although Real-ESRGAN+ is not optimized for NIQE, it achieves lower NIQE scores on most testing datasets.

C. More Qualitative Comparisons

Qualitative comparisons show Real-ESRGAN outperforming previous approaches in artifact removal and texture restoration. Real-ESRGAN+ further increases visual sharpness when trained with sharpened ground truths.

Related papers

Loading 2107.10833v2…