Source-linked AI summary
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
Xintao Wang, Liangbin Xie, Chao Dong, Ying Shan
TL;DR
Real-world blind super-resolution must handle degradations more complex than idealized synthetic assumptions. Real-ESRGAN addresses this with pure-synthetic training pairs built from high-order degradation modeling, artifact-aware filtering, and a U-Net discriminator with spectral normalization, achieving stronger visual restoration across real-world images. Its scope remains limited by artifacts and out-of-distribution degradations that it cannot reliably remove.
Problem
Conventional and blind SR methods remain limited by the mismatch between simple synthetic degradations and unknown, complex real-world degradations.
Method
Real-ESRGAN trains ESRGAN with pure synthetic pairs using high-order degradation modeling, sinc filters for ringing and overshoot artifacts, and a U-Net discriminator with spectral normalization.
Results
Real-ESRGAN trained with synthetic data restores most real-world images, enhancing details while removing artifacts and achieving better visual performance than previous works.
Takeaways & Limitations
Pure synthetic training can support practical real-world blind SR when degradation synthesis and adversarial training are adapted to complex artifacts and degradation spaces.
Takeaways & Limitations
The method cannot remove out-of-distribution complicated real-world degradations and may amplify their artifacts.
Abstract
from arXiv · showhide
Though many attempts have been made in blind super-resolution to restore low-resolution images with unknown and complex degradations, they are still far from addressing general real-world degraded images. In this work, we extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data. Specifically, a high-order degradation modeling process is introduced to better simulate complex real-world degradations. We also consider the common ringing and overshoot artifacts in the synthesis process. In addition, we employ a U-Net discriminator with spectral normalization to increase discriminator capability and stabilize the training dynamics. Extensive comparisons have shown its superior visual performance than prior works on various real datasets. We also provide efficient implementations to synthesize training pairs on the fly.
1. Introduction
Real-ESRGAN addresses the mismatch between idealized SR degradations and complex real-world images by combining high-order synthetic degradation modeling with architectural changes for stable, detailed restoration.
- Most SR methods assume bicubic downsampling, creating a degradation mismatch that limits their practicality on real-world images.
- Blind SR methods model unknown degradations explicitly or implicitly, but simple degradation combinations fail on complex samples and learned models generalize poorly outside their training distributions.
- Real-ESRGAN extends ESRGAN by synthesizing training pairs with a high-order degradation process that repeats classical degradation operations using different hyper-parameters.
- Sinc filters are incorporated to simulate common ringing and overshoot artifacts during degradation synthesis.
- A U-Net discriminator with spectral normalization increases discriminator capability and stabilizes training, supporting local detail enhancement and artifact suppression.
2. Related Work
Related blind SR work uses explicit degradation representations, real or unpaired data, and synthesized pairs, but these approaches remain constrained by degradation assumptions or data coverage.
- Blind SR methods using explicit degradation representations commonly separate or jointly perform degradation prediction and conditional restoration.
- Explicit approaches rely on predefined degradation types and levels and usually model simple synthetic degradations.
- Other methods construct training pairs from camera captures, unpaired data, or estimated blur kernels and extracted noise patches.
3. Methodology
Real-ESRGAN synthesizes training pairs with repeated classical degradation processes to better represent complex real-world degradations, while adding artifact modeling and architectural changes for practical restoration.
- Classical degradation model: The classical degradation model applies blur, resizing, noise, and JPEG compression to synthesize low-resolution inputs.Blur is modeled with convolutional kernels; resizing includes downsampling and upsampling, while Gaussian and Poisson noise are considered.
- High-order degradation model: Because first-order synthesis leaves a gap from realistic degraded images, Real-ESRGAN uses repeated degradation processes with different hyper-parameters.Each repeated process adopts the classical model, and the paper empirically uses a second-order process for a balance between simplicity and effectiveness.
- High-order degradation model: The second-order pipeline replaces fixed downsampling with random area, bilinear, or bicubic resizing to retain a reasonable image-resolution range.Nearest-neighbor interpolation is excluded because it introduces misalignment.
- Artifact modeling: Sinc filtering is used to synthesize ringing and overshoot artifacts that occur in real images and are not sufficiently represented by simpler degradation synthesis.The paper reports that different sinc cutoff frequencies produce filtered images visually similar to real ringing and overshoot examples.
- Networks and training: Real-ESRGAN retains the ESRGAN generator and uses pixel-unshuffle for ×2 and ×1 super-resolution to reduce spatial size before the main architecture.The reduced-resolution computation lowers GPU memory and computational-resource consumption.
- Networks and training: An U-Net discriminator with spectral normalization increases discriminative capability for local textures while stabilizing training over the larger degradation space.The training procedure first obtains a PSNR-oriented Real-ESRNet with L1 loss, then trains Real-ESRGAN with L1, perceptual, and GAN losses.
4. Experiments
Experiments evaluate Real-ESRGAN on diverse real-world images, ablate degradation and discriminator choices, and document both practical strengths and limitations.
- Comparisons with Prior Works: Real-ESRGAN is compared with ESRGAN, DAN, CDC, RealSR, and BSRGAN across diverse real-world testing datasets.The evaluation includes RealSR, DRealSR, OST300, DPED, ADE20K validation, and Internet images.
- Comparisons with Prior Works: Real-ESRGAN outperforms previous approaches in removing artifacts and restoring texture details on representative real-world samples.Real-ESRGAN+ trained with sharpened ground-truths further boosts visual sharpness.
- Ablation Studies: Second-order degradation improves removal of wall noise and wheat-field blur compared with the classical first-order degradation model.The ablation uses Real-ESRNet to make degradation effects more controllable.
- Ablation Studies: Sinc filters prevent restored results from amplifying ringing and overshoot artifacts, especially around text and lines.Without sinc filters, these artifacts are amplified; with them, the artifacts can be removed.
- Ablation Studies: A U-Net discriminator improves local texture details, while its use can also introduce unnatural textures and increase training instability.The broader discriminator ablation examines the trade-off between detail restoration and artifact or stability concerns.
- Limitations: Limitations include twisted lines from aliasing, unpleasant GAN artifacts, and inability to remove out-of-distribution complicated degradations.The paper states that these drawbacks substantially affect practical application and require future work.
5. Conclusion
Real-ESRGAN is trained with pure synthetic pairs for practical real-world blind super-resolution. It combines high-order degradation modeling, sinc filters, and a spectrally normalized U-Net discriminator.
- Real-ESRGAN is trained with pure synthetic training pairs for real-world blind super-resolution.
- High-order degradation modeling and sinc filters synthesize complex degradations, including ringing and overshoot artifacts.
- A U-Net discriminator with spectral normalization increases discriminator capability and stabilizes training dynamics.
A.1. Blur
The blur model uses Gaussian kernels alongside generalized Gaussian and plateau-shaped distributions to represent diverse blur shapes. Kernel shape parameters alter the resulting blur patterns.
- A.1. Blur: Isotropic and anisotropic Gaussian filters are used as common choices for blur kernels.Figure 12 shows Gaussian kernels and their corresponding blurry images.
- A.1. Blur: Generalized Gaussian and plateau-shaped distributions broaden the diversity of blur kernel shapes.Figure 13 illustrates shape changes associated with the parameter β.
- A.1. Blur: Including diverse blur kernels produced sharper outputs for several real samples.
A.2. Noise
The noise model considers additive Gaussian and Poisson noise. Poisson noise depends on image intensity, producing lower noise intensity in dark areas.
- A.2. Noise: The degradation examples include additive Gaussian noise and Poisson noise.Figure 14 compares their visual appearances.
- A.2. Noise: Poisson noise has intensity proportional to image intensity and independent noise across pixels.
- A.2. Noise: Poisson noise has low noise intensity in dark areas.
A.3. Resize
The resize analysis compares nearest-neighbor, area, bilinear, and bicubic operations across downsampling and upsampling combinations. Images are downsampled by four before being restored to their original size.
- A.3. Resize: The compared resize operations are nearest-neighbor, area, bilinear, and bicubic interpolation.
- A.3. Resize: Each experiment downsamples an image by a scale factor of four and then upsamples it to its original size.
- A.3. Resize: Different downsampling and upsampling algorithm combinations produce different visual effects.Figure 15 presents the resulting combinations for comparison.
- A.3. Resize: Table 1 reports NIQE scores on diverse real-world image datasets, with lower scores indicating better results.
A.4. JPEG compression
The section highlights JPEG compression as a source of artifacts and a remaining gap between synthetic and real image degradation. It also notes that different JPEG implementations can produce different compressed images.
- JPEG compression introduces unpleasing block artifacts into digital images.
- Using different JPEG implementations may create an extra gap between synthetic and real samples.
- DiffJPEG and cv2 produce slightly different JPEG-compressed images at quality factor q = 50.
B. Quantitative Comparisons
The quantitative comparison evaluates Real-ESRGAN against several state-of-the-art methods across diverse real-world datasets. Although Real-ESRGAN+ is not optimized for NIQE, it achieves lower NIQE scores on most testing datasets.
- NIQE is provided for reference because perceptual-quality metrics may not reflect fine-grained human perceptual preferences.
- Real-ESRGAN is compared with ESRGAN, DAN, CDC, RealSR, and BSRGAN on six diverse real-world image datasets.
- Real-ESRGAN+ produces lower NIQE scores on most testing datasets despite not optimizing for NIQE.
C. More Qualitative Comparisons
Qualitative comparisons show Real-ESRGAN outperforming previous approaches in artifact removal and texture restoration. Real-ESRGAN+ further increases visual sharpness when trained with sharpened ground truths.
- Real-ESRGAN outperforms previous approaches in removing artifacts and restoring texture details.
- Real-ESRGAN+ can further boost visual sharpness when trained with sharpened ground truths.
- Other methods typically fail on complicated artifacts, overshoot artifacts, or realistic natural textures across representative real-world samples.