Source-linked AI summary
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, Xiaoou Tang
TL;DR
SRGAN can generate realistic textures for single-image super-resolution, but its results can still contain artifacts and differ from ground-truth images. ESRGAN revises its architecture and adversarial training, achieving better perceptual quality and first place in the PIRM-SR Challenge.
Problem
SRGAN results retain a gap from ground-truth images and may contain unstable or unpleasant artifacts, limiting perceptual quality.
Method
ESRGAN combines BN-free Residual-in-Residual Dense Blocks with a relativistic discriminator and revised perceptual supervision to improve super-resolution quality.
Results
ESRGAN won first place in the PIRM-SR Challenge region 3 with the best perceptual index and consistently outperformed prior methods in sharpness and details.
Takeaways & Limitations
ESRGAN produces sharper, more natural textures and avoids artifacts observed in previous GAN-based super-resolution methods.
Takeaways & Limitations
The reasons behind batch-normalization artifacts and potential solutions remain to be further studied.
Abstract
from arXiv · showhide
The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge. The code is available at https://github.com/xinntao/ESRGAN .
1 Introduction
ESRGAN revisits SRGAN’s architecture, adversarial loss, and perceptual loss to improve super-resolution visual quality, producing sharper and more detailed textures. Its enhanced model consistently outperforms SRGAN and state-of-the-art methods, while a variant wins first place in the PIRM-SR Challenge’s region 3.
- Introduction: ESRGAN improves SRGAN in three aspects: network structure, adversarial loss, and perceptual loss.The paper introduces an RRDB-based architecture without batch normalization, relative-realness discrimination, and perceptual features before activation.
- Introduction: The proposed ESRGAN consistently outperforms state-of-the-art methods in sharpness and details.The introduction reports these gains in extensive experiments and illustrates them against SRGAN and ground-truth images.
- Introduction: ESRGAN won first place in region 3 of the PIRM-SR Challenge and achieved the best perceptual index.The challenge evaluates perceptual quality using non-reference measures, and lower perceptual index indicates better perceptual quality.
- Introduction: Network interpolation continuously adjusts reconstruction style and smoothness to balance visual quality with RMSE/PSNR.Image interpolation is used as an alternative strategy for participation in regions 1 and 2.
2 Related Work
Deep neural super-resolution evolved from SRCNN toward increasingly diverse architectures and stabilization techniques, while perceptual quality was pursued through feature-based losses and adversarial training. Evaluation commonly used distortion and non-reference perceptual measures, reflecting a tension between distortion and perceptual quality.
- SRCNN pioneered end-to-end learning of the mapping from low-resolution to high-resolution images and achieved superior performance over previous methods.
- Subsequent super-resolution research explored deeper residual, Laplacian pyramid, recursive, densely connected, back-projection, and residual-dense architectures, including EDSR without unnecessary batch-normalization layers.
- Residual paths, residual scaling, and robust initialization were proposed to stabilize training in very deep networks, motivating a compact residual-in-residual dense block.
- Perceptual loss improved visual quality by comparing feature-space errors, while contextual loss targeted natural image statistics through feature-distribution objectives.
- Adversarial training commonly targeted photo-realism, with later GAN work improving discriminator regularization and introducing relativistic discrimination.
- PSNR and SSIM are widely used distortion measures, whereas Ma’s score and NIQE assess perceptual quality; studies find distortion and perceptual quality are at odds.
3 Proposed Methods
ESRGAN improves SRGAN through changes to the generator architecture, discriminator, perceptual loss, and a network-interpolation strategy for balancing perceptual quality and fidelity.
- Generator architecture: The generator removes batch-normalization layers and replaces SRGAN’s residual block with a deeper Residual-in-Residual Dense Block combining multi-level residual learning and dense connections.Removing BN layers supports stable, consistent training while reducing computational complexity, memory usage, and improving generalization.
- Generator architecture: The RRDB-based deep network uses residual scaling and smaller initialization to facilitate stable training.Residual scaling multiplies residuals by a constant between 0 and 1 before adding them to the main path.
- Discriminator: The discriminator is replaced with a relativistic average discriminator that predicts whether a real image is more realistic than fake images on average.Its generator adversarial loss receives gradients from both generated and real data, unlike SRGAN’s generated-only contribution.
- Perceptual loss: The perceptual loss compares features before activation rather than activated features, providing denser information than the sparse post-activation representation.The total generator loss also includes an L1 content loss weighted with other loss terms by λ and η.
- Network interpolation: Network interpolation combines parameters from PSNR-oriented and GAN-based networks, allowing continuous balancing of perceptual quality and fidelity without retraining.For any feasible α, the interpolated model produces meaningful results without introducing artifacts.
4 Experiments
Experiments evaluate ESRGAN under ×4 super-resolution on diverse training and benchmark datasets, showing improved texture realism, sharper details, and fewer artifacts. Ablations identify the contributions of BN removal, pre-activation features, RaGAN, and deeper RRDB networks, while interpolation and challenge results demonstrate controllable fidelity–perceptual tradeoffs and strong perceptual performance.
- Experimental setup: Experiments use ×4 upscaling, with LR images generated by MATLAB bicubic downsampling from HR images.Training uses mini-batches of 16 and 128 × 128 cropped HR patches.
- Datasets and evaluation: Training combines DIV2K, Flickr2K, and OST images, while evaluation uses Set5, Set14, BSD100, Urban100, and the PIRM self-validation dataset.Models are trained in RGB with random horizontal flips and 90 degree rotations.
- Qualitative comparison: ESRGAN produces sharper, more natural textures and more detailed structures than PSNR-oriented and previous GAN-based methods, which yield blur, unnatural textures, noise, or artifacts.Examples include baboon whiskers, grass, building structures, and reduced facial artifacts.
- Ablation study: Removing BN, using features before activation, applying RaGAN, and adopting deeper RRDB models progressively improve sharpness, texture richness, structure recovery, and artifact reduction.The deeper RRDB model is reported to train easily and reduce unpleasing noise compared with SRGAN’s deeper-model difficulty.
- Interpolation and challenge: Network interpolation reduces artifacts while maintaining textures and provides smooth control between perceptual quality and fidelity, whereas image interpolation removes artifacts less effectively.In PIRM-SR Challenge region 3, ESRGAN achieves the best perceptual index and wins first place.
5 Conclusion
ESRGAN achieves consistently better perceptual quality than previous super-resolution methods and won first place in the PIRM-SR Challenge by perceptual index. Its approach combines RDDB blocks without batch normalization with training techniques and relativistic GAN components.
- Conclusion: ESRGAN achieves consistently better perceptual quality than previous super-resolution methods and won first place in the PIRM-SR Challenge by perceptual index.The conclusion identifies perceptual index as the challenge criterion.
- Conclusion: The model uses a novel architecture containing several RDDB blocks without batch-normalization layers.The conclusion names RDDB blocks without BN as the architectural foundation.
- Conclusion: Residual scaling and smaller initialization facilitate training of the proposed deep model.These techniques are presented as useful additions for training stability.
- Conclusion: The method also introduces relativistic GAN as part of its improvements.
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks Supplementary File
The supplementary file examines BN-related artifacts, techniques for training very deep models, dataset and patch-size effects, and additional qualitative comparisons.
- Supplementary file overview: The supplementary file covers BN-related artifacts, methods for training very deep models, dataset and training patch-size effects, and further qualitative visual comparisons.These topics are organized into Sections 1–5.
1 BN artifacts
Batch-normalization layers can introduce unstable visual artifacts, especially under deeper architectures, high-resolution placement, mismatched train-test statistics, and GAN training. Because these artifacts can appear, disappear, or change across iterations, ESRGAN removes batch normalization for stable training and consistent performance.
- BN artifacts: BN artifacts occur intermittently across iterations and settings, undermining stable training performance.The artifacts can appear, disappear, and change during training.
- BN artifacts: Deeper networks and BN layers in HR space make BN artifacts more likely than the 16-RB baseline with BN confined to LR space.The baseline is similar to SRResNet and is unlikely to introduce artifacts in these experiments.
- BN artifacts: Mismatched training and testing dataset statistics can cause BN artifacts and limit generalization ability.BN uses batch statistics during training but estimated whole-training-dataset statistics during testing.
- BN artifacts: Replacing L1 loss with VGG + GAN + L1 loss increases the occurrence probability of BN artifacts under the GAN framework.This comparison uses the same network structure as the baseline.
- BN artifacts: The authors therefore remove BN layers to achieve stable training and consistent performance.The reasons behind the artifacts and potential solutions remain open for further study.
2 Useful techniques to train a very deep network
Without batch normalization, training very deep networks becomes difficult despite RRDBs, so ESRGAN uses smaller initialization and residual scaling to improve stability. In experiments with 64 residual blocks, smaller initialization helped escape a poor local minimum, while combining both techniques provided a further slight improvement.
- Training challenges and techniques: Removing batch-normalization layers makes stable training and consistent performance difficult even with Residual-in-Residual Dense Blocks.The authors identify smaller initialization and residual scaling as techniques that ease training very deep networks.
- Residual scaling: Residual scaling multiplies residual features by 0.2 before adding them to the main path to prevent instability.This approach scales residuals by a constant between 0 and 1 and can correct improper initialization.
- Experimental comparison: In a 64-RB network, MSRA initialization falls into an extremely bad local minimum, whereas smaller initialization (×0.1) achieves good performance.The comparison is shown in Fig. 4a, where smaller initialization helps the network escape the poor local minimum.
- Experimental comparison: Smaller initialization achieves higher PSNR than residual scaling, and applying both techniques yields a further slight improvement.These results are reported for the very deep network used in the experiments.
3 The influence of different datasets
Larger and more diverse training datasets improve performance for PSNR-oriented super-resolution methods. For perceptual-driven methods, OST supports grass restoration, whereas ImageNet introduces new artifact types.
- PSNR-oriented methods: Larger training datasets lead to better performance for PSNR-oriented methods using a large 23-RRDB model.The model places 23 Residual-in-Residual Blocks before upsampling, followed by two convolution layers for reconstruction.
- PSNR-oriented methods: Combining DIV2K’s 800 images with Flickr2K’s 2650 diverse 2K images into DF2K increases PSNR performance.DIV2K is widely used, while Flickr2K contains high-resolution images collected from Flickr.
- Perceptual-driven methods: For perceptual-driven methods, OST’s diverse natural textures benefit grass restoration, while ImageNet introduces new artifact types.The comparison uses a large model with 23 RRDB blocks and an ImageNet subset of about 450k images.
4 The influence of training patch size
Larger training patches benefit both shallow and deep models, with greater gains for the deeper network. Because larger patches require more time and computing resources, the authors use different patch sizes for PSNR-oriented and perceptual-driven methods.
- Experimental setup: Training uses 96 × 96, 128 × 128, and 192 × 192 patches for models with 16 RBs and 23 RRDBs, evaluated on Set5 with RGB channels.The deeper model has larger capacity, and the training curves are reported for both architectures.
- Observed influence: Both models benefit from larger training patches, while the deeper model gains ∼0.12dB compared with ∼0.04dB for the shallower model.The larger-capacity network better exploits the enlarged training patch size.
- Trade-off: Larger training patches increase training time and computing-resource consumption.The patch-size choice therefore involves a trade-off between performance benefits and computational cost.
- Trade-off: The authors use 192 × 192 patches for PSNR-oriented methods and 128 × 128 patches for perceptual-driven methods.These settings are selected as a practical compromise between patch-size benefits and resource demands.
5 More qualitative comparison
Figure 7 presents additional qualitative results, with PSNR on the Y channel and perceptual index reported for reference. Four PSNR/perceptual-index pairs are provided alongside the comparisons.
- More qualitative comparison: The reported PSNR/perceptual-index pairs are (26.27 / 5.37), (23.40 / 2.75), (24.03 / 1.70), and (23.81 / 1.78).The passage labels these values as PSNR / Perceptual Index.
- More qualitative comparison: Figure 7 shows more qualitative results and reports PSNR on the Y channel together with the perceptual index for reference.The figure caption specifies that both metrics accompany the qualitative comparisons.