PaperlayerView exact revision on arXiv

Source-linked AI summary

RankSRGAN: Generative Adversarial Networks with Ranker for Image Super-Resolution

Wenlong Zhang, Yihao Liu, Chao Dong, Yu Qiao

arXiv:1908.06382v2cs.CV

TL;DR

Perceptual SR methods need objective guidance, but widely used perceptual metrics are nondifferentiable despite correlating with human ratings. RankSRGAN trains a Ranker to reproduce metric ranking behavior and adds a rank-content loss to GAN training; experiments report state-of-the-art perceptual-metric performance and visually pleasing results.

  • Problem

    Nondifferentiable perceptual metrics prevent existing perceptual SR methods from being stably optimized toward objective perceptual criteria.

  • Method

    RankSRGAN trains a Siamese-like Ranker on pairwise outputs from different SR algorithms and uses its learned ranking signal as a rank-content loss for GAN optimization.

  • Results

    RankSRGAN achieves superiority over state-of-the-art methods in perceptual metrics and recovers more realistic textures.

  • Takeaways & Limitations

    The framework can combine strengths from different SR methods and produce flexible results for different rank datasets, perceptual metrics, and loss combinations.

  • Takeaways & Limitations

    The method assumes that choosing a better algorithm than SRGAN to guide training can lead to better performance.

Abstract

from arXiv · show

Generative Adversarial Networks (GAN) have demonstrated the potential to recover realistic details for single image super-resolution (SISR). To further improve the visual quality of super-resolved results, PIRM2018-SR Challenge employed perceptual metrics to assess the perceptual quality, such as PI, NIQE, and Ma. However, existing methods cannot directly optimize these indifferentiable perceptual metrics, which are shown to be highly correlated with human ratings. To address the problem, we propose Super-Resolution Generative Adversarial Networks with Ranker (RankSRGAN) to optimize generator in the direction of perceptual metrics. Specifically, we first train a Ranker which can learn the behavior of perceptual metrics and then introduce a novel rank-content loss to optimize the perceptual quality. The most appealing part is that the proposed method can combine the strengths of different SR methods to generate better results. Extensive experiments show that RankSRGAN achieves visually pleasing results and reaches state-of-the-art performance in perceptual metrics. Project page: https://wenlongzhang0724.github.io/Projects/RankSRGAN

1. Introduction

Perceptual SR methods can produce realistic textures, but nondifferentiable perceptual metrics prevent stable metric-directed optimization. RankSRGAN addresses this with a differentiable Ranker and rank-content loss, while combining outputs from different SR methods.

  • 1. Introduction: CNN-based SR methods trade reconstruction accuracy against visual quality, with MSE-based models tending toward overly smoothed or sharpened images.Perceptual SR instead uses perceptual loss and adversarial learning to generate realistic textures and details.
  • 1. Introduction: NIQE and PI correlate with human ratings, but their nondifferentiable operations prevent direct use as loss functions.NIQE has correlation 0.76 and PI has correlation 0.83 with human ratings.
  • 1. Introduction: Existing perceptual SR methods therefore cannot show stable performance in the direction of objective perceptual criteria.The limitation follows from not incorporating NR-IQA metrics into optimization.
  • 1. Introduction: RankSRGAN trains a Siamese CNN Ranker to learn perceptual-metric ranking orders and uses its rank-content loss to guide GAN optimization.Ranking orders are learned instead of absolute metric values, accommodating different dynamic ranges across NR-IQA metrics.
  • 1. Introduction: The Ranker is trained from outputs of SRResNet, SRGAN, and ESRGAN, allowing the framework to use complementary strengths of different SR algorithms.SRGAN and ESRGAN can have mixed image-level orders under NIQE, so the Ranker favors different algorithms on different images.
  • 1. Introduction: Experiments report that RankSRGAN surpasses ESRGAN and achieves state-of-the-art performance across several perceptual metrics.The evaluated metrics include NIQE, Ma, and PI.

2. Related work

Related work spans CNN-based super-resolution, learning-based no-reference image-quality assessment, and learning-to-rank methods. RankSRGAN connects these areas by using a Ranker within a GAN-based SR framework.

  • 2. Related work: CNN-based super-resolution methods improved PSNR, while SRGAN introduced GAN-based generation of more realistic images.Subsequent work explored texture matching and related approaches.
  • 2. Related work: Learning-based NR-IQA methods estimate image quality using hand-crafted natural-scene features or CNNs that predict quality from image regions.Examples include NIQE, Ma, regression networks, and local-region quality prediction.
  • 2. Related work: Learning-to-rank has been applied to relative attributes and image generation, providing precedent for modeling pairwise quality relationships.RankCGAN is identified as a closely related use of ranking in image generation.
  • 2. Related work: RankSRGAN extends GAN-based SR by adding a Ranker alongside the generator and discriminator to constrain the generator in SR space.The framework also includes a fixed feature extractor and applies adversarial learning.

3. Method

RankSRGAN augments SRGAN with a differentiable Ranker that learns perceptual-metric ranking orders and constrains generator optimization. Its metric-ranking strategy uses outputs from different SR algorithms and can exceed their individual upper bound.

  • Rank dataset: The rank dataset pairs super-resolved images from different SR methods and labels each pair using a chosen perceptual metric.Images of the same content are ranked according to their metric-derived quality scores.
  • Ranker: The Siamese Ranker learns ranking orders rather than absolute perceptual-metric values, using margin-ranking loss to optimize pairwise scores.Its branches process paired inputs, and the margin controls the distance between their ranking scores.
  • RankSRGAN: RankSRGAN adds a well-trained Ranker to standard SRGAN, constraining the generator in super-resolution space while retaining adversarial learning.The generator and discriminator are jointly optimized, and the framework avoids complex GAN architectural changes to isolate the Ranker’s effect.
  • Training losses: The training objective combines perceptual, adversarial, and rank-content constraints, with the generator receiving discriminator-based adversarial supervision.The perceptual loss measures feature-space distance between HR and SR images using VGG19 features.
  • Rank-content loss: The rank-content loss feeds generated images into the Ranker and uses its score to guide perceptual-quality optimization.After sigmoid transformation, the loss ranges from 0 to 1; a lower Ranker score indicates better perceptual quality.
  • Analysis of Ranker: Metric rank labels images by perceptual scores, whereas model classification labels them by originating SR method; RankSRGAN-MR outperforms RankSRGAN-MC and ESRGAN in NIQE.Using SRGAN and ESRGAN outputs, metric rank can combine their stronger image-specific results and exceed the upper bound of the chosen algorithms.

4. Experiments

The experiments train Ranker on pairwise outputs from multiple SR algorithms and evaluate RankSRGAN across perceptual metrics, distortion, convergence, and ablations. Results show improved perceptual quality, stable training, and trade-offs controlled through rank-dataset and loss choices.

  • Training details of Ranker: Ranker training uses DIV2K/Flickr2K-derived image pairs, with SRResNet, SRGAN, and ESRGAN providing three perceptual levels labeled by NIQE order.Each perceptual level produces patches for training and validation; the best NIQE value receives label 1.
  • Training details of Ranker: A validation SROCC of 0.88 indicates that Ranker accurately reproduces the ordering of perceptual-quality labels.SROCC measures the monotonic relationship between labels and Ranker output scores.
  • Comparison with the-state-of-the-arts: RankSRGAN achieves the best NIQE and PI on Set14, BSD100, and PIRM-Test against SRGAN and ESRGAN, while improving perceptual scores without reducing PSNR.On PIRM-Test, it also obtains the highest PSNR among perceptual SR methods and generates more realistic textures without additional artifacts.
  • Comparison with the-state-of-the-arts: RankSRGAN remains relatively stable during training, with comparable PSNR and consistently better NIQE than SRGAN by a large margin.The convergence curves assess variation across training iterations.
  • Ablation study: Using SRResNet, SRGAN, and ESRGAN to construct the rank dataset motivates the main configuration, while alternative algorithm choices change performance.The selected algorithms are intended to provide ranking relationships around the SRGAN baseline and stronger guidance from ESRGAN.
  • Ablation study: RankSRGAN-HR improves PSNR by almost 0.4 dB while balancing perceptual quality and distortion when HR images replace ESRGAN in the rank dataset.The result suggests additional guidance from stronger algorithms can provide further improvement space.
  • Ablation study: Rank-based training separates SRGAN and ESRGAN outputs more than regression and achieves better performance on the selected perceptual metric.The ranking strategy produces larger output distances and improves the corresponding RankSRGAN results.
  • Ablation study: Increasing the MSE-loss weight raises PSNR but also increases NIQE, exposing a trade-off that loss weighting can adjust.The method is evaluated with different perceptual metrics, including NIQE, Ma, and PI, and with ranking versus regression strategies.

5. Conclusion

RankSRGAN optimizes super-resolution models toward perceptual metrics through a Ranker that learns their behavior. The framework combines strengths of different SR methods and produces flexible, perceptually superior results.

  • RankSRGAN optimizes an SR model in the orientation of perceptual metrics.
  • A Ranker learns perceptual-metric behavior through a learning-to-rank approach.
  • The framework combines strengths of different SR methods to generate better results.
  • Extensive experiments show superiority over state-of-the-art methods in perceptual metrics and recovery of more realistic textures.

Supplementary File

The supplementary file lists the paper’s authors, institutional affiliations, and contact email addresses.

  • The paper lists Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao as authors.
  • The authors are affiliated with the Shenzhen Key Lab of Computer Vision and Pattern Recognition and related Chinese Academy of Sciences institutions.
  • Yihao Liu is additionally affiliated with the University of Chinese Academy of Sciences.
  • The listed contact addresses use the siat.ac.cn domain.

Abstract

The supplementary file expands the Ranker experiments, reports ablation-study curves, and adds qualitative comparisons with state-of-the-art methods.

  • Additional Ranker details and experimental results are presented in the supplementary file.
  • Performance curves for different RankSRGAN models are included for the ablation study.
  • The file includes qualitative comparisons with state-of-the-art methods.

1. Details of Ranker

RankSRGAN’s Ranker is evaluated through convergence curves, dataset-size and architecture comparisons, ranking-score histograms, and SROCC-based accuracy analysis.

  • Convergence curves compare RankSRGAN using Ranker1 and Ranker2 across NIQE and PSNR.
  • Ranker1 uses rank dataset1 with 15 K image pairs, whereas Ranker2 uses rank dataset2 with 150 K image pairs.
  • Ranker architectures use VGG-inspired networks with depths VGG-8, VGG-12, and VGG-16; VGG-8 and VGG-12 are applied to RankSRGAN.
  • Figure 2 compares RankSRGAN with Ranker-VGG-8 and Ranker-VGG-12 using NIQE and PSNR convergence curves.
  • Ranker histograms compare NIQE and Ma labels with regression and ranking scores to assess separation of perceptual levels.

2. Details of RankSRGAN

The convergence results show that Ranker-guided RankSRGAN consistently improves perceptual metrics over SRGAN and generalizes across NIQE, Ma, and PI. Using HR in the ranking dataset further improves NIQE while also improving PSNR.

  • Convergence curves for RankSRGAN-(NIQE, Ma, and PI): Ranker-guided RankSRGAN achieves constant improvement over the baseline SRGAN during optimization.The convergence curves cover RankSRGAN-N, RankSRGAN-M, and RankSRGAN-PI, using NIQE, Ma, and PI rankers respectively.
  • Convergence curves for RankSRGAN-(NIQE, Ma, and PI): Ranker helps RankSRGAN reach state-of-the-art performance across different perceptual metrics and accelerates convergence for RankSRGAN-N versus regression-based Ranker.For RankSRGAN-M, Ranker reaches a value less than 1.40 in ESRGAN, whereas the regression Ranker cannot outperform ESRGAN at 1.40.
  • Convergence curves for RankSRGAN-HR: RankSRGAN-HR consistently improves NIQE over SRGAN while also continuously improving PSNR.RankSRGAN-HR trains Ranker with a ranking dataset generated from SRResNet, SRGAN, and ground-truth HR images, balancing perceptual quality and distortion.

3. More Qualitative Results

The qualitative results compare RankSRGAN with perceptual and PSNR-oriented super-resolution methods using enlarged images and both perceptual and distortion metrics.

  • Qualitative comparison: The qualitative evaluation presents ×4 enlarged results for RankSRGAN alongside SRGAN, ESRGAN, and SRResNet.SRGAN and ESRGAN are perceptual SR methods, while SRResNet is PSNR-oriented.
  • Evaluation metrics: NIQE and PSNR are used together to evaluate the compared super-resolution methods.The evaluation combines a perceptual metric with a distortion-oriented metric.
  • Evaluation metrics: Lower NIQE indicates better perceptual quality, whereas higher PSNR indicates less distortion from the ground-truth image.
Loading 1908.06382v2…