Source-linked AI summary
PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration
Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, Chao Dong
TL;DR
GAN-based image restoration exposes a growing mismatch between quantitative IQA results and perceptual quality. The paper introduces PIPAL, a large-scale human-rated dataset and benchmark, and finds that existing IQA methods face challenges on GAN-based outputs while anti-aliasing pooling improves IQA networks on these distortions.
Problem
GAN-based restoration creates realistic-looking textures that challenge existing IQA methods and contribute to inconsistency between quantitative results and perceptual quality.
Method
The paper constructs PIPAL with GAN-based IR outputs, Elo-based subjective scores from more than 1.13 million judgments, and benchmarks for IQA and IR algorithms.
Results
Existing IQA methods face challenges evaluating perceptual IR, especially GAN-based algorithms, while anti-aliasing pooling improves IQA networks on GAN-based distortions.
Takeaways & Limitations
IQA methods need adjustment as new IR technologies develop, and PieAPP and LPIPS are more suitable than PSNR and PI for GAN-based IR evaluation.
Takeaways & Limitations
Most traditional and deep-learning-based IQA methods require well-aligned inputs, partially reducing their performance on GAN-based distortions.
Abstract
from arXiv · showhide
Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent IR methods based on Generative Adversarial Networks (GANs) have achieved significant improvement in visual performance, but also presented great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. Then we raise two questions: (1) Can existing IQA methods objectively evaluate recent IR algorithms? (2) When focus on beating current benchmarks, are we getting better IR algorithms? To answer these questions and promote the development of IQA methods, we contribute a large-scale IQA dataset, called Perceptual Image Processing Algorithms (PIPAL) dataset. Especially, this dataset includes the results of GAN-based methods, which are missing in previous datasets. We collect more than 1.13 million human judgments to assign subjective scores for PIPAL images using the more reliable "Elo system". Based on PIPAL, we present new benchmarks for both IQA and super-resolution methods. Our results indicate that existing IQA methods cannot fairly evaluate GAN-based IR algorithms. While using appropriate evaluation methods is important, IQA methods should also be updated along with the development of IR algorithms. At last, we improve the performance of IQA networks on GAN-based distortions by introducing anti-aliasing pooling. Experiments show the effectiveness of the proposed method.
3 SenseTime Research
The listed affiliation is SIAT at the Shenzhen Institute of Artificial Intelligence and Robotics for Society, and the paper concerns perceptual image restoration and image quality assessment.
- The authors are affiliated with SIAT, Shenzhen Institute of Artificial Intelligence and Robotics for Society.
- The paper addresses perceptual image restoration, image quality assessment, generative adversarial networks, and perceptual super-resolution.
1 Introduction
The introduction identifies a mismatch between quantitative IQA results and perceptual quality, especially for GAN-based restoration, and introduces PIPAL to address this evaluation challenge.
- 1 Introduction: Image restoration reconstructs high-quality images from distorted low-quality inputs, including super-resolution, denoising, and enhancement.
- 1 Introduction: Deep-learning IR algorithms produce less-distorted or perceptual-friendly images, but fairly measuring their visual quality remains challenging for algorithms.
- 1 Introduction: Existing IQA metrics can disagree with perceptual quality, as higher PSNR values do not always correspond to better visual quality.
- 1 Introduction: GAN-based IR methods fabricate realistic-looking details and textures that existing IQA methods may confuse with noise or real details.
- 1 Introduction: PIPAL is an IQA dataset containing distorted images with human-annotated visual quality levels for measuring agreement between IQA predictions and human judgments.
- 1 Introduction: PIPAL includes traditional, deep-learning-based, and GAN-based IR outputs, with GAN-based results appearing in an IQA dataset for the first time.
- 1 Introduction: PIPAL uses the Elo rating system and more than 1.13 million human judgments to assign subjective scores.
- 1 Introduction: PIPAL contains 29k images, including 250 high-quality references and 116 distortions per reference, making it the largest IQA dataset with complete subjective scoring.
2 Related Work
The related work reviews image restoration and IQA methods and datasets, emphasizing that PIPAL extends prior resources with GAN-based distortions.
- Image Restoration: Image restoration research progressed from hand-crafted features and image priors to CNN-based methods that dominate because of their performance and usability.
- Image Quality Assessment Datasets: PIPAL compares with previous datasets by including outputs of GAN-based algorithms as a novel distortion type.
- Image Restoration: Recent IR outputs differ from traditional IQA distortions, making evaluation increasingly challenging for low-resolution, noisy, and jointly degraded images.
- Image Quality Assessment: IQA methods measure perceptual image quality and include full-reference methods that compare information or perceptual-feature similarity between images.
- Image Quality Assessment Datasets: Existing IQA datasets provide distorted images with subjective scores and differ in reference-image collection, distortion coverage, and subjective-score collection strategy.
- Image Quality Assessment Datasets: PIPAL includes 19 different GAN-based algorithm distortions in addition to existing distortion types.
3 Perceptual Image Processing ALgorithms Dataset
PIPAL is a large IQA dataset designed to evaluate image restoration across traditional, deep-learning, and GAN-based distortions. It combines 29k distorted images with Elo-based human scoring to benchmark IQA and restoration methods.
- Reference images: PIPAL contains 250 texture-focused reference patches selected from DIV2K and Flickr2K.The patches emphasize areas that are relatively hard to restore, including high-frequency textures.
- Image distortions: GAN-based outputs resemble natural images while containing incorrect details, challenging metrics that distinguish textures from noise and real detail.This distortion differs from distortions that do not follow the natural image distribution.
- Image distortions: 40 distortion types and 116 distortion levels produce 29k distorted images for benchmarking IQA and image restoration.The dataset includes traditional distortions, image-restoration outputs, and GAN-based results.
- Elo Rating for Mean Opinion Score: PIPAL assigns subjective scores with an Elo system that combines pairwise preference probabilities with image ratings.The method assumes preferences follow a logistic distribution parameterized by Elo scores; M = 400 and K = 16 in the dataset.
- Elo Rating for Mean Opinion Score: After many human judgements, averaged late-stage Elo scores become each image’s MOS subjective score.The averaging operation reduces randomness in Elo updates, and the dataset contains 1.13 million human judgements.
- Elo Rating for Mean Opinion Score: Elo’s extensibility allows new distortion types to be added while automatically adjusting scores for existing distortions without re-rating them.This supports a dynamic dataset as new image-generation technologies emerge.
4 Results
PIPAL benchmarks IQA and image-restoration evaluation, revealing that existing metrics struggle particularly with GAN-based distortions. The experiments also examine metric behavior, optimization artifacts, and anti-aliasing as a possible improvement direction.
- IQA benchmark: PIPAL evaluates IQA methods across traditional, denoising, and super-resolution distortions, including outputs from traditional, deep-learning, PSNR-oriented, and GAN-based algorithms.The benchmark uses commonly used full-reference and no-reference IQA methods, evaluated mainly with SRCC and KRCC.
- IQA benchmark: About 0.71 SRCC on PIPAL for PieAPP, versus about 0.90 on TID2013, shows that PIPAL is challenging for existing IQA methods.High overall correlation also does not guarantee strong performance on every distortion subtype.
- IQA benchmark: Almost all IQA methods suffer a severe performance drop on GAN-based algorithms despite performing well on traditional and PSNR-oriented SR algorithms.Several recent deep-network-based IQA methods still outperform others on GAN-based algorithms.
- IQA benchmark: PSNR, SSIM, and IFC are anti-correlated with subjective scores for GAN-based SR, whereas LPIPS and PieAPP are the most correlated.NIQE and PI show moderate performance; IFC performs better on denoising, traditional SR, and PSNR-oriented SR than on GAN-based distortions.
- IR benchmark: A 2.6dB PSNR decrease relative to EDSR coincided with an MOS increase of about 50 for a post-2017 GAN-based SR result.ESRGAN had the highest subjective performance, while RankSRGAN achieved state-of-the-art PI and NIQE performance; improving PI did not continuously improve MOS after ESRGAN.
- IR benchmark: With PSNR fixed, gradient optimization produced metric-specific best cases whose visual effects differed, and even LPIPS and DISTS best cases contained serious artifacts.These experiments show that numerical optimization of individual IQA methods can produce visually undesirable outcomes.
- GAN-based distortion: FSIMC largely underestimates GAN-based distortion quality because GAN-based samples barely intersect with Gaussian-noise and Gaussian-blur samples in its scatter plots.For Gaussian noise and lossy compression, FSIMC produces samples clustered along the fitted curve, indicating adequate characterization for those distortions.
- GAN-based distortion: Most traditional and deep-learning-based IQA methods require well-aligned inputs, which partially accounts for their performance drop on GAN-based distortion.The paper explores anti-aliasing pooling to improve robustness to small spatial misalignment.
5 Conclusion
The paper constructs PIPAL and benchmarks IQA and image restoration methods, finding challenges in evaluating perceptual, especially GAN-based, restoration. It also reports that anti-aliasing pooling improves IQA networks.
- PIPAL provides a dataset and benchmarks for evaluating IQA methods and image restoration algorithms.
- Existing IQA methods face challenges when evaluating perceptual image restoration, especially GAN-based algorithms.
- Anti-aliasing pooling layers improve IQA networks for GAN-based distortions.