Source-linked AI summary
The 2018 PIRM Challenge on Perceptual Image Super-resolution
Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, Lihi Zelnik-Manor
TL;DR
Perceptual SR needs evaluation that reflects both fidelity to the ground truth and natural-looking output, because these objectives can conflict. The paper reports the PIRM challenge, which jointly evaluates RMSE distortion and no-reference perceptual quality, and analyzes how metrics relate to human judgments. The challenge advanced perceptual SR beyond prior methods, while revealing both a strong low-distortion tradeoff and limitations in current perceptual quality assessment.
Problem
Reconstruction accuracy and perceptual quality can disagree, while human-opinion studies are difficult to reproduce and standard metrics inadequately quantify perceptual quality.
Method
The PIRM challenge ranks perceptual SR methods on a plane combining full-reference RMSE distortion with a no-reference perceptual index, alongside human-opinion and metric-correlation studies.
Results
The challenge advanced perceptual SR beyond existing baselines, and its selected perceptual index was highly correlated with human ratings.
Takeaways & Limitations
Perceptual SR methods can improve quality within distortion constraints, but common image quality measures are generally inadequate for quantifying perceptual quality.
Takeaways & Limitations
At 4x SR, current outputs generally remain unrealistic to human observers, and perceptual-quality methods have yet to appear for blind real-world SR.
Abstract
from arXiv · showhide
This paper reports on the 2018 PIRM challenge on perceptual super-resolution (SR), held in conjunction with the Perceptual Image Restoration and Manipulation (PIRM) workshop at ECCV 2018. In contrast to previous SR challenges, our evaluation methodology jointly quantifies accuracy and perceptual quality, therefore enabling perceptual-driven methods to compete alongside algorithms that target PSNR maximization. Twenty-one participating teams introduced algorithms which well-improved upon the existing state-of-the-art methods in perceptual SR, as confirmed by a human opinion study. We also analyze popular image quality measures and draw conclusions regarding which of them correlates best with human opinion scores. We conclude with an analysis of the current trends in perceptual SR, as reflected from the leading submissions.
1 Introduction
Single-image SR has advanced along two partly conflicting goals: reconstruction accuracy and perceptual quality. The PIRM challenge introduced a benchmark that evaluates both dimensions jointly, allowing perceptual-driven methods to compete with PSNR-oriented methods.
- Motivation: Reconstruction accuracy and perceptual quality increasingly disagree: low-distortion models often look unpleasing, while perceptually strong models score poorly on PSNR and SSIM.The paper describes this as a fundamental tradeoff between low distortion and closeness to natural-image statistics.
- Motivation: Perceptual-quality reports are difficult to reproduce because human studies expose each user to only a small subset of methods and images.Existing challenges therefore largely emphasize PSNR and SSIM rather than perceptual quality.
- Challenge design: PIRM evaluates SR on a two-dimensional plane using full-reference RMSE distortion and a perceptual index combining no-reference quality measures.The scheme measures visual naturalness independently of similarity to a ground-truth image.
- Outcomes: Participants introduced methods that improved perceptual SR beyond baselines, while the selected perceptual index was highly correlated with human ratings.The submissions varied losses, architectures, GAN variants, and training schemes.
- Outcomes: At low distortion, participants improved perceptual quality over EDSR with only a small RMSE increase; at high perceptual quality, they improved both quality and distortion over E-Net.These results indicate a severe tradeoff in the low-distortion regime and substantial room relative to the perception-distortion bound in the high-quality regime.
2 Perceptual Super Resolution
Perceptual SR emerged as a response to increasingly blurry or unnatural outputs from PSNR/SSIM optimization. Its methods use objectives such as perceptual, adversarial, and texture-matching losses, but generally reduce standard distortion scores.
- Origins: Convolutional SR methods initially optimized ℓ1/ℓ2 losses for reconstruction accuracy, but rising PSNR/SSIM did not consistently improve perceptual quality.The resulting outputs could become increasingly blurry and unnatural.
- Origins: Perceptual SR shifted optimization from PSNR maximization toward perceptual quality maximization.The paper uses “perceptual SR” for this newer line of work.
- Key objectives: Johnson et al. introduced a deep-feature ℓ2 perceptual loss, and Ledig et al. combined perceptual and adversarial losses for SR.Later work added texture-matching loss to further develop this direction.
- Tradeoff: Perceptual SR methods produce superior perceptual quality but substantially lower PSNR and SSIM, preventing straightforward participation in standard distortion-based benchmarks.The paper argues that these metrics do not faithfully quantify perceptual quality.
3 The PIRM Challenge on Perceptual SR
The PIRM challenge compared 4× single-image super-resolution methods using a perception-distortion evaluation scheme rather than distortion measures alone. Algorithms were ranked within three RMSE-defined regions by mean perceptual quality, with lower perceptual index indicating better quality.
- Task: The challenge task was 4× super-resolution of single images down-sampled with a bicubic kernel.
- Datasets: Validation and testing used separate sets of 100 diverse images, with high-resolution ground truths withheld from participants during the challenge.
- Evaluation: The evaluation used the perception-distortion plane, combining RMSE distortion with a perceptual index based on no-reference image-quality measures.
- Evaluation: Three regions were defined by RMSE ≤11.5, 12.5, and 16, and each region was optimized for the best mean perceptual quality.
- Evaluation: Lower perceptual index values indicated better perceptual quality.
- Evaluation: RMSE was computed as the square root of the aggregate mean squared error across all pixels and images, rather than as the mean of per-image RMSE values.
4 Challenge Results
The challenge submissions advanced perceptual SR beyond prior baselines, while exposing a perception–distortion trade-off and image-dependent method strengths. Human ratings confirmed improved perceptual quality, but no method consistently appeared fully real.
- Challenge results: Twenty-one teams submitted methods that exceeded previous SR methods across all regions, advancing the state of the art in perceptual SR.Region 3 submissions surpassed EnhanceNet and CX, while several methods improved both perceptual quality and reconstruction accuracy.
- Challenge results: 45% better PI than EDSR came with only a 7% RMSE increase, equivalent to 0.77 gray-levels per pixel, in Region 1.This result illustrates substantial perceptual improvement at low distortion with a small accuracy sacrifice.
- Challenge results: Beyond some point in Region 3, increasing RMSE yielded only slight perceptual gains, allowing similar perceptual quality at considerably lower RMSE.The perception–distortion trade-off was stronger in the low-distortion regime than in the high-perceptual-quality regime.
- Human opinion study: 35 raters evaluated 12 algorithms on a 1–4 realism scale, and judged challenge submissions significantly better than state-of-the-art baselines.Region 3 and Region 2 methods were notably better than EnhanceNet, while Region 1 methods were visually better than EDSR with a slight RMSE increase.
- Human opinion study: Human ratings showed that attainable perceptual quality increased with RMSE, progressing from Region 1 to Region 3.The study assessed perceptual quality only because raters did not see the original ground-truth images.
- Image variability: All methods received relatively few “definitely real” votes, and image content affected which methods looked best.Structure-dominated images favored accuracy-targeting methods, whereas texture-rich images favored Region 3 methods; all methods still produced unnatural features at 4× magnification.
5 Analyzing Quality Measures
The paper compares image-quality measures against human opinion for perceptual SR. PI and NIQE performed best overall and in the high-quality regime, whereas several conventional or alternative measures failed to rank perceptual quality reliably.
- Overall correlations: RMSE, SSIM, and IFC were anti-correlated with perceptual quality, making them inappropriate for evaluating perceptual SR.These measures are widely used for reconstruction quality but do not track human-opinion scores in this setting.
- Overall correlations: PI was the most correlated measure with human-opinion scores, while LPIPS and NIQE were also highly correlated.Ma et al. and BRISQUE showed moderate correlation.
- High-quality regime: In the high-perceptual-quality regime, only NIQE and PI correctly evaluated perceptual quality among the tested measures.LPIPS, Ma et al., BRISQUE, VIF, FSIM, MS-SSIM, CORNIA, and BLIINDS failed to faithfully quantify or rank quality there.
- Single-image analysis: 480 challenge outputs were analyzed on single images after subtracting each image’s mean score to remove between-image variation.The resulting trends were similar to those observed in the aggregate analysis.
- Single-image analysis: NIQE and PI were most correlated with human opinion in single-image analysis, especially at high perceptual quality.RMSE, SSIM, and IFC remained anti-correlated in this analysis.
6 Current Trends in Perceptual Super Resolution
PIRM submissions explored alternative loss functions, network architectures, and methods for controlling the perception-distortion tradeoff in perceptual SR.
- Overview: Twenty-one PIRM teams used deep networks and explored loss functions, architectures, and methods for traversing the perception-distortion tradeoff.The report summarizes challenge trends rather than reviewing the entire SR field.
- Loss functions: Adversarial training became the main loss-function trend because it learns natural-image statistics and supports realistic reconstructions.Participants used standard GAN, Relativistic GAN, and focal-loss-wrapped relativistic GAN variants.
- Loss functions: Combining adversarial and distortion objectives remained necessary because perceptual realism alone does not ensure affinity to the input.Participants also used perceptual, contextual, texture-transfer, and alternative distortion losses to address texture generation and smoothing.
- Architecture: Most teams adopted PSNR-oriented architectures while replacing their loss functions, especially EDSR generators and SRGAN discriminators.Other submissions used RRDB, DBPN, EUSR, and MGBP architectures; RRDB was associated with deeper models and improved recovered textures.
- Perception-distortion control: Four control strategies were considered: retraining, pixel-output interpolation, parameter interpolation, and an additional network input.The working point is image- and domain-specific, and its prediction is difficult with complex objectives or adversarial training.
7 Conclusions
The challenge established a perceptual-quality-driven SR benchmark and found that submitted methods advanced the state of the art, while common image-quality measures often inadequately captured perceptual quality.
- Conclusions: PIRM established the first benchmark for perceptual-quality-driven SR and enabled ranking alongside PSNR-maximization methods.Its evaluation jointly considered perceptual quality and reconstruction accuracy.
- Conclusions: Submitted algorithms pushed perceptual SR beyond existing baselines under the challenge’s evaluation scheme.The report compares submissions with existing baselines and describes the proposed methods as advancing the field’s state of the art.
- Conclusions: Most common image-quality measures were found inadequate for quantifying perceptual quality.The report presents this conclusion as the outcome of a study of common image-quality measures.
- Future challenges: In challenging settings such as 4x SR, current outputs were generally unrealistic to human observers, leaving substantial room for improvement.The conclusion also identifies blind real-world SR as a setting where perceptual-quality-driven algorithms had yet to appear.
- Future directions: The perception-distortion tradeoff motivates adaptive SR schemes and possible performance measures such as area-under-the-curve in the perception-distortion plane.The report suggests extending perceptual-quality objectives to other restoration tasks such as deblurring.
A Participating teams
Table 3 lists the participating teams in alphabetical order.
- Participating teams: Table 3 presents the participating teams in alphabetical order.
B Test phase results
Table 4 reports challenge results for the top submission from each group in each region, with tie-breaking rules based on perceptual index and RMSE.
- Test phase results: Table 4 reports the top submission from each group in each challenge region.Submissions differing by up to 0.01 in perceptual index are ranked by lower RMSE; marginal differences in both metrics are tied.
C More results
Figures 11 and 12 present additional visual super-resolution results, showing several top methods in each region alongside baselines [19, 31].
- Figure 11 compares additional visual SR results from several top methods in each region with baselines [19, 31].
- Figure 12 compares additional visual SR results from several top methods in each region with baselines [19, 31].