Source-linked AI summary
PieAPP: Perceptual Image-Error Assessment through Pairwise Preference
Ekta Prashnani, Hong Cai, Yasamin Mostofi, Pradeep Sen
TL;DR
Perceptual image-error assessment aims to match human judgments, but existing models and datasets are inadequate. PieAPP uses robust pairwise preference labels to train a pairwise-learning model, then estimates single-image perceptual error; it significantly outperforms existing IQA methods and generalizes to unseen distortions.
Problem
Existing IQA methods struggle to model human perceptual differences, while available datasets are small, noisy, or dependent on inconsistent scoring procedures.
Method
PieAPP trains an error-estimation function from pairwise human preference probabilities and applies it to a distorted image and reference after training.
Results
PieAPP significantly outperforms existing state-of-the-art IQA methods, achieving PLCC 0.842 and SRCC 0.831 versus Bosse et al. (FR).
Takeaways & Limitations
Pairwise preference labels and pairwise learning provide a perceptual image-error metric that is more consistent with human opinion and can generalize to new distortions.
Takeaways & Limitations
Prior learning-based IQA evaluations commonly overlap training and test distortions, making performance on unseen distortions difficult to determine.
Abstract
from arXiv · showhide
The ability to estimate the perceptual error between images is an important problem in computer vision with many applications. Although it has been studied extensively, however, no method currently exists that can robustly predict visual differences like humans. Some previous approaches used hand-coded models, but they fail to model the complexity of the human visual system. Others used machine learning to train models on human-labeled datasets, but creating large, high-quality datasets is difficult because people are unable to assign consistent error labels to distorted images. In this paper, we present a new learning-based method that is the first to predict perceptual image error like human observers. Since it is much easier for people to compare two given images and identify the one more similar to a reference than to assign quality scores to each, we propose a new, large-scale dataset labeled with the probability that humans will prefer one image over another. We then train a deep-learning model using a novel, pairwise-learning framework to predict the preference of one distorted image over the other. Our key observation is that our trained network can then be used separately with only one distorted image and a reference to predict its perceptual error, without ever being trained on explicit human perceptual-error labels. The perceptual error estimated by our new metric, PieAPP, is well-correlated with human opinion. Furthermore, it significantly outperforms existing algorithms, beating the state-of-the-art by almost 3x on our test set in terms of binary error rate, while also generalizing to new kinds of distortions, unlike previous learning-based methods.
1 Introduction
Perceptual image-error assessment seeks to match human judgments, but existing hand-coded and learning-based IQA methods are limited by model complexity and inadequate datasets. PieAPP addresses these limitations with pairwise preference labels and a pairwise-learning framework that predicts perceptual error without explicit error-score supervision.
- Perceptual image-error estimation is important for computer vision applications including compression, restoration, and adaptive reconstruction.
- Hand-coded IQA models remain limited because accurately modeling the complexity of the human visual system is difficult.
- Existing learning-based IQA methods depend on small, noisy datasets, restricting their accuracy and generalizability to new distortions.
- 88% of MTurk participants preferred image B, while 13 existing IQA methods failed to identify it as closer to the reference.
- The proposed dataset records pairwise preference probabilities instead of subjective quality scores, aiming to provide larger and more robust supervision.
- PieAPP trains on pairwise probabilities, then applies its learned error-estimation function to one distorted image and a reference without explicit perceptual-error labels.
2 Pairwise learning of perceptual image error
The framework represents each distorted image by a perceptual-error score relative to its reference and learns those scores from human pairwise preferences. A Bradley-Terry probability model connects score differences to preferences, enabling single-image error estimation after pairwise training.
- The dataset labels each pair of distorted images by the percentage of subjects preferring image A over B relative to reference R.
- Each reference defines a separate one-dimensional perceptual-error axis, with the reference at zero and more similar distorted images placed closer to the origin.
- The Bradley-Terry function h converts errors s_A and s_B into the probability p_AB of preferring A over B.
- The model learns an error-estimation function f whose scores are constrained by observed human preference probabilities.
- The pairwise-learning system processes A, B, and R, predicts pairwise preference probabilities, and trains by minimizing squared error against human labels.
3 New DCNN for image-error assessment
The paper introduces a flexible pairwise-learning framework and implements its error-estimation function with a deep convolutional architecture. The network extracts multiscale patch features, predicts patch errors and weights, and aggregates them into an image-level perceptual error.
- The pairwise-learning framework can train different error-computation models by replacing the error-estimation function f.The authors also report improved performance when existing architectures are integrated into the framework.
- Each feature-extraction subnet has 11 convolutional layers with skip connections that compute features from input patches.The layers use 3×3 filters, ReLU nonlinearities, and 2×2 max-pooling after every even layer.
- The implementation uses three weight-shared feature-extraction subnets and two weight-shared score-computation subnets for the distorted images and reference.Together, two feature-extraction subnets and one score-computation subnet form f.
- Features from multiple convolutional depths provide different image scales, while skip connections support gradient backpropagation.The extracted feature maps are flattened and concatenated into feature vectors.
- The score-computation subnet feeds distorted-reference feature differences into fully connected networks to predict patch errors and patch weights.Weighted averaging of these patchwise quantities produces the overall image error.
- Randomly sampling 36 patches of size 64×64 from 256×256 training images gives each pixel a 0.900 probability of appearing in at least one patch.This sampling strategy is used to prevent over-fitting and improve learning.
4 Large-scale image distortion dataset
The paper constructs a large-scale pairwise-preference dataset to address unreliable labels, limited content, and narrow distortion coverage in existing IQA datasets. It separates training and testing by both reference images and distortion types, and uses statistical modeling to reduce labeling costs.
- Existing IQA datasets have unreliable quality labels and limited image-content and distortion variety, making high-quality perceptual-error metric training difficult.They omit important real-world artifacts such as deblurring and dehazing distortions.
- The proposed dataset uses pairwise preference probabilities and broad distortion coverage, with a separate test set containing non-overlapping images and distortion types.This design supports rigorous evaluation of algorithm generalizability.
- The dataset contains 200 unique reference images, with 160 used for training and 40 for testing.The reference images are selected for high quality and varied real-world content.
- Training examples pair two distorted versions A and B with a reference R and assign a probabilistic human-preference label.Comparisons include inter-type pairs with different distortions and intra-type pairs using different parameters of the same distortion.
- The test set contains 40 reference images and 31 distortions, none of which overlap with the training set.It uses 15 randomly sampled distorted images per reference and exhaustive pairwise comparisons with probabilistic labels.
- Forty human responses per pair achieve tolerance η = 0.15 with target probability Ptarget ≥ 0.94, supporting the estimated preference labels.The responses are collected through Amazon Mechanical Turk.
- Maximum-likelihood modeling estimates missing pairwise labels from a subset of comparisons, with each image appearing in at least k comparisons.Empirically, k = 10 yields a binary error rate of 0.0006 over the estimated subset.
5 Results
The evaluation validates the Bradley-Terry and maximum-likelihood estimation assumptions, then shows that PieAPP substantially outperforms existing IQA methods and generalizes to unseen distortions. The pairwise-learning framework also improves existing learning-based architectures.
- 5.1 Consistency of the BT model and the estimation of probabilistic human preference: Fig. 5 shows that Bradley-Terry probabilities closely match ground-truth human preferences, validating the model for pairwise image comparisons.Using all pairwise labels, the fitted relationship is close to the identity line.
- 5.1 Consistency of the BT model and the estimation of probabilistic human preference: 40 responses per comparison increased noise slightly, but the fit remained good, validating the procedure used to supplement hand-labeled training data.The experiment retained 10 comparisons per image.
- 5.2 Performance on our unseen test set: 0.842 PLCC and 0.831 SRCC were achieved by PieAPP, exceeding Bosse et al. (FR) by 48.24% and 54.75%, respectively.These results were measured on the proposed test set.
- 5.2 Performance on our unseen test set: 9.25% BER was obtained by PieAPP versus 24.85% for Bosse et al. (FR), a 2.7× improvement on comparisons with stronger human preference.The test set used images and distortion types disjoint from training, indicating generalization to new content and distortions.
- 5.3 Testing our architecture on other IQA datasets: PieAPP’s DCNN outperformed or matched existing methods on CSIQ and TID2013, except for TID2013 PLCC, where it was 0.11% worse than Kim et al.Performance was averaged over five random 60%/20%/20% train-validation-test splits.
- 5.4 Improving other learning-based IQA methods: Pairwise-preference training considerably improved existing learning-based IQA methods, although PieAPP’s DCNN still performed better.The framework trained Kim et al. and Bosse et al. architectures using probabilistic-preference labels.
6 Conclusion
The paper presents a perceptual image-error metric that surpasses existing metrics through pairwise preference data and pairwise learning, with potential to improve future IQA methods.
- PieAPP is a novel perceptual image-error metric that surpasses existing metrics.
- The approach uses pairwise preference to create larger, higher-quality IQA datasets.
- A pairwise-learning framework trains an error-estimation function for perceptual image error.
- The approach could enable new and improved learning-based IQA methods in the future.