Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild

Weixia Zhang; Kede Ma; Guangtao Zhai; Xiaokang Yang

Source-linked AI summary

Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild

Weixia Zhang, Kede Ma, Guangtao Zhai, Xiaokang Yang

arXiv:2005.13983v6May 28, 2020cs.CVcs.LGcs.MMeess.IV

TL;DR

The paper addresses BIQA’s weak generalization between synthetic and realistic distortions and the incompatible perceptual scales of IQA databases. It trains a unified uncertainty-aware model using probabilistic image-pair rankings and a hinge regularizer, with favorable results across six databases and cross-scenario evaluations.

Problem
BIQA models trained for synthetic distortions remain weak on realistic distortions and vice versa, while IQA databases use incompatible perceptual scales and human judgments contain informative uncertainty.
Method
The method samples image pairs within databases, derives higher-quality probabilities from MOSs, DMOSs, and variances, optimizes a DNN with fidelity loss, and regularizes uncertainty with a hinge constraint.
Results
UNIQUE performs favorably against state-of-the-art BIQA models on six databases and shows strong cross-distortion-scenario behavior in cross-database and gMAD evaluations.
Takeaways & Limitations
The training strategy provides a unified way to learn from multiple distortion scenarios while aligning images on a learned perceptual scale and modeling human uncertainty.

Abstract

from arXiv · show

Performance of blind image quality assessment (BIQA) models has been significantly boosted by end-to-end optimization of feature engineering and quality regression. Nevertheless, due to the distributional shift between images simulated in the laboratory and captured in the wild, models trained on databases with synthetic distortions remain particularly weak at handling realistic distortions (and vice versa). To confront the cross-distortion-scenario challenge, we develop a \textit{unified} BIQA model and an approach of training it for both synthetic and realistic distortions. We first sample pairs of images from individual IQA databases, and compute a probability that the first image of each pair is of higher quality. We then employ the fidelity loss to optimize a deep neural network for BIQA over a large number of such image pairs. We also explicitly enforce a hinge constraint to regularize uncertainty estimation during optimization. Extensive experiments on six IQA databases show the promise of the learned method in blindly assessing image quality in the laboratory and wild. In addition, we demonstrate the universality of the proposed training strategy by using it to improve existing BIQA models.

I. INTRODUCTION

The paper targets BIQA models that generalize across synthetic and realistic distortions despite database scale mismatches and uncertain human judgments. It proposes unified pairwise training, uncertainty regularization, and the UNIQUE model for cross-scenario assessment.

BIQA models trained on synthetic distortions remain weak on realistic distortions, while models trained on realistic distortions do not work well on synthetic distortions.
Combining IQA databases is difficult because their subjective scores use different perceptual scales, making direct training unreliable without realignment.CSIQ uses DMOSs in [0, 1], whereas LIVE Challenge uses MOSs in [0, 100].
A hinge regularizer uses human opinion variances to constrain learned uncertainty and make BIQA models mimic uncertainty in human assessments.
The training strategy samples image pairs within each database and converts MOSs, DMOSs, and variances into probabilities that one image has higher perceptual quality.This pairwise supervisory signal bypasses additional subjective testing for perceptual scale realignment.
UNIQUE is a DNN-based unified BIQA model that performs favorably against state-of-the-art BIQA models across six databases covering synthetic and realistic distortions.The evaluation also includes cross-database testing and gMAD competition.

II. RELATED WORK

Related BIQA work evolved from distortion-specific models to general-purpose, deep, and ranking-based approaches. The paper positions UNIQUE as a unified model using continuous human-derived ranking information across heterogeneous databases.

Early BIQA methods targeted specific synthetic distortions, whereas later general-purpose methods learned quality from natural-scene statistics.
NIQE sought to capture arbitrary distortions, but its natural-scene-statistics model was not sensitive to unnaturalness introduced by realistic distortions.
Deep learning advanced BIQA through joint optimization of feature engineering and quality regression, alongside transfer learning and patch-based training strategies.
B. BIQA as Ranking: Ranking-based BIQA methods obtain relative quality information from distortion specifications, full-reference predictions, or human data.
B. BIQA as Ranking: UNIQUE differs from earlier ranking methods by using continuous ranking information derived from MOSs and variances during final quality prediction.

C. Uncertainty-Aware BIQA

The proposed training strategy combines image pairs from multiple IQA databases and infers probabilistic rankings from human quality distributions. This supports joint training while avoiding additional perceptual-scale realignment and enables future databases to be added at essentially no cost.

The method combines IQA databases and estimates a pairwise learning-to-rank model for unified BIQA.
A. IQA Database Combination: It randomly samples image pairs within each database and infers relative rankings from the corresponding MOSs and variances.
A. IQA Database Combination: Combining pairs from multiple databases builds a training set without additional subjective experiments for perceptual-scale realignment.
A. IQA Database Combination: Under Thurstone’s model, each image’s true perceptual quality is represented by a Gaussian distribution, so pairwise quality differences are also modeled probabilistically.
A. IQA Database Combination: The database-combination approach allows future IQA databases to be added with essentially no cost.

B. Model Estimation

UNIQUE estimates image quality and uncertainty jointly from pairwise perceptual preferences. Its objective combines fidelity loss for preference probabilities with a hinge regularizer for uncertainty.

Model formulation: UNIQUE learns differentiable quality and uncertainty functions from images of arbitrary input size.The functions are parameterized by w and estimate perceptual quality and uncertainty, respectively.
Training strategy: Training combines randomly sampled within-database image pairs before pairwise learning-to-rank estimation.The strategy consists of IQA database combination followed by model estimation.
Loss design: Fidelity loss measures similarity between ground-truth and predicted preference distributions while avoiding stated drawbacks of cross-entropy.The passage identifies nonzero minima near p=0.5 and unbounded penalties for hard examples as cross-entropy drawbacks.
Uncertainty regularization: A hinge regularizer addresses scaling ambiguity by directly supervising predicted uncertainty with human-opinion variance.The hinge term uses a margin ξ and is added to the fidelity objective with trade-off parameter λ.

C. Specification of UNIQUE

UNIQUE uses a shared ResNet-34-based pairwise architecture to produce both image quality and uncertainty estimates. Its second-order bilinear pooling summarizes spatial statistics before a two-output prediction layer.

Backbone and streams: UNIQUE uses ResNet-34 as its backbone because it balances model complexity and capability.The pairwise framework contains two streams implemented by DNNs with shared parameters.
Backbone and streams: Each stream combines convolution, batch normalization, ReLU, max-pooling, and four residual blocks.This structure follows the pairwise learning-to-rank design shown in Fig. 2.
Feature representation: Second-order bilinear pooling replaces first-order average pooling to summarize higher-order spatial statistics.The feature representation is flattened after the last convolution before bilinear pooling.
Uncertainty motivation: Human quality opinions across six databases are more consistent at low and high quality than at mid quality.The means-versus-standard-deviations plots therefore form arch-like shapes.
Prediction heads: A fully connected layer with two outputs represents fw(x) and σw(x), while the two stream parameters remain shared.The outputs correspond to predicted quality and uncertainty.

IV. EXPERIMENTS

The experiments compare UNIQUE with state-of-the-art BIQA models and test the proposed training strategy beyond the main benchmark. Evaluations include alternative training schemes, cross-database testing, and fine-tuning an existing model.

Evaluation program: The study compares UNIQUE with several state-of-the-art BIQA models on existing IQA databases and in the gMAD competition.The experiments are designed to assess performance across the paper’s evaluation settings.
Training-strategy evaluation: The training strategy is evaluated against alternative schemes and in a cross-database setting.These comparisons test the strategy separately from the main model comparison.
Model generalizability: The strategy is also applied to improve an existing differentiable BIQA model.DB-CNN is used as the example model for this model-agnostic test.

A. Experimental Setups

The experiments use six databases spanning synthetic and realistic distortions, with database-specific train/test splits and SRCC and PLCC evaluation. UNIQUE is trained with a specified pair count and training schedule.

Databases: Six databases cover synthetic distortions in LIVE, CSIQ, and KADID-10K and realistic distortions in LIVE Challenge, BID, and KonIQ-10K.All selected databases provide standard deviations of subjective quality scores.
Databases: TID2013 is excluded because its MOS construction lacks suitable psychometric scaling and does not satisfy the Gaussian assumption.Its MOS is based on winning counts from at most nine paired comparisons.
Data splits and metrics: 80% of images from each database form training data and 20% form testing data.Synthetic databases are split by reference images to ensure content independence.
Data splits and metrics: SRCC measures prediction monotonicity, whereas PLCC measures prediction precision.Both criteria are adopted for performance evaluation.
Training details: UNIQUE is trained on 270,000 image pairs using Adam with hinge margin ξ=0.025 and trade-off parameter λ=1.The reported performance was empirically insensitive to these two hyperparameters.
Training details: Training lasts twelve epochs with warm-up optimization followed by full-network fine-tuning.The first three epochs train only the last fully connected layer; subsequent epochs fine-tune the entire method.

1) Correlation Results:

UNIQUE is evaluated against BIQA and full-reference models across synthetic and realistic distortion databases, including cross-database and gMAD tests. The results indicate favorable performance across scenarios and robustness against competing models.

UNIQUE performs favorably against all competing BIQA models across six databases containing synthetic and realistic distortions.
UNIQUE also achieves competitive performance against state-of-the-art full-reference IQA models on synthetic databases.
gMAD evaluates IQA models by seeking image pairs that appear similarly scored by one model but substantially different to another, with human MOS used to quantify outcomes.
UNIQUE consistently spots PQR's failures in gMAD tests, supporting quality assessment across laboratory and wild images.
Against DB-CNN, UNIQUE survives attacks with human-consistent image quality and finds counterexamples that falsify DB-CNN.

3) Uncertainty Estimation Results:

The hinge regularizer improves the interpretability of UNIQUE's learned uncertainty. With the regularizer, uncertainty follows the human-like pattern of greater confidence at the quality extremes.

The hinge regularizer makes UNIQUE's uncertainty human-like, with lower uncertainty at both ends of the quality range.
Without the hinge regularizer, the learned uncertainty is less interpretable.

4) Qualitative Results:

Qualitative analyses show that UNIQUE aligns images from different databases on a meaningful perceptual scale and generalizes across synthetic and realistic distortions in cross-database testing.

UNIQUE aligns images from different databases in a perceptually meaningful quality scale despite forming training pairs within individual databases.
In cross-database training, UNIQUE achieves significantly better performance than four knowledge-driven and two DNN-based models.
Training on KADID-10K and KonIQ-10K simultaneously provides mutual regularization and supports generalization across synthetic and realistic distortions.

2) Performance of Different Training Strategies:

The proposed pairwise training strategy outperforms alternative training approaches and can also improve existing differentiable BIQA architectures. Its reported gains are attributed to continuous ranking supervision, fidelity loss, and hinge-based uncertainty modeling.

The proposed training method is statistically superior to binary labeling on LIVE, CSIQ, KADID-10K, and BID, and statistically indistinguishable on the remaining two databases.
The model-agnostic training strategy improves DB-CNN performance by as much as 151.37% over original versions trained on CSIQ and LIVE Challenge.
The proposed strategy uses continuous ranking probabilities derived from (D)MOSs and variances, fidelity loss, and a hinge uncertainty regularizer.
The authors attribute the performance improvements to more accurate continuous supervision, appropriate fidelity-loss penalties, and improved uncertainty modeling.