PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Rank for Blind Image Quality Assessment

Fei Gao, Dacheng Tao, Xinbo Gao, Xuelong Li

arXiv:1309.0213v3cs.CV

TL;DR

BIQA needs scalable alternatives to subjective scores, which are imprecise, biased, inconsistent, and costly to collect. The paper learns BIQA from preference image pairs by classifying relative-quality labels with MKLGL and estimating quality scores from the learned preferences. Experiments report comparable performance to state-of-the-art BIQA methods and easy extension to new distortion categories.

  • Problem

    Subjective BIQA scores are imprecise, content-biased, inconsistent across sessions, and difficult to obtain at large scale.

  • Method

    The method learns to rank images from preference pairs, classifies preference labels using MKLGL with fused NSS features, and estimates perceptual quality scores.

  • Results

    The method achieves performance comparable to state-of-the-art BIQA algorithms and can be extended to new distortion categories.

  • Takeaways & Limitations

    Preference image pairs offer a low-cost basis for learning BIQA models while retaining high correlation with human quality perceptions.

Abstract

from arXiv · show

Blind image quality assessment (BIQA) aims to predict perceptual image quality scores without access to reference images. State-of-the-art BIQA methods typically require subjects to score a large number of images to train a robust model. However, subjective quality scores are imprecise, biased, and inconsistent, and it is challenging to obtain a large scale database, or to extend existing databases, because of the inconvenience of collecting images, training the subjects, conducting subjective experiments, and realigning human quality evaluations. To combat these limitations, this paper explores and exploits preference image pairs (PIPs) such as "the quality of image $I_a$ is better than that of image $I_b$" for training a robust BIQA model. The preference label, representing the relative quality of two images, is generally precise and consistent, and is not sensitive to image content, distortion type, or subject identity; such PIPs can be generated at very low cost. The proposed BIQA method is one of learning to rank. We first formulate the problem of learning the mapping from the image features to the preference label as one of classification. In particular, we investigate the utilization of a multiple kernel learning algorithm based on group lasso (MKLGL) to provide a solution. A simple but effective strategy to estimate perceptual image quality scores is then presented. Experiments show that the proposed BIQA method is highly effective and achieves comparable performance to state-of-the-art BIQA algorithms. Moreover, the proposed method can be easily extended to new distortion categories.

I. INTRODUCTION

BIQA relies on subjective scores that can be imprecise, content-biased, inconsistent across sessions, and costly to collect at scale. This paper addresses these limitations by learning from preference image pairs, using classification with MKLGL and a quality-score estimation strategy.

  • I. INTRODUCTION: Subjective image-quality scores may miss subtle differences because observers choose uncertain numerical values and require realignment across subjects and distortion types.The LIVE examples show that raw DMOS can misrepresent relative quality, while realigned DMOS improves correspondence.
  • I. INTRODUCTION: Quality judgments can be biased by image content, since observers may assign different scores to equally undistorted images based on aesthetic preference.Two images with different content are both described as having perfect quality, yet personal preferences can alter their scores.
  • I. INTRODUCTION: Quality scales can be inconsistent across sessions, so similar realigned scores may still correspond to perceptually different impairments across distortion categories.The FF and WN examples have nearly equal realigned DMOS values despite visibly different annoyance levels.
  • I. INTRODUCTION: Large-scale subjective databases are difficult to build because they require collecting diverse distorted images, repeated evaluations under controlled conditions, and many trained observers.These burdens constrain the reliability and extension of BIQA methods trained on subjective quality scores.
  • I. INTRODUCTION: Preference image pairs provide relative-quality labels that are generally precise, consistent, inexpensive, and less sensitive to content, distortion type, viewing conditions, or subject identity.Pairs can also compare images differing in distortion type, content, or both.
  • I. INTRODUCTION: The proposed learning-to-rank BIQA method formulates preference-label prediction as classification, investigates MKLGL, and estimates perceptual quality scores from the learned preferences.The paper positions the framework as an approach for robust BIQA and describes its organization around PIP generation, modeling, experiments, and subjective validation.

B. Learning to Rank

The paper adapts pairwise learning to rank by representing image pairs with difference features and preference labels, reducing ranking to binary classification for BIQA.

  • B. Learning to Rank: Pairwise learning to rank constructs document or image pairs with difference feature vectors and relative-preference labels, reducing ranking-function learning to classification.This formulation permits established classifiers such as SVMs and neural networks to solve the preference-learning problem.
  • A. Paired Comparisons: Preference image pairs can be labeled efficiently because large impairment differences are easy to judge across corruption types, content, and viewing conditions.In such cases, one observer may suffice because different observers can provide unanimous judgments.
  • A. Paired Comparisons: Pairs with similar distortion strengths are difficult to judge when content or distortion types differ, making forced or “same” labels unreliable and costly.These cases can recreate the need for many comparisons and observers that motivates preference-based BIQA.
  • A. Paired Comparisons: The proposed procedure therefore asks observers to label only easily distinguishable pairs, since a complete comparison graph is unnecessary for learning from PIPs.The method requires only a sample of all possible image pairs.
  • A. Paired Comparisons: Paired-comparison experiments avoid practical scale-mismatch and contextual-effect problems, allowing labels from different sessions to be aggregated directly.The paper presents an extensive subjective paired-comparison study using this flexible procedure.

B. Collecting PIPs from Existing IQA Databases

The paper generates PIPs from existing IQA databases by selecting image pairs whose quality-score differences exceed a threshold, then assigning preference labels according to MOS or DMOS.

  • B. Collecting PIPs from Existing IQA Databases: PIPs from existing databases are formed only when the absolute quality-score difference exceeds threshold T, reducing reliance on ambiguous near-ties.The threshold satisfies T ≥ 0 and controls which labeled-image pairs enter the preference set.
  • B. Collecting PIPs from Existing IQA Databases: For each selected pair, the preference label is +1 when the first image is better and −1 when the second is better.Because higher MOS means better quality whereas higher DMOS means poorer quality, label assignment depends on the database score convention.
  • B. Collecting PIPs from Existing IQA Databases: PIPs extracted from different IQA databases can be combined without realignment, and database-derived and paired-comparison PIPs can form one training set.The paper characterizes this generation process as easy and convenient relative to acquiring human quality scores.

IV. THE PROPOSED BIQA METHOD

The proposed BIQA method learns preference labels from difference feature vectors using fused NSS features and MKLGL, then estimates perceptual quality scores from the learned ranking model.

  • IV. THE PROPOSED BIQA METHOD: The method formulates mapping difference feature vectors to preference labels as classification, uses MKLGL, and then estimates perceptual quality scores.Its framework is inspired by pairwise learning-to-rank and is illustrated in the proposed-method diagram.
  • A. Integrated NSS Features: Fused NSS features from BLIINDS-II, BRISQUE-L, and SRNSS represent images because individual feature families correlate weakly with some distortion categories.The fusion is intended to provide broader distortion representation than any single feature family.
  • A. Integrated NSS Features: The integrated feature vector has dimension 84, comprising 8×3 BLIINDS-II, 18×2 BRISQUE-L, and 8×3 SRNSS features.The stated dimensionality follows the numbers of scales or feature groups specified for the three components.

B. Preparation of Training Data

Training data consists of images, integrated feature vectors, and PIPs represented by difference feature vectors with binary preference labels; reversed pairs enforce classifier symmetry.

  • B. Preparation of Training Data: The training set contains n images, one integrated feature vector per image, N PIPs, and a difference vector-label pair for every PIP.Each feature vector lies in R^d, while the PIPs are drawn from ordered image pairs.
  • B. Preparation of Training Data: For every pair with difference vector x_k and label y_k, the reversed pair contributes −x_k and −y_k so the classifier respects pair-order symmetry.The augmented sets contain 2N difference vectors and binary labels, and classification learns their mapping.

C. Preference Learning via MKLGL

The method uses multiple kernel learning based on group lasso to classify preference labels from difference-feature vectors, combining feature portions and kernels to model their similarities.

  • C. Preference Learning via MKLGL: The difference feature vector contains eight portions, each representing an image scale or particular property.
  • C. Preference Learning via MKLGL: MKLGL learns preference labels from difference-feature vectors by measuring similarities across multiple feature portions with different kernels.The approach adopts MKLGL because of its reported efficiency and effectiveness across applications.
  • C. Preference Learning via MKLGL: The implementation uses 45 Gaussian kernels: five bandwidths for each of eight portions plus five for the entire difference vector.The full-vector kernels encode potential correlations between different feature portions while limiting memory use.
  • C. Preference Learning via MKLGL: For a test pair, the learned MKLGL model predicts its preference label from the corresponding difference feature vector.A zero difference vector is assigned the preference label 0, indicating equal predicted quality.

D. Quality Prediction

The paper converts pairwise preference predictions into quality scores by summing an image’s relative gains against training images and linearly mapping those gains to a bounded scale.

  • D. Quality Prediction: Each training image receives a gain equal to the sum of its ideal pairwise preference labels, reflecting quality relative to the other training images.A label of 1 indicates that one image is better than the other, making gain proportional to perceived quality.
  • D. Quality Prediction: The gain-to-quality mapping assumes training images span the full quality range and that extreme gains correspond to the highest and lowest scores.These assumptions provide the basis for estimating the linear mapping parameters.
  • D. Quality Prediction: A test image is paired with every training image, and predicted preference labels are summed into its test-image gain before quality conversion.The test image’s feature vector is compared with each training image, and MKLGL supplies the pairwise labels.
  • D. Quality Prediction: The predicted BIQA score is bounded by [−50/(n−1), 100 + 50/(n−1)], where n is the number of training images.The expanded bounds allow scores below the poorest or above the best training image.
  • D. Quality Prediction: In most cases scores fall within [0, 100], while extreme test images can receive values outside that interval.The training set can be extended with emerging extreme-quality images, or scores can be rescaled to [0, 100] using logistic regression.
  • D. Quality Prediction: Despite its simple construction, the quality prediction approach correlates highly with human perceptions in the reported experiments.The authors state that competitiveness with MOS/DMOS-trained models would be a successful outcome despite PIPs providing a coarser description.

V. IQA EXPERIMENTS

Experiments across LIVE, TID2013, CSIQ, and LIVEMD evaluate parameter behavior, session variability, effectiveness, database independence, and extension to new distortions against established BIQA methods.

  • V. IQA EXPERIMENTS: The experiments assess parameter sensitivity, session variability, within-database effectiveness, database independence, and extension to emerging distortion categories.These objectives are tested through Parts A–E using the listed databases and training configurations.
  • V. IQA EXPERIMENTS: The study compares the proposed method with BLIINDS-II, BRISQUE, BRISQUE-L, and CORNIA as state-of-the-art BIQA baselines.The baselines are implemented using author-provided codes and corresponding literature.
  • V. IQA EXPERIMENTS: Performance is measured with KRCC, PLCC, and SRCC after nonlinear logistic mapping, where higher values indicate greater consistency with human quality judgments.SVR and SVM parameters are selected by five-fold cross-validation, and experiments are repeated 100 times for robustness.

A. Variation with Algorithm Parameters

Performance improves with more training images, while threshold selection balances noisy comparisons against lost information; the method remains stable across sessions and competitive with established BIQA algorithms.

  • A. Variation with Algorithm Parameters: Thresholding first improves performance by removing noisy pairs, but overly large thresholds discard information and reduce accuracy.The method remains acceptable even at T = 70, and similar behavior appears across other parameter settings and databases.
  • A. Variation with Algorithm Parameters: Performance improves monotonically as training images or PIPs increase, with training-image count having the larger impact on quality estimation.With more training images, each test image receives more comparisons; the authors therefore recommend many images but only a small random subset of possible pairs.
  • C. Consistency with Human Opinions: Predicted scores vary by less than 5 points in most sessions and remain highly consistent with DMOS across 100 sessions.The joint histogram on LIVE indicates robustness to session variability.
  • C. Consistency with Human Opinions: Using only 1.3% of all possible image pairs at T = 10 and Ng = 20, the proposed method still outperforms several baselines and remains comparable to others.On LIVE, it is much better than BRISQUE-L, BLIINDS-II, and f SRNSVR with Ng = 5, and remains competitive as training images increase.
  • C. Consistency with Human Opinions: The proposed method is highly comparable to state-of-the-art BIQA algorithms and statistically achieves the best result across the evaluated databases.The comparison covers LIVE, TID2013, CSIQ, and LIVEMD, with fused NSS features helping address weak correlations affecting individual feature sets.

D. Database Independence

Training on preference image pairs yields a BIQA model that transfers across databases, outperforming MOS/DMOS-trained baselines in cross-database testing while retaining strong agreement with human opinions.

  • D. Database Independence: The proposed method achieves the best performance on TID2013 and acceptable, comparable performance on CSIQ, but NSS-based methods fail on LIVEMD’s GblurWN distortion.For GblurJPEG, the learned model works well because its feature deviations resemble those of singly distorted images.
  • D. Database Independence: Across databases, predicted scores remain highly consistent with DMOS, supporting the conclusion that the learned parameters are not over-fitted and are insensitive to database differences.The authors report the best SRCC values over all databases.
  • D. Database Independence: On the same database, Proposed competes favorably with f allMKL, while ProposedSVM is slightly worse than f allSVR.This supports competitive performance from models trained on PIPs despite their different supervision.
  • D. Database Independence: When trained on LIVE and tested elsewhere, Proposed and ProposedSVM statistically outperform f allSVR and f allMKL, indicating lower dependence on the training database.The cross-database evaluation covers TID2013, CSIQ, and LIVEMD.

E. Easy Extension of the Proposed Method

The framework extends to new distortion categories by adding a small number of preference image pairs (PIPs), but cross-distortion comparisons are needed to avoid quality-scale mismatch. Experiments on LIVE and hybrid databases show performance comparable to state-of-the-art methods, with further gains possible from more PIPs.

  • E. Easy Extension of the Proposed Method: The method is easily extended to emerging distortion categories by adding corresponding PIPs to the training set.The authors report that only hundreds of PIPs per distortion category can produce satisfactory performance.
  • E. Easy Extension of the Proposed Method: Training separately within distortion subsets performs worse than using 2000 PIPs from the entire LIVE database because cross-distortion quality relations are omitted.The resulting mismatch is visible for the WN distortion category, motivating both intra-distortion and inter-distortion PIPs.
  • E. Easy Extension of the Proposed Method: On hybrid databases, Proposed1 and Proposed2 are highly comparable to state-of-the-art BIQA methods across LIVE, TID2013, and CSIQ.They outperform other algorithms for most listed TID2013 distortion categories, but remain below several methods on LIVE and CSIQ.
  • E. Easy Extension of the Proposed Method: Proposed2 outperforms Proposed1 because its training data include relative-quality information between images from different databases.Proposed1 and Proposed2 differ in whether PIPs are formed across the hybrid databases.
  • E. Easy Extension of the Proposed Method: Unlike score-based extensions requiring realignment and many subjective ratings, PIPs can be generated cheaply and labeled by only a few subjects.The authors identify realignment as imprecise and dependent on databases covering quality ranges with small intervals.

VI. SUBJECTIVE STUDY PIPS

The subjective study evaluates whether preference labels from paired comparisons are reliable and inexpensive relative to DMOS-based judgments. Single-subject labels closely match multi-observer and DMOS-derived judgments, especially when quality differences are large.

  • VI. SUBJECTIVE STUDY PIPS: The preference-label study used approximately 240,000 randomly selected pairs from 808 LIVE images, with 10,000 pairs evaluated by all subjects.Most pairs were evaluated by one subject, while the multiply labeled subset served as a benchmark.
  • VI. SUBJECTIVE STUDY PIPS: Each image-pair evaluation takes 3 seconds on average, demonstrating the efficiency of generating PIPs.The study used paired comparisons of LIVE images, with sessions limited to 30 minutes to reduce observer fatigue.
  • VI. SUBJECTIVE STUDY PIPS: When DMOS differences exceed 10, DMOS-derived relative-quality labels generally agree with human judgments; small differences are less precise.Small DMOS differences produce larger distribution variances and more uncertain preference labels.
  • VI. SUBJECTIVE STUDY PIPS: A single subject's preference judgments achieve accuracy highly comparable to DMOS labels collected from multiple subjects.The authors conclude that one subjective judgment per image pair can provide valid preference labels while reducing training-data collection cost.
  • VI. SUBJECTIVE STUDY PIPS: The proposed BIQA method performs almost identically when trained with subjective preference labels or labels derived from DMOS.This validates paired-comparison labels and supports generating PIPs from existing IQA databases with a suitable threshold.

B. IQA Performance

The paper evaluates whether subjectively labeled preference pairs support BIQA quality prediction on LIVE. The resulting method achieves nearly the same performance as with DMOS-derived labels, while predicted-score comparisons are slightly more accurate than direct pair classification.

  • B. IQA Performance: The framework learns from difference feature vectors of subjectively labeled test-image pairs and predicts their preference labels with a classifier.The evaluation uses AccAll and Acccnst for direct preference-label prediction, alongside accuracies based on predicted quality scores.
  • B. IQA Performance: The proposed BIQA method obtains almost the same performance from subjective preference labels as from labels derived from DMOS.This demonstrates the validity of preference labels obtained through paired comparisons.
  • B. IQA Performance: Predicted quality-score comparisons are slightly more accurate than direct preference-label predictions because more comparisons are performed for each test image.This result supports the efficacy of the proposed quality-prediction approach.
  • B. IQA Performance: The paper reports that the framework correlates highly with human quality perceptions and can be extended to emerging distortions.The conclusion is based on experiments across four large standard databases and the subjective PIP study.
Loading 1309.0213v3…