PaperlayerView exact revision on arXiv

Source-linked AI summary

dipIQ: Blind Image Quality Assessment by Learning-to-Rank Discriminable Image Pairs

Kede Ma, Wentao Liu, Tongliang Liu, Zhou Wang, Dacheng Tao

arXiv:1904.06505v1cs.CVcs.MM

TL;DR

Blind image quality assessment must learn quality without pristine references, yet reliable MOS training data are expensive and sparse in the enormous image space. The paper generates uncertain quality-discriminable pairs, trains RankNet-based dipIQ, and extends it with ListNet-based dilIQ. dipIQ outperforms prior opinion-unaware models across benchmarks, with improved robustness, while dilIQ adds further gains.

  • Problem

    BIQA lacks abundant reliable training labels because subjective MOS collection is slow, cumbersome, and expensive relative to the high-dimensional image space.

  • Method

    The paper automatically generates quality-discriminable image pairs from diverse databases, assigns perceptual uncertainty, and trains an OU-BIQA model with RankNet.

  • Results

    dipIQ outperforms previous OU-BIQA models on four benchmark databases, with improved robustness confirmed by gMAD.

  • Takeaways & Limitations

    ListNet training on quality-discriminable image lists produces dilIQ, which achieves additional performance gains.

Abstract

from arXiv · show

Objective assessment of image quality is fundamentally important in many image processing tasks. In this work, we focus on learning blind image quality assessment (BIQA) models which predict the quality of a digital image with no access to its original pristine-quality counterpart as reference. One of the biggest challenges in learning BIQA models is the conflict between the gigantic image space (which is in the dimension of the number of image pixels) and the extremely limited reliable ground truth data for training. Such data are typically collected via subjective testing, which is cumbersome, slow, and expensive. Here we first show that a vast amount of reliable training data in the form of quality-discriminable image pairs (DIP) can be obtained automatically at low cost by exploiting large-scale databases with diverse image content. We then learn an opinion-unaware BIQA (OU-BIQA, meaning that no subjective opinions are used for training) model using RankNet, a pairwise learning-to-rank (L2R) algorithm, from millions of DIPs, each associated with a perceptual uncertainty level, leading to a DIP inferred quality (dipIQ) index. Extensive experiments on four benchmark IQA databases demonstrate that dipIQ outperforms state-of-the-art OU-BIQA models. The robustness of dipIQ is also significantly improved as confirmed by the group MAximum Differentiation (gMAD) competition method. Furthermore, we extend the proposed framework by learning models with ListNet (a listwise L2R algorithm) on quality-discriminable image lists (DIL). The resulting DIL Inferred Quality (dilIQ) index achieves an additional performance gain.

I. INTRODUCTION

BIQA is needed when pristine references are unavailable, but supervised models face costly subjective labeling and sparse coverage of the high-dimensional image space. The paper proposes automatically generated quality-discriminable pairs and ranking-based opinion-unaware learning, with dipIQ and dilIQ showing strong benchmark performance.

  • BIQA evaluates image quality without access to a pristine reference, making it desirable for real-world applications where references are unavailable.
  • MOS-based BIQA training is slow, expensive, and sparse relative to the image space, limiting confidence in model generalizability.TID2013 provides only 3,000 images with MOSs, while image-space dimensionality is typically in the millions.
  • The paper automatically generates large numbers of quality-discriminable image pairs from diverse databases at low cost and assigns each pair a perceptual uncertainty measure.
  • RankNet learns an opinion-unaware BIQA model from uncertain DIPs, producing the dipIQ index without subjective opinions for training.
  • Across four benchmark databases, dipIQ outperforms previous OU-BIQA models, while gMAD confirms improved robustness and ListNet-based dilIQ provides additional gains.

A. Existing BIQA Models

Existing BIQA research combines feature extraction with learned quality prediction, using natural-image, distortion, visual-system, and increasingly learned representations. Model training commonly uses regression, probabilistic methods, or ranking approaches, but MOS-dependent pair generation limits prior pairwise methods.

  • BIQA features exploit natural-image statistics, degradation characteristics, and human visual-system knowledge.
  • Spatial, wavelet, Fourier, and DCT representations provide features for detecting blur, ringing, blocking, contrast, and structural statistics.
  • Learned BIQA features include clustered quality filters, supervised filters, quality-aware high-frequency clustering, and convolutional neural networks.
  • BIQA quality prediction commonly uses SVR, neural networks, example-based prediction, and probabilistic modeling.
  • Earlier pairwise BIQA methods rely on MOS to generate DIPs, limiting the number of training pairs and producing inferior performance to existing BIQA methods.
  • Pairwise L2R methods learn from relative orders, while listwise methods can optimize ranking losses over instance lists.

III. PROPOSED PAIRWISE L2R APPROACH FOR OU-BIQA

The proposed pairwise framework automatically constructs DIPs using trusted FR-IQA models and quantifies confidence in their quality ordering. Its uncertainty measure decreases with score separation, reaching zero beyond the selected threshold.

  • A. DIP Generation: The DIP generation engine uses MS-SSIM, VIF, and GSMD to assign quality ordering to candidate image pairs without subjective testing.Their predictions are mapped approximately to the LIVE MOS scale, where higher scores indicate better perceptual quality.
  • A. DIP Generation: Each DIP receives a nonnegative score difference T and an uncertainty U(T) in [0, 1], with larger uncertainty indicating less confidence in discriminability.
  • A. DIP Generation: T > 20 makes a DIP clearly discriminable, supporting the implementation choice Tc = 20.The paper relates this threshold to LIVE MOS variability and subject consistency for pairwise quality judgments.
  • A. DIP Generation: Fig. 1 illustrates that DIPs with similar model scores can have substantially different perceptual discriminability.

B. RankNet [16]

RankNet learns a shared quality function from image pairs by converting score differences into preference probabilities and weighting cross-entropy by DIP uncertainty. The framework supports large-scale training and both linear and nonlinear dipIQ models.

  • B. RankNet [16]: RankNet uses two streams with shared weights to apply one quality prediction function to both images in each DIP.
  • B. RankNet [16]: The difference between two predicted quality scores is converted into a probability representing the preference for one image over the other.
  • B. RankNet [16]: RankNet minimizes cross-entropy between predicted pairwise probabilities and binary DIP labels indicating which image has better quality.
  • B. RankNet [16]: Higher-uncertainty DIPs contribute less to the batch loss through uncertainty-based weighting.
  • B. RankNet [16]: A linear dipIQ reduces to logistic regression, whose convex loss provides global optimality for the solution.
  • B. RankNet [16]: RankNet is selected because stochastic or mini-batch gradient descent can handle millions of training samples, unlike slower pairwise alternatives.

IV. EXPERIMENTS

This section describes the experimental setup for evaluating dipIQ, including its RankNet-based architecture and diverse natural-image training sources.

  • The experiments evaluate dipIQ using a RankNet-based architecture and a protocol comparing it with state-of-the-art BIQA models.The paper also extends the pairwise approach to listwise learning after the main experiments.
  • 840 high-quality, high-resolution natural images represent seven scene categories: human, animal, plant, landscape, cityscape, still-life, and transportation.Each source image is down-sampled so its maximum height or width is 768 pixels.
  • Four distortion types are added to the source images: JPEG, JP2K, white Gaussian noise, and Gaussian blur.

1) Training Set Construction:

The training and evaluation setup uses CORNIA features, RankNet models, large-scale distortion data, and four benchmark IQA databases.

  • 3) RankNet Instantiation:: CORNIA features provide a shared testing basis for comparing pairwise-L2R dipIQ with regression-based BLISS.The CORNIA representation has 20,000 dimensions in the RankNet input.
  • 3) RankNet Instantiation:: The study investigates linear dipIQ* and nonlinear dipIQ models within RankNet.The nonlinear model uses three hidden layers with ReLU activations, while dipIQ* connects input directly to output.
  • 3) RankNet Instantiation:: The nonlinear architecture choices are partly ad hoc, and alternative architectures could potentially improve performance.
  • 3) RankNet Instantiation:: RankNet training uses shared two-stream weights, mini-batch gradient descent with momentum, and cross-entropy loss.The reported settings include batch size 512, momentum 0.9, weight decay 5 × 10^-4, and learning rate 10^-4.
  • 1) Training Set Construction:: More than 80 million DIPs support training, while evaluation uses LIVE, CSIQ, TID2013, and Waterloo Exploration Database.The first three databases provide MOS values; the Exploration database requires criteria designed for databases without MOS.

2) Evaluation Criteria:

The evaluation combines correlation, distortion-level ranking, pristine-versus-distorted separation, and pairwise preference consistency criteria.

  • SRCC measures agreement between MOS rankings and model-prediction rankings, while PLCC measures linear correlation between MOS and predictions.SRCC and PLCC are applied to LIVE, CSIQ, and TID2013.
  • The D-test measures how well an IQA model separates pristine and distorted images using an optimized classification threshold.Its discriminability index lies in [0, 1], with larger values indicating better separability.
  • The L-test evaluates ranking consistency across distortion levels for images sharing source content and distortion type.It assumes image quality decreases monotonically as distortion level increases and uses average SRCC.
  • The P-test measures whether an IQA model gives concordant preferences for quality-discriminable image pairs.Its preference consistency ratio lies in [0, 1], with higher values indicating better performance.
  • SRCC and PLCC use repeated random splits, with 80% of reference images for fitting and 20% for testing; median values are reported over 1,000 repetitions.PLCC first maps raw predictions to the MOS scale using a nonlinear function.

1) Comparison with FR and OU-BIQA Models:

Across benchmark databases, dipIQ generally outperforms previous opinion-unaware BIQA models, while its learned representations organize images by distortion and perceptual quality. Statistical testing and large-scale preference evaluation further support its robustness and generalizability.

  • Benchmark comparisons: dipIQ and dipIQ* outperform all previous OU-BIQA models on LIVE and CSIQ, while remaining comparable to ILNIQE on TID2013.
  • Benchmark comparisons: dipIQ* shows consistent performance gains over BLISS across LIVE, CSIQ, and TID2013, which the authors associate with uncertainty-weighted DIPs rather than noisier synthetic scores.
  • Statistical significance: At 95% confidence on LIVE PLCC values, dipIQ is statistically better than dipIQ*, which is better than all previous OU-BIQA models.
  • Robustness evaluation: On the Waterloo Exploration Database, dipIQ and dipIQ* outperform previous OU-BIQA models in the D-test and P-test and are competitive in the L-test.
  • Feature visualization: The learned third-hidden-layer features cluster images by distortion type while aligning them by perceptual quality, with high-quality images grouped regardless of content.

2) Comparison with OA-BIQA Models:

The pairwise dipIQ framework is compared with opinion-aware BIQA models using matched feature representations and training settings. It shows improved generalizability and robustness, including in gMAD comparisons with established baselines.

  • Comparison setup: dipIQ is compared with BRISQUE, DIIVINE, and CORNIA using the same feature representations and MOS-based training settings.BRISQUE and DIIVINE provide spatial- and wavelet-domain features, while CORNIA supplies the default dipIQ input.
  • Feature representations: CORNIA features generally outperform BRISQUE features, which outperform DIIVINE features under the same pairwise L2R framework.
  • Robustness: gMAD comparisons show that dipIQB and BRISQUE can disagree substantially on perceptual quality, with dipIQB judged more robust under matched feature representations and MOS training.The comparison uses image pairs selected by fixing one model's quality while maximizing or minimizing the other model's scores.
  • Conclusion: The pairwise L2R approach improves generalizability and robustness relative to opinion-aware BIQA models using the same feature representations and MOS for training.

V. LISTWISE L2R APPROACH FOR OU-BIQA

The paper extends pairwise learning-to-rank for OU-BIQA to listwise learning by constructing uncertain, ordered image lists and training ListNet. The resulting dilIQ improves over dipIQ on some databases while remaining comparable on others.

  • DIL construction: Three-element DILs are constructed by concatenating DIPs with compatible uncertainty levels and ordered quality relations.For example, ⟨i, j⟩ and ⟨j, k⟩ yield ⟨i, j, k⟩ with i better than j and j better than k.
  • ListNet model: ListNet extends RankNet with multiple shared-weight streams for list inputs and a loss based on permutation probabilities.The implementation uses three streams for three-element DILs.
  • Listwise loss: When n = 2, ListNet's loss is equivalent to RankNet's loss; for three-element DILs, the ground-truth permutation is the ordered sequence ⟨i, j, k⟩.
  • Uncertainty weighting: Perceptual uncertainty weights the listwise loss, and ListNet introduces no new parameters relative to the underlying model.The list uncertainty U_ijk is transferred from the corresponding DIPs.
  • Training: 50 million DILs are generated for ListNet training, using the same training procedure as RankNet and selecting the weights with the lowest validation loss for testing.
  • Results: dilIQ shows remarkable improvements over dipIQ on CSIQ and TID2013, while remaining comparable on LIVE and the Exploration database.The authors suggest that explicit ranking-position information may account for the improvements.

VI. CONCLUSION AND FUTURE WORK

The paper concludes that dipIQ provides an opinion-unaware BIQA framework based on automatically generated DIPs and RankNet, while dilIQ extends it with listwise learning. Future work includes broader pair and list generation, advanced ranking algorithms, handling indistinguishable-quality pairs, and end-to-end models.

  • Conclusion: dipIQ uses RankNet to learn OU-BIQA models from large numbers of DIPs automatically generated at low cost with trusted FR-IQA models.
  • Conclusion: Experiments report higher accuracy and improved robustness to content variations for dipIQ, while dilIQ achieves an additional performance gain through listwise learning.
  • Future work: Future work may develop pair and list generation engines for settings where reference images are unavailable or do not exist.
  • Future work: The current framework needs improvement to learn from quality-indiscriminable image pairs, which could be identified through subjective paired comparisons or image-source information.
  • Future work: End-to-end BIQA models may eventually bypass feature extraction and achieve stronger robustness and generalizability by combining the DIP engine with deep convolutional neural networks.
Loading 1904.06505v1…