PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Quality Assessment using Contrastive Learning

Pavan C. Madhusudana, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan C. Bovik

arXiv:2110.13266v1cs.CVcs.MMeess.IV

TL;DR

Image-quality representation learning is challenging because NR-IQA must handle diverse distortions while labeled datasets remain small. CONTRIQUE uses distortion type and degree prediction with contrastive learning on unlabeled data, then freezes its CNN backbone for linear quality regression. Experiments report competitive performance across synthetic and authentic distortions without additional backbone fine-tuning.

  • Problem

    NR-IQA requires robust quality predictions for diverse, content-dependent distortions, but available labeled IQA datasets are too small for effective CNN training.

  • Method

    CONTRIQUE trains a deep CNN with a contrastive objective to predict distortion type and degree from unlabeled images containing synthetic and authentic distortions.

  • Results

    CONTRIQUE achieves competitive performance against state-of-the-art and supervised IQA models across datasets with synthetic and authentic distortions, without additional CNN-backbone fine-tuning.

  • Takeaways & Limitations

    The learned representations are robust and generalize across synthetic and realistic distortions while requiring no large labeled subjective IQA dataset for representation learning.

Abstract

from arXiv · show

We consider the problem of obtaining image quality representations in a self-supervised manner. We use prediction of distortion type and degree as an auxiliary task to learn features from an unlabeled image dataset containing a mixture of synthetic and realistic distortions. We then train a deep Convolutional Neural Network (CNN) using a contrastive pairwise objective to solve the auxiliary problem. We refer to the proposed training framework and resulting deep IQA model as the CONTRastive Image QUality Evaluator (CONTRIQUE). During evaluation, the CNN weights are frozen and a linear regressor maps the learned representations to quality scores in a No-Reference (NR) setting. We show through extensive experiments that CONTRIQUE achieves competitive performance when compared to state-of-the-art NR image quality models, even without any additional fine-tuning of the CNN backbone. The learned representations are highly robust and generalize well across images afflicted by either synthetic or authentic distortions. Our results suggest that powerful quality representations with perceptual relevance can be obtained without requiring large labeled subjective image quality datasets. The implementations used in this paper are available at \url{https://github.com/pavancm/CONTRIQUE}.

I. INTRODUCTION

NR-IQA must predict perceptual quality despite diverse, content-dependent distortions and limited labeled data. CONTRIQUE addresses this through contrastive self-supervision on unlabeled images with synthetic and authentic distortions, using distortion prediction as an auxiliary task.

  • NR-IQA estimates degraded-image quality without pristine references or known distortion types, while accounting for diverse artifacts and image content.
  • Synthetic-distortion datasets often apply a single artifact, unlike newer databases containing authentically distorted images with realistic combinations of distortions.
  • NSS-based models perform well on synthetic artifacts but are limited on images with unknown, commingled authentic distortions.
  • Small labeled IQA datasets make CNN training difficult, while fine-tuning can require dataset-specific hyperparameters and reduce generalizability through overfitting.
  • CONTRIQUE learns image-quality representations from unlabeled data using contrastive learning and distortion type-and-degree prediction as an auxiliary task.
  • The frozen CONTRIQUE backbone provides competitive quality predictions without additional fine-tuning and generalizes across synthetic and realistic distortions.

A. NR-IQA Models

NR-IQA models use handcrafted or deep features to map images to quality scores, but realistic and combined distortions remain difficult. Self-supervised representation learning offers an unlabeled-data alternative, and CONTRIQUE uses transform-domain CNN representations for this setting.

  • Blind image-quality models commonly extract features and then regress them to quality values, using either traditional handcrafted methods or deep CNNs.
  • Traditional models can perform strongly on synthetic distortions but often have limited capability on realistic distortions and their combinations.
  • CNN-based NR-IQA models commonly use ImageNet transfer learning because large-scale image-quality data are unavailable.
  • Self-supervised learning derives representations from unlabeled images by exploiting structural information, including instance discrimination and auxiliary tasks.
  • CONTRIQUE treats its CNN as a transform-domain mapping from an image x in R3×H×W to a representation h in Rd.
  • B. Self-Supervised Learning: The training pipeline seeks robust quality-predictive representations without ground-truth quality scores and is illustrated in Fig. 1.

A. Auxiliary Task

CONTRIQUE converts quality representation learning into distortion classification, identifying both distortion type and degradation degree. An encoder-projector network is trained with a pairwise contrastive objective that groups images sharing auxiliary-task classes.

  • A. Auxiliary Task: The auxiliary task classifies images by similar distortion type and quality-degradation degree to learn discriminative quality representations.
  • A. Auxiliary Task: CONTRIQUE uses an encoder followed by an MLP projector, with h=f(x) and z=g(h)=g(f(x)) representing encoder and projected embeddings.
  • A. Auxiliary Task: The objective encourages similar z representations for images in the same class using a normalized temperature-scaled cross-entropy loss.
  • A. Auxiliary Task: Unlike ordinary cross-entropy, the contrastive objective computes pairwise similarities between every image pair in a batch.

B. Multiscale Learning and Cropping

CONTRIQUE uses multiscale processing and quality-preserving transformations to learn representations robust to distortion and image content. Its contrastive objective distinguishes distortion classes while enforcing invariance across suitable transformations and scales.

  • B. Multiscale Learning and Cropping: Multiscale representations capture quality cues from both local characteristics and global image details.
  • B. Multiscale Learning and Cropping: Random crops are assumed to preserve an image’s distortion class, although their perceived quality may differ from the original.
  • B. Multiscale Learning and Cropping: The contrastive objective learns embeddings that discriminate distortion classes while remaining invariant to quality-preserving transformations.
  • B. Multiscale Learning and Cropping: Quality-preserving transformations include horizontal flipping and color-space conversion, with RGB, LAB, HSV, grayscale, and a band-pass transform used for complementary quality information.
  • B. Multiscale Learning and Cropping: For authentic distortions, transformed versions of the same user-generated image form the positive class because distortion-type labels are generally unavailable.
  • B. Multiscale Learning and Cropping: Training batches randomly contain equal numbers of synthetic and authentic distorted images to avoid sampling bias.

E. Patch Features

CONTRIQUE extracts local patch features and evaluates the resulting frozen representations with multiscale ridge regression. The encoder is not fine-tuned using ground-truth quality scores during evaluation.

  • E. Patch Features: Each input image is partitioned into nonoverlapping P × P patches to capture local distortion and quality characteristics.
  • E. Patch Features: Patch features are obtained from the encoder and used to represent local image information relevant to global picture quality.
  • E. Patch Features: After training, the projector is discarded and a regularized linear regressor maps frozen encoder representations to quality scores.
  • E. Patch Features: Evaluation concatenates features computed at full-scale and half-scale resolutions without additional fine-tuning of encoder weights.

IV. EXPERIMENTS AND RESULTS

The experiments evaluate CONTRIQUE across IQA databases containing synthetic and authentic distortions. Training uses unlabeled data that combines synthetic artifacts with realistic distortions from multiple datasets.

  • IV. EXPERIMENTS AND RESULTS: The evaluation includes multiple IQA databases and ablation experiments examining distortion types and color spaces.
  • IV. EXPERIMENTS AND RESULTS: Training data combines images impaired by synthetic and authentic distortions.
  • IV. EXPERIMENTS AND RESULTS: KADIS supplies 700k synthetically distorted images derived from 140k pristine images, spanning 25 distortion types and 5 degradation degrees.
  • IV. EXPERIMENTS AND RESULTS: Authentic-distortion training data combines AVA, COCO, CERTH-Blur, and VOC datasets intended to capture realistic image distortions.

Training Details:

The paper trains CONTRIQUE with a ResNet-50 encoder and evaluates it on eight IQA databases spanning synthetic and authentic distortions. The evaluation includes authentic and synthetic datasets with varied contents, distortion types, devices, and quality annotations.

  • Training Details:: CONTRIQUE uses a ResNet-50 encoder and a two-layer MLP projector whose final output dimension is K = 128.
  • Training Details:: Table I compares CONTRIQUE with NR models on authentic-distortion databases, grouping models by feature-extraction type and bolding the first and second best entries.
  • Training Details:: Training uses 256 × 256 crops, P = 64 patches, four patches per input image, temperature τ = 0.1, and 25 training epochs.
  • Training Details:: Experiments span 8 large IQA databases covering both synthetic and authentic distortions.
  • Training Details:: Authentic-distortion evaluation includes KonIQ, CLIVE, FLIVE, and SPAQ, with images sourced from public media, mobile devices, real-world collections, and smartphones.
  • Training Details:: FLIVE analysis uses images and quality scores but excludes its patch information, while SPAQ analysis excludes additional tag information.
  • Training Details:: SPAQ images are resized before evaluation so that their shorter side is 512.
  • Training Details:: Synthetic-distortion evaluation includes LIVE-IQA, CSIQ-IQA, TID2013, and KADID databases with multiple distortion types and degraded images.

Compared Methods:

CONTRIQUE is compared with nine state-of-the-art no-reference image-quality models spanning handcrafted, codebook-based, and CNN-based feature extraction.

  • Compared Methods:: The comparison includes nine state-of-the-art NR-IQA models across three feature-extraction categories.The categories are traditional/hand-crafted features, codebook-based features, and CNN-based models.
  • Compared Methods:: BRISQUE and NIQE represent traditional or hand-crafted features, while CORNIA and HOSA use codebook-based features.Except NIQE, these compared methods use a support vector regressor for quality prediction.
  • Compared Methods:: DB-CNN, PQR, BIECON, PaQ-2-PiQ, and HyperIQA represent CNN-based image-quality models.

Evaluation Protocol:

Performance is evaluated with SROCC and PLCC under repeated dataset splits, using comparison tables for synthetic, authentic, and cross-dataset settings. The representation visualization additionally examines learned feature organization.

  • Evaluation Protocol:: SROCC and PLCC measure agreement between predicted quality scores and image-quality judgments.PLCC is computed after applying a four-parameter logistic non-linearity to the predictions.
  • Evaluation Protocol:: Each dataset is split into 70% training, 10% validation, and 20% test data, with validation selecting the ridge-regression regularization coefficient.Synthetic-distortion splits are based on reference images to prevent content overlap, and the split procedure is repeated 10 times with median performance reported.
  • Evaluation Protocol:: Table II compares CONTRIQUE with NR models on databases containing synthetic distortions, with first- and second-best entries boldfaced.Unavailable results are marked with dashes.
  • Evaluation Protocol:: On authentic-distortion datasets, CONTRIQUE achieves performance comparable to fine-tuned CNN models without fine-tuning and outperforms ResNet-50 features.
  • Evaluation Protocol:: On synthetic-distortion datasets, CONTRIQUE achieves superior performance among the compared models, supporting generalization across synthetic and authentic distortions.
  • Evaluation Protocol:: Cross-dataset evaluation compares training and testing on different datasets, while keeping the encoder backbone fixed and changing only the linear-regressor weights.The reported comparison covers four datasets, including two synthetic and two realistic-distortion datasets.

D. Visual Comparison of Representations

CONTRIQUE representations form distortion- and degradation-level clusters in t-SNE visualizations, unlike scattered ResNet-50 features, while mixed training and combined color-space training improve generalization and correlation.

  • D. Visual Comparison of Representations: t-SNE visualizations use white noise, Gaussian blur, JPEG, JPEG2000, natural, and user-generated-content images to inspect representation structure.Each group contains 150 images, with synthetic distortions from CSIQ-IQA and natural or UGC images from KADIS and KonIQ.
  • D. Visual Comparison of Representations: CONTRIQUE features cluster by distortion type, whereas ImageNet-pretrained ResNet-50 features are scattered and not separable.
  • D. Visual Comparison of Representations: White-noise and JPEG-compression CONTRIQUE features also separate by degradation level.The figure includes zoomed views of these two distortion clusters.
  • E. Significance of Training Data: Mixed synthetic-and-authentic training improves generalization, with negligible performance loss relative to training separately on the corresponding distortion type.Synthetic-only training boosts synthetic-dataset performance, while UGC-only training does so for authentic IQA datasets.
  • F. Importance of Different Color Spaces: Combined color-space training yields higher correlations than training with any individual color space.
  • F. Importance of Different Color Spaces: Evaluation performance remains approximately unchanged across input color spaces, reducing the need to change color spaces during evaluation.

V. CONTRIQUE FULL-REFERENCE MODEL

CONTRIQUE-FR extends CONTRIQUE to full-reference IQA by incorporating reference-image information through feature differences without retraining the encoder. It achieves performance comparable to state-of-the-art full-reference models on synthetic IQA datasets.

  • Model extension: CONTRIQUE-FR modifies the quality regressor to use absolute differences between reference and distorted-image features.The reference and distorted images are encoded, and their feature difference is used to predict quality.
  • Model extension: The CONTRIQUE encoder is reused for full-reference IQA without additional training or CNN fine-tuning.Only the regressor is modified to incorporate reference information.
  • Evaluation: CONTRIQUE-FR achieves performance comparable to state-of-the-art full-reference IQA models on synthetic datasets.The comparison includes traditional models such as PSNR and SSIM and deep models such as LPIPS and DISTS.
  • Evaluation: Reference information improves CONTRIQUE correlation values relative to the no-reference setting.The paper attributes the comparison between Tables II and IV to gains from access to high-quality reference images.
Loading 2110.13266v1…