PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network

Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, Zhou Wang

arXiv:1907.02665v1eess.IVcs.CVcs.MM

TL;DR

Blind image quality assessment must handle both synthetic and authentic distortions despite limited subjective training data and differing distortion characteristics. The paper proposes DB-CNN, which combines distortion-specific CNN features through bilinear pooling and fine-tunes the unified model for quality prediction. Experiments report superior performance across synthetic and authentic databases, with additional generalizability and robustness validation on Waterloo using gMAD.

  • Problem

    BIQA needs an end-to-end approach that handles both synthetic and authentic distortions, while deep methods face limited subjective ground truths and differing distortion characteristics.

  • Method

    DB-CNN separately learns synthetic-distortion and authentic-distortion features, bilinearly pools them into one representation, and fine-tunes the complete model on subject-rated databases.

  • Results

    DB-CNN achieves superior performance on synthetic and authentic IQA databases and shows generalizability and robustness across database validations and Waterloo gMAD experiments.

  • Takeaways & Limitations

    The results support separate distortion modeling, pre-training, and bilinear feature combination as the paper’s basis for a scalable, generalizable, and robust BIQA model.

Abstract

from arXiv · show

We propose a deep bilinear model for blind image quality assessment (BIQA) that handles both synthetic and authentic distortions. Our model consists of two convolutional neural networks (CNN), each of which specializes in one distortion scenario. For synthetic distortions, we pre-train a CNN to classify image distortion type and level, where we enjoy large-scale training data. For authentic distortions, we adopt a pre-trained CNN for image classification. The features from the two CNNs are pooled bilinearly into a unified representation for final quality prediction. We then fine-tune the entire model on target subject-rated databases using a variant of stochastic gradient descent. Extensive experiments demonstrate that the proposed model achieves superior performance on both synthetic and authentic databases. Furthermore, we verify the generalizability of our method on the Waterloo Exploration Database using the group maximum differentiation competition.

I. INTRODUCTION

BIQA must assess images without reference information across synthetic and authentic distortions, but existing deep methods and training strategies have limited coverage. DB-CNN addresses this by learning separate distortion-specific features and combining them for end-to-end quality prediction.

  • Motivation: BIQA is important because image-processing stages can introduce distortions that degrade perceptual quality, often without an available reference image.The motivation spans capture, compression, transmission, and storage pipelines.
  • Research gap: Deep BIQA has been limited by insufficient subjective ground truths such as mean opinion scores for training.This shortage has constrained end-to-end optimization with deep CNNs.
  • Research gap: ImageNet fine-tuning performs reasonably on authentic LIVE Challenge images but does not stand out on synthetic LIVE and TID2013 databases.The cited databases represent different distortion scenarios.
  • Proposed direction: DB-CNN learns separate feature sets for synthetic and authentic distortions, bilinearly pools them, and fine-tunes the unified model for quality prediction.Synthetic features use distortion type-and-level pre-training, while authentic features come from an ImageNet-pre-trained VGG-16.
  • Proposed direction: The synthetic pre-training data extend Waterloo and PASCAL VOC images with five additional distortion types, complementing the four original types.The added types are contrast stretching, pink noise, color quantization with dithering, over-exposure, and under-exposure.

III. DB-CNN FOR BIQA

The synthetic-distortion stream is pre-trained on a large synthesized dataset to classify distortion type and degradation level. Its CNN uses a modified VGG-style architecture and cross-entropy training to learn features for DB-CNN.

  • Pre-training data: The synthetic pre-training set combines 4,744 Waterloo images and 17,125 PASCAL VOC images into 21,869 source images.Nine distortion types are synthesized, with five levels for most types and two levels for over- and under-exposure.
  • Pre-training data: The resulting pre-training set contains 852,891 distorted images, avoiding the need for subjective scores for every synthesized image.Known distortion type and degradation level labels provide the supervision.
  • Pre-training objective: S-CNN is pre-trained as a multi-class classifier whose classes encode distortion type at a specific degradation level.This uses information available directly from the synthesis process.
  • Network architecture: S-CNN follows a modified VGG-16-style design with resized and cropped 224 × 224 × 3 inputs, global average pooling, three fully connected layers, and a softmax output.Its convolutional layers use 3 × 3 kernels and stride two, while ReLU layers are omitted from the architecture illustration.
  • Optimization: The training objective is empirical cross-entropy over mini-batch predictions from the softmax classifier.The model produces an M-dimensional probability vector for each input.

B. CNN for Authentic Distortions

Authentic distortions are difficult to simulate and provide limited training data, so the method uses ImageNet-pre-trained VGG-16 features as the authentic-distortion stream.

  • Authentic-distortion features: VGG-16 is used instead of training from scratch because authentic-distortion data are limited and small-sample CNN training often overfits.ImageNet contains realistic natural images with varied perceptual quality.

C. DB-CNN by Bilinear Pooling

DB-CNN combines synthetic- and authentic-distortion representations through bilinear pooling, then predicts quality from the unified representation. This design supports whole-image inputs of arbitrary size during training and testing.

  • DB-CNN models synthetic and authentic distortions as two-factor variations using S-CNN and VGG-16 representations.
  • The two CNN outputs are bilinearly pooled after matching their spatial dimensions, producing a d1 × d2 representation.
  • The transformed bilinear representation enters a fully connected layer with one output for final quality prediction.
  • DB-CNN is fine-tuned on target IQA databases using MOS-based ℓ2 loss between predicted and subjective quality scores.
  • Bilinear pooling summarizes spatial information and allows DB-CNN to accept whole images of arbitrary size instead of cropped patches.

IV. EXPERIMENTS

The experiments compare DB-CNN with state-of-the-art BIQA models across individual and cross-database settings, then test robustness and justify design choices.

  • Experiments compare DB-CNN with state-of-the-art BIQA models on individual databases and across databases.
  • Robustness is evaluated on the Waterloo Exploration Database using discriminability, ranking consistency, and gMAD competition criteria.
  • Ablation experiments assess the rationality of DB-CNN’s design.

A. Experimental Setups

The evaluation covers synthetic and authentic IQA databases using repeated train-test splits and SRCC and PLCC as performance criteria.

  • The main evaluation uses LIVE, CSIQ, TID2013, LIVE MD, and the authentic LIVE Challenge Database.
  • LIVE Challenge contains 1,162 real-world images with complex realistic distortions and MOS collected from over 8,100 unique human evaluators.
  • The target databases are split into 80% fine-tuning and 20% testing, with content independence guaranteed for synthetic databases.
  • The splitting procedure is randomly repeated ten times, and average results are reported across databases.
  • SRCC measures prediction monotonicity, whereas PLCC measures prediction precision after a nonlinear logistic transformation.

3) Implementation Details:

S-CNN is trained from scratch with Adam, while DB-CNN uses Adam fine-tuning with database-specific learning rates and whole-image inputs.

  • Implementation Details: S-CNN uses He initialization, Adam, mini-batches of 64, and 30 epochs with the learning rate decaying from 10^-3 to 10^-5.
  • Implementation Details: Inputs to S-CNN are scaled to 256 × 256 × 3 and cropped to 224 × 224 × 3.
  • Implementation Details: Table I reports average SRCC and PLCC results across ten sessions and highlights the top two results.
  • Implementation Details: DB-CNN fine-tuning uses Adam with learning rates of 10^-6 for LIVE and CSIQ and 10^-5 for TID2013, LIVE MD, and LIVE Challenge.
  • Implementation Details: DB-CNN uses mini-batches of eight, batch normalization, and original-size images during fine-tuning and testing.

B. Experimental Results

DB-CNN performs strongly across synthetic and authentic IQA databases, including individual distortion types. Its pre-training also supports generalization to unseen distortions with similar perceptual artifacts, although several TID2013 distortions remain difficult for all BIQA models.

  • Performance on Individual Databases: DB-CNN achieves superior performance on both synthetic and authentic IQA databases, including the authentic LIVE Challenge database.The authors attribute its LIVE Challenge success to the relevance of high-level VGG-16 features for authentic distortions.
  • Performance on Individual Distortion Types: DB-CNN ranks among the top two models in 34 of 46 individual-distortion evaluations across LIVE, CSIQ, and TID2013.On CSIQ, it performs especially strongly for pink noise and contrast change.
  • Performance on Individual Distortion Types: Pre-training on representative synthetic artifacts enables DB-CNN to generalize to unseen distortions with similar perceived appearances.Examples include generalization from Gaussian noise, Gaussian blur, and color-quantization distortions to related unseen artifacts.
  • Performance on Individual Distortion Types: All BIQA models fail on non-eccentricity pattern noise, local block-wise distortions, and mean shift in TID2013.The paper characterizes these distortion types as difficult to model.

3) Performance across Different Databases:

Across databases and annotation-free robustness tests, DB-CNN maintains comparatively strong performance despite the difficulty of transferring from synthetic to authentic distortions. gMAD comparisons expose specific weaknesses in competing models and in DB-CNN itself.

  • Performance across Different Databases: Models trained on synthetic databases do not generalize to the authentic LIVE Challenge database, yet DB-CNN achieves higher prediction accuracies in this cross-database setting.This evaluates transfer without retraining on the authentic target database.
  • Results on the Waterloo Exploration Database: The Waterloo Exploration Database evaluates DB-CNN with discriminability, ranking-consistency, and pairwise-concordance criteria without requiring human annotations.The study retrains the synthetic-distortion stream using PASCAL VOC images only to preserve image-content independence.
  • Results on the Waterloo Exploration Database: In gMAD, DB-CNN correctly identifies higher-quality images in deepIQA challenge pairs, while deepIQA fails to falsify DB-CNN.The competition generates extremal image pairs by maximizing one model’s responses while fixing the other.
  • Results on the Waterloo Exploration Database: gMAD finds no clear winner between DB-CNN and MEON, but reveals weaknesses in both models.MEON struggles with JPEG compression in images containing few structures, while it also finds a blur-related counterexample to DB-CNN.

5) Ablation Experiments:

The ablation study evaluates DB-CNN’s streams, bilinear pooling, initialization, and synthetic pre-training choices under the same experimental protocol.

  • Ablation Experiments: The ablation baseline uses either the S-CNN or VGG-16 stream alone while retaining bilinear pooling.With one stream, bilinear pooling becomes the outer product of the final convolutional activations with themselves.
  • Ablation Experiments: The study also replaces bilinear pooling with feature concatenation to assess the contribution of the fusion module.Other ablations vary initialization and the information used to pre-train the S-CNN stream.

V. CONCLUSION

DB-CNN models synthetic and authentic distortions as two factors, combining specialized CNN streams for BIQA. The authors report strong performance and broad validation, while identifying unified early feature extraction as future work.

  • DB-CNN models synthetic and authentic distortions as two-factor variations within a CNN-based BIQA framework.
  • Its reported performance is attributed to two-stream distortion modeling, pre-training, and bilinear pooling for feature combination.
  • Validation across databases, Waterloo Exploration experiments, and gMAD results support the model’s reported scalability, generalizability, and robustness.
  • Adding distortion types, stronger CNN backbones, and alternative bilinear-pooling variants are identified as possible extensions.
  • The current approach fine-tunes separately on synthetic or authentic databases rather than learning a unified model from early feature extraction onward.
Loading 1907.02665v1…