Source-linked AI summary
On the Use of Deep Learning for Blind Image Quality Assessment
Simone Bianco, Luigi Celona, Paolo Napoletano, Raimondo Schettini
TL;DR
The paper addresses distortion-generic blind image quality assessment by testing CNN-based representations and quality-specific fine-tuning. DeepBIQ uses fine-tuned CNN features, SVR prediction, and average pooling across image crops, achieving strong agreement with human scores across authentic and synthetic-distortion databases.
Problem
The study investigates how to predict perceptual image quality without a reference image when real-world distortions are complex mixtures not necessarily modeled by synthetic distortions.
Method
DeepBIQ fine-tunes a CNN for five category-based image-quality classes, extracts features from multiple image crops, predicts crop scores with SVR, and averages them.
Results
DeepBIQ reaches an LCC of almost 0.91 on LIVE In the Wild, 0.20 higher than the best state-of-the-art solution, and outperforms compared methods on four benchmark databases.
Takeaways & Limitations
DeepBIQ’s quality predictions are often closer to the average observer than those of a generic human observer, while remaining effective when only the SVR is retrained on new databases.
Takeaways & Limitations
The approach relies on transfer learning because blind-IQA datasets are relatively small, with earlier CNN layers typically kept fixed while higher-level layers are fine-tuned.
Abstract
from arXiv · showhide
In this work we investigate the use of deep learning for distortion-generic blind image quality assessment. We report on different design choices, ranging from the use of features extracted from pre-trained Convolutional Neural Networks (CNNs) as a generic image description, to the use of features extracted from a CNN fine-tuned for the image quality task. Our best proposal, named DeepBIQ, estimates the image quality by average pooling the scores predicted on multiple sub-regions of the original image. The score of each sub-region is computed using a Support Vector Regression (SVR) machine taking as input features extracted using a CNN fine-tuned for category-based image quality assessment. Experimental results on the LIVE In the Wild Image Quality Challenge Database and on the LIVE Image Quality Assessment Database show that DeepBIQ outperforms the state-of-the-art methods compared, having a Linear Correlation Coefficient (LCC) with human subjective scores of almost 0.91 and 0.98 respectively. Furthermore, in most of the cases, the quality score predictions of DeepBIQ are closer to the average observer than those of a generic human observer.
1 Introduction
The paper investigates deep learning for distortion-generic blind image quality assessment, evaluating CNN feature choices, image sub-regions, pooling strategies, and fine-tuning. It proposes a CNN-based approach using SVR to predict perceptual quality from image content and learned distortion features.
- Motivation: Automatic image quality assessment predicts an image’s perceptual quality, commonly represented by the Mean Opinion Score (MOS) from human ratings.Such prediction can help handle low-quality images or correct quality during capture.
- Related approaches: Blind IQA methods learn quality measures without requiring a reference image, using approaches including natural-scene statistics and machine learning.Existing examples include NIQE, DIIVINE, C-DIIVINE, BLIINDS-II, FRIQUEE, and related learned methods.
- Research hypothesis: CNN features pretrained for object and scene recognition are investigated as image representations for modeling dependencies between image content and subjective quality.The motivating hypothesis is that these features also capture distortions such as JPEG compression, noise, and blur.
- Contributions: The study evaluates pretrained CNN features, multiple image sub-regions, feature and score pooling strategies, and a CNN fine-tuned for category-based image quality assessment.The fine-tuned CNN classifies crops into five quality categories before serving as a feature extractor.
- Evaluation: Experiments use the LIVE In the Wild database of authentic mobile-camera distortions and also test applicability on LIVE, CSIQ, TID2008, and TID2013.The study reports accurate prediction with high agreement relative to human subjective scores.
2 Deep Learning for BIQ assessment
The proposed BIQ pipeline uses CNNs as learned image descriptors, optionally applies transfer learning for quality-specific representations, and pools information from multiple image crops. SVR models map these representations to image-quality scores.
- CNN foundations: CNNs use stacked convolutional, nonlinear, pooling, and fully connected layers learned jointly for a target task.Convolutional layers extract local features, while pooling aggregates nearby feature statistics.
- CNN foundations: Transfer learning adapts a CNN pretrained on a large dataset instead of training the entire network from scratch on limited BIQ data.Earlier layers may remain fixed while higher-level layers are fine-tuned for the target domain.
- Feature and prediction pooling strategies: Multiple 227×227 crops avoid image resizing, each covering almost 21% of a 500×500 image, to evaluate local quality and preserve localized artifacts.Crop pooling can distinguish globally blurred images from high-quality depth-of-field images.
- Feature and prediction pooling strategies: Prediction pooling fuses crop-level SVR quality scores using minimum, average, or maximum operators.Feature pooling and feature concatenation provide alternative ways to combine crop information before prediction.
- Image description using a fine-tuned CNN: Fine-tuning replaces the pretrained CNN’s last fully connected layer, trains the replacement from scratch, and updates other weights using available MOS-labeled quality data.The procedure uses back-propagation for image quality assessment.
3 Experimental results
Experiments evaluate CNN feature choices, crop-based fusion, and fine-tuning for blind IQA on authentic and legacy distortion databases. DeepBIQ's fine-tuned CNN with multiple crops and average-pooled predictions achieves the strongest reported performance and closely matches human-average judgments.
- Experimental setup: 1,162 authentic-distortion images and repeated 80/20 splits support evaluation against diverse real-world artifacts and human subjective scores.The LIVE In the Wild database includes over 350,000 opinion scores from more than 8,100 unique human subjects; splits are repeated 10 times.
- Experiment I: pre-trained CNNs: ImageNet+Places-CNN features outperform ImageNet-CNN and Places-CNN features, supporting broader concept coverage for generic image-content modeling.The comparison uses 4096-dimensional fc7 features from the central crop of each subsampled image.
- Experiment II: feature and prediction pooling: Feature average-pooling is statistically better than feature concatenation and the evaluated alternative fusion schemes for randomly cropped ImageNet+Places-CNN inputs.The best configurations vary crop counts from 5 to 50 and compare feature pooling, feature concatenation, and prediction pooling.
- Experiment III: fine-tuned CNN: Fine-tuning the CNN to classify crops into five MOS-based distortion classes provides the basis for the strongest DeepBIQ configuration.The five classes are bad, poor, fair, good, and excellent; class weighting improves results over batch-balancing by producing more heterogeneous batches.
- Experiment III: fine-tuned CNN: 4.7·10^-4: prediction average-pooling performs statistically better than feature average-pooling for the fine-tuned CNN.The comparison uses randomly selected crops and the best configurations selected across crop counts.
- Overall results: 0.20 and 0.21: fine-tuned CNNs with multiple crops and average-pooled predictions improve LCC and SROCC over the best compared state-of-the-art method.DeepBIQ also predicts ground-truth MOS with RMSE 8.59% and MAE 6.42%, while achieving 97.2% of predictions within one subjective-score standard deviation.
4 Conclusions
The paper evaluates deep learning designs for distortion-generic blind image quality assessment and identifies DeepBIQ as its best proposal. DeepBIQ combines a fine-tuned CNN, SVR-based sub-region scoring, and average pooling, outperforming compared state-of-the-art methods across real-world and synthetic-distortion databases.
- Experimental scope: The study reports three experiments spanning generic features from pre-trained CNNs and features from CNNs fine-tuned for image quality.These experiments examine different design choices for distortion-generic blind image quality assessment.
- DeepBIQ: DeepBIQ uses a CNN fine-tuned for category-based image quality assessment to extract features for an SVR that predicts image quality.The CNN was originally trained to discriminate 1,183 visual categories.
- Results: Almost 0.91 LCC is achieved by DeepBIQ on the reported real-world evaluation, 0.20 higher than the best state-of-the-art solution.The score pools predictions from multiple image crops using the average operator.
- Results: In many cases, DeepBIQ predictions are closer to the average observer than predictions from a generic human observer.This comparison is reported alongside the real-world quality-assessment result.
- Results: DeepBIQ outperforms all compared state-of-the-art methods on LIVE, CSIQ, TID2008, and TID2013 synthetic-distortion databases.Only the SVR is retrained for these datasets while the CNN remains unchanged.