PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment

Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, Wojciech Samek

arXiv:1612.01697v2cs.CV

TL;DR

The paper addresses the challenge of learning perceptually meaningful image quality without hand-crafted domain knowledge across both FR and NR settings. It develops a deep end-to-end CNN with feature fusion and optional learned patch weighting, and reports superior IQA performance with competitive cross-database generalization. However, the authors note that generalization still has considerable room for improvement.

  • Problem

    Existing IQA approaches lacked an end-to-end trained FR method, while perceived quality and patch importance vary spatially and NR assessment has no reference image.

  • Method

    A deep CNN uses Siamese FR feature extraction, adaptable NR processing, feature fusion, and optional joint learning of patch quality and patch weights.

  • Results

    The proposed methods outperform state-of-the-art NR and FR IQA approaches and achieve cross-database generalization competitive with state-of-the-art data-driven methods.

  • Takeaways & Limitations

    Purely data-driven neural networks show strong potential for IQA without requiring hand-crafted features or prior domain knowledge.

  • Takeaways & Limitations

    Generalization performance still offers considerable room for improvement, as with current data-driven IQA methods.

Abstract

from arXiv · show

We present a deep neural network-based approach to image quality assessment (IQA). The network is trained end-to-end and comprises ten convolutional layers and five pooling layers for feature extraction, and two fully connected layers for regression, which makes it significantly deeper than related IQA models. Unique features of the proposed architecture are that: 1) with slight adaptations it can be used in a no-reference (NR) as well as in a full-reference (FR) IQA setting and 2) it allows for joint learning of local quality and local weights, i.e., relative importance of local quality to the global quality estimate, in an unified framework. Our approach is purely data-driven and does not rely on hand-crafted features or other types of prior domain knowledge about the human visual system or image statistics. We evaluate the proposed approach on the LIVE, CISQ, and TID2013 databases as well as the LIVE In the wild image quality challenge database and show superior performance to state-of-the-art NR and FR IQA methods. Finally, cross-database evaluation shows a high ability to generalize between different databases, indicating a high robustness of the learned features.

I. INTRODUCTION

The paper frames IQA as important for human-centered image communication and develops a deep, end-to-end approach spanning FR and NR settings. It also addresses spatially varying patch importance and evaluates generalization across multiple databases.

  • IQA is essential because digital images are commonly intended for human viewing in communication and processing systems.
  • NR IQA is more difficult than FR IQA because it has no access to the original reference, despite its practical relevance when references are unavailable.
  • 10 convolutional layers, 5 pooling layers, and 2 fully connected layers form the proposed deep CNN for general IQA.
  • The FR model uses Siamese feature extraction and feature fusion for joint regression of reference and distorted-image features.
  • Patchwise relative weights address nonuniform local quality and importance, enabling joint optimization of local assessment and global pooling.
  • Removing one feature-extraction path adapts the FR architecture to NR IQA, producing DIQaM-NR and WaDIQaM-NR variants.
  • Experiments cover LIVE, TID2013, CISQ, and CLIVE, with additional cross-database analyses of generalization ability.

II. RELATED WORK

Prior IQA methods use hand-crafted image statistics, visual-system-inspired models, or learned representations, but end-to-end learning remained limited for FR IQA. Saliency-based pooling also generally modeled attention rather than jointly optimizing perceived quality.

  • MSE is widely used but correlates poorly with perceived visual quality, motivating more perceptually aligned IQMs.
  • Traditional IQA methods commonly model distortion-related feature changes in FR, RR, and NR settings, with SSIM as a prominent example.
  • II. RELATED WORK: Before this work, feature learning and regression had been used in FR IQA, but no end-to-end trained method was known for that task.
  • II. RELATED WORK: NR methods such as DIIVINE, BLIINDS-II, BRISQUE, NIQE, and FRIQUEE model natural-image statistics or distortion-related parameters.
  • II. RELATED WORK: CORNIA combines feature and regression training for NR IQA using codebook-based features, pooling, and SVR.
  • II. RELATED WORK: Earlier CNN-based IQA work combined feature extraction and regression but used a shallow architecture and average patch pooling.
  • C. Salience and Attention for IQA: Saliency-weighted IQMs combine local quality with attention estimates, yet saliency models generally predicted attention rather than perceived quality through joint optimization.

III. DEEP NEURAL NETWORKS FOR IMAGE QUALITY ASSESSMENT

The proposed FR IQA network uses parallel Siamese CNN processing, feature fusion, regression, and patch aggregation. Its architecture supports alternative fusion strategies, regularized deep layers, and simple or weighted spatial pooling.

  • A. Neural network-based FR IQA: A Siamese network processes reference and distorted inputs in parallel, sharing connection weights for feature extraction.
  • B. Feature Fusion: Feature fusion combines reference and distorted feature vectors by concatenation, difference, or concatenation supplemented with their difference.
  • A. Neural network-based FR IQA: The fused features enter the regression module to estimate local image quality for FR IQA.
  • A. Neural network-based FR IQA: VGGnet provides the deep CNN basis, while ReLU activations, zero-padding, max pooling, and dropout regularize the architecture.
  • A. Neural network-based FR IQA: Images are divided into 32 × 32 patches whose local qualities are pooled into global quality by simple or weighted averaging.
  • A. Neural network-based FR IQA: The resulting models are named DIQaM-FR and WaDIQaM-FR for simple and weighted average patch aggregation, respectively.

B. Feature Fusion

The FR IQA network fuses reference and distorted-image features before regression, then pools patchwise quality estimates either uniformly or using learned spatial weights. The weighted approach jointly models local quality and relative patch importance.

  • Feature Fusion: Reference and distorted feature vectors are fused by concatenation or their feature-space difference before regression.The vectors share identical structure, making their difference a meaningful distance representation.
  • Pooling: Simple averaging aggregates patchwise qualities under the assumption that every image region has identical relative importance.The global estimate averages the locally estimated qualities across sampled patches.
  • Pooling: MAE is selected instead of MSE because simple average pooling introduces label noise and MAE reduces the influence of outliers.The noise arises because local quality is implicitly treated as identical to the global image quality.
  • Weighted Pooling: Spatially non-uniform distortions, summation, and saliency effects motivate learning patch-specific weights alongside local quality.Global and local perceived quality need not coincide under these conditions.
  • Weighted Pooling: A parallel regression branch outputs α_i, transforms it into positive weights, and uses normalized weights to compute the global quality estimate.The weighted pooling method is formally equivalent to linear saliency weighting.
  • Weighted Pooling: Joint end-to-end training minimizes a loss based on the weighted global estimate rather than treating patch quality and weighting as separate procedures.The quality and weight branches share the feature-extraction pathway but have different regression parameters.

D. Network Adaptations for NR IQA

The FR architecture becomes an NR IQA model by removing the reference-image branch and feature fusion, while retaining either simple or weighted spatial pooling. Training uses grouped patches and adaptive optimization procedures suited to the network’s structure.

  • D. Network Adaptations for NR IQA: NR adaptation removes the Siamese branch that extracts reference-patch features, so feature fusion is no longer required.The resulting models are named DIQaM-NR and WaDIQaM-NR for simple and weighted pooling, respectively.
  • D. Network Adaptations for NR IQA: Both simple-average and weighted-average spatial pooling methods remain applicable in the NR setting.The NR variants use the same loss functions as the corresponding FR approaches.
  • Training: Weighted-average pooling requires patches from the same image to remain together during batch construction.This differs from simple average pooling, where each image patch can be treated as a separate sample.
  • Training: The learning rate is adapted per parameter with ADAM using β1 = 0.9, β2 = 0.999, ϵ = 10^-8, and α = 10^-4.Validation loss is computed after each epoch in evaluation mode.
  • Training: The patch-quality and patch-weight regression branches do not have identical weights because their updates use gradients with respect to different parameters.The distinction applies to the two parallel regression branches.

IV. EXPERIMENTS AND RESULTS

Experiments evaluate the proposed IQA networks across established synthetic-distortion databases and a real-world NR database, using reference-image-based splits and cross-database testing. The evaluation also standardizes score ranges and retains all TID2013 distortion types.

  • Datasets: The experiments use LIVE, TID2013, CSIQ, and the LIVE In the Wild Image Quality Challenge Database.The NR approach is additionally evaluated on the real-world CLIVE database.
  • Datasets: LIVE contains 779 images from 29 reference images with five distortion types, including compression, noise, blur, and fast-fading channel distortions.Its DMOS ratings range from 0 to 100, with lower values indicating better visual quality.
  • Datasets: TID2013 contains 3000 images from 25 references, covering 24 distortion types at five distortion levels.Its distortions range from common noise, blur, and compression to more exotic pattern noise.
  • Datasets: CSIQ contains 866 images from 30 references with compression, blur, noise, and contrast distortions, and DMOS values spanning 0 to 1.Lower CSIQ DMOS values indicate better visual quality.
  • Datasets: CLIVE contains 1162 naturally captured images with mixed real-world impairments and no undistorted reference images.The images vary in scenes, objects, luminance conditions, and cameras.
  • Evaluation Protocol: Reference-image-based splits prevent distorted or undistorted versions of test and validation images from appearing in training.Cross-dataset evaluation uses CSIQ to test models trained on LIVE or TID2013.
  • Evaluation Protocol: The evaluation uses the full TID2013 database, maps scores to the LIVE DMOS range, and quantifies prediction accuracy with Pearson correlation.The mapping makes errors and gradients comparable across databases and treats higher local scores as higher distortion.

C. Performance Evaluation

The proposed models are evaluated against state-of-the-art methods across full-reference and no-reference settings, databases, and distortion subsets. Weighted pooling is especially beneficial for full-reference IQA and spatially inhomogeneous distortions, while its no-reference advantage depends on the database and distortion type.

  • Full-reference IQA: The FR models outperform state-of-the-art methods on LIVE, while WaDIQaM-FR is best overall on TID2013 and across its grouped distortion types.DIQaM-FR is comparable to existing methods on TID2013, whereas weighted patch aggregation improves performance consistently across grouped distortions.
  • No-reference IQA: DIQaM-NR achieves the best evaluated LCC on LIVE and the best LCC and SRCC on TID2013, with SROCC on LIVE slightly below SOM.The comparison includes other state-of-the-art no-reference methods, although some methods lack TID2013 results.
  • No-reference IQA: On CLIVE, WaDIQaM-NR outperforms most evaluated models but is clearly outperformed by FRIQUEE, while still surpassing DIQaM-NR.CLIVE is more difficult than LIVE and TID2013, and all evaluated methods perform worse there.
  • Distortion dependence: For NR IQA, weighted pooling decreases performance for Gaussian blur, JPEG, and JP2K distortions but increases it for local block-wise distortions.The authors conjecture that reference information is usually important for assigning local pooling weights, whereas strongly inhomogeneous distortions can be weighted from the distorted image.
  • Distortion dependence: Weighted patch aggregation can compensate for spatially inhomogeneous distortions, explaining its large performance increase on the exotic TID2013 subset.The paper links this effect to the greater amount of inhomogeneous distortion types in that subset.

D. Local Weights

The weighted models learn local quality estimates together with spatially varying weights, whose usefulness depends on distortion structure and reference availability. The visualizations show that FR weighting can roughly segment salient image regions, while NR weighting succeeds particularly when distortion dominates image structure.

  • Local weights: The influence of weighted pooling depends on distortion type and on whether a reference image is available.The paper illustrates this dependence with JP2K, LBDDI, and CLIVE examples.
  • JP2K example: For JP2K distortion, WaDIQaM-FR improves the prediction over DIQaM-FR and its weight map roughly segments object-containing regions, unlike WaDIQaM-NR.The example has MOS 34; predictions are 54, 42, 60, and 70 for DIQaM-FR, WaDIQaM-FR, DIQaM-NR, and WaDIQaM-NR, respectively.
  • LBDDI example: For LBDDI, both weighted models assign higher weights to distorted regions and improve prediction over simple averaging, while FR weighting again roughly segments the image.NR segmentation still fails, but the image structure is less important for this extreme distortion.
  • CLIVE examples: CLIVE’s spatially concentrated scenes can let WaDIQaM-NR identify patches contributing most to overall image structure, whereas globally underexposed scenes make impairment localization difficult.This spatial variance is proposed as one reason weighted NR pooling helps CLIVE relative to LIVE and TID2013.
  • Local weights: Weighted average patch aggregation affects both predicted quality maps and local weights because joint optimization couples yi with α* i.The method jointly learns local quality and relative local importance rather than averaging local qualities uniformly.

E. Cross-Database Evaluation

Cross-database evaluation shows that generalization depends on the training database and distortion overlap. Weighted models can improve adaptation, but performance on unseen images and distortions remains limited.

  • Cross-database FR evaluation: WaDIQaM-FR generalizes better than DIQaM-FR across the reported training–test combinations and performs comparably to DOG-SSIM.The comparison covers models trained on LIVE or TID2013 and tested on TID2013, CSIQ, or LIVE.
  • Cross-database NR evaluation: DIQaM-NR outperforms BRISQUE and CORNIA on the CSIQ subset but is outperformed by other state-of-the-art methods on the TID2013 subset.These subsets contain the four distortions shared across LIVE, CSIQ, and TID2013.
  • Cross-database NR evaluation: WaDIQaM-NR appears better able than DIQaM-NR to adapt to unseen distortions, consistent with its results on CLIVE.The paper characterizes this comparison as suggestive rather than definitive.
  • Generalization limits: None of the compared learning-based methods exceeds SROCC 0.5 when a LIVE-trained model is tested on the diverse TID2013 distortion setting.Only four of TID2013’s 24 distortions are represented in the LIVE training data.
  • Generalization limits: On CSIQ, the proposed methods reach SROCC 0.733, which the authors describe as still far from satisfactory for adaptation to unseen images.Training on TID2013 retains the same shortcoming as training on LIVE in this respect.
  • Evaluation caveat: Reported comparisons may vary slightly because learning-based evaluations use different random splits, split ratios, and pooling procedures.The paper notes that this makes evaluation partly random and can influence comparisons between methods.

F. Convergence Evaluation

Prediction performance improves with more sampled patches and then saturates, while training and validation losses reveal different convergence behavior for the two pooling strategies.

  • Patch sampling: FR SROCC increases monotonically with the number of sampled patches toward saturation on LIVE and TID2013.Weighted average pooling improves over simple averaging only when Np > 8; WaDIQaM-FR saturates near Np ≈32.
  • Optimization convergence: Training, validation, and testing losses follow the typical iterative-gradient-descent pattern for both DIQaM-NR and WaDIQaM-NR.WaDIQaM-NR attains lower training loss but maintains that level less well during validation.

2) No-Reference Image Quality Assessment:

The NR and FR variants use patch pooling and feature-fusion choices that affect performance, while reference information can be progressively reduced from FR toward NR without retraining.

  • Feature fusion: Explicit feature differences improve FR fusion: concatenating fr, fd, and fd − fr performs better than mere concatenation of the two feature vectors.The result is reported consistently on LIVE and TID2013.
  • Reference reduction: The evaluated FR models are technically reduced-reference because they use Np = 32 reference patches rather than the full image.Reference information can be reduced further by lowering Np without retraining, and each reference-patch feature vector has 512 dimensions.
  • Reference reduction: PCA can reduce the reference-patch feature vector of a trained FR model during testing, including an extreme case with no reference information.PCA is estimated from 4000 reference patches sampled from the training set.
  • Reference reduction: A fair comparison with existing reduced-reference indices would require analyzing the interaction between feature dimensionality and the number of patches Np.The paper identifies this interaction as an unresolved comparison issue.

V. DISCUSSION & CONCLUSION

The proposed framework supports end-to-end FR and NR IQA with optional weighted patch aggregation, outperforming state-of-the-art methods while retaining limited generalization and data-related challenges.

  • The framework supports end-to-end feature learning and regression for both FR and NR IQA, with optional weighted average patch aggregation.
  • The approach outperforms state-of-the-art NR and FR IQA methods, while its generalization performance still leaves considerable room for improvement.
  • Purely data-driven IQA remains constrained by limited annotated data, although the approach suggests neural networks have substantial potential without domain knowledge.
  • Further work should optimize feature dimensionality, parameter balance, evaluation losses, and IQA-specific architectural adaptations.
  • The learned local weights reflect image structure, distortion type, and distortion distribution rather than necessarily corresponding to visual saliency.
  • The weighting architecture could learn local weights for existing IQMs and combine them with conventional regression.
Loading 1612.01697v2…