PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptual Quality Prediction on Authentically Distorted Images Using a Bag of Features Approach

Deepti Ghadiyaram, Alan C. Bovik

arXiv:1609.04757v1cs.CV

TL;DR

Blind image quality models lose discriminative power on authentically distorted images, motivating models for complex real-world distortion mixtures. The paper presents a first effort toward such models, and its success supports further development for related applications.

  • Problem

    Generalized statistical models lose power to discriminate high-quality from distorted images when distortions are authentic, motivating blind IQA models for real-world images.

  • Method

    The paper designs a blind IQA model for images corrupted by complex mixtures of authentic distortions and uses diverse statistical image features.

  • Results

    The first effort is successful enough to encourage exploration of analogous blind video quality assessment models and practical real-world adaptation.

  • Takeaways & Limitations

    Authentic distortions require capturing more diverse statistical image features than generalized models provide.

  • Takeaways & Limitations

    Scope remains image quality assessment; analogous blind video models and practical real-world adaptation are identified as future work.

Abstract

from arXiv · show

Current top-performing blind perceptual image quality prediction models are generally trained on legacy databases of human quality opinion scores on synthetically distorted images. Therefore they learn image features that effectively predict human visual quality judgments of inauthentic, and usually isolated (single) distortions. However, real-world images usually contain complex, composite mixtures of multiple distortions. We study the perceptually relevant natural scene statistics of such authentically distorted images, in different color spaces and transform domains. We propose a bag of feature-maps approach which avoids assumptions about the type of distortion(s) contained in an image, focusing instead on capturing consistencies, or departures therefrom, of the statistics of real world images. Using a large database of authentically distorted images, human opinions of them, and bags of features computed on them, we train a regressor to conduct image quality prediction. We demonstrate the competence of the features towards improving automatic perceptual quality prediction by testing a learned algorithm using them on a benchmark legacy database as well as on a newly introduced distortion-realistic resource called the LIVE In the Wild Image Quality Challenge Database. We extensively evaluate the perceptual quality prediction model and algorithm and show that it is able to achieve good quality prediction power that is better than other leading models.

Introduction

No-reference image quality assessment aims to predict human-perceived quality without a reference image, but legacy databases mainly use isolated synthetic distortions. This paper targets authentic images containing complex, interacting distortions by developing a more robust blind IQA approach and evaluating it on both legacy and real-world databases.

  • No-reference IQA predicts an image’s visual quality from the image alone, without additional information or a reference.
  • Authentic vs. Inauthentic Distortions: Legacy IQA databases primarily contain pristine images altered by controlled, synthetic single distortions, unlike real-world camera distortions.
  • Authentic vs. Inauthentic Distortions: Authentically captured images commonly contain unpredictable mixtures of interacting distortions that cannot be accurately described by simple or separable distortion models.
  • Authentic vs. Inauthentic Distortions: The paper seeks richer quality-aware features and a robust generic predictor that performs on both legacy synthetic-distortion databases and real-world images.
  • Authentic vs. Inauthentic Distortions: The LIVE In the Wild Image Quality Challenge Database contains real scenes captured by diverse mobile devices and affected by unknown mixtures of multiple authentic distortions.
  • Authentic vs. Inauthentic Distortions: Generalized statistical models and their quality-aware features lose discriminative power when applied to authentically distorted images.

Motivation behind using Natural Scene Statistics

Natural scene statistics (NSS) models capture statistical regularities of natural images and quantify departures associated with distortion. These features distinguish many isolated synthetic distortions, but authentic mixtures can preserve near-pristine statistics, motivating richer feature representations.

  • NSS foundations: NSS models capture statistical regularities in normalized natural-image coefficients and quantify their perturbations for blind image quality assessment.Bandpass debiasing and divisive normalization reduce spatial dependencies, after which normalized luminance coefficients can be modeled statistically.
  • NSS foundations: The normalized coefficients of 29 diverse pristine LIVE images follow an approximately Gaussian distribution, with fitted GGD shape parameter α = 2.15.The Gaussian-like regularity persists across substantially different image content.
  • Synthetic distortions: Single synthetic distortions alter normalized-coefficient statistics characteristically, producing GGD shape parameters of 1.12 for JP2K compression and 3.02 for white noise.These deviations from Gaussianity have been established across distorted-image datasets.
  • Authentic distortions: Authentically distorted images can contain mixtures of blur, sensor noise, illumination changes, and other nonlinear distortions that are difficult to model individually.Such composite distortions differ from the isolated processes used to generate legacy benchmark images.
  • Authentic distortions: A mixed authentic distortion can retain near-Gaussian normalized-coefficient statistics, with α = 2.12 despite multiple distortions.This behavior can make conventional NSS features unreliable for judging perceptual quality in authentic images.
  • Why richer features are needed: On the legacy LIVE database, α separates many high- and low-quality images, whereas heavily distorted LIVE Challenge images can still have α values near 2.0.The contrast motivates modeling authentic distortions in a higher-dimensional space of perceptually relevant features.
  • Why richer features are needed: Prior state-of-the-art no-reference IQA algorithms perform poorly on LIVE Challenge images, while the feature-driven model yields good prediction performance.The results support the need for quality-aware NSS features predictive of authentic distortion severity.
  • Why richer features are needed: The proposed response is a bag of feature maps spanning complementary luminance, chrominance, color-space, and transform-domain statistics.It adds previously unused detail and chromatic features while avoiding assumptions about which distortion types are present.

Related Work

Existing blind IQA models largely rely on features learned from synthetically and usually singly distorted images, limiting their direct applicability to authentic, nonhomogeneous distortions. The paper motivates a predictor designed for complex real-world impairments.

  • Distortion-specific models assume a particular impairment, whereas authentic images contain different distortion types affecting different regions with varied severities.
  • NSS-based blind IQA models avoid prior distortion assumptions using spatial, wavelet, DCT, or other statistical features, but remain evaluated mainly on legacy data.
  • CNN patch-labeling and score-pooling strategies developed for homogeneous benchmark distortions cannot be directly extended to LIVE Challenge images.
  • Most blind IQA models are trained on synthetic, usually singly distorted benchmark images.
  • Top-performing existing models perform poorly on the LIVE Challenge Database.

LIVE In the Wild Image Quality Challenge Database

The LIVE In the Wild Image Quality Challenge Database contains naturally captured mobile-camera images with diverse, mixed impairments and extensive human quality ratings. Its subjective scores show high reliability across rating splits and study conditions.

  • The database contains 1,163 images with randomly occurring distortions and genuine capture artifacts from smartphones and tablets.
  • Images include diverse distortion types, mixtures, and severities introduced during capture, processing, and storage rather than artificially added.
  • Authentically distorted images commonly contain multiple impairments that resist categorization into discrete distortion types.
  • More than 350,000 quality ratings from over 8,100 unique subjects provided approximately 175 ratings per image.
  • Subject-Consistency Analysis: 0.9896 average Pearson correlation between MOS values from repeated split-half ratings demonstrates high subject consistency.
  • Subject-Consistency Analysis: 0.9851 correlation between crowdsourced and traditional-study scores on five gold-standard images supports the reliability of the test framework.

Feature Maps Based Image Quality

Authentic distortion mixtures produce statistical variations that are difficult for conventional distortion-sensitive models to predict. The proposed solution represents images through feature maps spanning multiple perceptual spaces and extracts their scene statistics.

  • Complex authentic distortion mixtures create large, hard-to-predict statistical variations relative to synthetically distorted images.
  • The model uses multiple perceptual image representations to distinguish different aspects of perceived quality loss.
  • For each image, it constructs feature maps in multiple color spaces and transform domains, then extracts individual and collective scene statistics.
  • The extracted statistics are used as quality-aware features for training a quality predictor.

Statistical Modeling of Normalized Coefficients

The method models divisively normalized coefficients from perceptually relevant feature maps using flexible parametric distributions and statistics across two scales. These features quantify departures from natural-image coefficient behavior for quality prediction.

  • Divisive Normalization: For luminance map L, normalized luminance coefficients are computed from local mean and variance using the NLC operation.
  • Divisive Normalization: Divisive normalization reduces statistical dependencies, models nonlinear neighboring responses, and accounts for contrast masking relevant to distorted-image perception.
  • Most feature maps undergo divisive normalization before their coefficients are statistically modeled and converted into quality-predictor features.
  • Statistical Distributions: GGD parameters model coefficient-distribution shape and variance, while AGGD extends GGD to skewed normalized distributions.
  • Statistical Distributions: Distortion-related non-Gaussianity is captured through GGD or AGGD parameters together with kurtosis, skewness, and goodness-of-fit statistics.
  • Multiscale Features: Features are extracted at the original and half-resolution scales because images and distortions have multiscale structure; additional scales were not beneficial on the large dataset.

Feature Maps

The paper constructs a heterogeneous bag of perceptually motivated statistical features from luminance, color, and transform-domain feature maps. These maps capture natural-scene regularities and their departures under distortion without assuming a specific distortion type.

  • Feature-map design: The bag of features combines perceptually motivated natural-scene statistics from diverse color-space representations and image transforms.The features are grounded in perceptual-processing models, natural picture statistics, or parametric and sample statistics of relevant image distributions.
  • Luminance maps: Luminance processing begins with a normalized luminance coefficient map modeled using a generalized Gaussian distribution and sample statistics.The normalized luminance map applies divisive normalization, while the fitted model parameters and additional statistics become features.
  • Luminance maps: Neighboring paired products estimate local directional correlations, with 24 parameters per scale producing 48 additional features across two scales.The products use four directional neighbors, and their distributions are modeled with an AGGD alongside kurtosis and skewness.
  • Luminance maps: Sigma-map features measure structural anomalies through kurtosis, skewness, and mean statistics at two scales.The sigma field is described as statistically regular in pristine images but disturbed by distortion; this feature map had not previously been used for visual modeling.
  • Luminance maps: DoG-of-sigma features capture deviations in multiscale center-surround statistics, including estimated shape, standard deviation, skewness, and kurtosis.A further sigma field is extracted from DoGsigma, whose normalized coefficients also exhibit statistical regularities that distortions disrupt.
  • Transform and color maps: Additional maps include the luminance Laplacian, complex steerable-pyramid wavelets, and CIELAB chroma representations.The wavelet features are strongly multiscale and multi-orientation, while C-DIIVINE-derived features use three scales and six orientations.

Advantages of the proposed Feature Maps

The proposed feature maps address cases where standard normalized luminance statistics fail to distinguish authentic distortions from pristine content. Yellow-channel and DoGsigma distributions provide more useful separation across singly and authentically distorted images.

  • Distortion-realistic comparisons: Authentically distorted images may contain noise, blur, and compression artifacts, unlike the isolated JPEG2000 and additive-white-noise distortions applied to a pristine image.The comparison uses images from both the legacy LIVE database and the LIVE In the Wild Image Quality Challenge Database.
  • Distortion-realistic comparisons: Normalized luminance histograms can make an authentically distorted image resemble a pristine image, limiting their usefulness for that case.The example contrasts the standard mean-subtraction and divisive-normalization operation with the proposed yellow-channel and DoGsigma maps.
  • Feature-map advantages: Yellow-channel and DoGsigma histograms distinguish pristine images from both singly and authentically distorted images.The distributions of the proposed feature maps vary between pristine and authentically distorted images in the illustrated examples.
  • Feature-map advantages: The authors report observing the usefulness of all proposed feature maps on the large, comprehensive LIVE Challenge database.This supports evaluating a broad collection of feature maps rather than relying on one diagnostic representation.
  • Feature-map advantages: A bag-of-features strategy is used because predicting the perceptual severity of authentic distortions is difficult and distortion types may be diverse.The approach aims to capture distortion-perception information distributed across multiple color representations.

Regression

The extracted feature statistics and real-valued mean opinion scores are used to train a nonlinear support vector regressor. The trained model maps test-image feature vectors to predicted quality scores, with parameters selected by cross-validation.

  • Training: Perceptually relevant feature statistics and corresponding real-valued MOS values form the training data for a quality regressor.The features summarize the proposed feature maps, while MOS supplies the target quality label.
  • Regression model: The database size is described as insufficient to motivate deep learning methods, supporting the use of an SVM classifier and regressor.The passage also notes SVM’s flexibility with high-dimensional data and diverse information sources.
  • Regression model: The regression model uses an SVR with a radial basis kernel to learn a nonlinear mapping from image features to a single quality score.SVR is presented as a common tool in image- and video-quality assessment for mapping high-dimensional feature vectors to quality scores.
  • Regression model: Cross-validation determines the optimal learner parameters, with fitting error on validation data as the selection objective.The stated aim is to minimize the learner’s validation fitting error.

Experiments

The experiments evaluate FRIQUEE and its feature-space variants against established no-reference IQA models on authentic and synthetic distortions. FRIQUEE-ALL performs significantly better on the LIVE Challenge Database and remains competitive across legacy benchmarks, while models trained only on synthetic distortions degrade on authentic images.

  • Model and evaluation setup: FRIQUEE combines diverse perceptually relevant statistical features across multiple domains to train a blind image-quality regressor.FRIQUEE-ALL uses all feature maps together with HSI and yellow-channel maps, while Luma, Chroma, and LMS variants use corresponding subsets.
  • Model and evaluation setup: The model uses an RBF-kernel SVR trained on 80% of images and tested on non-overlapping 20% splits, repeated across 50 random trials.Median PLCC and SROCC, along with mean outlier ratio, are reported across the iterations.
  • Authentically distorted images: FRIQUEE-ALL is significantly better than current top-performing no-reference IQA models on unseen LIVE Challenge Database test data.The comparison uses identical train/test settings and repeated cross-validation trials.
  • Authentically distorted images: FRIQUEE’s perceptually driven natural-scene-statistics features outperform features used by top-performing models such as C-DIIVINE, BRISQUE, and DIIVINE.The authors attribute the comparison to median correlation scores and evaluate statistical significance with paired t-tests at 95% confidence.
  • Authentically distorted images: Most algorithms perform poorly on the LIVE Challenge Database, making it a difficult test of model generalizability to authentic distortions.The database uses MOS values from a crowdsourced study treated as ground-truth quality scores.
  • Cross-database evaluation: All evaluated models lose performance when trained only on synthetic, inauthentic distortions, whereas FRIQUEE competes favorably on both authentic and legacy synthetic databases.On the legacy LIVE Database, FRIQUEE reaches or exceeds the performance saturation level achieved by tested prior models.

Conclusions and Future Work

The paper presents a first effort toward blind IQA for images containing complex mixtures of authentic distortions. Its success motivates future work on blind video quality assessment and practical perceptual optimization of camera capture.

  • Conclusions: The work is a first effort toward blind IQA models that predict the quality of images corrupted by complex mixtures of authentic distortions.The conclusion frames the approach as an initial step rather than a completed solution to all real-world IQA settings.
  • Future work: The reported success motivates exploring analogous blind video quality models based on space-time natural video statistics.The proposed future direction extends the natural-statistics approach from images to video.
  • Future work: The authors also propose adapting the model to real-world applications such as perceptual optimization of digital-camera capture.This is presented as a practical future application rather than an evaluated result in the paper.
Loading 1609.04757v1…