Source-linked AI summary
NIMA: Neural Image Assessment
Hossein Talebi, Peyman Milanfar
TL;DR
Image quality assessment needs human-correlated judgments despite subjective ratings, while many existing methods predict only mean opinion scores. NIMA uses a CNN to predict human rating distributions for aesthetic and technical quality, achieving strong reported correlations and supporting no-reference enhancement tuning. The approach’s current enhancement experiments require multiple evaluations, limiting real-time application.
Problem
Existing image-quality methods commonly predict only mean opinion scores despite the subjective, distributional nature of human ratings.
Method
NIMA retrains CNN architectures to predict histograms of ordered human ratings for aesthetic and technical quality using squared Earth Mover’s Distance.
Results
NIMA achieves high reported accuracy and rank correlation on aesthetic assessment, near-state-of-the-art correlation on TID2013, and cross-dataset evaluation favors models trained on AVA.
Takeaways & Limitations
The predicted quality models can guide denoising and tone-enhancement parameters to produce perceptually superior results without a reference image.
Takeaways & Limitations
The current enhancement setup requires evaluating the enhancement operator multiple times, limiting real-time application.
Abstract
from arXiv · showhide
Automatically learned quality assessment for images has recently become a hot topic due to its usefulness in a wide variety of applications such as evaluating image capture pipelines, storage techniques and sharing media. Despite the subjective nature of this problem, most existing methods only predict the mean opinion score provided by datasets such as AVA [1] and TID2013 [2]. Our approach differs from others in that we predict the distribution of human opinion scores using a convolutional neural network. Our architecture also has the advantage of being significantly simpler than other methods with comparable performance. Our proposed approach relies on the success (and retraining) of proven, state-of-the-art deep object recognition networks. Our resulting network can be used to not only score images reliably and with high correlation to human perception, but also to assist with adaptation and optimization of photo editing/enhancement algorithms in a photographic pipeline. All this is done without need for a "golden" reference image, consequently allowing for single-image, semantic- and perceptually-aware, no-reference quality assessment.
I. INTRODUCTION
Image quality assessment seeks human-correlated judgments of technical degradations and aesthetic properties, but existing approaches often predict only mean scores. NIMA instead predicts rating distributions with CNNs and applies them to no-reference assessment and enhancement tuning.
- Motivation: Image quality assessment covers technical degradations and semantic aesthetic characteristics, using full-reference or no-reference approaches.Full-reference metrics use a reference image, whereas blind methods model distortions statistically.
- Our Contributions: NIMA predicts distributions of human ratings as histograms rather than classifying images or regressing only to mean scores.The approach uses squared Earth Mover’s Distance to account for ordered score classes.
- Our Contributions: The same CNN architecture can be trained for both aesthetic and technical quality assessment using different datasets.The paper reports state-of-the-art performance for both tasks.
- Our Contributions: The predicted distribution also supports estimating score standard deviation, which is related to image non-conventionality in aesthetic assessment.This extends prediction beyond the average human rating.
- Our Contributions: NIMA quality predictions are used to tune image denoising and tone-enhancement parameters without requiring a reference image.The paper positions this as a way to produce perceptually superior enhancement results.
C. A Large-Scale Database for Aesthetic Visual Analysis (AVA) [1]
AVA is a large aesthetic-rating dataset built from photography contests, with many human ratings per image and broad contest coverage. Its rating distributions show disagreement patterns associated with extreme aesthetic scores.
- Dataset: AVA contains about 255,000 images rated for aesthetic quality by amateur photographers.Each image receives an average rating from 200 people and belongs to one of nearly 900 contest themes.
- Ratings: AVA scores range from 1 to 10, with 10 representing the highest aesthetic score.Mean ratings are concentrated around an overall mean of approximately 5.5.
- Rating Distributions: Ratings for images with very low or very high mean scores show greater standard deviations than ratings near the overall mean.The dataset reports an overall mean standard deviation of 1.43.
- Rating Distributions: Divergence of opinion is more consistent for AVA images with extreme aesthetic qualities.Ratings with means between 2 and 8 can be approximated by Gaussian distributions, while highly skewed ratings can use Gamma distributions.
- Comparison Dataset: TID2013 contains 3,000 distorted images generated from 25 clean reference images across 24 distortion types and five levels.Examples include compression artifacts, noise, blur, and color artifacts.
- Comparison Dataset: TID2013 scores come from pairwise forced-choice comparisons in which each distorted image is evaluated in nine random comparisons.Observers can view the reference image while choosing between distorted images.
- Observed Ratings: In TID2013, increasing JPEG distortion generally lowers mean quality scores, while contrast stretching can produce relatively higher perceived quality.The contrast trend differs from the more consistent pattern reported for other distortions.
E. LIVE In the Wild Image Quality Challenge Database [26]
LIVE provides mobile-device photographs with human quality ratings, while the paper contrasts its score statistics with those of AVA and TID2013. Because TID2013 and LIVE lack full rating distributions, the method approximates them for training.
- Dataset: LIVE contains 1,162 photographs captured by mobile devices, each rated by an average of 175 unique subjects.The paper scales LIVE scores to the range [1, 10].
- Rating Statistics: LIVE images rated near the overall mean show higher rating standard deviations in the joint histogram.This describes a different mean–deviation pattern from the extreme-score disagreement reported for AVA.
- Training Targets: TID2013 and LIVE provide only mean and standard deviation opinion scores, unlike AVA, which provides each image’s rating distribution.This difference affects how training targets are constructed.
- Training Targets: For TID2013 and LIVE, score distributions are approximated through maximum entropy optimization because the original distributions are unavailable.The approximation supplies score probabilities required by the proposed training method.
- Dataset Visualizations: The AVA and TID2013 figures display mean-score, standard-deviation, and joint mean–deviation histograms.These plots summarize the datasets’ rating statistics rather than individual image examples.
II. PROPOSED METHOD
The proposed method predicts human rating distributions for image quality using retrained CNN classifiers, replacing their final layers with ten quality-score outputs. It derives mean and standard deviation from the predicted distribution and uses image resizing and augmentation during training.
- CNN architecture: The framework explores VGG16, Inception-v2, and MobileNet classifier architectures for quality assessment.These architectures provide alternative CNN backbones for the proposed predictor.
- CNN architecture: The baseline CNN’s final layer is replaced with a ten-neuron fully connected layer followed by soft-max activations.Baseline weights are initialized from ImageNet training, while end-to-end quality-assessment training updates the network.
- Training procedure: Training rescales images to 256 × 256, randomly crops 224 × 224 regions, and horizontally flips crops.The resizing and cropping strategy is intended to lessen over-fitting, whereas random crops without rescaling produced unconvincing results because composition changed.
- Rating-distribution prediction: The predicted distribution uses ten ordered score buckets, with AVA scores spanning 1–10 and TID2013 scores spanning 0–9.The distribution assigns probabilities to ordered score buckets whose probabilities sum to one.
- Rating statistics: Mean and standard deviation are computed from the predicted score probabilities to summarize image-quality ratings.The mean is the probability-weighted score sum, while the standard deviation measures score dispersion.
- Rating-distribution prediction: The method estimates a probability mass function of human ratings rather than only a mean score.Each image is paired with ground-truth user ratings, and the model learns an estimated distribution.
A. Loss Function
The loss function accounts for the ordered nature of quality-score classes by comparing the cumulative distributions of ground-truth and predicted ratings with squared Earth Mover’s Distance.
- Motivation: Cross-entropy ignores relationships between ordered score buckets, whereas EMD-based losses penalize misclassifications according to class distance.Quality ratings satisfy an intrinsic ordering, making distance between score classes relevant to training.
- Earth Mover’s Distance: Earth Mover’s Distance measures the minimum cost of moving one rating distribution into another.The distance uses the r-norm separation between ordered score classes.
- Earth Mover’s Distance: The normalized EMD is expressed through differences between the ground-truth and estimated cumulative distribution functions.The closed-form expression sums these cumulative-distribution differences across score buckets.
- Normalization: The closed-form EMD solution requires the ground-truth and predicted distributions to have equal total mass.The model’s soft-max output guarantees that the predicted probabilities sum to one.
- Normalization: The training framework sets r = 2 to penalize Euclidean distance between the cumulative distributions and facilitate gradient-descent optimization.The chosen norm defines the class-distance penalty used by the loss.
III. EXPERIMENTAL RESULTS
The experiments train separate aesthetic and technical-quality models on AVA, TID2013, and LIVE, using held-out test sets for comparison with existing methods.
- Experimental setup: Two separate models are trained for aesthetics and technical quality assessment on AVA, TID2013, and LIVE.The experiments evaluate the proposed models separately for the two assessment tasks.
- Experimental setup: Each dataset is split so that 20% of its data is reserved for testing.Performance is discussed on the test sets and compared with existing methods.
- Implementation: The CNN implementation initializes baseline weights from ImageNet and randomly initializes the final fully connected layer.Training uses momentum values of 0.9, dropout of 0.75 on the last baseline layer, and learning rates of 3 × 10^-7 and 3 × 10^-6 for baseline and final layers, respectively.
A. Performance Comparisons
NIMA predicts aesthetic and technical quality ratings while modeling full opinion-score distributions. On AVA and TID2013, its models achieve competitive correlations and distributional agreement, with AVA mean-score distributions closely matched but standard deviations harder to predict.
- Distributional prediction: NIMA predicts rating distributions rather than only mean opinion scores, using histogram-based outputs for both aesthetic and technical assessment.The TID2013 comparison evaluates correlations with mean and standard-deviation scores, plus EMD between predicted and ground-truth distributions.
- TID2013 technical assessment: NIMA(VGG16) achieves TID2013 correlations close to state-of-the-art results while using one rescaled image instead of multiple image crops.Most compared methods regress to mean opinion scores, whereas NIMA also predicts the rating distribution.
- AVA distribution prediction: The AVA ground-truth mean-score distribution is closely predicted by NIMA(Inception-v2), whereas the standard-deviation distribution remains more challenging.The paper attributes variation in standard deviations partly to unconventional subject matter or style.
B. Cross Dataset Evaluation
Cross-dataset evaluation shows that models trained on AVA transfer most effectively across the tested datasets. Training on LIVE transfers poorly to AVA, while AVA’s much larger sample count supports training without significant overfitting.
- Cross-dataset performance: Training on AVA gives the best average cross-dataset performance among the evaluated training datasets.For AVA-trained and LIVE-tested models, linear and rank correlations are 0.552 and 0.543, respectively.
- Transfer asymmetry: 0.238 and 0.2 linear and rank correlations result when training on LIVE and testing on AVA.The paper contrasts these values with the stronger transfer from AVA to LIVE.
- Dataset scale: AVA contains roughly 250 times more examples than LIVE, allowing NIMA models to be trained without significant overfitting.The paper presents dataset size as an explanation for AVA-trained models’ stronger generalization to other test examples.
C. Photo Ranking
NIMA scores support qualitative ranking of aesthetic and technically distorted images and can guide perceptual tuning of enhancement operators. The examples show sensitivity to tone, contrast, composition, framing, color, texture, and smoothing strength.
- Photo ranking: AVA-trained models qualitatively rank photos using predicted mean scores that reflect tone, contrast, composition, framing, and color palette.The landscape and sky examples compare predicted NIMA scores with ground-truth AVA scores.
- Photo ranking: The TID2013-trained model accurately ranks test images containing varied distortion types and levels.Figure 12 compares predicted and ground-truth scores for the TID2013 test images.
- Enhancement tuning: NIMA scores can serve as a prior for perceptually tuning image enhancement operators without requiring a reference image.The paper applies the approach to tone enhancement and denoising parameter selection.
- Tone enhancement: The aesthetic model guides a multi-layer Laplacian filter toward near-optimal settings across 726 combinations of detail, shadow, and brightness parameters.The model tends to prefer high-contrast images with boosted details, consistent with AVA ground-truth results.
- Image denoising: For denoising, maximum quality scores select different spatial-smoothing parameters across images and tend to preserve texture rather than over-smooth details.The experiment adds Gaussian noise with standard deviation 30 and averages scores over 50 random crops.
E. Computational Costs
NIMA model complexity varies substantially across architectures. MobileNet provides the lightest and fastest inference, with a slight performance reduction relative to the heavier models.
- Inference efficiency: NIMA(MobileNet) is significantly lighter and faster than the other evaluated models for one pass on a 224 × 224 × 3 image.The measurements use TensorFlow on specified CPU and GPU hardware.
- Accuracy-efficiency trade-off: The MobileNet efficiency advantage comes at the expense of a slight performance drop.The paper refers to performance comparisons reported in Tables I and II.
IV. CONCLUSION
The paper presents CNN-based assessment models that predict rating distributions for aesthetic and technical quality, and uses them to guide image enhancement. However, the current enhancement setup requires repeated operator evaluations, limiting real-time use.
- CNN-based models are trained on aesthetic and pixel-level quality datasets to predict rating distributions rather than only mean scores.The approach targets both high-level aesthetics and low-level technical quality.
- The trained models guide denoising and tone-enhancement parameters toward perceptually superior results.
- Repeated evaluation of the enhancement operator limits the method's real-time application.
- For enhancement operators with well-defined derivatives, using NIMA as the loss function may be more efficient.