Source-linked AI summary
Effective Aesthetics Prediction with Multi-level Spatially Pooled Features
Vlad Hosu, Bastian Goldlucke, Dietmar Saupe
TL;DR
AQA models must handle subjective judgments and AVA’s variable, high-resolution images without discarding information through cropping or rescaling. The paper addresses this with staged training on MLSP features extracted from pretrained Inception networks, achieving a higher SRCC than prior work while reducing training cost.
Problem
Variable-resolution, high-resolution AVA images force many AQA methods to crop or rescale inputs, potentially discarding information and hindering efficient training.
Method
The method extracts and stores MLSP features from pretrained Inception networks, then trains a shallow CNN head separately to predict image MOS.
Results
0.756 SRCC versus the existing reported 0.612 substantially improves correlation performance on AVA.
Takeaways & Limitations
Using original-resolution information with pretrained MLSP features enables stronger full-score-range AQA performance while requiring faster training and less GPU memory.
Takeaways & Limitations
The model can overemphasize technical quality when technically flawed images receive high ratings because their subject matter remains interesting.
Abstract
from arXiv · showhide
We propose an effective deep learning approach to aesthetics quality assessment that relies on a new type of pre-trained features, and apply it to the AVA data set, the currently largest aesthetics database. While previous approaches miss some of the information in the original images, due to taking small crops, down-scaling or warping the originals during training, we propose the first method that efficiently supports full resolution images as an input, and can be trained on variable input sizes. This allows us to significantly improve upon the state of the art, increasing the Spearman rank-order correlation coefficient (SRCC) of ground-truth mean opinion scores (MOS) from the existing best reported of 0.612 to 0.756. To achieve this performance, we extract multi-level spatially pooled (MLSP) features from all convolutional blocks of a pre-trained InceptionResNet-v2 network, and train a custom shallow Convolutional Neural Network (CNN) architecture on these new features.
1. Introduction
AQA on AVA is difficult because human judgments depend on diverse content, style, and technical factors, while variable high-resolution inputs hinder efficient deep-learning training. The proposed staged MLSP approach preserves original-image information, reduces training costs, and improves correlation performance.
- AQA is difficult because aesthetic judgments reflect photography rules, affective preferences, content, and style.
- AVA contains about 250 thousand images, each rated by an average of 210 users.
- Variable resolutions and high-resolution AVA images make fixed-size batching inefficient and motivate rescaling or cropping that loses information.
- The framework extracts and stores MLSP features from variable-sized high-resolution images, then trains aesthetic assessment separately on those features.
- 0.756 SRCC versus the previous 0.612 improves state-of-the-art AQA correlation performance on AVA.
- MLSP features are compact and resolution-independent, while shallow head networks reduce GPU memory requirements.
- Wide MLSP training takes about 10 minutes per AVA epoch, narrow MLSP training less than a minute, and fine-tuning averages about 4.5 hours on one GPU.
2. Related work
Prior AVA methods used multi-column networks, crops, rescaling, ranking losses, or multi-level features to address aesthetic assessment. This work builds on multi-level perceptual features while studying spatial pooling across Inception architectures for AQA.
- AVA AQA methods face two linked constraints: more than 25 thousand resolutions and images up to 800 × 800 pixels.
- Multi-column methods combine global rescaled views with local or saliency-selected crops, but still train on transformed image regions.
- Talebi et al. replaced classification with a regression head predicting rating distributions while training on 224 × 224 crops from rescaled images.
- Ranking-based methods learn aesthetic ordering with pairwise or triplet-style objectives using low-resolution inputs.
- Multi-level features capture both low-level degradations and higher-level content-related factors in aesthetic quality.
- Prior perceptual methods used pooled ImageNet features, whereas this work studies spatially pooled multi-level features from Inception-v3 and InceptionResNet-v2.
3. Network architecture and training
The method extracts multi-level spatially pooled features from Inception networks and learns MOS predictions with shallow regression heads. It supports narrow and wide feature representations through distinct architectures and uses staged feature storage for efficient training.
- MLSP feature extraction: MLSP features resize activation outputs to fixed 1 × 1 or 5 × 5 spatial resolutions before concatenating them across network levels.Narrow features use global average pooling, while wide features use 5 × 5 spatial average pooling.
- Training pipeline: The approach stores augmented features from variable-resolution AVA images, using proportional crops and horizontal flips.Four crop positions and two mirroring states produce eight stored feature sets per image.
- Training strategy: The staged design separates feature extraction from head training, reducing the memory burden of end-to-end learning at high image resolutions.The paper reports that end-to-end training is too demanding at high resolutions, whereas shallow heads are more efficient.
- Regression heads: A custom 3FC head predicts MOS with mean squared error, using dropout and batch normalization during training.The head is shared across several proposed architectures.
- Narrow MLSP architectures: Single-1FC and Single-3FC learn from concatenated narrow MLSP features, while Multi-3FC predicts from separate feature blocks and combines the outputs linearly.Multi-3FC uses an independent fully connected head for each Inception-module output.
- Wide MLSP architecture: Pool-3FC reduces wide MLSP dimensionality with a modified Inception module, pools the concatenated features, and applies a 3FC head.The module uses 1 × 1 and 3 × 3 convolutions plus average pooling, with 1024 kernels per column selected as best in experiments.
4. Results
Experiments compare feature-learning architectures, input treatments, and network depths on AVA. The best results come from wide MLSP features extracted from original images, while broader feature-level coverage improves performance and correlation with MOS.
- Architecture comparison: Single-3FC is selected for later feature-extraction comparisons because it is the best and simpler than Multi-3FC.This selection follows experiments on rescaled-image features.
- Feature depth: Using all 11 Inception-v3 levels benefits performance: the last 8 layers have the strongest effect, while the first 3 still provide additional benefit.The analysis increases the number of included levels from the final content-oriented blocks toward earlier low-level blocks.
- Backbone comparison: InceptionResNet-v2 narrow features perform about as well as Inception-v3 spatially pooled features, with 16,928 versus 10,048 extracted kernels.The deeper network contributes more kernels across more levels.
- Original-image features: The best performance is obtained by Pool-3FC trained on wide MLSP features from the AVA original images.Table 3 compares features extracted from original images using Inception-v3 and InceptionResNet-v2.
- Evaluation metrics: Reported SRCC can diverge from binary accuracy because thresholding MOS at 5 evaluates only low-versus-high quality classification.The paper characterizes this threshold as an arbitrary and limiting choice for assessing the full score range.
- Comparison to previous methods: 0.756 SRCC is achieved by Pool-3FC at 0.8161 accuracy, compared with 0.612 SRCC at 0.8151 accuracy for Talebi et al.The authors report the higher SRCC-to-accuracy ratio as better generalization across the score range.
5. Discussion
The discussion argues that correlation metrics better reflect aesthetics ranking across the score range than binary accuracy, while illustrating both typical successes and failure cases.
- Performance evaluation: Correlation metrics such as SRCC and PLCC better represent image-quality ranking across the full score range than binary classification accuracy.They avoid arbitrary thresholds and are not subject to the same class-imbalance issue described for binary accuracy.
- Performance evaluation: Binary classification accuracy is threshold-dependent because images above a MOS of 5 are labeled high quality, despite AVA’s average MOS being 5.5.Changing the binary split changes accuracy and makes the measure unrepresentative of the full score range.
- Performance evaluation: About 70% of random test images are high quality and 30% are low quality, making roughly 70% the minimum accuracy of an all-high-quality classifier.Reported binary accuracy does not account for this class imbalance.
- Performance evaluation: The paper reports binary accuracy alongside correlation metrics, achieving results comparable to the state of the art.Correlation metrics are the optimization target, while binary accuracy remains a commonly reported measure.
- Failure cases: The model’s largest errors occur on high-MOS images with technical faults but interesting subject matter, suggesting technical quality is overemphasized in unusual cases.The reviewed examples were selected from the highest-MOS test images by absolute prediction error.
6. Conclusions
The paper presents MLSP features from pretrained ImageNet CNNs as a general transfer-learning approach, demonstrated on AQA with strong correlation performance and efficient high-resolution training.
- Conclusions: The approach substantially outperforms the state of the art on correlation metrics while matching it on binary classification accuracy.AQA is presented as a challenging perceptual problem, and correlation metrics are emphasized as the more discriminating evaluation.
- Conclusions: The method uses features extracted from pretrained ImageNet CNNs, with MLSP features introduced as a new type of perceptual feature.The approach is general but is showcased on aesthetics quality assessment.
- Conclusions: Pretrained networks reduce training speed and GPU-memory demands, while spatially wide and multi-level features provide further performance gains.Large feature sets can be stored on disk, offsetting memory limitations.
- Conclusions: Using original-resolution information yields 0.756 SRCC versus the existing reported 0.612, while binary accuracy remains 81.7%.The paper attributes the improvement partly to avoiding rescaling and cropping effects on technical quality and composition.
- Conclusions: MLSP features from InceptionResNet-v2 consistently outperform those from Inception-v3.Both architectures were trained on a 1,000-class ImageNet subset.
- Limitations and future work: Deeper architectures and models trained on larger datasets may provide further performance gains with pretrained features.This is presented as a direction suggested by the comparison between InceptionResNet-v2 and Inception-v3.