Source-linked AI summary
Image Aesthetic Assessment: An Experimental Survey
Yubin Deng, Chen Change Loy, Xiaoou Tang
TL;DR
Image aesthetic assessment seeks to computationally distinguish high- and low-quality photographs despite subjective judgments, complex photographic factors, and limited annotated data. This survey systematically reviews representations, datasets, metrics, and deep-learning strategies, and evaluates baseline settings for aesthetic scoring and manipulation. It reports competitive baselines, including a 73.59% balanced-accuracy CNN configuration and faster CNN-based cropping than a prior method.
Problem
Image aesthetic assessment must model subjective photographic judgments across diverse genres and techniques while obtaining sufficient human-annotated data for robust evaluation.
Method
The survey organizes methods by visual representations and evaluation criteria, reviews their contributions, and systematically evaluates deep-learning settings including data preparation, fine-tuning, and multi-column architectures.
Results
The evaluated baselines are competitive with prior state-of-the-art methods, including 73.59% balanced accuracy for a triplet-pretraining plus multi-task one-column CNN and approximately 0.2 seconds GPU cropping.
Takeaways & Limitations
The survey provides a reference for image-aesthetic research and identifies richer, more diverse annotations as a direction for future models and meaningful statistics.
Takeaways & Limitations
Individual users may hold different aesthetic preferences, making adaptation from generic aesthetics to user-specific taste a challenging problem.
Abstract
from arXiv · showhide
This survey aims at reviewing recent computer vision techniques used in the assessment of image aesthetic quality. Image aesthetic assessment aims at computationally distinguishing high-quality photos from low-quality ones based on photographic rules, typically in the form of binary classification or quality scoring. A variety of approaches has been proposed in the literature trying to solve this challenging problem. In this survey, we present a systematic listing of the reviewed approaches based on visual feature types (hand-crafted features and deep features) and evaluation criteria (dataset characteristics and evaluation metrics). Main contributions and novelties of the reviewed approaches are highlighted and discussed. In addition, following the emergence of deep learning techniques, we systematically evaluate recent deep learning settings that are useful for developing a robust deep model for aesthetic scoring. Experiments are conducted using simple yet solid baselines that are competitive with the current state-of-the-arts. Moreover, we discuss the possibility of manipulating the aesthetics of images through computational approaches. We hope that our survey could serve as a comprehensive reference source for future research on the study of image aesthetic assessment.
1 INTRODUCTION
Image aesthetic assessment computationally models human judgments to distinguish high- from low-quality photographs, motivated by image-search needs and challenged by complex rules, genre differences, techniques, and annotation demands. The survey organizes prior methods, reviews deep-learning settings, and identifies directions for future research.
- Motivation: Image aesthetic assessment distinguishes high-quality photographs from low-quality ones using photographic rules and computational classification or regression.Human judgments can depend on lighting, contrast, and composition.
- Motivation: Growing online visual data increases demand for systems that return professional photographs rather than random snapshots in image-search applications.The paper illustrates this with searches for colorful, pleasing mountain scenery instead of gray or blurry snapshots.
- Challenges: The task is difficult because photographic rules are intertwined, aesthetics vary across genres and capture techniques, and robust testing requires substantial human-annotated data.The paper lists genres such as scenery and night scenes, techniques such as HDR and depth-of-field, and data scarcity as challenges.
- Prior approaches: Earlier rule-based systems used low-level features, whereas later work found data-driven approaches more effective as training data grew from hundreds to millions of images.Transfer learning from data-rich source tasks to smaller target tasks is also reported as feasible.
- Survey scope: The survey categorizes methods by handcrafted or learned representations and by classifiers or regressors, addressing a gap left by a 2011 review that excluded deep-learning methods.It also distinguishes image aesthetics from objective image-quality metrics designed for reference-based restoration.
- Contributions: The authors systematically evaluate data preparation, fine-tuning, and multi-column architectures for robust deep aesthetic scoring, including strategies for distribution bias and false positives.Their CNN baselines are assessed with balanced accuracy.
- Contributions: The paper reviews datasets, handcrafted and deep features, evaluation criteria, and aesthetic manipulation, including aesthetic-based image cropping.It reports the survey’s organization across these topics and discusses future directions.
2 BACKGROUND
Deep neural networks learn data-driven feature representations through layered forward propagation, nonlinear activation, loss evaluation, and gradient-based weight updates.
- Deep neural networks: Deep neural networks learn feature representations through stacked parameterized layers that provide increasingly abstract representations.Common layers include convolutional and fully connected layers, inspired by hierarchy in the human visual cortex.
- Forward propagation: A layer’s forward operation maps input x to output y, while nonlinear functions such as sigmoid or ReLU produce activation outputs.For ReLU, the activation is z = max(0, y).
- Learning: Training defines a loss between network output z and objective t, then uses backpropagation to compute weight updates that reduce the loss.The update is expressed as W ← W − η∆W.
- Learning: Batch stochastic gradient descent averages gradients over training examples to support optimization of increasingly deep networks.The paper points readers to a broader overview of deep-learning methodologies.
Reference PSNR / SSIM / VIF
PSNR, SSIM, and VIF quantify objective image fidelity against a high-quality reference, but they are not designed to measure subjective human aesthetic quality.
- Objective quality metrics: PSNR, SSIM, and VIF are higher-is-better quality metrics typically measured against a reference groundtruth image.They are useful indicators in image-restoration applications.
- Deep-learning context: The surveyed deep-learning background connects supervised training to loss minimization and gradient-based optimization rather than to the reference-image metrics used in restoration.Batch procedures compute more stable gradients from training examples.
- Objective quality metrics: These reference-based metrics can produce misleading aesthetic assessments because objective fidelity does not reflect human-perceived aesthetic values.The paper contrasts restoration examples with building images whose metric measurements fail to capture perceived aesthetics.
3 A TYPICAL PIPELINE
A typical aesthetic-assessment pipeline learns a representation from labeled images and then applies a decision component for classification or regression.
- Pipeline formulation: Supervised pipelines learn f : g(X) → Y from image-label pairs, with binary labels for classification or continuous scores for regression.The feature representation is denoted by g(x_i).
- Pipeline components: The pipeline has two main components: feature extraction for aesthetic representation and a decision component for classification or regression.Feature extraction aims to describe photographic or artistic aspects robustly enough to distinguish image qualities.
- Representations and decisions: Handcrafted and deep features are the majority feature types, while Naive Bayes, SVM, boosting, and deep classifiers support binary classification.Support vector regressors are used for ranking or scoring aesthetic quality.
4 DATASETS
Image aesthetic assessment uses datasets ranging from small private collections to large, score-annotated benchmarks. CUHK-PQ provides binary labels across scene categories, while AVA supplies dense ratings but includes ambiguous mid-score images.
- Earlier private datasets typically contain a few hundred to a few thousand images with binary labels or aesthetic scores.
- Photo.Net contains 20,278 images, each receiving at least 10 ratings on a 0–7 aesthetic scale.
- CUHK-PQ contains 17,690 binary-labeled images across seven scene categories, with roughly a 1:3 positive-to-negative ratio.
- AVA contains approximately 250,000 images, each receiving 78–549 votes on a 1–10 scale whose average commonly supplies the ground-truth label.
- AVA includes ambiguous examples near mean score 5, whereas binary classification selects positives using a threshold above 5+σ.
- IAD contains 1.5 million scored images, using threshold-selected positives for training and evaluation on AVA.
CRAFTED FEATURES
Hand-crafted aesthetic assessment represents images with global, regional, compositional, and task-specific features before classification or regression. These methods encode photographic rules and adapt features to particular image categories or use cases.
- Hand-designed feature extractors require substantial engineering skill and domain expertise.
- Global features: Early systems combined low- and high-level global features with SVMs, Naïve Bayes, or boosting to classify professional photographs versus snapshots.
- Regional features: Regional features complement global representations by modeling subject-region clarity, lighting, exposure, sharpness, texture, and subject–background relationships.
- Composition features: Composition features model salient-object position, visual weight, rule-of-thirds, low depth-of-field, and opposing colors using regressors or classifiers.
- Combined representations: Other approaches fuse global and regional descriptors or encode local and global spatial structure with graphlets and multiple visual channels.
- Task-specific features: Task-specific features target faces, landscapes, geographic scenes, or Chinese calligraphy by exploiting category-relevant perceptual, spatial, or rule-based properties.
- Task-specific features: The reviewed systems use classifiers such as SVM, AdaBoost, and neural networks, alongside regressors for aesthetic ranking or scoring.
6 DEEP LEARNING APPROACHES
Deep-learning approaches learn aesthetic representations from data using generic transferred features, single- or multi-column CNNs, category information, patches, attributes, and auxiliary tasks. Architectures increasingly combine complementary visual scales or semantic signals.
- Deep learning learns feature representations from large datasets, reducing reliance on highly engineered feature designs required by shallow models.
- Generic and single-column CNNs: Generic deep features can be transferred from other tasks, such as AlexNet penultimate-layer representations with spatial pyramid pooling, then classified using an SVM.
- Generic and single-column CNNs: Single-column CNNs learn aesthetic features directly, with variants trained from scratch, using compact representations, or aggregating multiple sampled patches.
- Category-aware CNNs: Category-specific CNN branches exploit scene categories by pre-training parallel convolutional layers and transferring their weights into an end-to-end aesthetic model.
- Multi-column CNNs: Multi-column CNNs combine global images with local patches, multiple scales, style attributes, or learned textual-attribute regions.
- Ranking and auxiliary tasks: Siamese and multi-task architectures incorporate image-pair ranking, attributes, content, scene categories, or semantic predictions to assist aesthetic learning.
7 EVALUATION CRITERIA AND EXISTING RESULTS
Evaluation practices vary across datasets and metrics, preventing direct comparison of all reviewed methods. The survey therefore summarizes results by benchmark and uses balanced accuracy alongside overall accuracy to address class imbalance.
- Evaluation metrics: The literature reports classification accuracy, precision–recall curves, Euclidean distance, and residual sum of squares across aesthetic tasks.
- Cross-study comparison: Because datasets and evaluation criteria differ, the survey summarizes results separately for CUHK-PQ, AVA, and other datasets rather than directly comparing every method.
- Class imbalance: Overall accuracy is intuitive but can be biased on imbalanced data: predicting every AVA example as positive reaches about 70% accuracy.
- Class imbalance: Balanced accuracy weights sensitivity and specificity equally and penalizes classifiers that predict only the dominant class.
- Class imbalance: 50% balanced accuracy corresponds to the all-positive naïve predictor on the cited AVA class distribution.
- Baseline evaluation: The survey reports both overall and balanced accuracy for its proposed baseline to provide a more reasonable performance measure.
8 EXPERIMENTS ON DEEP LEARNING SETTINGS
The experiments systematically examine CNN training choices for aesthetic classification, showing that careful initialization, sampling, auxiliary learning, and input design substantially affect performance. Results also reveal trade-offs involving global composition, local patches, model depth, category, and personalization.
- Deep learning approaches dominate image aesthetic assessment performance, motivating systematic analysis of CNN training settings.
- 8.2 Training From Scratch vs Fine-Tuning: Fine-tuning pretrained CNNs improves initialization, while DAN fine-tuned from VGG-16 achieves the best balanced and overall accuracy.Because aesthetic datasets are typically small, freezing earlier layers may help prevent overfitting during fine-tuning.
- 8.3 Mini-Batch Sampling: Balanced mini-batches yield better balanced accuracy than randomly sampled mini-batches, despite lower overall accuracy.Balanced accuracy tracks both true-positive and true-negative prediction rates under class imbalance.
- 8.4 Triplet Pre-Training and Multi-Task Learning: 73.59% balanced accuracy is reached by the 1-column CNN using triplet pre-training plus multi-task learning.About 25% of AVA training images lack category labels, so the auxiliary loss is ignored for those instances.
- 8.4 Triplet Pre-Training and Multi-Task Learning: The 1-column CNN suppresses false positives while retaining reasonable overall classification accuracy, and its qualitative errors show high-quality trends or low image tones.The DAN-1 examples include images correctly classified despite being misclassified by BDN and DMA-net; false positives often have high contrast or depth of field.
9 IMAGE AESTHETIC MANIPULATION
The survey connects image aesthetic assessment with aesthetic manipulation through automatic cropping. It reviews attention-, aesthetics-, and change-based cropping, then adapts an aesthetic-classification CNN to generate crops directly or through learned crop regression.
- Aesthetic-based Image Cropping: Automatic image cropping improves aesthetic composition by removing undesired regions, and the survey focuses on it as a major aesthetic-enhancement operation.
- Aesthetic-based Image Cropping: Change-based cropping accounts for removed content and the starting composition, but evaluating many candidate crops makes it computationally expensive.
- DAN-1-original without cropping data: The modified DAN-1-original removes fully connected layers, producing a fully convolutional network whose response identifies crop regions for arbitrary-sized inputs.
- DAN-1-regression with cropping data: DAN-1-regression adds a window-regression layer trained with cropping data to predict more precise crop coordinates using Euclidean loss.
- Results: The DAN-1-original response reveals object composition rather than only salient objects, producing competitive crops without additional cropping data.
- Results: DAN-1-regression surpasses the previous state of the art, especially on boundary displacement error, while generating crops in ∼0.2 seconds on GPU and ∼2 seconds on CPU.
10 CONCLUSION AND POTENTIAL DIRECTIONS
The survey reviews image aesthetic assessment, proposes an alternative baseline, and extends aesthetic-classification CNNs to automatic aesthetic-based cropping. It concludes that the field remains open, with ambiguity, limited auxiliary information, and dataset diversity still posing challenges.
- The survey systematically reviews major image aesthetic assessment methods and proposes an alternative baseline for understanding image aesthetics.
- Automatic image cropping is presented as an extension that adapts a learned aesthetic-classification CNN to aesthetic-based image manipulation.
- Future progress is challenged by ambiguous ground truth, limited auxiliary data for category-specific aesthetics, and insufficiently diverse annotations.