PaperlayerView exact revision on arXiv

Source-linked AI summary

Photo Aesthetics Ranking Network with Attributes and Content Adaptation

Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, Charless Fowlkes

arXiv:1606.01621v2cs.CVcs.IRcs.MM

TL;DR

The paper addresses fine-grained photo-aesthetic ranking beyond binary classification by learning relative aesthetics with photographic attributes and content. It introduces a rater-aware dataset and unified model, achieving state-of-the-art AVA classification through score thresholding while matching average-worker performance but trailing more consistent raters.

  • Problem

    Existing image-aesthetics methods primarily focus on coarse binary classification, despite applications needing automatic fine-grained aesthetic rankings.

  • Method

    A Siamese CNN jointly predicts relative rankings and aesthetic scores while incorporating photographic attributes, content adaptation, and mixed within- and cross-rater image-pair sampling using AADB annotations.

  • Results

    The unified model achieves state-of-the-art classification performance on AVA by thresholding estimated scores and performs comparably to average mechanical-turk workers, though below more consistent workers.

  • Takeaways & Limitations

    AADB and rater-aware ranking training support aesthetic models that align with human ratings while motivating future adaptation to individual user preferences.

  • Takeaways & Limitations

    Models trained on AADB or AVA show very limited cross-dataset transferability, associated with differing rater tastes and visual-characteristic distributions.

Abstract

from arXiv · show

Real-world applications could benefit from the ability to automatically generate a fine-grained ranking of photo aesthetics. However, previous methods for image aesthetics analysis have primarily focused on the coarse, binary categorization of images into high- or low-aesthetic categories. In this work, we propose to learn a deep convolutional neural network to rank photo aesthetics in which the relative ranking of photo aesthetics are directly modeled in the loss function. Our model incorporates joint learning of meaningful photographic attributes and image content information which can help regularize the complicated photo aesthetics rating problem. To train and analyze this model, we have assembled a new aesthetics and attributes database (AADB) which contains aesthetic scores and meaningful attributes assigned to each image by multiple human raters. Anonymized rater identities are recorded across images allowing us to exploit intra-rater consistency using a novel sampling strategy when computing the ranking loss of training image pairs. We show the proposed sampling strategy is very effective and robust in face of subjective judgement of image aesthetics by individuals with different aesthetic tastes. Experiments demonstrate that our unified model can generate aesthetic rankings that are more consistent with human ratings. To further validate our model, we show that by simply thresholding the estimated aesthetic scores, we are able to achieve state-or-the-art classification performance on the existing AVA dataset benchmark.

1 Introduction

The paper addresses the limits of binary aesthetics classification by proposing fine-grained ranking with joint photographic attributes, content information, and rater-aware training data.

  • Binary classification separates obviously high- and low-aesthetic images but provides limited insight into borderline cases.This motivates rating and ranking aesthetics instead of assigning only binary labels.
  • The proposed Siamese CNN predicts relative aesthetic rankings alongside overall aesthetic scores.Pairwise training enables sampling strategies and auxiliary side information for regularization.
  • The model jointly incorporates aesthetic attributes and photo content because attribute importance varies across thematic content.Content-similar image pairs help learn content-specific relations between attributes and aesthetics.
  • AADB records score distributions, meaningful attributes, and anonymized rater identities across images.These annotations support within-rater ranking strategies and comparison with individual human ratings.
  • Mixed within- and cross-rater image pairs with pairwise ranking loss substantially improve ranking correlation.The dataset and sampling strategy address differing aesthetic tastes among individuals.

2 Related Work

Prior work separately explored whole-image aesthetics, describable attributes, and content adaptation; this paper unifies these components in one CNN framework.

  • CNN for aesthetics classification: The proposed approach combines regression and ranking rather than treating aesthetic prediction solely as classification.It warps the whole input image instead of relying on original high-resolution patches.
  • Attribute-adaptive models: Hand-crafted or text-derived attributes can be difficult to engineer, poorly defined, or incomplete for aesthetics analysis.The paper instead builds on meaningful style attributes learned in a CNN framework.
  • Content-adaptive models: Earlier content-adaptive methods assumed category labels were available during both training and testing.This limits their use when image categories are unknown at prediction time.
  • Content-adaptive models: The unified model jointly incorporates photo content, meaningful attributes, and aesthetic ratings in a single CNN.Soft category weights combine multiple content-adaptive branches while reusing shared front-end computations.

3 Aesthetics and Attributes Database

AADB is a Flickr-derived aesthetics dataset with diverse photographic imagery, multi-rater scores, meaningful attributes, and rater identities tracked across images.

  • Data collection: AADB collects varied real-scene photographs from Flickr under a Creative Commons license and removes non-photographic images.The curation excludes cartoons, drawings, paintings, advertisements, and adult-content images.
  • Dataset properties: AADB tracks score distributions, meaningful style attributes, and anonymized rater identities across images.Rater identity provides side information for training improved aesthetic ranking models.
  • Annotation: Five workers independently annotate each image with an aesthetic score and eleven meaningful attributes.The ground-truth aesthetic score averages the five ratings.
  • Annotation: The eleven attributes cover photographic principles involving content, lighting, color, focus, and composition.Examples include interesting content, good lighting, color harmony, shallow depth of field, rule of thirds, and symmetry.
  • Dataset comparison: Existing datasets often lack full score distributions or style attributes, or contain examples that are biased or easy for binary classification.Some datasets provide only binary labels, which are not readily suited to rating prediction.
  • Dataset comparison: AADB differs from AVA through a more balanced professional-consumer photo distribution and more diverse photo qualities.AVA contains many heavily edited or synthetic images, while AADB emphasizes photographic imagery of real scenes.

4 Fusing Attributes and Content for Aesthetics Ranking

The model combines regression and pairwise ranking with attribute and content adaptation to produce more consistent fine-grained aesthetics estimates. It uses shared CNN features, auxiliary attribute supervision, and content-weighted predictions while exploiting multiple pair-sampling strategies.

  • 4.1 Regression Network for Aesthetics Rating: The training pipeline fine-tunes AlexNet first for aesthetic regression, then adds Siamese ranking, attribute, and content-category branches through joint optimization.The regression model predicts continuous ratings; later stages incorporate pairwise ranking and auxiliary tasks.
  • 4.2 Pairwise Training and Sampling Strategies: The joint Euclidean-ranking loss anchors predicted scores to user-rating scale while explicitly penalizing relative-order errors.The ranking-loss weight ωr is selected using validation data, and the margin α is specified for pairwise training.
  • 4.2 Pairwise Training and Sampling Strategies: Pair sampling favors large score differences for stable ordering and uses within-rater comparisons to obtain more consistent training signals despite differing aesthetic tastes.The model uses rater identities to compare ratings assigned by an individual across multiple images.
  • 4.3 Attribute-Adaptive Model: The attribute-adaptive model shares front-end features with the aesthetic predictor and uses attribute supervision as deep regularization, even when annotations cover only part of the training data.Attribute predictions are concatenated with the base model to produce the final aesthetic score; missing annotations can be handled by freezing the attribute layer.
  • 4.4 Content-Adaptive Model: Because attribute relevance depends on image content, the content-adaptive model predicts category scores and uses them as weights to combine multiple content-specific branches.Each branch combines attribute and content-specific features, and soft category weighting avoids hard category selection.
  • 4.4 Content-Adaptive Model: The model assumes low-level features can be shared between aesthetics and category prediction, while content categories may be non-exclusive.During whole-network fine-tuning, the content-classification branch is frozen.

5 Experimental Results

Experiments evaluate the proposed ranking model on AADB and AVA, including architecture variants, sampling strategies, hyperparameters, human consistency, and cross-dataset transfer. The unified model improves ranking and achieves strong AVA classification after thresholding estimated scores, while transfer across datasets remains limited.

  • Benchmark Datasets: AADB contains 10,000 images split into 8,500 training, 500 validation, and 1,000 testing images, with aesthetic scores, attributes, and anonymized rater identities.Content groups are obtained by unit-normalized fc7 features and unsupervised k-means because AADB lacks ground-truth content tags.
  • Performance Evaluation: Spearman’s ρ measures agreement between predicted and ground-truth rankings, while thresholding predicted scores produces binary AVA classifications.The ranking coefficient ranges from −1 to 1 and is invariant to monotonic transformations of predicted scores.
  • Model Architecture and Loss Functions: Attributes and content adaptation both improve ranking performance, and the unified model with ranking, attribute, and content branches performs best among the compared models.The content-adaptive model uses unsupervised k-means clusters, while attribute gains are larger on AVA than on AADB.
  • Classification Benchmark Performance: 77.33% accuracy on AVA exceeds the 72.85% low-resolution SPP baseline by 4.48% after simply thresholding estimated aesthetic scores.The model uses whole warped down-sampled images and was trained for aesthetics ranking rather than classification.
  • Human Consistency: The model exceeds average human consistency but remains below highly consistent “power raters” in rank correlation.Human consistency is evaluated by comparing each individual’s ratings with the ground-truth average score.
  • Cross-Dataset Evaluation: Models trained on AADB or AVA have limited transferability because the datasets differ in rater tastes and visual-characteristic distributions.AVA contains more professional or heavily edited images, whereas AADB includes many casual daily photos.

6 Conclusion

The proposed CNN unifies photo style attributes and content information for aesthetic rating, using individual aesthetic rankings from a new annotated dataset. It achieves state-of-the-art classification performance on AVA by thresholding estimated scores, while matching average-worker performance but trailing more consistent workers.

  • The CNN unifies photo style attributes and content information to rate image aesthetics.
  • Individual aesthetic rankings from a dataset with aesthetic and attribute scores support training the architecture.
  • State-of-the-art classification performance on the AVA benchmark is achieved by simple thresholding, despite not using high-resolution image patches.
  • The model performs as well as the average mechanical Turk worker but lags behind more consistent workers who label large image batches.

Attributes in AADB

AADB uses eleven aesthetics-related attributes selected with professional photographers. They cover traditional photographic principles and provide vocabulary for applications such as automatic photo editing and image retrieval.

  • Eleven attributes highly related to image aesthetics were selected after consultation with professional photographers.
  • The attributes include balancing element, content, color harmony, depth of field, lighting, motion blur, object emphasis, rule of thirds, and vivid color.
  • The attributes span traditional photographic principles of color, lighting, focus, and composition.
  • They provide a natural vocabulary for applications such as automatic photo editing and image retrieval.

Data Collection By Amazon Mechanical Turk

AADB was collected from Creative Commons Flickr photographs filtered to retain photographic images, then annotated on Amazon Mechanical Turk for aesthetics and eleven attributes. Five qualified workers labeled each image, with averaged scores normalized to [0, 1].

  • Images were downloaded from Flickr under Creative Commons licenses and manually curated to remove non-photographic content.
  • Multiple Mechanical Turk workers independently annotated each image with an overall aesthetic score and eleven meaningful attributes.
  • For each attribute, workers marked positive, negative, or null according to whether it enhanced, degraded, or did not affect image aesthetics.
  • 10,000 images were labeled by five different workers each, with workers required to pass instructions and a qualification exam.
  • There were 190 workers, and images were split into batches of ten so raters annotated different numbers of batches.
  • Averaged aesthetic and attribute scores were used as ground truth, and aesthetic scores were normalized to [0, 1].The normalized ratings follow a Gaussian distribution.

Appendix: Statistics of AADB

The final AADB contains 10,000 images with aesthetic-quality ratings and attribute assignments from five individual raters per image. Unlike AVA, AADB provides rating scores for attributes rather than only binary attribute labels.

  • The final AADB contains 10,000 images, each rated for aesthetics and attributes by five individual raters.
  • AADB provides rating scores for attributes, whereas AVA provides only binary attribute labels.

Appendix: Consistency Analysis

The consistency analysis finds statistically significant agreement among AADB annotations, using Kendall’s W and Spearman’s ρ with permutation-based significance testing.

  • Agreement Measures: Kendall’s W measures agreement among multiple raters while accounting for tied ranks, ranging from 0 for no agreement to 1 for complete agreement.
  • Attribute Annotation: Attributes repetition and symmetry receive only presence annotations, while augmentation samples rare cases to address classifier data imbalance.
  • Global Consistency: Permutation tests evaluate the null distribution of global Kendall’s W and show that empirical agreement in AADB is statistically significant.
  • Batch Consistency: 0.5322 average Kendall’s W across batches indicates significant consistency among AMT annotations.Each batch contains ten images rated using five possible ratings, so tied ranks can occur.
  • Batch Consistency: 99.07% of 1,013 batches had significant agreement at FDR level Q = 0.05 under the Benjamini-Hochberg procedure.
  • Batch Consistency: 98.45% of batches showed significant pairwise-rank agreement under Spearman’s ρ at FDR level Q = 0.05.Batch p-values were computed from exact permutation distributions and averaged across rater pairs.

Appendix: Analysis of Content-Aware Model

The content-aware analysis compares alternative ways to combine content-specific aesthetic branches on AVA and qualitatively examines predictions on external personal photos.

  • AVA Analysis: Table 8 analyzes content-aware aesthetic scoring methods on the AVA dataset.
  • AVA Analysis: The compared methods select branches using ground-truth or predicted content, average branch scores, or weight scores by content-classification confidence.The weightedSum FT variant fine-tunes the network while freezing the classification branch.
  • Qualitative Evaluation: External personal photos are evaluated qualitatively using estimated aesthetic scores and predictions for all eleven attributes.These images were downloaded online and were not part of AADB.
  • Qualitative Evaluation: Attribute estimates are labeled negative, null, or positive using thresholds of −0.2 and 0.2.
  • Qualitative Evaluation: Figures 13–15 present external images with high, low, and medium estimated aesthetic scores.
Loading 1606.01621v2…