Source-linked AI summary
Recognizing Image Style
Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, Holger Winnemoeller
TL;DR
Visual style is important to image meaning but has received limited computer-vision research attention. The paper evaluates image features for style prediction, introduces large photographic and painting datasets, and finds strong performance from deep features, including features trained for object recognition. The learned classifiers support style-aware image search and achieve results comparable to human performance in one evaluation.
Problem
Visual style is important to image meaning, but computer vision has explored it only minimally and existing photographic style data cover a narrow range.
Method
The paper evaluates image features for style classification, builds large Flickr and Wikipaintings datasets, and uses learned classifiers for style-aware search.
Results
Deep convolutional features generally perform best, achieve accuracy essentially matching Turkers for Group membership, and improve state-of-the-art AVA aesthetic prediction results.
Takeaways & Limitations
Style classifiers can extend image search with stylistic constraints and provide mid-level features for other image-understanding tasks.
Takeaways & Limitations
The paper leaves unresolved why object-detection features work so well for style and notes possible mismatch between Flickr-group definitions and Turk-task definitions.
Abstract
from arXiv · showhide
The style of an image plays a significant role in how it is viewed, but style has received little attention in computer vision research. We describe an approach to predicting style of images, and perform a thorough evaluation of different image features for these tasks. We find that features learned in a multi-layer network generally perform best -- even when trained with object class (not style) labels. Our large-scale learning methods results in the best published performance on an existing dataset of aesthetic ratings and photographic style annotations. We present two novel datasets: 80K Flickr photographs annotated with 20 curated style labels, and 85K paintings annotated with 25 style/genre labels. Our approach shows excellent classification performance on both datasets. We use the learned classifiers to extend traditional tag-based image search to consider stylistic constraints, and demonstrate cross-dataset understanding of style.
1 Introduction
The paper frames visual style as important to image meaning but underexplored in computer vision, then introduces broad style datasets and deep-feature classifiers. These classifiers perform strongly against human observers and support style-aware image search.
- Visual style influences image meaning across art, cinematography, advertising, and photography, yet remains underexplored in computer vision.
- The authors define styles spanning photographic techniques, composition, moods, genres, colors, and scene types, with labels that are not mutually exclusive.
- Deep convolutional neural network features outperform hand-tuned features for style classification, despite being trained on object categories.
- The classifiers predict Group membership at essentially the same accuracy as Turkers and improve state-of-the-art performance on the AVA aesthetic prediction task.
- Style-aware search enables queries such as finding images similar to a reference but more Film Noir, while style tags can support other image-understanding tasks.
2 Related Work
Prior computer-vision work largely emphasizes recognition, reconstruction, aesthetics, or a small set of narrowly defined visual properties. Painting-style datasets have also been limited in scale, style count, or task scope.
- Most computer-vision research addresses recognition and reconstruction independently of image style, with some work on beauty, interestingness, and memorability.
- Earlier aesthetic-prediction systems used hand-designed features for properties such as colorfulness, saturation, rule-of-thirds, and depth-of-field.
- The AVA dataset includes photographic ratings and some style labels derived from challenge titles and descriptions.
- Painting-style studies generally covered fewer than ten visually distinct styles and fewer than 60 images per style, while a larger artwork dataset did not target style classification.
3 Data Sources
The paper constructs large, broad-coverage datasets for photographic and painting style because existing photographic style data were narrow and large-scale painting-style data were absent. Flickr groups provide curated photographic labels, while Wikipaintings supplies expert-labeled artwork.
- Existing photographic style data covered only a narrow range, motivating broader labels across genres, composition, and moods.
- Flickr groups serve as community-curated sources of visual concepts instead of unreliable free-form user tags.
- The Flickr dataset contains 20 styles across optical techniques, atmosphere, mood, composition, color, and genre.
- 4,000 positive examples per label produce a total of 80,000 Flickr images, with positive labels considered clean but negative labels potentially noisy.
- The Wikipaintings dataset contains 100,000 expert-labeled artworks, from which 25 styles with more than 1,000 examples yield 85,000 images spanning Renaissance to modern art.
4 Learning algorithm
The learning setup uses linear classifiers over high-dimensional image features, trained with stochastic gradient optimization and validated regularization and loss choices. Multiclass prediction is reduced to one-versus-all binary classifiers.
- The method uses linear classifiers because the datasets are large and some features are high-dimensional.
- Training uses an open-source stochastic-gradient implementation with adaptive subgradient optimization.
- L1 and L2 regularization, hinge or logistic loss, and the initial learning rate are selected or set through the stated validation and optimization procedure.
- The multiclass setup uses a One vs. All reduction to binary classifiers.
5 Image Features
The paper evaluates image features spanning color, composition, content, and deep representations for visual-style classification, including second-stage feature fusion. It also tests content-dependent style modeling by combining features with aggregate content classifiers.
- Feature Families: Style prediction is evaluated using features that represent low-level statistics, color, composition, content, and object-recognition information.The evaluation includes single-feature performance and second-stage fusion of multiple features.
- Feature Families: The 784-dimensional L*a*b color histogram captures whole-image color distributions using 4, 14, and 14 bins across its channels.It is motivated by styles with strong color dependence, including Noir, Horror, and Vintage.
- Feature Families: GIST and graph-based visual saliency represent image composition through low-resolution structure and visual attention, respectively.GIST uses a 960-dimensional color descriptor, while saliency uses a 1024-dimensional feature on 256 by 256 images.
- Feature Families: MC-bit is a 15,000-dimensional binary feature learned by stacking classifiers and pooling operations over lower-level image features and ImageNet synsets.It represents image content as a handcrafted deep architecture.
- Deep Features: The deep convolutional features DeCAF5 and DeCAF6 come from an eight-layer ImageNet network trained on over a million images across 1,000 classes.The paper investigates features from two different network levels, despite their object-class training objective.
- Content-Dependent Modeling: Content-dependent style modeling forms the outer product of a feature channel with four aggregate classifiers for animals, vehicles, indoor objects, and people.These aggregate classes are intended to distinguish image types where different style signals may apply.
6 Experiments
Experiments evaluate style classification on Flickr, AVA, and Wikipaintings, compare learned features with humans, and examine content–style relationships and applications. Deep features perform strongly across datasets, while content-dependent representations improve Flickr results and support cross-dataset and style-based search.
- Flickr Style: 80,000 Flickr images are used to learn and predict 20 visual style labels, with 20% reserved for testing and 20% for validation.
- Flickr Style: 0.368 mean AP is achieved by feature fusion on Flickr Style, exceeding DeCAF6’s 0.336 mean AP; average per-class accuracy is 78%.
- Flickr Style: Content explains some style errors and correlations: Macro is associated with animals, while Hazy and Long Exposure commonly align with vehicles.
- Flickr Style: Fusion × Content features obtain the best results on Flickr, supporting the hypothesis that style depends on image content.
- Human Comparison: Turkers achieve 75% mean accuracy versus 78% for the best method, although humans outperform the algorithm on Macro and Horror.
- Wikipaintings: On Wikipaintings, feature fusion reaches 0.473 mean AP versus 0.441 for MC-bit, with per-class accuracies from 72% to 94%.
- AVA Style: On AVA, DeCAF6 reaches 0.579 mean AP and fusion 0.581, both surpassing the previous 0.538 mean AP state of the art.
- Applications: The learned classifiers enable style-filtered image search and transfer style rankings between photographs and paintings.