Source-linked AI summary
Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature
Babak Saleh, Ahmed Elgammal
TL;DR
Growing digitized art collections require visual similarity measures suited to paintings, but suitable features and metric-learning strategies remain difficult to determine. The paper compares visual features and metric-learning methods for task-specific style, genre, and artist classification. Experiments report that learned metrics improve these classification tasks, with LMNN on Classeme features outperforming the state of the art for style classification while reducing feature-vector size by 90%.
Problem
Large digitized art collections need domain-optimized visual similarity, while engineering or learning features for artistic concepts is difficult because comprehensive expert annotations are costly.
Method
The paper compares four visual features, five metric-learning approaches, and three learning methodologies, including fused optimized feature spaces for style, genre, and artist classification.
Results
LMNN achieves the best performance across classification experiments, and LMNN learned on Classeme features outperforms the state of the art for style classification while reducing feature-vector size by 90%.
Takeaways & Limitations
Supervised metric learning can transform raw painting features into similarity representations that improve style, genre, and artist classification.
Takeaways & Limitations
Applicability of the learned representation to image retrieval and recommendation systems remains future work, as does learning metrics from other annotations such as time period.
Abstract
from arXiv · showhide
In the past few years, the number of fine-art collections that are digitized and publicly available has been growing rapidly. With the availability of such large collections of digitized artworks comes the need to develop multimedia systems to archive and retrieve this pool of data. Measuring the visual similarity between artistic items is an essential step for such multimedia systems, which can benefit more high-level multimedia tasks. In order to model this similarity between paintings, we should extract the appropriate visual features for paintings and find out the best approach to learn the similarity metric based on these features. We investigate a comprehensive list of visual features and metric learning approaches to learn an optimized similarity measure between paintings. We develop a machine that is able to make aesthetic-related semantic-level judgments, such as predicting a painting's style, genre, and artist, as well as providing similarity measures optimized based on the knowledge available in the domain of art historical interpretation. Our experiments show the value of using this similarity measure for the aforementioned prediction tasks.
1 Introduction
Digitized painting collections create a need for domain-optimized visual similarity measures that support archival, retrieval, recommendation, and semantic classification. The paper addresses this by comparing visual features and metric-learning strategies for predicting style, genre, and artist.
- Motivation: Digitized fine-art collections are expanding, increasing the need for multimedia systems that archive, retrieve, and recommend paintings using domain-appropriate visual similarity.Metadata commonly includes artist, style, date, and genre, while recommendation systems need paintings that users may find similar.
- Motivation: Human viewers make semantic judgments about paintings, including genre, style, date, and likely artist, but computationally encoding these concepts is difficult.Such judgments depend partly on a viewer’s art-historical knowledge and exposure.
- Research challenge: Engineering features for all artistic concepts is cumbersome, while learning them would require extensive expert annotations unavailable through typical crowdsourcing.The paper notes that art-historical expertise is needed to annotate these concepts reliably.
- Approach: The study investigates low-level through semantic-level visual elements and learns task-specific similarity metrics for style, genre, and artist classification.The selected tasks use annotations that are widely available and more often agreed upon by art historians and critics.
- Study design: The evaluation compares four visual features, five metric-learning approaches, and three learning methodologies across three artistic prediction tasks.The paper presents this as a comprehensive comparative study using a large digitized art dataset.
2 Related Work
Prior work explored handcrafted low-level features, brushstroke representations, local edge descriptors, and semantic features for painting analysis. This paper broadens the comparison across tasks, feature levels, and metric-learning approaches.
- Handcrafted features: Earlier painting-classification studies mainly used low-level features encoding color, shadow, texture, edges, and brushstrokes.Texture statistics were commonly used to represent brushstrokes, although digitization resolution affects texture features.
- Handcrafted features: Researchers also investigated local edge-orientation descriptors such as SIFT and HOG for analyzing paintings.These descriptors provide alternatives to color-, texture-, and brushstroke-based representations.
- Semantic features: A prior style-classification study found that semantic-level Classemes features outperformed low-level features on a small dataset of seven styles with 70 paintings per style.The earlier evaluation was narrower than the present study’s scope.
- Study scope: The present evaluation spans three prediction tasks, visual features from low-level to semantic-level, and five metric-learning approaches.This scope is explicitly contrasted with prior work evaluating three approaches over low-level HOG features for painter influence paths.
3 Methodology
The methodology evaluates visual features and learns task-specific similarity metrics for classifying paintings by style, genre, and artist. It also combines projected features or metrics to form richer representations for classification.
- 3.2 Classification Methodology: The pipeline extracts visual features, learns a metric optimized for each prediction task, and projects raw features into the resulting task-specific space.The learned projection converts high-dimensional visual information into a space intended to support classification.
- 3.1 Dataset and Proposed Tasks: The study uses WikiArt, comprising 81,449 paintings by 1,119 artists across 27 styles and 45 genres.The collection spans fifteen centuries to contemporary artists and is larger than collections used in prior work.
- 3.2 Classification Methodology: Feature fusion separately projects four feature types with task-specific metrics and concatenates the projected vectors for classification.This strategy combines different types of visual information after metric learning.
- 3.2 Classification Methodology: Metric fusion projects each visual feature with five metrics and concatenates the resulting optimized spaces to incorporate different similarity-learning criteria.The criteria include information theory and neighborhood analysis.
- 3.3 Visual Features: The visual representations span low-level GIST features and learned semantic-level Classeme, Picodes, and CNN-based features.GIST provides a 512-dimensional representation, while the semantic-level representations encode confidence in generic object categories.
- 3.4 Metric Learning: Metric learning uses a positive semi-definite matrix M, decomposed as M = G^T G, to project raw features so similarity becomes Euclidean distance in the projected space.The learned metric can also reduce feature dimensionality when M is low rank, and LMNN enforces nearby same-class examples while separating impostors.
4 Experiments
Experiments compare visual representations and metric-learning methods across style, genre, and artist classification. Learned metrics improve classification, with feature fusion achieving the strongest reported style result while also reducing representation dimensionality.
- Experimental setup: The study compares four visual features, five metric-learning approaches, and three learning methodologies across style, genre, and artist classification.The visual features include low-level GIST and higher-level Classeme, Picodes, and CNN-based representations.
- Experimental setup: All feature vectors were projected to 512 dimensions with PCA to enable a fair comparison for metric learning.For CNN features, the first 500 coefficients account for 95.88% of the total eigenvalue sum.
- Style classification: Boost metric learning and ITML generally provide the highest style-classification accuracy across visual features, with Boost on Classeme producing the greatest baseline improvement.Style classification uses SVM classifiers evaluated after metric projection, alongside a raw-feature baseline.
- Style classification: Style-classification confusions often align with art-historical relationships, including Action painting as a subgenre of Abstract expressionism and Mannerism’s relation to the Renaissance.The confusion matrix also captures similarities involving Expressionism, Fauvism, Minimalism, and Color field paintings.
- Genre and artist classification: Learning Boost metric generally improves artist classification more than other metrics, except that ITML performs best with CNN features.Artist classification uses one-vs-all SVM classifiers for 23 artists, assigning each test image to the classifier with maximum confidence.
- Feature fusion: 45.97% accuracy is achieved for style classification by applying LMNN to GIST, Classeme, PiCoDes, and CNN features and concatenating the projected vectors.This exceeds a reported 43% result using two PiCoDes variants and two CNN layers.
- Feature fusion: The best fused representation uses 400 dimensions rather than 3882, reducing space by 90% while achieving higher reported style-classification accuracy.The paper identifies image retrieval as a future application for this compact representation.
- Prototype image search: A prototype search using LMNN and feature fusion finds cross-style matches by forcing the retrieved image to have a different style from the query.The experiment evaluates whether a metric learned from style labels can identify similarity across styles.
5 Conclusion and Future Works
The comparative study finds that Classeme features perform best across style, genre, and artist classification, while LMNN is strongest for fused features. Learning LMNN on Classeme features also yields state-of-the-art style classification with a 90% smaller feature vector, while retrieval validation remains future work.
- Classeme features show superior performance across style, genre, and artist classification, independently of the learned metric.
- Boost and ITML improve classification accuracy across all individual visual features.
- LMNN achieves the best performance across classification experiments when different feature types are fused.
- 90% smaller feature vectors result from learning LMNN on Classeme features while outperforming the state of the art for style classification.The representation reduces the feature-vector size relative to the reported state-of-the-art representation.
- Applicability of the learned representation to image retrieval and recommendation systems remains future work.