Source-linked AI summary
BAM! The Behance Artistic Media Dataset for Recognition Beyond Photography
Michael J. Wilber, Chen Fang, Hailin Jin, Aaron Hertzmann, John Collomosse, Serge Belongie
TL;DR
Computer vision systems remain focused on photographic imagery, while contemporary artwork presents diverse representations that existing systems often fail to recognize. The paper constructs and annotates a large Behance dataset spanning content, emotions, and artistic media, then uses it for recognition, style, and domain-adaptation studies. The reported experiments show value for improving object-classifier generality and related artistic-image tasks.
Problem
Photographic datasets do not adequately represent contemporary artistic imagery, limiting study of recognition across artistic styles, media, and emotions.
Method
The authors collect almost 65 million Behance images, annotate content, artistic media, and emotions, and evaluate recognition, style classification, domain adaptation, and retrieval tasks.
Results
The dataset supports representation-gap analysis, improved object-classifier generality, style classification, domain adaptation, and retrieval by content, media, or emotion.
Takeaways & Limitations
Behance Artistic Media provides a starting foundation for studying artistic imagery and extending machine vision beyond photographic representations.
Takeaways & Limitations
The dataset reports 90% label precision, and its initial label set is only a small proof-of-concept rather than a PASCAL/COCO-sized labeling effort.
Abstract
from arXiv · showhide
Computer vision systems are designed to work well within the context of everyday photography. However, artists often render the world around them in ways that do not resemble photographs. Artwork produced by people is not constrained to mimic the physical world, making it more challenging for machines to recognize. This work is a step toward teaching machines how to categorize images in ways that are valuable to humans. First, we collect a large-scale dataset of contemporary artwork from Behance, a website containing millions of portfolios from professional and commercial artists. We annotate Behance imagery with rich attribute labels for content, emotions, and artistic media. Furthermore, we carry out baseline experiments to show the value of this dataset for artistic style prediction, for improving the generality of existing object classifiers, and for the study of visual domain adaptation. We believe our Behance Artistic Media dataset will be a good starting point for researchers wishing to study artistic imagery and relevant problems.
1. Introduction
The paper introduces a large contemporary-art dataset from Behance to address the gap between photographic computer vision and artistic imagery. It targets object categories, artistic media, and emotions while supporting studies of recognition generality and domain transfer.
- Motivation: Current object classifiers trained on photographs often fail to recognize objects rendered in artistic media.This representation gap limits generalization beyond everyday photographic imagery.
- Dataset scope: Behance provides a broad cross-section of contemporary art and design spanning creative direction, fine art, technical diagrams, graffiti, and concept design.The dataset is intended to represent contemporary rather than historical classical art.
- Annotation scope: The dataset targets object categories, artistic media, and emotions because these facets describe contemporary artwork and are not apparent in current photographic datasets.The annotation goals include visually diverse object depictions, professional media such as sketches and watercolor, and the relatively unexplored facet of emotion.
- Scope and caveat: Quality tests report 90% label precision, reflecting a scalability challenge in annotating millions of images.The authors identify scalable annotation and vocabulary design as central challenges.
- Contribution: Almost 65 million images and an expert-defined vocabulary of binary artistic attributes form the Behance Artistic Media Dataset.The dataset includes quality-assurance thresholds and was planned for release through bam-dataset.org.
- Research uses: The dataset supports representation-gap analysis, domain adaptation across artistic media, style classification, and retrieval by content, media, or emotion.The authors also report that their object classifier scores above 0.95 on the illustrated artistic examples.
2. Related Work
Prior artistic and attribute datasets largely emphasize photography or classical paintings and often lack broad contemporary-art annotations. The paper positions BAM as a large-scale contemporary-art resource combining content, media, and emotion labels.
- Existing attribute datasets: Earlier attribute research studied aesthetics and emotional qualities mainly in photographic imagery.Examples include attributes such as interestingness, symmetry, exposure, depth of field, and mood.
- Existing artistic datasets: Related artistic datasets include photographs, classical paintings, cubist art, or Behance imagery without descriptive attributes.Open Images includes some media labels, but its labeling quality is unclear and each cited label has fewer than 400 human-verified images.
- Annotation challenges: A classifier trained directly on project-level “Cat” tags reaches only 36% cat precision in its top 100 images because tags are noisy and may cover unrelated project images.The classifier tends to learn many small animals rather than fine-grained cats.
- Paper positioning: BAM focuses on non-photorealistic contemporary art and combines emotion, media, and content annotations at large scale.The authors describe it as the first dataset, to their knowledge, seeking this combination for a broad range of contemporary artwork.
3. The Behance Media Dataset
The Behance Media Dataset combines contemporary artwork from a large portfolio site with binary labels for artistic media, emotions, and object categories. A human-in-the-loop pipeline expands expert and crowd annotations into automatically labeled data while applying validation-based quality thresholds.
- Dataset: Behance contains over ten million projects and 65 million images spanning fields including sculpture, painting, graphic design, graffiti, illustration, and advertising.Projects are the fundamental categorization unit and include noisy user-supplied metadata.
- Attribute categories: The dataset annotates media, emotion, and entry-level object categories using binary attributes designed to cover visually distinguishable contemporary artistic styles and content.Media labels include 3D computer graphics, comics, oil painting, pen ink, pencil sketches, vector art, and watercolor; emotion labels cover four affective categories.
- Annotation pipeline: Tags alone are insufficient because some attributes lack corresponding tags and project-level tags cannot identify which individual image they describe.This motivates collecting image-level human annotations rather than relying solely on noisy user-supplied metadata.
- Dataset statistics: The final dataset contains positive and negative examples for 20 attributes, with median counts of 54,000 positive and 8.7 million negative images per attribute.People has the most positive images at 1.74 million, while Cat has the fewest at 19,244.
- Quality and scale: 90.4% mean precision was measured across attributes, while automatic labeling amplified crowd-positive labels 17.4-fold and negative labels about 505-fold.Quality assurance used worker agreement on sampled automatically labeled images, alongside agreement and reputation requirements for Mechanical Turk workers.
4. Experiments
Experiments use BAM! to test recognition across artistic representations, unseen media, stylistic attributes, transfer to other datasets, and task-specific retrieval. Results expose a photography-to-art representation gap while showing complementary feature strengths and cross-dataset gains.
- Object recognition: Existing detectors and classifiers trained on photography recognize artistic objects worse than systems trained on artwork, revealing a representation gap.A fusion of ImageNet and Behance scores performs slightly better than the compared individual models on all but two attributes.
- Domain adaptation: Unseen-medium generalization is highest for iconic objects within a medium and lowest for objects portrayed inconsistently, such as watercolor cars or 3D flowers.The classifier learns from six media and is evaluated on the seventh.
- Style and aesthetics: StyleNet features outperform ImageNet-derived features on all four emotion attributes and four of six media attributes, whereas ImageNet features perform better on all nine content attributes.Performance is evaluated using average precision on held-out human labels.
- Style and aesthetics: BAM! imagery improves style classification on three external challenges compared with both the original ResNet-50 and StyleNet.The evaluated external datasets include Flickr and AVA, which focus on photographic style.
- Visual subspace learning: Task-specific 64-dimensional subspaces retrieve content, media, and emotion neighbors with precision-at-10 of 0.71, 0.91, and 0.84, respectively.Media and content embeddings align more clearly with their query attributes than the emotion embedding.
5. Conclusion
The paper introduces BAM!, a large-scale contemporary-art dataset with rich content, emotion, and media attributes. Its hybrid annotation process addresses noisy metadata and crowdsourcing scale, and experiments demonstrate usefulness across several vision tasks.
- Dataset: BAM! collects contemporary artwork from Behance, a large repository of professional and commercial artists.The dataset is intended to represent diverse artistic styles and content.
- Dataset: The dataset provides visually distinctive attributes covering emotion, artistic media, and content.These attributes are designed to represent the diversity found in Behance imagery.
- Annotation: A hybrid human-in-the-loop labeling system uses deep learning to scale annotation while meeting quality guarantees despite noisy tags and Behance’s size.The approach addresses the limitations of direct tag learning and brute-force crowdsourcing.
- Applications: BAM! supports object recognition, domain transfer, style classification, aesthetic classification, and content- or media-based image retrieval.The paper uses these tasks to study artistic representations and related vision problems.
- Conclusion: The authors position BAM! as a foundation for further research on large-scale artistic imagery.The conclusion frames artistic imagery as an underexplored research area.