Source-linked AI summary
CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification
Marcos V. Conde, Kerem Turgutlu
TL;DR
Fine-grained artwork recognition is challenging, and expert annotation is costly; iMet labels are also treated as noisy because each object has a single annotator without verification. The paper converts artwork attributes into natural-language descriptions and contrastively fine-tunes CLIP for classification and image-text retrieval. It reports better zero-shot fine-grained classification than few-shot supervised SOTA models and presents the approach as a way to perform artwork retrieval and recognition from natural-language instructions.
Problem
Fine-grained artwork recognition is difficult because visually similar subcategories require costly expert labeling, while iMet attributes are treated as noisy because each object has one annotator without verification.
Source: §1 · p. 1, §1.1 · p. 2Method
Results
Takeaways & Limitations
The approach supports fine-grained artwork retrieval and recognition through natural-language instructions without directly optimizing for iMet data.
Source: §4 · p. 4iMet attributes are recommended as noisy labels because each object is annotated by a single annotator without verification.
Source: §1.1 · p. 2
Abstract
from arXiv · showhide
Existing computer vision research in artwork struggles with artwork's fine-grained attributes recognition and lack of curated annotated datasets due to their costly creation. To the best of our knowledge, we are one of the first methods to use CLIP (Contrastive Language-Image Pre-Training) to train a neural network on a variety of artwork images and text descriptions pairs. CLIP is able to learn directly from free-form art descriptions, or, if available, curated fine-grained labels. Model's zero-shot capability allows predicting accurate natural language description for a given image, without directly optimizing for the task. Our approach aims to solve 2 challenges: instance retrieval and fine-grained artwork attribute recognition. We use the iMet Dataset, which we consider the largest annotated artwork dataset. In this benchmark we achieved competitive results using only self-supervision.
1. Introduction
Fine-grained artwork recognition is difficult because subcategories can look similar and expert labeling is costly, leaving limited training data. The paper explores CLIP-based multimodal representations for classification and retrieval, reporting zero-shot results better than few-shot supervised SOTA models.
1. Introduction
- Fine-grained visual classification distinguishes subcategories within broader classes, but similar-looking categories and costly expert labeling make accurate recognition difficult and limit training data.Examples include identifying a painting’s author, a sculpture’s material, or an instrument’s country of origin.Source: §1 · p. 1
1.1. Dataset and Benchmark
- The iMet collection spans paintings, instruments, prints, clothing, sculpture, furniture, and metalwork, among other artwork types.The dataset describes attributes related to what is seen in an artwork or inferred about its utility.Source: Figure 1 · p. 1, §1.1 · p. 1
1. Introduction
1.1. Dataset and Benchmark
- iMet groups artwork attributes into five parent classes and includes 3,471 unique attributes, with labels curated and verified by museum experts.Each object is annotated by one annotator without a verification step, so the authors recommend treating attributes as noisy labels.Source: §1.1 · p. 2
2. Approach
The approach turns noisy artwork attributes into multiple natural-language descriptions, then contrastively fine-tunes CLIP to learn aligned image-text representations. The domain-adapted model can be further fine-tuned for downstream fine-grained recognition.
2. Approach
- Noisy fine-grained attributes are converted into natural-language descriptions using templates and permutations, yielding more than 15 descriptions per iMet image.For categories with multiple values, the generation process uses different combinations of attribute values.Source: §2 · p. 2
2.1. Contrastive Pre-training
- CLIPart is trained with a symmetric image-text InfoNCE objective, using one randomly sampled description per image and attribute-value dropout when categories have multiple values.The model uses ViT-B/32 image and text encoders, pairwise cosine similarity of L2-normalized embeddings, and a learnable temperature.Source: §2 · p. 2
- Contrastive pre-training can use free-form text despite noise, support retrieval with natural-language queries beyond predefined labels, and improve downstream convergence.The models are trained for 20 epochs; the authors compare text pairs generated with all five categories against pairs excluding tags.Source: §2.1 · pp. 2–3
2.2. Fine-tuning
- Downstream recognition is treated as multilabel classification, with encoder weights frozen for five epochs and all model weights updated for the remaining 15.The same downstream setup is used across experiments for fair comparison.Source: §2.2 · p. 3
3. Experiments
The experiments assess zero-shot, few-shot, and fully supervised fine-grained classification alongside multimodal retrieval, using CLIP-based ViT models and other benchmark models.
3.1. Zero-Shot Experiments
- Zero-shot evaluation uses 512-dimensional image representations from CLIPbase and CLIPart to assign each hold-out image the label of its nearest training-set neighbor.The training set contains 142,119 iMet 2020 images, and the hold-out set contains 20,000 images.Source: §3.1 · p. 3
3.2. Multimodal Retrieval Experiments
- Multimodal retrieval evaluation embeds 20,000 validation images and their text descriptions, then measures retrieval percentages at 5 and 20 and mean and median retrieval rank.The study compares CLIPart versions trained with different text descriptions.Source: §3.2 · p. 3
3.3. Few-Shot and Fully Supervised
- Few-shot experiments train on fractions of the data, while fully supervised comparisons use the full training set and include CLIPart, CLIPbase, ImageNet-pretrained ViT, ResNets, and EfficientNets.The experiments compare few-shot and fully supervised performance with zero-shot performance.Source: §3.3 · p. 3
3.4. Results
- Results use F2-score on a 20,000-image validation set, with the metric chosen to provide robustness to noisy labels and favor recall over precision.The reported evaluation covers zero-shot, few-shot, and fully supervised training on iMet.Source: §3.4 · p. 3
- CLIPart without task-specific supervision outperforms few-shot ResNet-50 and EfficientNet, while fine-tuned CLIPart ViT matches fully supervised CNNs and beats ImageNet-pretrained ViT in half the epochs.The ResNet-50 and EfficientNet comparisons use models trained on a fraction of the complete dataset and optimized for the task.Source: §3.4 · p. 4
- CLIPart retrieves the complete text pair within the first 20 of 20,000 ranked predictions 54% of the time, and removing descriptive text hurts retrieval performance.The retrieval evaluation compares image and text embeddings on 20,000 validation images.Source: §3.4 · p. 4
4. Conclusion
The conclusion presents CLIP-based natural-language supervision as an approach to artwork retrieval and fine-grained attribute recognition. It also describes zero-shot use and text generation from closed-set labels.
4. Conclusion
- The approach uses CLIP with artwork images and natural-language supervision to address retrieval and fine-grained attribute recognition.The conclusion frames these as the main art-related computer-vision challenges.Source: §4 · p. 4
- The network can perform fine-grained artwork retrieval and recognition from natural-language instructions in a zero-shot manner without direct optimization for iMet data.Source: §4 · p. 4
- The paper proposes constructing natural-language text from available closed-set attribute labels by augmenting them.Source: §4 · p. 4
A1. Training convergence
The appendix compares fine-grained art classification after fine-tuning ViT-B/32 from three pretraining strategies and reports improved convergence and F2-score for CLIPArt in the first training stage.
A1. Training convergence
- The comparison fine-tunes ViT-B/32 for fine-grained art classification from CLIP, CLIPArt, and ImageNet pretraining.Source: A1. Training convergence · p. 5
- ImageNet pretraining is the third initialization strategy in the ViT-B/32 fine-tuning comparison.Source: A1. Training convergence · p. 5
- The CLIP baseline uses contrastive pretraining on 400 million images, while CLIPArt pretraining uses artwork images and their natural-language descriptions.Source: A1. Training convergence · p. 5
- CLIPArt improves first-stage convergence and F2-score when experiments use the same training setup and images and are trained to convergence.Source: Figure 6 · p. 5
A2. Large Scale Art Dataset
The appendix describes a pipeline for expanding artwork training data by combining predicted labels and expert-written descriptions into image-text pairs.
A2. Large Scale Art Dataset
- Supervised CNN teachers infer pseudo-labels on unlabeled artwork scraped from the internet, creating data for self-training or distillation into smaller noisy student models.Source: A2. Large Scale Art Dataset · p. 5
- Table 3 covers supervised SOTA CNNs trained on iMet and evaluated on the 2020 Benchmark.Source: Table 3 · p. 5
- Expert-written free-form descriptions are described as transferring human visual attention into textual attention, which can help distinguish significant categorization features.Source: A2. Large Scale Art Dataset · p. 5
- The proposed pipeline combines teacher-predicted pseudo-labels and scraped expert descriptions into image-text pairs for CLIP visual-attention learning.The authors aim to build an artwork dataset with more than 1 million image-text pairs.Source: A2. Large Scale Art Dataset · p. 5