Source-linked AI summary
Style or Signature? Artist-Disjoint Evaluation of Style Classification in Frozen Vision Embeddings
Rory Ashton
TL;DR
Frozen vision embeddings may classify artistic style by recognizing individual artists rather than shared movement characteristics. This paper evaluates style classification with artists held out and finds that accuracy declines unevenly, especially for Surrealism.
Problem
Random-split style evaluations cannot distinguish recognition of artistic style from recognition of individual artists when the same painter appears on both sides.
Method
The paper evaluates frozen vision embeddings using artist-disjoint nearest-neighbour classification on 320 paintings spanning four twentieth-century movements.
Results
5-NN style accuracy falls from 0.87 to 0.77, with Impressionism and Cubism largely stable while Surrealism falls twenty points across encoders.
Takeaways & Limitations
Artist-disjoint evaluation reveals whether style classification reflects shared visual form or dependence on recognizable painters.
Takeaways & Limitations
The absolute accuracies are specific to the small, curated dataset and should not be generalized to paintings broadly.
Abstract
from arXiv · showhide
Frozen image embeddings from models such as CLIP are increasingly used to classify paintings by art-historical style, with high reported accuracy. We ask whether this accuracy reflects an understanding of style or the recognition of individual artists. Standard evaluation uses random splits in which works by the same artist appear on both sides, so a classifier can succeed by recognising the painter rather than the movement. We re-evaluate style classification under an artist-disjoint protocol, holding out every artist in turn so that no work is ever classified using other works by its own painter. On a balanced dataset of 320 paintings across four twentieth-century movements, 5-NN style accuracy falls from 0.87 to 0.77 under this protocol, and the drop is sharply uneven. Impressionism and Cubism barely move, while Surrealism falls twenty points. The pattern holds across four image encoders, including a vision-only self-supervised model, which places the effect in visual structure rather than language. Where an encoder captures genuine shared form, individual artists are barely recognisable yet style is robust, while Surrealism shows the opposite. We argue that artist-disjoint evaluation is necessary to measure stylistic understanding in frozen embeddings.
1 Introduction
The paper argues that high style-classification accuracy in frozen vision embeddings may reflect artist recognition because standard random splits allow same-artist works on both sides. It introduces artist-disjoint evaluation and finds that accuracy declines unevenly across movements, revealing fragility concealed by random splits.
- Motivation: Standard random splits can place works by the same artist in reference and query sets, making artist identity a confound in style classification.A classifier may recognize the painter rather than the movement when same-artist works are available as neighbours.
- Motivation: Frozen CLIP features support high-accuracy artist identification, providing a strong neural-signature signal that can inflate apparent style performance.This makes the confound substantive rather than hypothetical.
- Method: Artist-disjoint evaluation holds out whole artists from the neighbour pool, preventing paintings from being classified using other works by their own painter.The protocol is presented as a safeguard against identity leakage and as a way to expose fragility hidden by random-split evaluation.
- Results: 0.87 to 0.77: 5-NN style accuracy falls on a balanced dataset of 320 paintings across four twentieth-century movements under artist-disjoint evaluation.The aggregate decline masks a sharply uneven effect: Impressionism and Cubism barely move, whereas Surrealism falls twenty points.
2 Related work
Prior work questions whether frozen vision embeddings capture stylistic nuance in paintings, while WikiArt evaluations often leave dataset-specific artifacts and artist identity uncontrolled. Artist-disjoint evaluation follows established group-disjoint practices in adjacent fields to prevent identity leakage.
- Frozen vision embeddings for art: Frozen CLIP features transfer to art without adaptation, but prior studies find weaker performance on stylistic nuance and fine-grained style than on semantic content.Off-the-shelf CLIP struggles with Wölfflin’s principles and requires fine-tuning.
- Style classification on WikiArt: WikiArt movement classification commonly uses held-out splits within the same dataset, making transferable stylistic features difficult to distinguish from dataset-specific artefacts.A survey calls for duplicate removal, artist- or source-disjoint splits, and transparent class distributions.
- Artist identity as a confound: 89.3% of 372 WikiArt artists are matched by a classifier on frozen features, demonstrating artist identity as a strong confound in random-split style evaluation.These recurring artist characteristics are described as neural signatures.
- Group-disjoint evaluation: Group-disjoint evaluation is established in medical imaging and speech recognition, where holding out patients or speakers prevents identity leakage and can reveal substantial variability.Speech-recognition results can vary substantially depending on which speaker is held out.
3 Data
The study uses a balanced dataset of 320 paintings from four twentieth-century movements, sampled from a frozen WikiArt release with eight artists and ten works per movement artist. The design prioritizes reproducibility and conventional movement membership, while acknowledging that its artist selection reflects the historical canon rather than complete membership.
- Source and reproducibility: The dataset comes from a frozen, versioned Artificio/WikiArt release with single style labels, avoiding live-site drift and multi-tagging inflation.The exact image set is pinned by filenames and SHA-256 hashes in the released manifest and checksum list.
- Sampling: Each movement includes eight artists selected by strong standard-reference association and a minimum of ten qualifying works, with ten fixed-seed random works sampled per artist.The study lists all 32 artists and merges designated Cubist and Abstract Expressionist sub-styles into unified movement labels.
- Preprocessing: All images are received by encoders as identical 256×256 anisotropically resized thumbnails that preserve content while distorting aspect ratio.The preprocessing retains framing, composition, and figure-to-ground boundaries, though the resulting images differ from typical pretraining inputs.
- Limitations: The selection represents the conventional canon rather than objective or complete movement membership because association reflects historical collecting, exhibition, and digitisation biases.Kahlo was excluded from Surrealism because her membership is contested, while Newman, Still, and Hofmann replaced unavailable canonical Abstract Expressionists.
4 Method
The method encodes each painting once with frozen, L2-normalised image embeddings, then evaluates style using full-pool and artist-disjoint 5-NN protocols. An artist-recognisability control separately measures whether style performance may reflect recognition of individual painters.
- Embeddings: Each painting is encoded once by a frozen pretrained image encoder, with gradients disabled and features L2-normalised so cosine similarity becomes an inner product.CLIP ViT-B/32 is the reference encoder unless otherwise stated; the full pipeline is repeated with additional encoders.
- Full-pool style classification: Full-pool 5-NN assigns each of 320 works the majority style among its five nearest neighbours, excluding the query itself.Ties use the closest neighbour and then alphabetical order; accuracy is stable for k ∈{3, 5, 7, 9}, and mean silhouette scores provide a classifier-free measure.
- Artist-disjoint style classification: Leave-one-artist-out classification holds out each of 32 artists, using their ten works as queries and the remaining 310 works as the reference pool.No query is classified using another work by the same painter; the exhaustive deterministic protocol yields eight accuracy figures per movement and reports their standard deviation.
- Artist-recognisability control: The artist-recognisability control predicts artist identity rather than style while permitting same-artist neighbours, measuring recognition against the 1/32 chance rate.Results include overall accuracy, movement means, per-artist values, and comparison with the same artists’ artist-disjoint style accuracies.
5 Results
Artist-disjoint evaluation reduces 5-NN style accuracy from 0.869 to 0.766, with the decline concentrated in Surrealism rather than shared uniformly across movements. Strong artist recognisability and movement robustness are not simply inversely related, and the pattern persists across four encoders, including vision-only DINOv2.
- Artist-disjoint style classification: 0.869 to 0.766: Overall 5-NN style accuracy falls under artist-disjoint evaluation, with Impressionism and Cubism barely changing while Surrealism falls 20.0 points.The protocol holds each of the 32 artists out in turn.
- Movement-specific robustness: 0.1 and 0.2: Miró and Masson have the lowest Surrealism style accuracies when their own works are removed, while the other six Surrealists remain between 0.5 and 0.8.Surrealism’s high variance is driven by a minority of artists rather than evenly distributed across the movement.
- Artist recognisability: 0.466 against 1/32 = 0.031 chance: 5-NN artist classification shows that individual painter identity is strongly represented in the embeddings.Same-artist neighbours are permitted for this artist-recognition measurement.
- Artist recognisability: Pearson r = 0.06: Artist recognisability is essentially unrelated to style robustness across the 32 artists.The results include recognisable artists that classify well or collapse, and unrecognisable artists that classify well or perform weakly.
- Movement-specific robustness: 0.71 and 0.85: Abstract Expressionism has the highest mean artist-recognisability but remains fairly robust under hold-out, showing that distinctive artists and coherent movement signals can coexist.Pollock, Rothko, and Newman each classify perfectly under hold-out, whereas Gorky reaches 0.20 style accuracy.
- Cross-encoder validation: 0.513, 0.638, 0.375, and 0.500: Surrealism is the most fragile movement under artist-disjoint evaluation for all four encoders.The pattern persists across CLIP, DINOv2, and supervised ResNet-50; DINOv2 reproduces the collapse despite using images alone.
6 Discussion
The discussion argues that style-classification accuracy can reflect artist recognition rather than shared stylistic form, with artist-disjoint evaluation revealing substantial and uneven overstatement. It also cautions that comparisons should rely on within-encoder movement ordering because absolute accuracies are not comparable across encoders.
- Movements are separable for different reasons: 0.87 conceals a range from 0.51 to 0.90 once artists are held out, showing that aggregate style accuracy masks movement-specific robustness.Impressionism and Cubism survive artist hold-out more strongly than Surrealism.
- Shared form versus artist lookup: Cubism combines the lowest artist-recognisability with among the highest hold-out robustness, contradicting the idea that style classification is merely artist classification.The encoder cannot reliably identify the painter yet still places the work correctly.
- Movements differ in what unifies them: The same movement ordering across every tested encoder, including a vision-only model, indicates that the pattern reflects the movements rather than one model or language.The discussion links this result to visual differences between Impressionism and Cubism and Surrealism’s looser shared appearance.
- Implications for evaluation: Random-split evaluation can overstate style understanding by up to twenty points for one movement, with uneven overstatement hidden by aggregate performance.Allowing same-artist works on both sides risks certifying partly artist recognition as style understanding.
- Two cautions: Absolute accuracies are not comparable across encoders, so the analysis relies on movement ordering within each encoder.The discussion notes a departure for the ImageNet-trained ResNet-50, which classifies Abstract Expressionism far less robustly than transformer encoders.
7 Limitations
The study’s conclusions are limited by the small, curated dataset, uncontrolled content, inherited labeling and canon biases, preprocessing distortions, and the narrow frozen-embedding nearest-neighbour probe. Accordingly, reported accuracies concern this evaluation setting rather than style in isolation or the models’ full representational limits.
- Scale and balance: 320 works across 32 artists make absolute accuracies specific to this curated, balanced dataset rather than representative of broader encoder performance.A single held-out painter can move a per-style figure by as much as 0.125, so the study reports spread across held-out artists alongside means.
- Scale and balance: 0.125 is the maximum movement a single held-out painter induces in a per-style figure, motivating spread reporting alongside means.Equal artist and work counts per movement aid interpretation, but do not make the absolute accuracies broadly generalisable.
- Content is not controlled: Unmatched subject matter and composition leave style entangled with content, so per-movement accuracies do not measure style in isolation.The confound applies across movements and cannot by itself explain why holding out artists costs Surrealism twenty points and Impressionism four.
- Labels, canon, and preprocessing: Crowd-sourced labels, a conventional Western and male-skewed canon, and anisotropic resizing limit the dataset’s neutrality and may affect geometry-dependent movements differently.The resizing distorts aspect ratio uniformly across movements and encoders, but differential effects cannot be ruled out.
- Scope of the probe: The probe uses frozen image embeddings as fixed feature extractors with nearest-neighbour classification, excluding fine-tuning, intermediate layers, and text towers.The claims therefore concern off-the-shelf image embeddings under nearest-neighbour comparison, not the models’ limits after adaptation.
8 Conclusion
Artist-disjoint evaluation shows that frozen vision embeddings’ high style-classification accuracy partly depends on artist overlap, with the effect varying sharply across movements. The paper therefore argues that holding out whole artists is necessary to test stylistic understanding.
- Conclusion: 0.87 to 0.77: holding out whole artists lowers 5-NN style accuracy, despite high accuracy under artist-overlapping splits.Standard splits allow works by the same artist on both sides, enabling artist recognition to support style classification.
- Conclusion: Twenty points: Surrealism’s accuracy falls sharply, while Impressionism and Cubism are essentially unaffected by artist-disjoint evaluation.An artist-recognisability control accounts for this unevenness.
- Conclusion: Artist-disjoint evaluation is necessary because splits allowing artist overlap test something other than style, while holding whole artists out blocks that route.The paper identifies larger, less curated datasets; more movements and artists; alternative classifiers; content-controlled subsets; text-encoder probing; and fine-tuning as natural extensions.