Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style

Motivation

• VLM style classification is difficult because artworks combine fine-grained content and formal features, while models may rely on pretraining patterns rather than faithfully reasoning over the input.The motivating example highlights scene content alongside engraving and crosshatching as potentially relevant to Northern Renaissance classification.Source: §1 · p. 1

Research Questions

• The study evaluates whether VLM concepts answer three questions: what visual concepts drive style predictions, whether they are causally related, and whether they align with art historians.These questions cover model mechanisms, causal influence, and human-aligned interpretation.Source: §1 · p. 1

Approach

• The method jointly decomposes full images and constituent patches, grounding broad image motifs in interpretable patch-level concepts that distinguish content from form.The approach adapts prior full-image concept decomposition and uses patch concepts to explain image-level style predictions.Source: §1 · pp. 1–2, Figure 1 · p. 2

Expert Evaluation

• Art historians assessed extracted concepts for semantic and stylistic quality, providing expert analysis of whether model-identified features correspond to meaningful content and form dimensions.The six-person collaboration examined features such as objects, scenes, color palette, texture, and lighting.Source: §1 · p. 2

Findings

• 73% of the model’s top associated concepts were judged relevant to style prediction, indicating considerable agreement while also revealing divergences in the visual cues used by models and historians.The analysis treats disagreement as informative about model cues that may fall outside conventional art-historical categorization.Source: §1 · pp. 2–3

Art Style Recognition

• Artistic-style recognition has progressed from handcrafted color features to learned local and global CNN representations, improving classification accuracy over time.The cited literature spans early color histograms and later convolutional approaches.Source: §2 · p. 3

Interpretability Gap

• The paper’s novelty is to compare the mechanisms underlying VLM style predictions directly with art historians’ judgments, rather than evaluating accuracy alone.This reframes the central question from whether VLMs classify style to how they do so.Source: §2 · p. 3

Interpretability Methods

• The approach builds on interpretability methods including linear probing, activation patching, dictionary learning, sparse autoencoders, and Semi-NMF for decomposing dense activations into human-understandable concepts.Parekh et al. (2024) previously applied Semi-NMF to VLM object concepts; this work extends the approach to art-style prediction.Source: §2 · p. 3

Evaluation Scope

• The experiments use artwork and architectural-style datasets, reflecting the paper’s comparison of VLM mechanisms across artistic and architectural domains.The supplied dataset caption identifies Table 1 as summarizing datasets and associated style categories.Source: Table 1 · p. 4

Data

• The study uses three five-category datasets: two WikiArt collections with 2,500 images each and an architecture dataset with 1,500 images sampled evenly across styles.Each image is divided into 16 patches, producing 40,000 and 24,000 patch samples for the art and architecture data respectively.Source: §3.1 · p. 4

Concept Decomposition

• Semi-NMF decomposes VLM residual-stream representations into a learned concept dictionary and nonnegative image activations, with sparsity encouraging only a few active concepts per image.The representation matrix contains columns corresponding to latent states used to predict styles, and the decomposition approximates it as Z ≈ UV.Source: §3.2 · p. 4

Patch Concepts

• Images are split into a 4x4 patch grid before decomposition because localizing features makes concepts easier to interpret in detailed artworks.The method also experiments with representation layers and concept counts, selecting configurations using linear-probe accuracy.Source: §3.2 · p. 4

Concept Representation

• Concepts are represented by the top 24 image patches with maximal activation and can additionally receive automatically generated textual labels.Prototype patches provide visual evidence for what each discovered concept represents.Source: §3.2 · pp. 4–5

Style Associations

• A linear classifier uses concept activations to predict the VLM’s style output, and its positive weights identify concepts most associated with each style.This probing step measures concept-style associations without using the original image directly.Source: §3.3 · p. 5

Causal Analysis

• The causal analysis masks the four patches with the highest target-style classifier weights and compares their effect with random or alternative masks, expecting targeted masking to reduce style probability most.The intervention assumption is expressed as ∆1,y < ∆2,y < ∆3,y for the specified intervention ordering.Source: §3.4 · p. 5

From Patches to Images

• PMI links an activated full-image concept to distinctive patch concepts by ranking patch co-occurrences, then retrieves the highest-PMI patches for a test image’s strongest image-level concept.The mapping addresses the mismatch between simple local patches and full-resolution artworks.Source: §3.5 · p. 5

4.1 Model Benchmarking

• Qwen3 and GPT5 achieve the best reported art-style classification accuracies, whereas Molmo2 and LLaVA-1.5 show a notable drop-off.Models perform better on architecture than WikiArt datasets, while a five-style control set produces strong performance, suggesting closely related WikiArt styles are harder to distinguish.Source: §4.1 · p. 5

4.2 Concept Decomposition

• Across layers, style-specific concepts emerge, but some align with model predictions rather than ground-truth labels, especially for overlapping Romanticism and Realism examples.Renaissance, Rococo, and Baroque concepts align with ground truth in one region, while other concepts are associated with Romanticism despite Realism annotations.Source: How do models associate concepts with… · p. 6, Figure 3 · p. 6

4.3 Concept-Style Associations

• Later-layer concept activations predict the model’s output style more accurately, reaching 0.95 linear-probe accuracy at the final layer.The probe uses concept activations from image patches to predict the model’s style output on the Early Modern WikiArt subset.Source: How do models associate concepts with… · p. 6, Figure 4 · p. 7
• The linear probe’s weights identify concepts correlated with each style, and these associations are additionally validated through latent-direction interventions.The masking analysis tests whether the concepts identified from probe weights affect full-image style prediction.Source: §4.3 · pp. 6–7
• Targeted concept masking causes the greatest drop in correct-style log-probability across settings, exceeding random and random non-overlapping masks.Differences between concept and random masks are statistically significant in almost all cases, supporting the importance of the identified concepts.Source: §4.3 · p. 7, Figure 5 · p. 7

5.2 Expert Analysis

• The expert assessment used artwork, model predictions, and three associated concepts, with top-associated and randomly selected concepts compared under a blind design.Six art historians ranked concept relevance and provided style predictions; agreement on style labels was moderate, with Fleiss’ kappa 0.52 and mean Cohen’s kappa 0.55 against WikiArt.Source: §5 · p. 7, §5.2 · p. 9, §5.2.1 · p. 9

5.1 Intrinsic Evaluation: Concept Quality

• 73% of 128 extracted concepts received a majority coherence score of at least 3, with moderate interannotator agreement (Krippendorff’s alpha = 0.52).The experts assessed semantic and stylistic coherence using representative image patches.Source: §5.1 · p. 7, §5.1.1 · p. 8

5.1.1 Results

• Patch-level concepts were usually organized by form or content, whereas image-level concepts were more often labeled as stylistic themes.Many patch concepts combined form, content, and style, while only a few lacked a clear human-interpretable basis.Source: §5.1.1 · p. 8, Figure 6 · p. 8

5.1 Intrinsic Evaluation: Concept Quality

• GPT5.4-as-a-Judge’s summary-based labeling method achieved a 92% win rate against other models, compared with 21% for the baseline.The evaluation used art historians’ labels as the reference for comparing automatic concept-labeling methods.Source: Alignment between expert and automatic… · p. 8

5.2.1 Results

• Only 17% of top-associated concepts were judged absent from the artwork, compared with 70% of randomly selected concepts.Among model-associated concepts, 29% were rated irrelevant to the predicted style and 33% irrelevant to the historians’ own style prediction; random concepts were mostly irrelevant at 93% and 96%.Source: §5.2.1 · p. 9
• Art historians identified concept multiplicity and content bias as key sources of divergence between expert and model style recognition.Concepts could combine form and content and be reused across styles, while content such as trees and forests could be over-associated with Romanticism.Source: §5.3 · pp. 9–10

6 Discussion and Conclusion

• The paper contributes a method for extracting interpretable visual concepts from latent representations and a framework for comparing VLM and art-historical style reasoning.The analysis addresses concept identification, causal grounding, and alignment with expert judgments.Source: §6 · p. 10
• Activated concepts were strongly correlated with model predictions, reaching 95% accuracy in the final layers of Qwen3.Masking top-associated concepts reduced style-prediction accuracy significantly more than masking random concepts.Source: §6 · p. 10
• Art historians judged the model’s top-associated concepts relevant to style prediction in 73% of analyzed cases.Misalignments showed that models can combine multiple concept aspects and rely on patterns that differ from art historians’ visual reasoning.Source: §6 · p. 10

7 Limitations and Future Work

• The expert analyses were limited in scale, and larger studies with additional relevant patch concepts could deepen interpretation of model–art historian differences.The authors also note that some patch concepts shown to experts were not directly present in the image.Source: §7 · p. 10
• Future work could use causal analysis to identify important patches, although this approach is computationally intensive.This is proposed as an alternative to the two-step image–patch concept mapping process.Source: §7 · p. 10
• A larger-scale study of automatic labeling with human evaluation is needed for more rigorous validation of candidate approaches.The paper presents this as a future direction rather than a completed evaluation.Source: §7 · p. 10

Effective Labels

• Effective Labels ranges from 1 to 5 and measures how purely concepts align with style labels across their activating images.A value of 1 means all activating images share one style, whereas 5 indicates uniform mixing.Source: §A.2 · p. 14

Sparsity

• Sparsity ranges from 0 to 1 and measures how selectively concept activations contribute to each image representation.Zero means all concepts contribute equally; one means a single concept dominates.Source: §A.2 · p. 14

Reconstruction Error

• Reconstruction error is the mean per-sample relative ℓ2 error between original representations Z and reconstructions UV.Lower values indicate that the concept dictionary more faithfully recovers the original representations.Source: §A.2 · p. 14

Automatic Labeling Methods

• Four automatic labeling methods are compared: BASELINE, SUMMARY, IMAGE-SIM, and CONTRASTIVE.They respectively use Logit Lens tokens, descriptions of highly activated images, CLIP-selected candidate labels, and negative sampling against nearby concepts.Source: B Concept Labeling Methods · p. 14

SUMMARY

• SUMMARY selects the top 20 activated images, generates art-historical descriptions, and combines them into one label for the concept.The descriptions focus on content and stylistic properties such as color, texture, composition, lighting, and pose.Source: B Concept Labeling Methods · pp. 14–15

IMAGE-SIM and CONTRASTIVE

• IMAGE-SIM generates up to five candidate labels and selects the one with the highest average CLIP cosine similarity to the selected images.CONTRASTIVE repeats this procedure while penalizing similarity to images from the five nearest concepts.Source: B Concept Labeling Methods · p. 14

Causal Analysis

• The causal analysis measures style log-probability before and after removing top activated concepts or masking their corresponding image patches.Hidden-state interventions subtract a concept direction, while image experiments compare targeted, random, and non-overlapping random masks; targeted masks identify regions such as ships, water, and birds.Source: §C.1 · p. 15, §C.2 · p. 15

Human Evaluation

• Human studies show concepts through grids of highly activating patches and ask annotators to describe their coherence and relevance to style predictions.Study 1 evaluates intrinsic concept quality; Study 2 evaluates whether concepts are reflected in images and relevant to participants’ predictions before and after revealing the model’s answer.Source: Figure 12 · p. 16, Figure 13 · p. 17

Accuracy Benchmarking

• Per-style and overall VLM accuracies are reported across models and datasets, with Table 3 distinguishing best, second-best, and overall results.For LLaVA-1.5, high accuracy can reflect prediction bias toward particular styles.Source: Table 3 · p. 19, F Model Benchmarking · p. 19

Configuration Analysis

• The study varies extraction layer, concept count K, and activation-binarization threshold to evaluate decomposition quality and active concepts per image.Tables 4–6 cover these configuration effects and lower-K settings for Wikiart (early modern).Source: §G.1 · p. 19, §G.2 · p. 19, Table 4 · p. 19

Concept Predictiveness

• Linear probing measures how well concept activations predict model output style across the Architecture and Wikiart datasets.Figures 16 and 17 report patch-level style-prediction results, using raw and binarized activations for Architecture.Source: Figure 16 · p. 19, §G.2 · p. 19, Figure 17 · p. 20

Configuration Selection

• K = 32 is selected for one setting because it covers the dataset’s main image motifs and 94.7% of SVD variance.This choice provides a compact representation while retaining most of the reported variance.Source: §G.3 · p. 20

Aggregate Evaluation

• All three automatic labeling methods outperform BASELINE, with SUMMARY the clear winner in pairwise judgments against art-historian references.The evaluation covers all 128 visual concepts and compares relative precision, recall, and overall quality.Source: H Concept Labeling Evaluation · p. 20, Table 7 · p. 20

Specificity

• SUMMARY’s advantage is driven by increased specificity, which often captures detailed stylistic descriptions more precisely.For a coherent concept, a prompt label describing hazy skies, loose brushwork, muted tones, and soft gradients better captured the reference label.Source: H Concept Labeling Evaluation · p. 20

Limitation

• When a concept is incoherent, the same specificity can produce an overly elaborate but incorrect label.For concept 53, art historians found no discernible correlation, whereas the generated label asserted dimly lit interiors, chiaroscuro, backlighting, and loose texture.Source: H Concept Labeling Evaluation · p. 20

Recommendation

• The authors recommend relying on automatic labels when the underlying concepts are coherent.This recommendation follows from the contrast between specificity’s benefits for coherent concepts and its errors for incoherent ones.Source: H Concept Labeling Evaluation · p. 20

• Figure 18 relates each concept’s linear probe weight for a target style to the mean causal effect of removing that concept.Source: I Patch-Level Causal Effects · p. 20
• Concepts with higher probe weights causally suppress the model’s prediction of the target style when removed.Source: I Patch-Level Causal Effects · p. 20
• Spearman’s ρ = −0.714 (p = 1.13 × 10−100) indicates a strong negative correlation between probe weights and removal effects.Source: Figure 18 · p. 20