Source-linked AI summary
Beyond Vision-Language Models: Visual-Semantic Affect Graphs for Emotion Recognition in Artworks
Ngoc-Duy Mai, Ha-Thanh Nguyen, Hideaki Takeda
TL;DR
Artwork emotion recognition must relate formal and semantic cues to affective response, while direct image-to-emotion prediction may leave those relations implicit. The paper proposes Visual-Semantic Affect Graphs to model those cues explicitly, and reports that hierarchical valence-arousal VSAG performs best overall on EmoArt-5k. The results also indicate that the benefit of valence-arousal structure depends on graph aggregation strategy.
Problem
Artwork emotion recognition must connect formal and semantic cues to affective response, which is subjective and often implicit in visual content.
Source: §1 · p. 2Method
VSAG represents artworks with interpretable visual-semantic feature nodes and learns their associations with emotion labels, optionally using hierarchical valence-arousal modeling.
Source: §1 · p. 2, §3.4 · p. 8Results
On EmoArt-5k, VSAG+VA achieved the highest accuracy, macro-F1, and weighted-F1 among evaluated models, improving standard VSAG but not VSAG-TopK.
Source: §5 · p. 13, §6 · p. 14Takeaways & Limitations
VLMs and graph-based models capture partly complementary signals, while VSAG's interpretability depends on feature quality and coverage.
Source: §5 · p. 13The graph primarily uses visual-semantic features and omits contextual information such as artist, year, medium, style period, and cultural background.
Source: §5 · p. 13
Abstract
from arXiv · showhide
Emotion recognition in artworks remains challenging because affective meaning is often subjective, implicit, and shaped by relationships between visual form and semantic associations. While vision-language models provide strong multimodal representations, direct image-to-emotion prediction may overlook explicit relations among formal, semantic, and affective cues that support emotional interpretation. This paper proposes Visual-Semantic Affect Graphs (VSAG), a graph-based framework that represents artworks through interpretable feature nodes, including color, brushstroke, composition, light-shadow, line quality, and extracted visual entities. The graph learns feature-emotion associations for dominant emotion prediction, with valence-arousal structure modeled through an optional hierarchical auxiliary affect module. We evaluate VSAG and VSAG-TopK against direct VLM inference using Qwen3-VL and InternVL3.5 on EmoArt-5k. Results show that direct VLMs provide useful but modest baselines, while graph-based models improve dominant emotion prediction. In particular, the hierarchical valence-arousal variant of VSAG achieves the best overall performance, improving accuracy, macro-F1, and weighted-F1 over both direct VLM inference and the emotion-only VSAG baseline. Per-class results indicate that graph-based models and VLMs capture partly complementary signals, while valence-arousal structure is most useful when integrated as a coarse-to-fine affect hierarchy rather than as a simple post-hoc correction. These findings suggest that explicit visual-semantic structure can support both prediction and interpretation in artwork emotion recognition.
1 Introduction
Artwork emotion recognition must connect visual form and semantic associations to affective response, not stop at recognizing depicted content. The paper tests whether explicit visual-semantic graphs improve emotion prediction over direct VLM inference.
1 Introduction
- Emotion recognition in art requires connecting visual form and semantic associations to affective response, beyond recognizing depicted objects or motifs.Relevant cues include color, composition, texture, rhythm, light, abstraction, and symbolism.Source: §1 · p. 2
- VLMs can struggle with implicit, context-dependent affect, including prompt sensitivity, frequent-label bias, and confusion between rare and neighboring emotions [11] [2].Strong general visual-language alignment may not suffice for robust emotion understanding.Source: §1 · p. 2
- Artwork emotion may depend on formal and aesthetic features such as color, texture, composition, light, line, and brushwork, as well as semantic content.The cited accounts frame art perception as involving emotion, valuation, memory, and interpretation rather than visual recognition alone.Source: §1 · p. 2
- The paper compares graph-based visual-semantic representations with direct VLM inference and asks which attributes associate most strongly with particular emotions.Its framework represents attributes such as color, brushstroke, composition, light, line quality, and visual entities, with optional hierarchical valence-arousal modeling.Source: §1 · p. 2
2 Related Work
Related work places artwork emotion recognition at the intersection of affective modeling, VLM-based understanding, and structured visual-semantic representations. It identifies whether graph learning over artwork attributes improves emotion prediction as an open question.
2 Related Work
- Artwork emotion combines categorical and dimensional affect, and datasets including WikiArt Emotions, ArtEmis, and FeelingBlue support its study.The section characterizes emotion prediction as interpretive, involving visual form, semantic association, and affective response.Source: §2 · p. 3
- Strong general VLM performance does not guarantee reliable affective understanding, which can be sensitive to prompts and label formulations.The text highlights this concern for art, where emotional meaning may depend on relations among color, form, composition, and symbolic content.Source: §2 · p. 3
- Scene graphs, knowledge graphs, and art resources such as ArtGraph and ArtKB make visual or artwork structure explicit, but graph-based emotion prediction remains unclear.The paper addresses this question by constructing an affect-focused artwork-feature graph and comparing it with direct VLM inference.Source: §2 · p. 3
3 Method
VSAG treats artwork emotion prediction as structured prediction over interpretable visual-semantic features. It converts formal attributes and depicted entities into feature nodes, then uses those representations for emotion prediction, with an optional valence-arousal hierarchy.
3 Method
- VSAG represents each artwork through interpretable visual-semantic features before learning how those features support dominant emotion prediction.The +VA variant predicts a four-way valence-arousal group and a within-group emotion distribution; VSAG-TopK+VA instead uses these outputs as auxiliary regularization while decoding directly.Source: §3 · p. 4, Figure 1 · p. 4
3.1 Problem Setup and Graph Construction
- Each artwork is represented using visual attribute descriptions and a general image description, with dominant emotion as the main target and valence-arousal group as an optional auxiliary target.The framework describes its inputs and target labels in the graph-construction setup.Source: §3.1 · p. 4
Attribute Categorization
- Predefined lexical rules convert brushstroke, color, composition, light-shadow, and line-quality descriptions into controlled, interpretable feature nodes.Fallback categories include Other and Unknown / Not Applicable when no rule matches or an attribute is absent.Source: Attribute Categorization · p. 5, Table 1 · p. 5
Entity Extraction
- A text LLM extracts concrete depicted entities or visual resemblances from artwork descriptions, while excluding formal visual-analysis terms.Post-processing removes generic or formal-analysis terms and deduplicates repeated entities.Source: Entity Extraction · pp. 5–6
Graph Definition
The graph links artwork nodes to visual-semantic feature nodes and affect-label nodes. Its emotion-only and hierarchical variants differ in how affect labels are represented.
Graph Definition
- The graph contains artwork, visual-semantic feature, and affect-label nodes; the hierarchical variant adds valence-arousal group and within-group emotion labels.The emotion-only model uses dominant emotion nodes.Source: Graph Definition · p. 6
- An artwork-feature edge indicates that the feature is observed in that artwork, defining the artwork’s feature neighborhood.The model learns feature-label associations from these observed edges.Source: Graph Definition · p. 6
- Predictions use heads such as direct emotion, valence-arousal group, or within-group emotion, with candidate labels for each task decomposed into explicit graph paths.Source: Graph Definition · p. 6
3.2 Visual-Semantic Affect Graph
VSAG predicts affect by aggregating explicit artwork-feature-label paths, with learned feature-label associations and label-conditioned attention. An optional residual can correct scores, while explicit graph paths remain the primary prediction mechanism.
3.2 Visual-Semantic Affect Graph
- VSAG initializes learnable feature-label associations with smoothed log-odds scores estimated from the training set.The scores are updated during training.Source: §3.2 · p. 6
- For each candidate label, VSAG assigns attention weights to observed features and aggregates their artwork-feature-label paths into a graph score.The attention is label-conditioned and the graph score aggregates the paths.Source: §3.2 · pp. 6–7
- An optional residual adjacency encoder supplies a correction, while explicit graph-path scoring remains the primary prediction mechanism.The residual strength is controlled by λ.Source: §3.2 · p. 7
- VSAG assumes affective meaning is distributed across observed visual-semantic features, whose contributions depend on edge scores and label-conditioned attention.Source: §3.2 · p. 7
3.3 VSAG-TopK
VSAG-TopK retains the visual-semantic graph but aggregates only the strongest k feature paths for each candidate label. This variant assumes that a small number of informative cues may determine an artwork’s emotion.
3.3 VSAG-TopK
- The variant assumes that an artwork’s emotion may be determined by a small number of highly informative visual-semantic cues.Source: §3.3 · p. 7
- VSAG-TopK selects the strongest k observed feature paths for each candidate label instead of aggregating all observed paths.Its path score includes a learned feature-label edge score and a contextual compatibility component whose strength is controlled by δ.Source: §3.3 · p. 7
- VSAG and VSAG-TopK share a graph structure but differ in aggregation: all observed paths versus the strongest feature-label paths.Source: §3.3 · p. 7
3.4 Hierarchical Valence-Arousal Auxiliary Modeling
The hierarchical valence-arousal variant uses dataset-defined affect groups to structure dominant-emotion prediction. It can decode first to a group and then to an emotion within that group, rather than applying a simple additive correction.
3.4 Hierarchical Valence-Arousal Auxiliary Modeling
- The emotion labels are organized into valence-arousal groups defined by the dataset label scheme.Each dominant emotion maps to a group.Source: §3.4 · p. 8
- The model computes a direct emotion distribution, a valence-arousal group distribution, and within-group emotion distributions.Within-group distributions cover the emotions belonging to each group.Source: §3.4 · p. 8
- The hierarchical distribution first estimates an affective group and then selects an emotion within that group, narrowing the candidate emotions.For example, a Positive-Low group narrows choices to Calm, Contentment, and Glad.Source: §3.4 · p. 8
- VSAG+VA decodes from the hierarchical distribution, whereas VSAG-TopK+VA uses valence-arousal losses during training but decodes from the direct emotion head.Source: §3.4 · p. 8
3.5 Training and Inference
Training adds valence-arousal group and within-group supervision, with a consistency term for the hierarchical variant. Inference can use direct emotion scores or hierarchical decoding, and graph paths provide feature-level explanations.
3.5 Training and Inference
- The hierarchical +VA variant adds group, within-group, and consistency losses to the emotion-only dominant-emotion loss.The group loss supervises four-way group prediction; the within-group loss supervises emotion prediction within the gold group.Source: §3.5 · p. 8
- The model jointly optimizes direct emotion, valence-arousal group, and within-group emotion heads rather than using valence-arousal as a post-hoc additive correction.Source: §3.5 · p. 9
- At inference, emotion-only VSAG uses the highest direct emotion probability, while VSAG+VA decodes from the hierarchical distribution and VSAG-TopK+VA from the direct head.Source: §3.5 · p. 9
- Graph paths support explanations of which visual-semantic attributes support a predicted emotion or valence-arousal group.VSAG-TopK explanations use the highest-scoring paths selected by TopK pooling.Source: §3.5 · p. 9
4 Results
On EmoArt-5k, graph-based models improve dominant-emotion prediction, with hierarchical valence-arousal modeling in VSAG achieving the strongest overall results. The analysis also finds emotion-linked feature patterns, but these are descriptive associations rather than causal feature importance.
4.1 Dataset
- EmoArt-5k pairs abstract artworks with descriptions, visual-attribute annotations, and affective labels; dominant emotion is the prediction target, with valence-arousal groups used in auxiliary variants.The dataset is a publicly available subset of EmoArt.Source: §4.1 · p. 9
- The dataset is highly imbalanced: Calm accounts for more than half of artworks, while several classes are rare, motivating accuracy, weighted-F1, and macro-F1 reporting.The rare classes named include Glad, Aroused, Bored, Tired, and Annoyed.Source: §4.1 · p. 9
4.2 Emotion Prediction Performance
- The best direct VLM, InternVL3.5-8B, has accuracy and weighted-F1 only slightly above 0.5; larger parameter size does not guarantee better prediction.Qwen3-VL-8B performs worse in accuracy than its 2B and 4B variants.Source: §4.2 · pp. 10–11
- VSAG+VA achieves the best scores across accuracy, macro-F1, and weighted-F1, outperforming direct VLMs and emotion-only VSAG.The reported advantage applies to the hierarchical valence-arousal variant of VSAG; VSAG-TopK+VA underperforms VSAG-TopK.Source: Table 4 · p. 11, §4.2 · p. 11
- VSAG+VA improves Alarmed, Frustrated, and Sad over emotion-only VSAG, while rare classes remain difficult and VLMs are stronger for some classes.The authors describe the models as capturing partly complementary affective signals.Source: Table 5 · p. 11, §4.2 · p. 11
4.3 Feature-Emotion Association Analysis
- Alarmed and Frustrated associate with intense visual features, while Calm and Contentment associate with softer visual properties.Examples include strong contrast and dramatic lighting for Alarmed, rough or energetic brushwork for Frustrated, soft lighting and balanced composition for Calm, and warm colors for Contentment.Source: §4.3 · p. 12
- Feature-emotion scores are learned associations, not causal feature importance, and rare-class associations such as Glad warrant caution.The authors characterize the analysis as descriptive evidence of which feature groups are associated with each emotion label.Source: §4.3 · p. 12
- Feature-group associations are relatively balanced across emotions; color, light-shadow, brushstroke, and entity nodes are prominent for different labels.Figure 2 includes positive associations for feature nodes appearing in at least five training artworks and normalizes mean association strengths within each emotion.Source: §4.3 · p. 12, Figure 2 · p. 13
5 Discussion
VSAG and direct VLMs capture partly complementary signals, while valence-arousal structure helps standard VSAG but not VSAG-TopK. Interpretation remains limited by single-label supervision, rare classes, missing contextual information, and the scope of current graph-model comparisons.
5 Discussion
- Because each artwork has one dominant-emotion label, models match one annotation choice rather than the full range of viewer responses; graph scores still expose secondary emotions and feature-emotion paths.The discussion frames multi-label annotation as a possible better fit for artworks' range of affective responses.Source: §5 · p. 13
- Valence-arousal structure improves standard VSAG, mainly for several negative or less frequent classes, but this benefit does not transfer to VSAG-TopK.The discussion suggests VA structure is most useful with distributed feature-path aggregation, while TopK pooling may already emphasize dominant affective cues.Source: §5 · p. 13
- VLMs and graph-based models capture complementary signals, but VSAG's interpretability depends on the quality and coverage of its explicit visual-semantic features.The discussion contrasts VLMs' global impressions and broad semantic priors with VSAG's feature nodes.Source: §5 · p. 13
- Rare labels remain a challenge, with near-zero performance motivating investigation of class reweighting, focal loss, or graph-based sampling.The discussion also notes that multi-label annotation may better reflect the range of affective responses artworks evoke.Source: §5 · p. 13
- The current graph omits artist, year, medium, style period, and cultural background, and future work could add external art knowledge and evaluate against existing knowledge-graph methods.The evaluation primarily compares VSAG variants rather than independently developed graph-based approaches.Source: §5 · pp. 13–14
6 Conclusion
On EmoArt-5k, graph-based models improve dominant emotion prediction over direct VLM inference, and VSAG+VA achieves the highest accuracy, macro-F1, and weighted-F1 among evaluated models. The results also show that VA's benefit depends on aggregation strategy, while rare classes remain difficult.
6 Conclusion
- VSAG+VA achieves the highest accuracy, macro-F1, and weighted-F1 among evaluated models, while graph-based models improve dominant emotion prediction over direct VLM inference.Direct VLM inference provides a useful but modest baseline.Source: §6 · p. 14
- Hierarchical valence-arousal modeling improves standard VSAG but not VSAG-TopK, with per-class gains on several difficult negative emotions.Rare classes remain challenging because of severe label imbalance.Source: §6 · p. 14
- Feature-emotion analysis identifies interpretable associations between emotions and cues including contrast, brushwork, composition, lighting, and entities.Source: §6 · p. 14
- Future work could extend VSAG toward multi-label emotion modeling, richer art-historical metadata, and hybrid models combining VLM representations with interpretable graph structure.Source: §6 · p. 14