Source-linked AI summary
ArtEmis: Affective Language for Visual Art
Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, Leonidas Guibas
TL;DR
Existing computer-vision datasets largely emphasize objective image content, leaving the relationship between visual stimuli, emotional experience, and linguistic explanations underexplored. ArtEmis addresses this gap with a large-scale artwork dataset pairing emotions with grounded explanations and trains models for emotion prediction and affective captioning. The corpus is richer in affective and abstract language than existing datasets, while the neural speakers can produce plausible grounded explanations, though their quality remains far below human ground truth.
Problem
Existing computer-vision annotations emphasize visible objects and actions, whereas the link between artworks, emotional experience, and linguistic affective explanations is underexplored.
Method
The authors collect ArtEmis, pairing artworks with human emotions and explanations, then train emotion-prediction models and neural speakers for affective captioning.
Results
ArtEmis contains explanations that are significantly more affective, abstract, and rich in metaphors and similes than existing datasets, while neural speakers produce plausible grounded emotion explanations.
Takeaways & Limitations
Computationally modeling emotional attributes of images opens a direction for more nuanced emotional image understanding and human-computer communication.
Takeaways & Limitations
The best neural speakers remain far below human ground truth in the diversity, accuracy, and creativity of synthesized utterances.
Abstract
from arXiv · showhide
We present a novel large-scale dataset and accompanying machine learning models aimed at providing a detailed understanding of the interplay between visual content, its emotional effect, and explanations for the latter in language. In contrast to most existing annotation datasets in computer vision, we focus on the affective experience triggered by visual artworks and ask the annotators to indicate the dominant emotion they feel for a given image and, crucially, to also provide a grounded verbal explanation for their emotion choice. As we demonstrate below, this leads to a rich set of signals for both the objective content and the affective impact of an image, creating associations with abstract concepts (e.g., "freedom" or "love"), or references that go beyond what is directly visible, including visual similes and metaphors, or subjective references to personal experiences. We focus on visual art (e.g., paintings, artistic photographs) as it is a prime example of imagery created to elicit emotional responses from its viewers. Our dataset, termed ArtEmis, contains 439K emotion attributions and explanations from humans, on 81K artworks from WikiArt. Building on this data, we train and demonstrate a series of captioning systems capable of expressing and explaining emotions from visual stimuli. Remarkably, the captions produced by these systems often succeed in reflecting the semantic and abstract content of the image, going well beyond systems trained on existing datasets. The collected dataset and developed methods are available at https://artemisdataset.org.
1. Introduction
ArtEmis studies how visual artworks evoke emotions and how people explain those emotions in language. It introduces a dataset and models designed to capture subjective, abstract, and grounded affective responses beyond objective image content.
- Why visual art?: Visual art is used because it is often created to provoke emotion and can resist simple subject-based descriptions.
- ArtEmis links visual artworks with human emotional reactions and natural-language explanations of those emotions.
- Novelty of ArtEmis: The explanations connect visible attributes to psychological interpretations, abstract concepts, personal experiences, and metaphors; over 20% contain similes.
- Affective neural speakers: Using ArtEmis, the authors develop emotion-prediction models and neural speakers that produce grounded emotion explanations from visual stimuli.
- Subjectivity of responses: The corpus is more affective, abstract, and rich in metaphors and similes than existing datasets, while often retaining agreement and references to visible elements.
- Subjectivity of responses: ArtEmis preserves subjective variation: 61% of annotated artworks have at least one positive and one negative emotional reaction.
2. Background and related work
Prior captioning datasets and models largely describe objects or sentiments, while ArtEmis targets emotion explanations grounded in visual artworks.
- ArtEmis adds emotion-focused explanations to image captioning, emphasizing affective language grounded in visual stimuli.
- COCO-captions describes common objects in natural images, whereas other datasets collect discriminative references for colors or 3D objects.
- Existing captioning approaches commonly build on recurrent networks, LSTMs, and teacher forcing.
- Sentiment-oriented captioning uses either emergent sentiment representations or explicit positive-versus-negative supervision.
3. ArtEmis dataset
ArtEmis pairs diverse WikiArt images with emotion selections and free-text explanations, producing linguistically rich, subjective, and visually specific affective annotations.
- 81,446 WikiArt artworks from 1,119 artists span 27 styles and 45 genres, covering art from the fifteenth through twenty-first centuries.
- Annotators select one of eight emotions or “something-else,” then provide a detailed free-text explanation for their choice.
- 439,121 utterances from 6,377 annotators form a 36,347-word corpus collected through 10,220 aggregate annotation hours.
- Linguistic analysis: 15.8 words per caption and higher occurrence of nouns, pronouns, adjectives, verbs, and adpositions indicate rich language use.
- Linguistic analysis: Only 16.5% of ArtEmis captions are neutral by VADER, compared with 77.4% of COCO-captions.
- Emotion analysis: 61% of artworks receive both positive and negative annotations, while “something-else” accounts for 11.7% of responses.
- Linguistic analysis: ArtEmis explanations include abstract, subjective, sentimental, imaginative, and metaphorical language, including similes in over 20% of the corpus.
- User studies: 97.5% of sampled utterances were judged appropriate, and users identified the target artwork from an explanation 94.7% of the time.
4. Neural methods
The neural methods combine auxiliary emotion classifiers with captioning architectures and ArtEmis supervision to generate visually grounded emotional explanations.
- Auxiliary classification tasks: A 9-way text classifier predicts the emotion associated with an ArtEmis explanation using an LSTM or fine-tuned BERT.
- Auxiliary classification tasks: An ImageNet-pretrained ResNet-32 predicts empirical distributions of artwork-induced emotions by minimizing KL-divergence.
- Baselines: The ANP baseline trains on COCO captions and injects sentiment-bearing adjectives selected using the image emotion classifier.
- Neural speakers: SAT and M2 provide the principal speaker backbones, combining visual encoders with attentive LSTMs or meshed-memory transformers.
- Neural speakers: Emotion-grounded SAT speakers receive the annotator-selected emotion as an additional feature at each decoding step.
- Implementation: Models share encoders, schedules, and LSTM dimensions; ArtEmis uses an 85%/5%/10% train-validation-test split and beam size 5 for generation.
5. Evaluation
Evaluation combines linguistic similarity, novelty, affective properties, emotion alignment, and human judgments to assess ArtEmis neural speakers.
- Auxiliary classifiers are evaluated with average accuracy, while speakers are assessed across three complementary metric categories.
- Speaker metrics measure similarity to held-out human captions, generation novelty, simile frequency, and alignment between generated and majority emotions.
- CIDEr-D is considered unsuitable because ArtEmis annotations are diverse and inherently subjective.
- User studies evaluate whether generations appear human-like, addressing the limitations of algorithmic metrics for subjective affective captions.
6. Experimental results
The experiments show that fine-grained emotion inference is difficult, while emotion-grounded neural speakers produce more emotionally aligned and human-like explanations than simpler baselines. However, their diversity, accuracy, and creativity remain far below human ground truth.
- Emotion prediction: 63.1% and 65.7% accuracy were achieved by LSTM and BERT, respectively, when predicting fine-grained emotions from text, compared with 53.0% for AMT users and 60.3% for experts.For binary positive-versus-negative sentiment, experts, LSTM, and BERT reached 85.9%, 87.4%, and 91.0%, respectively.
- Emotion prediction: 60.0% accuracy was attained when image classifiers predicted the dominant emotion for test images with a unimodal emotion distribution.This evaluation covered 38.5% of the split, and the data were highly unbalanced across emotion labels.
- Neural speakers: ArtEmis-trained Basic and Grounded speakers outperform simpler baselines on linguistic metrics, while the emotion-grounded variant performs significantly better on emotion alignment.The Grounded variant also produces a more appropriate proportion of similes than the non-grounded alternatives.
- Neural speakers: The emotion-grounded speaker generates sophisticated explanations incorporating nuanced emotional understanding and visual analogies on unseen artworks.Examples include analogies such as “looks like blood” and nuanced descriptions such as “sad and lonely” and “expressive eyes.”
- Turing test: The emotion-grounded variant surpassed or tied human annotations more often than the basic speaker in the AMT Turing test, which surpassed or tied them 40% of the time.The authors hypothesize that emotion grounding produced more common and fitting explanations that were easier to judge as human-made.
- Limitations: The best neural speakers remain far from human ground truth in the diversity, accuracy, and creativity of their synthesized utterances.The authors identify bridging this gap as requiring significant further research.
7. Conclusion
The paper introduces ArtEmis to study how visual content relates to affect through language and demonstrates neural speakers that express emotions with associated explanations. It positions computational handling of emotional image attributes as a direction for human-computer communication and interaction.
- Conclusion: ArtEmis connects human emotions and natural-language explanations of affective responses triggered by visual artworks.The dataset is intended to support analysis of how image content relates to affect.
- Conclusion: The paper releases ArtEmis and demonstrates neural speakers that express emotions and provide associated explanations.These contributions represent the paper’s first step toward data-driven emulation of human emotional responses.
- Conclusion: Computationally handling images’ emotional attributes opens a direction for human-computer communication and interaction.