PaperlayerView exact revision on arXiv

Source-linked AI summary

Forging Emotions: a deep learning experiment on emotions and art

Amalia Foka

arXiv:2610.32163v1cs.CVcs.CYcs.HC

TL;DR

Affective computing has largely emphasized detecting emotion rather than generating visual affect, despite the need for images that convey or trigger specified emotions. Forging Emotions explores social-media emotion datasets and GANs, concluding that current dataset methods and simple color associations are inadequate and recommending association-based datasets and deeper GAN understanding.

  • Problem

    Affective computing has focused more on detecting emotions than generating or triggering them, while visual affect generation with GANs remains unexplored.

    Source: Introduction · p. 2, §1.2 · p. 3

  • Method

    The experiment creates Instagram image datasets using #sad and #happy without verification and explores GAN processes and their outputs for visual affect generation.

    Source: §2 · p. 4, §3.2 · pp. 6–7

  • Results

    The experiment concludes that current methods for building emotion image datasets are insufficient and that basic color–emotion correspondences do not seem adequate for visual affect generation.

    Source: Conclusion · p. 8, §3.3 · p. 8

  • Takeaways & Limitations

    The authors recommend emotional association networks for datasets and research into trained and compositional GAN structures to generate novel compositions that convey emotion through subject matter.

    Source: Conclusion · p. 8

    Single-word social-media searches cannot guarantee coverage of the range of emotional experience, and some sadness-related imagery may be removed under platform policies.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Affective computing is an interdisciplinary field that studies computational methods that relate to or influence emotion. These methods have been applied to interactive media artworks, but they have focused on affect detection rather than affect generation. For affect generation, computationally creative methods need to be explored that have recently been driven by the use of Generative Adversarial Networks (GANs), a deep learning method. The experiment presented in this paper, Forging Emotions, explores the use of visual emotion datasets and the working processes of GANs for visual affect generation, that is, for generating images that can convey or trigger specified emotions. This experiment concludes that the methodology used so far by computer science researchers to build image datasets for describing high-level concepts such as emotions is insufficient and proposes utilizing emotional networks of associations according to psychology research. Forging Emotions also concludes that to generate affect visually, merely corresponding to basic psychology findings, such as bright or dark colours, does not seem adequate. Therefore, research efforts should aim to understand the structure of trained GANs and compositional GANs in order to produce genuinely novel compositions that can convey or trigger emotions through the subject matter of generated images.

Resumen

Forging Emotions explores GANs and visual emotion datasets for generating images intended to convey or trigger specified emotions. It finds current dataset-building methods and reliance on basic color–emotion associations inadequate, and points to emotional association networks and GAN interpretation as research directions.

Resumen

  • The experiment explores visual emotion datasets and GAN processes to generate images intended to convey or trigger specified emotions.Source: Resumen · p. 2
  • The dataset-building methodology used by computer science researchers is insufficient for describing high-level concepts such as emotions.Source: Resumen · p. 2
  • Merely matching basic psychology findings, such as bright or dark colors, does not seem adequate for visual affect generation.Source: Resumen · p. 2
  • The paper calls for emotional association networks and better understanding of trained and compositional GANs to create novel compositions that convey emotion through subject matter.Source: Resumen · p. 2

Introduction

Emotions matter to interactive media artworks, where systems can orchestrate emotional experiences, but affective computing has focused more on detecting emotions than generating or triggering them. Forging Emotions examines visual emotion datasets and GAN processes as a route to visual affect generation, while questioning single-word dataset searches and emotion-label validation.

Introduction

  • Interactive artworks can use programmed feedback to orchestrate recipients’ emotional experiences, making affective computing pertinent to their design.Source: Introduction · p. 2
  • Affective computing research and interactive artworks have mainly used computational methods for emotion detection rather than generation or triggering.Source: Introduction · p. 2
  • The experiment explores visual emotion datasets and GAN processes, questioning whether single-word social-media searches and label validation represent emotional experience or convey specified emotions.Source: Introduction · p. 2

1. Background

The background surveys affective computing, GAN-based visual generation, and visual emotion datasets. It identifies a gap in computational affect generation and questions whether datasets assembled through social-media searches and emotion labels adequately represent how emotions are experienced.

1.1. Affective computing

  • Affective computing commonly detects emotions through sensors and modalities such as facial expressions, gestures, and speech; research on generation includes synthesized expressions and affective games, but games trigger emotions by design rather than computational methods.Source: §1.1 · p. 3
  • Research on generating affective triggers remains largely unexplored, including in interactive artworks using affective computing.Source: §1.1 · p. 3

1.2. Deep learning methods

  • Visual affect generation with GANs remains unexplored, although prior work used emotion-labelled paintings to generate images with features such as red for anger and dark colors for sadness and fear.Source: §1.2 · p. 3
  • GAN training datasets often contain visually similar examples for specific objects or concepts, leaving open how to build a suitable dataset for higher-level concepts such as emotion.Source: §1.2 · p. 3

1.3. Visual emotion datasets

  • Emotion researchers hand-select images for datasets such as IAPS, GAPED, and NAPS, while computer-science datasets include user-labelled or participant-rated images such as IAPSa, DeviantArt photographs, and OASIS.Source: §1.3 · p. 4
  • Computer-science researchers commonly collect social-media images through single-word emotion searches and use crowd validation, with dataset scale often framed as the remaining task.Source: §1.3 · p. 4

2. Forging Emotions

Forging Emotions questions whether single-word social-media tags capture emotions as people experience them and creates unverified #sad and #happy datasets. It also explores GAN outputs and black-box analysis as ways to examine what the models learn.

2. Forging Emotions

  • Psychology research frames an emotion as a network of associations with past experiences, and these networks are individual while sharing a common basic structure.Source: §2 · p. 4
  • The experiment creates Instagram datasets from #sad and #happy without verifying the collected images, questioning single-word searches and MTurk validation.Source: §2 · p. 4
  • The study tests whether GANs can compose images that visualize or evoke happiness and sadness, and uses black-box exploration to examine what a GAN learns from visual emotion data.Source: §2 · p. 5
  • GAN interpretability research includes examining whether models learn composition or memorize pixel patterns; one cited study found learned compositional aspects and domain-specific features in some neuron units.Source: §2 · p. 5
  • Black-box exploration uses the training dataset and model outputs to examine behavior and gain insight into model interpretations.Source: §2 · p. 5

3. Experiments

The experiments train DCGANs on hashtag-collected Instagram images and a sadness dataset to test how well these data and models support visual affect generation. The generated images reflect dataset content and show that dark or bright color cues alone do not ensure emotion is conveyed.

3.1. Dataset creation

  • 30,000 unique #sad and 30,000 unique #happy Instagram images were collected without human validation, alongside a 2,635-image You et al. sadness dataset.Images were deduplicated but otherwise unprocessed; all models used the same DCGAN parameters and 100,000 training epochs.Source: §3.1 · p. 5, §3 · p. 5

3.2. Forging sadness

  • Both sadness models generated images reminiscent of abstract paintings, while their largest color clusters were dark grey (26.84%) and charcoal (30.63%), respectively.The Instagram-trained model produced more text-like images, reflecting differences in the training datasets; captions often described blurry images or collages rather than sadness.Source: §3.2 · pp. 6–7
  • The Instagram sadness set contained predominantly text images and varied subjects, including images using “sad” to mean inadequate or unfashionable; single-word social-media searches also missed many experiences of sadness.Neither hashtag-based dataset contained images such as funerals, wars, disasters, or mutilated bodies; the authors note that MTurk validation cannot restore omitted aspects of sadness.Source: §3.2 · p. 7

3.3. Forging Happiness

  • The happiness model produced brighter, varied images and some captions described happy occasions, yet the authors question whether these images visualize or trigger happiness.They argue that basic color associations are insufficient and call for larger datasets and research into trained GAN structure and compositional GANs to direct image subject matter.Source: §3.3 · p. 8

Conclusion

The experiment concludes that current methods for building emotion datasets do not capture the full range of emotional experience. It proposes association-based datasets and deeper study of GAN structure to support novel affective compositions.

Conclusion

  • Single-word emotion queries can yield images related to an emotion but miss the full range of happiness or sadness.The paper proposes exploring networks of associations that describe emotional experience, considering more online sources, and involving artists in dataset creation.Source: Conclusion · p. 8
  • The paper proposes emotional association networks as a way to represent how each emotion is experienced in visual datasets.The proposed approach would move beyond images collected solely through single-word emotion queries.Source: Conclusion · p. 8
  • Artists’ contributions to creating visual emotion datasets would be valuable.The paper also proposes using a greater variety of online sources rather than social media alone.Source: Conclusion · p. 8
  • For visual affect generation, the paper recommends better understanding trained and compositional GAN structures.The authors propose that interpretation methods could reveal learned features and inform GANs that combine elements into novel compositions directed at affect through image subject matter.Source: Conclusion · p. 8

Related papers

Loading 2610.32163v1…