PaperlayerView exact revision on arXiv

Source-linked AI summary

AffectGAN: Affect-Based Generative Art Driven by Semantics

Theodoros Galanos, Antonios Liapis, Georgios N. Yannakakis

arXiv:2109.14845v1cs.CVcs.LG

TL;DR

AffectGAN addresses how generative systems can create artistic images intended to express affective states from semantic prompts. It combines CLIP with a WikiArt-trained VQGAN and evaluates 32 generated images with 50 participants. Most images matched the intended affect for at least half of participants, although happiness was difficult to identify and generated images exhibited GAN-related glitches.

  • Problem

    Affective computing commonly expresses affect through human-centric signals, leaving a need for artistic image generation that embeds affective information.

    Source: §I · p. 1

  • Method

    AffectGAN combines CLIP for semantic guidance with a VQGAN trained on WikiArt to iteratively synthesize images from affective and semantic prompts.

    Source: §I · p. 1, §III · p. 2

  • Results

    65% of generated images were matched to the affect prompt by at least half of participants, with depression achieving 68% accuracy and happiness 36%.

    Source: §I · p. 2, 1) Classification of elicited emotion · pp. 3–4, §V · p. 5

  • Takeaways & Limitations

    The proof-of-concept supports using semantic affective image generation for emotion elicitation, affective-database augmentation, and affective interaction applications.

    Source: §VI · p. 6

    The study was small-scale, and generated images commonly contained facial-expression, emoji-like, and other GAN-related glitches.

    Source: §V · p. 5

Abstract

from arXiv · show

This paper introduces a novel method for generating artistic images that express particular affective states. Leveraging state-of-the-art deep learning methods for visual generation (through generative adversarial networks), semantic models from OpenAI, and the annotated dataset of the visual art encyclopedia WikiArt, our AffectGAN model is able to generate images based on specific or broad semantic prompts and intended affective outcomes. A small dataset of 32 images generated by AffectGAN is annotated by 50 participants in terms of the particular emotion they elicit, as well as their quality and novelty. Results show that for most instances the intended emotion used as a prompt for image generation matches the participants' responses. This small-scale study brings forth a new vision towards blending affective computing with computational creativity, enabling generative systems with intentionality in terms of the emotions they wish their output to elicit.

I. INTRODUCTION

AffectGAN generates artistic images intended to express affect through semantic prompts, linking affective computing with computational creativity. In a 32-image study evaluated by 50 participants, intended emotions generally matched responses, though happiness was difficult to identify.

  • AffectGAN generates semantic images that embed affective information and express it through visual art, bridging affective computing and computational creativity.Source: §I · p. 1
  • The system combines OpenAI’s CLIP with VQGAN trained on WikiArt, using semantics to guide image synthesis.Source: §I · p. 1
  • 50 participants evaluated 32 images generated from four emotions across eight painting types for elicited emotion, quality, and novelty.Source: §I · pp. 1–2
  • Emotion-matching accuracy was 68% for depression, 65% for anger, 62% for calmness, and 36% for happiness.Source: §I · p. 2

II. BACKGROUND

Prior computational-creativity systems generated images representing concepts, while affective computing commonly modeled human affect through facial expressions or body stance. AffectGAN instead generates affective art from semantic prompts without requiring user images or a human-provided seed image.

  • AffectGAN emphasizes affect-infused image generation from arbitrary semantic prompts, unlike DARCI’s filter-programming approach from an original seed image,.Source: §II · p. 2
  • The system follows a user-agnostic affective-computing direction by using affect labels on WikiArt images rather than images of users.Source: §II · p. 2
  • Earlier text-to-image approaches often produced unsatisfactory results for complex domains, whereas newer multimodal models made high-quality language-based image generation feasible.Source: §II · p. 2

III. AFFECTGAN

AffectGAN combines CLIP and a WikiArt-trained VQGAN in an iterative codebook-sampling process. CLIP scores semantic alignment while VQGAN synthesizes images from quantized visual codes.

  • A custom VQGAN trained on 81,444 WikiArt images learns quantized image-patch embeddings and is fine-tuned at higher resolution to improve reconstructions.Training used 7 million steps followed by 1 million fine-tuning steps.Source: §III · p. 2
  • Codebook sampling initializes logits over 16x16-pixel cells, samples VQGAN codes, and minimizes CLIP cosine loss against the prompt embedding.The WikiArt VQGAN codebook contains 1024 possible codes.Source: §III · p. 2
  • The final image is produced by selecting the argmax visual codes rather than sampling them.Source: §III · p. 3
  • CLIP and VQGAN jointly enable semantically guided artistic generation by combining language-image alignment with conditional visual synthesis.Source: §III · p. 3

IV. RESULTS

The evaluation tested whether affective prompts were recognizable and whether emotion and painting type influenced judgments of generated images. A 32-image dataset spanning four emotions and eight painting types was assessed through a structured questionnaire.

IV. RESULTS

  • The study assessed the efficacy of semantic image generation from affective descriptions by generating images with different genres and emotional contexts and surveying their qualities.Source: A. Experimental Protocol · p. 3

A. Experimental Protocol

  • The dataset contained 32 images formed by crossing four circumplex emotions with eight WikiArt painting types.The emotions were depression, calmness, anger, and happiness.Source: A. Experimental Protocol · p. 3
  • Each participant viewed all images and answered an emotion question plus 5-item Likert-scale quality and novelty questions.Source: A. Experimental Protocol · p. 3
  • Participants were recruited through the authors’ contacts using combined snowball and convenience sampling.Source: A. Experimental Protocol · p. 3

B. Participants

The study surveyed 50 participants, most of whom were 25–35 years old and identified as male, with limited reported generative-art experience overall.

  • 50 participants completed the survey, with 60% aged 25–35 and 70% identifying as male.Participants averaged 2.96 out of 5 for generative-art experience, and 32% rated themselves 4 or 5.Source: B. Participants · p. 3

C. Findings

The findings examine whether viewers could identify AffectGAN’s intended affective prompts and how they responded to the generated images. Participants often matched the intended emotion, although classification varied substantially by affective state.

1) Classification of elicited emotion:

C. Findings

  • The study also measured perceived image quality and novelty alongside emotion classification, with Table II reporting averages over eight images per affective prompt.Quality and novelty responses used rating questions spanning very poor to very good and not novel at all to very novel.Source: 1) Classification of elicited emotion · p. 3, Table II · p. 4

1) Classification of elicited emotion:

  • Depression prompts achieved the highest emotion-matching accuracy at 68%, followed by anger at 65%, calmness at 62%, and happiness at 36%.Happiness was often labelled as “other” or confused with calmness.Source: 1) Classification of elicited emotion · pp. 3–4
  • Low-valence prompts were classified more accurately at 72% than high-valence prompts at 65%, while arousal-level distinctions produced greater confusion.Viewers generally distinguished positive from negative affect more easily than high from low arousal.Source: 1) Classification of elicited emotion · p. 4

2) Impact of affective prompts on user responses:

Affective prompts influenced both the emotions viewers reported and their evaluations of the generated images. Calmness images received higher quality but lower novelty ratings, while happiness and anger sets drew more varied and novel responses.

2) Impact of affective prompts on user responses:

  • Calmness prompts received higher quality but lower novelty ratings than the other affective prompt sets, although differences were not significant at 95% confidence.Source: 2) Impact of affective prompts on user… · p. 4
  • Table III reports quality, novelty, and emotion responses by painting type using averages from four generated images per type.Source: Table III · p. 4
  • Happiness and anger prompts produced more unique responses and higher overall novelty ratings, plausibly alongside more confusing elicited emotions.The analysis links the larger number of unique responses to higher “other” response rates.Source: 2) Impact of affective prompts on user… · p. 4

3) Impact of painting types on user responses:

Painting type substantially shaped emotion classification, perceived quality, and novelty. Abstract and sketch-study prompts yielded the most accurate affect predictions, whereas landscapes and portraits scored differently on quality and novelty.

3) Impact of painting types on user responses:

4) Qualitative analysis of generated images:

AffectGAN images varied in color composition, realism, affective interpretation, quality, and novelty across emotional and painting-type prompts.

V. DISCUSSION AND NEXT STEPS

The user study demonstrates both AffectGAN’s ability to elicit intended affect and its limitations, especially for happiness recognition and image artifacts.

  • 65% of generated images were matched to the affect prompt by at least half of participants, but happiness was frequently mislabeled as calm.Source: §V · p. 5
  • Still Life prompts had the lowest accuracy at 42%, while Abstract Paintings reached 66% accuracy but received the lowest quality and novelty ratings.Source: §V · p. 5
  • More interesting images were often harder to label correctly when they had high quality, novelty, or elicited participants’ own affective labels.Source: §V · p. 5
  • Generated images commonly contained facial or emoji-like glitches and out-of-place elements, motivating custom discrimination or post-generation filtering.Source: §V · p. 5, Figure 4 · p. 5
  • The initial study reused existing architectures and pretrained models, leaving tailored emotion inputs, image-conditioned generation, latent-vector control, and interactive applications as future directions.Source: §V · pp. 5–6

VI. CONCLUSION

AffectGAN combines CLIP and VQGAN to generate artistic images from semantic prompts that express affective states. The proof-of-concept supports applications as emotion elicitors, affective training data, and interactive creative systems.

  • AffectGAN combines CLIP and VQGAN to generate high-resolution artistic images driven by semantic prompts and intended emotional states.Source: §VI · p. 6
  • The study demonstrates potential for affective computing and computational creativity by enabling systems to generate and assess emotions intended for audiences.Source: §VI · p. 6
Loading 2109.14845v1…