PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

Zixuan Xing, Lancheng Gao, Shengyan Li, Huiyu Duan, Zhaolin Cai, Guangtao Zhai, Xiongkuo Min

arXiv:2610.18455v1cs.CV

TL;DR

Emotion in AI-generated images may differ from the intended affect, and MLLM interpretation has not been established as consistent with human perception. EmoGenie benchmarks expressed and evoked emotion, and EmoProbe combines emotion question answering with continuous score prediction. The paper reports state-of-the-art performance for EmoProbe across coarse-grained perception and fine-grained affective assessment.

  • Problem

    Affective benchmarks predominantly use natural images, while generated-image studies have largely focused on expressed emotion in portraits, leaving a unified benchmark for expressed and evoked emotion in AI-generated images unavailable.

    Source: §1 · pp. 1–2

  • Method

    EmoGenie pairs question-answer data and continuous human ratings for AI-generated images, while EmoProbe combines emotion question answering with response-aligned regression heads.

    Source: §1 · p. 2, §3.1 · p. 4, §3.4 · p. 5

  • Results

    EmoProbe achieves state-of-the-art performance on coarse-grained emotion perception and fine-grained affective assessment of AI-generated images.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments reveal substantial variation in current T2I models’ emotion-generation capabilities across affective perspectives and emotion categories.

    Source: §5.2 · p. 8, §6 · p. 9

Abstract

from arXiv · show

Rapid advances in artificial intelligence-generated content (AIGC) have significantly promoted using text-to-image (T2I) models to generate images with specific affective intentions. However, the emotional effects conveyed by generated images frequently deviate from target expectations, and it remains unclear whether multimodal large language models (MLLMs) can reliably interpret the emotion of AI-generated visual content. Existing affective benchmarks are predominantly built on natural images, while affective studies of generated images have largely focused on emotions expressed in portraits, lacking a unified benchmark for studying the emotions expressed and evoked by generated images. We therefore introduce EmoGenie, a benchmark of emotion for AI-generated images that evaluates capabilities in T2I generation and MLLM understanding across expressed and evoked emotions. It comprises 6,400 images generated by eight T2I models using 800 emotion-targeted prompts, annotated with question-answer (QA) pairs, termed EmoGenie-QA, and continuous human ratings, termed EmoGenie-Assess. EmoGenie-QA contains 90,000 closed-set questions in single-image and image-pair settings, as well as 1,057 open-ended questions for emotion description. EmoGenie-Assess benchmarks continuous affective and aesthetic prediction using 13,175 mean opinion scores (MOSs) aggregated from human ratings in terms of valence, arousal, dominance, and aesthetic quality. We further develop EmoProbe, which combines emotion question answering with response-aligned, dimension-specific regression heads for continuous score prediction. Experimental results show that EmoProbe achieves superior results on emotion-related QA tasks, while its affective and aesthetic predictions correlate better with human ratings, demonstrating its state-of-the-art capabilities of unified coarse-to-fine affective understanding for AI-generated images.

1 INTRODUCTION

EmoGenie addresses the lack of a unified benchmark for expressed and evoked emotions in AI-generated images, while EmoProbe combines emotion question answering with continuous assessment. The reported results show state-of-the-art EmoProbe performance and variation across T2I models, emotion categories, and evaluation tasks.

1 INTRODUCTION

  • Existing affective benchmarks predominantly use natural images, while generated-image studies have focused mainly on expressed emotion in synthetic portraits.The paper identifies a remaining need to assess both perspectives in AI-generated images and MLLM interpretation against human perception.Source: §1 · pp. 1–2
  • EmoGenie is a large-scale benchmark for emotion analysis in AI-generated images, comprising 6,400 images and over 91K question-answer pairs, covering expressed and evoked perspectives for emotion perception, description, and assessment.Its construction includes 90K closed-set and 1,057 open-ended question-answer pairs, plus 13,175 affective and aesthetic scores.Source: §1 · p. 2, Figure 1 · p. 3
  • Benchmark results reveal marked variation across affective perspectives, emotion categories, and evaluation tasks.The benchmark evaluates representative T2I models and MLLMs in a unified framework.Source: §1 · p. 2
  • EmoProbe achieves state-of-the-art performance on coarse-grained emotion perception and fine-grained affective assessment of AI-generated images.It integrates emotion question answering with response-aligned regression in a two-stage training procedure.Source: §1 · p. 2

2 RELATED WORK

Prior visual-emotion datasets and MLLM benchmarks cover diverse annotations and tasks, but their coverage of AI-generated images remains incomplete. Existing work on generated images often focuses on controllability, portraits, or one affective perspective.

2.1 VISUAL EMOTION DATASETS

  • Visual-emotion datasets and benchmarks cover discrete emotions, affective distributions, and tasks such as emotion ranking and dimensional assessment.Some recent benchmarks distinguish expressed from evoked emotions, but these paradigms remain largely absent for AI-generated images.Source: §2.1 · p. 3

2.2 BENCHMARKS FOR AIGIS

  • AIGI emotion-generation studies include category-conditioned generation, continuous valence–arousal control, affective editing, and synthetic-portrait evaluation.Joint evaluation of expressed and evoked emotion in AI-generated images remains underexplored.Source: §2.2 · p. 3

2.3 EMOTION-ORIENTED MLLMS

  • Emotion-oriented MLLMs support tasks including reasoning, explanation, comparison, and continuous assessment, but most are developed or evaluated on natural images or one affective perspective.Their alignment with human judgments of expressed and evoked emotion in AI-generated images remains unclear.Source: §2.3 · p. 3

3 EMOGENIE BENCHMARK CONSTRUCTION

EmoGenie pairs two emotion perspectives with human annotations to form perception, description, and continuous-assessment benchmarks. Its QA and assessment subsets derive from annotations of images generated from separate expressed- and evoked-emotion prompt pools.

3 EMOGENIE BENCHMARK CONSTRUCTION

  • EmoGenie comprises EmoGenie-QA for emotion perception and evidence-grounded description, and EmoGenie-Assess for continuous affective and aesthetic scores.Together, the subsets support a unified benchmark construction for emotion analysis.Source: §3 · p. 4

3.1 DATA COLLECTION

  • The benchmark uses separate prompt pools for expressed and evoked emotion, generating 6,400 images from 800 prompts across eight T2I models.The 400 prompts per perspective use distinct emotion taxonomies and prompt designs.Source: §3.1 · p. 4

3.2 SUBJECTIVE ANNOTATION

  • Human annotation covers emotion category, graded affective attributes, and aesthetic quality, with evidence-grounded descriptions elicited in a second round.The first round collects 15 annotations per image; the second round asks annotators to justify judgments with observable visual cues and compare salient differences for image pairs.Source: §3.2 · pp. 4–5

3.3 EMOGENIE-QA

  • EmoGenie-QA contains 90K closed-set perception questions and 1,057 open-ended description questions for single images and image pairs.Description questions require emotion judgments grounded in visual cues or salient image differences; their reference answers are reserved for evaluation.Source: §3.3 · p. 5

3.4 EMOGENIE-ASSESS

  • EmoGenie-Assess provides continuous ratings for valence, arousal, dominance, and aesthetic quality, with each score averaging 15 independent ratings.VAD scores retain a 1–9 scale, while aesthetic scores retain a 1–5 scale.Source: §3.4 · p. 5

4 THE EMOPROBE APPROACH

EmoProbe combines emotion perception, evidence-grounded description, and continuous assessment in one model. Its two-stage training moves from categorical QA to regression against human ratings.

4.1 MODEL ARCHITECTURE

  • EmoProbe uses a shared multimodal backbone for emotion perception, evidence-grounded description, and continuous affective and aesthetic assessment.For image pairs, both images are jointly encoded in a shared context for direct comparison.Source: §4.1 · p. 6
  • Four dimension-specific regression heads map the hidden state aligned with the final numerical response token to valence, arousal, dominance, and aesthetic scores.The model generates a structured numerical judgment, and the representation aligned with its final numerical token is passed to the corresponding head.Source: §4.1 · p. 6

4.2 TWO-STAGE TRAINING

  • Stage 1 instruction-tunes on EmoGenie-QA for categorical single-image and image-pair judgments; Stage 2 trains continuous score regression on EmoGenie-Assess.Stage 2 freezes the multimodal projector while jointly optimizing vision and language LoRA parameters and the four regression heads; VAD supervision is limited to evoked-emotion images, while aesthetic supervision covers both perspectives.Source: §4.2 · p. 6

5 EXPERIMENTS

The experiments compare eight T2I models and EmoProbe with 14 MLLMs across generation, perception, description, and continuous assessment. EmoProbe leads the reported QA and human-rating comparisons, while generation patterns vary across emotion perspectives.

5.1 EXPERIMENTAL SETUP

  • The evaluation compares eight T2I models and 14 MLLMs, using image-disjoint 4:1 train/test splits for perception and assessment and reserving description for evaluation.Perception uses exact-match accuracy, assessment uses SRCC and PLCC, and description combines correctness, completeness, relevance, and conciseness.Source: §5.1 · p. 7

5.2 EVALUATION OF T2I EMOTION GENERATION

  • GPT-Image-2 leads expressed-emotion alignment, while Qwen-Image leads evoked-emotion alignment; seven of eight T2I models score better for expressed emotion.GPT-Image-2 also receives the highest aesthetic ratings from both perspectives; seven models receive higher aesthetic ratings for evoked images, with GPT-Image-2 the exception.Source: §5.2 · pp. 7–8

5.3 EVALUATION ON MLLM EMOTION UNDERSTANDING

  • 75.54% overall, EmoProbe ranks first on QA perception and description, and its mean SRCC/PLCC correlations with human ratings are 0.645/0.681.It scores 82.87% and 77.62% on expressed- and evoked-emotion perception, respectively, and ranks first in description for both perspectives.Source: §5.3 · pp. 8–9, Table 4 · p. 8, Table 5 · p. 9

5.4 ABLATION STUDY

  • Ablations associate multimodal adaptation with better perception and description, and Stage 2 with gains on all four assessment dimensions while maintaining perception and further improving description.Stage 2 also improves results with a Qwen3.5-VL backbone; InternVL3.5-8B achieves stronger overall results in the backbone comparison.Source: §5.4 · p. 9, Table 6 · p. 9

6 CONCLUSION

EmoGenie evaluates emotion generation and understanding across expressed and evoked perspectives, while EmoProbe combines categorical perception with continuous assessment. Its prompt design and matched generation setup support comparison across varied T2I models.

6 CONCLUSION

  • EmoGenie benchmarks expressed- and evoked-emotion generation and understanding, while EmoProbe integrates coarse-grained perception with fine-grained continuous assessment through two-stage training.Experiments reveal substantial variation in T2I model emotion-generation capabilities across affective perspectives and emotion categories.Source: §6 · p. 9
  • The authors report informed consent, compensation, and privacy-conscious annotation, document the benchmark-to-evaluation pipeline, and state that AI assistance was reviewed by authors.They plan to release the benchmark, code, prompts, annotations, and evaluation resources upon publication.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10, AI USE STATEMENT · p. 10

A.1 EMOTION TAXONOMIES AND PROMPT DESIGN

  • Prompts state an affective target and concrete visual content while leaving composition and rendering flexible, and identical semantic prompts go to every T2I model.Expressed- and evoked-emotion prompts use separate pools because they rely on different visual evidence.Source: §A.1 · p. 13
  • The expressed-emotion pool covers seven categories and three intensity levels, with coordinated facial and bodily cues varied across prompts.Its 400 prompts include 156 mild, 110 moderate, and 134 intense expressions.Source: §A.1 · p. 13
  • The evoked-emotion pool uses eight categories and scene-centred prompts that elicit affect through events, objects, environments, and visual organization.It contains 50 prompts per category, 400 in total.Source: §A.1 · pp. 13–14

A.2 T2I MODELS AND IMAGE GENERATION

  • Eight diverse T2I models each generated one image for every prompt, yielding 6,400 images without emotion- or aesthetic-based post-generation filtering.The selection spans open-source checkpoints and proprietary APIs, as well as diffusion and rectified-flow systems.Source: §A.2 · p. 14

A.3 VISUAL-QUALITY VARIATION

Human aesthetic ratings vary across T2I models and affective perspectives: expressed-emotion images show a wider spread, while evoked-emotion ratings cluster more tightly.

A.3 VISUAL-QUALITY VARIATION

  • The benchmark retains one output per prompt–model combination without aesthetic filtering, preserving the quality variation produced under its generation settings.Figure 5 depicts the structured annotation fields for expressed- and evoked-emotion assessments.Source: §A.3 · p. 15, Figure 5 · p. 16
  • 3.88 was GPT-Image-2’s mean aesthetic rating for expressed-emotion images, versus 3.49 for Qwen-Image, 3.46 for Seedream 5.0, and 2.02 for PixArt-Σ.SD3 and SD3.5 also had broad distributions with pronounced lower-quality tails.Source: §A.3 · p. 15
  • Evoked-emotion mean ratings ranged from 3.42 to 3.71, with extensively overlapping interquartile ranges and smaller separation between GPT-Image-2 and other models.Seven models had higher mean ratings for evoked-emotion images; GPT-Image-2 alone had a higher mean for expressed emotion.Source: §A.3 · p. 15

B DETAILS OF HUMAN ANNOTATION AND BENCHMARK ASSEMBLY

Human annotation distinguishes emotion expressed by a depicted person from emotion evoked in a viewer, then collects structured judgments and evidence-grounded descriptions for both.

B.1 ANNOTATION DIMENSIONS AND INTERFACE

  • Annotators received definitions, scale anchors, and examples distinguishing expressed from evoked emotion, then completed applicable fields for randomized images.The first round collected 15 independent annotations for each of 6,400 images, yielding 96,000 image-level records.Source: §B.1 · p. 16
  • For expressed emotion, annotators selected a depicted person’s emotion, rated expression intensity, and judged aesthetics on a 1–5 scale.Intensity reflected facial and bodily cues; aesthetic judgments considered composition, colour, illumination, clarity, realism, and overall visual impact.Source: §B.1 · p. 16
  • For evoked emotion, annotators rated category, valence, arousal, dominance, and aesthetics, while a second round gathered evidence-based explanations for selected images and pairs.The second round included 300 single-image cases and 100 image pairs, with four annotators completing it.Source: §B.1 · p. 16

B.2 AGGREGATION AND AGREEMENT ANALYSIS

The benchmark aggregates annotations and reports agreement for categorical and ordered judgments, then organizes closed-set questions, open-ended descriptions, continuous scores, and staged EmoProbe training objectives.

B.2 AGGREGATION AND AGREEMENT ANALYSIS

  • The pipeline checks records before aggregation, uses majority vote for emotion categories, retains tied leaders, and averages ordered ratings on their original scales.It uses AC1 for nominal categories and quadratic-weighted AC2 for ordered ratings.Source: §B.2 · pp. 16–17
  • Expressed-emotion categories showed higher agreement than evoked-emotion categories, while ordered dimensions showed consistently strong weighted agreement.Evoked-image aesthetic quality had the highest AC2; Table 7 reports agreement for the principal dimensions.Source: §B.2 · p. 17, Table 7 · p. 18

B.3 DETAILED CONSTRUCTION OF EMOGENIE-QA

  • EmoGenie-QA contains 90,000 closed-set questions and 1,057 evaluation-only description questions, with image-disjoint splits for the closed-set task.The closed-set questions include single-image and image-pair formats; description questions require visible evidence for emotion judgments.Source: §B.3 · p. 18

B.4 DETAILED CONSTRUCTION OF EMOGENIE-ASSESS

  • EmoGenie-Assess contains 13,175 mean human ratings for VAD on evoked images and aesthetic quality across both perspectives.Scores retain their original 1–9 VAD and 1–5 aesthetic scales and use image-disjoint training, validation, and test partitions.Source: §B.4 · p. 18

C EMOPROBE LOSS FUNCTIONS

  • EmoProbe trains first on categorical QA responses, then retains language supervision while adding dimension-specific regression for available continuous targets.VAD supervision applies only to evoked-emotion images, while aesthetic supervision covers both perspectives.Source: C EMOPROBE LOSS FUNCTIONS · pp. 18–19

D DETAILED EVALUATION SETTINGS AND EXTENDED ANALYSIS

The evaluated MLLMs span general-purpose multimodal systems and models specialized for emotion understanding. Their descriptions highlight different visual architectures, training strategies, and affective capabilities.

D.1 DETAILED INFORMATION OF COMPARED MLLMS

  • General-purpose systems include open and proprietary models with varied designs for visual encoding, multi-image input, grounding, and multimodal reasoning.The descriptions cover Janus-Pro, LLaVA-OneVision, Qwen2.5-VL, Qwen3-VL, Qwen3.5-VL, InternVL3, InternVL3.5, GPT-5.5, and Claude-Sonnet-4.6.Source: §D.1 · pp. 19–20
  • Emotion-Qwen, EmoCaliber, and EEmo-Logic are emotion-oriented MLLMs, respectively emphasizing emotion-specific features, confidence calibration, and evoked-emotion tasks.Emotion-Qwen combines general and affective experts; EmoCaliber adds confidence estimation; EEmo-Logic trains for ranking, VAD prediction, and dominant-emotion classification.Source: §D.1 · p. 20
  • EmoVIT adapts InstructBLIP for visual emotion understanding by training its instruction-aware Q-Former while freezing the visual encoder and language model.Its training corpus includes categorical, conversational, and reasoning instructions for affective image analysis.Source: §D.1 · p. 20
  • AffectGPT targets affective understanding across audio, video, and text, using pre-fusion modules to integrate modality-specific evidence before language-model processing.Training on MER-Caption supports natural-language emotion descriptions grounded in multimodal evidence.Source: §D.1 · p. 20

D.2 EXTENDED EVALUATION OF T2I EMOTION GENERATION

The extended generation analysis finds perspective-specific emotion substitutions across T2I models, while the open-ended description evaluation scores responses on four defined dimensions. The matrices expose category-level patterns beyond aggregate target-emotion alignment.

D.2 EXTENDED EVALUATION OF T2I EMOTION GENERATION

  • Anger–disgust and fear–surprise substitutions characterize expressed-emotion errors, while evoked-emotion confusions include amusement–contentment and unstable anger.Happiness and surprise are consistent expressed-emotion targets; contentment, awe, and excitement are comparatively stable evoked targets, while some generators shift fear or sadness toward awe or contentment.Source: §D.2 · pp. 20–21, Figure 7 · p. 21
  • The normalized confusion matrices place prompted emotions in rows and annotator-assigned emotions in columns, with diagonal mass indicating target alignment.Off-diagonal mass shows emotion substitutions; expressed categories are A–G and evoked categories A–H.Source: Figure 7 · p. 21
  • Expressed-emotion errors track similar facial or bodily cues, whereas evoked-emotion errors more often link related valence or scene semantics.These category-level substitutions reveal weaknesses that aggregate alignment scores can obscure.Source: §D.2 · p. 21

D.3 EVALUATION CRITERIA FOR OPEN-ENDED DESCRIPTION

  • DeepSeek-Flash evaluates correctness, completeness, and relevance using the question, candidate response, reference answer, and rubric, without image or model-identity access.Conciseness is computed deterministically from the candidate-to-reference word-count ratio.Source: §D.3 · pp. 21–22
  • Open-ended responses are scored for correctness, completeness, relevance, and conciseness, with each dimension contributing equally to the final score.The first three dimensions use 0–2 ratings and are averaged across five evaluations; conciseness is assigned by candidate-to-reference word-count ratio.Source: §D.3 · p. 22

D.4 EVALUATION CRITERIA FOR CONTINUOUS ASSESSMENT

EmoProbe leads most dimension-by-format closed-set QA comparisons, with exceptions in single-image emotion recognition. The section also defines continuous-assessment correlations and illustrates predictions and perception cases against human judgments and comparison models.

D.4 EVALUATION CRITERIA FOR CONTINUOUS ASSESSMENT

  • Continuous predictions are evaluated with SRCC for rank agreement and PLCC for linear agreement against human-aggregated scores.Scores are computed separately for valence, arousal, dominance, and aesthetic quality, then averaged unweighted across the four dimensions.Source: §D.4 · p. 22
  • The probability assigned to a dimension’s high-end descriptor is mapped to its original interval [ad, bd], yielding a continuous score without training an additional regressor.These intervals are [1, 9] for VAD and [1, 5] for aesthetic quality.Source: §D.4 · p. 22

D.5 DIMENSION-WISE CLOSED-SET QA

  • 90.42% single-image dominance accuracy for evoked emotion and 87.36% pairwise target-alignment accuracy are EmoProbe’s strongest reported wins, while it trails on single-image emotion recognition.EmoProbe leads 9 of 10 evoked and 5 of 6 expressed comparisons; InternVL3-8B-Instruct and EmoCaliber lead the stated single-image recognition exceptions.Source: Table 8 · p. 23, Table 9 · p. 23, §D.5 · p. 23

E QUALITATIVE CASE STUDIES

  • In the illustrated continuous-assessment cases, EmoProbe predictions are close to human-aggregated scores across valence, arousal, dominance, and aesthetic quality.Figure 8 compares those scores with EmoProbe and evaluated baselines.Source: Figure 8 · p. 24, E QUALITATIVE CASE STUDIES · p. 24
  • EmoProbe answers the illustrated challenging closed-set questions correctly, including single-image judgments and image-pair comparisons.Figures 9 and 10 show evoked- and expressed-emotion cases, respectively, with green and red markers denoting correct and incorrect responses.Source: E QUALITATIVE CASE STUDIES · p. 24, Figure 9 · p. 25, Figure 10 · p. 26
Loading 2610.18455v1…