PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoAgent: A Multi-Agent Framework for Diverse Affective Image Manipulation

Qi Mao, Haobo Hu, Yujie He, Difei Gao, Haokun Chen, Libiao Jin

arXiv:2503.11290v3cs.CVeess.IV

TL;DR

Existing AIM methods often tie each emotion to a fixed visual expression, despite the variety of cues through which emotions may be perceived. EmoAgent defines D-AIM and uses collaborative planning, editing, and critique to produce diverse edits for one image and target emotion. The paper reports that it outperforms existing AIM methods in emotional alignment and visual diversity.

  • Problem

    Existing AIM methods rely on fixed emotion-to-visual mappings, despite the subjective and diverse ways people perceive and express emotion.

    Source: §I · p. 1, B. Affective Image Manipulation · p. 3

  • Method

    EmoAgent defines D-AIM and uses Planning, Editing, and Critic agents to generate, execute, and refine diverse editing strategies for one source image and target emotion.

    Source: §I · p. 1

  • Results

    EmoAgent significantly outperforms existing AIM methods in emotional alignment and visual diversity.

    Source: §I · p. 2, §VI · p. 12

  • Takeaways & Limitations

    The framework supports multiple semantically grounded and emotionally faithful transformations for the same affective goal.

    Source: §VI · p. 12

Abstract

from arXiv · show

Affective Image Manipulation (AIM) aims to alter visual elements within an image to evoke specific emotional responses from viewers. However, existing AIM approaches rely on rigid \emph{one-to-one} mappings between emotions and visual cues, making them ill-suited for the inherently subjective and diverse ways in which humans perceive and express emotion.To address this, we introduce a novel task setting termed \emph{Diverse AIM (D-AIM)}, aiming to generate multiple visually distinct yet emotionally consistent image edits from a single source image and target emotion. We propose \emph{EmoAgent}, the first multi-agent framework tailored specifically for D-AIM. EmoAgent explicitly decomposes the manipulation process into three specialized phases executed by collaborative agents: a Planning Agent that generates diverse emotional editing strategies, an Editing Agent that precisely executes these strategies, and a Critic Agent that iteratively refines the results to ensure emotional accuracy. This collaborative design empowers EmoAgent to model \emph{one-to-many} emotion-to-visual mappings, enabling semantically diverse and emotionally faithful edits.Extensive quantitative and qualitative evaluations demonstrate that EmoAgent substantially outperforms state-of-the-art approaches in both emotional fidelity and semantic diversity, effectively generating multiple distinct visual edits that convey the same target emotion.

I. INTRODUCTION

AIM seeks to alter images to evoke emotional responses, but prior methods often map each emotion to a fixed visual expression, despite the diversity of affective perception. EmoAgent introduces D-AIM and a collaborative three-agent framework for generating varied, emotionally consistent edits; the paper also situates its approach within affective computing’s use of multimodal models for emotion analysis.

I. INTRODUCTION

  • Prior AIM methods use fixed emotion-to-visual mappings, while people may interpret the same emotion through varied visual cues.The paper frames generating emotionally consistent yet visually diverse edits as a central AIM challenge.Source: §I · p. 1
  • D-AIM targets multiple visually distinct yet emotionally consistent edits from one source image and target emotion.EmoAgent treats this as exploring multiple semantic editing paths.Source: §I · pp. 1–2
  • EmoAgent assigns planning, editing, and iterative emotional critique to collaborative agents.The Planning Agent uses a vision-language model and emotion-factor knowledge retriever to create varied plans; the Critic Agent helps refine outputs that do not convey the target emotion accurately and coherently.Source: §I · pp. 1–2

A. Affective Computing

  • Affective computing studies machine perception and interpretation of emotion across modalities, including images and text.Emotion representations include categorical states and dimensional spaces; recent work also includes multimodal models such as GPT-4V for visual and textual contexts.Source: A. Affective Computing · p. 2
  • The paper incorporates multimodal large language models to analyze and evaluate emotional consistency during editing.It reports quantitative and qualitative evaluations showing improved emotional accuracy and visual diversity within the same emotion.Source: A. Affective Computing · p. 2, §I · p. 2

B. Affective Image Manipulation

Semantic image manipulation methods edit according to text prompts, but are not generally designed for emotional reasoning. Prior affective image manipulation methods progressed from low-level adjustments toward emotion-aware editing, yet the paper identifies fixed emotion-to-visual mappings as a remaining limitation; it presents EmoAgent’s collaborative agents as an approach to diverse, emotionally grounded editing.

B. Affective Image Manipulation

  • EmoAgent’s workflow generates diverse plans, executes them, and sends emotionally unsatisfactory outputs to iterative refinement.Its agents use emotion-grounded plan generation, hierarchical action execution, and chain-of-thought-based emotional critique.Source: Figure 2 · p. 3
  • Semantic image manipulation methods are generally not designed to interpret or express emotional semantics in prompts.AIM work moved from low-level lighting or style adjustments to emotion cues and EmoEdit’s Emotion Adapter in diffusion-based editing.Source: B. Affective Image Manipulation · p. 3
  • Existing AIM methods commonly map each emotion to fixed visual features, although different edits can evoke the same emotion depending on scene semantics, composition, and narrative intent.The paper introduces D-AIM to target multiple semantically distinct yet emotionally consistent edits for one input and target emotion.Source: B. Affective Image Manipulation · p. 3

C. LLM-As-Agent

  • GenArtist uses a single agent for sequential text-prompt generation and editing, whereas AIM calls for emotional reasoning, grounded edits, and multi-step refinement.The paper proposes Planning, Editing, and Critic agents for collaborative emotion-aware editing.Source: C. LLM-As-Agent · pp. 3–4

III. MULTI-AGENT FRAMEWORK FOR DIVERSE-AIM

The framework defines D-AIM as generating multiple distinct edits of one image that share a target emotion, while preserving the source’s high-level semantic structure and layout. EmoAgent first creates and evaluates candidate edits, then iteratively refines those that fail its emotional criteria.

III. MULTI-AGENT FRAMEWORK FOR DIVERSE-AIM

  • EmoAgent divides D-AIM into Planning, Editing, and Critic roles that translate emotional intent into strategies, execute modifications, and assess emotional correctness.The division supports a closed-loop pipeline for refining results toward emotional and visual goals.Source: §III · p. 4

A. Task Definition: Diverse-AIM

  • D-AIM seeks multiple visually and semantically distinct versions of a source image that express the same target emotion.The outputs should preserve the source image’s high-level semantic structure and visual layout while expressing the intended emotion.Source: A. Task Definition: Diverse-AIM · p. 4

B. Agent Collaboration Workflow

  • In pre-creation, the Planning Agent generates distinct editing plans and the Editing Agent executes them; the Critic independently evaluates each result.Unsatisfactory results enter an independent feedback loop in which the Critic identifies revisions and the Editing Agent applies them until a final result is produced.Source: B. Agent Collaboration Workflow · p. 4

C. Planning Agent

The Planning Agent maps one source image and target emotion to multiple semantically distinct, emotionally aligned editing plans. It retrieves emotion-relevant visual elements and organizes them into structured instructions.

C. Planning Agent

  • The Planning Agent takes a source image and target emotion and produces multiple distinct editing plans, each a coherent path aligned with the image semantics and emotion.Together, the plans express the same emotional intent through diverse visual solutions.Source: C. Planning Agent · pp. 4–5
  • It analyzes source-image semantics and emotion, then retrieves a diverse pool of editing elements from an external Emotion-Factor Knowledge database.The database is organized into emotion–element–method triples.Source: C. Planning Agent · p. 4, Figure 4 · p. 5
  • The plan space has emotion-type roots, editing-element branches, and manipulation-method leaves.Elements include objects, attributes, expressions, backgrounds, and hue; methods include replacement, addition, removal, and other edits.Source: C. Planning Agent · p. 5
  • The agent pairs context-aware elements with suitable methods, formats them as natural-language instructions, and assembles different instruction sets into candidate plans.Each plan represents an independent visual manipulation path.Source: C. Planning Agent · p. 5

D. Editing Agent

The Editing Agent translates each plan into concrete, emotionally aligned image edits using a library of specialized tools. It executes instructions through preprocessing, editing, and validation, with failed validations prompting re-execution or adjustment.

D. Editing Agent

  • The Editing Agent takes editing plans and the source image as input and produces visually diverse edited images aligned with the target emotion.Its tools include text-guided, text-plus-mask, and mask-plus-reference-image modalities.Source: D. Editing Agent · p. 5
  • Each instruction is handled through preprocessing, visual editing, and validation for semantic consistency and emotional alignment.Failed validation triggers re-execution or adjustment of the instruction.Source: D. Editing Agent · pp. 5–6
  • EmoAgent more effectively conveys target emotions than SIM- and AIM-based methods while preserving high-level semantic content.The qualitative comparison is shown in Fig. 5.Source: Figure 5 · p. 6

E. Critic Agent

The Critic Agent evaluates whether edits convey the target emotion and coordinates with the Editing Agent to refine inaccurate or incorrectly executed instructions. The loop continues until the output conveys the target emotion.

E. Critic Agent

  • The Critic Agent identifies mismatches between the target emotion and the emotion perceived in an edited image, then gives the Editing Agent feedback for refinement.It uses structured reasoning across both pre-creation and optimization stages.Source: E. Critic Agent · p. 6
  • If an initial result does not convey the target emotion, the Critic diagnoses whether instructions are emotionally ineffective or incorrectly executed and sends revisions back for re-application.The revised image is re-evaluated, and the loop continues until the output conveys the target emotion.Source: E. Critic Agent · p. 6
  • Critic–Editing Agent collaboration is described as improving reliability while keeping final results emotionally aligned and semantically faithful.Table II is captioned as a quantitative comparison on semantic consistency and emotion alignment; its values are not given here.Source: E. Critic Agent · p. 6, Table II · p. 6

IV. EXPERIMENTS

The experiments use EmoEditSet images and assess semantic consistency, emotion alignment, and emotion-aware diversity. The evaluation protocol compares methods on emotional and semantic outcomes and tests multiple outputs per image for diversity.

A. Implementation Details

  • The implementation uses Qwen2.5-VL as the default VLM, text-embedding-ada-002 for retrieval encoding, and the EmoEditSet test set of 405 real-world images at 512 × 512 resolution.The Emotion-Factor Knowledge database is constructed from clustered EmoSet images and descriptions; the Planning Agent retrieves the top five elements as candidates.Source: A. Implementation Details · p. 6

B. Evaluation Details

  • For one source image, Fig. 6 contrasts EmoEdit’s fixed editing patterns with EmoAgent’s varied, context-aware elements that yield emotionally consistent yet visually distinct edits.Across varied source images, Fig. 7 describes EmoAgent as adapting edits to image semantics while achieving greater emotional fidelity and visual diversity than EmoEdit.Source: Figure 6 · p. 7, Figure 7 · p. 8
  • The evaluation measures semantic consistency with CLIP-I, emotion alignment with Emo-A, Emo-S, and ESR, and emotion-aware diversity with LPIPS and Sem-D.Higher LPIPS indicates greater visual diversity; Sem-D measures average cosine distance between CLIP embeddings of output variants.Source: B. Evaluation Details · p. 7
  • For semantic and emotional evaluation, the protocol randomly assigns a target emotion to each of 405 test images; for diversity, it generates five outputs for each of 50 randomly selected source images.The diversity comparison is with EmoEdit, while semantic and emotional comparisons include representative SIM and AIM methods.Source: B. Evaluation Details · pp. 7–8

C. Comparison Results on Semantic Consistency and Emotion Alignment

EmoAgent outperforms comparison methods on emotion alignment while retaining semantic consistency, and it produces more diverse emotion-aware edits than EmoEdit. Its outputs also adapt visual elements to image context rather than reusing fixed cues.

C. Comparison Results on Semantic Consistency and Emotion Alignment

  • For excitement, anger, and disgust, EmoAgent adds emotion-relevant cues while preserving image content and layout, unlike examples of ineffective or structurally degrading baseline edits.Examples include fireworks, crying expressions and lightning, and dirt, insects, and a greenish hue.Source: C. Comparison Results on Semantic… · p. 8
  • EmoAgent achieves the highest scores on Emo-A, ESR, and Emo-S, and surpasses style-based CLVA on CLIP-I despite a slight semantic-consistency decrease.The comparison also reports that EmoEdit lacks an explicit optimization mechanism for emotional alignment and relies on rigid emotion-to-feature mappings.Source: C. Comparison Results on Semantic… · p. 9

D. Comparison Results on Emotion-Aware Diversity

  • Across different inputs with the same target emotion, EmoAgent varies elements to fit each image, while EmoEdit often repeats cues such as graves or running people.For awe from one source image, EmoAgent also varies foreground and background elements, including waterfalls, sunlight, lakes, skies, and mountains.Source: D. Comparison Results on Emotion-Aware… · p. 9
  • 0.488 LPIPS for EmoAgent versus 0.205 for EmoEdit, and 0.180 Sem-D versus 0.086, indicate greater perceptual and semantic diversity.The authors attribute the richer outputs to adapting visual expressions to the input-image context.Source: Table III · p. 9, D. Comparison Results on Emotion-Aware… · p. 10

E. Ablation Studies

Ablations show that the retriever, editing tools, and critic each contribute to EmoAgent’s emotional alignment or diversity. In a user study, EmoAgent receives higher preference rates than EmoEdit across all four evaluation dimensions.

E. Ablation Studies

  • Without the EFK Retriever, Emo-A declines from 61% to 40% and Sem-D from 0.180 to 0.161, alongside less relevant or emotionally ambiguous concepts.The retriever supplies emotion-grounded visual knowledge for selecting semantically appropriate editing elements.Source: E. Ablation Studies · p. 10
  • Removing the editing tools library lowers Emo-A from 61% to 35% and Emo-S from 0.318 to 0.159, while LPIPS decreases from 0.488 to 0.425.With the full toolset, the Editing Agent selects suitable tools for each instruction, enabling transformations from subtle facial adjustments to large-scale background changes.Source: E. Ablation Studies · p. 10
  • Removing the Critic Agent lowers Emo-A from 61% to 28%, the largest drop among ablation settings, while the critic iteratively identifies and corrects emotionally inaccurate edits.The figure describes iterative evaluation and feedback with the Editing Agent as improving emotional alignment.Source: E. Ablation Studies · p. 11, Figure 10 · p. 11

F. User Study

  • 80.77% Emotion Accuracy, 76.92% Semantic Plausibility, 76.30% Structural-Affective Balance, and 80.74% Emotion-Aware Diversity preferences favor EmoAgent over EmoEdit.The study involved 60 participants and 30 randomly selected test images.Source: F. User Study · p. 11, Table VI · p. 11

V. DISCUSSIONS

EmoAgent edits images toward multiple emotion categories and can address novel emotional intents without additional training. The conclusion frames its planning, editing, and critic stages as supporting diverse, emotionally faithful outputs.

A. Manipulation in Various Emotion Categories

  • EmoAgent modifies images in eight emotional directions using category-specific semantic factors, as illustrated by amusement and awe edits to a dilapidated house.Examples include posters, a phonograph, wooden flooring, chandeliers, murals, and golden lighting.Source: A. Manipulation in Various Emotion… · p. 11, Figure 11 · p. 12

B. Generalization to Out-of-Domain Emotions

  • EmoAgent supports zero-shot generalization to arbitrary emotions by adapting editing strategies to novel targets without additional training.Examples include flowers, puppies, and warm tones for happiness, and night scenes, dark clouds, fallen leaves, and umbrellas for depression.Source: B. Generalization to Out-of-Domain… · p. 12, Figure 12 · p. 12

VI. CONCLUSIONS

  • The conclusion presents D-AIM as generating distinct yet emotionally consistent edits from one image and target emotion, addressing prior one-to-one emotion-to-feature mappings.It describes planning, editing, and critic stages as enabling structured exploration and reports gains over existing AIM methods in emotional alignment and visual diversity.Source: §VI · p. 12
  • Future work will explore user-in-the-loop mechanisms and adaptive modeling to personalize affective outputs because emotional perception varies across individuals.This is stated as a future direction rather than an evaluated capability.Source: §VI · p. 12

APPENDIX

The appendix adds implementation details, ablation results, and further experimental findings. It describes construction of the emotion-factor resources, the editing and evaluation setup, and additional AIM and D-AIM examples.

APPENDIX

  • The supplementary material lists implementation details, ablations, additional AIM and D-AIM comparisons, and workflow visualizations across its sections.It also includes additional experimental findings.Source: APPENDIX · p. 14

A. Implementation Details

  • The emotion-factor tree is built by clustering CLIP embeddings of EmoSet images, filtering clusters, and organizing visual cues hierarchically; the EFK database matches node text with image-semantic embeddings.The appendix also describes the editing-tool implementations and questionnaire materials for subjective evaluation.Source: A. Implementation Details · p. 14, Figure 14 · p. 15, Figure 15 · p. 16

B. Additional Experimental Results

  • Additional material provides cluster examples for sadness and excitement and further qualitative comparisons for affective image manipulation and D-AIM.The supplementary findings include more subjective experimental results and emotion-aware diversity examples.Source: Figure 13 · p. 14, B. Additional Experimental Results · p. 16

C. Workflow Visualization of EmoAgent

EmoAgent’s workflow is presented as an interpretable sequence of agent actions, with a Self-Critic mechanism correcting initial editing errors. The visual examples include both a correctly executed workflow and a workflow in which errors are corrected.

C. Workflow Visualization of EmoAgent

  • EmoAgent follows a workflow of thinking, discovering, and executing, with each action adhering to that sequence.The paper describes the agents as operating with the ReAct logic.Source: C. Workflow Visualization of EmoAgent · p. 16
  • When initial editing actions are incorrect, the Self-Critic mechanism corrects them until all steps are completed correctly.Fig. 19 shows the workflow when editing errors occur; Fig. 18 shows an example without editing or emotional errors.Source: C. Workflow Visualization of EmoAgent · p. 16, Figure 18 · p. 19, Figure 19 · p. 20
  • The workflow examples are presented as making decision-making and emotion processing traceable at each step.The paper identifies this transparency as an interpretability advantage.Source: C. Workflow Visualization of EmoAgent · p. 16
  • EmoAgent’s qualitative comparisons show stronger target-emotion expression while preserving image structure than existing SIM and AIM methods.The comparison is presented in Fig. 17.Source: Figure 17 · p. 18

Related papers

Loading 2503.11290v3…