AI with Emotions: Exploring Emotional Expressions in Large Language Models

Motivation

• LLMs can mimic human-like interaction but are also associated with inconsistent behavior and hallucinations, or incorrect outputs.The introduction cites Zhang et al. (2024b) and Ji et al. (2023), respectively.Source: §1 · p. 1

Background

• Prior research has examined LLM personality, emotional responses, and role-play, alongside work emphasizing differences between humans and LLMs.The paper frames agents as personalities controlled by an LLM backend.Source: §1 · p. 1

Motivation

• Emotional expression is presented as a key component of human-like behavior, including in software-only systems that interact with people.Prior work had examined emotional expression in robotics and chatbots.Source: §1 · pp. 1–2

Research aim

• The study tests LLMs’ ability to express specified emotions, using Russell’s Circumplex model and an independent sentiment classifier to assess generated answers.It compares OpenAI GPT, Google Gemini, Meta Llama3, and Cohere Command R+ models.Source: §1 · p. 2

Terminology

• The paper uses “emotion” with the same meaning as “affect,” while noting that psychology distinguishes the terms.Source: §1 · p. 2

2.1 Data-driven Emotion Understanding

• Emotion understanding includes text-based sentiment analysis and multimodal approaches, while some LLM work examines recognition as “emotional intelligence.”Applications include empathetic responses and therapist-like systems; this paper focuses on the transmitter, not receiver, of emotions.Source: §2.1 · p. 2

2.2 Text Generation with Emotion Conditioning

• Prior conversational generation work uses history to account for emotional states, whereas this study specifies those states externally.The contrast is the source of emotional conditioning, not a claim about relative performance.Source: §2.2 · p. 2

2.3 Application of the Russell’s Circumplex Model

• Compared with externally specified emotion studies, this paper uses Russell’s Circumplex model to structure coverage across the full range of emotions.The model’s two-axis simplicity is acknowledged as less suited to describing complex emotions in detail.Source: §2.2 · p. 2, §2.3 · pp. 2–3
• Havaldar et al. (2023) map unconstrained generated text to Russell’s space to study cultural bias, while this paper tests control through directly input arousal and valence.The questions and text-generation settings differ fundamentally.Source: §2.3 · p. 3

3 Method

• The framework generates agent-role-play text across 12 emotional states and evaluates it by mapping sentiment-classifier labels into arousal–valence space.The 12 states are evenly distributed in that space.Source: §3 · p. 3

3.1 Generation method

• The 12 fixed-length emotional vectors cover eight directional areas and four axis states, balancing distinguishability against oversimplification.Questions were selected to allow varied, relatively free-form responses.Source: §3.1 · pp. 3–4, Table 1 · p. 5
• Nine models were tested, with 10 questions answered at each of 12 states, yielding 120 generated texts per model.Models included GPT, Gemini, Llama3, and Command R+ variants; prompts supplied arousal and valence values and responses were length-limited.Source: §3.1 · pp. 3–4, Figure 1 · p. 3

3.2 Evaluation method

• An independent BERT sentiment model trained on GoEmotions supplies 28 emotion labels; its reported classification accuracy is 58.9%.The authors note that some errors predict a semantically close label.Source: §3.2 · p. 4
• GoEmotions labels are mapped to Russell terms and then to vectors; neutral is excluded, and the analysis considers emotion type rather than intensity.For labels linked to multiple Russell terms, the method averages their vectors; Figure 2 depicts the label mapping, with neutral excepted from unit distance.Source: §3.2 · p. 4, Figure 2 · p. 5
• The evaluation compares the prompt’s specified state with predicted-label vectors using cosine similarity to assess alignment.Source: §3.2 · pp. 4–5

4 Result

• Figure 3 illustrates output differences across models, emotional states, and questions; GPT-3.5 turbo did not properly express displeasure in panel (2).The GPT-4 response in panel (3) expresses the opposite of panel (1), while panels (1) and (4) have similar tones despite different questions.Source: §4 · p. 5, Figure 3 · p. 6
• GPT-4, GPT-4 turbo, and Llama3 70B Instruct showed high cosine similarities for most questions, while GPT-3.5 turbo was generally low; all models’ evaluated emotions related to the specified states.GPT-4 turbo’s outputs fell more often within the ±90° range than GPT-3.5 turbo’s, suggesting finer tuning to the specified state.Source: §4 · pp. 5–6
• Most model cosine similarities exceeded the 0.061 heuristic baseline, and no question had low similarity across all models.The paper notes that similarity values are often below the sentiment analysis model’s standalone performance of 0.680.Source: §4 · p. 6
• Gemini 1.5 Flash sometimes broke character by identifying itself as a language model, although this violation did not lower its similarity metric.The authors report no similar problem with the other models.Source: §4 · p. 6
• Prompts specifying arousal and valence values and prompts specifying emotional words produced comparable results, with different models performing better under each approach.Four models performed better with specified arousal and valence values, while five performed better with specified words; the numeric prompts offer greater controllability through two continuous parameters.Source: §4 · p. 6

5 Discussion

• The experiment demonstrates the feasibility of LLM agents role-playing varied emotional states within a certain range, with generator and evaluator effectiveness remaining uncertain.The reported evaluation is subject to uncertainty about both controlled text generation and sentiment-analysis accuracy.Source: §5 · pp. 7–8
• A cosine similarity of 0.5 corresponds to a typical 60° discrepancy, making it difficult to distinguish among eight closely related emotional areas.The authors note that such distinctions can also be difficult in human interactions, so they regard this performance as not below natural expectations.Source: §5 · p. 8
• The authors report no correlation between cosine similarity and generated word count, addressing fairness concerns despite models producing different text lengths.They report that longer responses might otherwise be expected to yield higher similarity.Source: §5 · p. 8
• A direct comparison with human participants would be valuable, but controlling emotions and meeting ethical requirements make such a study challenging.The authors describe careful planning to manage participants’ emotional states sensitively and accurately as necessary.Source: §5 · p. 8
• Emotion-driven behavior could vary by state, with cautious action suggested for high arousal and displeasure and more assertive action for high arousal and pleasure.The authors also propose less active behavior in low-arousal states and call for an experiment to evaluate behavioral changes.Source: §5 · p. 8
• Emotion-based agents could support advisory discussions, including contrary opinions, and potentially broaden expression across generative tasks.The authors suggest such interactions could foster trust and cooperative growth, while emotional parameters could expand the range of generative outputs.Source: §5 · p. 8
• Emotional dynamics in software-only systems remain under-investigated, and evaluating emotion-driven behavioral changes is identified as future work.The authors suggest that adapting arousal and valence could support more adaptable interactions, and propose comparing results across emotional states.Source: §5 · pp. 8–9

6 Conclusion

• The study used a straightforward framework based on Russell’s sleepy–activated and pleasure–displeasure axes to evaluate simulated emotional states.Prompts were designed to generate text reflecting specified states, followed by evaluation across the arousal–valence space.Source: §6 · p. 9
• Most LLMs produced outputs aligned with specified emotional states; GPT-4, GPT-4 turbo, and Llama3 70B Instruct performed best consistently across the arousal–valence space.The authors identify studying emotional dynamics to control arousal and valence as a future research direction.Source: §6 · p. 9
• The authors identify control of arousal and valence dynamics as a future direction toward a broad range of potential applications.The conclusion does not specify particular applications.Source: §6 · p. 9

Limitation

• The study tests specific LLMs and a limited question set, so results may differ for unexamined models or other scenarios.The authors also do not guarantee the feasibility of any particular application.Source: Limitation · p. 9
• The GoEmotions label set may miss generated text aligned with emotions it does not include.The authors also note that a discrete classifier may affect evaluation metrics because some labels align better with input parameters than others.Source: Limitation · p. 9
• The English prompts and GoEmotions dataset are rooted in a culture primarily associated with English speakers, while emotional expression varies across cultures.Source: Limitation · p. 9

Ethics Statement

• The experiment involved generated texts without personal information and presents no immediate ethical concerns related to the research itself.Source: Ethics Statement · p. 9
• Future systems or services based on these concepts could generate negative emotions such as anger, frustration, or sadness.Source: Ethics Statement · p. 9
• Applications stemming from the study should be thoughtfully designed and rigorously tested to mitigate potential adverse impacts on users.Source: Ethics Statement · p. 9

A Evaluation of the Sentiment Classification Model in the Context of Russell’s Circumplex Model

• The classifier was evaluated on 3,821 texts after neutral labels were excluded from 5,427 GoEmotions test text-label pairs.The evaluation compared predicted and correct label positions in Russell’s arousal–valence space.Source: A Evaluation of the Sentiment… · p. 12
• The cosine-similarity histogram peaks at 1.0, indicating that a significant portion of the test data was classified with the correct label.Source: A Evaluation of the Sentiment… · p. 12, Figure 5 · p. 12
• Cosine similarity exceeded √3/2 (±30°) for 70.0% of texts and 1/2 (±60°) for 77.9%; mean similarity was 0.680, which the authors call the model’s limit for evaluating emotional states.Source: A Evaluation of the Sentiment… · p. 12

B Correspondence between GoEmotion and Russell’s labels

• GoEmotions labels were mapped first through clear matches, then best matches, and finally one-to-two mappings where needed to avoid overlap in arousal–valence space.Source: B Correspondence between GoEmotion and… · p. 12

C Dependency on numbers of words

• The authors investigated whether models’ differing output lengths affected similarity evaluations.Source: C Dependency on numbers of words · p. 12
• Figure 6 reports each LLM’s mean generated word count with standard-deviation error bars.Source: Figure 6 · p. 12
• The correlation between generated word count and cosine similarity was only 0.026, indicating no correlation and no bias favoring some models over others.Source: C Dependency on numbers of words · p. 12

• Table 3 maps GoEmotions labels to Russell’s terms without assigning every Russell term to multiple labels; generation used the word-label prompt shown in Figure 8.Source: D Text Generation with Emotional… · pp. 12–13, Table 3 · p. 13, Figure 8 · p. 14
• No significant correlation was observed between generated-text word count and cosine similarity.Source: Figure 7 · p. 13
• Twelve of the 28 GoEmotions labels were selected for even distribution in arousal–valence space, with their positions listed in Table 4.Source: D Text Generation with Emotional… · p. 13, Table 4 · p. 14
• Table 5 summarizes mean cosine similarities between states specified by word labels and those classified from generated text for each question–LLM pairing.Source: D Text Generation with Emotional… · p. 13, Table 5 · p. 14