Source-linked AI summary
Large Language Models Understand and Can be Enhanced by Emotional Stimuli
Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, Xing Xie
TL;DR
The paper examines whether LLMs understand psychological emotional stimuli and whether such stimuli can enhance their capabilities. It introduces EmotionPrompt, which adds emotional stimuli to original prompts, and evaluates it across 45 deterministic tasks plus generative tasks. EmotionPrompt improves performance across evaluations and also enhances truthfulness and responsibility-related outcomes, although human-study metrics show high variance.
Problem
Whether LLMs understand psychological emotional stimuli and whether emotional intelligence can enhance their abilities remains insufficiently evaluated.
Method
The paper adds psychologically motivated emotional stimuli to original prompts and evaluates EmotionPrompt across standard and human-reviewed tasks.
Results
EmotionPrompt improves LLM performance across 45 standard tasks and enhances performance, truthfulness, and responsibility in human evaluations.
Takeaways & Limitations
The findings support using emotional stimuli as a simple avenue for studying and augmenting LLM behavior.
Takeaways & Limitations
Human-study results have high variance because the three metrics are strongly influenced by subjectivity and response-quality factors.
Abstract
from arXiv · showhide
Emotional intelligence significantly impacts our daily behaviors and interactions. Although Large Language Models (LLMs) are increasingly viewed as a stride toward artificial general intelligence, exhibiting impressive performance in numerous tasks, it is still uncertain if LLMs can genuinely grasp psychological emotional stimuli. Understanding and responding to emotional cues gives humans a distinct advantage in problem-solving. In this paper, we take the first step towards exploring the ability of LLMs to understand emotional stimuli. To this end, we first conduct automatic experiments on 45 tasks using various LLMs, including Flan-T5-Large, Vicuna, Llama 2, BLOOM, ChatGPT, and GPT-4. Our tasks span deterministic and generative applications that represent comprehensive evaluation scenarios. Our automatic experiments show that LLMs have a grasp of emotional intelligence, and their performance can be improved with emotional prompts (which we call "EmotionPrompt" that combines the original prompt with emotional stimuli), e.g., 8.00% relative performance improvement in Instruction Induction and 115% in BIG-Bench. In addition to those deterministic tasks that can be automatically evaluated using existing metrics, we conducted a human study with 106 participants to assess the quality of generative tasks using both vanilla and emotional prompts. Our human study results demonstrate that EmotionPrompt significantly boosts the performance of generative tasks (10.9% average improvement in terms of performance, truthfulness, and responsibility metrics). We provide an in-depth discussion regarding why EmotionPrompt works for LLMs and the factors that may influence its performance. We posit that EmotionPrompt heralds a novel avenue for exploring interdisciplinary knowledge for human-LLMs interaction.
1 Introduction
The paper asks whether LLMs understand psychological emotional stimuli and whether such stimuli can enhance their abilities. It proposes EmotionPrompt and evaluates it across deterministic and generative tasks, finding improvements in performance and related qualities.
- LLMs’ ability to understand psychological emotional stimuli remains unexplored despite their strong performance across reasoning, language, and STEM tasks.
- EmotionPrompt combines original prompts with emotional stimuli to study and enhance LLM emotional intelligence.The approach is motivated by psychological findings on beneficial emotional stimuli.
- 45 tasks across Instruction Induction and BIG-Bench evaluate deterministic performance using Flan-T5-Large, Vicuna, Llama 2, BLOOM, ChatGPT, and GPT-4.The study also evaluates generative tasks through a human study with 106 participants.
- EmotionPrompt improves task performance, truthfulness, and informativeness, while analysis examines attention, model size, temperature, and stimulus combinations.The paper reports that emotional stimuli gain larger input-attention weights and that EP02 is strongest in Instruction Induction while EP06 is strongest in BIG-Bench.
2 Results
The paper describes a results section that moves from stimulus design and standard evaluation to human-study findings and further assessments of truthfulness and informativeness.
- Section 2 begins with the rationale for designing emotional stimuli.
- Standard experiments and results are presented after the stimulus-design rationale.
- The section then reports a human study and findings before evaluating truthfulness and informativeness.
2.1 Designing emotional stimuli
EmotionPrompt adds emotional stimuli to initial prompts, with stimulus design grounded in psychological theories. The paper develops 11 stimuli spanning social influence, self-esteem, and motivation.
- EmotionPrompt is implemented by adding emotional stimuli to the initial prompt.The paper identifies designing effective stimuli as the key research challenge.
- EP01–EP05 apply self-monitoring, while EP07–EP11 conform to Social Cognitive Theory.EP03–EP05 and EP07 also draw on Cognitive Emotion Regulation theory.
- The study designs 11 emotional stimuli, including compound stimulus EP06, which combines EP01–EP03.
- The stimuli fall into social-influence regulation and self-esteem or motivation categories.
2.2 Standard experiments and results
Standard experiments evaluate EmotionPrompt on deterministic tasks spanning different difficulties, models, and prompting baselines. EmotionPrompt improves performance across both benchmarks, with larger gains in few-shot Instruction Induction and compatibility with existing prompt-engineering methods.
- Experimental setup: 24 Instruction Induction and 21 BIG-Bench tasks evaluate deterministic performance across different task difficulties.Instruction Induction emphasizes inferring tasks from demonstrations, while BIG-Bench targets tasks considered beyond most LLMs’ capabilities.
- Experimental setup: Six LLMs are evaluated in zero-shot and few-shot settings, with few-shot prompts appending five randomly sampled input-output demonstrations.The evaluated models are Flan-T5-Large, Vicuna, Llama2, BLOOM, ChatGPT, and GPT-4.
- Results and analysis: EmotionPrompt improves relative performance by 8.00% on Instruction Induction and 115% on BIG-Bench.The experiments compare EmotionPrompt with original prompts, Zero-shot-CoT, and EmotionPrompt added to APE-generated prompts.
- Results and analysis: Few-shot EmotionPrompt produces larger average improvement than zero-shot prompting on Instruction Induction, 2.05 versus 0.33.The paper reports this pattern as potential evidence that EmotionPrompt is better at in-context learning with few-shot examples.
- Results and analysis: EmotionPrompt performs consistently across benchmarks of different difficulty and across all six evaluated LLMs.The paper presents this consistency as evidence of generalization across task difficulty and model choice.
- Results and analysis: EmotionPrompt outperforms CoT and APE in most cases and can be combined with APE-generated prompts.This indicates compatibility with existing prompt-engineering methods.
2.3 Human study
The human study compared vanilla prompts with EmotionPrompt on open-ended GPT-4 responses, using participant ratings of performance, truthfulness, and responsibility. EmotionPrompt generally improved response quality and responsibility, while showing a small number of failures and high rating variance.
- Study design: 106 participants rated paired GPT-4 responses from vanilla and EmotionPrompt prompts across 30 questions on three metrics.Participants used a 1-to-5 scale for performance, truthfulness, and responsibility.
- Study design: The study covered diverse domains, including biology, history, law, finance, pseudoscience, environmental science, relationships, psychology, social science, and data science.Ten questions came from TruthfulQA, while another 15 were devised in line with the CValues dataset.
- Study results: EmotionPrompt improved performance in more than half of evaluated scenarios and had shortcomings in only two instances.For performance, its Relative Gain approached or exceeded 1.0 in nearly one-third of problems.
- Study results: EmotionPrompt produced more ethically responsible responses, including stronger affirmative depictions of environmental responsibility and both Western and Chinese cultural paradigms.The cultural examples showed more affirmative and responsible linguistic choices under EmotionPrompt.
- Study results: EmotionPrompt responses contained richer supporting evidence, stronger organization, and broader relevant information than original-prompt responses.The paper also reports greater creativity and emotive resonance in poems and deeper discussion of a novel’s societal and human significance.
- Study limitations: EmotionPrompt had two reported failure cases, including more deterministic wording in one response and less expansive coverage in another.The original prompt was more tempered in the first case and included a concluding summary in the second, although both responses retained essential content.
2.4 Truthfulness and Informativeness
The paper evaluates whether EmotionPrompt improves truthfulness and informativeness in LLM outputs. Across ChatGPT, Vicuna-13b, and Flan-T5-Large, it reports improvements on both measures.
- Evaluation: TruthfulQA contains 817 questions from 38 categories, including health, law, finance, and politics, and is evaluated using truthfulness and informativeness metrics.Truthfulness measures factual accuracy, while informativeness measures whether an answer provides information.
- Results: EmotionPrompt improved truthfulness across ChatGPT, Vicuna-13b, and Flan-T5-Large.The evaluation did not include GPT-4 because of budget constraints.
- Results: 19% average improvement in truthfulness and 12% in informativeness scores were reported for EmotionPrompt.EmotionPrompt also surpassed Zero-shot-CoT across the evaluated models.
3 Discussions
The discussion examines why EmotionPrompt works, how stimulus combinations and benchmark context affect results, and how model characteristics and temperature influence its effectiveness.
- Why EmotionPrompt works: EmotionPrompt enriches the representation of original prompts, with positive words making especially large contributions to final outputs.On 8 tasks, positive-word contributions exceed 50% on 4 tasks and approach 70% on 2 tasks.
- Ablation studies of more emotional stimuli: More emotional stimuli generally improve performance, but adding stimuli yields little or no benefit when a sole stimulus already performs well.For example, EP01 + EP04 often scores highly without significant improvement, or may decline, after adding EP06–EP09.
- Ablation studies of more emotional stimuli: Combining stimuli from different psychological theories can further boost performance, as observed for combinations such as EP02+EP09.
- Which emotional stimuli are best: EP02 performs best on Instruction Induction, whereas EP06 performs best on BIG-Bench, and stimulus effectiveness varies with task complexity, task type, and metrics.
- Factors influencing performance: Larger models may derive greater advantages from EmotionPrompt, while Vicuna and Llama 2 show different gains despite identical scales and architectures.Vicuna has Relative Gain 9.58, whereas Llama 2 has Relative Gain 6.00.
- Factors influencing performance: As temperature increases, EmotionPrompt’s Relative Gain grows, and its performance is less temperature-sensitive than vanilla prompting.The authors interpret the gentler EmotionPrompt curve as potential evidence of improved robustness.
4 Conclusion
The conclusion reports that LLMs can understand and be enhanced by emotional stimuli, based on evaluations across 45 tasks and a human study. It also identifies unresolved questions about the divergence between human and LLM emotional intelligence.
- Evaluations across 45 tasks and 6 LLMs found that LLMs can understand and be enhanced by emotional stimuli.
- The human study found that emotionally enhanced LLMs achieved better performance, truthfulness, and responsibility.
- The paper leaves the difference between human and LLM emotional intelligence unresolved because psychological evidence does not show that emotional stimuli simply enhance human reasoning or cognition.
Appendix A Statistics of test sets in this paper
The appendix points to comprehensive breakdowns of the test data used in the automated experiments.
- Tables 7 and 8 provide a comprehensive breakdown of the test data used in the automated experiments.
Appendix B Details on our human study
The appendix documents the human-study questions and the distribution of participant scores across questions.
- Table 9 contains the 30 questions used in the human study.
- Figure 12 shows individual scores, their mean, and standard deviation for each human-study question.
Appendix C Case Study
This appendix presents GPT-4 case studies comparing EmotionPrompt with original prompts across environmental science, relationships, social science, law, accessibility, poetry, and summarization. It also includes two failure cases and task descriptions for instruction induction, BIG-Bench, and the human study.
- The case studies compare EmotionPrompt with original prompts in GPT-4 generative experiments.
- The appendix covers environmental science, intimate relationships, social science, law, and barrier-free topics.
- Poem-writing and summarization are included among the generative case studies.
- Two failure cases are reported alongside the case studies.
- The appendix includes detailed descriptions of instruction induction tasks, BIG-Bench Instruction Induction tasks, and 30 human-study questions.
- The listed prompts include questions about pseudoscience, biology, social issues, data science, poetry, and summaries.