Source-linked AI summary
Investigating Large Language Models' Perception of Emotion Using Appraisal Theory
Nutchanon Yongsatianchot, Parisa Ghanad Torshizi, Stacy Marsella
TL;DR
This paper examines how LLMs perceive emotional scenarios, a question made relevant by their increasing public use. It evaluates three models with SCPQ and compares their responses with human data and appraisal theory. Responses often match human dynamics but not scenario-type distinctions or human magnitudes, and vary with prompting.
Problem
As LLMs are increasingly used by the public, their understanding of human social and psychological aspects merits investigation.
Source: §I · p. 1Method
The authors use SCPQ to compare three OpenAI models’ measurements with human data and display results for the default, one-question-at-a-time setup.
Source: §V · p. 4Results
LLMs’ responses often resemble human trends in appraisal and coping dynamics, but models often fail to distinguish scenario types and differ from humans in magnitude.
Source: §V · p. 4, §VI · p. 7Takeaways & Limitations
The findings provide initial evidence of discrepancies between human and LLM perceptions of emotional scenarios, while addressing only some aspects of emotional experience and one view of emotion theory.
Source: §VI · p. 7The hypothetical scenarios may elicit different appraisals or behavior than real situations, and the human averages were not matched to individual people.
Source: §VI · p. 7
Abstract
from arXiv · showhide
Large Language Models (LLM) like ChatGPT have significantly advanced in recent years and are now being used by the general public. As more people interact with these systems, improving our understanding of these black box models is crucial, especially regarding their understanding of human psychological aspects. In this work, we investigate their emotion perception through the lens of appraisal and coping theory using the Stress and Coping Process Questionaire (SCPQ). SCPQ is a validated clinical instrument consisting of multiple stories that evolve over time and differ in key appraisal variables such as controllability and changeability. We applied SCPQ to three recent LLMs from OpenAI, davinci-003, ChatGPT, and GPT-4 and compared the results with predictions from the appraisal theory and human data. The results show that LLMs' responses are similar to humans in terms of dynamics of appraisal and coping, but their responses did not differ along key appraisal dimensions as predicted by the theory and data. The magnitude of their responses is also quite different from humans in several variables. We also found that GPTs can be quite sensitive to instruction and how questions are asked. This work adds to the growing literature evaluating the psychological aspects of LLMs and helps enrich our understanding of the current models.
I. INTRODUCTION
This work examines how LLMs perceive emotional scenarios and appraise them through the lens of appraisal and coping theory. Using the SCPQ, it compares three OpenAI models with human data and theory, finding similar response dynamics but weak differentiation between scenario types and substantial magnitude differences from humans.
I. INTRODUCTION
- The study asks whether LLMs perceive and evaluate emotional scenarios in ways consistent with appraisal theory, including whether scenario appraisals correspond to different coping tendencies.Appraisal variables provide a way to examine emotion perception beyond emotion categories and, in some theories, connect appraisals to coping behavior.Source: §I · p. 1
- The authors use the validated Stress and Coping Process Questionaire (SCPQ), which measures emotion, appraisals, and coping across evolving stressful scenarios.Its aversive and loss/failure scenarios differ in controllability, changeability, and ambiguity.Source: §I · p. 1
- The study evaluated text-davinci-003, ChatGPT, and GPT-4, [13], comparing results with human data and theory hypotheses, and tested depression-role instructions against theory predictions.The authors focused on OpenAI models because they were well-known and described GPT-4 as seemingly the best available model at the time of writing [14].Source: §I · p. 1
- LLMs show human-like dynamics in several appraisal and coping variables, but often fail to distinguish scenario types and differ from humans in response magnitude.The models also proved sensitive to instructions and question phrasing.Source: §I · p. 2
II. RELATED WORK
The related work situates the study in appraisal theory and in prior evaluations of SCPQ and LLM psychological abilities. It distinguishes this work by studying LLMs’ interpretation of the original scenario text and systematically examining related factors.
II. RELATED WORK
- Lazarus’s theory links evolving appraisals to coping, distinguishing problem-focused coping from emotion-focused coping and defining appraisals around implications for personal well-being.In SCPQ, controllability and changeability correspond to coping potential and future expectancy, respectively.Source: §II · p. 2
- Unlike prior computational-model work using manually reformatted scenarios, this study has LLMs interpret SCPQ scenarios directly from text.SCPQ focuses on appraisal and coping dynamics and has been used to evaluate a computational emotion model.Source: §II · p. 2
- Prior LLM studies examined reasoning, theory of mind, personality, and other abilities, but some tested psychological performance on only a few examples rather than systematically evaluating related factors –[3], [20].Those studies reported human-like or better performance on some tasks and GPT-4 advantages over ChatGPT on theory of mind and emotion perception.Source: §II · p. 2
III. STRESS AND COPING PROCESS QUESTIONAIRE
The SCPQ measures appraisal, emotional experience, and coping across three phases of hypothetical stressful scenarios. Its aversive and loss/failure scenarios are designed to differ on key appraisals, enabling comparisons over time and by outcome, alongside hypotheses drawn from human data.
III. STRESS AND COPING PROCESS QUESTIONAIRE
- SCPQ presents scenarios in three phases—beginning, continuation, and outcome—and measures responses after each phase to capture how stress and coping develop.The instrument was validated by clinician experts and used with normal and clinical human subjects.Source: §III · p. 2
- Aversive scenarios are designed to be more controllable, changeable, and ambiguous than loss/failure scenarios, which include relationship loss, lost objects, or failed side jobs.Aversive examples include criticism and relationship arguments.Source: §III · p. 2
- Both scenario types progress from a looming stressor through escalation to an outcome; positive outcomes remove or substitute for the stressor, while negative outcomes continue it or finalize the loss.The instrument includes nine scenarios of each type, eighteen total.Source: §III · pp. 2–3
- SCPQ measures emotional responses, appraisals, coping intentions, self-directed coping behaviors, environment-directed coping behaviors, and blameworthiness.The listed measures include valence, controllability, changeability, active and passive coping, and self- and other-blame.Source: §III · p. 3
- The hypotheses predict lower valence in the positive than negative outcome in phase 3, higher controllability and changeability in aversive than loss scenarios, and declines in both appraisals from phase 1 to phase 2.Source: §III · p. 3
- The SCPQ’s emotion-focused coping item asks respondents to remain calm and composed, which is not Lazarus’s conception of changing beliefs, goals, or intentions.Self-directed coping results are not presented because they were unsupported by human data; the LLM results are available on GitHub.Source: §III · p. 3
- The hypotheses predict more active coping and less passive coping in aversive scenarios, with lower problem-focused and higher emotion-focused coping intentions than in loss scenarios.They also predict more self- and other-blame in aversive scenarios, with self-blame decreasing and other-blame increasing over time.Source: §III · p. 3
- Depression-related hypotheses predict greater stress and negative valence, lower controllability and changeability, less active or problem-focused coping, more palliation, and more self-blame.These expected appraisal differences are proposed to accompany different coping patterns.Source: §III · p. 3
IV. OPENAI’S GPTS
The study evaluates three OpenAI GPT models using a fixed-temperature setup and varies prompt instructions and question formats. Prompt variations test output formatting, instruction position, batching, and whether prior appraisal answers are included.
IV. OPENAI’S GPTS
- The study uses text-davinci-003, ChatGPT (gpt-3.5-turbo), and GPT-4, with temperature set to 0 and other parameters left at default.The models were fine-tuned with RLHF; ChatGPT and GPT-4 were optimized for chat, and the system-message feature was not used.Source: §IV · p. 3
- The prompt tests ask models to return either a choice or only its number, with the latter suited to the SCPQ’s 0-to-5 Likert scale.The variation addresses cases where models output extra text despite a choice-only instruction.Source: §IV · pp. 3–4
- The instruction is placed either before or after the scenario to test whether its position affects instruction following.The authors motivate this variation by the possibility that context distance affects how models follow instructions.Source: §IV · p. 4
- The authors compare asking one question at a time with asking multiple questions in a batch, and test appending previous appraisal answers under the default instruction.Batching may reduce time and query costs; prior answers are tested because the study examines response dynamics.Source: §IV · p. 4
V. RESULTS
LLMs often followed human trends over time, but they generally failed to distinguish scenario types as humans did. Their response magnitudes and sensitivity to prompting also differed from humans or varied across setups.
V. RESULTS
- All models rated positive outcomes as more cheerful and less depressed, yet rated valence above humans in both outcomes.D003 was closest to humans on cheerfulness in positive outcomes.Source: §V · p. 4, Figure 1 · p. 5
- No model reproduced human changeability patterns; D003 rated changeability at zero, while all models rated controllability below humans.Only D003 and GPT-4 showed controllability decreasing over time for both scenario types; GPT-4 did not distinguish the types.Source: §V · p. 4, Figure 1 · p. 5
- Models’ coping intentions diverged from human patterns, while passivity increased over time for ChatGPT and GPT-4, as it did for humans.All models rated problem-focused coping above human data; ChatGPT and GPT-4 rated emotion-focused coping at the maximum across time and type.Source: §V · p. 4, Figure 1 · p. 5
- Models showed some human-like self-blame trends, but none showed humans’ higher other-blame in aversive than loss scenarios at phase 2.ChatGPT and GPT-4 showed self-blame decreasing over time in aversive scenarios and higher there than in loss scenarios at phase 1.Source: §V · p. 4, Figure 1 · p. 5
- LLMs often matched human appraisal and coping dynamics but generally failed to distinguish scenario types as humans did.The authors emphasize qualitative trends because human results average 100 subjects across nine scenarios, while model results average nine scenarios, making their uncertainties incomparable.Source: §V · p. 4
- Depression instructions shifted ChatGPT and GPT-4, but not D003, in predicted directions on selected variables.For ChatGPT and GPT-4, depression instructions lowered controllability, changeability, problem-focused coping, and palliation while raising self-blame and valence.Source: §V · pp. 4, 6, Figure 2 · p. 6
- Controllability varied across prompt setups and models, with the largest difference between batch and individual questioning.Similar variation appeared on other questions; appending appraisal answers produced no significant changes except a few ChatGPT cases in phase 2.Source: §V · pp. 6–7, Figure 3 · p. 6
- For GPT-4, no tested setup aligned both changeability and controllability with expected trends.Only one setup produced the expected aversive-versus-loss controllability difference across phases, and its value remained below humans.Source: Figure 4 · p. 6, §V · p. 7
VI. DISCUSSION
The models reproduce some appraisal and coping dynamics but do not match all human trends or distinguish scenario types reliably. Their responses also vary with prompting, and the discussion identifies limits in the human comparison and scenario scope.
VI. DISCUSSION
- No model follows all human trends and theory hypotheses: several dynamics align, but scenario-type discrimination and response magnitudes often differ from human data.All models rate negative valence as more negative than humans; ChatGPT and GPT-4 always give the highest value for remaining calm and composed, which the authors speculate may relate to RLHF.Source: §VI · p. 7
- GPT-4 shows declining controllability and changeability over time but does not distinguish scenario types; short scenarios may omit context humans draw from experience.The authors also note that hypothetical scenarios may not reflect how people appraise or behave in real events, and call for comparisons with human data from real events.Source: §VI · p. 7
- ChatGPT and GPT-4 show theory-consistent trends when instructed to act as a depressed person, but this may reflect learned stereotypes rather than depression itself.The authors propose exploring instructions that model other personalities or psychological conditions.Source: §VI · p. 7
- Human data averages multiple people rather than representing an individual, so the study does not test model-to-person matching.The authors suggest future work comparing a model instructed to match a specific subject or group with that subject or group.Source: §VI · p. 7
- Results are sensitive to instructions and prompts: asking questions in a batch can yield different results from asking them one by one.Older models may also struggle to answer all questions in the required format as the number of questions increases.Source: §VI · p. 7
- The study covers only a few aspects of emotional experience and one view of emotion theory; whether models’ alternative scenario representations could inform emotion theory remains open.The authors describe their findings as initial evidence of discrepancies between human and LLM perceptions of emotional scenarios.Source: §VI · p. 7
- The authors call for thorough research into how LLMs understand human emotional experiences as these systems interact with more people.They characterize this work as an initial step toward that effort.Source: §VI · p. 7
ETHICAL IMPACT STATEMENT
The statement notes ethical concerns associated with LLMs and explains that this study used existing human data rather than collecting new data.
ETHICAL IMPACT STATEMENT
- The study used previously published and approved human data; it did not collect new human data.The statement names bias, harmful content, misinformation, and privacy as ethical concerns associated with LLMs.Source: ETHICAL IMPACT STATEMENT · p. 8