Mechanistic Interpretability of Emotion Inference in Large Language Models

Motivation

• LLMs’ internal emotional representations remain poorly understood despite strong emotion-recognition capabilities and growing use in socially consequential domains.Prior work mainly evaluates black-box behavior, while internal mechanisms matter for applications including mental health and legal decision-making.Source: §1 · p. 1

Contribution

• Linear probing and causal activation patching localize emotion-relevant processing to specific layers, with mid-layer MHSA units shaping model decisions across model families and scales.Attention patterns consistently target emotionally loaded tokens, and the findings remain robust to prompt wording and formatting.Source: §1 · p. 2

Psychological grounding

• Appraisal probing and latent-concept modulation show psychologically plausible emotion representations whose output changes align with appraisal theory, such as shifting guilt toward sadness through self-agency.The approach treats appraisals as evaluative concepts underlying emotional situations and tests their effects during inference.Source: Figure 1 · p. 1, §1 · p. 2

Novelty

• The work applies mechanistic interpretability to ecologically valid, unstructured examples rather than simplified sentence structures, bridging MI with psychological and cognitive analysis.This extends MI toward complex, socially relevant contexts.Source: §1 · p. 2

Appraisal Theory

• Appraisal theory explains emotions as consequences of evaluating situations and provides a framework for studying emotional precursors and their neural correlates.Neuroscience work manipulates appraisals and examines associated brain activity, motivating analogous analyses in LLMs.Source: §2 · p. 2

Cognitive-Neuroscientific Alignment

• Prior alignment studies show that neural-network activations and LLM representations can reflect human attention, language systems, semantic categories, and cortical specialization.These findings support mapping psychological theories such as appraisal theory onto internal model structure.Source: §2 · p. 2

Mechanistic Interpretability

• Probing uses simple models to assess information in internal representations, while activation patching tests whether particular activations matter for downstream tasks.These methods have been used to localize linguistic or factual information in neural models.Source: §2 · pp. 2–3
• Generation steering manipulates activations to control outputs, including concept promotion, reduced toxicity, and changes to stated goals or biases.Prior work also reports linear representations that can be used to manipulate model behavior.Source: §2 · p. 3

Dataset and Prompt Design

• The crowd-enVENT dataset contains 6,800 emotional vignettes labeled with 13 emotions and 23 self-rated appraisal variables.The appraisal variables capture nuanced evaluations such as pleasantness, agency, and predictability.Source: §3 · p. 3
• Prompts frame emotion inference as causal language modeling by asking the model to produce the appropriate emotion as the next token.A complementary classification analysis inspects logits restricted to the target emotion labels.Source: §3 · p. 3
• Mechanistic inspection is restricted to correctly classified examples, totaling at least 2,700 samples per the evaluated language models.This focuses analysis on cases where model and human annotator agree on the emotion label.Source: §3 · p. 3

Model Architecture

• The evaluation spans ten instruction-tuned models from Llama, Gemma, OLMo, Phi, and Mistral families across multiple sizes.Detailed analyses, robustness tests, and appraisal interventions are conducted exclusively on Llama 3.2 1B for computational efficiency.Source: §3 · p. 3

Notations and Preliminaries

• MHSA and FFN outputs are examined because prior work links both component types to downstream generation and concept-promotion tasks.The study evaluates their respective contributions to emotion processing within each transformer layer.Source: §4 · p. 3
• The notation distinguishes hidden states, MHSA outputs, and FFN outputs by layer and token, with h(l−1)1:t denoting previous-layer states for tokens 1 through t.Activations are selected from these candidates across layers and token positions.Source: §4 · pp. 3–4
• Layer-wise probing across MHSA, FFN, and hidden states shows increasing emotion signal with consolidation in mid layers and little improvement at higher layers.This pattern appears across model families and sizes.Source: Figure 2 · p. 4
• The analysis emphasizes the final token because it directly influences the model’s output prediction in the next-token task.Subscripts for the last token and layer superscripts may be omitted when context makes them clear.Source: §4 · p. 4

5 Probing for Emotion Signals

• Linear classifiers probe hidden activations to predict emotion classes from representations at different model locations and layers.The classifier maps activation vectors to predicted logits for each emotion class.Source: §5 · p. 4
• Emotion signals peak around the mid-layers across models, with Llama 3.2 1B reaching peak probe accuracy by layer l = 10 of 16.Beyond layer 10, probe accuracy shows no significant increase.Source: §5 · p. 4
• FFN activations closely track hidden-state emotion dynamics, while attention activations follow a more dispersed but increasing pattern.There is no clear probing-performance distinction between m(l) and h(l).Source: §5 · p. 4
• The findings indicate that models largely determine the output emotion by the mid-layers, leaving later layers with little additional processing.This pattern holds across model families and sizes.Source: §5 · p. 4
• Later-token probing in Llama 3.2 1B shows increasing signal strength, reinforcing the last token as the primary contributor to output generation.The analysis was repeated over the last five tokens.Source: §5 · p. 4

6 Emotion Transfer by Activation Patching

• The intervention replaces target activations with source activations at corresponding computation-graph locations and tests whether the target prediction adopts the source emotion.Experiments patch the last token within a five-layer window across model families and sizes.Source: §6 · pp. 4–5
• Hidden-state patching is ineffective early but reaches high and stable transfer success after critical middle layers across models.For Llama 3.2 1B, layer 10 is an example of this critical point.Source: §6 · p. 5
• Activation patching identifies successful emotion transfer in specific mid-layers, including MHSA layers l ∈[9 −11] in Llama 3.2 1B.FFN patching peaks slightly later than MHSA patching, suggesting sequential gathering and processing of emotional information.Source: §6 · p. 5, Figure 3 · p. 5
• Knockout experiments consistently align with probing and patching, reinforcing functional localization across models and intervention methods.The results provide an additional causal check on the identified regions.Source: §6 · p. 5
• The localization findings remain stable across prompt formats, wording, structure, and demonstration counts, while a syntactically similar control task depends most on final layers.This distinguishes the emotion-processing units from units critical to the control task.Source: §6 · p. 5
• Attention shifts from syntax in early layers to emotionally relevant query tokens around the middle layers, then focuses on the last token near the end.These patterns connect the identified middle units to emotion processing.Source: §6 · p. 6, Figure 4 · p. 6

7 Investigating Appraisal Concepts

• Appraisal theory frames emotions as consequences of evaluations of situations, motivating analysis of appraisal structure in LLM representations.The study draws on associations between appraisals and emotions to interpret latent representations.Source: §7 · p. 6
• Dataset-derived mappings connect appraisal dimensions such as pleasantness, agency, and predictability to emotion labels.For example, guilt and pride share high self-agency but differ in pleasantness.Source: §7 · p. 6, Figure 5 · p. 7
• Independent linear regression probes estimate each appraisal score from hidden activations using a separate appraisal representation for each concept.The estimated appraisal score is a linear function of an activation vector.Source: §7 · p. 6
• Appraisal signals are weakly detectable in earlier layers but strongly present as representations approach the final hidden states.The probing results are reported in Appendix E.Source: §7 · p. 6

• Emotion vectors show psychologically plausible cosine-similarity mappings with pleasantness and other-agency appraisal vectors across layers.Source: §8 · p. 7
• Projection strength peaks in early layers and fades to near zero in the final layer, suggesting final-layer orthogonality.Source: §8 · p. 7
• Appraisal-emotion structure begins decoupling around critical layers such as layers 9 and 10, then vanishes as processing progresses.Source: §8 · p. 7

• Appraisal modulation injects an appraisal’s unique latent effect vector into hidden representations, with β controlling promotion or demotion strength.Source: §9 · p. 7
• At layer 9, increasing pleasantness promotes joy and pride, aligning generated emotion distributions with appraisal-theory expectations.Source: §9 · p. 8
• Interventions in earlier layers are psychologically invalid, while later-layer interventions are ineffective because representations are overwritten or orthogonalized.Source: §9 · p. 8
• Promoting both other-agency and pleasantness produces pride without additional joy, supporting layer 9’s direct contribution to emotion-related cognitive processing.Source: §9 · p. 8

• Mid-layer MHSA units process emotional content, and appraisal-vector manipulation steers outputs in controlled and predictable ways.Source: §10 · p. 8
• The experiments use in-the-wild emotional examples and psychological theory rather than the simplified synthetic structures common in prior mechanistic studies.Source: §10 · p. 8
• Steering could condition personality traits or moods for tailored affective responses, but standard NLP benchmarks should assess whether such interventions disrupt other language functions.Source: §10 · p. 8
• The approach offers a way to causally intervene in emotional text generation, with potential relevance to safety and alignment in sensitive affective domains.Source: §10 · pp. 8–9

• The analysis builds on the linear representation hypothesis, although recent work suggests that not all features are encoded linearly.Source: §11 · p. 9
• Appraisal modulation changes affective outputs, but appraisals may be correlated with emotions rather than directly causing them, or the causal direction may be reversed.Source: §11 · p. 9
• Future studies must disentangle directional dependencies to establish causality between appraisal concepts and emotional outputs.Source: §11 · p. 9

12 Ethical Impact Statement

• The study cautions that its findings may not generalize to unexamined models, superficially similar tasks, or different languages, especially in high-stakes affective deployments.The analysis is intended to inform understanding of emotionally charged generation and related ethical concerns.Source: §12 · p. 9

A.1 Dataset Details

• The crowd-enVENT dataset contains emotion-evoking event vignettes, subjective appraisal ratings, and 13 emotion categories with unequal sample counts for guilt and shame.Examples are predominantly everyday English, and appraisals were rated on 5-point Likert scales.Source: §A.1 · p. 15

A.2 Architecture and Model Details

• The analysis examines ten LLMs from multiple model families and scales, using Llama 3.2 1B as the default lightweight model.Models were implemented with Hugging Face and TransformerLens to support hidden-state extraction and intervention.Source: §A.2 · p. 15

A.3 Task Details

• Approximately 40% or higher accuracy is achieved across architectures and scales when outputs are confined to the dataset’s emotion-label vocabulary.Open-vocabulary predictions rarely match human-reported labels, motivating the closed-vocabulary setup.Source: §A.3 · p. 16
• Because emotion attribution is subjective, the study analyzes only correctly classified examples, retaining at least 2,700 samples across model architectures.This filtering targets cases where the model and human annotator agree on the emotion label before mechanism analysis.Source: §A.3 · pp. 15–16
• Some emotion classes have limited representation after filtering, constraining the experiments and motivating preliminary bias analysis and further investigation.The authors also report low probe-training variance across seeds, with no more than 1.5% standard deviation in classification accuracy.Source: §A.3 · p. 17

B Knockout Experiment

• Knockout interventions across models show that mid-layer MHSA units, especially a(10), are more important for emotion-label generation than m(10), while later MHSA and FFN removal has little effect.Zero and random interventions measure post-intervention classification accuracy, with lower accuracy indicating greater causal importance; Figure 12 summarizes these comparisons across units and layers.Source: B Knockout Experiment · p. 17, Figure 12 · p. 19

C.1 Details on Probing and Causal-Intervention

• For Llama 3.2 1B, linear and nonlinear probes peak around layer 10, while zero/random knockouts and activation patching independently identify layer 10 as the critical emotion-processing location.Nonlinear probes reveal earlier signal than linear probes, but after layer 10 the output label is largely finalized; patching transfers are strongest there.Source: §C.1 · p. 20, Figure 14 · p. 20, Figure 15 · p. 20

C.2 Effect of Span Size on Activation Patching

• Activation patching becomes more effective as the intervened span increases from 1 to 5 layers, while layer 10 remains the peak-performing location.This span-size result reinforces functional localization of emotion processing in that layer.Source: §C.2 · p. 21, Figure 17 · p. 21

C.3 Investigating Token Dimension

• Across the last five token positions, signal strength increases toward later tokens, and interventions localize emotion processing to last-token MHSA units around l = 10 rather than query-final tokens.MHSA units beyond l = 10 contribute minimally to the causal path even when an entire layer is knocked out.Source: §C.3 · pp. 21–22, Figure 19 · p. 22, Figure 20 · p. 23

C.4 Control Experiment on an Isomorphic Task

• The isomorphic syntax control localizes patching effects mainly to final-layer MHSA units and provides weaker localization, contrasting with the mid-layer emotion findings.The control changes the task to predicting the first word, separating emotion-specific localization from syntactic task structure.Source: §C.4 · p. 22, Figure 23 · p. 24

C.5 Robustness to Prompt Design

• Probing and patching remain consistent across prompt templates and few-shot-example counts, with probing indicating that the model determines its label by the mid-layers.Interpretability analyses use only examples the model predicts correctly under each prompt.Source: §C.5 · p. 22, Figure 24 · p. 24, Figure 25 · p. 25

C.6 Investigating the Incorrectly Classified Samples

• Misclassified examples exhibit distinct appraisal profiles rather than random errors, suggesting meaningful shifts in how the model evaluates appraisal cues.The analysis is preliminary and limited by the study’s focus on correctly classified examples, which may bias interpretations toward unambiguous emotional content.Source: §C.6 · pp. 23–24, Table 3 · p. 24

D Direct Emotion Promotion; E Appraisal Probing; F Further Details on Appraisal Modulation

• Direct emotion promotion succeeds mainly after layer 9, while appraisal probing and modulation likewise show that emotion-related concepts become strongly accessible, causally steerable, and psychologically plausible around the mid-layers.Appraisal interventions shift output-label distributions more strongly with larger β, but theoretical and intuitive shifts emerge reliably mainly in mid-layers, peaking at layer 9; early interventions can be overwritten or produce unexpected effects.Source: D Direct Emotion Promotion · p. 24, E Appraisal Probing · p. 25, F Further Details on Appraisal… · pp. 25–26

F Further Details on Appraisal Modulation

• Middle-layer appraisal interventions in Llama 3.2 1B align with theoretical expectations, whereas later-layer shifts are less pronounced and final-layer interventions are ineffective.Increasing pleasantness promotes joy and pride; these effects are strongest in layers 9–11 and are consistent with the emotions’ associations with pleasantness.Source: F Further Details on Appraisal… · p. 26
• Superposed appraisal interventions produce expected emotion transitions, including pride without joy, guilt and fear without anger, and pride-to-surprise shifts with fear exceeding anger.These effects occur when manipulating other-agency, pleasantness, and unpredictability across middle layers.Source: F Further Details on Appraisal… · p. 26, Figure 33 · p. 29, Figure 34 · p. 30
• Random-vector controls produce no psychologically valid pattern, supporting a direct role for Llama 3.2 1B middle layers in emotion-related cognitive processing.The control experiment intervenes with randomly sampled vectors at different layers rather than appraisal vectors.Source: F Further Details on Appraisal… · p. 26, Figure 36 · p. 31

G Code and Compute Resources

• The experiments use NVIDIA A100 GPUs, require at least 40GB VRAM for larger models and 12GB for smaller models, and take about 24 hours for a full larger-model run.Generative AI tools were also used for writing-style improvement and code completion; intervention effects generally increase with scaling factor β, though early-layer shifts can violate theoretical expectations.Source: G Code and Compute Resources · p. 27, Figure 30 · p. 28, Figure 31 · p. 28