PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs "Feel"? Emotion Circuits Discovery and Control

Chenxi Wang, Yixuan Zhang, Ruiji Yu, Yufei Zheng, Lang Gao, Zirui Song, Zixiang Xu, Gus Xia, Huishuai Zhang, Dongyan Zhao, Xiuying Chen

arXiv:2510.11328v1cs.CLcs.AI

TL;DR

The paper addresses the gap between detecting emotion-related signals in LLMs and understanding the mechanisms that generate emotional expression. It extracts emotion directions, identifies and integrates local components into circuits, and reports 99.65% test-set emotion-expression accuracy with circuit modulation, outperforming prompting- and steering-based methods.

  • Problem

    Prior studies identify emotion-related features, but the mechanisms behind emotional expression remain poorly understood, while existing control methods leave those mechanisms unexplained.

    Source: §1 · p. 1

  • Method

    The paper extracts context-agnostic emotion directions, identifies neurons and attention heads through analytical and causal analysis, and integrates them into emotion circuits.

    Source: §1 · p. 2

  • Results

    99.65% overall emotion-expression accuracy is achieved on the test set, surpassing prompting- and steering-based baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that emotional expression in large models arises from distributed internal computation that can be traced and modulated.

    Source: §8 · p. 8

    The study is limited to English inputs and six basic emotions, and circuit stability under fine-tuning or transfer learning remains unexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As the demand for emotional intelligence in large language models (LLMs) grows, a key challenge lies in understanding the internal mechanisms that give rise to emotional expression and in controlling emotions in generated text. This study addresses three core questions: (1) Do LLMs contain context-agnostic mechanisms shaping emotional expression? (2) What form do these mechanisms take? (3) Can they be harnessed for universal emotion control? We first construct a controlled dataset, SEV (Scenario-Event with Valence), to elicit comparable internal states across emotions. Subsequently, we extract context-agnostic emotion directions that reveal consistent, cross-context encoding of emotion (Q1). We identify neurons and attention heads that locally implement emotional computation through analytical decomposition and causal analysis, and validate their causal roles via ablation and enhancement interventions. Next, we quantify each sublayer's causal influence on the model's final emotion representation and integrate the identified local components into coherent global emotion circuits that drive emotional expression (Q2). Directly modulating these circuits achieves 99.65% emotion-expression accuracy on the test set, surpassing prompting- and steering-based methods (Q3). To our knowledge, this is the first systematic study to uncover and validate emotion circuits in LLMs, offering new insights into interpretability and controllable emotional intelligence.

1 Introduction

The paper asks how LLMs generate emotional expression and whether its internal mechanisms can support emotion control. It proposes an interpretability-driven framework and reports 99.65% test-set expression accuracy with circuit modulation, surpassing prompting- and steering-based baselines.

1 Introduction

  • Prior work found emotion-related activation features but did not explain the mechanisms behind emotional expression; existing steering and prompting methods manipulate surface signals without explaining those mechanisms.The paper frames this gap as leaving interventions black-box and unreliable.Source: §1 · p. 1
  • The framework combines context-agnostic emotion-direction extraction, local neuron and attention-head identification, and global circuit integration.It uses controlled contexts, analytical decomposition, causal interventions, and sublayer influence to assemble circuits.Source: §1 · pp. 1–2
  • 99.65% overall emotion-expression accuracy on the test set, surpassing prompting- and steering-based baselines, is achieved without explicit prompting or instruction.Circuit modulation drives hidden states into distinct emotion clusters across layers and yields coherent, natural emotional expressions.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors characterize emotion in LLMs as structured, stable internal mechanisms rather than mere surface reflections of training data.They present traceable emotion circuits as mechanistic evidence for emotional generation.Source: §1 · p. 2

2 Related Research

Prior research has identified emotion-related representations and developed emotion-control methods, but the paper distinguishes recognizing emotion from explaining how LLMs generate emotional expression. It positions its contribution as uncovering circuit mechanisms behind that generation.

Emotion-related mechanisms in LLMs

  • Prior work found emotion-related signals in LLMs, but did not construct the circuit mechanisms that drive emotional expression.Broekens et al. (2023) and Yongsatianchot et al. (2023) linked LLMs to psychological emotion dimensions; Li et al. (2024) and Tigges et al. (2024) studied causal or linear emotion representations; Tak et al. (2025) and Lee et al. (2025) examined layer-wise repair and emotion neurons.Source: §2 · pp. 2–3
  • The paper distinguishes prior work on recognizing emotions in text from its focus on the internal processes that generate emotional expression.The cited recognition studies probe emotion recognition rather than generation mechanisms.Source: §2 · p. 3

Methods for emotion control

  • Most emotion-control methods demonstrate technical feasibility but remain black-box and do not uncover the internal mechanisms of emotion generation.The section discusses dialogue frameworks, style vectors, latent-space manipulation, and more recent emotion-control approaches.Source: §2 · p. 3

3 Background

The background introduces Transformer forward computation to support later experiments. It describes the residual stream as the space for tracing emotion representations through attention and MLP sublayers.

3 Background

  • The background outlines Transformer forward computation to provide context for the later experiments.The residual stream carries information across layers and is used to analyze how emotion representations propagate.Source: §3 · p. 3, §3.1 · p. 3

3.1 Transformer Architecture

  • The study measures residual-stream activations after each attention or MLP sublayer output is added back, treating the residual stream as the primary analysis space.Each Transformer layer has a multi-head self-attention sublayer and a feed-forward MLP sublayer, each followed by a residual connection.Source: §3.1 · p. 3
  • Attention heads let tokens attend to contextual information from previous tokens, while the MLP uses up-projection, gating, and down-projection operations.Gated activations serve as inputs to the down-projection and targets for neuron-level ablation and enhancement analyses.Source: §3.1 · pp. 3–4

4 SEV Dataset

The paper constructs SEV as a controlled dataset for probing emotion under matched everyday contexts, then uses an independent test set to assess generalization beyond the discovery data. Main analyses use LLaMA-3.2-3B-Instruct, with the framework also reproduced on Qwen2.5-7B-Instruct.

Dataset Construction

  • Emotionally explicit words are prohibited so that emotional variation comes from event semantics rather than lexical emotion cues.The dataset is designed to keep inputs domain-neutral and universal.Source: §4 · p. 4
  • SEV contains 480 event descriptions across eight everyday domains, with 20 scenarios per domain and three outcome variants per scenario.Each record pairs a neutral scenario with positive, neutral, and negative outcomes that share participants, time, and context but differ in result valence.Source: §4 · p. 4

Test Set

  • An independent test set of the same size is constructed to test whether discovered mechanisms generalize beyond the training context.The paper describes this as out-of-domain validation on unseen, neutral input text.Source: §4 · p. 4

Model Selection

  • Main analyses use LLaMA-3.2-3B-Instruct, and the framework is also reproduced on Qwen2.5-7B-Instruct to check robustness and generality.LLaMA-3.2-3B-Instruct was selected for its transparent architecture, moderate scale, and documented open-source implementation.Source: §4 · p. 4

5 Extracting Context-Agnostic Emotion Directions

The study extracts emotion directions by removing shared scenario–event semantics from activations, then validates them through emotion steering. The resulting vectors separate emotions across layers, and steering induces intended emotions on held-out examples.

Emotion Elicitation via Prompting

  • 98.85% emotion expression accuracy on SEV and 98.96% on the held-out test set were achieved with prompting-based elicitation.Successful generations supplied the activations used to derive emotion directions.Source: §5 · p. 4
  • Distinct emotion clusters emerge from prompting-based hidden states from layer 9, with anger and disgust, and sadness and fear, appearing close by layer 12.The representation clusters remain stable in subsequent layers.Source: §5 · p. 4

Context-Agnostic Emotion Vectors Extraction

  • Emotion directions are extracted by subtracting each scenario–event group’s mean activation across six emotions, then centering and normalizing per-emotion means.This formulation treats the group mean as a neutral baseline to cancel shared semantics; the vectors show separated clusters as early as layer 0, with linear-probe F1 = 1.0.Source: §5 · p. 5

Emotion Steering Validation

  • 91.22% success rate was achieved by steering with the extracted emotion vectors on the held-out test set.The steering scaling by local RMS maintains activation magnitude while enforcing the target emotional direction.Source: §5 · p. 5

6 Local Mechanisms of Emotion Representation

The section identifies emotion-relevant neurons and attention heads, then tests their roles by ablation, enhancement, and random-intervention controls. Targeted components show concentrated effects, while random interventions have negligible impact.

6 Local Mechanisms of Emotion Representation

  • Neuron contributions are ranked by their mean contribution to the layerwise emotion direction, while attention heads are identified by causal ablation and change in emotion projection score.The selected top-k components are used in subsequent ablation and enhancement experiments.Source: §6 · p. 5, §6.1 · p. 6, §6.2 · p. 6

6.3 Causal Validation via Ablation

  • Ablating only 2 or 4 top-ranked neurons sharply decreases the emotion score, after which the effect plateaus as k increases by over 30×.The result indicates a long-tail effect in which only a few neurons dominate emotion expression.Source: §6.3 · p. 6
  • Attention-head ablation produces a steep early decline in emotion score followed by saturation, indicating that only a handful of heads decisively shape emotion representation.The experiment removes the top-k heads per layer and measures the change in emotion score.Source: §6.3 · p. 6

6.4 Causal Validation via Enhancement

  • Enhancing top-ranked neurons causes emotion scores to rise sharply for small k and then saturate, showing that a small subset can evoke emotional representations.The prompts omit emotion instructions, so the measured response follows internal modulation rather than prompting.Source: §6.4 · p. 7
  • Ablating identified neurons or attention heads sharply reduces emotion scores, while enhancement shows that stimulating selected components can elicit emotion without explicit instructions.Targeted enhancement effects rise sharply for small k and quickly saturate; only a few neurons or heads can produce strong responses.Source: §6.3 · p. 6, §6.4 · p. 7

6.5 Control Experiments with Random Interventions

  • Random interventions have negligible impact, while targeted interventions remain clearly separated from random ones across k.Random sets are sampled uniformly and results are averaged over 10 random seeds.Source: §6.5 · p. 7

7 Beyond Locality: Global Emotion Circuits

The section assembles local emotion mechanisms into global circuits by measuring sublayer influence and combining components across layers. Modulating these circuits controls emotional expression, with high test-set accuracy and performance above prompting- and steering-based approaches.

7 Beyond Locality: Global Emotion Circuits

  • The section first quantifies each sublayer’s causal influence on the global emotional state, then integrates local mechanisms into circuits for expression control.Source: §7 · p. 7, §7.1 · p. 7

7.1 Layerwise Importance of Emotion Generation

  • Sublayer importance estimates how a perturbation along a local emotion direction shifts the final state toward a stable emotion reference, with rankings consistent across α values.The reference direction is formed by sign-aligning and averaging attention and MLP directions from layers 21–25.Source: §7.1 · pp. 7–8

7.2 Global Emotion Circuit Assembly

  • Global circuits combine local component scores with sublayer importance, giving each sublayer a minimum quota before allocating the remaining budget proportionally to importance.This balances distributed expression with greater allocation to sublayers that more strongly influence the final emotion basis.Source: §7.2 · p. 8
  • Across emotions, neuron overlap is 0.056±0.033 and attention-head overlap is 0.454±0.047, indicating emotion-specific MLP subcircuits alongside shared attention pathways.Source: §7.2 · p. 8

7.3 Controlling Emotion Expression via Circuit Modulation

  • 99.65% overall emotion-expression accuracy on the test set, outperforming prompting- and steering-based approaches, is achieved by circuit modulation.Generated text also exhibits natural affective tone without explicit prompting.Source: §7.3 · p. 8

8 Conclusion

The conclusion presents emotional expression as coordinated computation across neurons and attention heads, rather than a surface artifact. Tracing, assembling, and modulating these circuits supports precise emotional control and motivates extending circuit-level interpretability to other domains.

8 Conclusion

  • Emotion in LLMs arises from coordinated activations of neurons and attention heads organized as interpretable emotion circuits.Source: §8 · p. 8
  • Tracing, assembling, and modulating the circuits enables precise and natural emotional control that outperforms prompting- and steering-based baselines in accuracy and expressiveness.Source: §8 · p. 8
  • The framework is presented as a foundation for extending circuit-level interpretability and controllable generation to broader cognitive and stylistic domains.Source: §8 · p. 9

Limitations

The paper identifies boundaries in language, emotion coverage, and circuit stability, and its appendices describe dataset-generation prompts, annotation, evaluation settings, visualizations, and prompting templates.

Limitations

  • The analysis is limited to English and Ekman’s six basic emotions, while circuit stability under fine-tuning or transfer learning remains unexplored.Source: Limitations · p. 9

A Date Generation Prompt

  • The dataset-generation prompt covers eight everyday themes and requests 20 neutral first-person English scenarios, then three similarly structured valence variants without explicit emotion words.Source: A Date Generation Prompt · p. 10

B Annotation Prompt

  • The annotation prompt asks raters to judge whether a text’s style matches the target emotion, focusing on tone or attitude rather than content valence.Source: B Annotation Prompt · pp. 10–11

C Emotion Expression Accuracy of Prompting and Steering

D Emotion Expression Accuracy under Circuit Modulation

E Layer-wise Visualization of Emotion Representations

F The prompts for Prompting-based generation

G Layerwise Directional Consistency of Emotion Subspaces

Emotion directions become highly consistent across deeper sublayers, supporting a reference basis from layers 21–25. The appendix contrasts prompting and steering accuracy and identifies visualizations of hidden-state clusters.

G Layerwise Directional Consistency of Emotion Subspaces

H.1 Emotion Expression Accuracy of Prompting and Steering

  • 84-96% prompting accuracy across emotions contrasts with steering above 92% for happiness and surprise but below 5% for anger, sadness, fear, and disgust.The authors suggest safety alignments may underlie steering’s resistance to negative emotion.Source: §H.1 · p. 11

H.2 Layer-wise Visualization of Emotion Representations

  • The appendix presents layer-wise hidden-state clustering for samples successfully guided by prompting on Qwen2.5-7B-Instruct and clustering of pure emotion vectors.Source: §H.2 · p. 11

H.3 Results of MLP Neurons Intervention Experiments

Selected neuron and attention-head interventions alter emotion scores, while circuit-based steering at scale 2.0 surpasses steering-based elicitation across valence contexts.

H.3 Results of MLP Neurons Intervention Experiments

  • Figure 6 displays cosine-similarity heatmaps of emotion directions across all 56 residual positions for six emotions, showing stable similarity above 0.90 in deeper layers 21–25.Source: Figure 6 · p. 14
  • On Qwen2.5-7B-Instruct, top-ranked neuron interventions sharply change emotion scores while random interventions have negligible effects, matching LLaMA’s long-tail pattern.The results support the conclusion that only a few localized units dominate emotion expression.Source: §H.3 · p. 17, Table 4 · p. 18, Table 5 · p. 18

H.4 Results of Attention Head Intervention Experiments

  • Positive ∆s under top-k attention-head enhancement and negative ∆s under masking jointly support the identified heads’ causal importance for emotion expression.The experiments are summarized in Tables 6 and 7.Source: §H.4 · p. 17, Table 6 · p. 18, Table 7 · p. 18

H.5 Emotion Expression Accuracy with Scale 2.0 Steering

  • With scale 2.0, circuit-based steering on Qwen2.5-7B-Instruct surpasses steering-based elicitation across positive, neutral, and negative valence contexts.Per-valence accuracy for all six emotions is summarized in Table 8.Source: §H.5 · p. 17, Table 8 · p. 19

Related papers

Loading 2510.11328v1…