PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Emotion in the Deep: A Systematic Study of How LLMs Represent, Retain, and Express Emotion

Jingxiang Zhang, Lujia Zhong

arXiv:2510.04064v2cs.AI

TL;DR

Prior evaluations often treat LLMs as black boxes, leaving their internal emotional representations insufficiently examined. This study probes hidden states across layers and finds structured emotion signals that emerge early, often peak mid-network, and remain steerable and detectable after hundreds of tokens.

  • Problem

    Research has examined LLMs’ emotional capabilities, but many evaluations focus on outputs and treat models as black boxes rather than investigating their internal emotional representations.

    Source: Introduction · pp. 1–2

  • Method

    The study builds an approximately 400,000-utterance emotion-balanced corpus and uses lightweight probes on frozen Qwen3 and LLaMA hidden layers to read out emotion information.

    Source: Introduction · p. 2, Base Model and Probe Design · p. 4

  • Results

    Emotion structure emerges early, often peaks mid-network, and remains steerable and detectable after hundreds of tokens; final-layer probes also outperform zero-shot classification.

    Source: Results and Analysis · p. 4, Discussion · p. 8, Conclusion · p. 8

  • Takeaways & Limitations

    The released corpus, toolkit, and layer-wise map provide a practical foundation for future work on interpretability, safety, and alignment.

    Source: Conclusion · p. 8

    The study uses English-language Reddit-style text, a simplified Ekman taxonomy, and single-turn text-only prompts; cross-lingual and cross-cultural robustness remains untested.

    Source: Limitations and Future Work · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly expected to navigate the nuances of human emotion. While research confirms that LLMs can simulate emotional intelligence, their internal emotional mechanisms remain largely unexplored. This paper investigates the latent emotional representations within modern LLMs by asking: how, where, and for how long is emotion encoded in their neural architecture? To address this, we introduce a novel, large-scale Reddit corpus of approximately 400,000 utterances, balanced across seven basic emotions through a multi-stage process of classification, rewriting, and synthetic generation. Using this dataset, we employ lightweight "probes" to read out information from the hidden layers of various Qwen3 and LLaMA models without altering their parameters. Our findings reveal that LLMs develop a surprisingly well-defined internal geometry of emotion, which sharpens with model scale and significantly outperforms zero-shot prompting. We demonstrate that this emotional signal is not a final-layer phenomenon but emerges early and peaks mid-network. Furthermore, the internal states are both malleable (they can be influenced by simple system prompts) and persistent, as the initial emotional tone remains detectable for hundreds of subsequent tokens. We contribute our dataset, an open-source probing toolkit, and a detailed map of the emotional landscape within LLMs, offering crucial insights for developing more transparent and aligned AI systems. The code and dataset are open-sourced.

Introduction

Research on LLM emotion has emphasized outward capabilities, leaving internal representations comparatively underexplored. This study examines whether emotion forms a structured internal geometry and introduces a corpus and probing approach to investigate it.

Introduction

  • Prior work reports that LLMs simulate emotional expression rather than experience subjective feelings, while emotional prompts can influence performance.These findings motivate asking whether models also form structured internal representations of emotion.Source: Introduction · p. 1
  • Most evaluations focus on emotional outputs and treat models as black boxes, leaving the internal mechanisms that produce them underexamined.The paper frames coherent internal emotional geometry as important for more transparent and predictable AI systems.Source: Introduction · pp. 1–2
  • The study introduces an approximately 400,000-utterance, emotion-balanced corpus and probes frozen Qwen3 and LLaMA 3 hidden layers without changing model parameters.The corpus combines classified Reddit comments, rewritten neutral content, and synthetic examples across six Ekman emotions plus neutral.Source: Introduction · p. 2

Related Work

The related work spans neural representation probing, emotion signals across model depth, interpretability methods, and links between computational models and human-centered theories of emotion.

Related Work

  • The paper situates its contribution among work on internal representations, layer-wise emotion analysis, interpretability, and cognitive alignment.These are the four related-work strands highlighted in the section.Source: Related Work · p. 2

Understanding and Probing Neural Representations

  • Linear probes assess information in intermediate neural representations, while later work studies intrinsic dimensionality and representation compression.Intermediate layers are described as balancing task-relevant information with noise reduction.Source: Understanding and Probing Neural… · p. 2

Layer-Wise Analysis of Emotion in LLMs

  • Prior probing studies report that emotional signals are most distinct at middepth, with later layers adding little new emotional insight.The section describes later layers as refining outputs for specific tasks after intermediate layers construct semantic representations such as emotion.Source: Layer-Wise Analysis of Emotion in LLMs · p. 2

Interpretability and Explainability in Affective Computing

  • Activation patching has been used to transfer emotion between sentences by modifying mid-layer activations, connecting representations to model behavior.The section also describes attention and SHAP as transparency methods for identifying influential inputs or features.Source: Interpretability and Explainability in… · p. 2

Alignment with Cognitive and Neuroscience Theories

  • Related work connects emotion in LLMs to appraisal theory and reports alignments between model representations and human eye-tracking or fMRI activity.The paper positions its large-scale emotion analysis within this computational and human-centered research direction.Source: Alignment with Cognitive and… · p. 2

Method

The method constructs and filters an emotion-balanced Reddit corpus, then uses lightweight classifiers on frozen model hidden states to examine emotion representations across layers.

Dataset - Gathering and Cleaning

Corpus Construction

  • Qwen3-32B classified 300,000 sampled Reddit comments as one of six Ekman emotions or neutral; neutral utterances were assigned target emotions and rewritten, and approximately 60,000 prototypical utterances were synthetically generated.Source: Corpus Construction · p. 3

Filtering and Final Dataset

  • Filtering removed non-English text, utterances shorter than three words, and exact duplicates, yielding approximately 400,000 emotive utterances.Median length ranged from 12 words for surprise to 27 for sadness; rewritten examples tended to be longer than raw data.Source: Filtering and Final Dataset · p. 4

Model Probing and Evaluation

  • Lightweight classifiers attached to frozen model layers read out encoded emotion information and map its geometry across network depth.The approach leaves the pretrained model parameters unchanged.Source: Model Probing and Evaluation · p. 4

Base Model and Probe Design

  • At each selected layer, a two-layer ReLU MLP classifies the last non-padded token’s hidden state into seven emotion categories.The seven categories are the six Ekman emotions plus neutral.Source: Base Model and Probe Design · p. 4

Data Handling and Class Balancing

  • The corpus was split 90% for training and 10% for testing, with oversampling for training and undersampling for a balanced test set.Minority training classes were duplicated, while the majority neutral class was down-sampled.Source: Data Handling and Class Balancing · p. 4

Training and Evaluation Procedure

  • One-epoch probes treat LLM-generated annotations as reference labels; PCA projects probe probabilities to 2D for KDE visualization.The described optimizer uses a 10^-4 learning rate with linear warmup followed by cosine decay.Source: Training and Evaluation Procedure · p. 4

Emotion Classification from Final-Layer Representations

The section compares zero-shot emotion classification with probes trained on final-layer representations from pretrained and supervised fine-tuned models. SFT-Template probes perform best, with their advantage over zero-shot smaller at larger scale.

Emotion Classification from Final-Layer Representations

  • The comparison evaluates zero-shot accuracy and coverage alongside Pre-trained, SFT-Raw, and SFT-Template probes using final-layer representations.The SFT-Raw probe uses raw utterances; the SFT-Template probe uses the full chat template.Source: Emotion Classification from… · p. 4

Results and Analysis

  • +20.74 pp for Qwen3-0.6B (0.7170 vs. 0.5096), shrinking to +1.86 pp for Qwen3-8B (0.8058 vs. 0.7872), as the SFT-Template Probe beats zero-shot accuracy.Across models, Table 2 ranks SFT-Template above SFT-Raw, which in turn exceeds the Pre-trained Probe.Source: Results and Analysis · p. 4, Table 2 · p. 5
  • Qwen models have better zero-shot coverage than LLaMA, which the paper suggests may reflect using Qwen’s own labels during dataset filtering.Source: Results and Analysis · p. 5

Visualizing the Internal Geometry of Emotion

Final-layer probe outputs form structured emotion clusters whose separation increases with model scale and whose arrangement reflects semantic relationships among emotions.

Visualizing the Internal Geometry of Emotion

Analysis of Emotional Clusters

  • Larger 8B models show tighter, better-separated emotion clusters than Qwen3-0.6B, whose contours overlap broadly.Source: Analysis of Emotional Clusters · p. 5
  • Anger and disgust nearly overlap, while joy and surprise, anger and disgust, and fear and sadness form positive, negative, and downcast groups, respectively, around central neutral.Source: Analysis of Emotional Clusters · p. 5

Layer-Wise Emergence of Emotional Signal

Probes detect emotional information early in the network, with their strongest accuracy in middle-to-late layers rather than at the final layer.

Layer-Wise Emergence of Emotional Signal

Prompt-Based Emotion Generation

The study tests how system prompts shape emotion expressed in generated replies from Qwen3-4B and LLaMA 3.2-3B. A single-line prompt shifts that expression, with emotional prompts increasing sadness recall while reducing sadness precision.

Prompt-Based Emotion Generation

Temporal Persistence of Emotional Tone

Offset-aware probes track whether a reply retains traces of the user’s original emotion. Those traces decay at different rates by emotion, and the measure concerns detectable activations rather than subjective feeling.

Temporal Persistence of Emotional Tone

Analysis of Emotional Signal Decay

Discussion

The discussion describes emotion as a layered, steerable, and persistent representation in LLMs, while noting safety implications and limits on the study’s scope.

Discussion

  • Probe results show that emotion is classifiable from final-layer states, with semantically meaningful clusters that become more distinct as model scale increases.The study also reports that lightweight probes achieve high accuracy.Source: Discussion · p. 8
  • Emotional signals emerge early and often peak in middle layers, while an initial user emotion remains detectable across hundreds of generated tokens.Figure 7 plots smoothed probe accuracy at different decoding positions.Source: Discussion · p. 8, Figure 7 · p. 8

Implications for Alignment and Safety

  • A single-line system prompt can influence the model’s expressed emotional tone, while clear emotional separability may support post-hoc safety mechanisms and also enable misuse to manipulate users.The authors call for auditing and control of internal emotional representations in alignment research.Source: Implications for Alignment and Safety · p. 8, Discussion · p. 8

Limitations and Future Work

  • Findings are limited to English Reddit-style text, Ekman’s six basic emotions, possible Qwen classifier circularity, and single-turn, text-only prompts.Cross-lingual and cross-cultural robustness remains untested; real-world multi-turn, multimodal, and tool-use settings were not examined.Source: Limitations and Future Work · p. 8

Conclusion

The conclusion presents a released corpus and probing toolkit alongside a layer-wise account of emotional representations, which the authors position as a foundation for interpretability, safety, and alignment work.

Conclusion

  • The authors release a 400k-utterance emotion-balanced corpus, an open-source probing toolkit, and a large-scale layer-wise study of how LLMs encode emotion.The paper describes the study as the first large-scale layer-wise investigation of this topic.Source: Conclusion · p. 8
  • Emotion-related structure is present early, peaks before the final layer, and remains steerable and detectable after hundreds of tokens.Source: Conclusion · p. 8
  • The authors describe these results as a practical foundation for future work on model interpretability, safety, and alignment.Source: Conclusion · p. 8
Loading 2510.04064v2…