PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear Representations of Sentiment in Large Language Models

Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, Neel Nanda

arXiv:2310.15154v1cs.LGcs.AIcs.CL

TL;DR

The paper asks how sentiment is represented inside LLMs. It combines linear-direction discovery with causal interventions and circuit analysis, finding a causally relevant sentiment direction and a summarization motif that stores sentiment at intermediate positions. The authors also identify scope limitations concerning universality and incomplete circuit explanations.

  • Problem

    The paper addresses the open question of how sentiment, a variable relevant across language tasks, is represented within LLMs.

  • Method

    The study finds sentiment directions with toy data, evaluates them across natural-language datasets, and uses causal interventions, activation patching, and ablations to analyze their mechanisms.

  • Results

    Sentiment is represented linearly and causally, while a summarization motif stores sentiment at intermediate tokens and contributes to sentiment processing.

  • Takeaways & Limitations

    The findings provide an interpretable case study of sentiment circuitry and suggest that transformers form sentence- or clause-level sentiment summaries at information bottlenecks.

  • Takeaways & Limitations

    The authors question whether the direction is universal or instead a principal component of more specific sentiment features, and note that their abstractions do not explain all task performance.

Abstract

from arXiv · show

Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs). In this study, we reveal that across a range of models, sentiment is represented linearly: a single direction in activation space mostly captures the feature across a range of tasks with one extreme for positive and the other for negative. Through causal interventions, we isolate this direction and show it is causally relevant in both toy tasks and real world datasets such as Stanford Sentiment Treebank. Through this case study we model a thorough investigation of what a single direction means on a broad data distribution. We further uncover the mechanisms that involve this direction, highlighting the roles of a small subset of attention heads and neurons. Finally, we discover a phenomenon which we term the summarization motif: sentiment is not solely represented on emotionally charged words, but is additionally summarized at intermediate positions without inherent sentiment, such as punctuation and names. We show that in Stanford Sentiment Treebank zero-shot classification, 76% of above-chance classification accuracy is lost when ablating the sentiment direction, nearly half of which (36%) is due to ablating the summarized sentiment direction exclusively at comma positions.

1 INTRODUCTION

The paper investigates how LLMs represent sentiment and finds a causally significant linear direction, alongside a summarization motif that stores sentiment at intermediate, non-valenced tokens.

  • 1 INTRODUCTION: The study examines sentiment representations in LLMs using causal mediation analysis.It treats sentiment as a variable in the data generation process and investigates its internal representation.
  • 1 INTRODUCTION: A single sentiment direction is presented as evidence that models represent extracted input properties as directions in activation space.The authors argue that such linear representations are especially amenable to reverse-engineering.
  • 1 INTRODUCTION: The summarization motif aggregates sentiment on intermediate tokens such as commas, periods, and particular nouns rather than only on valenced tokens.The authors distinguish this usage of “summarization” from standard NLP summarization tasks.
  • 1 INTRODUCTION: Sentiment stored on summarization tokens is causally relevant for final prediction and can function as an information bottleneck.The paper compares this structure to an explicit classification token and describes sentence- or clause-level summaries.
  • 1 INTRODUCTION: The paper combines toy-data direction finding, natural-language correlation, causal interventions, activation patching, and ablations to study sentiment circuitry.These methods are used to investigate what a single interpretable direction means across the full data distribution.

2 METHODS

The methods build sentiment directions from toy datasets, evaluate them across models and natural text, and use patching and ablation to test causal importance.

  • 2.1 DATASETS AND MODELS: ToyMovieReview uses positive or negative adjective–verb prompts with held-out adjective combinations and sentiment-specific expected completions.The dataset samples from 85 adjectives, split 55/30 for train/test, and 8 verbs.
  • 2.1 DATASETS AND MODELS: ToyMoodStory uses multi-subject prompts and evaluates model output with the logit difference between “excited” and “nervous.”The supplied passages identify the prompt structure and evaluation metric but do not provide the full template.
  • 2.1 DATASETS AND MODELS: The evaluation uses SST, OpenWebText, GPT-2, and Pythia models spanning 85M to 2.8b parameters.SST contains 10,662 one-sentence movie reviews with phrase-level human sentiment labels; OpenWebText supplies random text for correlational evaluation.
  • 2.3 CAUSAL INTERVENTIONS: Figure 2 compares cosine similarity among directions learned from ToyMovieReview adjective representations, while Figure 3 evaluates activation-based sentiment classification.Figure 3 uses the top/bottom 0.1% of GPT2-small first-layer activations and GPT-4 classification; ablations remove components or selected directions.
  • 2.2 FINDING DIRECTIONS: Five methods derive sentiment directions, including PCA, mean difference, K-means, linear probing, and Distributed Alignment Search.Linear probing uses normalized logistic-regression weights, while DAS learns a direction through a logit-difference objective.
  • 2.3 CAUSAL INTERVENTIONS: Activation patching swaps activations between matched prompts with flipped sentiment to estimate the importance of model components.Directional activation patching restricts the intervention to a single vector-space direction; evaluation uses logit difference and logit-flip accuracy.

3 FINDING AND EVALUATING A ‘SENTIMENT DIRECTION’

The paper finds that sentiment is captured by a largely shared direction in activation space whose projections track positive and negative language, generalize beyond toy data, and support causal interventions. Evaluations further show that this direction changes with negation and is most useful at intermediate layers.

  • Finding a sentiment direction: Different methods recover highly similar sentiment directions, suggesting noisy approximations of one singular direction.The directions are learned from the ToyMovieReview dataset and exhibit extremely high cosine similarity.
  • Correlational evaluation: Sentiment activations separate negative and positive tokens at opposite extremes, while neutral tokens occupy the middle.This bipolar pattern was observed across sampled OpenWebText tokens classified by GPT-4.
  • Correlational evaluation: Negated words such as ‘fail’, ‘doubt’ and ‘uncertain’ can flip from negative in early layers to positive after further processing.The authors use the layerwise sentiment activation as a view of how the model updates sentiment during the forward pass.
  • Causal evaluation: Directional patching shifts logit differences on out-of-sample adjectives, and DAS can completely flip the prediction.The direction was not trained on any verbs, yet unsupervised methods still changed the output-related logit differences.
  • Causal evaluation: Toy-trained sentiment directions generalize most strongly to Stanford Sentiment Treebank at intermediate model layers.The paper interprets this pattern as evidence that sentiment knowledge is most prominent where the residual stream forms abstract concepts.
  • Causal evaluation: The sentiment circuit combines direct use of sentiment-laden words with a summarization motif at an intermediate position.In the ToyMovieReview circuit, heads 7.1 and 7.5 write to the summary position, whose information contributes causally to summary readers at END.

4 THE SUMMARIZATION MOTIF FOR SENTIMENT

The paper finds that sentiment processing uses intermediate summarization tokens and interpretable circuits, rather than transferring information only directly from valenced words to the final output. Ablation and patching experiments show these summaries are causally important, including on punctuation in multi-subject tasks and SST.

  • 4.1 CIRCUIT ANALYSES: The ToyMovieReview circuit identifies valenced words, writes sentiment to the final movie token as SUM, then reads ADJ, VRB, and SUM at END.The circuit comprises 9 attention heads, including sentiment summarizers that write to and read from SUM.
  • 4.1 CIRCUIT ANALYSES: 54.8% of the logit-difference drop and 58.3% of logit flips result when the circuit is patched along the sentiment direction at ADJ and VRB.Patching along all directions produces 97% logit flips and a 75% logit-difference drop, indicating that the sentiment direction accounts for most circuit function.
  • 4.1 CIRCUIT ANALYSES: In Pythia-2.8b mood stories, attention heads use preference-phrase commas and other tokens to write summaries to repeated names and “feels” tokens.Path patching indicates that some heads partially rely on the comma token, while later heads read the repeated name and feels tokens for outputs important to END.
  • 4.2 EXPLORING AND VALIDATING SUMMARIZATION BEHAVIOR IN PUNCTUATION: Reliance on period summaries increases as irrelevant text separates preference phrases from the final prompt region.The period-to-pre-period logit-difference-change ratio rises from 0.29 with 0 irrelevant tokens to 1.15 with 22 tokens.
  • 4.3 VALIDATING SUMMARIZATION BEHAVIOR IN SST: 18% drops in both logit difference and zero-shot accuracy follow directional ablation at every comma, while mean ablation produces a 19% accuracy drop.The comma intervention indicates that nearly half of sentiment-direction-mediated task ability is mediated through comma information.
  • 4.2 EXPLORING AND VALIDATING SUMMARIZATION BEHAVIOR IN PUNCTUATION: The summarization motif aggregates sentiment at non-valenced tokens such as commas, periods, and particular nouns, creating partial information bottlenecks that matter causally.The authors describe this as sentiment information being stored at intermediate positions before contributing to the final prediction.

5 RELATED WORK

The related work situates this study within mechanistic interpretability, internal-representation research, summarization motifs, and causal intervention methods for language models.

  • Sentiment Analysis: Sentiment analysis remains a widely used NLP task and benchmark following its early transformation by deep learning.The paper references surveys and benchmark work spanning sentiment-analysis research.
  • Understanding Internal Representations: The paper draws on mechanistic interpretability’s goal of reverse-engineering learned algorithms inside models.It connects this agenda to research on representations and world-modeling behavior in transformers.
  • Summarization Motif: The summarization-motif analysis follows research on information bottlenecks and abstraction levels in model representations.The paper also relates its use of “motif” to systems biology and prior observations of information storage after relevant tokens.
  • Causal Interventions in Language Models: The experiments use causal mediation analysis and circuits-analysis tools including mean ablation and activation patching.These methods are presented as approaches for identifying computational subgraphs that use feature representations.

6 CONCLUSION

The paper identifies linear sentiment representations and summarization as its two central findings, while outlining unresolved questions about universality, circuitry, and broader applicability.

  • The paper’s two central findings are a linear sentiment representation and summarization that stores sentiment information.
  • The proposed sentiment direction generalizes from a tiny toy dataset to diverse natural-language datasets, especially in larger models with longer contexts.
  • The study presents interpretable-feature analysis as an efficient alternative to enumerating many features through dictionary learning.
  • The authors question whether the identified direction is universal or instead the first principal component across task-specific sentiment directions.
  • The authors note that causal abstractions do not explain 100% of sentiment-task performance, potentially because of distributed representations, superposition, or self-repair.
  • Small datasets and many hyperparameters create a risk of gaming measures, although larger SST results and consistency across models increase confidence.
  • DAS outperformed on most metrics but may overfit a dataset and take activations out of distribution.
  • Future work should examine summarization across broader contexts and determine which model or task factors influence its use.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility by documenting its datasets, models, training procedures, algorithms, and analysis techniques.

  • The authors provide detailed descriptions of the datasets, models, training procedures, algorithms, and analysis techniques used.

A.1.1 CLUSTERING

Clustering analyses test whether sentiment-related activations form separable structures that support out-of-sample classification.

  • The authors first test whether the residual stream is clustered, a basic requirement for a sentiment direction.
  • PCA on 30 positive and 30 negative adjectives produces well-separated clusters, with unseen adjectives and verbs fitting their appropriate class.
  • 2-means classification remains generally strong on held-out adjectives and out-of-distribution verb tokens across models.

A.1.2 ACTIVATION ADDITION

Activation addition tests whether manipulating the learned sentiment direction can steer generated text toward different sentiment outcomes.

  • Activation addition on GPT2-small adds multiples of the sentiment direction to the first-layer residual stream to steer a positive review continuation.
  • At steering coefficient -17, outputs become extremely negative, with intermediate coefficients producing a gradual transition.

A.1.3 MULTI-LINGUAL SENTIMENT

The paper traces sentiment processing from a shared residual-stream direction to circuits of attention heads and neurons, including mechanisms that summarize sentiment at commas and other intermediate positions. Cross-lingual evidence is limited, while circuit analyses identify partially distinct phrase-processing and summary-reading pathways.

  • Multilingual evidence: French results showed hints of cross-lingual generalization only in the smallest model exhibiting such evidence, with early-layer representations obscured by poor tokenization.The models were not very good at French, and the representation was not evident in the first couple of layers.
  • Circuit analysis: Four primary GPT-2 Small heads—10.4, 9.2, 10.1, and 8.5—contribute at least 5% damage to the logit difference in the movie-sentiment circuit.Heads 10.4, 9.2, and 10.1 attend mainly to adjective and verb positions, while 8.5 attends primarily to the second “movie” token.
  • Circuit analysis: 7.1 and 7.5 causally transmit adjective and verb information through receivers 9.2 and 8.5, shifting the logit difference by 7% and 4%, respectively.The corresponding causal effect was not observed for other token positions.
  • Summarization circuitry: Comma representations form a designated COMMASUM position, and many heads both read summaries there and write summaries from names and feeling-related positions.Later heads can read summaries produced by earlier heads while writing their own summary information.
  • Summarization circuitry: Distinct attention-head groups respond disproportionately to phrase patching or comma patching, suggesting semi-separate summary-reading circuitry.One head showed a strong response in either patching case, whereas other groups were situation-specific.
  • Neuron mechanisms: Neuron out-directions aligned with the sentiment direction have interpretable tails, including neurons responding to negation, contrast markers, and negative words after contractions.Examples include “not hesitate,” “however” after something negative, and “but” following “not bad.”
Loading 2310.15154v1…