Source-linked AI summary
Linear representations in language models can change dramatically over a conversation
Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini, Sangnie Bhardwaj, Murray Shanahan
TL;DR
The paper asks whether linear directions for high-level concepts retain stable meanings as language models adapt to conversational context. It identifies such directions and tracks them across turns, finding that conversation-relevant representations can reverse while generic representations are generally preserved. These dynamics challenge static interpretability and steering approaches while motivating context-sensitive analyses.
Problem
The paper investigates whether linear representations of high-level concepts remain stable as language models adapt to conversational context.
Method
The authors identify concept directions with balanced yes/no questions and logistic regressions, then extract layer-wise answer-token representations across conversation turns and prompt contexts.
Results
Conversation-relevant representations can flip from factual to non-factual or ethical to unethical, and vice versa, across turns, including under robust regressions and across layers.
Takeaways & Limitations
Representation dynamics challenge static feature interpretations, probes, and steering methods that assume directions retain fixed meanings throughout a conversation.
Takeaways & Limitations
The study evaluates relatively few conversations and a narrow subset of concepts, and cannot establish the causal role of the measured representations.
Abstract
from arXiv · showhide
Language model representations often contain linear directions that correspond to high-level concepts. Here, we study the dynamics of these representations: how representations evolve along these dimensions within the context of (simulated) conversations. We find that linear representations can change dramatically over a conversation; for example, information that is represented as factual at the beginning of a conversation can be represented as non-factual at the end and vice versa. These changes are content-dependent; while representations of conversation-relevant information may change, generic information is generally preserved. These changes are robust even for dimensions that disentangle factuality from more superficial response patterns, and occur across different model families and layers of the model. These representation changes do not require on-policy conversations; even replaying a conversation script written by an entirely different model can produce similar changes. However, adaptation is much weaker from simply having a sci-fi story in context that is framed more explicitly as such. We also show that steering along a representational direction can have dramatically different effects at different points in a conversation. These results are consistent with the idea that representations may evolve in response to the model playing a particular role that is cued by a conversation. Our findings may pose challenges for interpretability and steering -- in particular, they imply that it may be misleading to use static interpretations of features or directions, or probes that assume a particular range of features consistently corresponds to a particular ground-truth value. However, these types of representational dynamics also point to exciting new research directions for understanding how models adapt to context.
1. Background
Linear representations in language models have been studied as structured directions for high-level concepts, but their meanings may depend on context and evolve during in-context adaptation. This work examines how conversational context changes these representations and what that implies for interpretability.
- Large language models can encode high-level concepts such as factuality or honesty along linear representation dimensions.
- Existing interpretability work has raised concerns that linear representations may become unfaithful under distribution shift, including shifts caused by accumulating conversational context.
- Prior work links linear representations to in-context learning of tasks, functions, structures, and personas.
- The authors frame their work as evidence that interpretability methods should account for representation dynamics over context.
2. Methods
The experiments identify linear dimensions for concepts such as factuality, then track how question-answer representations change across conversation turns and prompt settings. Balanced yes/no datasets, logistic regression, and layer-wise activation extraction distinguish generic from context-relevant effects.
- The main experiments use Gemma 3 models, especially the largest 27B-IT model, with broader model sizes and families examined in appendices.
- The study uses conversations, stories, and other prompts sourced from prior research, written by hand, generated through model interaction, or drafted by another language model.
- A conversation turn consists of a user query together with a model response; a single message is called a ply when needed.
- Balanced yes/no datasets contain generic questions expected to be context-independent and context-relevant questions tailored to each conversation or story.
- The researchers evaluate both possible answers in empty and extended contexts, extracting residual-stream representations while the model processes the answering Yes or No token at every layer.
- Regularized logistic regressions trained on 90% of balanced generic question-answer representations identify directions that predict factuality, with held-out data used for evaluation.
- The analysis uses generic-question prompts, an opposite-day prompt, and target prompts to separate conceptual representation from superficial behavioral changes.
- The margin score measures how strongly a representation direction separates positive from negative answers, becoming negative when classifications reverse relative to ground truth.The score is based on classifier logit differences for paired answers.
3. Experiments
The experiments show that linear representations of factuality and ethics can shift or invert during contextualized conversations, especially for conversation-relevant content. These dynamics persist across prompting settings, layers, and some model sizes, while explicit fictional stories produce weaker adaptation.
- Opposite-day representations: Factuality and ethics representations can invert after a few opposite-day conversation turns.Initially separated factual and non-factual answers become reversed along the identified dimensions; ethics shows a similar flip.
- Interpretation: The experiments expose construct-validity risks for interpreting linear directions as stable factuality or ethics features.The initial opposite-day result may reflect representations of behavior the model considers correct rather than factuality or ethics themselves, motivating robust regressions.
- Conversation-dependent changes: Robust factuality dimensions still invert for conversation-relevant questions while generic-question representations remain roughly preserved.The effect occurs despite regressions designed to separate factuality from superficial response patterns.
- Prompting settings: Replayed off-policy and on-policy conversations produce qualitatively similar changes in context-relevant representations.The findings therefore do not require conversations generated by the evaluated model.
- Role-play dynamics: Representations oscillate as a model alternates between opposing roles in a consciousness debate and can shift back after a corrective message.These results are consistent with adaptation to the role cued by the conversation.
4. Discussion
The discussion frames contextual representation changes as both evidence about model adaptation and a challenge for interpretability and safety. It also identifies limitations in experimental scope and opportunities for studying adaptation more broadly.
- Representations of factuality and ethics can flip during conversations, with changes observed across model families and appearing more dramatic in larger models.
- Similar effects in on-policy and off-policy settings are consistent with models adapting representations to the role they are playing.
- Context-dependent representation changes may connect beneficial in-context learning with harmful behavior in long contexts.
- Static interpretations can misclassify information across contexts, challenging probing, control, and representational lie detectors.
- Sparse autoencoder features should not be assumed to have consistent meanings throughout a sequence, motivating methods that account for representation dynamics.
- The findings motivate research on representation changes in conversations, codebases, and other applications to improve interpretability, safety, or adaptability.
- The experiments cover relatively few conversations and a narrow subset of concepts, limiting the breadth of the evaluation.
A. Supplemental Methods
The supplemental methods describe how conversations and prompts were formatted, sourced, and replayed for model evaluation. They also specify the opposite-day exchange used in the experiments.
- The opposite-day prompt instructs the model to answer user queries with the opposite of their real answers.
- The example exchange labels Fortune 500 companies as Antarctic and claims Earth is larger than the Sun.
- Turn 1 is the initial instruction-response exchange, while the first question-answer exchange is turn 2 and the final exchange is turn 3.
- Prompt conversations were sourced from prior work, handwritten, generated through model interaction, or drafted by another language model and edited for formatting.
- Evaluations followed model-specific chat formats, appending each question and candidate answer to the conversation context.
B.1. Accuracy on main question sets
This supplemental section reports Gemma V3 27B IT accuracy on the main question sets across the evaluated contexts.
- Gemma V3 27B IT accuracy is reported for the main question sets in each experimental condition.
B.2. Answer-wise scores and non-robust factuality representations without opposite day
Supplemental analyses separate factual and non-factual answers, compare robust and non-robust factuality dimensions, and examine layerwise consistency. The effects remain relatively similar across regression choices and layers.
- Factual and non-factual answers show relatively similar effects with robust and non-robust factuality regressions.
- Figure 7 shows that factual and non-factual answers invert even with some negative factuality bias across the consciousness conversation.
- Layerwise results remain relatively robust, with consistent condition rankings across the analyzed layers.
B.4. Contrast-Consistent Search (CCS)
CCS-derived representation directions generalize in empty contexts but often become unreliable after longer conversations, including for generic factual questions and conversation-specific questions.
- CCS directions often generalize in empty contexts but can classify generic factual questions below chance after longer conversations.This suggests the inferred directions are not robust to accumulated conversational context.
- Conversation-specific questions frequently shift from above-chance classification in empty prompts to at-or-below chance in conversation contexts.
- CCS representations are not immune to contextual effects and may be less robust overall after conversations.
- These context-dependent flipping patterns resemble those observed with supervised representation methods.
B.5. End-of-conversation correction partially reverses the representational shifts
A single end-of-conversation evaluation and critique exchange partially reverses topic-relevant factuality shifts, but does not fully restore the representations.
- Changing the model's conversational role toward a critic can rapidly alter representational directions.
- A corrective critique exchange substantially shifts topic-relevant factuality representations back toward correct answers, without complete recovery.The experiment changes the conversation's direction by having the model review and criticize its preceding responses.
- Generic factuality margins also degrade after critique, although behavioral performance remains high at 94.8%.
- The evaluation exchange produces only partial correction and does not recover an above-chance factuality margin.
B.6. Smaller Gemma models
Across Gemma V3 scales, representational changes are qualitatively stronger in larger models, while the 4B model shows little reliable change.
- Larger 27B and 12B Gemma models show more dramatic context-driven representational changes than the 4B model.
- These results suggest representation-change effects may be exacerbated at scale, though scale is difficult to disentangle from context-processing ability.
- The 12B model shows representations flipping near the conversation's end, whereas the 4B model shows little significant change on target questions.
B.7. Analyzing Qwen3 14B on opposite day
Qwen3 14B reproduces factuality and ethics representation flips on opposite day, but its lower baseline accuracy makes the magnitude of change harder to assess.
- Qwen3 14B's lower baseline accuracy limits confidence in measuring representational change from a reliable starting point.Its consciousness-question accuracy is 52.2% in an empty context, not significantly different from the 50% chance level.
- Qwen3 14B shows the same opposite-day flipping of factuality and ethics representations observed in other models.
- The cross-model replication suggests these representational changes are not entirely model-specific.
- The observed effects are noisier, likely because lower accuracy makes correct answers less cleanly represented.
B.8. Causal interventions can have opposite effects at different points in the context
Causal steering interventions can behave differently depending on the conversation context, including reversing their intended behavioral effect. An intervention that biases factual answers in an empty context can instead bias non-factual answers after a chakras conversation.
- Interventions bias answers toward factuality in an empty context but toward non-factuality after the chakras conversations.The intervention is applied to representations identified before the model answers.
- These results show that representation changes over conversation can cause interventions to yield different behavior from the contexts for which they were designed.The finding concerns causal steering before the model produces an answer.
- The intervention's effects after consciousness questions remain more consistent across the conversation than after the chakras conversations.Consistency in one long-context setting does not establish consistent behavior across other contexts.