Source-linked AI summary
Emergent Linear Representations in World Models of Self-Supervised Sequence Models
Neel Nanda, Andrew Lee, Martin Wattenberg
TL;DR
The paper addresses whether OthelloGPT’s board-state world model is genuinely nonlinear or can be represented linearly. It identifies a player-relative coding scheme, uses linear probes and vector interventions to test it, and finds accurate representation and controllable predictions, while noting that the identified circuit does not explain the entire model.
Problem
Prior work reported nonlinear board-state representations in OthelloGPT, leaving the linearity and interpretability of its internal world model unresolved.
Method
The paper probes board states using MINE, YOURS, and EMPTY directions relative to the current player, then intervenes on residual-stream activations with corresponding linear vectors.
Results
The board state is linearly represented relative to the current player, and simple vector arithmetic accurately controls the model’s behaviour.
Takeaways & Limitations
Mechanistic understanding of linear representation directions can support interpretable control and analysis of emergent world models.
Takeaways & Limitations
The identified board-state circuit does not explain the entire model, and the authors find hints that multiple circuits may predict legal moves in end games.
Abstract
from arXiv · showhide
How do sequence models represent their decision-making process? Prior work suggests that Othello-playing neural network learned nonlinear models of the board state (Li et al., 2023). In this work, we provide evidence of a closely related linear representation of the board. In particular, we show that probing for "my colour" vs. "opponent's colour" may be a simple yet powerful way to interpret the model's internal state. This precise understanding of the internal representations allows us to control the model's behaviour with simple vector arithmetic. Linear representations enable significant interpretability progress, which we demonstrate with further exploration of how the world model is computed.
1 Introduction
The paper asks how sequence models represent decision-making and challenges the claim that OthelloGPT encodes board states nonlinearly. It argues that a player-relative linear representation supports accurate probing, behavioral steering, and further mechanistic analysis.
- The paper investigates whether features in sequence-model activations are represented as linear directions, a question with major interpretability implications.It frames large language models as feature extractors but notes that the linear representation hypothesis remains controversial.
- Unlike the original nonlinear-probe interpretation, the authors demonstrate that OthelloGPT’s closely related world model is linearly encoded.
- OthelloGPT represents tiles relative to the current player as MINE, YOURS, or EMPTY rather than using absolute BLACK, WHITE, and EMPTY labels.The mapping reverses between odd and even timesteps.
- A linear projection learned using this player-relative insight derives board states with near perfect accuracy.
- Vector arithmetic along MINE, YOURS, or EMPTY directions changes the model’s internal board belief and predictions.The intervention method is presented as simpler and more interpretable than gradient-based editing.
- Additional linear analyses examine how empty tiles are detected and how tiles are represented as FLIPPED at each timestep.
2 Preliminaries
The paper studies OthelloGPT, an autoregressive transformer trained only to predict valid moves from move sequences. It introduces the model architecture and probe formulations used to analyze its internal board representations.
- 2.1 Othello: Othello is a two-player game on an 8x8 grid where players place black or white discs and flip enclosed opponent discs.The objective is to finish with a majority of one’s coloured discs.
- 2.2 OthelloGPT: OthelloGPT is an 8-layer GPT with 8 attention heads per layer and a 512-dimensional hidden space.Its vocabulary contains 60 playable-move tokens.
- 2.2 OthelloGPT: The model autoregressively predicts the next valid move from prior moves without being given the game or its rules.Training sequences use uniformly chosen valid moves rather than strategic moves trained to win games.
- 2.3 Notations: Each transformer layer adds attention-head and MLP outputs to the residual stream between embedding and unembedding.The final prediction applies the unembedding layer followed by a softmax.
- 2.3 Notations: Linear probes are projections from the residual stream, while nonlinear probes are two-layer MLPs used for three-way classification.The three classes are evaluated either as absolute colours or relative MINE, YOURS, and EMPTY labels.
3 Linearly Encoded Board States
The experiments test whether OthelloGPT’s board state is linearly represented relative to the current player. Linear probes reach high accuracy by layer 4 under this formulation.
- 3.1 Experiment Setup: The experiment classifies each tile as MINE, YOURS, or EMPTY relative to the player at that timestep, rather than as BLACK, WHITE, or EMPTY.The BLACK-to-MINE and WHITE-to-YOURS mapping reverses on alternating timesteps.
- 3.1 Experiment Setup: The probes train on 3,500,000 game sequences and are evaluated on a held-out set of 1,000 games.A validation set of 512 games is used, and probes typically converge after about 100,000 training samples.
- 3.1 Experiment Setup: The study compares the learned probes with randomly initialized-model, probabilistic, and absolute-colour probe baselines.The absolute-colour baselines classify BLACK, WHITE, and EMPTY.
- 3.1 Experiment Setup: The board-state probe predicts all 64 tiles using a D × 64 × 3 output shape.
- 3.2 Results: The intervention methodology adds EMPTY, MINE, or YOURS directions to residual-stream layers and tracks changed tiles and newly predicted legal moves.Red squares mark intervened tiles, while teal tiles mark new legal moves after intervention.
- 3.2 Results: Linear probes achieve high accuracy by layer 4, demonstrating that the emergent board state is linearly encoded.
4 Intervening with Linear Directions
The authors intervene on OthelloGPT’s board-state representation by adding linear directions to the residual stream, testing whether predictions reflect altered states. These single-vector interventions match gradient-based editing while enabling both colour flips and tile erasure.
- Intervention setup: The experiment tests whether intervening on encoded board states changes predictions toward an altered target state B′ rather than the original state B.Each case uses a partial game sequence and targeted board state, with efficacy evaluated over 1,000 test cases.
- Intervention mechanism: The intervention adds a scaled MINE, YOURS, or EMPTY vector to the residual stream at every layer.Pushing in MINE flips YOURS tiles to MINE, while EMPTY erases previously played tiles.
- Comparison with prior editing: A single vector addition replaces Li et al.’s iterative gradient-descent editing of OthelloGPT’s activation space.The linear-direction approach is presented as simpler and more interpretable than editing from a nonlinear probe.
- Evaluation: Intervention efficacy is measured as error rate by comparing top-N post-intervention predictions with the legal moves at target state B′.The metric counts false positives and false negatives in a multi-label classification formulation.
- Results: The interventions are equally effective as gradient-based editing, confirming that the interpreted linear directions match how the model uses them.The experiment also includes erasing a previously played tile to validate the EMPTY direction beyond colour flipping.
5 Additional Linear Interpretations
Linear probes reveal how OthelloGPT represents empty and flipped tiles, while interventions test their behavioral relevance and expose changing board-versus-move computation across games.
- 5.1 Interpreting Empty Tiles: The model broadcasts which moves have been played through attention heads, encoding PLAYED information in residual streams via linear projections.Each move is embedded and transformed through attention value and output projections.
- 5.1 Interpreting Empty Tiles: An average cosine similarity of -0.862 across all 60 squares indicates that pEMPTY encodes NOT PLAYED as a linear function of token embeddings.The analysis compares the PLAYED representation with the empty-tile probe direction.
- 5.1 Interpreting Empty Tiles: The EMPTY probe reaches 98.9% accuracy from x0_mid, compared with 76.8% from X0_pre, showing that empty-tile information is available before the MLP layer.The evaluation uses EMPTY versus NOT-EMPTY classification on 1,000 test games.
- 5.1 Interpreting Empty Tiles: Attention heads selectively attend to MY or YOUR moves, alternating across timestep parity and supporting a MINE/YOURS rather than BLACK/WHITE board representation.Figure 4 gives examples of heads attending to YOUR moves on the left and MY moves on the right.
- 5.3 Additional Linear Concepts: FLIPPED: The model linearly encodes FLIPPED tiles, and intervening on a flipped tile reduces average error rate to 0.486 from a null-intervention baseline of 1.686.The intervention tests whether the flipped feature is causally relevant for predicting moves in the altered board state.
- 5.4 Multiple Circuits Hypothesis: MOVEFIRST appears around move 30 and becomes more frequent toward the end, while board-state accuracy drops and legal moves may precede perfect board computation.The authors hypothesize multiple circuits, but state that the reason for this end-game behavior remains unclear.
6 Discussions
The discussion highlights that probing choices shape the interpretation of model representations, and that linearity depends on the feature basis used. It also situates these questions within broader debates about why linear representations emerge.
- The features selected for probing can produce different conclusions, as BLACK/WHITE and MINE/YOURS classifications yield different takeaways.The authors warn that human-intuitive features may not match the model’s own representation.
- Whether a representation is linear or nonlinear can depend on the coordinate system used to describe the underlying concept.The discussion uses distance and gravitational-force representations to illustrate this relativity.
- Prior work found robust linear image representations in iGPT, leaving open why linear features emerge and which linear features language models encode.One proposed reason is that matrix multiplication can extract different subsets of linear features for each neuron.
7 Related Work
Related work spans internal representations, interventions, and mechanistic interpretability across sequence, language, vision, and game-playing models. These studies provide precedents for linear probing, world-model discovery, activation control, and circuit analysis.
- The related literature covers three areas: internal representations, interventions, and mechanistic interpretability.
- Internal representations: Sequence-model studies have uncovered world representations, grounded concepts such as direction and colour, and linear vectors associated with truthfulness.
- Figure 6 visualizes layerwise linear-probe projections alongside legal-move predictions produced by applying the unembedding layer at each layer.
- Internal representations: Linear representations have been studied in word embeddings, linguistic probing, vision-language mappings, and interpretable chess or Hex concepts.
- Interventions: Intervention research controls model behavior by altering activations, using either optimization-based methods or inference-time linear arithmetic.
- Mechanistic interpretability: Mechanistic interpretability reverse-engineers neural-network behavior and representations to assess underlying computations and alignment with researcher intent.The framework has been applied to grokking, superposition, and individual-neuron analysis.
8 Conclusion
The paper concludes that Othello-playing sequence models contain multiple linear representations that support board-state interpretation, behavioral control, and mechanistic analysis. It also reports evidence for multiple end-game circuits and iterative feature refinement across layers.
- The board state is linearly represented relative to the current player as MINE, YOURS, and EMPTY rather than as absolute colours.The authors report accurate behavioral control through simple vector arithmetic on this internal world model.
- The analysis extends beyond board state to empty-tile detection and linear representations of pieces being FLIPPED.
- End-game behavior provides hints of multiple legal-move circuits and further evidence that residual networks iteratively refine features across layers.
10 Author Contributions
The paper’s contributions were distributed across discovery, experimentation, writing, editing, and implementation reporting. The listed contributions distinguish conceptual discoveries from experimental leadership and supporting editorial work.
- Neel Nanda discovered the relative-board-state linear representation, demonstrated vector-arithmetic interventions, led initial experiments and write-ups, and advised throughout.
- Andrew Lee led the write-up and performed all experiments, including discoveries concerning flipped representations, empty tiles, and multiple-circuit hypotheses.
- Martin Wattenberg edited and distilled the paper and contributed the analogy comparing linear and quadratic distance representations.
- Table 4 reports the hyperparameters used for the linear probes.
- The authors observe that intervening on a sufficient number of layers is needed for OthelloGPT to alter its predictions.They hypothesize that this requirement relates to the residual structure of transformer models.
C Multiple Circuits
At end games, OthelloGPT may predict moves before fully computing the board, while probe accuracy also declines. These findings motivate hypotheses involving alternative or multiple circuits.
- OthelloGPT sometimes computes moves before boards at end games.The analysis examines whether legal moves can be predicted without the entire board.
- Probe accuracy drops during the end game, leaving two explanations for the incomplete board state.The model may either avoid computing a perfect board or use an alternative circuit because correct board computation becomes difficult.
- Board-state accuracy, FLIPPED accuracy, and next-move predictions improve across layers, while probes share similar directions.These patterns provide multiple evidence of iterative feature refinement.
E On Principled Ways of Probing
The paper highlights three challenges for probing: establishing causality, choosing principled features, and avoiding probes that compute or memorize their targets. It argues that linear probes offer a more interpretable route while retaining useful control and analysis.
- Probe findings may correlate with model behavior without proving that the model uses the discovered features.Intervention is used to study causality, but prior intervention results have been mixed.
- Figure 8 compares when 90% of the board state is computed before or after move predictions.The figure tests the timing relationship between broad board-state computation and move prediction.
- Figure 9 examines when the minimum necessary set of tiles is computed relative to move predictions.Figure 10 illustrates an example of the minimum tile set that makes a move legal.
- Choosing the feature categories to probe can lead to different takeaways and pitfalls.For OthelloGPT, classifying absolute colors versus player-relative colors illustrates the danger of projecting preconceptions.
- Expressive probes may compute or memorize the desired feature rather than extract it, creating a validity risk.The paper presents linear probes as easier to interpret because their features correspond to vector directions.
- Linear probes are presented as having interpretable and controllable power, with anticipated applicability to larger language models.The claim is framed as evidence and an expectation rather than an established general result.