PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg

arXiv:2210.13382v5cs.LGcs.AIcs.CL

TL;DR

The paper asks whether sequence models merely memorize surface statistics or instead form internal representations of the processes generating their sequences. It trains a GPT variant on Othello move transcripts and finds evidence for a nonlinear board-state representation that can be causally linked to predictions and used for latent saliency maps.

  • Problem

    The paper addresses whether language-model competence reflects memorized surface statistics or internal representations of the processes generating observed sequences.

  • Method

    The authors train Othello-GPT without prior game knowledge, probe its activations for board state, and intervene on those activations to represent counterfactual boards.

  • Results

    The experiments provide evidence that Othello-GPT maintains a nonlinear representation of game board states that is causally linked to its predictions.

  • Takeaways & Limitations

    Interventional experiments can produce board-level latent saliency maps that provide insight into how the network makes predictions.

  • Takeaways & Limitations

    The study’s future work is to extend the investigation to more complex games and compare transcript-trained models with systems given a priori board knowledge.

Abstract

from arXiv · show

Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.

1 INTRODUCTION

The paper studies whether sequence models learn internal, interpretable representations of the processes generating their data rather than merely memorizing surface statistics. Using Othello as a controlled setting, it probes and intervenes on a GPT model’s board-state representation to examine its role in predictions.

  • Language models can display capabilities beyond next-word prediction, but how such performance emerges from sequence modeling remains debated.
  • Memorizing surface statistics may fail on out-of-distribution data because such correlations do not reflect a causal model of sequence generation.
  • Othello provides a controlled, synthetic testbed for examining world representations because its rules and state are simpler than chess.
  • Othello-GPT predicts legal moves from game transcripts despite receiving no prior knowledge of Othello or its rules and no training objective to play strategically.
  • Probes infer board states from internal activations, while interventions modify activations toward counterfactual states to test whether the representation affects predictions.
  • The paper also uses board-level interventions to create latent saliency maps that explain predictions at a higher level than individual input tokens or moves.

2 “LANGUAGE MODELING” OF OTHELLO GAME TRANSCRIPTS

The authors train Othello-GPT on move transcripts using minimal input structure and evaluate whether it learns legal-move behavior rather than memorizing sequences. Models trained on both championship and synthetic data predict legal moves accurately, motivating further probing of internal game-state representations.

  • Othello is used because its game tree is too large to memorize while its rules and state remain simpler than chess.
  • Othello-GPT extends partial game transcripts with legal moves using only token sequences and no a priori knowledge of the game or rules.
  • The championship dataset contains expert human games, whereas the much larger synthetic dataset contains legal but otherwise random moves.
  • The model receives sequential tile indices as tokens, with geometrical relationships among tiles left for training to learn rather than explicitly provided.
  • 0.01% synthetic-data error and 5.17% championship-data error show far better legal-move prediction than the 93.29% error of the untrained model.
  • The study then turns from legal-move performance to probing whether Othello-GPT computes internal representations of game state.

3 EXPLORING INTERNAL REPRESENTATIONS WITH PROBES

The paper probes Othello-GPT’s internal activations to test whether they encode board state. Linear probes perform poorly, whereas nonlinear probes recover board-state information more accurately, supporting a nontrivial internal representation.

  • Probe setup: Probes classify each tile’s black, white, or empty state from Othello-GPT activations after a move sequence.The probe input is an autoregressive activation from a chosen layer, and validation pairs are split 8:2.
  • 3.1 LINEAR PROBES HAVE HIGH ERROR RATES: Linear probes never achieve below 20% error, with only marginal improvement over probes on a randomized network.This suggests any board-state representation is not encoded in a simple linear form.
  • Memorization control: The skewed-game-tree analysis is intended to test whether performance reflects memorized transcripts rather than board-state computation.The supplied passages note that removing one opening branch does not fully prevent repeated board states because different move sequences can converge to the same state.
  • Probe comparisons: The evaluation compares randomized Othello-GPT with models trained on different datasets across layers using tile-state error rates.Table 1 reports linear-probe error rates, while Table 2 reports nonlinear-probe error rates and standard deviations are provided in Appendix H.
  • 3.2 NONLINEAR PROBES HAVE LOWER ERROR RATES: Nonlinear probes achieve significantly better accuracy on trained networks than linear probes, while randomized-network nonlinear probes show almost no comparable improvement.The contrast indicates that trained activations contain a nontrivial board-state representation recoverable through nonlinear computation.

4 VALIDATING PROBES WITH INTERVENTIONAL EXPERIMENTS

The authors intervene on internal activations to change a probed board state by one tile and then test whether legal-move predictions change accordingly. Across natural and unreachable board positions, the intervention substantially reduces prediction error relative to a null intervention.

  • 4.1 INTERVENTION TECHNIQUE: Interventions change a baseline board state B into B′ by modifying activations so a probe reports a different state for one targeted tile.The intervention records predictions for B, alters the internal representation, and evaluates predictions associated with B′.
  • 4.1 INTERVENTION TECHNIQUE: Because later GPT layers incorporate earlier representations, the intervention modifies the targeted activation from an initial layer Ls through the final layer.The procedure acts at the temporally last token and alternates intervention with subsequent computation.
  • 4.1 INTERVENTION TECHNIQUE: Gradient descent on the probe’s class score changes the internal activation for the tile whose state is being modified.The method optimizes the activation rather than the probe weights, using cross-entropy against the desired board state.
  • 4.2 INTERVENTION RESULTS: The benchmark contains 1000 natural and 1000 unnatural cases, with the latter using positions unreachable by legal play.Unnatural positions provide a stringent test because they are far from the training distribution.
  • 4.2 INTERVENTION RESULTS: 0.12 and 0.06 average errors are achieved on the two benchmarks with Ls = 4, versus baseline errors of 2.68 and 2.59.The metric sums false positives and false negatives when comparing top-N predictions with the post-intervention legal-move set.

5 LATENT SALIENCY MAPS: ATTRIBUTION VIA INTERVENTION

The paper uses interventions on latent board-state representations to attribute Othello-GPT predictions to individual tiles. Synthetic-trained and championship-trained models produce distinct saliency patterns reflecting legality versus strategic move selection.

  • Attribution via intervention: Interventions change one tile’s latent state and measure how much the probability of an attributed legal move changes.The resulting values are normalized by subtracting the board-wide mean.
  • Synthetic dataset: Synthetic-trained Othello-GPT assigns high saliency to tiles required for move legality, while most other tiles receive lower values.This pattern can reveal that the model’s goal is learning legal moves even without access to the training procedure.
  • Championship dataset: Championship-trained Othello-GPT produces more complex maps, with high saliency on legality-related tiles and many other tiles.The broader pattern is consistent with expert moves depending on complex global board features.
  • Interpretation: Latent saliency maps provide board-based visualizations of how interventions affect the model’s top-1 predictions.The maps distinguish the underlying information used by models trained on synthetic and championship data.

6 RELATED WORK

Related work studies internal world representations in sequence models and board-game systems. This paper extends those directions by examining the form of representations learned without explicit game structure or rules.

  • World representations: Prior studies found that sequence models can encode semantic world-state information, including color, direction, and Boolean states.Some work also reports geometric connections between model activations and standard color space.
  • Board-game models: Chess-language-model research showed strong evidence that transformers track board state but did not investigate the form of that internal representation.That gap motivates the present study.
  • Board-game models: Unlike many game-playing systems, this study examines a model trained on move transcripts without a priori knowledge of board structure or game rules.This contrasts with systems such as AlphaZero, which receive built-in game knowledge.
  • Controllable generation: The intervention approach is related to methods that steer frozen autoregressive or diffusion models by optimizing desired attributes alongside generation quality.These related methods involve a trade-off between the target attribute and fluent generation.

7 CONCLUSION

The paper concludes that Othello-GPT develops a nonlinear, causally relevant representation of board states and that interventions can visualize its predictive reasoning. The authors identify generalization to more complex games and natural language as future work.

  • Conclusion: Experiments provide evidence that Othello-GPT maintains a nonlinear representation of board states and that this representation is causally linked to predictions.The representation concerns the Othello world encoded by the sequence model.
  • Conclusion: Interventional experiments produce latent saliency maps that show how predictions relate to the Othello board.Maps from synthetic- and championship-trained models reveal dramatic differences in their underlying representations.
  • Future work: Future work includes testing more complex games and comparing transcript-trained models with systems given a priori Othello knowledge.The proposed comparison includes latent saliency maps versus standard saliency maps from a program that receives the actual board state.
  • Future work: The generalization of these findings to models trained on natural language remains an open direction.The authors suggest synthetic language tasks as one possible stepping stone toward natural-language settings.
  • Probe geometry: The probe-weight geometry shows clearer correlations with board-grid structure for trained models than for the randomized baseline.The randomized network is not completely random, consistent with above-chance linear-probe performance.

B ABLATION ON NONLINEAR PROBE ACCURACIES

The ablations examine where and when nonlinear probes recover board concepts. Probe accuracy increases with capacity, peaks at intermediate layers, and follows a time-dependent pattern indicating delayed updating and forgetting.

  • What-how-when: Nonlinear-probe accuracy increases as the number of hidden units increases, because greater capacity captures more information from the hidden space.The analysis varies hidden-unit count and the layer from which representations are extracted.
  • What-how-when: Probe accuracy peaks at an intermediate Othello-GPT layer rather than at the deepest layer.The paper notes that this resembles findings that linguistic properties are often best probed in middle layers.
  • What-how-where and what-how-when: Figure 6 compares probe accuracy across game progression and across Othello-GPT layers for randomized, championship-trained, and synthetic-trained models.Game progression is indexed by how long a tile has remained in its current state.
  • What-how-where: A parabolic accuracy curve shows that concepts are best captured after existing for some time but before persisting too long.The authors interpret this as evidence of uncertainty before board-state updates and forgetting when representations change.
  • Intervention result: In one intervention case, pre-intervention and post-intervention predictions both have zero errors even though the ground-truth legal moves change.Systematic results are reported over 1,000 cases.

D INTER-PROBE INTERACTION

Intervening at the fifth layer can alter the probed board state, but later layers may correct or preserve the changed tile differently. The figures compare board-state probes and move heatmaps before and after intervention.

  • D INTER-PROBE INTERACTION: At layer 5, intervention corrected the flipped C4 disc in the immediately following layer.When the same intervention was applied later, the model became more convinced that C4 should be black and stopped correcting it.
  • D INTER-PROBE INTERACTION: Figure 7 compares pre-intervention and post-intervention move-probability heatmaps, with legal moves underscored in each condition.
  • D INTER-PROBE INTERACTION: Figure 8 compares probed world states before and after intervention across layers, with color indicating confidence in each tile’s top-1 prediction.

E DISCUSSION OF LATENT SALIENCY MAPS

Latent saliency maps can expose rule-relevant board structure, but their attribution becomes incomplete when a move is legal through multiple lines. The reported maps therefore align better with single-line than multi-line Othello logic.

  • E DISCUSSION OF LATENT SALIENCY MAPS: When multiple discs are flipped, the intervention-based map identifies only the first contributing disc, diverging somewhat from human understanding of Othello.Other flipped discs can still make the current prediction legal, even when they are in opposite states.
  • E DISCUSSION OF LATENT SALIENCY MAPS: The method visualizes the AND-logic for flipping one straight line: opponent discs must lie between the move and a same-color disc.
  • E DISCUSSION OF LATENT SALIENCY MAPS: For moves flipping more than one line, intervening on one line does not nullify the prediction because Othello permits any one of eight lines to justify legality.The authors identify extracting this OR-over-AND structure with unlimited interventions as future research.

F ALTERNATIVE METRICS FOR THE INTERVENTION EXPERIMENT

The intervention experiment was evaluated with multiple metrics because top-1 accuracy was nearly saturated on the synthetic model. Error, F1, and KL divergence nevertheless showed similar trends, supporting the intervention technique and the proposed causal interpretation.

  • F ALTERNATIVE METRICS FOR THE INTERVENTION EXPERIMENT: The evaluation treats intervention success as multi-class classification rather than top-1 prediction because top-1 performance is nearly saturated on the synthetic model.The benchmark intervention changes the legal-tile set by an average of 2.12 tiles.
  • F ALTERNATIVE METRICS FOR THE INTERVENTION EXPERIMENT: KL divergence measures the predicted next-move distribution against a discrete uniform prior over legal next moves.Baseline F1 and KL values are averaged across 1000 intervention cases.
  • F ALTERNATIVE METRICS FOR THE INTERVENTION EXPERIMENT: Error, F1, and KL divergence exhibit similar trends, supporting the usefulness of the intervention technique and the causality from world representations to predictions.

G ABLATIONS ON INTERVENTION HYPER-PARAMETERS

The intervention procedure is robust to optimizer settings, while its starting layer and regularization weight have important practical effects. Intervening too shallowly is hazardous, whereas intervening only deeply leaves too little computation for the model to adapt.

  • G ABLATIONS ON INTERVENTION HYPER-PARAMETERS: Intervention optimization is robust across different optimizers, learning rates α, and numbers of steps.
  • G ABLATIONS ON INTERVENTION HYPER-PARAMETERS: Regularizing cross-entropy losses for non-target tiles is intended to prevent unintended tile changes, but sweeping β at Ls = 5 did not clearly help.
  • G ABLATIONS ON INTERVENTION HYPER-PARAMETERS: Intervening across more than five layers touches shallow layers without reliable world representations, making the intervention hazardous.
  • G ABLATIONS ON INTERVENTION HYPER-PARAMETERS: Intervening only at the deepest layers can successfully modify the world representation but leaves insufficient computation for predictions to adapt to it.
  • G ABLATIONS ON INTERVENTION HYPER-PARAMETERS: Repeating probe experiments across 100 random seeds supported robustness to randomness and significant differences between linear and nonlinear probe accuracies.
Loading 2210.13382v5…