Source-linked AI summary
Planting a Latent Variable in Natural-Looking Text: a More Realistic Test of Belief States in LLMs and Their Link to Concept Geometry
Alexandru-Iulius Jerpelea
TL;DR
The paper addresses whether belief-state tracking and latent-state geometry extend beyond synthetic toy languages. It plants a controllable Markov latent variable in natural-looking text and finds that a student transformer tracks its posterior while arranging states in the chain’s ring order.
Problem
Existing belief-state evidence relies mainly on synthetic HMM tokens, and probing posterior beliefs does not establish how latent-variable states are geometrically arranged.
Method
An LLM teacher generates ordinary text steered along eight sparse autoencoder directions whose active states follow a controlled ring-shaped Markov chain, then a small transformer is trained on it.
Results
The student tracks a noisier version of the optimal observer’s belief state; on first-dwell documents, its performance becomes indistinguishable from control2, supporting belief-state computation.
Takeaways & Limitations
The eight latent states form a ring in the exact transition order: the fitted pattern explains 38% of centroid variance and ranks 1st among 2520 orderings.
Takeaways & Limitations
The planted variable is not fully isolated because the SAE directions may themselves be latent variables or attributes, and the ring geometry is not dominant.
Abstract
from arXiv · showhide
LLMs are thought to track "belief states," i.e., running probability distributions over the latent variables that govern language (Shai et al., 2024; Sarfati et al., 2026), but so far this has only been comprehensively demonstrated on toy synthetic data and in a few isolated case studies. It has also never been empirically connected to the geometry of LLM features (the concepts interpretability finds in model activations). In this work, we plant a controllable latent variable inside natural-looking text. An LLM teacher writes ordinary text while we "subliminally" steer it along one of K = 8 unrelated sparse autoencoder directions at each token, with the active directions following a ring-shaped Markov chain. A small transformer model trained on this corpus does indeed track the Bayesian posterior belief about our planted latent variable. Moreover, it also arranges the 8 states themselves on a ring, in the exact order of the Markov chain, which is supporting evidence that a concept's geometry can be formed by the statistical dynamics of the latent variable behind it.
1 Introduction
Prior work shows transformers can encode Bayesian belief states on synthetic latent-variable sequences, but it remains unclear whether this extends to natural language or connects to concept geometry. This paper addresses both gaps by planting a controllable Markov latent variable in natural-looking text and testing whether a student transformer recovers its beliefs and state geometry.
- Prior evidence: Transformers trained on HMM-generated token sequences linearly encode a running probability distribution over the latent state.This belief state updates after each token and represents the model’s inferred state of the data-generating process.
- Open gap: Existing demonstrations rely mainly on synthetic HMM data, leaving their relevance to real natural language unresolved.The HMM dictates the whole toy language, while constructing a controlled natural-language experiment is difficult.
- Open gap: Interpretability work finds that concepts can occupy low-dimensional manifolds, but there is no full explanation for why these geometries form.Examples include circular and helical arrangements such as days of the week on a circle and calendar years on a helix.
- Open gap: Belief-state geometry and concept geometry are distinct residual-stream objects, and their empirical relationship remains unclear.The former represents probability vectors over latent-variable values; the latter arranges the values themselves.
- Approach: The paper generates ordinary text by steering an LLM teacher along K = 8 unrelated orthogonal SAE directions whose transitions follow a controllable ring-shaped Markov chain.A small transformer is then trained from scratch on the resulting corpus to test belief tracking and whether the eight states inherit the Markov geometry.
- Contribution: The student tracks the planted variable’s belief state and arranges its eight states on a ring in the exact neighbor order imposed by the Markov chain.These results directly target the gap between belief-state dynamics and concept geometry in a more natural-looking setting.
2 Method
The paper creates natural-looking text with a controllable, secretly labeled latent variable by steering a teacher LLM along eight SAE directions whose states follow a ring-shaped Markov chain. It then compares a from-scratch student transformer with an exact Bayesian observer and examines the resulting belief geometry.
- The Generative Process: The teacher generates ordinary-looking text while adding the active state’s SAE decoder direction to its residual stream.Steering is applied during generation, producing natural-looking text with a hidden state label at every token.
- The Corpus: The corpus uses one active direction per token, while filtering selects SAE latents that are causal, near-orthogonal, and non-co-activating on natural text.The generated documents contain 256 tokens, and the corpus includes a control condition with steering disabled.
- The Generative Process: A controllable latent variable is created by imposing Markov dynamics over eight otherwise unrelated SAE directions.The directions become states of a new latent variable, with transitions constrained by a chosen ring structure.
- The Corpus: The latent state follows an eight-state ring, with stay probability 0.95 and an average dwell time of approximately 20 tokens before moving to a neighbor.Each document’s 256-token state path is sampled before generation, so the imposed transition structure is known.
- The Optimal Observer: The optimal observer performs Bayes inference to maintain the exact posterior bt(i) = P(zt = i | x1:t) over the eight states at every token.Because the KV cache is reset at each token, the observer is computable and its posterior is used as the student’s probe target.
- The Optimal Observer: PCA of the optimal observer’s posteriors reveals a ring-shaped geometry, with the eight states arranged in the transition chain’s order.The student is a 110M-parameter Llama-style transformer trained from scratch, and controls test unsteered data and switching without neighbor structure.
3 Results
The student tracks the planted latent variable’s Bayesian belief state, while control comparisons isolate the benefit of learning ring transitions. Its state centroids also form the Markov chain’s ring geometry, although first-dwell and memory controls qualify the interpretation.
- 3.1 The Student Tracks the Belief States: The best layer reaches R2 = 0.49 for the Bayesian posterior, with 0.577 state-argmax accuracy versus the optimal observer ceiling of 0.762.The student had no direct access to the steered teacher model.
- 3.1 The Student Tracks the Belief States: Control2 reaches R2 = 0.41 and 0.52 argmax accuracy because tokens eventually reveal the current state even without ring-transition structure.The student’s approximately 0.08 R2 advantage is attributed to using how the previous state constrains the next one.
- 3.1 The Student Tracks the Belief States: On first-dwell tokens, the ring student and control2 become indistinguishable, consistent with transition information helping only after a state switch.With no prior information, the transition map is useless even to the optimal observer.
- 3.1 The Student Tracks the Belief States: The student’s probe read-out has the same ring arrangement as the optimal observer’s posterior geometry, but with a noisier cloud.The comparison uses a layer-12 PCA view.
- 3.2 The Eight States of the Latent Variable Sit on a Ring: Across layers 9–11, the eight state centroids follow the Markov chain’s exact neighbor order, with pairwise-distance correlation 0.45 versus 0.11 for control1.The true ring ranks first among 2520 possible orderings by explained centroid variance for the student, while control models land mid-distribution.
- 3.2 The Eight States of the Latent Variable Sit on a Ring: A centroid-memory confound can itself draw a ring, so first-dwell averaging removes previous-state leakage; the top-2 PCA circle then does not survive.Fourier analysis is used to test for ring structure in another plane, while raw similarities are warmest between ring neighbors.
4 Conclusion
The paper introduces a method for planting a controllable Markov latent variable in natural-looking text and connects belief-state learning to concept geometry. It reports that the planted states follow the intended ring structure, although the clean ring is not visible in the leading PCA components of first-dwell centroids.
- The method plants a latent variable in natural-looking text by steering an LLM teacher along SAE directions whose activity follows a chosen Markov chain.The teacher writes ordinary text while the active direction changes according to the controlled latent dynamics.
- The clean ring is no longer visible in the leading principal components of first-dwell centroids.
- The paper connects belief-state geometry to concept geometry, arguing that latent-variable geometry is influenced by the dynamics of the data-generating process.This extends the belief-state result to the geometry of the latent variable itself.
5 Future Work
The paper proposes extending the setup to multiple interacting latent variables, semantic-dynamics conflicts, manifold entanglement, hierarchical concepts, and predictions for real LLMs. These directions aim to test whether controlled latent dynamics explain broader concept geometries.
- A natural follow-up is planting multiple HMMs simultaneously, including more complicated interactions between them.
- The setup could directly test whether feature manifolds reflect semantic similarity when semantically similar states follow a Markov chain unrelated to that similarity.
- The authors propose testing whether states participate in multiple latent-variable geometries, producing possible manifold entanglement.The multiple-HMM setup could test this scenario directly.
- Future work includes studying how hierarchical concepts fit into the proposed relationship between latent dynamics and geometry.
- The authors aim to estimate a real latent variable’s transition dynamics from a training corpus and predict its concept geometry in an LLM such as OLMo.
Limitations
The study remains limited by its imposed latent variable, correlational evidence, narrow experimental configurations, and incomplete isolation of the planted structure. These constraints leave open how latent variables should be injected into natural-looking text.
- The experiment imposes a latent variable on language, so it does not fully prove the broader belief-state hypothesis.Its more natural text setting retains the limitation of earlier synthetic demonstrations.
- The evidence for concept geometry is correlational because the paper does not perform causality experiments.
- The study needs broader configurations, including more HMM structures, SAE latent choices, and teacher models.
- The SAE directions used to construct the latent variable may themselves encode latent variables or attributes of other variables, preventing full isolation of the planted variable.The ring geometry was not dominant, leaving open how to inject latent variables more cleanly.