Source-linked AI summary
The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors
Raphaël Sarfati, Eric Bigelow, Daniel Wurgaft, Siddharth Boppana, Jack Merullo, Atticus Geiger, Owen Lewis, Tom McGrath, Ekdeep Singh Lubana
TL;DR
The paper asks how LLMs encode, update, and manipulate implicit posterior beliefs in representation space. Using Llama 3.2 on in-context normal-distribution inference, it finds curved belief manifolds and shows that geometry-aware interventions better preserve intended belief families than standard linear steering. The results support treating beliefs as geometric objects rather than globally linear representations.
Problem
The paper addresses the lack of a mechanistic account of how LLMs encode beliefs over latent variables, update them with evidence, and reshape them through representation interventions.
Method
The authors study Llama 3.2 inferring normal-distribution parameters from in-context samples and use linear field probes to characterize and intervene on belief manifolds.
Results
The study finds curved representation manifolds and shows that standard linear steering can move activations off-manifold, causing unintended coupled shifts, whereas geometry-aware interventions better preserve the intended belief family.
Takeaways & Limitations
LLM beliefs are best understood as structured geometric objects, making manifold-aware probing and intervention more faithful than globally linear control.
Takeaways & Limitations
The analysis uses true activations from unf finetuned base models and is currently constrained to numerical settings, leaving broader natural-language extensions open.
Abstract
from arXiv · showhide
Large language models (LLMs) form implicit beliefs (posteriors over latent variables) from prompts, but we lack a mechanistic account of how these beliefs are encoded in representation space, how they update with new evidence, and how interventions reshape them. We study a controlled setting in which Llama-3.2 infers the parameters of a normal distribution from in-context samples. We show that parameter posteriors are encoded as curved manifolds in representation space, and trace how they evolve along the prompt. Standard linear steering moves representations off-manifold, inducing unintended, coupled changes, whereas geometry-aware methods preserve the target belief family. Our work demonstrates an example of linear field probing (LFP) as a principled approach to tile the data manifold and make interventions that respect the underlying geometry. Our results suggest that LLM beliefs are inherently geometric objects, and that globally linear representations are often inadequate abstractions.
1 Introduction
The paper studies how LLMs encode, update, and manipulate implicit beliefs using a controlled normal-distribution inference task. It finds curved representation manifolds, evolving belief trajectories, and geometry-aware interventions that better preserve intended belief families.
- Natural-language ambiguity requires models to represent uncertainty over latent states and update that uncertainty as evidence arrives.
- The paper studies how models encode posterior beliefs, update them with new evidence, and respond to interventions on representations.
- The controlled task feeds Llama 3.2 comma-delimited integers sampled from a generating distribution Gθ, whose parameters provide known latent ground truth.
- Geometry: Posterior beliefs over (µ, σ) form smooth but curved manifolds that can be read out using linear field probes.
- Dynamics: When the generating distribution switches mid-sequence, the study quantifies belief-update trajectories and the timescales required for equilibration.
- Interventions: Geometry-aware interventions better preserve the intended belief family and avoid the coupled shifts produced by standard linear steering.
- Conclusion: The authors conclude that LLM beliefs are structured geometric objects rather than single linear directions, with manifolds supporting more reliable probing and intervention.
2 Experiments and analysis
The experiments use stochastic numerical time series and probability-aware dimensionality reduction to compare model outputs with generating distributions. They analyze activations, softmax behavior, and linear field probes within this controlled setup.
- Stochastic numerical time series provide a continuous family of belief states whose generating distributions can be compared directly with model outputs.
- The setup includes comma-predicting-number and number-predicting-comma tokens, with com2num used unless otherwise noted.
- The authors make code for data generation, analysis, and figure reproduction available online.
- PCA is used for high-dimensional vectors, while inPCA is used for softmax probability vectors constrained to the unit d-simplex.
- Activations, representations, and hidden states refer interchangeably to the model’s residual stream, indexed by token, sequence, and layer.
- Logits are treated as a behavior proxy, normalized with softmax at temperature T = 1.0, and linear field probes detect specified input-domain values.
3 Results
The results map Llama-3.2’s inferred Gaussian beliefs onto curved representation manifolds, trace their dynamics after distribution switches, and use linear field probes to read and intervene on them. The findings show that local linear readout can coexist with curved geometry, limiting globally linear steering.
- 3 Results: The study uses linear field probes to expose structured belief geometry and support interventions that target belief parameters without treating them as single linear directions.The broader workflow covers manifold mapping, switching dynamics, probing, and geometry-based intervention design.
- 3.1 Shape of beliefs: Llama-3.2 acquires posterior beliefs over input distributions and encodes them in smooth, substantially curved activation manifolds.The manifolds vary independently along mean and standard deviation, forming two approximately orthogonal belief families.
- 3.2 Belief dynamics: After a switch from N(300,100) to N(700,100), the model rapidly adjusts its mean while variance equilibrates more slowly.The variance remains elevated for roughly 300 tokens before relaxing toward its true value.
- 3.3.1 Linear field probes: Linear field probes tile the belief manifold: mean-indexed representations are linearly separable, probe vectors vary smoothly, and intermediate probes can be interpolated.Probe accuracy rises from 0.87 at layer 0 to 0.99 at layer 15, while interpolation works without retraining.
- 3.3.1 Linear field probes: Probe transfer is local, with performance decaying as mean distance increases, indicating curvature and limiting any single global linear direction.The transfer decay matches the Gram-matrix decay lengthscale.
- 3.3.2 Field geometry: The probe Gram matrix captures readout geometry, which can remain low-dimensional even when the underlying activation manifold is highly curved.Intrinsic dimensionality generally increases across layers but drops at the final layer, while principal directions dominate the readout geometry.
4 Interventions
The paper compares linear, activation-manifold, and field-aware steering methods for changing inferred distribution parameters. Linear interventions can leave the Gaussian belief manifold and distort coupled properties, whereas geometry-aware paths better preserve the intended distribution family.
- Linear steering can push next-token predictions off the data manifold and misalign them with the target distribution.
- Activation-aware steering (primal space): Because preimage manifolds vary nonlinearly, linear steering toward a target mean can produce large unintended changes in output variance.
- Activation-aware steering (primal space): Manifold-aware steering follows prototypes across the activation manifold, preserving class-conditioned structure while shifting the output mean toward the target.
- Field-aware steering (dual space): Linear probe steering uses a normalized difference between probe vectors at µ = 700 and µ = 300 as its steering direction.
- Field-aware steering (dual space): A single probe direction can move activations far from the σ0 = 100 manifold, whereas field-geometric travel preserves standard deviation while increasing the mean between 300 and 500.
- Field-aware steering (dual space): Field-aware steering uses probe geometry and kernel-regressed weights to keep induced distributions closer to the intended N(µ, σ0) family within a certain range.
5 Conclusion
The paper frames LLM beliefs as structured manifolds whose dynamics and interventions reveal limitations of globally linear representations. Its numerical, unfine-tuned setting clarifies geometric implications while leaving extension to broader language tasks open.
- The framework maps posterior predictions to structured activation manifolds and tracks belief dynamics between attractor regions.
- Even when a target feature is locally linearly decodable, representations can remain substantially curved and jointly nonlinear across belief parameters.
- Linear steering can cut across curved belief manifolds, causing unintended coupled shifts despite linear separability and readability.
- The analysis uses true activations from unfine-tuned base models in numerical settings, so broader natural-language applications remain open.
A Convergence to the input distribution
The model’s output distribution converges toward the distribution generating the input numbers as more samples appear in the prompt. The convergence is reached after roughly 100 numbers, although that sample count is insufficient to fully sketch a distribution with σ = 100.
- After about 100 numbers, the model’s logits adequately reproduce the underlying input distribution.
- Even 100 input numbers are insufficient to sketch a distribution with σ = 100, despite faithful output predictions.
- The model’s output distribution is described as converging toward the normal distribution underlying the input numbers.
- Figure 7 tracks convergence using KL divergence, entropy, and output distributions transitioning from a uniform integer distribution to the true normal distribution.
B Belief manifolds for various models
The same analysis reveals qualitatively similar structured manifolds across several foundational model families. Manifold resolution increases in deeper layers.
- The methodology is applied to Llama, LFM, and OLMo models using input time series and PCA of residual-stream representations.
- Across models, the representations show qualitatively similar structured manifolds, with increasing resolution toward deeper layers.
C Ideal observer
The model’s switching-time-series beliefs differ from an ideal Bayesian observer: early outputs transition toward a stationary distribution, while Llama-3.2 adapts faster than the ideal observer.
- Early outputs are far from outputs at t ≥100, which encode the stationary distribution according to pairwise Hellinger distances.
- The model does not conform to an online ideal Bayesian observer under standard assumptions.
- The model’s posterior predictive distribution is Student-t under a stationary i.i.d. Gaussian process with unknown mean and variance.
- Llama-3.2 converges much faster than the ideal observer on the switching-time-series trajectory.
D Meta-in-context learning
On sequences with repeated distribution switches, the model shows evidence of meta-in-context learning: its responses become progressively faster after each change.
- The model’s response to distribution changes becomes faster and faster across repeated switches.
E.1 Data and feature geometry
The paper distinguishes the geometry of the activation data manifold from the geometry of feature fields defined on it, and operationalizes those fields with linear probes.
- Data geometry: A data manifold can be curved while classes remain linearly separable along a coordinate direction.
- Field dimensionality: A rank-1 field has collinear probe vectors, whereas two non-parallel separating hyperplanes yield field dimensionality two.
- Field geometry: Linear field probes are families of local linear readouts whose varying directions reveal the geometry of feature fields over a data manifold.
- Probe construction: Multiclass probes learn C class logits with W ∈ R^C×d, while one-vs-rest probe sets train C binary classifiers.
- Probe construction: The study uses multiclass probes because one-vs-rest probes are less efficient for separating a specific class from all others.
- Intervention directions: Steering directions are computed from differences between point-probe vectors, optionally after centering multiclass rows across classes.
- Probe interpolation: The paper compares arithmetic, spherical geodesic, and Gram-matrix-based eigenbasis interpolation of probe vectors.
E.4 Mixture of manifolds
The paper tests whether mean and variance effects can be combined as independent additive components, but spline-based interpolation fails to reproduce the true two-parameter manifold geometry.
- E.4 Mixture of manifolds: The Linear Representation Hypothesis motivates modeling mean and variance effects as separate subspaces U and V around an anchor.
- E.4 Mixture of manifolds: The proposed decomposition treats the prototype surface as an approximately additive product manifold.
- E.4 Mixture of manifolds: Spline-based interpolation of mean and variance centroids fails to capture the true geometry of the (µ, σ) sheet.
- E.4 Mixture of manifolds: The failure may reflect nonlinear interactions between µ and σ, leaving other decompositions possible.
- Related visualizations: The broader experiments visualize trajectories and activations across models, input distributions, and geometric representations.