PaperlayerView exact revision on arXiv

Source-linked AI summary

When Models Manipulate Manifolds: The Geometry of a Counting Task

Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, Joshua Batson

arXiv:2601.04480v1cs.LG

TL;DR

The paper asks how a language model perceives and performs fixed-width linebreaking despite receiving only token sequences. It mechanistically studies Claude 3.5 Haiku using sparse features and geometric analyses, finding curved count manifolds, attention-based boundary estimation, and linearly separable newline decisions. These results show that geometric and feature-based views jointly clarify distributed computation in a natural language-model behavior.

  • Problem

    The paper investigates how language models acquire perceptual mechanisms for tracking document position and line width when their input is only a sequence of tokens.

  • Method

    The authors analyze Claude 3.5 Haiku’s linebreaking behavior using synthetic fixed-width text, dictionary features, attribution graphs, geometric manifold analysis, and targeted interventions.

  • Results

    Character counts are represented on curved one-dimensional manifolds, while attention heads manipulate these representations to estimate line-boundary distance and support newline prediction.

  • Takeaways & Limitations

    Feature and geometric perspectives provide complementary descriptions, with geometry simplifying interpretation of scalar representations and computations in this natural task.

  • Takeaways & Limitations

    Geometric manifold analysis is expensive in researcher time and becomes difficult for complex concepts that are not straightforward to parameterize.

Abstract

from arXiv · show

Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.

1 Introduction

The paper investigates how Claude 3.5 Haiku performs fixed-width linebreaking from token sequences, revealing sparse feature and geometric mechanisms for representing counts, detecting boundaries, and deciding when to break lines.

  • Motivation: Claude 3.5 Haiku must learn text perception from token sequences to handle spatial tasks such as fixed-width linebreaking.The task requires tracking character position and line-width constraints that are not explicit in the token input.
  • Representations: The model represents token lengths, current position, line width, and remaining characters on curved one-dimensional manifolds embedded in low-dimensional residual-stream subspaces.Sparse discrete features tile these manifolds and provide approximate local coordinates for different counts.
  • Prediction: The newline decision combines estimated remaining space with the predicted next-word length by placing the counts on near-orthogonal subspaces.This arrangement makes correct linebreak predictions linearly separable.
  • Distributed computation: Many attention heads cooperatively construct the curved counting geometry because an individual head cannot produce sufficient output variance.The distributed computation lets separate components contribute pieces of the overall representation.
  • Interpretability: The geometric perspective can simplify feature-based interpretation, but parameterizing manifolds is expensive and difficult for complex concepts.The paper presents geometry as especially workable for known continuous variables such as integer counts.

2 Representing Character Count

The model represents character count through complementary sparse-feature, subspace, manifold, and probe views rather than a single linear coordinate. These representations form a curved low-dimensional geometry that supports causal manipulation and explains characteristic ringing patterns.

  • A linear probe after layer 1 predicts character count with R2 = 0.985, but this does not imply a single-line representation.
  • Character count is analyzed through sparse features, a low-dimensional subspace, a continuous one-dimensional manifold, and logistic probes.Each perspective supports a different analysis: orientation, causal intervention, geometric construction, or attention-matrix analysis.
  • 2.1 Character Count Features: Ten sparse features activate over character-count ranges, with neighboring features typically active together and receptive fields widening at larger counts.Their profiles suggest local coordinates for a curved continuous representation and resemble place-cell-like coding.
  • 2.2 The Model Represents Character Count on a Continuous Manifold: The top 6 principal components capture 95% of character-count variance, forming a twisting curve whose feature reconstruction closely approximates the mean residual-stream trajectory.The feature vectors discretize the curve, while interpolation between neighboring features provides a smoother approximation.
  • 2.5 Rippled Representations are Optimal: The representation exhibits ringing: probe responses and vector similarities rebound away from their maxima, including positive and negative similarities at different distances.This pattern is presented as a consequence of embedding a rippled manifold in low dimensions.
  • 2.5 Rippled Representations are Optimal: A physical simulation likewise produces rippling and ringing when many points are packed into six dimensions with local attraction and global repulsion.Reducing the attractive-zone width or embedding dimension increases curvature and ringing, though the simulation can reach local minima.

3 Sensing the Line Boundary

Haiku detects approaching line boundaries by comparing character count with line width through geometrically transformed counting representations. Multiple boundary heads combine offset responses to estimate characters remaining with higher resolution.

  • Boundary detection compares current character count with the overall line-width constraint to estimate characters remaining.
  • A boundary head’s QK matrix twists the character-count manifold so count i aligns with line width k = i + ϵ, producing strong attention near the boundary.
  • The boundary-head transformation preserves a specific offset and generalizes to arbitrary line widths and separator types.
  • Multiple boundary heads use distinct offsets and overlapping response curves to tile characters-remaining values.
  • Combining heads improves resolution across the full relevant range because individual heads vary most strongly over different character-remaining intervals.
  • Ablation and activation substitution experiments causally test the characters-remaining subspace and its effect on newline prediction.

4 Predicting the Newline

The final linebreak decision combines boundary information with the predicted next-token length. Orthogonal count representations make the decision linearly separable, while attribution analysis identifies features that promote or suppress newline prediction.

  • The final decision combines estimated characters remaining with the predicted next-word length to determine whether the word fits.
  • Late break-predictor features activate when the next token would exceed the line width, increasing newline probability and decreasing the competing word’s probability.
  • Break-suppressor features activate when the next token would just barely fit, reducing newline probability.
  • Characters remaining and next-token length occupy near-orthogonal subspaces, making the break condition i −j ≥0 a separating hyperplane.
  • A PCA-derived separating hyperplane achieves an AUC of 0.91 for predicting whether the next token should be a newline.
  • Haiku does not appear to redirect all probability mass from overlong words to the newline at line endings.

5 A Distributed Character Counting Algorithm

Haiku computes character counts through distributed attention mechanisms that accumulate token-length information across layers. Early heads provide coarse estimates, while later heads refine curved counting representations.

  • Many attention heads across multiple layers cooperatively compute increasingly accurate estimates of character count.
  • Layer 0 heads write near-rays whose sum forms a curved manifold, while Layer 1 heads output curves that further refine it.
  • The five key Layer 0 heads achieve R2 = 0.93 for character-count prediction, compared with R2 = 0.97 using 11 heads across the first two layers.
  • Token embeddings organize character lengths 1–14 in a circular pattern with an oscillating component, alongside overlapping short-, medium-, and long-word features.
  • The analysis focuses on attention because its outputs affect the character-count representation 4× more than MLPs, and on the first two layers for tractable refinement.
  • Individual heads use QK circuits to select newline-centered context and OV circuits to convert token information into heuristic character-count estimates.
  • The model appears to count characters between adjacent newlines with a distributed algorithm, but how it aggregates global line width remains unresolved.

6 Visual Illusions

Contextual character sequences can hijack Haiku’s counting mechanism and alter line-length judgments. The strongest effects occur when inserted strings redirect attention patterns used for newline-based counting.

  • The study asks whether contextual cues can create language-model analogues of visual illusions in line-length perception.
  • The @@ delimiter can redirect counting heads from previous-newline attention toward the inserted string.
  • Inserting @@ changes the predicted next token and disrupts newline prediction without changing the line length.
  • Most inserted two-character sequences have moderate effects, while code-related or delimiter-like pairs can substantially alter newline probability.
  • The impact on newline prediction is associated with how strongly inserted tokens modulate the relevant attention patterns.
  • The illusion also generalizes to explicit length comparisons, where contextual insertion can reverse which alternative is judged shorter.
  • The paper treats the analogy to human visual illusions as suggestive rather than a direct equivalence, emphasizing contextual cues and learned priors.

7 Related Work

This work connects linebreaking and character-count representations to prior studies of positional mechanisms, number geometry, feature manifolds, and biological perception. It extends token-position analyses toward character-based position schemes and finds parallels with place and boundary cells.

  • Methodological context: The study uses attribution graphs with QK attributions built on crosscoders to reverse-engineer linebreaking mechanisms.The authors frame the work as combining empirical model biology with low-level neural-network reverse engineering.
  • Position: Prior positional interpretability work largely studies token position, whereas this work examines non-token-based position schemes based on character counts.Earlier studies identified periodic structure in MLP neurons, SAE features, and learned position embeddings for absolute token position.
  • Geometry and Feature Manifolds: Research on number geometry has found periodic representations, helix-like number arrangements, and matrix-multiplication-based addition algorithms in language models and toy transformers.The related literature spans geometric representations of numbers in toy models and arithmetic in LLMs.
  • Geometry and Feature Manifolds: Feature manifolds provide a formal lens in which cosine similarity can encode intrinsic geometry, including structured manifolds with ripples and dilation.These observations create a methodological challenge because data may have different geometric structures.
  • Biological Analogues: Character-count features parallel place cells on a one-dimensional track and boundary cells, while their dilation resembles dilation in biological number representations.The analogy concerns both geometric and algorithmic patterns in the model’s representations.

8 Discussion

The discussion presents linebreaking as a naturalistic perceptual behavior requiring latent character-count computations and semantic integration, while arguing that feature and geometric views jointly clarify model mechanisms. It also emphasizes that current explanations remain incomplete and need more automated structural-discovery methods.

  • Discussion: Linebreaking requires representing scalar positions in character-count units that are absent from the token input and integrating them with semantic word-prediction circuits.The task involves predicting newline placement while selecting the next semantic token.
  • Naturalistic Behavior and Sensory Processing: Perceptual tasks that occur naturally in pretraining and are easy to supervise can provide especially clear targets for deep mechanistic case studies.The discussion recommends behaviors with consistent model performance because they are more likely to reveal crisp mechanisms.
  • The Utility of Geometry: Geometry reveals elegant structure in counting manifolds and attention-head transformations, but the final breaking decision and semantic operations remain only partly characterized.The authors specifically note unresolved complexity for multitoken outputs and rely mainly on feature views for semantic operations.
  • Limitations: The analysis leaves open how the model handles uncertainty, variable line widths, multiple plausible token lengths, multitoken words, and special formatting cases.Examples include LaTeX footnotes and markdown links.
  • Unsupervised Discovery: Unsupervised sparse features were crucial for discovering what to study and where to look, after probing and patching alone proved ineffective.The authors initially did not know to distinguish line width from character count or expect line width to be represented only on newline tokens.
  • Feature-Manifold Duality: Feature and geometric descriptions are dual views of the same representation, but geometric structure can reduce the complexity produced by a discrete feature decomposition.The feature description can be reconstructed from identified feature activities, while attention actions can be expressed through virtual weights.
  • A Call for Methodology: The geometric analysis serves as an existence proof rather than a general recipe, motivating unsupervised methods for automatically detecting simpler structures.The authors seek tools that reduce interpretation burden across other computations.

C Haiku Task Performance

The authors evaluate Haiku’s ability to predict line breaks across synthetically wrapped prose with varied fixed line widths. Haiku adapts to each tested width and predicts newline positions with high probability by the third line.

  • Haiku Task Performance: Haiku predicts newlines at the correct positions with high probability by the third line for every tested line length k.The evaluation uses 200 synthetically wrapped prose sequences with k = 20, 40, . . . , 140.
  • Haiku Task Performance: The evaluation measures mean log probability of newline and top-1 accuracy as a function of the number of context lines.The figure reports task performance across increasing context length.

D Feature Splitting and Universality

Character-count and line-width features form structured, partly canonical tilings of feature manifolds across dictionary sizes. The results also reveal feature splitting, ringing geometry, and a dynamical explanation for ripples in packed low-dimensional representations.

  • Feature Splitting and Universality: Dictionaries of different sizes learn character-count features with very similar receptive fields, suggesting a canonical sparse parameterization.The authors hypothesize that boundary constraints anchor feature development near the start of the line.
  • Feature Splitting and Universality: Decoder directions remain geometrically consistent across dictionary sizes and exhibit characteristic ringing.Pairwise comparisons across dictionaries support consistency of the learned geometry.
  • Feature Splitting and Universality: Three features activating over roughly 20–45 characters split by line width: one is width-general, one favors long lines, and one favors proximity to the boundary.The features share an activation interval but differ in their dependence on line width.
  • Feature Splitting and Universality: The observed cross-dictionary tiling argues against pathological dictionary behavior in which increasing size merely creates finer feature tilings.The comparison is presented as evidence that such behavior does not occur in this setting.
  • Feature Splitting and Universality: Line-width features tile their space similarly to character-count features.This extends the observed feature-family structure beyond character counts.
  • Dynamical System Model: A six-dimensional hypersphere simulation with local attraction and global repulsion produces rippling and ringing when many vectors are packed into few dimensions.The simulation uses 100 points, attractive forces toward six neighbors on each side, and circular topology to avoid boundary conditions.

G Analytic Construction of Ringing and Fourier Modes

The paper analyzes ringing in low-dimensional character-count manifolds through an analytic Fourier construction. Fourier components nearly match PCA in explained variance, while attention heads geometrically transform probe representations through alignment and offsets.

  • Analytic construction: A discretized circle embedded in lower dimensions develops ringing when its similarity matrix is approximated by truncating the top eigenvectors.The construction uses a circulant similarity matrix and low-rank projection to model the character-count geometry.
  • Boundary geometry: A boundary head applies a linear QK transformation that slides one character-count curve along itself, pairing counts separated by a target offset.This operation enables a count to align with a slightly larger count when the line boundary is near.
  • Fourier modes: The Fourier decomposition is close to optimal for the character-count curve, explaining at most 10% less variance than an equal number of PCA components.PCA remains optimal for variance capture because the Fourier basis does not account for dilation.
  • Layerwise sharpening: Across layers, character-count representations become more peaked, improving count estimation while producing higher curvature and more pronounced ringing.The progression is visible in cosine-similarity cross-sections across layers 0 through 3.
  • Head transformations: Different heads preserve alignment or shift representations left or right, producing distinct geometric transformations visible in cosine similarities, SVDs, and PCA projections.The heads therefore access and manipulate the probe space in specialized ways.

I Break Predictor Features

Break prediction combines character-count representations with the anticipated length of the next token. Separate feature families increase or suppress newline prediction depending on whether that token would exceed the remaining line capacity.

  • Boundary detection: Boundary detector features at about one-third model depth estimate line-boundary proximity without considering the next token’s length.These features operate before later token-conditioned break decisions.
  • Break prediction: Later features activate when the most likely next token is longer than the characters remaining, increasing newline prediction.The condition corresponds to points below the red diagonal in the reported visualization.
  • Break suppression: Other features suppress newline prediction when the predicted next token is shorter than the characters remaining.These are the converse of the break-prediction features.
  • Logit effects: Break predictors and suppressors also affect other token logits by favoring tokens short enough to fit when the next-token estimate is wrong.The newline is the top effect, but longer tokens are systematically suppressed as well.
  • Token-length features: Layer 0 contains overlapping and non-exhaustive features that activate according to individual-token character counts.Some common tokens activate multiple length features, while others activate none because length information may be absorbed into token-specific features.

K The Mechanics of Head Specialization

Attention heads construct character-count estimates by using previous newlines as sinks and accumulating token-length information. Heads specialize in offsets, while later heads refine and sharpen the resulting representations.

  • Toy counting mechanism: A toy head places default attention on the previous newline and distributes attention across subsequent tokens, whose output magnitudes scale with token character counts.The construction yields a representation whose total ray length is proportional to the line’s character count.
  • Head specialization: Actual Layer 0 heads use newlines as attention sinks but specialize in different offsets across prompts and line widths.Their attention patterns are measured by tokens since the newline and by individual token character length.
  • OV aggregation: Heads also bias attention toward longer tokens, so their OV responses adjust character-count estimates according to the lengths of attended tokens.The output combines a newline-relative character offset with a token-length-dependent adjustment.
  • Example head: L0H1 writes short-range count features when attending to the newline, defaults toward a larger count when it does not, and interpolates when attention is mixed.Short and long attended tokens shift the estimate in opposite directions.
  • Layerwise refinement: Layer 1 heads combine token character lengths with Layer 0’s initial line-length estimate to create a more refined count.These repeated computations sharpen the representations.

M Full Layer 0 Attention Results

The paper extends its analysis from linebreaking to early line endings and structured tables. It finds specialized representations for remaining line capacity and for table row and column indices, with related geometric signatures.

  • Early linebreaks: Features sensitive to early-ended lines encode how far the character count falls below the line width and help predict the token following a linebreak.These features are less useful for linebreaking itself but support post-break prediction.
  • Interpretation: The roles of these feature families are not apparent from quickly inspecting dataset examples, but careful analysis reveals interpretable behavior.The paper emphasizes mechanistic analysis over surface-level labeling of them as generic newline features.
  • Table representations: Synthetic markdown tables expose feature families specialized to particular separator-token rows and columns.The dataset contains 20 markdown tables arranged by location in the table.
  • Geometric signatures: Larger-table experiments show row- and column-index representations with ringing in probe similarities and a characteristic baseball-seam shape in PCA space.These geometric patterns resemble those found in character-count representations.
Loading 2601.04480v1…