PaperlayerView exact revision on arXiv

Source-linked AI summary

When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task

Sai Sumedh R. Hindupur, Hadas Orgad, Thomas Fel, Demba Ba

arXiv:2609.37680v1cs.LGcs.AIcs.CL

TL;DR

Although low-dimensional manifolds have been observed in concept representations, how models use representation geometry in particular computations remains unclear. The paper analyzes number comparison in Qwen2.5-7B-Instruct and finds that linear number representations support the computation despite curved manifold structure, including in three-number comparisons.

  • Problem

    How models use representation geometry to perform specific computations remains unclear, including whether comparison manipulates nonlinear number manifolds or uses linear structure.

    Source: §1 · pp. 1–2

  • Method

    The paper gives a causal geometric account of number comparison in Qwen, tracing how the model represents and combines numbers and performs comparisons.

    Source: §1 · p. 2

  • Results

    Qwen uses linear number representations despite curved manifold structure, and the linear-representation comparison algorithm extends to three-number sequences.

    Source: §1 · pp. 2–3, §4 · p. 9

  • Takeaways & Limitations

    For ordered, non-periodic concepts, manifold structure can coexist with an underlying linear structure used in a computation.

    Source: §1 · p. 2, §4 · p. 9

    The analysis is restricted to number inputs and maximum computation; its claims concern ordered concepts that can be mapped to the number line without periodicity.

    Source: §4 · p. 10

Abstract

from arXiv · show

One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can tell us how models perform computations, and how to effectively intervene on them. While low dimensional manifolds have been observed for multiple concepts in the literature (e.g. numbers encoded on helices, days of the week on a circle, ...), with structure believed to reflect properties of data and tasks, the extent to which models rely on them for computation, and how they manipulate them, remains unclear. We characterize precisely the geometry of computation in a number-comparison task, as an abstraction of comparison for decision making, and how models utilize geometry in an elegant fashion to implement it. Specifically, we study the causal geometry of number comparison in Qwen2.5-7B-Instruct, a capable and widely studied open-weight model, and find Qwen largely uses linear representations of numbers despite the presence of curved geometry. To compare two numbers, the model first encodes each number along a vector and adds the two representations using attention and the residual connection, bringing them into a shared space in the residual stream. Then, the model uses MLP neurons to compare the pair of numbers on local regions in this shared space, which correspond to smaller intervals of input numbers, and combines these to obtain the position of the maximum. In fact, this reliance on linear representations for comparison also persists when the model compares three numbers. Our findings demonstrate that the manifold hypothesis can co-exist with linear representations: while concepts that are ordered may have manifold structure in representations, the model may use an underlying linear structure of the concept in certain computations.

1 INTRODUCTION

Although ordered concepts can have low-dimensional manifold structure, how models use that geometry in a specific computation remains unclear. The paper gives a causal account of number comparison in Qwen, finding that linear representations support the computation despite curved activations.

1 INTRODUCTION

  • The paper situates its question within algorithmic interpretability, which investigates variables and how models manipulate them, alongside mechanistic work focused largely on implementation-level circuits.It describes the work as contributing to algorithmic interpretability.Source: §1 · p. 2
  • The study asks which aspects of number representations models use for comparison, against a background of reports of nonlinear number manifolds and linear representations of ordered concepts.The introduction frames the unresolved issue as whether a computation manipulates nonlinear manifolds or uses simpler linear representations despite their presence.Source: §1 · pp. 1–2
  • Earlier comparison studies addressed circuits, behavioral performance, or probing, but did not examine causal representation geometry and how models use it to compare numbers.The paper presents its geometric account as causal and says it can predict effects of causal interventions on behavior.Source: §1 · p. 2
  • Qwen causally uses linear number representations for comparison despite underlying curved manifold structure, combining numbers additively before local comparisons are integrated into a global answer.The account describes an attention head and residual connection creating a shared space, where MLP neurons compare local regions corresponding to intervals of input numbers.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The algorithm extends to three-number sequences, showing that linear representations also support comparisons beyond pairs.The introduction identifies this as an extension to three-length sequences.Source: §1 · p. 3

2 PAIRWISE COMPARISON OF NUMBERS

The section traces how Qwen2.5-7B-Instruct compares numbers, from position-specific number directions through a shared representation to staged MLP comparisons. It also extends the comparison algorithm to three-number sequences.

2.1 ALGORITHM FOR PAIRWISE COMPARISONS

  • Each number is encoded along its own direction, then attention copies y1 to y2’s position and residual addition forms a shared representation where y1 > y2 is linearly decodable.The shared representation spans the two directions v1 and v2; MLP neurons compare within local regions before later neurons combine the comparisons.Source: §2.1 · pp. 3–4

2 PAIRWISE COMPARISON OF NUMBERS

  • Using clean-to-corrupt interchange interventions on 2,000 ordered pairs, the analysis tests causal patching and restricts its target to argmax-position computation.The ordered pairs comprise 1,000 randomly chosen pairs and their reflections; the model’s answer position, rather than the answer value, is the analysis target.Source: §2.1 · p. 4, §2.2 · p. 5

2.2 INDIVIDUAL NUMBER REPRESENTATIONS

  • Despite curved number geometry, causal direction u affects answers; attention head H14 copies y1 to y2’s position, and directions u, v1, v2 encode number magnitudes.The directions have causal effects measured by interchange intervention accuracy; H14 attends to y1’s position when copying its information.Source: §2.2 · pp. 5–6

2.3 SHARED REPRESENTATION ENABLES COMPARISON

  • In layer-14’s pre-MLP residual stream, v1 and v2 form a two-dimensional shared plane; two rank-one patches nearly match the plane patch’s IIA.The individual directions originate at different positions in the computation, yet their combined patch has a similar causal effect to patching their shared plane.Source: §2.3 · pp. 6–7

2.4 GLOBAL COMPARISON IS A COMBINATION OF LOCAL COMPARISONS

  • Freezing 12 layer-14/15 MLP neurons significantly reduces IIA from patching span(v1, v2), supporting a two-stage local-to-global comparison circuit.Layer-14 neurons perform localized comparisons; layer-15 neurons combine them into global comparators and write the answer position to a single direction.Source: Figure 4 · p. 6, §2.4 · p. 7
  • For three numbers, v1, v2, v3 encode the values, and local MLP comparisons produce running-maximum flags at y2 and y3 before the final argmax is read.The answer position is represented at the last token, and the flags and final answer representation causally affect model logits.Source: §2.3 · p. 7, Figure 5 · p. 8

3 MULTIPLE COMPARISONS: THE CASE WITH THREE NUMBERS

The three-number extension represents numbers linearly and compares each new number with those seen so far, then combines comparison flags to recover the maximum’s position. Evidence in Qwen2.5-7B-Instruct supports this computation while leaving parts of the final decoding mechanism unresolved.

3.1 ALGORITHM FOR MULTIPLE COMPARISONS

  • At each token, additive linear representations of the numbers seen so far feed local comparisons and a global comparator that stores whether the current number is the running maximum.The flag is α_t = I(y_t = max(y_1, . . . , y_t)), stored along direction d_t.Source: §3.1 · p. 8
  • A zero flag says the current number is not the maximum, but does not identify where an earlier maximum occurred; final attention combines the flags into an answer-position representation.The model also computes whether the first number is maximum at the y3 position.Source: §3.1 · pp. 8–9
  • The authors hypothesize that decoding scans flags backward for the first α_t = 1, while the full attention-head operation and answer decoding remain future work.The final-token attention head was empirically observed; the proposed backward scan is a hypothesis.Source: §3.1 · p. 9

3.2 EVIDENCE FOR THE THREE NUMBER COMPARISON ALGORITHM

  • At y2 and y3, separate computations produce causal flags for whether each number is the largest so far; a layer-20 representation combines them into the answer position.The two principal components of this representation have the same position-recovery score as the full residual stream at that position.Source: §3.2 · p. 9
  • At y3, the model builds a three-dimensional space from directions encoding all three numbers; layer-14 MLP neurons compare locally and form a direction encoding whether y3 is maximum.Patching the directions or their span affects outputs, while perturbing y2 has minimal effect in this computation.Source: §3.2 · p. 9
  • At y2, a separate pairwise computation compares y2 with y1; position-recovery tests find the y2 and y3 flags and the final two-dimensional answer subspace causal.The two principal components of the layer-20 answer representation match the full residual stream's position-recovery score.Source: §3.2 · p. 9

4 DISCUSSION AND LIMITATIONS

The paper finds that Qwen2.5-7B-Instruct uses linear number representations and additive mixing for maximum comparison despite curved manifold structure. It cautions that the evidence is scoped to ordered numerical inputs and establishes sufficiency rather than necessity.

DISCUSSION

  • Qwen2.5-7B-Instruct combines linear number representations additively, compares local regions with MLP neurons, and aggregates them into a global maximum comparison.The paper reports that this algorithm also extends to three-number comparisons.Source: §4 · p. 9
  • Curved representation geometry can coexist with simpler linear structures that are sufficient to affect computation; detailed geometry alone need not explain how concepts are used.The authors present this as nuance to the view that detailed representation geometry necessarily clarifies computation.Source: §4 · p. 9

LIMITATIONS

  • The analysis covers numbers and maximum computation; its claims about ordered concepts apply to concepts mappable to a non-periodic number line.The authors also rely on DAS for causal directions in some cases and note that their evidence shows sufficiency, not necessity; other information-processing paths may remain.Source: §4 · p. 10
  • Patching can show that identified directions and subspaces modify outputs, but removing them may not change outputs because other processing paths may solve the task.The authors describe this distinction as sufficiency without a claim of necessity.Source: §4 · p. 10

A.1 MODEL, TASK AND PROMPTS

The experiments use frozen Qwen2.5-7B-Instruct on raw completion prompts asking for the maximum of two or three numbers. Tokenization and sampling are controlled so operand positions and number ranges are specified for each regime.

A.1 MODEL, TASK AND PROMPTS

  • 28 layers, 28 attention heads, and SwiGLU MLPs define the frozen Qwen2.5-7B-Instruct model, run in float16 on one NVIDIA A100 40GB GPU.The model width is 3584 and the MLP width is 18944.Source: §A.1 · p. 14
  • A raw completion prompt uses a one-shot exemplar to fix the answer format for maximum questions over two or three operands.The two-number prompt uses “The maximum of 12 and 4 is 12”; the three-number exemplar uses 12, 437, and 5.Source: §A.1 · p. 14
  • Operands are digit-tokenized, and activations are read at each number’s last token, where the whole number is first visible under causal attention.Within each experiment, operands share a digit count and sampled operands have distinct leading digits.Source: §A.1 · p. 14
  • Two-digit operands are sampled from [10, 100) with minimum gap 10; three-digit operands are from [100, 1000) with minimum gap 100, solely to replicate number-representation analysis.The three-digit regime is not used for the comparison analysis.Source: §A.1 · p. 14

A.2 COUNTERFACTUAL PAIRS

The counterfactual pairs change which operand wins while preserving the comparison setup; the three-number cases vary operand order and corruption position to examine the comparison.

A.2 COUNTERFACTUAL PAIRS

  • For K = 2, each triple satisfies a > b + g > r + 2g, and replacing a with r changes the winning operand while keeping b fixed.Patching a clean state into the corrupted run tests whether the model treats r as the maximum, rather than whether it outputs the clean value a.Source: §A.2 · p. 14
  • The K = 2 triples are evaluated in both value arrangements, y1 > y2 and y2 > y1.Intervention scores compare t(r) against t(b), so a patch that merely re-inserts a cannot raise the score by reproducing the clean value.Source: §A.2 · p. 14
  • For K = 3, quadruples satisfy a > b > c > r with consecutive gaps above g; replacing a with r makes the clean runner-up the corrupted winner.The head-identification groups vary the runner-up or maximum position one at a time; other analyses use three corruption cases patched at y3's position.Source: §A.2 · p. 14

A.3 POPULATIONS AND SPLITS

The analyses use shared, disjoint fitting and evaluation examples, with separate clean-prompt clouds for geometry and probes; evaluation procedures vary by operand count.

A.3 POPULATIONS AND SPLITS

  • Intervention scores use 400 evaluation examples, while directions and neuron rankings are fitted on 128 disjoint examples from the same deduplicated pool.For K = 3, 1,341 of 1,400 sampled quadruples pass the four-order correctness filter, and the first 528 are used.Source: §A.3 · p. 14
  • Geometry and probe analyses use separate clouds of clean prompts, and the two-number clouds include each unordered pair in both orders.No filter is applied for K = 2; in the two-digit regime the model answers every ordered pair of distinct two-digit numbers correctly.Source: §A.3 · p. 14
  • Behavioral accuracy uses all 8,010 ordered pairs of distinct two-digit numbers and 4,000 three-digit pairs drawn with replacement.The operand-count sweep uses 200 prompts per length and Wilson score intervals; Table 1 summarizes sample sizes and hyperparameters.Source: §A.3 · p. 15, Table 1 · p. 15

B METHODS

The methods measure causal effects by replacing activations from a clean run into a corrupted run at selected model sites, using both full-rank and rank-restricted patches.

B METHODS

  • Interchange interventions replace a component of the corrupted-run activation at one site with its value from the clean run.The method uses x for the corrupted activation, xclean for the clean activation, and P for an orthonormal basis.Source: §B.1 · p. 15
  • Rank-restricted patches are compared with full-rank patches at the same site as a reference, not as an upper bound, because they can exceed it.The full-rank restoration uses P = I.Source: §B.1 · p. 15
  • Patch sites include residual-stream positions, attention-head contributions, pre-MLP residuals, and coordinates of the post-SwiGLU vector feeding the down-projection.The pre-MLP residual can be patched through the attention output, giving the same result as an end-of-layer patch at full rank.Source: §B.1 · p. 15

B.2 METRICS

The metrics distinguish recovery of the patched operand’s position in the logits from whether the model’s next token or full decoded answer is correct.

B.2 METRICS

  • Position recovery is normalized restoration of the logit difference, computed per example and averaged across examples.The logit difference contrasts t(r) and t(b) at the last token, using the same token pair on clean, corrupted, and patched runs.Source: §B.2 · p. 15
  • The denominator averages 10–13 logits and enables comparisons across sites and cases, but position recovery need not lie in [0, 1].When IIA is near zero in three-number cases y1 and y2, position recovery measures logit-difference movement rather than a change of answer.Source: §B.2 · p. 16
  • IIA scores whether the model’s next token is t(r), while ablation accuracy checks the greedily decoded full answer against the true maximum.IIA uses the full vocabulary at the last token; full-number accuracy decodes one token beyond the operands’ digit count.Source: §B.2 · p. 16

B.3 DIRECTIONS

This appendix describes how number directions are identified and evaluated, how causal tracing localizes computation, and how MLP neurons are selected and tested.

B.3 DIRECTIONS

  • DAS fits rank-k directions for a causal variable, with directions signed so their component rises with the carried number.A 128-example fit is used after 16 examples yielded held-out IIA 0.688; a second initialization gives held-out IIA 0.94 for two-digit operands.Source: §B.3 · p. 16
  • Linear decodability uses ridge regression on log numbers and a ridge classifier for comparison, with held-out R2 or accuracy; causal directions and principal components are also compared.Multi-dimensional patches use the Gram–Schmidt-orthogonalized span of the directions; its score is a lower bound for a rank-k patch at that site.Source: §B.3 · p. 16

B.4 LOCALIZATION

  • Causal tracing restores clean residuals across layers and token positions; head interchanges use co-patches, and layer-20 readout localization raises mean position recovery from 0.09 to 0.55.The whole attention sublayer serves as a reference when individual layer-20 heads are patched.Source: §B.4 · pp. 16–17

B.5 NEURONS

  • The shared neuron set is the intersection of the top-20 attribution rankings for the two perturbation cases: 12 neurons, six in each MLP.The three-number analysis reuses this set unchanged.Source: §B.5 · p. 17
  • Neuron rankings use attribution patching, then evaluate the top-k neurons by freezing them on evaluation examples against two random k-neuron sets.Freezing pins selected outputs to their corrupted-run values; injection writes selected clean MLP outputs into the corrupted run.Source: §B.5 · p. 17

B.6 DOSE-RESPONSE SWEEPS, RECEPTIVE FIELDS AND CONNECTIVITY

The appendix details the dose-response sweep, receptive-field measurements, connectivity metrics, and a specificity control for the three-number directions.

B.6 DOSE-RESPONSE SWEEPS, RECEPTIVE FIELDS AND CONNECTIVITY

  • The dose-response sweep sets both pre-MLP layer-14 residual coordinates at y2 to chosen-number means (±2 cloud window) on a 9 × 9 grid, scoring first-operand argmaxes among answers naming either operand, with diagonal masked.The pre-MLP site is used because a per-number mean explains R2 = 0.87 of the v1 coordinate there, versus 0.35 at the layer end.Source: §B.6 · p. 17
  • A receptive field is a neuron's post-SwiGLU activation over a prompt grid; three-number fields are pairwise marginals averaged over the third number and divided by their own peak.At the y2 position, y3 is held fixed at 55.Source: §B.6 · p. 17
  • Connectivity is measured with causal activation shifts between MLP neurons, gate-projection virtual weights, and alignment of neuron write directions with learned subspaces.The alignment is compared with the distribution over all 18,944 neurons in the layer.Source: §B.6 · p. 17
  • As a specificity control, each three-number direction is evaluated without refitting when its carried number does not change, where it should have no effect.Source: §B.6 · p. 17

C FURTHER EXPERIMENTAL RESULTS

The appendix presents further experiments in the order of the comparison algorithm, using held-out counterfactuals unless a figure caption specifies otherwise. It covers behavior, localization, number representations, comparator neurons, three-number comparison, and whole-number intervention scoring.

C.1 BEHAVIOUR AND WHAT AN INTERVENTION CHANGES

C.2 WHERE THE COMPUTATION HAPPENS

C.3 INDIVIDUAL NUMBER REPRESENTATIONS

  • The individual-representation analyses compare direction u with principal components, across earlier layers and three-digit operands, and examine y2 representations and the L14.H14 output.Source: §C.3 · p. 21, Figure 12 · p. 21, Figure 13 · p. 21

C.4 THE SHARED REPRESENTATION

C.5 THE COMPARATOR NEURONS

C.6 THREE NUMBERS

C.7 IIA ON THE WHOLE GENERATED NUMBER

  • Whole-number scoring shows that only full-rank patches with y2 perturbed lose IIA on the whole generated number; full-rank patches mostly generate a number other than r that starts with its leading digit.Source: Figure 37 · p. 31, Figure 36 · p. 31, Table 4 · p. 32
Loading 2609.37680v1…