Concept Subspaces Compute Beyond the Logit Lens: A Weights-Only Test for Locating Representations Upstream of Readout

• Language models express reasoning concepts in prose, code, and mathematical notation, raising whether concept identity persists across forms and how recovered subspaces overlap output readout.Concept-direction research motivates these questions (Park et al., 2024; Arditi et al., 2024; Turner et al., 2024).Source: §1 · p. 2
• FARS is extracted by concept-centroid PCA over eighteen reasoning concepts in six surface forms, averaging within concept to suppress the form confound.The rank is k = 10, an empirical operating point where marginal gains plateau, not an intrinsic-dimensionality claim.Source: §1 · p. 2
• The readout-orthogonality test compares a probed basis with the top-k right singular vectors of the unembedding matrix using model weights.Positive and depth-matched controls test whether the diagnostic registers both low and high readout loading.Source: §1 · p. 2
• Final-layer PCA exceeds FARS in 25/26 models, and a depth-matched same-layer next-token control does so in 25/25 cases.These comparisons calibrate the geometric diagnostic against output-oriented subspaces.Source: §1 · p. 2
• Four activation-derived concept estimators carry 0.38–0.80% mean readout energy, while re-extracted FARS achieves 62–100% cross-format retrieval on 24 generative models.The latter tests transfer of the extraction procedure, not transport of a fixed basis.Source: §1 · p. 2
• Intervention effects are model dependent and substantially weaker than full replacement, so geometric separation and intervention sensitivity do not establish causal sufficiency.The intervention comparison includes random, norm-matched random, full-vector, and no-intervention controls.Source: §1 · p. 2

• Concept-direction methods include labelled concept vectors, truth directions, steering interventions, learned subspace interventions, erasure, and sparse autoencoder features.The paper uses centroid PCA as a lightweight estimator and evaluates its readout geometry alongside alternative constructions.Source: §2 · p. 2
• Tuned lenses, function-vector steering, logit-lens accessibility, and Patchscopes address intermediate readout, while this diagnostic adds positive controls and a depth-matched linear reference.Low overlap with the specified span does not exclude lower singular directions, learned decoders, or subsequent nonlinear processing.Source: §2 · p. 2
• Cross-model alignment work motivates testing aligned concept centroids and their relations, which this paper examines on a fixed inventory.The paper frames this as a test of the cross-architecture representation question.Source: §2 · p. 2
• Multilingual representation studies address several distinct questions, but collectively do not establish invariance across prose, code, and mathematical notation.The benchmarks hold concept identity fixed while varying forms, without claiming exhaustive priority over cross-format datasets.Source: §2 · pp. 2–3
• Causal intervention methods can localise circuits, but intervention effects need not recover the naturally used mechanism; the geometric control addresses readout adjacency, not that broader identification problem.The paper distinguishes decodability from intervention efficacy.Source: §2 · p. 3

• TriForm contains 324 programmatically generated stimuli: 18 concepts, three instances, and six forms spanning prose, code, mathematical notation, and step-by-step structure.The forms vary linguistic, symbolic, and structural presentation while keeping reasoning content exactly constant.Source: §3 · p. 3
• The model testbed includes 26 LLMs from four architecture families plus reasoning-tuned variants, ranging from 0.4B to 70B parameters.RoBERTa-large is excluded from readout analyses because form-RSA exceeds concept-RSA throughout, leaving the test prerequisite unmet.Source: §3 · p. 3
• FARS is extracted by averaging each concept’s activations across forms and instances, then applying PCA to the 18 centroids; all estimators are rank-matched at k=10.The setup also uses permutation RSA, cross-form ridge probing, subspace patching, ablation, amplification, and intervention controls.Source: §3 · p. 3

4 Extracting FARS

• FARS is the top-k principal subspace of the mean-centred concept-centroid matrix, selected at the concept-RSA peak within the 30–75% depth band.The reasoning-position analysis instead selects by centroid variance, a distinct extraction setting.Source: §4 · p. 3
• A shuffled-centroid null trails FARS by 17–29σ in concept-RSA, while multi-class LDA matches it within ∆≤0.02.These comparisons support interpreting the subspace as concept-carrying rather than a variance artefact.Source: §4 · p. 3
• FARS affects reasoning performance in several models, but argmax preservation alone is less informative because random-subspace patches also preserve it.The paper therefore tests readout overlap with positive controls and examines transfer, rank and layer sensitivity, and reasoning-model analysis boundaries.Source: §4 · pp. 3–4
• Under 10-d cross-form patching, FARS ablation costs −20, −28, −17, and −10 pp on GSM8K for four named models relative to matched-rank random.Llama-8B requires amplification c⋆=2 with a −22 pp change; Llama-70B is inert at c≤2.Source: Table 1 · p. 4

4.1 What a concept-subspace intervention can change

• Intervention tests use four models, three seeds, and 120 directed concept pairs per seed, comparing FARS with random, norm-matched random, full replacement, and no intervention.Bases and model layers are fixed across seeds.Source: §4.1 · p. 4
• Last-token FARS patching produces seed-mean source-directed rates of 10.8%, 3.3%, 0.8%, and 0.8% across four models, versus 19.7–47.8% for full replacement.Random controls and no intervention yield zero events; the effects are not reliable substitution for the full state.Source: §4.1 · p. 4, Figure 2 · p. 4

5 Readout-Orthogonality Test and Method Validation

• The test compares a probed basis with the top-k right singular vectors of WU, using the model’s weights rather than new model runs.The study evaluates rank-matched estimators and controls across 26 models; the raw test uses WU, while the depth-matched test uses a fitted translator.Source: §5 · pp. 4–5
• 3.56% for final-layer PCA versus 0.80% for FARS in the top-10 readout span; final-layer PCA is more readout-adjacent in 25/26 models.The exception, R1-Distill-Qwen-7B, has raw FARS loading of 5.78%.Source: §5 · p. 5
• 5.88% for same-layer next-token PCA versus 0.46% for FARS: the depth-matched comparison separates them by 13× in all 25 models.A fitted linear translator pulls the readout back to the FARS layer; concept estimators remain near the Haar floor.Source: §5 · p. 5
• In this dataset, form-dominant subspaces have greater top-10 readout loading than concept-dominant ones: FormPCA 1.26% and FullPCA 1.04%, versus FARS 0.80%, LDA 0.41%, and probe weights 0.38%.Source: §5 · p. 5
• Difference-of-means and FARS yield the same top-k subspace for the same class means, corpus, layer, and balanced pairwise construction.Their Grassmann distance is below 10^-7 on every model; this equivalence does not extend to arbitrary steering directions.Source: §5 · p. 6
• Published refusal and truth directions have low loading at the FARS block but higher loading when extracted at the final block.At the FARS block, their median loadings are 0.44% and 0.20%; at the final block they are 1.98% and 4.09%, respectively.Source: §5 · p. 6
• Vocabulary-derived concepts load 8.43% in the readout span versus 0.80% for FARS, despite only 0.56% overlap between their subspaces.The mean per-model loading ratio is 16.9×; the vocabulary-derived direction has greater loading in all 26 models.Source: §5 · p. 6
• The tested concept-feature subset of one pretrained residual-stream SAE loads 1.06%, close to FARS at 0.96%, while the full dictionary loads 3.93%.This comparison covers one model.Source: §5 · p. 6

6 Cross-Inventory Transfer: Novelty Benchmark

• 62–100% top-1 cross-format retrieval on 24 generative models, 6–57 percentage points above matched-rank Haar baselines on every tested model.The encoder DeBERTa-v3 reaches 32%; retrieval compares stimuli with centroids from the other five forms.Source: §6 · p. 6
• 22.8–40.0% frozen TriForm-basis retrieval across three audited models is above sampled random means of 20.2–35.3% but below full-space retrieval of 48.3–90.6%.This is partial transfer, not evidence for a universal fixed basis.Source: §6 · p. 7
• Re-extraction on Novelty reproduces cross-format structure, but the result supports transfer across two inventories, not every possible concept set.Novelty uses ten concepts from five domains absent from TriForm, each rendered in the same six surface forms; k=10 is compared with a matched-rank random baseline.Source: §6 · pp. 6–7

7 Dimension Choice and Layer Localization

• k=10 captures about 90% of full-rank centroid variance and cross-format retrieval, with marginal gains of at most 10 percentage points beyond it.The sweep covers k=1–17 across dense, state-space, and encoder-MLM models.Source: §7 · p. 7
• Retrieval peaks at depth fractions of 0.29–0.97 across models, so no universal middle-layer location is established.Peak selection favors higher retrieval; peak-versus-final comparisons show lower final-layer retrieval and generally smaller distance to the readout span.Source: §7 · p. 7
• The CoT-tail subspace carries 58–61% top-1 retrieval, 34–39 percentage points above a matched-rank Haar-random baseline.The table also reports nonzero, sub-ceiling base-to-distilled distances and large aggregate distances between last-token and CoT-tail subspaces.Source: Table 3 · p. 8
• Across 27 extracted models, X-FARS accounts for 97.6% of cross-model centroid variance, 36.8 standard deviations above a random-projection control at 61.7±1.0%.The shared frame supports analogies, preserves pairwise differences, and enables training-free cross-architecture retrieval.Source: §7 · p. 8

8 Two Concept Subspaces in Reasoning-Tuned Models

• The CoT-tail analysis reads the last position generated under a 512-token budget in two open-weight R1-Distill models, their base counterparts, and QwQ.The reported 64–88% budget-hit rates come from a different three-model cohort and concern reasoning-stream length, not its end.Source: §8 · p. 8
• 56–57% of non-English prompts yield English traces, and regrouping by generated form lowers cross-format score from 30.6–44.4% to 26.9–28.1%.The regrouped score remains 3.3–5.1× a matched-rank Haar control; this comparison uses two models and a fresh generation set.Source: §8 · p. 8
• Position, layer choice, and output format remain potential contributors, so these observations do not isolate the computation responsible for reasoning.QwQ-32B has aggregate distance dG=4.33 and 59.0% retrieval.Source: §8 · p. 8

9 Discussion

• FARS has low overlap with the dominant linear readout span, while final-layer and same-layer next-token controls carry more readout energy.Together with intervention results, this is consistent with concept representations contributing before final output readout, but does not identify the complete computation or establish sufficiency.Source: §9 · p. 8
• The full diagnostic pipeline is not compute-free: extraction requires activations, and depth matching requires fitting a translator.The translator explains only part of final-layer variation (held-out R2 = 0.23–0.63), making it an approximate linear reference.Source: §9 · p. 8
• Rank sweeps assess overlap with a chosen readout span, not information in every output direction; at k = D, every subspace has unit loading.The diagnostic is most useful alongside targeted interventions and decoding tests.Source: §9 · p. 8
• The empirical scope uses supervised labels and a fixed concept inventory, and re-extraction transfer does not establish a universal fixed basis.Fixed-basis transfer is partial and task dependent; CoT-tail analyses sample a token-budget-limited reasoning stream, and an output-facingness prediction did not replicate.Source: §9 · p. 8

Ethics Statement

• The analyses use only publicly released open-weight LLMs.No human-subjects data are collected or released.Source: Ethics Statement · p. 9
• TriForm and Novelty stimuli contain canonical reasoning patterns with no harmful content.Source: Ethics Statement · p. 9
• An AdvBench-prompt experiment found no safety-refusal gap from the tested prose-versus-code representational asymmetry.Only aggregate refusal counts are released; no model completions or new attack strings are released.Source: Ethics Statement · p. 9

Reproducibility Statement

• The public code and research snapshot includes benchmark and core analysis code but omits the September 28–29 supplementary runs.The reported compute description has not been independently remeasured.Source: Reproducibility Statement · p. 9
• Full reproduction additionally requires exact stimulus files, activation caches, extraction and alignment implementation, and checkpoint revisions.Source: Reproducibility Statement · p. 9
• The raw diagnostic needs an extracted basis and unembedding weights, while depth matching additionally needs fitted activation-based translators.Source: Reproducibility Statement · p. 9

AI Use Statement

• LLM assistants contributed to code, literature retrieval, experimental planning, prose, figures, and supplementary model experiments; authors retain responsibility for design, verification, interpretation, and submitted content.Source: AI Use Statement · p. 9

Appendix: Evidence and Reproduction

• The appendix separates evidence by analysis and retains negative and non-replicating results without promoting claims beyond their evidence.Model counts refer to each stated analysis, not one shared cohort; some reconstructed figures use plotting-script summaries rather than raw trial records.Source: Appendix: Evidence and Reproduction · p. 12

A Implementation

• The raw diagnostic code operates on a supplied basis and readout weights; it does not reproduce extraction or validate an X-FARS optimizer, while the appendix includes FARS-extraction and subspace-patching code.Source: A Implementation · p. 12

B Stimulus Examples

• Stimulus examples illustrate a controlled inventory but do not establish coverage of natural reasoning tasks.Table 4 gives a modus ponens example across six surface forms.Source: B Stimulus Examples · p. 13, Table 4 · p. 13

C TriForm Benchmark Concept Inventory

• TriForm contains 18 concepts across five reasoning domains, while the separate Novelty inventory contains 10 concepts absent from TriForm.Novelty uses six surface forms and three instances per concept, totaling 180 stimuli.Source: Table 5 · p. 13, Table 6 · p. 14

D Subspace Extraction Details

• The extraction details note that protocol-specific layer choices differ in the historical CoT analysis; the reported cluster-purity table characterizes FARS and form-control subspaces as orthogonal information carriers.Source: D Subspace Extraction Details · p. 14, Table 7 · p. 14

D.1 Instruction-tuning robustness

• The five-model dimensionality sweeps reach 97.1–97.9% of k=17 concept RSA at k=10, while the MBPP comparison identifies concept-distribution-specific performance.The MBPP-sanitized results are below full activations for problems outside the 18 TriForm concepts.Source: Figure 4 · p. 15, Table 9 · p. 15

D.2 Out-of-distribution retrieval (scope limit)

• The out-of-distribution retrieval section frames Table 10 as a comparison of 10-dimensional FARS with a 5-dimensional form control.Source: Table 10 · p. 15

E.1 Proof of Lemma 1

• Pairwise-difference scatter is 2n times centroid scatter, making Diff-of-means and FARS top-k subspaces identical; concept-centroid PCA matches LDA RSA within 0.02.The shuffled-centroid null trails FARS by 17–29σ. CCS is 1-d unsupervised binary, DAS learns a rotation toward a causal target, and LEACE erases concepts, so these are not equivalent methods to FARS.Source: §E.1 · p. 16, F Methodological Position: FARS vs.… · p. 17

G Readout Orthogonality at Matched Depth

• FARS carries 0.46% energy versus 0.30% for random subspaces under the pulled-back readout, while the same-layer next-token control carries 5.88%—12.9× FARS—and exceeds all four published estimators in 25/25 models.FARS’s raw-test residual enrichment over the Haar floor, 0.79%, shrinks to 0.46% after depth matching.Source: Table 13 · p. 19, G Readout Orthogonality at Matched Depth · p. 19

H Readout-Rank Sensitivity

• Final-layer PCA is more enriched than FARS in 25/26 models at k = 10 and k = 32, and 24/26 at k = 100; the comparison does not establish zero readout information at any rank.R1-Distill-Qwen-7B reverses the comparison at all three ranks, and Mamba-2.8B also reverses it at k = 100; at full ambient rank, energy and enrichment equal 1 for every subspace.Source: H Readout-Rank Sensitivity · p. 19, Table 14 · p. 21

I Published Concept Directions Under the Test

• Validated refusal and truth directions show low readout loading at the FARS block in 16/22 and 19/20 models, while vocabulary-anchored directions load more.The vocabulary construction loads 8.43% on average against 5.20% for random-token directions and 0.80% for FARS, a 16.9× separation; random tokens exceed it in 4/26 models, and the effect varies by model. Refusal and truth directions become readout-adjacent at the final block, with median energy increases of 5.3× and 8.2×, respectively; missing AUC checks qualify some pairs.Source: I Published Concept Directions Under… · pp. 20–21, §I.1 · pp. 22–23, Table 16 · p. 24

I.2 Sparse-autoencoder feature directions

• On Llama-3.1-8B-Inst, SAE concept-selective features have 1.06% readout loading, indistinguishable from same-layer FARS at 0.96% and random features at 0.88%.The SAE and FARS subspaces have 26% mutual energy overlap; this is a one-model result because only this model had a matching published residual-stream SAE.Source: §I.2 · p. 23, Table 17 · p. 24

J Layer-wise Localization of Concept Structure

• Three analyses locate format-agnostic concept structure in middle layers, where concept RSA peaks, cross-format probes recover concepts, and format-agnostic neurons are most prevalent.Cross-format probing reaches 60.3% for Mistral-7B against 5.6% chance; EN→Math top-10 overlap (0.65–0.73) is 3–4× EN→Code (0.16–0.22) across five models, with every 95% bootstrap CI above 2.4×.Source: J Layer-wise Localization of Concept… · p. 24, Table 18 · p. 25

K Empirical Dimensionality: the k = 10 Plateau

• The k = 10 dimension is an empirical elbow, and its plateau replicates across dense decoders, state-space models, and the DeBERTa encoder-MLM rather than being specific to one architecture.The sweep used k = 1–17 on an 18-concept TriForm benchmark and assessed captured centroid variance and cross-format retrieval; the supplied result passage gives no saturation percentages.Source: K Empirical Dimensionality: the k = 10… · p. 25

L Inter-Layer FARS Subspace Stability

• Within a model’s plateau, consecutive FARS bases overlap 0.89 versus approximately 0.05 for random subspaces, with overlap 0.78 across ±2–3 layers, consistent with a slowly rotating frame.Boundary-crossing overlap is 0.80 ± 0.18; RoBERTa is excluded because its plateau has width one.Source: L Inter-Layer FARS Subspace Stability · pp. 25–26

M A Prediction That Did Not Replicate: Concept-Family Ordering

• The expected concept-family ordering is fully monotonic in only 1 of 6 models, with 41 of 60 pairwise orderings correct versus 30 expected by chance.The inference-validity direction is near chance only on gpt2-xl (1.03); on larger models it reaches 38×, and final-layer saturation likely washes out family differences.Source: M A Prediction That Did Not Replicate:… · p. 26, Table 23 · p. 28

N Multi-seed cross-concept intervention audit

• The four-model, three-seed audit comprises 24 completed runs with 120 directed concept pairs per run, but its KL-proximity event is not a measure of semantic correctness.Seeds vary both pair sampling and the random subspace; prefix-aligned positions need not match semantics, whereas the last-token protocol maps source and target last tokens at the same layer.Source: N Multi-seed cross-concept… · p. 29

N.1 Logged dose replication on Qwen2.5-3B-Instruct

• On Qwen2.5-3B-Instruct, logged dose sweeps show chat accuracy falling from 90% baseline to 6% with FARS at c = 2, while plain-prompt accuracy is 55% at c = 0.5 and 20% at c = 2.The runs differ in prompts, sampled items, and generation limits, so their difference cannot be attributed to any one factor or establish semantic sufficiency.Source: Figure 12 · p. 30, §N.1 · p. 30

N.2 Tokenwise norm-matched task intervention

• At c = 2, FARS reaches 6% accuracy versus 40–58% for norm-matched random controls, with output-limit rates of 2% versus 12–60%.The comparison uses the same 50 questions, retains truncated generations in the accuracy denominator, and conditions on one model and three sampled random bases; the high-dose gap supports directional sensitivity, not causal sufficiency.Source: Figure 13 · p. 31, §N.2 · p. 31

O Causal Validation: Cross-Form Activation Patching

• Cross-form FARS patching preserves the model’s argmax in 89.7–97.3% of cases across five models, exceeding same-rank full-PCA patching on every model (Holm-corrected p<10^-22).Full-PCA preserves 37.3–87.7% and full-activation replacement 22.0–83.3%; preservation alone does not establish that a patch installs a concept.Source: O Causal Validation: Cross-Form… · p. 31, Table 25 · p. 31

O.1 Cross-concept patching: the sufficiency question

• The historical one-model cross-concept pilot records an intervention effect but rarely moves the output distribution nearer the source, and its Haar control is not norm-matched.Its KL event does not measure semantic accuracy and does not establish necessity or that a source concept was installed.Source: §O.1 · pp. 31–32, Table 26 · p. 32

P.1 Probe level: cross-form concept identifiability

• FARS ablation reduces cross-form probe accuracy by 17–50 percentage points on all 15 tested models, while random 10-dimensional ablation changes it by ±0.2 points.The same dimensions that recover concept identity therefore carry cross-form concept identifiability; Llama-70B-Inst is probe-necessary despite being task-level inert.Source: §P.1 · p. 32

P.2 Token level: FARS vs. random ablation

• FARS ablation changes top-1 argmax in 22 percentage points more cases than matched-rank random ablation and produces 130× larger median KL.The supplied passage reports this token-level comparison without specifying model or protocol conditions.Source: §P.2 · p. 32

P.3 Task level: GSM8K and MATH

• On Qwen2.5-7B-Inst, FARS ablation lowers GSM8K accuracy from 90.0% to 70.0% (−20.0 pp) and MATH from 71.7% to 53.3% (−18.4 pp).The GSM8K FARS-versus-random comparison has paired Wilcoxon p=4.0×10−5; on MATH, FARS-versus-random p=5×10−3.Source: §P.3 · p. 33
• At c=1.0, Qwen-Inst FARS amplification yields 71% accuracy versus 90% for random amplification; Mistral also shows bidirectional effects, unlike the reported Llama-family inertia.On Mistral-7B-Inst, FARS accuracy is 42% versus random 52% at c=−1.0 and 8% versus random 59% at c=+2.0.Source: §P.3 · p. 33
• FARS shows no significant advantage over random subspaces on three BIG-Bench Hard tasks (−3.3 pp, p=0.23), and concept-targeted steering on Qwen-Inst GSM8K falls to 0% at α=2.The authors characterize FARS as most informative for mathematical-symbolic reasoning; its GSM8K interventions are out of distribution for the 18 concepts and support general concept-bearing computation rather than benchmark-specific structure.Source: §P.3 · pp. 33, 35

Q.2 Declarative-Python control

• Of the +0.506 total prose-to-code gap, encoding style explains ∼11% and format surface ∼89% on Qwen2.5-7B-Instruct.The procedural-axis difference is +0.054 (p=2.5×10−9), compared with +0.452 for the surface axis.Source: §Q.2 · p. 36, Table 33 · p. 36

R Cross-Format Safety Refusal Test (Falsified Hypothesis)

• Cross-format refusal gaps are unsupported in two instruction-tuned models: Qwen’s gap is +1 pp (p=0.50), and Llama’s is −3 pp in the opposite direction (p=0.996).The test covers two open-weight models, a simple Python wrap, aggregate refusal counts, and a conservative classifier.Source: R Cross-Format Safety Refusal Test… · p. 37, Table 34 · p. 37

S Head-Level Decomposition at the FARS Writer Layer

• At the writer layer, attention contributes 75.6% on Llama and 69.6% on Mistral; the replicated profile is layer-localised and assembled across many heads with an MLP contribution.The MLP shares are 24.4% and 30.4%, respectively; head indices differ across the two architectures.Source: S Head-Level Decomposition at the FARS… · pp. 37–38, Table 35 · p. 37
• Joint ablation of Llama-8B’s top five heads leaves FARS RSA unchanged (∆=−0.002, matching random-5), so contribution magnitude does not establish head-level causal necessity.The authors state that multi-layer scrubbing would be required.Source: S Head-Level Decomposition at the FARS… · p. 38

S.1 Axis-specific ablation: FARS is distributed

• Full FARS ablation drops Qwen-7B-Inst GSM8K accuracy 14 pp (91% →77%), while each canonical-axis ablation stays within ±2 pp; cross-model effects vary, supporting a distributed representation.On Qwen-Inst, cumulative top-K ablation is monotone (92% →80%) with ∼83% of the effect by K=4, whereas random-K is inert. Llama results are mixed: single axes remain near baseline, and larger-model top-K interventions are inert despite removing a residual component.Source: §S.1 · p. 38, Table 36 · p. 38

T Where Is FARS Written? A Per-Layer Decomposition

• Across 15 models from four architecture families, FARS is written 0–5 layers before its peak and remains recoverable across a multi-layer plateau.The writer-before-peak gap is widest in Mamba (5 layers). Across 14 models, consecutive-layer plateau overlap averages 0.892 ± 0.036, versus a random 10-d baseline of ∼0.05; FARS is recoverable within the plateau up to small rotation.Source: T Where Is FARS Written? A Per-Layer… · pp. 38–39, Table 37 · p. 39

U.1 Frozen TriForm basis on unseen concepts

• Held-out concept RSA reaches 0.533, 2.6× the random baseline, while Novelty-FARS exceeds random by 6–57 pp on every model, supporting extraction-method transfer rather than a universal fixed basis.The held-out comparison is against a random 10-d projection on the same split, and every model has positive ∆. Frozen-basis transfer is partial: the bases lose substantial accuracy relative to full representations and do not establish a universal rank-10 space.Source: §U.1 · pp. 39–40, Table 40 · p. 40, Table 39 · p. 40

V Capability Prediction: FARS Plateau Width vs. Downstream Benchmarks

• Plateau width correlates positively with all three downstream benchmarks, with MMLU strongest (ρ=+0.66), but its partial correlation after controlling for log10 P is marginal (ρ=+0.48, p=0.094).With n=13, this is a population-scale observation rather than a per-model predictor. Peak RSA-C saturates at 0.36–0.37 across 12 modern generative models (3B+), despite a 5–95% GSM8K accuracy range.Source: Table 41 · p. 41, V Capability Prediction: FARS Plateau… · p. 41

W.1 Format-collapse control: does the CoT-tail really cross formats?

• Regrouping reasoning traces by their generated form lowers cross-format scores from 30.6–44.4% to 26.9–28.1%, but they remain 3.3–5.1× matched-rank Haar controls.More than half of non-English prompts yield English traces, yet six surface forms survive. This control covers two models, and traces are 79–86% budget-truncated, so the result concerns the reasoning stream rather than completed chains.Source: §W.1 · pp. 41–42, Table 42 · p. 42

W.2 The three questions

• Last-input-token and CoT-tail FARS differ substantially in the distilled models, while the native QwQ-32B observation has dG=4.33 and CoT-tail retrieval of 59.0% versus 22.8% Haar.Base-to-distilled comparisons show nonzero geometric changes, not preservation of a causal reasoning mechanism. Selected CoT-tail layers are earlier, and these comparisons change both token position and layer selection. Adding Llama-Distill raises X-FARS retrieval by 1.74 pp; adding Qwen-Distill lowers it relative to that intermediate pool, though each pool remains above its Haar mean. The audited sampled tails are often budget-limited; these results do not establish subspace sufficiency.Source: §W.2 · p. 42, §W.4 · p. 43, Table 44 · p. 43

X Historical Cross-Model Alignment and Retrieval

• Across 27 models, X-FARS explains 97.6% of cross-model centroid variance versus 61.7 ± 1.0% for a same-rank random projection; FARS-projected retrieval reaches 71.7% versus 50.8% full-space.Cross-model difference vectors and analogies also exceed shuffled controls, while compositions vary by architecture. Retrieval gains are scoped to the extracted concept distribution: on MBPP-sanitized, FARS retrieval falls to 1–6% versus 6.5–17% for full activations.Source: Figure 17 · p. 44, X Historical Cross-Model Alignment and… · pp. 44, 46, Table 48 · p. 46

Y Historical Canonical Concept Axes

• Refitted across 27 models, X-FARS explains 97.6% of centroid variance, and its canonical axes show interpretable concept contrasts; alignment remains high across a nine-depth-percentile sweep.Axis 1 captures the declarative–procedural distinction, and the top five axes carry 76% of canonical variance. Cross-architecture stability is higher for selected declarative than procedural concepts. The passage introduces replacing the two-step pipeline with a joint objective but supplies no result for that objective here.Source: Table 49 · p. 46, Y Historical Canonical Concept Axes · pp. 46–47
• 0.881 mean cosine across 351 cross-model pairs: 97.7% are positive and 91.2% exceed 0.5, versus a shuffled null of −0.001 ± 0.004.The separation is 200.2σ; widening the pool from 15 to 27 models lowers the mean cosine from 0.961 while raising separation.Source: Figure 22 · p. 49, Y Historical Canonical Concept Axes · p. 49
• 0.613±0.248 mean Pearson r: X-FARS preserves cross-model centroid-distance landscapes across 351 pairs, with 74% above 0.5 and a 94.9σ separation from null.Curated Word2vec analogies reached 37.4% top-1 accuracy (chance 6.7%; 17.0σ) and 54.4% top-3 accuracy; the effect came from within-domain analogies.Source: Y Historical Canonical Concept Axes · p. 49
• 69.8% top-1 cross-architecture concept retrieval across 30 ordered pairs and six forms, essentially matching within-model cross-form retrieval at 70.5%.Same-family pairs reached 84–87%, and similar-scale cross-family dense models reached 80–87%; GPT-2 XL and Mixtral-as-query were weakest.Source: Table 52 · p. 50, Y Historical Canonical Concept Axes · p. 50
• 97.0% variance: X-FARS exceeds FARS+GPA by 5.1 pp across 15 models, with ∆RSA ∈[−0.008, +0.006].Alignment remains robust under concept subsampling (9 concepts: 98.0%; 15 concepts: 93.6%) and instruction tuning (Mistral-base to Mistral-Inst Procrustes: 92.4%; cross-basis RSA loss: 1–3%).Source: Y Historical Canonical Concept Axes · pp. 47–48, Table 54 · p. 51

Z Composite Concepts and Constituent-Span Reconstruction

• Pairwise composite activations had span ratios above both random controls on every tested architecture (p<10^-9 per row), with both coefficients positive in 66–85% of stimuli.The result replicated on Mamba-2.8B (span 0.77) and RoBERTa-large (span 0.83); strict equal-weight sums failed (p≈1), and coefficients varied by instance.Source: Z Composite Concepts and… · pp. 51, 53, Table 55 · p. 51
• Triplet spans significantly exceeded random-direction controls on all six generative architectures, including Mamba (p<10^-4 per model).Encoder MLMs failed triplet reconstruction: DeBERTa-v3-large had span 0.66 versus random-concept 0.76 and no all-positive triplets, although pair composition held on both encoders.Source: Z Composite Concepts and… · pp. 51–52, Table 56 · p. 52
• Only Mamba-2.8B clearly beat the random-concept control for quartet composition (span 0.87 vs. 0.79, p=0.023); dense transformers tied or lost while beating random-direction controls.This pattern is consistent with composition saturating as arity approaches FARS dimensionality.Source: Table 57 · p. 52, Z Composite Concepts and… · p. 52

Cross-Model FARS Geometry Predicts Other Models’ Behavior

• ρ=−0.750 across 204 entries links cross-model FARS distances with patching overlap, but random 10-d projections recover much of the association (ρ=−0.714).The association is exploratory: entries share models and form pairs, and it does not establish FARS-specific computation or causal necessity.Source: Table 58 · p. 53, Cross-Model FARS Geometry Predicts… · p. 53, Table 59 · p. 54