PaperlayerView exact revision on arXiv

Source-linked AI summary

Gluing Local Contexts into Global Meaning: A Sheaf-Theoretic Decomposition of Transformer Representations

Bryce Grant, Peng Wang

arXiv:2610.17117v1cs.CLcs.LG

TL;DR

Existing interpretability tools do not test whether representations keep content consistent across paraphrases. This paper uses a cellular sheaf on paraphrase graphs to derive content and influence directions, then extends the graph across layers to measure harmonic mass. Its content subspace outperforms LEACE on held-out retrieval, and harmonic mass ranks architectures by steering fragility.

  • Problem

    Existing tools do not test whether a representation encodes the same information when phrasing changes but meaning stays constant.

    Source: §1 · p. 1

  • Method

    The paper builds a cellular sheaf over paraphrase pairs and uses its Laplacian to derive a globally consistent content subspace and a spectral complement, without training the frozen model.

    Source: §1 · p. 1, §8 · p. 10

  • Results

    At 20 dimensions, sheaf H0 beats LEACE’s preserved subspace on held-out CounterFact retrieval across eight architectures by a mean +17.1 pp.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper’s kernel, spectral complement, and depth-induced cycles provide content, influence, and steering-fragility measures from the same framework.

    Source: §8 · pp. 10–11

    Graphs joining activations of different dimensions require per-node maps of different widths, which the paper leaves for future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Existing interpretability tools do not test whether a representation keeps the same content when only the phrasing changes. We treat each paraphrase pair as a local witness that two activations should agree and build a cellular sheaf over the paraphrase graph; its Laplacian $L_\mathcal{F}$ answers the question on a frozen model with no training. The kernel $H^0$ is a content subspace: at 20 dimensions it beats the preserved subspace of the concept eraser LEACE on held-out CounterFact fact retrieval across eight architectures by a mean $+17.1$\,pp, and constraining all paraphrases of a fact jointly adds a further $+6.5$\,pp. The spectral complement concentrates causal influence: ablating the most paraphrase-variable coordinates, a coordinate-wise proxy for it, changes model outputs $5.6$ to $26.5\times$ more than ablating coordinates of matched per-coordinate variability, across nine models from 124M to 13B parameters. Stacking layers into a grid creates $b_1 = N(L-1)$ algebraic 1-cycles, and the resulting model-derived harmonic mass varies by nearly three orders of magnitude across six architectures and ranks them as their steering fragility does (Spearman $\rho = 0.90$, exact one-sided permutation $p = 0.014$). The same paraphrase graph yields the content subspace, the influence directions, and the fragility measure. Project page: https://cwru-aism.github.io/gluing-lc-page/

1 Introduction

The paper frames paraphrase consistency as a missing interpretability question: whether a representation preserves content when phrasing changes. It uses a sheaf Laplacian to separate stable content from phrasing-dependent variation and evaluates the resulting subspaces and cycle-based diagnostic.

Motivation

  • Existing interpretability tools do not test whether a feature encodes the same information when phrasing changes but meaning does not.The paper links this gap to variable steering outcomes across models and inputs.Source: §1 · p. 1

Approach

  • Paraphrase pairs act as local agreement witnesses in a cellular sheaf, whose Laplacian separates globally consistent content from high-disagreement directions.The analysis is performed on a frozen model without training.Source: §1 · p. 1

Key results

  • At 20 dimensions, sheaf H0 beats LEACE’s preserved subspace on held-out CounterFact retrieval across eight architectures by a mean +17.1 pp.The introduction also reports that jointly constraining all paraphrases of a fact adds +6.5 pp.Source: §1 · p. 2

Functional influence and fragility

  • Across nine models from 124M to 13B parameters, ablating paraphrase-variable coordinates changes outputs 5.6 to 26.5× more than matched-variability controls.The paper also reports that grid-derived harmonic mass ranks six architectures by steering fragility (ρ = 0.90, p = 0.014).Source: §1 · p. 2

2 Cellular Sheaves on Paraphrase Graphs

The paper models paraphrase agreement with a cellular sheaf on a graph. Its coboundary measures local disagreement, while the Laplacian kernel represents globally consistent sections that can be read as content across equivalent contexts.

Sheaf construction

  • A cellular sheaf assigns vector-space stalks to graph nodes and edges, linked by linear restriction maps at each node-edge incidence.Here hidden states occupy node stalks, while paraphrase pairs form edges.Source: §2 · pp. 2–3

Local disagreement

  • The coboundary operator δ0 measures disagreement between neighboring node values after mapping them into the shared edge stalk.When δ0x is zero everywhere, the local data glues into a global section.Source: §2 · p. 3

Global consistency

  • The sheaf Laplacian LF = (δ0)⊤δ0 has kernel H0, the space of globally consistent sections.The paper reads H0 as content consistent across semantically equivalent contexts and the spectral complement as phrasing-dependent variation.Source: §2 · p. 3

Relation to prior tools

  • Sparse autoencoders decompose one activation and CKA or SVCCA compare whole spaces, whereas this framework splits a space into phrasing-invariant and phrasing-varying parts.The resulting split is used for interventions intended to move variable directions while leaving content intact.Source: §2 · p. 3

3 Local Construction on Matching Graphs

On a matching graph, the paper projects paraphrase-pair activation differences into a shared comparison space and uses their Laplacian spectrum to define content and high-disagreement directions. The resulting spectral problem is low-dimensional relative to the model’s hidden state.

Sheaf construction

  • The matching-graph construction assigns hidden states to node stalks and a shared k-dimensional comparison stalk to each paraphrase edge, using P⊤ as the restriction map.The paper gives joint PCA, CCA, and contrastive Fisher as choices for learning P.Source: §3 · p. 3

Restriction maps

  • Contrastive Fisher chooses the k generalized eigenvectors with largest within-pair variance relative to total activation variance, orienting P toward maximal within-pair variation.The resulting spectral complement is intended to concentrate variation while H0 captures residual content.Source: §3 · p. 3

Laplacian and decomposition

  • The empirical sheaf Laplacian is the second-moment matrix of projected edge disagreements, and its low- and high-eigenvalue directions define H0 and the H1 directions.The eigenproblem has size k × k; with k = 128, the spectral step has the same cost from 124M to 13B models.Source: §3 · pp. 3–4

Matching-graph limitation

  • A matching graph has no cycles, so algebraic H1 vanishes; the paper uses “H1 directions” as a mnemonic for the top-Laplacian-eigenvalue directions, not cohomology classes.On this graph, every edge cochain is a coboundary.Source: §3 · p. 4

Matching-graph reduction

  • For a matching graph with one shared P and identity transports, LF reduces to projected within-pair covariance; the Fisher baseline solves a generalized eigenproblem against projected total covariance.This reduction depends on the shared-map and identity-transport conditions.Source: §3 · p. 4

Per-node maps

  • With per-node restriction maps and Procrustes transports, LF generally no longer reduces to a projected covariance built only from hidden-state second moments.The different maps on the two sides of an edge change the cross-terms.Source: §3 · p. 4

Restriction-map findings

  • Within the single-shared-P regime, random or identity maps recover most H0 retrieval, while the Fisher baseline degrades fact preservation from 29.8% to 15.4%.The paper also reports that joint multi-paraphrase constraints add +6.5 pp.Source: §3 · p. 4

4 Multi-Layer Matching Grids and Algebraic H1

The paper adds cycles to paraphrase graphs through network depth or relation structure, allowing harmonic components to capture obstructions to consistency. It tests whether model-derived harmonic mass tracks steering fragility and reports that relation-hypergraph holonomy does not depart significantly from a shuffled-paraphrase null.

4.1 Multi-layer matching grids

  • The grid connects paraphrase pairs horizontally within layers and each sentence vertically across adjacent layers, yielding b1 = N(L −1) square cycles.Each square tests whether the layer transition commutes with paraphrase equivalence.Source: §4.1 · p. 4
  • Ablating H1 coordinates changes outputs more than matched-variability controls across architectures, with reported ratios from 5.6× to 26.5×.Figure 2 distinguishes one-layer logit effects from three-depth effects on the final hidden state.Source: Figure 2 · p. 5
  • The model-derived cochain combines paraphrase differences and centered, Procrustes-aligned layer transitions; its harmonic mass measures the energy fraction in the Hodge harmonic subspace.With identity transport, the cochain would be a coboundary and its harmonic mass would be zero.Source: §4.1 · p. 5
  • At L = 2, model-derived harmonic mass spans [10−4, 0.08], while a random Gaussian cochain has expected harmonic fraction 0.25.The table caption states that all rows share b1 = 30, so the variation comes from model layer transitions.Source: §4.1 · p. 5, Table 1 · p. 5
  • For six architectures, harmonic mass ranks steering fragility with Spearman ρ = 0.90 and exact one-sided permutation p = 0.014.The three lowest-mass architectures are the three most robust, and the three highest are the three most fragile.Source: §4.1 · p. 5
  • Filling each grid square with two 2-simplices makes b1 and b2 zero in tested cases, causing harmonic mass to vanish; the coexact fraction preserves the architecture split.The paper therefore leaves the grid’s 2-cells unfilled for its harmonic-mass measure.Source: §4.1 · p. 6

4.2 Relation hypergraphs

  • A relation hypergraph with 60 facts, five paraphrases per fact, and six relations yields 66 cycles: 60 fact-level and six relation-level.The matching graph and filled fact cliques on the same nodes have H1 = 0.Source: §4.2 · p. 6
  • Relation-hypergraph cycle holonomy is 2.5–3.1 versus Haar RMS 4, and no architecture significantly differs from the shuffled-paraphrase null (one-sided p = 0.10–0.33).A node-space operationalization of H1 did not transfer to retrieval or steering.Source: §4.2 · p. 6

5 Experiments

The experiments evaluate the kernel as a content subspace, its spectral complement as a source of concentrated influence, and multi-layer harmonic mass as an architecture-level steering diagnostic. They test retrieval, activation ablation, and steering across transformer models.

5.1 Setup and Datasets

  • The evaluation spans nine transformer models from 124M to 13B parameters, using MRPC for restriction-map learning, CounterFact for steering and retrieval, and held-out fact splits for retrieval comparisons.The setup also uses functional validation on MRPC and PAWS sentences.Source: §5.1 · p. 6

5.2 Content from the Kernel: H0 Beats LEACE, and Joint Paraphrase Constraints Improve It

  • On held-out hard same-relation retrieval, 20-dimensional sheaf H0 beats full-dimensional LEACE-preserved subspaces across eight architectures by a mean +17.1 pp.The comparison uses held-out CounterFact facts; the reported gap ranges from +1.5 pp on OLMo-2 to +42.2 pp on GPT-2.Source: Table 2 · p. 7, §5.2 · p. 7
  • Joint constraints across all paraphrases improve ring-H0 over pair-H0 by a mean +6.5 pp across eight architectures.A five-edge acyclic path also gains +7.7 pp over pairs, while the ring exceeds the path by +0.5 pp, significant on none of the eight architectures.Source: §5.2 · p. 7

5.3 Influence from the Spectral Complement: H1 Carries Concentrated Causal Influence

  • H1-coordinate ablation changes outputs 5.6–26.5× more than ablation of controls matched for per-coordinate variability across all nine models.Every 95% confidence interval excludes 1; the matched controls have only 0.13–0.53 of the H1 set’s total variability, so part of the ratio reflects greater total variability removed by H1 ablation.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Fragility from Harmonic Mass: Sheaf Steering as an Architecture Diagnostic

  • Harmonic mass ranks steering fragility across six architectures with Spearman ρ = 0.90 and exact one-sided permutation p = 0.014.The three lowest-mass architectures are the three most robust and the three highest are the three most fragile; harmonic mass uses MRPC layer transitions, while fragility uses CounterFact fact preservation under random perturbation.Source: §5.4 · p. 8
  • At α = 0.3, steering outcomes vary by architecture: contrastive sheaf leads on fragile Llama-2-7B, while random steering is competitive on robust Mistral-7B and Llama-3-8B.On Llama-2-7B, contrastive sheaf preserves 31.0% of facts versus 4.2% for random; on Mistral-7B, CCA sheaf reaches 41.6% versus random at 39.4%, a statistical tie.Source: Table 4 · p. 9, §5.4 · p. 9
  • On the two swept robust architectures, H0-removed sheaf steering beats random by a mean +1.0–2.0 pp at α ∈ [0.10, 0.15], while random becomes competitive from α = 0.30.The reported recommendation is α ∈ [0.10, 0.15] on robust architectures and α ≈ 0.30 on fragile ones.Source: §5.4 · p. 9
  • PIXEL reproduces the architecture-dependent pattern: it preserves 21.1% of facts on Llama-2-7B versus 4.2% for random, but reaches 34.8% on Mistral-7B versus random at 39.4%.Contrastive sheaf leads on Llama-2-7B at 31.0%.Source: §5.4 · p. 9

6 Related Work

The paper situates its sheaf decomposition among work on sheaf methods, subspace interpretability, and steering. Its distinction is a post-hoc decomposition of frozen transformer representations by cross-context consistency.

Paper scope

  • This version adds multi-layer grids, harmonic-mass fragility diagnosis, a held-out LEACE protocol, and other experiments beyond the earlier workshop version.The earlier version contained only the single-layer construction and three pilot experiments.Source: §6 · p. 9

Sheaves in machine learning

  • Related transformer sheaf work targets circuit coherence or context-chart atlases; this paper instead uses paraphrase-graph hidden states and post-hoc H0/H1 decomposition for retrieval and steering.The paper also situates its construction within sheaf Laplacian and sheaf neural network research.Source: §6 · p. 10

Subspace interpretability

  • Unlike work targeting individual features, chosen-attribute decodability, or whole-space similarity, this approach splits representations into phrasing-invariant and phrasing-varying parts.The paper contrasts frozen-model invariance read from ker(LF) with training-time paired-view invariance objectives.Source: §6 · p. 10

Steering and linear representations

  • The paper compares empirically to PIXEL and frames its coordinate ablations as addressing the linear-representation concern only in part.Prior work documents model-dependent steering reliability across 36 models and input-level unreliability.Source: §6 · p. 10

7 Discussion

The discussion connects the construction to self-supervised regularization and identifies boundaries on its diagnostic scope. It also describes a lightweight use for harmonic mass while noting that broader validation remains needed.

Discussion

  • Harmonic mass can screen a model before a steering sweep using 30 MRPC pairs and one forward pass per sentence.The diagnostic’s reported ranking is based on six architectures, and the paper says more architectures are needed before it serves as a predictor.Source: §7 · p. 10

Self-supervised connection

  • Across seven architectures, the sheaf-regularized objective holds encoder rank at 57–60/64, while naive MSE collapses it to 14–42/64.The objective combines tr(LF), the within-pair MSE up to scale, with a log-determinant spread term that penalizes dimensional collapse.Source: §7 · p. 10

Limitations

  • The restriction map P is architecture- and partially domain-dependent, and node-space H1 does not transfer; logical equivalence and dialogue continuation remain untested settings.The paper reports a cross-lingual probe that transfers to translation pairs, while stating that more architectures are needed to validate harmonic mass as a predictor.Source: §7 · p. 10

8 Conclusion

The conclusion presents the sheaf condition as a way to read cross-context consistency from a frozen model. It summarizes the kernel, spectral complement, and layerwise cycles as yielding retrieval, influence, and fragility findings.

8 Conclusion

  • The sheaf Laplacian’s kernel glues local paraphrase-pair agreements into a content subspace, while its spectral complement represents phrasing-dependent variation in a frozen model.The decomposition is read off without training.Source: §8 · p. 10
  • The kernel beats LEACE’s preserved subspace on held-out retrieval, and the spectral complement concentrates causal influence.The conclusion presents these as outcomes of the decomposition.Source: §8 · pp. 10–11
  • Cycles across layers yield harmonic mass whose ranking tracks steering fragility across six architectures, with ρ = 0.90.The conclusion frames this as an architecture-level relationship.Source: §8 · p. 11

A.1 Algebraic H1̸ = 0 on Richer Topologies

The appendix compares paraphrase, clique, and relation-ring topologies to distinguish algebraic cycles from harmonic mass. It also contrasts sheaf H0 with LEACE on fact retrieval.

A.1 Algebraic H1̸ = 0 on Richer Topologies

  • 66 independent cycles arise in the unfilled relation hypergraph, whose expected harmonic fraction is 18.33% and measured fraction is 18.4%.The cycles include rings linking facts that share a relation.Source: §A.1 · p. 15
  • The matching graph and filled fact cliques both have algebraic H1 = 0, while the relation hypergraph has dim H1 = 66.Table 5 compares genuine cohomology and Hodge harmonic fraction across architectures and topologies.Source: §A.1 · p. 15, Table 5 · p. 16
  • The filled paraphrase cliques have zero harmonic fraction: their cycles are triangle boundaries, despite 60% of energy lying outside im(δ0).The comparison distinguishes energy outside the coboundary image from harmonic energy.Source: §A.1 · pp. 15–16
  • On the 60-fact retrieval set, sheaf H0 outperforms LEACE preserved by up to 10.7 pp on GPT-2 and Qwen-2.5-7B.Sheaf H0 uses 20 of 768–3584 dimensions; LEACE-erased retrieval is 1–3%.Source: §A.1 · p. 16, Table 6 · p. 16

A.2 Simplicial Complex Variants and Scale-Up

The appendix tests how topology and edge dimension affect harmonic mass, then relates that measure to steering fragility. It also describes the grid cycles and checks the numerical H0 estimate against an algebraic calculation.

A.2 Simplicial Complex Variants and Scale-Up

  • Filling each grid square with two triangles eliminates harmonic mass, while the coexact fraction preserves the architecture split: robust models are below 3% and fragile models are 5–20%.Table 7 reports the triangulated-grid quantities at edge dimension k = 64.Source: §A.2 · p. 16, Table 7 · p. 17
  • Filling each fact’s paraphrase clique leaves six relation-cycle harmonic 1-forms; the measured harmonic mass is 0.93%, near the expected 0.91%.The 2-complex has b1 = 6 and b2 = 240, but shared-PCA maps produce no model-implied degree-2 obstruction.Source: §A.2 · pp. 16–17
  • Across edge dimensions and L ≤ 8, the three robust architectures remain below the three fragile ones, with the span exceeding three orders of magnitude.At L = 2, reported values span approximately 2 × 10^-5 to 8 × 10^-2.Source: §A.2 · p. 17, Table 8 · p. 17

A.3 Harmonic Mass vs Steering Fragility (Per-Model)

  • Harmonic mass and CounterFact steering fragility rank six models similarly, with Spearman ρ = 0.90 and exact one-sided permutation p = 0.014.The measures come from disjoint MRPC layer-transition and CounterFact steering measurements; the sole rank inversion is within sampling error.Source: §A.3 · p. 18

A.4 Geometric Picture: Where H1 Lives

  • Each inter-layer square closes a path up one strut, across the next layer, and down another, yielding b1 = N(L − 1) cycles that carry harmonic mass.The slabs represent layers and struts connect the same sentence across adjacent layers.Source: Figure 3 · p. 18

A.5 Proof of the Reduction and Spectral Validation

  • On GPT-2 with 50 MRPC pairs, the spectral H0 estimate is 49 versus algebraic H0 dimension 50, with one eigenvalue just above the cutoff.The near-threshold eigenvalue is approximately 4 × 10^-6, above the 10^-6 threshold.Source: §A.5 · p. 18

B.1 Cross-Dataset Generalization

The appendices test the sheaf decomposition across transfer and design variants, then probe its retrieval, steering, and cross-lingual behavior.

B.1 Cross-Dataset Generalization

  • Out-of-domain P retains +5 to +30 pp over no-kernel baselines but loses ∼25 pp relative to in-domain P on held-out CounterFact retrieval.In-domain P averages 81.6%, versus 55–60% for out-of-domain P across eight architectures.Source: Table 9 · p. 19, §B.1 · p. 19

B.2 Steering Magnitude Sweep

  • At α=0.10–0.15, sheaf-derived steering beats random by +1.0–2.0 pp on Mistral-7B and Llama-3-8B; at α ≥0.30, random becomes competitive.The authors attribute the high-magnitude tradeoff on robust models to over-perturbation from concentrated H1 influence.Source: §B.2 · p. 19

B.3 Cycle-Aware H0: Multi-Paraphrase Rings

  • On hard-CounterFact retrieval, across all eight architectures, multi-paraphrase rings and full paraphrase cliques each beat the pair-only acyclic forest by mean +6.5 pp, performing within 0.1 pp.Ring exceeds the path by +0.5 pp on average, with the 95% CI including zero on all eight.Source: Table 10 · p. 20, §B.3 · p. 20

B.4 Restriction Map Sensitivity

  • Identity-P lands within 4.1 pp of PCA-P on average and beats PCA without kernel selection by +47 pp; random-P lands within 8 pp of PCA-P.The ablation indicates that kernel selection recovers content, while P itself does not select content directions.Source: Table 11 · p. 20, §B.4 · p. 20

B.5 Retrieval Baseline Comparison

  • Sheaf H0 beats every label-free retrieval alternative by +9 to +36 pp across eight architectures and averages within ∼2 pp of the label-requiring discriminative ceiling.It wins outright on Phi-2, Gemma-2, and Llama-3.Source: Table 12 · p. 21, §B.5 · p. 21

B.6 Sheaf Eigenbasis Ablation

  • The eigenbasis ablation compares projecting out H1 with random 20-dimensional subspaces using median and range of mean next-token KL ratios over layers, plus prompt-wise wins.It uses top-20 H1 eigenvectors and random controls at every other layer.Source: Table 13 · p. 22, §B.6 · p. 22

B.7 Dormancy and the Subspace-Patching Illusion

  • H1 is the least dormant measured subspace, and ablating it collapses recall while matched random ablation leaves recall at baseline.On GPT-2, the intervention KL ratio is 15.7×; the authors rely on recall collapse for the 7B models because their KL ratio saturates numerically.Source: Table 14 · p. 22, §B.7 · p. 22

B.8 Cross-Lingual Invariance Probe

  • On Llama-3-8B, H0 retains cross-lingual retrieval and matches or exceeds the full 4096-dimensional hidden state for EN→FR, while H1 discards it.The probe fits H0 on 100 English–French pairs and evaluates 100 disjoint held-out pairs.Source: Table 15 · p. 23, §B.8 · p. 23

B.9 Per-Architecture Layer Profiles

These appendices examine layerwise influence profiles and steering comparisons, then test holonomy, node-space retrieval, and sheaf-based training outcomes.

B.9 Per-Architecture Layer Profiles

  • Across three sampled depths, seven architectures peak at L/4 and OLMo-2-7B and Gemma-2-9B peak at L/2; three depths cannot resolve a layerwise shape.For four models, the table separately reports a 1000-sentence L/2 run under protocol a.Source: §B.9 · p. 23, Figure 5 · p. 23

B.10 McNemar Tests with Multiple Comparisons Correction

  • After Benjamini-Hochberg correction across 16 tests, 13 comparisons survive; the largest effect is on Llama-2-7B (Cohen’s h = +0.35, p < 0.001).Main sheaf-versus-random and sheaf-versus-Fisher comparisons use McNemar’s test with n=1000 per model.Source: §B.10 · pp. 23–24

B.11 PIXEL Head-to-Head on CounterFact

  • PIXEL preserves the architecture-dependent pattern: contrastive sheaf steering reaches 31.0% on Llama-2-7B, CCA sheaf reaches 41.6% on Mistral-7B, and no sheaf method beats random on Llama-3-8B at α=0.30.PIXEL improves over CAA on fragile Llama-2-7B, but random approaches or beats it on robust models.Source: §B.11 · p. 24

B.12 Holonomy Null Calibration

  • As token samples per node increase from N=2 to N=6, holonomy obstruction falls from 3.1–4.4 to 2.2–2.8 on all four architectures, indicating a finite-sample component.The Procrustes fits are underdetermined when N < k.Source: §B.12 · p. 24
  • Observed holonomy obstructions exceed the shuffled null by 0.10–0.29 on four architectures, but none of the differences is significant (one-sided p = 0.10–0.33).The null shuffles paraphrase assignments within each relation across 20 seeds.Source: §B.12 · p. 24, Table 16 · p. 25

B.13 Node-Space Operationalizations of Relation-Hypergraph H1

  • The relation-hypergraph harmonic projection retrieves at 36.3%, below GPT-2’s random-20 baseline of 49.1%; the authors interpret it as relation-stable rather than fact-specific.They bound this result to the immediate downstream usefulness of node-space projections for retrieval, not the algebraic content of H1.Source: §B.13 · p. 25

B.14 Sheaf-Regularized SSL Collapse Prevention (Per-Architecture)

  • Across seven architectures, naive MSE and tr(LF) collapse the encoder to effective rank 14–42/64, while tr(LF) − β log det Cov keeps rank at 57–60/64.The latter is the sheaf regularizer with a log-determinant covariance term.Source: §B.14 · p. 25, Table 17 · p. 26

B.15 MLST on STS-B (Full Table)

  • On GPT-2, MLST-nosheaf-concat exceeds VICReg by +0.046 Spearman and MLST-section exceeds it by +0.035; on Mistral-7B, MLST-section does not significantly exceed VICReg.Across eight trained methods, cross-architecture rank correlation is ρ = 0.74 (p = 0.037); the authors find the specific sheaf transitions and cycle apparatus neutral to slightly negative as a training signal at this scale.Source: §B.15 · pp. 25–26, Table 18 · p. 26

C.1 Hyperparameter Sensitivity

The sensitivity analyses find a stable range for selecting the H0 dimension, while retrieval performance also depends on pooling choice.

C.1 Hyperparameter Sensitivity

  • The spectral gap remains 91.7–95.6× across edge dimensions k ∈ {32, 64, 128}, leaving the content/context boundary stable.The eigenvector count m selects eigenvectors without changing the spectrum.Source: §C.1 · p. 26
  • m = 20–80 forms a retrieval plateau on all four tested architectures, while m = 128 = k leaves no kernel selection and retrieval collapses to 36–53%.At m = 40, retrieval is consistently +4 to +6 pp above m = 20; m = 20 was chosen a priori.Source: §C.1 · p. 27, Table 19 · p. 27
  • Mean pooling gives H0 retrieval of 80–87% across four architectures; first-token pooling collapses, and H0 loses to the full hidden state only on Mistral-7B with first-token pooling.Absolute retrieval rates depend on whether the pooling choice captures content.Source: §C.1 · p. 27, Table 20 · p. 27

C.2 Experimental Details

  • Experiments use mean pooling, joint-PCA restriction maps, k = 128, 20 top/bottom eigenvectors, and specified middle-to-late layers; sheaf construction takes under one minute per model.7B+ models use 4-bit quantization, and activation extraction is the dominant compute cost.Source: §C.2 · p. 27

C.3 Practitioner Guidance

  • The recommended workflow builds the sheaf from 200+ MRPC pairs and selects steering settings on a 50-example validation set, using H1 to avoid generation collapse in fragile models.For robust models, random steering can be competitive at α ≥ 0.30; moderate magnitudes are recommended for a sheaf advantage.Source: §C.3 · p. 28

C.4 NeurIPS Paper Checklist

  • The authors identify architecture- and domain-dependent restriction-map selection, ties with random steering on robust models at α = 0.30, and several unsuccessful or untested comparisons as limitations.These include no significant difference from a shuffled-paraphrase null for cycle holonomy and neutral-to-slightly-negative MLST cycle/transition terms as a training signal.Source: §C.4 · p. 28
  • The checklist reports public datasets, documented splits and hyperparameters, bootstrap confidence intervals, paired tests, and released analysis code with a reproduction guide.The code and paper specify the core pipeline and experimental settings.Source: §C.4 · p. 29
  • The work uses public benchmarks and existing models without human subjects, and frames steering as diagnostic while noting potential misuse against safety guardrails.The checklist says no new pretrained models or datasets are released.Source: §C.4 · pp. 29–30

Related papers

Loading 2610.17117v1…