PaperlayerView exact revision on arXiv

Source-linked AI summary

Relational Composition as Parallel Transport: What Per-Entity Frames Force in Attention

Aakash Sarkar, Ananya Mukherjee, Motoshige Sato

arXiv:2610.34945v1cs.LGcs.AIcs.FL

TL;DR

The paper asks how attention should compare entities whose coordinates are independently arbitrary. It derives transported attention and analyzes how width and readout govern exact composition; its results show that decoding depends on the readout as well as the learned states.

  • Problem

    A dot product is meaningful only for vectors in a shared coordinate system, which is not assured for a newly introduced entity whose role is specified through relations.

    Source: §1 · p. 2

  • Method

    The paper treats independent per-entity frame changes as a symmetry and derives transported attention together with representation-theoretic limits on composition determined by width and readout.

    Source: §1 · p. 2

  • Results

    On the same 72 frozen states, the cross-entropy-optimal affine head is provably uniform and the quadratic head exact; cover training found only under sign-blind readouts (6 of 48 versus 0 of 48 monotone).

    Source: §7 · p. 9

  • Takeaways & Limitations

    Within the studied model class, the realized algebra depends on the width-readout pair, and the decoder helps determine what the representation encodes.

    Source: §7 · p. 9

    The theory is established for small finite groups, parsed-relation benchmarks, and supplied relation operators; continuous or large groups need new tools, and the benchmark algebraic reading is tentative.

    Source: §7 · p. 9

Abstract

from arXiv · show

A dot product means something only when both vectors share a coordinate system. Content words have one; a fresh entity does not: rotating its coordinates alone changes nothing. We take this per-entity frame freedom as a symmetry and ask what it forces in attention. For orthogonal transports, invariant scores depend only on norms and the transported inner product $\langle x_i,T_{ij}x_j\rangle$; equivariant messages lie in the span of $x_i$ and $T_{ij}x_j$. Endpoint channel count bounds transport rank, so with one vector per entity relations must be supplied; standard attention is the flat sector of a shared frame. Representation theory attaches an integer to each mechanism; the readout decides which. With orthogonal transitions, a convex-cell readout composes a finite group exactly only at or above its smallest faithful real representation dimension. At this minimum, exactness forces a faithful representation. An even readout can decode a cover at smaller width, and the gap is unbounded. Below its threshold, the part of the group a mechanism cannot see sets an accuracy ceiling computable before training. 17/17 fitting seeds at minimal width passed tests through length 1000, versus 7/18 at twice the width; 12 models carry correctness certificates to explicit lengths, from the weights alone. On $C_3\times A_4$ (faithful width 5), descent at width 4 learned the 72-state double cover under sign-blind attention (6/48 seeds; monotone 0/48); on a frozen cover the cross-entropy-optimal affine head is uniform, a quadratic head exact. On parsed CLUTRR, transported composition beats a matched transformer by 41 points at 4-10 hops, but forcing isometries costs 65 points; on StepGame's commutative algebra, isometries help. Static embeddings gave each word one vector; contextual models gave each occurrence its own; per-entity frames make each relation an operator, the unit in which relations compose.

1 INTRODUCTION

Per-entity frame freedom makes relations, rather than shared coordinates, the basis for comparing entities. The paper derives the resulting attention constraints and shows that width and readout jointly determine which compositional memory can be realized.

1 INTRODUCTION

  • A dot product is meaningful only when its vectors share coordinates, motivating invariance to changing one entity’s frame while transforming its relation data consistently.The paper presents this as an inductive bias, not a claim that fresh names imply physical symmetry.Source: §1 · p. 2
  • Transporting a neighbor’s vector into the receiver’s frame before scoring or messaging makes standard attention the shared-frame case.The paper derives this form from per-entity frame freedom while noting that related forms have prior empirical precedent.Source: §1 · p. 2
  • With convex decision cells, exact finite-group composition requires at least the smallest faithful real-representation width, while smaller widths can support covers under sign-blind readouts.For C3 × A4, a double cover works at width four although faithful width is five.Source: §1 · p. 2
  • The paper develops forced score and message forms, width and accuracy limits, finite certificates, and experiments, while distinguishing existence proofs from successful optimization and finite-test accuracy from exactness.It reports unsuccessful runs and gives raw weights worst-case horizons.Source: §1 · p. 2

2 INDEPENDENT FRAMES AND TRANSPORTED ATTENTION

Independent entity frames require relations to transport vectors between coordinate systems before attention compares or combines them. The section characterizes those operations, limits where transports can come from, and identifies shared-frame attention as a flat special case.

2.1 WHAT THE SYMMETRY FORCES

  • For orthogonal transports, invariant scores depend on endpoint norms and the transported inner product, while equivariant messages lie in the span of the receiver and transported neighbor vectors.The theorem specifies admissible inputs and message form, not a particular nonlinearity or architecture.Source: §2.1 · p. 3

2.2 WHERE A TRANSPORT CAN COME FROM

  • Endpoint-only covariant maps have transport rank bounded by the ranks of both endpoint channel sets, so one vector per endpoint cannot supply a full-rank orthogonal transport when d ≥ 2.Relations must then supply the transport or additional endpoint channels must be available.Source: §2.2 · p. 3

2.3 SHARED FRAMES AND GRAPH COMPOSITION

  • A connection is flat exactly when its cycle products are identity, allowing per-entity frames in which every transport is identity.Standard attention operates in this shared-frame sector; path composition and loop holonomies describe graph-level composition.Source: §2.3 · p. 4

3 THE READOUT DETERMINES THE COMPOSITION WIDTH

The width needed for exact group composition depends on both the latent transitions and the decoder. Convex readouts enforce faithful-representation thresholds, while nonlinear readouts can decode smaller covers and mechanism-specific limits bound accuracy below threshold.

3 THE READOUT DETERMINES THE COMPOSITION WIDTH

  • An orthogonal recurrence composes target-group generators and decodes each reachable state to its word’s group element; convex-cell decoders include affine argmax and nearest-prototype rules.Arbitrary attention followed by value aggregation need not have convex decision cells.Source: §3 · p. 4

3.1 EXACTNESS FORCES A FAITHFUL REPRESENTATION

  • Exact convex-cell decoding requires width at least dmin(G), and at that minimum the latent transitions form a faithful representation of the target group.A faithful representation with a generic initial state also admits exact affine-argmax decoding; the threshold applies to all-word exactness.Source: §3.1 · p. 4
  • For SO(d) transitions, the minimum special-orthogonal width is dSOmin(G) ∈ {dmin, dmin + 1}; for D4, the minima are two and three.The supplied table summarizes group-specific ceilings and widths computed from Cayley tables before training.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 NONLINEAR READOUTS CAN DECODE A SMALLER COVER

  • Quadratic readouts can decode some finite covers below faithful width: C3 × A4 has quadratic width four versus linear width five, and the additive width gap is unbounded across A2m5.A generic initial state and sign-blind quadratic features can distinguish target labels while discarding the sign of a lifted state.Source: §3.2 · p. 5
  • For C3 × A4, a 72-element orthogonal double cover acts on R4, and squared-prototype decoding can read its labels although convex-cell heads cannot decode the cover exactly.The two lifts of each label are antipodal; a convex cell containing both also contains their midpoint.Source: §3.2 · p. 5
  • On a balanced antipodal cover, the cross-entropy-optimal affine-softmax head predicts uniformly, while quadratic features distinguish label pairs; this loss result does not imply zero best-case affine classification accuracy.A nearest-selected-lift classifier can reach 1/2 accuracy despite having worse cross-entropy.Source: §3.2 · p. 6

3.3 BELOW THE INTEGER: CEILINGS

  • Below a mechanism’s threshold, its unseen subgroup bounds accuracy by 1/|NG| plus a term vanishing as the joint walk mixes, and realizing the quotient attains the limit.Commutative mechanisms are capped by 1/|[G, G]|; derived and lower-central series give further limits for cascades and signatures.Source: §3.3 · p. 6
  • For A4, two cascaded commutative layers compose exactly, whereas signature readouts remain capped at 0.25 under the stated hypotheses.The group is solvable but not nilpotent, so no signature level decodes it exactly.Source: §3.3 · p. 6

4 FROM EXACT CONSTRUCTIONS TO LEARNED-MODEL GUARANTEES

Finite prototype certificates connect approximate learned recurrences to correctness over explicit word-length horizons. Their guarantees depend on residual error, readout margins, and real-arithmetic assumptions.

4 FROM EXACT CONSTRUCTIONS TO LEARNED-MODEL GUARANTEES

  • A finite prototype certificate bounds transition residuals and requires each prototype’s readout label to persist throughout a neighborhood.The certificate checks all prototype transitions and readout neighborhoods; for linear logits, the neighborhood radius is set by the minimum margin divided by the norm of the weight difference.Source: §4 · pp. 6–7
  • If every transition residual is at most ϵ and Lϵ < η, every word of length L is decoded correctly; when ϵ = 0, correctness holds for every word.Orthogonality limits accumulated distance from the prototype to at most the word length times ϵ.Source: §4 · pp. 6–7
  • The stated guarantees use real arithmetic with stored matrices; floating-point execution also requires arithmetic-error bounds.If transition norms are bounded by 1 + κ, the residual term must account for the resulting growth.Source: §4 · p. 7

5 EXPERIMENTS

Experiments test representability, learning, and certification across finite groups, a below-width cover, and parsed relational tasks. Results depend on width and readout, while the usefulness of isometric transports varies by task.

5.1 WIDTH AND VERIFICATION

  • Every group in the eight-group ladder fit at its predicted width, but raw length-300 accuracy varied, including 0.549 for D50, 0.746 for Q8, and 0.843 for S5.The ladder supports representability and extraction, not uniformly reliable optimization.Source: §5.1 · p. 7
  • At d = 3, 17/17 fitting A4 and D4 runs passed finite evaluation, compared with 7/18 at d = 6.A separate certificate cohort had all six seeds per group exact through length 1000; raw weights received finite horizons, not all-word guarantees.Source: §5.1 · p. 7

5 EXPERIMENTS

  • Finite evaluation is not all-word exactness: a Q8 run that passed length 1,000 failed at 2,000, while observed errors can occur beyond certified horizons.The reported evaluation criterion uses 2,000 fresh words per length; certificates bound raw weights to explicit horizons.Source: §5.1 · p. 7, §5 · p. 7, Figure 2 · p. 8

5.2 A LEARNED COVER BELOW FAITHFUL WIDTH

  • On C3 × A4, sign-blind width-four attention learned certified double covers in 6 of 48 seeds, while monotone width-four attention succeeded in 0 of 48.At width five, monotone attention succeeded in 13 of 36 pooled seeds; the confirmatory cosine variant reproduced cover learning on declared seeds.Source: §5.2 · p. 8
  • On the same 72 frozen cover states, affine cross-entropy was 3.5835 = log 36, while quadratic and squared-prototype heads reached accuracy 1.000 and cross-entropy at most 10^-5.The intervention changes only the head, isolating what the decoder can read from the states.Source: §5.2 · p. 8

5.3 TRANSPORT CONSTRAINTS ON PARSED RELATIONAL TASKS

  • The parsed-task comparisons concern composition given extracted relations, not text understanding or text-level state of the art.StepGame tests hops 6–10 after training through 5; CLUTRR tests hops 4–10 after training on 2- and 3-hop examples.Source: §5.3 · p. 8
  • On CLUTRR at 4–10 hops, unconstrained matrix scans score 0.848 OOD against the transformer’s 0.436, while orthogonal scans score 0.202.On StepGame, an additive composer scores 0.753 against the transformer’s 0.481; within a separate probe family, rotations raise the mean from 0.73 to 0.82.Source: §5.3 · pp. 8–9

6 RELATED WORK AND SCOPE

The paper places transported relational operators alongside prior local-frame, geometric, and composition models, while distinguishing its readout-dependent finite-group results from related width analyses.

6 RELATED WORK AND SCOPE

  • Local-frame and transported-feature methods include GET, Neural Sheaf Diffusion (Bodnar et al., 2022), Bundle Neural Networks (Bamberger et al., 2025), and geometry-aware attention (Miyato et al., 2024).The paper also situates bilinear scores and path products in earlier knowledge-graph models (Nickel et al., 2011; Guu et al., 2015; Sun et al., 2019; Xu and Li, 2019).Source: §6 · p. 9
  • Unlike Larson (2026), which trains toward a prescribed representation under a fixed readout, the paper’s theorems prescribe no target and show convex-readout exactness forces a faithful representation.It distinguishes its finite-group cover result and compact-closure necessity from concurrent readout-dependent separations.Source: §6 · p. 9
  • The paper frames orthogonal recurrences, state-tracking models, and group division as related approaches to composition, while noting that transformers can compute noncommutative composition by other mechanisms.It makes no claim that transformers are order blind.Source: §6 · p. 9

7 DISCUSSION AND LIMITATIONS

The discussion ties model behavior to per-entity frame symmetry, transport structure, width, and readout, while delimiting the results to explicit model classes and settings. The appendices give fuller statements and proofs for the paper’s main results.

7 DISCUSSION AND LIMITATIONS

  • The framework’s scope is explicit: its behavior is governed by precomputed integers within the stated model class, and whether that governance extends beyond transport models remains open.The paper identifies small finite groups, uncertain optimization, parsed-relation benchmarks, and the unbridged distance to transformer stacks as limitations.Source: §7 · p. 9
  • On the same 72 frozen states, the cross-entropy-optimal affine head is uniform and the quadratic head exact; the paper leaves open whether this readout dependence extends beyond transport models.In training, cover learning occurred under sign-blind readouts in 6 of 48 seeds and under monotone readouts in 0 of 48.Source: §7 · p. 9
  • With exact matched anchors, orthogonal transport is identified only on the anchored subspace; inferring transports from incomplete relational evidence is not evaluated.The experiments supply relation operators.Source: §7 · p. 9

A FULL STATEMENTS AND PROOFS

  • The appendix maps the main-text theorems on forced scores, endpoint carriers, width, readout dependence, ceilings, and certificates to their fuller statements and proofs.These references identify where the paper expands each result’s hypotheses and supporting arguments.Source: A FULL STATEMENTS AND PROOFS · p. 12

A.1 SETTING AND CONVENTIONS

  • Per-entity frame freedom is a modeling premise: rotating one entity’s coordinates alone leaves its content unchanged, and the paper adopts this as an inductive bias.The symmetry acts independently on each entity’s attached data.Source: §A.1 · p. 12

A.2 RELATIONS ARE SUPPLIED, NOT COMPUTED

  • With vertex data alone, invariants contain only per-entity Gram matrices and invariant attributes, not information about relative frame orientations.Information about relative orientation must enter through transports or frames; the premise is not implied by fresh names.Source: §A.2 · pp. 12–13

A.3 FORCED FORM

  • For orthogonal transports, invariant scores depend on the endpoint norms and transported inner product, while equivariant messages lie in the span of the receiver vector and transported neighbor vector.The stated forms are tight: every such score and span element is realizable.Source: §A.3 · p. 13
  • For general linear transports, the invariant and equivariant forms use additional powers of TT^⊤ acting on the receiver vector and transported neighbor vector.The orthogonal case TT^⊤ = I reduces to the simpler forced form.Source: §A.3 · p. 13

A.4 THE CARRIER THEOREM: WHAT A TRANSPORT CAN BE COMPUTED FROM

This section characterizes what endpoint channels and orthogonal transports permit, and how flatness, transported anchors, holonomy, and readout geometry constrain composition width. It distinguishes representational bounds from readout-dependent decoding and accuracy ceilings, and gives finite verification procedures for learned transports.

A.4 THE CARRIER THEOREM: WHAT A TRANSPORT CAN BE COMPUTED FROM

  • Endpoint-only transports have rank at most min(mi, mj, d); one channel per entity yields rank at most one, while a full-rank transport requires d independent channels at each endpoint.With no channels at either endpoint, invariant data alone cannot produce a nonzero transport; a single-vector rank-one transport is never orthogonal when d ≥ 2.Source: §A.4 · p. 14
  • Transported anchors span the gauge-equivariant vector outputs available at an entity; holonomy can expand one anchor to a full frame when its orbit spans the fiber.For a single anchor, the transported span is the holonomy orbit; a cyclic anchor spans the whole fiber, and holonomy words can synthesize an equivariant frame under the stated separability or irreducibility conditions. These restrictions concern endpoint-only inputs: supplied graph transports can bring additional covariant channels to an endpoint.Source: §A.4 · pp. 14–15

A.5 FLAT LIMIT AND HOLONOMY

  • A connection is flat exactly when its cycle holonomies are trivial; in that gauge, forced scores and messages reduce to attention in one global frame.Standard attention has identity transports and no explicitly parameterized connection; this does not by itself bound a standard stack’s ability to compute relational composition by other means.Source: §A.5 · p. 15

A.6 WIDTH LAW FOR CONVEX-CELL READOUTS

  • Convex-cell readouts require width at least dmin(G), the least dimension of a faithful real representation; at that minimum, exactness forces a faithful representation.A faithful representation at dmin(G), with a generic initial state and nearest-prototype readout, is sufficient. Without assuming a finite latent group, the same threshold conclusion holds for exactness on all positive words, but finite tests alone do not imply exactness.Source: §A.6 · pp. 16–17

A.7 READOUT-GRADED WIDTH

  • Readout choice changes the width bound: linear argmax requires dmin(G), arbitrary readouts require dcov(G), and wany ≤ wquad ≤ wlin can be strict.Here dcov(G) is the least dimension admitting a finite orthogonal group that surjects onto G; exact composition may therefore decode a cover at smaller width than convex-cell readouts allow.Source: §A.7 · p. 18
  • Readout choice changes width thresholds: the additive gap between linear and quadratic widths is unbounded, while widths coincide when dmin(G) ≤ 4.C3 × A4 has faithful width five and quadratic width four; A4 × A4 at width four is not exact with a linear readout but can be decoded from xx^T. Projective width need not remain a threshold at larger dimensions, although exact realizability is monotone by padding.Source: §A.7 · p. 19
  • On a balanced antipodal cover, every affine-softmax cross-entropy minimizer predicts uniformly, giving accuracy 1/M; a quadratic head can decode the cover exactly.The affine result assumes uniform sampling over signs within each labelled orbit; it does not predict joint backbone–head training or finite-length distributions unbalanced across lifts.Source: §A.7 · p. 19

A.8 CEILINGS

  • Asymptotic accuracy is capped at 1/|NG|, and width d further caps it at |Qd|/|G|; exact decoding requires dcov(G) ≤ d.The general ceiling comes from the part of the target group invisible to the joint target–latent process; the width bound is attained by a lookup readout on a cover.Source: §A.8 · pp. 20–21
  • Commutative mechanisms have asymptotic ceiling 1/|[G, G]|, attained by abelianization; at finite length their multiset-based accuracy bound cG(L) is exactly computable by integer dynamic programming.The stated convergence of cG(L) to that ceiling is conditional on conditional mixing, not a theorem for all groups and alphabets.Source: §A.8 · p. 20
  • A cascade of k commutative layers has asymptotic ceiling 1/|G(k)|, and the ceiling is one exactly when k reaches G’s derived length.A suitable cascade can attain the bound, but this is a statement about the model class, not every fixed choice of layer sizes.Source: §A.8 · pp. 20–21
  • With independent letter increments, level-k signatures decode exactly iff G is nilpotent of class at most k; unwired permutation-invariant transformers are multiset-based and bounded by cG(L).The signature accuracy ceiling tends to 1/|γk+1G| only under the stated conditional-uniformity hypothesis; dependent increments can identify distinct letters.Source: §A.8 · pp. 21–22

A.9 HORIZON AND CERTIFICATE

  • A verified snapped representation is exact at every length, while a learned model is certified through lengths L < η/(δ∥x0∥).Verification checks orbit labels, generator deviation, and a positive readout robustness radius; the finite-horizon guarantee requires all three.Source: §A.9 · p. 22
  • A verified prototype automaton certifies correctness whenever Lϵ < η, but approximate orbit search alone proves neither finite order nor exactness of learned dynamics.The certificate requires a finite total transition map with every transition-residual, label-consistency, and readout-radius check completed; snapping search and alignment are not themselves proof.Source: §A.9 · pp. 22–23

B EXPERIMENTAL PROTOCOLS AND COMPLETE REPORTED RESULTS

The appendix distinguishes finite evaluation, extracted snapped models, and certificates for the learned recurrence.

B EXPERIMENTAL PROTOCOLS AND COMPLETE REPORTED RESULTS

  • Finite evaluation tests accuracy on fresh words through length 1,000, while a raw-model certificate bounds the stored recurrence’s horizon in real arithmetic.A snapped model is a proposed exact representation or cover whose algebra and readout require separate checks.Source: B EXPERIMENTAL PROTOCOLS AND COMPLETE… · p. 23

B.1 WIDTH LADDER AND THE MINIMAL-WIDTH COMPARISON

The width experiments support representability at predicted widths and show stronger finite-evaluation results at minimal width than at twice that width, while raw generalization and optimization remain variable.

B.1 WIDTH LADDER AND THE MINIMAL-WIDTH COMPARISON

  • The ladder used training words of length at most ten and evaluated fresh words through length 1,000; extraction used known target-group representations rather than discovering groups without labels.The controlled A4/D4 comparison used anchor, fit, and polish stages; its certificate cohort comprised six width-three seeds per group.Source: §B.1 · p. 24
  • Table 2 reports eight ladder rows, with widths measured in O(d); the recorded row summaries are not averages over a common number of successful seeds.Source: Table 2 · p. 24
  • D50’s snapped model scored 1.000 through L = 1,000 with extraction residual about 0.18, while S5’s residual was 0.018 and its snapped-prototype readout scored 1.000.These are extraction results, not accuracy claims about the unmodified learned recurrence; successful optimization was uncommon in some cells.Source: §B.1 · p. 24
  • For Q8, seven width-four runs stayed near 0.50 ± 0.02 through L = 300, while a width-four run passing length 1,000 failed at length 2,000.The evidence is finite-horizon behavior, not an exact representation below the faithful-width bound.Source: §B.1 · p. 24
  • 17/17 fitting runs at d = 3 passed finite evaluation, versus 7/18 at d = 6; these conditional counts do not establish universal exactness or failure at greater width.Three nonfitting D4 runs at d = 3 remained at 0.50 across evaluated lengths.Source: §B.1 · p. 24
  • Under independent entity-frame rotations, transported scoring retained 1.000 accuracy, while fixed-frame bilinear scoring fell from 1.000 to 0.192 and an MLP from 1.000 to 0.159.Chance was 0.167; the shared-frame control gave 1.000 for all three scores.Source: §B.1 · p. 25
  • Every A4 and D4 certificate-cohort seed at d = 3 was exact through length 1,000, and snapped generators closed to a group with defect at most 6 × 10−13.Raw-model certificate horizons were 71–276 for A4 and 100–222 for D4; these were one to two orders of magnitude before first observed error.Source: Figure 3 · p. 25, §B.1 · p. 25

B.2 SPINOR CELLS AND ATTENTION-SCORE PARITY

The spinor experiments test whether attention-score parity affects which group structures can be learned, and distinguish raw finite evaluation from extracted candidate models and their certificates.

B.2 SPINOR CELLS AND ATTENTION-SCORE PARITY

  • The registered cells used four head types and twelve seeds per cell; exploratory cosine variants changed normalization, score initialization, and training budget, so they were not confirmatory replications.The confirmatory rerun used predeclared seeds and criteria, byte-identical code, and excluded no runs.Source: §B.2 · pp. 25–26
  • In the confirmatory replication, sign-blind width four fit and was exact in 3/24 runs, monotone width four in 0/24, and monotone width five in 7/24.Pooled with the sweep, the counts were 6/48 sign-blind and 0/48 monotone at width four, and 13/36 monotone at width five.Source: §B.2 · p. 26
  • The successful sign-blind width-four runs certified a 72-state double cover with kernel {±I}, while the successful width-five monotone runs were faithful.The width-four raw models passed finite evaluation through L = 1,000 but drifted later; snapped-model correctness requires algebraic and readout verification.Source: §B.2 · p. 26
  • The two registered width-four quadratic runs identified a candidate double cover C3 × 2T → C3 × A4, but approximate closure and small residuals alone do not prove finiteness.For the target double transposition b, the raw residual satisfied ∥R2 b + I∥F ≤ 3.6 × 10−3; after snapping it was about 10−15.Source: §B.2 · p. 26
  • Three width-five affine runs yielded faithful extracted models despite failing the raw length-1,000 criterion; none of twelve width-four quadratic A4 × A4 runs fit.The width-six affine control had two evaluation successes, so the results constrain the optimization claim.Source: §B.2 · p. 26
  • In the identified covers, the C3 letters and involution act in opposite su(2) factors, and a three-cycle’s mixed components are compatible with an A4-to-C3 abelianization twist.This diagnostic is more specific than accuracy, but the universal readout-width theorem does not require every approximate trained solution to use this parameterization.Source: §B.2 · pp. 26–27
  • A proposed snapped-model certificate checks a finite labeled orbit, exact transitions, an initial state, and readout correctness; these checks imply correctness by induction.Floating-point residuals are diagnostic evidence, not exact arithmetic proofs; a raw-recurrence certificate instead bounds a finite horizon.Source: §B.2 · p. 27

B.3 READOUT INTERVENTION ON FIXED COVER STATES

The fixed-state intervention isolates the effect of changing the decoder on cover states, contrasting affine and quadratic heads while keeping the dynamics fixed.

B.3 READOUT INTERVENTION ON FIXED COVER STATES

  • On the same cover states, quadratic and squared-prototype heads reached accuracy 1.000 and cross-entropy at most 10−5, while fitted affine heads attained cross-entropy 3.5835.The affine logit variation was about 10−4 on snapped states and 10−2 on raw states.Source: §B.3 · pp. 27–28
  • A selected-lift affine head scored 1/2, a constructive lower bound rather than a demonstrated optimum; the table separates achievable accuracy from loss minimization.On an exact balanced antipodal orbit, affine cross-entropy is minimized by uniform predictions, but this theorem does not establish the same optimum on raw approximate states.Source: §B.3 · pp. 27–28, Table 4 · p. 27

B.4 PARSED PUBLIC BENCHMARKS AND CONTROLS

On parsed relational benchmarks, isometries help StepGame but substantially harm CLUTRR. The evidence tables distinguish constructed witnesses from certified or snap-certified runs.

B.4 PARSED PUBLIC BENCHMARKS AND CONTROLS

  • The benchmarks use common parsed relational interfaces, so these experiments assess composition given extracted relations, not text understanding or text-level state of the art.StepGame’s parsed-interface audit covered more than 65,000 examples; its 100% agreement applies only to the clean strata tested.Source: §B.4 · p. 28
  • 27.2 percentage points: on StepGame, the additive composer’s mean OOD accuracy was 0.753 versus 0.481 for the transformer.The recorded difference was at least 23 points at every evaluated OOD hop count.Source: §B.4 · p. 28
  • 64.6 percentage points: on CLUTRR, the orthogonal control’s mean OOD accuracy was 0.202 versus 0.848 for the unconstrained scan.The orthogonal control fit its training data at about 0.98.Source: §B.4 · p. 29

B.5 EVIDENCE TABLES

  • Table 6 reports numerically verified constructed cover witnesses at width 4; Table 7 gives one row per certified or snap-certified run, listing constructed, snapped, raw-certified, and finite-evaluation evidence.Source: Table 6 · p. 29, §B.5 · p. 29, Table 7 · p. 29

B.6 SUPPORTING DEPTH, SIGNATURE, TRANSFORMER, AND GRAPH EXPERIMENTS

Supporting experiments contrast constructed algebraic witnesses with learned performance across cascades, signatures, order-blind and causal transformers, and a knowledge-graph diagnostic. Their ceilings and conclusions depend on the stated models, distributions, and protocols.

Depth experiments

  • Constructed cascades for D4, A4, and S4 scored 1.0000 through length 1,000, while learned cascades did not reach these witnesses; learned A5 accuracy was 0.0168 versus its 1/60 ≃ 0.0167 limiting value.The depth-k ceiling applies to the finite-group cascade model under stated joint mixing assumptions, not every diagonal state-space architecture.Source: §B.6 · p. 30, Table 8 · p. 30

Signature controls

  • A level-two D4 signature witness scored 1.000 through length 1,000, while learned A4 signature heads scored 0.068–0.091, near chance 1/12 and below 0.25.The 0.25 ceiling requires the stated factorization and distributional or mixing assumptions; it is not a blanket bound on arbitrary nonlinear functions of truncated signatures.Source: §B.6 · p. 30

Scope of bounds

  • The paper distinguishes exact-decoding obstructions from partial-accuracy ceilings, whose hypotheses include factorization through a finite quotient and relevant distributional or mixing assumptions.Finite-length ceilings require a mixing correction; normalization can also change information available to learned heads.Source: §B.6 · p. 30

Order-blind and causal transformers

  • At L = 10, order-blind bag accuracies were 0.43 for D4 and 0.34–0.37 for A4, below their respective ceilings of 73/128 ≃ 0.5703 and ≃ 0.4420.The ceiling depends on the alphabet and sampling distribution; causal models exceeded these bag ceilings in distribution, but nine out-of-range runs were near chance at tested longer lengths.Source: §B.6 · p. 31

Knowledge-graph null result

  • On FB15k-237, loop-residual norm correlations with Hits@10 were +0.055 raw and −0.039 filtered, so the registered predicted association was not supported.The exploratory route-divergence correlation of +0.345 concerns a different quantity and does not establish a causal mechanism.Source: §B.6 · p. 31

C REPRODUCIBILITY, USE OF AI, AND ETHICS

The appendix gives proofs and specifies the experimental protocols and evidence criteria. The authors report extensive generative-AI assistance, human review, and no new human-subject experiments or personal-data collection.

Reproducibility

  • Appendix A gives full proofs, while Appendix B specifies training stages, decoder families, finite-evaluation criteria, and certificate-verification procedures.Registered and exploratory cells, including unsuccessful cells, are reported separately; real-arithmetic guarantees are not validated floating-point execution bounds.Source: C REPRODUCIBILITY, USE OF AI, AND ETHICS · p. 31

Use of AI

  • Generative AI assisted with theory, proofs, hypotheses, experimental methods, implementation, interpretation, writing, figures, literature work, and reference formatting; the authors report reviewing all AI-assisted work.The authors state datasets were sampled by code and that theorem statements, proofs, code, reported numbers, and citations underwent specified checks.Source: C REPRODUCIBILITY, USE OF AI, AND ETHICS · p. 31

Ethics

Loading 2610.34945v1…