PaperlayerView exact revision on arXiv

Source-linked AI summary

The Rank the Task Demands: A Causal Rank Law for Matrix Memories Trained on Group Composition

Samuel Larson

arXiv:2609.12259v1cs.LG

TL;DR

The paper asks whether gradient descent recruits the representational dimension required by a task’s algebra, and whether that dimension is causally load-bearing. It trains matrix memories on group composition under a hard single-state bottleneck and a fixed decoder. Across five groups, recruited rank tracks minimal faithful representation dimension, the matched-dimension S4/A5 pair is equivalent, and force-rank tests find sufficiency at the target dimension.

  • Problem

    Prior work treated matrix-memory rank descriptively or through hand-built constructions, leaving open whether learning recruits the minimal dimension fixed by task algebra and whether that dimension is causally load-bearing.

    Source: §1 · p. 1

  • Method

    The study trains a single matrix state toward a chosen minimal faithful representation of finite-group composition and evaluates rank and held-out recovery with a fixed readout.

    Source: §2 · p. 2

  • Results

    Across five groups, recruited rank tracks dmin (ρ = 0.9747); S4/A5 ranks are equivalent within ±0.5 rank-units, and rank dmin suffices in all five groups.

    Source: §1 · p. 2, §4 · p. 4, §5 · pp. 4–5

  • Takeaways & Limitations

    Within this target and scoring setup, measured effective rank tracks representation dimension, and the matched-dimension S4/A5 comparison establishes equivalence within the pre-registered tolerance.

    Source: §4 · pp. 3–4

    Results are limited to sub-1M-parameter synthetic testbeds, not pretrained language models; the razor uses four seeds per group, and S3’s confirmation is directional because of a soft-convergence flag.

    Source: §6 · p. 6

Abstract

from arXiv · show

Matrix-valued memories make rank the natural budget of a learned representation: the number of independent directions a state spans bounds what it can bind, compose, and track. We report causal evidence, on a group-composition testbed trained under a hard single-state bottleneck with a fixed decoder that cannot launder rank, that gradient descent recruits precisely the rank the task's algebra demands. A companion paper [Larson, 2026a] establishes the analogous recruitment and causal necessity pattern on a $K$-pair associative-binding testbed, where exact recovery provably requires state rank at least $K$; this paper inherits that instrument and extends the rank law from a scalar capacity bound to a representation-theoretic one. We train toward chosen minimal faithful reference representations embedded in larger matrices. On group-composition state tracking over five finite groups spanning the solvable/non-solvable divide, the recruited rank equals the group's minimal faithful real representation dimension $d_{\min}$ (Spearman $\rho = 0.9747$, the design's tie-capped maximum), the dimension-matched solvable/non-solvable pair $S_4$/$A_5$ is statistically equivalent under a pre-registered test, and a pre-registered force-rank test separates a guaranteed similarity ceiling from empirical recovery at the target dimension: one rank below $d_{\min}$, cosine similarity is capped by the target's tied unit spectrum at $\sqrt{(d_{\min}{-}1)/d_{\min}} \le 0.894$, below the $0.9$ threshold in every group by construction, with observed cells at 86-95% (mean 91%) of that ceiling; at $d_{\min}$, not guaranteed a priori, recovery clears the pre-registered anchor-relative bar at four seeds per group in all five groups. Within this testbed, measured effective rank tracks representation dimension; the matched-dimension $S_4$/$A_5$ comparison establishes equivalence within the pre-registered tolerance.

1 Introduction

The paper asks whether gradient descent recruits the minimal representational dimension fixed by a task’s algebra, and whether that dimension is causally load-bearing. On group-composition state tracking, recruited rank follows minimal faithful representation dimension, and a force-rank test finds recovery at that dimension across five groups.

1 Introduction

  • The study tests whether gradient descent recruits the task’s minimal representational dimension and whether that dimension is causally load-bearing.Prior work treated rank descriptively or through hand-built constructions, leaving this question open.Source: §1 · p. 1
  • The test uses a hard single-state bottleneck and scores cosine similarity against the true continuous target at a 0.9 threshold.A gradient blank-out test verifies that the decoder reads only one matrix state.Source: §1 · p. 1
  • Across five groups, recruited rank tracks minimal faithful real representation dimension, and the force-rank test finds sufficiency at that dimension in all five.One rank below, maximum cosine is capped at 0.894, below the 0.9 recovery threshold by construction.Source: §1 · p. 2

2 Tasks, Models, Instrument, and the Binding Foundation

The paper studies supervised learning of finite-group word composition into a chosen minimal faithful reference representation embedded in a larger matrix. A fixed readout measures rank and recovery, extending an instrument previously used for associative binding.

2 Tasks, Models, Instrument, and the Binding Foundation

  • A word sampled as a random walk on a finite group’s Cayley graph must map to the product of its generators under a pinned orthogonal reference representation.The five groups are S3, S4, A5, S5, and A6, with dmin values 2, 3, 3, 4, and 5; S4/A5 share dimension but differ in solvability.Source: §2 · p. 2
  • Training targets a chosen minimal faithful representation embedded in a larger matrix, so the studied rank requirements apply to this target and scoring setup.They are not claims about storage across all possible encodings or decoders.Source: §2 · p. 2
  • A Transformer maps each word to one matrix state, and a fixed readout with no learned weights reads only that state.The state has dmin + 2 dimensions, leaving room for over-recruitment; decoder access is blank-out verified.Source: §2 · p. 2
  • The instrument estimates a dominant dmin-subspace and scores held-out, gauge-fitted recovery; rank remains invariant under the fitted gauge.For exact binding of K independent key-value pairs, the stated matrix factorization implies rank at least K; the companion paper reports learned rank recruitment and causal necessity on that task.Source: §2 · p. 2

3 The Rank Law on Group Composition, Observed

On the five-group word problem, measured restricted effective rank tracks the group’s minimal faithful representation dimension. The observed association reaches the maximum expressible in this design, whose S4/A5 tie limits the correlation.

3 The Rank Law on Group Composition, Observed

  • Per-group mean restricted effective rank is within 4.9–10.2% of dmin, and all 19 seeds fall inside the pre-registered [0.7, 1.3] · dmin band.The band is corroborating evidence; its upper half is non-binding by construction.Source: §3 · p. 3
  • Spearman correlation between recruited rank and dmin is ρ = 0.9747, the design’s tie-capped maximum because S4 and A5 share dmin.Under the exact permutation null, P(ρ ≥ 0.8) = 8/120 ≈ 6.7%; the pre-registration treats this leg as corroborating rather than independently decisive.Source: §3 · p. 3

4 Equivalence at Matched Dimension

A pre-registered equivalence test compares effective rank for S4 and A5, which share dmin = 3 but differ in solvability. The observed ranks meet the study’s ±0.5 rank-unit equivalence criterion.

4 Equivalence at Matched Dimension

  • The matched-dimension contrast tests whether effective rank is equivalent across S4 and A5 despite their opposite solvability.The paper frames representation dimension as an axis distinct from solvability.Source: §4 · p. 3
  • Figure 2 plots held-out crosscheck rec@0.9 against force-rank k relative to dmin, alongside the unconstrained anchor and the 0.9×anchor bar.S3’s four seeds unanimously show zero below dmin; the other groups’ four-seed sufficiency verdicts are in Table 1.Source: Figure 2 · p. 4
  • S4 and A5 differ by +0.019 rank-units, and the pre-registered two one-sided tests declare equivalence within ±0.5 rank-units.The comparison uses five seeds per group and a design-record power simulation; the test reliably rejects equivalence at a true gap of 1.0 rank-units.Source: §4 · p. 4

5 The Causal Razor

The force-rank intervention tests whether rank below dmin is insufficient and whether rank dmin recovers across all five groups. Below dmin, geometry forces failure at the 0.9 threshold; at dmin, all five groups meet the pre-registered anchor-relative criterion.

5 The Causal Razor

  • The encoder is spectrally truncated throughout training at k∈{dmin−1,dmin,dmin+1}, against a zero-padded target of rank exactly dmin and an unconstrained anchor.The below-dmin arm is an integrity control: any recovery would indicate an instrument leak.Source: §5 · p. 4
  • At k=dmin−1, rec@0.9 is 0.000 in every group; at k=dmin, all five groups confirm against their pre-registered bars.The rank-(dmin−1) cosine ceiling is ≤0.894, while observed cells reach 86–95% of that ceiling (mean 91%).Source: §5 · pp. 4–5, Table 1 · p. 5
  • Identity-padding confounds the razor by giving every capped arm a free rank-2 loss reduction; the reported intervention therefore uses zero-padding.Source: §5 · pp. 4–5

6 Related Work and Limitations

Prior work studies group composition, sequential state tracking, and state-rank dynamics, but does not connect learned state rank to a task’s minimal faithful representation dimension. The paper’s results are limited to small synthetic testbeds, with seed and convergence qualifications.

Related Work

  • Prior binary-operation studies do not measure sequential state rank, while sequential-composition work studies irreducible representations, expressivity, or grokking rather than a dmin necessity threshold.The paper measures and manipulates rank itself, with required dimension fixed by dmin.Source: §6 · p. 5
  • Observational state-rank studies lack task algebra fixing a required dimension, and the effective-rank estimator used here is standard rather than new.Earlier work reads low rank as under-used capacity; this paper treats it as the rank demanded by the task.Source: §6 · p. 5
  • The measured dimension axis is distinct from solvability or series-length accounts of expressivity, and the paper frames its result as a narrower algebra-driven instance of shared representation convergence.The paper also distinguishes its continuous-target cosine threshold from associative-memory results using argmax decoding.Source: §6 · p. 6

Limitations

  • Results concern sub-1M-parameter synthetic testbeds, not pretrained language models; razor cells use four seeds per group.S5 uses a 20k-step pin after a pre-registered re-test, and S3’s confirmation is directional because of a soft-convergence flag.Source: §6 · p. 6

A The Five Groups and Reference Representations

The task set comprises five finite groups with minimal faithful real representations; S4 and A5 are the matched-dimension pair on opposite sides of the solvability divide.

A The Five Groups and Reference Representations

  • The reference representations are orthogonal by construction and verified faithful, meaning injective on G, at build time.Source: Table 2 · p. 8
  • S4 and A5 form the matched-dmin pair despite lying on opposite sides of the solvability divide.Source: Table 2 · p. 8
  • Table 2 lists five task groups and identifies dmin as the minimal faithful real representation dimension.Source: Table 2 · p. 8

B Per-Seed Observational Rank

Restricted effective rank tracks dmin across groups, while whole-matrix measurements show that the recruited rank depends on the target’s padding. An observed S4 razor-versus-anchor inversion remains an unconfirmed hypothesis.

B Per-Seed Observational Rank

  • Every seed falls within the pre-registered [0.7,1.3]·dmin band; Spearman ρ=0.9747 reaches the design’s tie-capped maximum, and S4/A5 differ by 0.019 rank-units.Source: Table 3 · p. 8
  • Mean restricted-rank deviations from dmin range from 4.9% for S4 to 10.2% for S5.The reported deviations are S3 6.1%, S4 4.9%, A5 5.6%, S5 10.2%, and A6 5.3%.Source: B Per-Seed Observational Rank · p. 8
  • S4’s seed-0 razor cell exceeds its unconstrained anchor by 0.15, but the proposed skipped-dimensions explanation is unconfirmed.Across four seeds, the inversion is strict in 2 of 4 and tied in the other 2.Source: B Per-Seed Observational Rank · p. 8

C Whole-Matrix Effective Rank

  • On the eye-padded target, whole-matrix effective rank recruits dmin+2; on the zero-padded target, it recruits dmin and leaves two spare dimensions unused.The respective reported means are 3.77/4.84/4.78/5.51/6.72 against dstate=4/5/5/6/7, and 1.82/2.82/2.83/3.77/4.67 against dmin=2/3/3/4/5.Source: Table 4 · p. 8
  • In the zero-padded razor cells, whole-matrix effective rank stays at or below the imposed cap k and, for k≥dmin, matches the unconstrained anchor.Source: Table 5 · p. 9

D Instrument Details and the Two Defects

The force-rank ceiling is below the 0.9 recovery threshold at dmin−1, while recovery at dmin is empirical; several instrument details qualify how the rank evidence should be read.

D Instrument Details and the Two Defects

  • Centering is load-bearing: the uncentered lens scores a flawless synthetic model 0.705, versus 0.9996 with centering on identical data.For an orthogonal target, uncentered covariance is isotropic and carries no subspace information.Source: D Instrument Details and the Two Defects · p. 9
  • Scoring only words of length L≥2 changes per-group mean recovery cosine by at most 0.013 and per-seed cosine by at most 0.041, without group-selective divergence.The L=1 read is attention-degenerate.Source: D Instrument Details and the Two Defects · p. 9
  • The first 58-cell identity-padded sweep was inconclusive because I2 consumed two rank slots, leaving every capped arm below the representation’s dmin.The zero-padded causal wave was the registered fix; the initial arms otherwise trained near their rank-constrained ceilings, with two disclosed S5 optimization shortfalls.Source: D Instrument Details and the Two Defects · p. 9
  • Restricted effective rank cannot exceed dmin by construction, so only the pre-registered lower band bound of 0.7·dmin was empirically testable.Group means fall between 0.898 and 0.951·dmin.Source: D Instrument Details and the Two Defects · p. 9
  • At dmin−1, observed cosines reach 86.3%–94.9% of the geometric ceiling (mean 90.8%), which remains below rec@0.9 in every group.The ceiling itself is 0.707/0.816/0.816/0.866/0.894 for S3/S4/A5/S5/A6; dmin permits exact recovery but does not guarantee it.Source: D Instrument Details and the Two Defects · pp. 9–10
  • Stable rank preserves the group ordering (Spearman ρ=0.975) but gives 0.59–0.73·dmin, making the “equals dmin” point estimate entropy-estimator dependent.The causal razor caps algebraic rank and does not depend on the effective-rank estimator.Source: D Instrument Details and the Two Defects · p. 10

E S3 Per-Seed Causal Detail

  • In S3, k=dmin−1 gives rec@0.9=0 in all four seeds, while k=dmin clears the fixed seed-mean bar but each seed’s own bar in only two seeds.The seed-mean comparison is 0.5625 against the pre-fixed 0.495 bar; recomputing the bar from the four-seed anchor mean would make it 0.574.Source: Table 6 · p. 10

F Per-Seed Razor Detail for S4, A5, S5, A6

The four-seed extension confirms the dmin sufficiency result for S4, A5, S5, and A6 under the registered seed-mean comparisons.

F Per-Seed Razor Detail for S4, A5, S5, A6

  • S4, A5, S5, and A6 seed means at k=dmin are 0.6875, 0.6500, 0.7000, and 0.6750, each above its fixed bar and with own-bar clears in 4/4 seeds.At k=dmin−1, rec@0.9 is 0 in all 16 cells; S5’s original 8k pin under-trained it, so its registered rerun used 20k steps.Source: Table 7 · p. 11

G Soft Convergence at the Shortest Pinned Budgets

The shortest pinned S3 and S5 razor cells fall short of the convergence margin, qualifying how their sufficiency results should be read. The companion binding-task depth-21 probe repeats a target already tested at depth 5.

G Soft Convergence at the Shortest Pinned Budgets

  • All eight shortest-budget S3 and original-8k S5 cells miss the 0.92 convergence margin; S3 sufficiency is directionally consistent under soft convergence.S5’s 0.876–0.879 shortfall is larger than S3’s 0.900–0.914; at the revised 20k pin, all S5 cells pass at 0.959–0.981.Source: Table 8 · p. 11

H Depth-21 Periodicity Under the Single K-Cycle

  • The binding-task depth-21 composition probe shares its target exactly with the already-tested depth 5 under the single Hamiltonian K-cycle.The companion paper [Larson, 2026a] gives the periodicity analysis and depth-decay measurement.Source: H Depth-21 Periodicity Under the… · p. 11
Loading 2609.12259v1…