PaperlayerView exact revision on arXiv

Source-linked AI summary

Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

Burc Gokden

arXiv:2608.10288v1cs.LGcs.CL

TL;DR

The paper asks how input-learned bilinear attention generalizes the fixed Euclidean comparisons used in transformer language models. It combines structural proofs, reference-implementation checks, and measured diagnostics, establishing exact SDPA containment and reporting near-invariance of deductive outputs across inputs.

  • Problem

    The central question is how input-learned bilinear attention generalizes SDPA, whose fixed Euclidean comparisons underpin most transformer language models.

    Source: §1 · p. 4

  • Method

    The paper combines structural proofs, reference-implementation checks, and measured diagnostics to analyze input-generated bilinear attention and a conditional mechanism for invariance.

    Source: §1 · pp. 4–5

  • Results

    10^-6–10^-11 relative fluctuations characterize measured input invariance, while SDPA is contained exactly at G_LM = I.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Measured invariance suggests a cached bilinear operator could replace the nonlinear attention subnetwork at inference.

    Source: §1 · p. 4

    Transferring ideal-operator theorems to the floating-point implementation assumes unmasked scores remain many orders of magnitude above the negative mask value.

    Source: §3.2 · p. 14

Abstract

from arXiv · show

The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator $G_{LM}$, built from a positive tensor $A_{LM}$ by elementwise power laws. The architecture is fully specified, verified against pinned reference releases; claims are labeled theorem, conditional theorem, measurement, or conjecture. Unconditionally: PLGA contains SDPA exactly at $G_{LM}=I$; $A_{LM}$ and $A_P$ are strictly entrywise positive, with Perron-Frobenius structure on $A_{LM}$; the DAG regularizer has the NOTEARS walk-counting form and positivity obstructs exact acyclicity; and, under nonresonance (satisfied by standard rotary frequencies), a commutant criterion identifies which operators preserve relative-position dependence. An inference-collapse theorem: exact input invariance of deductive outputs collapses inference to generalized SDPA with a constant operator. Measured invariance: relative fluctuations of $10^{-6}$ and below; perturbation bounds quantify but do not certify cached inference; the assembled proxy misses the decoding margin. A conditional three-stage mechanism (rotary twirl, concentration, row-map contraction) is measured on a released checkpoint. Blockwise training and scoring under the global Gram are stated with explicit target exposure; on tested samples, block and sequential scoring select identical answers and agree on the published TruthfulQA probability-mass metric within $5\times 10^{-5}$ per item. Self-organized criticality enters as a phenomenological framework with an intrinsic order parameter; open claims become falsifiable conjectures. Selected proof cores are machine-checked in Lean 4.

1. Introduction

PLGA [10] [11] learns an input-generated bilinear attention operator and contains SDPA exactly at G_LM = I. Its inference-collapse theorem requires exact operator invariance, while invariance in trained models remains an empirical finding.

1. Introduction

  • PLGA [10] [11] contains SDPA exactly at G_LM = I, without establishing strict function-class separation or performance superiority.The construction maps an input-derived query Gram through positive A_LM, elementwise power laws, potential tensor A_P, and bilinear score operator G_LM. Intermediate tensors are exposed as deductive outputs alongside next-token probabilities.Source: §1 · p. 4
  • Across input changes, deductive outputs exhibit 10^-6–10^-11 relative fluctuations, reaching zero at floating-point resolution for the best models; exact invariance would permit replacing the nonlinear PLGA subnetwork with a cached tensor operator.Published cached-versus-uncached benchmark evaluations are unchanged [13], but use one-pass block scoring without a generation-time cache; substantive fidelity evidence comes from measured operator and logit agreement. The invariance mechanism remains conditional, with measurable diagnostics computed directly, while evaluated cache-fidelity constants do not close margins.Source: §1 · pp. 4, 5, 7
  • Rank one under an explicit singular-value tolerance, generator A has identical rows across heads, whereas A_LM remains strictly positive and generically near-full numerical rank.A’s sole nonzero eigenvalue is its common row sum and spectral radius; A_LM has Perron–Frobenius structure [13]. Reported floating-point-zero determinants [13] reflect determinant underflow for A_LM, not rank deficiency.Source: §1 · p. 4
  • An intrinsic fluctuation-based order parameter separates observed training phases [14]; generalization tracks critical-like dynamics, but self-organized criticality remains a phenomenological hypothesis.Maximum learning rate and warm-up steps serve as control parameters in the reported driving/dissipation dynamics.Source: §1 · p. 4

Epistemic conventions and machine-checked proofs

  • The paper verifies architectural definitions against implementations, distinguishes four epistemic categories, and machine-checks selected proof cores without validating surrounding unformalized claims.The categories are algebraic theorems, conditional theorems, empirical observations, and analogies/conjectures. Open claims concern invariant-operator rigidity, a spectral order parameter, and operator transfer across domains; selected proof cores are formalized in Lean 4.Source: §1 · pp. 4, 5, 7, 8

Related work

  • Unlike causal PaTH [46] and CCQ [22], PLGA shares one global-Gram-generated operator across rows, sacrificing historical-row prefix consistency.PaTH accumulates data-dependent positional transitions, while CCQ applies a prescribed contraction of running key covariance with a per-token scalar gate. PLGA instead uses a deep learned positive power-law generator inside ordinary softmax scores.Source: §1 · pp. 5–6
  • PLGA’s distinctive object is an explicit power-law head-space operator for regularization, measurement, and caching, rather than XCiT’s [8] mixing map or AoH’s [47] frozen form.XCiT mixes feature channels using its normalized cross-covariance matrix; PLGA retains token–token attention as the mixing mechanism. AoH has neither an input-conditioned generator nor an invariant-operator removal theorem.Source: §1 · p. 6
  • Under nonresonance, an inserted operator preserves offset-only rotary scores exactly when it commutes with position rotations; the mathematical contribution is application and synthesis.This PLGA-specific commutant criterion supports operator absorption, a head-space codimension count, and a commutant-residual diagnostic. The DAG regularizer is NOTEARS, and the remaining mathematical tools are classical.Source: §1 · p. 7

2. Preliminaries and Notation

The preliminaries represent contexts as embedded token graphs [11] [12] and distinguish learned global structure from inferred local tensors. They also separate ideal causal masking used in proofs from finite-penalty implementations.

2.1. Tokens, graphs, and the quantization/manifold duality.

  • 32,000 tokens form the reference vocabulary [12], and contexts of at most 1024 tokens [12] [13] [14] become matrices of dense token embeddings.Each context matrix has shape S × d_model, with one embedding vector per row.Source: §2.1 · p. 8
  • Contexts are weighted token graphs [11] [12], pairing discrete vocabulary and continuous feature-space descriptions while separating learned global parameters from inferred local tensors.The continuous feature space has dimension d_model, and its interaction structure is learned globally across instances.Source: §2.1 · p. 8
  • After pretraining at criticality, nominally local tensors become numerically instance-independent, an empirical result of [13] formalized in Section 5.These tensors include A_P, G_LM, and E_LM.Source: §2.1 · p. 8

2.2. Matrix conventions.

  • Matrix conventions distinguish ordinary products, elementwise operations, and row-wise softmax; causal-mask proofs use ideal masking rather than finite implementation penalties.Implementations use -10^9 or the dtype minimum, reproducing ideal masking only through floating-point underflow under ordinary score ranges. LayerNorm [2] acts on the feature axis with epsilon_LN = 10^-6 and learned affine parameters.Source: §2.2 · pp. 8–9

3. The Power Law Graph Attention Operator

This section defines PLGA’s input-generated bilinear operator, its recovery of SDPA and generalized SDPA, and the positive-tensor and row-map structures that characterize generator collapse without implying low-rank interactions. It also describes multi-head PLGA as block-diagonal scoring on separate subspaces, with a shared metric generator, distinct downstream parameters, and output mixing only after scoring.

3.1. Formal definition.

  • PLGA [11] [12] [14] constructs G_LM = a A_P + b_a using elementwise power-law potentials and a shared residual learner.The reference learner has eight residual units containing two complete SwiGLU blocks each; five full parameter matrices are head-specific. PLGA replaces CoulGAT’s [10] hand-engineered inverse-square adjacency with a learned interaction tensor.Source: §3.1 · pp. 9–10

3.2. Basic structural properties.

  • D_Q = Q^T Q is positive semidefinite with rank at most min(S, d_k), but A_LM and G_LM need not be symmetric or positive-definite.For S < d_k, downstream full-rank structure is generated by the learner or subsequent maps, not inherited as Gram rank. Such structure cannot be attributed to training without a trained-versus-initialized comparison.Source: §3.2 · pp. 10–11, §3.1 · p. 10
  • A_LM is strictly entrywise positive, making its elementwise power law smooth and giving it a simple positive Perron root dominating all other eigenvalue magnitudes.The Perron root has entrywise positive left and right eigenvectors.Source: §3.2 · p. 11
  • 62.5 of 64 is A_LM’s median numerical rank (range 1–64), despite floating-point-zero determinants; generator A, not A_LM, exhibits numerical rank-one collapse.These measurements concern the audited checkpoint. Entrywise nonlinearity and learned bias can produce full-rank A_LM from rank-one A, so its vanishing floating-point determinant carries no rank information.Source: §3.2 · p. 12
  • A shared row map makes the learner permutation-equivariant; input-independent identical generator rows occur exactly when this map is constant across all visited normalized-Gram rows.Constancy forces a common profile across heads, while a Lipschitz row map bounds generator changes by normalized-Gram changes. Observed cross-head equality alone does not prove local constancy.Source: §3.2 · p. 13
  • Ideal masked attention is row-stochastic with causal support, yet scores depend on all supplied rows; finite-mask implementations match ideal support only through conditional floating-point underflow.Value outputs are convex combinations of permitted values, and attention eigenvalues lie in the closed unit disk. Finite-logit implementations are dense in exact arithmetic; ideal-support equivalence requires sufficiently separated mask and unmasked-score magnitudes.Source: §3.2 · pp. 13–14

3.3. PLGA as attention with a learned bilinear form.

  • PLGA exactly recovers SDPA at G_LM = I; frozen operators give generalized SDPA, while input-generated operators generically yield a nonlinear score map.A frozen operator can be absorbed into the query projection without position-dependent transforms, but rotary embeddings generically prevent this equivalence. Degenerate parameters such as a = 0 make the operator constant.Source: §3.3 · p. 14

3.4. Multi-head structure as a local decomposition.

  • Multi-head PLGA splits model space into head-width subspaces, learns head-specific tensors, and concatenates outputs before linear mixing.The residual learner is shared across heads within each decoder layer.Source: §3.1 · p. 9, §3.4 · p. 14
  • The layer’s deductive state is block-diagonal, with mutually non-interacting components computing scores on their own subspaces [11].Source: §3.4 · p. 15
  • Heads share the metric-learner network Φres (Proposition 3.11), but W, bW, P, a, and ba are per-head tensors.Heads are differentiated after the common metric generator, not before.Source: §3.4 · p. 15
  • The output projection WO linearly mixes head outputs after scores are formed; it neither acts on nor conjugates the per-head score operators.Source: §3.4 · p. 15

4. The PLDR-LLM Architecture

A commutant criterion separates relative-position-only PLGA scores from absolute-position-aware scores, while RoPE also enters metric learning. The PLDR-LLM decoder [12] [13] preserves input sequence length.

4.1. Rotary position embeddings and a commutant characterization.

  • Under nonresonance, PLGA scores depend only on relative position exactly when the inserted bilinear operator G commutes with every RoPE rotation.Standard RoPE frequencies satisfy nonresonance; admissible operators have 2-by-2 blocks c_jI_2 + s_jJ_2. The proof uses distinct rotation eigenvalues, without requiring orbit density.Source: §4.1 · pp. 15–16
  • Generic learned G_LM operators violate the commutant condition, causing PLGA scores to carry absolute positional information through R_-n G_LM R_n.The characterization constrains an inserted bilinear operator, not projection reparameterizations, and requires no invertibility. Projecting trained G_LM onto the commutant is proposed as a diagnostic of demanded absolute positional structure.Source: §4.1 · pp. 15–16
  • RoPE [38] acts twice: on head-split queries and keys for bilinear scoring, and before density-operator construction as input to the metric learner.The position-dependent conjugation in metric learning supplies the rotary “twirl” analyzed later.Source: §4.1 · pp. 15–16

4.2. The full decoder map.

  • PLDR-LLM [12] [13] maps each length-S token sequence to S vocabulary-probability distributions, preserving sequence length.Length preservation is represented as an explicit predicate in the Lean development.Source: §4.2 · p. 16

Appendix C). The computation is as follows. With X(0) = LN

PLDR-LLM is causal when each generation call receives only the known prefix and consumes its final row, but its global query Gram does not generally guarantee historical-row prefix consistency. This distinction separates blockwise training and scoring from sequential likelihoods, while cached inference freezes the prompt-inferred metric generator.

4.3. Training objective and the DAG regularizer.

  • For T >= 3, exactly two of T-1 block-loss terms are guaranteed deployed conditionals; intermediate terms expose their targets through the global Gram.Cross-entropy is normalized over nonpadding labels, although padded rows still enter the Gram. Historical-row prefix consistency—not agreement of realized losses—justifies interpreting all rows as chain-rule factors; low block loss alone does not establish low autoregressive perplexity.Source: §4.3 · pp. 17–18
  • Strict positivity makes exact acyclicity unattainable for A_LM and A_P; reported zero DAG loss reflects floating-point underflow, not acyclic support.The NOTEARS characterization gives h(M) = tr exp(M ⊙ M) - d >= 0, with equality exactly for acyclic support. A_LM entries are at least 10^-9, while no positive DAG-loss floor is asserted for sign-unconstrained G_LM.Source: §4.3 · pp. 18–19
  • The DAG regularizer penalizes weighted closed walks through a normalized log heat trace, rather than establishing a causal DAG over feature coordinates.Spectral heat-trace and geometric closed-walk expansions describe the same penalty. Reported regularization recovers finite DAG losses where unregularized A_P and G_LM losses overflow, and can improve benchmark scores under the published one-pass block protocol [12].Source: §4.3 · pp. 18–20

4.4. Inference: KV-cache and G-cache.

  • ~3× caching speedup is reported [13]; G-cache adds no approximation beyond KV-cache's freezing of the prompt-inferred metric generator.Exact prefix recomputation defines the reference conditionals; deterministic maps from frozen A make G-cache exact relative to KV-cache. On the audited checkpoint, recomputed G_LM matched cached values bitwise and logit deviations stayed below decoding margins; published one-pass candidate benchmarks use no generation-time cache.Source: §4.4 · p. 20

Appendix C). The computation is as follows. With X(0) = LN

  • Final-row generation on exactly the known prefix is causal without invariance, collapse, or contraction assumptions.The metric learner uses the Gram of rotated queries, so positional phases enter its input. The released configuration enables attention-projection and unembedding biases, and the unembedding is not tied to the embedding.Source: Appendix C · p. 17, §4.5 · p. 21

4.5. The online generation contract and historical-row prefix consistency.

  • The global query Gram makes PLGA non-row-local: suffix tokens can alter historical outputs despite causal masking.Historical-row prefix consistency requires each parallel row to equal its own prefix-call final row, enabling one-pass sequential likelihoods; causally masked SDPA satisfies this property. Historical attention rows with multiple allowed keys generically change under suffix perturbations, while the first row's singleton attention is unaffected.Source: §4.5 · p. 21
  • 0 changed entries in 2,048,000 randomized historical-logit comparisons were observed on the collapsed checkpoint; 74.5% of compared entries moved on an earlier checkpoint.Historical-row invariance is therefore a checkpoint property tied to collapse, not a guarantee of the shared architecture.Source: §4.5 · p. 22
  • Attention-masked padding can change final-real-row outputs through the global Gram; stripping padding restores unpadded scoring bitwise.A validity-masked running Gram can restore historical-row prefix consistency, but an all-row pass requires up to S metric-learner evaluations per layer and head. This optional construction is not used in the released implementations, which retain the global Gram.Source: §4.5 · p. 22

5. Deductive Outputs as Invariant Operators

This section establishes how exact invariance of deductive outputs reduces inference to fixed-operator generalized SDPA, without implying equivalent training dynamics, and bounds deviations under approximate invariance. It examines rotary averaging, concentration, and conditional row-map contraction as a proposed mechanism under stated idealizations, supported by checkpoint diagnostics but without certification of cached decoding or an established account of its selection by training.

5.1. The steady state and the order parameter.

  • Exact G_LM invariance guarantees cache equivalence, but neither operator invariance nor score-row equality is necessary for identical full-model outputs.Equal softmax rows are equivalent to row-wise constant score shifts, which suffice for cache equivalence on every reachable decoding state. Exactness relative to frozen KV-cache semantics is unconditional, unlike full-prefix recomputation; historical-row prefix consistency is not established.Source: §5.1 · pp. 23–24
  • 10^-6 to 10^-11 near-critical m contrasts with approximately 1–50 sub-critical m in [14], but does not precisely rank reasoning ability.The per-tensor order parameter normalizes cross-run RMSE by RMS entry magnitude; exact-arithmetic m = 0 is equivalent to that tensor's input invariance. Floating-point zero only establishes agreement at numerical resolution, and the published sample lacks uncertainty estimates.Source: §5.1 · pp. 23–25

5.2. The inference-collapse theorem.

  • Exact input invariance collapses PLDR-LLM inference to generalized SDPA with constant learned bilinear forms; G_LM = I recovers SDPA exactly.Without RoPE, constant operators can be absorbed into the entire affine query projection, including its bias. With RoPE, commutant membership suffices; necessity holds for unrestricted score families, not necessarily a fixed model's reachable queries and keys.Source: §5.2 · p. 25
  • Full training retains a chain-rule gradient term absent from the collapsed parameterization; a nonzero term establishes different instantaneous gradients.This parameterization-level asymmetry does not prove strict function-class containment. Input-dependent operators can nevertheless realize score maps unavailable to constant-G heads, so collapse is not a family-wide expressivity equivalence.Source: §5.2 · pp. 25–26

5.3. Quantitative stability of the collapse.

  • C(θ) ε bounds cached-versus-uncached logit deviations under Lipschitz assumptions, but does not itself certify bit-identical decoding.Certification requires comparing evaluated end-to-end constants with realized logit margins. Identical benchmark scores reported in [13] used one-pass block scoring, not a cached-versus-recomputed comparison.Source: §5.3 · p. 26

5.4. The origin of the invariance: how the metric learner generates an invariant metric generator.

  • The proposed invariance mechanism combines rotary phase averaging, statistical concentration, and contraction of a shared row map, but remains a quantitative hypothesis.Its structural basis is rotated-query density construction, row-wise LayerNorm, and a row map shared across rows and heads. Stage 1 has an exact but loose bound, Stage 2 assumes independence, and Stage 3's contraction is measured rather than proved.Source: §5.4 · p. 27

5.4.1. Stage 1: the rotary twirl projects the density operator onto a commutant.

  • 43.9 is the vacuous uniform rotary-twirl bound at S = 1024; fast frequencies are suppressed while the slowest remain nearly unchanged.Under common-second-moment and nonresonance assumptions, the deterministic normalized density approaches the commutant at O(1/S), with scalar-block targets for symmetric moments. The worst frequency multiplier is approximately 0.99991 at S = 1024, making practical suppression depend on frequency-resolved energy.Source: §5.4.1 · pp. 27–28

5.4.2. Stage 2: statistical concentration of the normalized density operator.

  • Independent bounded queries sharing a second moment yield conditional concentration around a dataset-level object; contextual transformer queries need not satisfy these assumptions.The concentration variable S is context length within one forward pass, not pretraining token count. These quantities affect invariance through different mechanisms.Source: §5.4.2 · p. 28
  • O(S^-1/2) concentration confines normalized rows near a fixed set under Stage 1–2 idealizations, leaving invariance dependent on the learned row map.Implemented LayerNorm uses epsilon_LN = 10^-6: shift invariance is exact, scale invariance approximate, and its compact range is a ball rather than a sphere. Its epsilon floor gives a finite global Lipschitz bound; the scale-discrepancy expansion requires epsilon much smaller than row variance.Source: §5.4.2 · pp. 29–30

5.4.3. Stage 3: a conditional contraction bound for the trained row map.

  • Conditional contraction of every learned unit shrinks output diameter exponentially with depth, producing approximately identical rows across heads and layers sharing the row map.The contractive regime is an unproved assumption, not guaranteed by residual connections or LayerNorm; composition does not imply fixed-point dynamics. Conversely, a positive pairwise lower bound on output separation rules out collapse for that pair.Source: §5.4.3 · p. 30
  • Sampled composite Jacobians and pairwise contraction ratios test the trained row map locally, but do not establish a tube-uniform Lipschitz certificate.Stage 2 concentration and Stage 3 diagnostics remain separate evidence unless a tube-uniform bound permits their quantitative combination. Transfer must also account for measured LayerNorm scale discrepancies, whose small median conceals a nonuniform tail.Source: §5.4.3 · pp. 30–31
  • Tens of orders of magnitude too large, the assembled sample-extrema coefficient cannot certify decoding decisions despite empirically indistinguishable tested cache fidelity.Certification requires domain-valid factors and a logit perturbation bound B satisfying 2B < Delta, where Delta is the minimum realized top-two margin. Relative-RMS invariance must first be converted to the required spectral-norm bound through Frobenius norm.Source: §5.4.3 · p. 32

5.4.4. Why training at criticality might select the constant map: a hypothesis.

  • Constant-map selection at criticality remains a hypothesis: collapsed inference permits it, but proposed gradient and stability mechanisms are heuristic.In the collapsed instance, Q, K, V can carry instance-specific information while the operator remains constant. Concentration does not establish negligible instance-specific gradients, and input sensitivity does not generally imply larger loss curvature.Source: §5.4.4 · p. 33
  • 41B-token training yields m approaching 0 at float resolution [14], one signature consistent with the proposed constant-map selection mechanism.Other signatures are cross-head identity of A [13], supported by shared-row-map diagnostics, and increased caching perturbation under DAG regularization [13]. Training-token count is distinct from the context length S governing concentration.Source: §5.4.4 · p. 33

6. Power Laws, Scale Invariance, and Self-Organized Criticality

Power laws uniquely satisfy measurable scalar scale covariance, but this does not establish global scale covariance for PLGA. The spectral account of criticality is conditional, while self-organized criticality remains a phenomenological framework rather than established physics.

6.1. Why power laws: the unique scale-equivariant interaction.

  • Power laws are the only measurable positive scalar maps satisfying scale covariance, making the elementwise potential stage a minimal scale-covariant interaction.The proof reduces scale covariance to the multiplicative Cauchy equation, whose measurable solutions are powers [1, Ch. 2].Source: §6.1 · p. 33
  • Elementwise scale covariance does not imply global architectural covariance: heterogeneous exponents, additive biases, numerical floors, normalization, and mixing introduce incompatible transformations or scales.Calling learned exponents physical scaling dimensions would require an identified symmetry action, identifiability analysis, and empirical covariance tests; these are unavailable.Source: §6.1 · pp. 33–34

6.2. Criticality of the attention dynamics: a conditional spectral dictionary.

  • Uniform gaps at both spectral edges yield exponential correlation decay for reversible Markov families; applying this dictionary to trained PLDR-LLM remains conjectural.A fixed finite matrix cannot support an accumulating spectral density, and trained attention operators are neither reversible nor constant across layers and tokens. No near-critical spectral measurement of trained operators has been published; applicability remains Conjecture 8.2.Source: §6.2 · pp. 34–35
  • Spectral-edge comparability determines decay exponents but not exact coefficients; an exact edge density supports stronger asymptotic constants.With density (c + o(1))s^(ϑ−1) and no singular edge component, autocorrelation is asymptotic to c Γ(ϑ)t^-ϑ. The norm decays with half the autocorrelation exponent; comparability alone permits oscillations preventing an asymptotic equivalent.Source: §6.2 · p. 35, §6.3 · p. 35
  • Power-law spectral accumulation near the upper edge yields scale-free correlation decay only under explicit family-limit and lower-edge-gap conditions.Spectral measures must converge weakly, with the family limit taken before the large-time limit; no uniform-in-time convergence is claimed. Without the lower-edge gap, an atom at −1 can invalidate the conclusion.Source: §6.2 · pp. 34–35

6.3. The SOC training picture as a phenomenological framework.

  • ≤10^-5 is typical for m(θ) in phenomenologically near-critical models with non-degenerate generation; sub-critical models have m(θ) = O(1) or larger [14].The empirical near-critical range is ≲10^-2; maximum learning rate and warm-up duration are controls, token batches are drive, and gradient updates are dissipation. Sub-critical models show lower loss but degenerate generation and near-chance benchmarks [14]; dragon-king spikes can damage final states despite loss recovery [14].Source: §6.3 · p. 36
  • Published training evidence supports schedule-tuned critical-like phenomenology, not established self-organized criticality or a demonstrated self-tuning mechanism.Experiments use single runs per condition without uncertainty quantification; phase labels partly depend on the outcomes they explain [14]. Drive–relaxation separation and standard criticality discriminants remain unmeasured.Source: §6.3 · pp. 35–36, §6.2 · p. 35
  • The renormalization-group interpretation is an analogy, not a derived map, and motivates rather than establishes the operator-transfer conjecture.Depth is interpreted as coarse-graining, learned exponents as scaling dimensions, and instance-invariant operators as fixed-point behavior; no semigroup is claimed. Tokenizer-insensitivity reported in [12] motivates the universality analogy associated with RG fixed points and Conjecture 8.3.Source: §6.3 · p. 36

7. Advantages of PLDR-LLM over SDPA-LLM · 8. Conjectures

This section reviews PLDR-LLM’s exact inclusion of SDPA, learned operator sector, and measured training and caching benefits, alongside additional training cost and limited validation of diagnostic and transfer advantages. It then presents the transport conjecture of target-domain benchmark parity through limited refitting with the nonlinear PLGA network frozen, noting that operator congruence alone establishes neither parity nor low refitting cost.

7.1. A larger parameterized family with distinct training dynamics.

  • Learnable G_LM achieves the best average benchmark scores among matched transferred-, identity-, and random-constant models under the published one-pass block protocol [13].SDPA is exactly the G_LM = I point; PLGA generically has different training dynamics where the gradient nonvanishing hypothesis holds. Under nonresonance and unrestricted query/key reachability, d_k = 64 yields 4032 additional head-level score-function dimensions outside the RoPE commutant, not a full-model function-class separation.Source: §7.1 · pp. 37–38, §7 · p. 37

7.2. An inspectable law representation.

  • Exposed operators support DAG regularization, exponent readout, and transplantation; DAG regularization improves benchmark scores without scaling model or data [12].Their structure includes strictly positive A_LM with Perron–Frobenius properties, scale-equivariant potentials, rank-one collapse, and empirically invariant G* with quantified caching perturbations. Exponents and couplings are directly monitorable, but interpreting them as training-domain scaling laws remains unestablished.Source: §7.2 · pp. 38–39

7.3. An intrinsic evaluation diagnostic (prospective).

  • The intrinsic order parameter m(θ) separates published near-critical and sub-critical runs in phase-level agreement with benchmarks [14], but broader evaluation remains unvalidated.It is identically zero for SDPA because G_LM = I; finer within-phase rankings remain unresolved without uncertainty estimates. Potential applications to small models, non-language domains, and data-limited settings require validation with multiple seeds, held-out thresholds, and an independent phase criterion.Source: §7.3 · p. 39

7.4. Phase-aware, self-instrumented training.

  • Training loss can miss failed PLDR phases, while m(θ), DAG losses, and dragon-king events provide internal signals for rejecting bad runs [12] [14].Sub-critical models can achieve lower training loss while generating token salad, and similar loss/accuracy can conceal sharply different deductive behavior. Gap-closing versus gapped spectra offer a candidate interpretation, but until Conjecture 8.2 is tested these diagnostics remain predictive rather than interpreted.Source: §7.4 · pp. 39–40

7.5. Inference efficiency and a deployment asymmetry.

  • ∼3× faster cached inference preserves published one-pass block benchmark evaluations [13], but reported 27–39% advantages over SDPA are confounded by different software stacks.G-cache is exact relative to KV-cache; freezing A at the prompt remains approximate, and evaluated perturbation bounds cannot certify bit-identical decoding. Executing PLGA once per prompt permits separately packaged weight omission, but withholding the training network is not established security without a threat model and attack evaluation.Source: §7.5 · p. 40

7.6. Inductive bias and transfer. · 8. Conjectures

  • Scale-equivariant power laws and extractable (G*, P) motivate conjectural operator alignment and whole-model domain transfer, not established cross-domain capabilities.Conjecture 8.3 proposes constrained common-frame congruence between near-critical operators from distributions in the same universality class; congruence invariants, not eigenvalues, would encode that class. Conjecture 8.4 separately proposes an explicit transport procedure freezing the source pair (G*, P).Source: §7.6 · p. 40, §8 · p. 42

7.7. Costs, trade-offs, and honest limits.

  • Published PLDR-LLM experiments show average benchmark parity rather than dominance over SDPA references, with a learned-operator advantage in matched ablations and the longer 41B-token run overtaking SDPA on average [12] [13] [14].Reference configurations have #ResL/#A ≈129–149 [13]; training costs more per step than SDPA, and near-critical settings can fail via dragon-king events [13] [14]. Evidence comes from single training runs per condition within one research program, without uncertainty estimates or same-stack matched baselines; billion-parameter behavior remains an open empirical question.Source: §7.7 · pp. 40–41

8. Conjectures

  • Conjectures 8.1–8.2 posit exact operator rigidity in a joint width/training limit and a spectral distinction between near-critical and sub-critical models.Rigidity assumes fixed depth and m(θ) → 0; observed fluctuation reductions are schedule-confounded, and context-length bounds do not establish training-token exponents. The spectral conjecture requires a reversible operator family with power-law accumulation at 1 versus a uniform gap, proposes reasoning ability monotone in the spectral exponent, and could enable single-pass spectral diagnostics.Source: §8 · p. 41
  • The proposed transport procedure refits only tokenizer/embedding maps, linear projections, and readout while keeping the nonlinear PLGA network frozen.Which components require refitting, and at what cost, are part of the conjecture rather than consequences of operator congruence.Source: §8 · p. 42
  • Conjecture 8.4 proposes target-domain benchmark parity, within a stated tolerance, against a from-scratch control matched for parameters, tokens, and FLOPs.The proposed test compares the transport procedure against its matched from-scratch control.Source: §8 · p. 42
  • Operator congruence alone, Conjecture 8.3, does not imply successful transport or the conjectured small refitting cost.Congruence of G_* does not transport tokenizer, embeddings, projections, FFN, normalization, or readout. The proposed congruence test trains on two same-class synthetic sources, extracts G_*, and checks alignment of constrained congruence-invariants.Source: §8 · p. 42

9. Discussion and Open Problems … Appendix B. Code, Models, and Verification Resources

The discussion separates exact architectural and algebraic results from measured invariance and unresolved theoretical and empirical claims. The accompanying sections disclose AI assistance and document notation, proof dependencies, pinned verification resources, and benchmark scoring.

9. Discussion and Open Problems

  • PLDR-LLM exactly includes SDPA at G_LM = I, while inference collapse requires invariance and interpreting invariant operators as domain laws remains an open ambition.Results include scope-qualified scale covariance, positive-tensor Perron–Frobenius structure, rank-one generator algebra, perturbation bounds, and walk-counting and commutant identities. Rotary twirling, concentration, and row-map contraction form a conditional mechanism; SOC remains phenomenological, and the positional-operator codimension claim is conditional.Source: §9 · pp. 42–44
  • Open theoretical problems concern collapse dynamics, explicit renormalization coarse-graining, and learned operators’ departures from the RoPE commutant.Goals include proving attraction to the constant-map manifold at critical control settings and constructing token coarse-graining under which the layer map is approximately covariant. Existing per-layer/head commutant measurements need extension across scales and trajectories, alongside off-commutant twirl tests; operator classification requires constrained congruence invariants rather than similarity or eigenvalues.Source: §9 · pp. 42–43
  • Empirical claims rest on single training runs per condition within one research program, requiring broader validation before benchmark-superiority or causal-phase claims.Priorities include multiple seeds, scales and corpora; independently fixed phase criteria; parameter-, token-, and FLOP-matched baselines; cache-equivalence tests; commutant-projection interventions; and SOC scaling, fluctuation, and self-tuning diagnostics. Training trajectories must track prefix consistency, sequential-versus-block scores, blockwise-CE-versus-sequential-NLL target exposure, spectra and contraction diagnostics, with sequential benchmark rescoring at every checkpoint used for causal claims.Source: §9 · pp. 43–44

Disclosure of the use of AI tools

  • Claude assisted drafting, proofs, implementation checks, and numerical audits; Codex supported revision verification, while the author reviewed all claims and retains responsibility.The disclosed models are Claude Fable 5 from Anthropic and GPT-5.6-Sol from OpenAI. Selected Lean 4 proofs were mechanically checked.Source: Disclosure of the use of AI tools · p. 44

Appendix A. Notation

  • The notation distinguishes positive interaction and potential tensors from the bilinear score operator, and blockwise global-context cross-entropy from sequential autoregressive NLL.It also identifies the density operator, shared metric-learner row map, and projection onto the RoPE torus commutant.Source: Appendix A · p. 45

Result dependency map.

  • The dependency map records direct proof inputs and hypotheses, distinguishing proved results from Hypothesis 5.14 and falsifiable, unproved conjectures.Inputs include elementary linear algebra, Perron–Frobenius theory, matrix Bernstein concentration under an idealization, and Lipschitz composition under a contraction hypothesis.Source: Result dependency map · p. 46

Appendix B. Code, Models, and Verification Resources

  • Architectural verification is tied to pinned implementation snapshots, with released model configurations, dataset revisions, machine-checked proof cores, and numerical-audit resources documented.The manifest records repository commits, model revisions, and selected file hashes; the sequential audit additionally pins corpus and benchmark data. Released resources include Lean 4/mathlib proof cores building without unproved obligations, Python dependencies, full audit results, and raw arrays.Source: Appendix B · pp. 47–48
  • Published benchmark scores use uncached whole-candidate block scoring, not generation-time caching, with exact sequential equivalence established for one-token candidates.The wrapper concatenates context and candidate, evaluates all but the last token in one call, and sums candidate log-probabilities. Multi-token agreement on the audited checkpoint is qualified by Appendix D’s measured gaps.Source: Appendix B · p. 48

Appendix C. Lean Formalization: Exact Coverage · Appendix D. Numerical Audit on a Released Checkpoint · D.1. Setup and provenance.

Lean kernel-checks selected proof cores, not the full decoder or checkpoint measurements. A separate numerical audit distinguishes directly computed quantities from bounds and proxies and uses semantic tests to guard sensitive computations.

Appendix C. Lean Formalization: Exact Coverage

  • Lean proofs are kernel-checked relative to mathlib’s classical principles, not axiom-free or constructive, and do not validate adjacent unformalized claims.The repository builds without sorry, admit, axiom, or unsafe declarations. Continuous integration audits every exported theorem, permitting only propext, Classical.choice, and Quot.sound.Source: Appendix C · p. 49
  • Kernel-checked algebra covers positivity, softmax characterizations, Gram-matrix structure, positive entrywise-power group laws, and rank-one identities.Softmax includes ideal masking and equality exactly under common shifts; Gram proofs establish symmetry, positive semidefiniteness, and rank equality. Rank-one proofs include powers and nilpotency, while architecture-level scale covariance is explicitly not claimed.Source: Appendix C · p. 49
  • LayerNorm invariances, single-block rotary commutants, and analytic twirl bounds are checked; multi-block commutants, codimension counts, and matrix-level twirl projections are not.LayerNorm includes its exact norm identity and scale invariance at epsilon = 0 on positive-variance rows; the single-block commutant criterion requires sin theta != 0. Twirl proofs cover the chord identity and geometric-sum bound, excluding frequency-resolved constants and stochastic applications.Source: Appendix C · p. 49
  • DAG walk inequalities, a positivity obstruction, and abstract contraction bounds are checked, but trained-unit contractivity is measured rather than proved.The normalized-loss floor log(1 + epsilon^2) follows only in prose; the NOTEARS converse and spectral-side identity are not checked. Contraction proofs establish multiplicative diameter bounds and K^N bounds for iterated K-Lipschitz maps.Source: Appendix C · p. 50
  • Inference formalization verifies identity-operator reduction, constant-operator absorption, and abstract causal-decoder contracts, not the full PLDR-LLM decoder.Absorption includes affine biases; score-level and masked-row sufficiency, length preservation, prefix consistency, and row-locality implications are checked. Unrestricted RoPE necessity, model-level decoding induction, gradient decomposition, and checkpoint measurements remain unformalized.Source: Appendix C · p. 50
  • Perron–Frobenius theory, matrix concentration, perturbation bounds, and all hypotheses, analogies, and conjectures lie outside the formalization.Other prose-only components include softmax Lipschitz bounds, finite-mask underflow semantics, rank-one singular-value determinant and spectral-radius claims, and quantitative LayerNorm scale-error and Lipschitz bounds. The reversible-family spectral dictionary is also outside scope.Source: Appendix C · pp. 49–50

Appendix D. Numerical Audit on a Released Checkpoint

  • The checkpoint audit computes named quantities directly or explicitly labels bounds and proxies, with semantic tests guarding exponent indexing and perturbation-chain assembly.Diagnostics cover spectra, LayerNorm error, twirl energy, commutant residuals, contraction, decoding-margin proxies, cached deviations, DAG loss, and order parameters. Online-contract tests cover suffix changes, Gram-boundary padding, and sequential-versus-block scoring on two released checkpoints.Source: Appendix D · p. 51

D.1. Setup and provenance.

  • Eight fixed English prompts of 32–43 tokens probe the pinned, unmodified PLDR-LLM-v51-SOC-110M-5 checkpoint [14], with direct measurements distinguished from bounds and proxies.The five-layer, 14-head model uses dk = 64 and float32, running eager attention under transformers 4.56.1 on one consumer GPU. Spectra use float64; min/median/max generally aggregate across layers, heads, and prompts.Source: §D.1 · p. 51, Appendix D · p. 51

D.2. Singular values and numerical rank (in place of determinants), trained versus initialized.

Singular-value measurements show that the trained generator A collapses to rank one with nearly identical heads, while A_LM remains generally near full numerical rank despite float-zero determinants. Initialization controls identify the collapse as a training outcome rather than an architectural artifact.

Trained singular-value results

  • 560 trained layer–head–prompt instances all have generator A numerical rank 1, with σ2/σ1 ≤1.4 × 10^-8.The median σ2/σ1 is 8.7 × 10^-17 at tolerance 64 ε_f32 σ1. Within-head row-variance/entry-variance is approximately 4 × 10^-13, while cross-head relative RMS is ≤1.6 × 10^-8.Source: §D.2 · p. 51
  • 62.5 of 64 is A_LM’s median numerical rank, with median σ2/σ1 of 0.27, despite its float determinant being zero.Numerical ranks range from 1–64; determinant underflow reflects the singular-value scales, not rank-one collapse. The entrywise nonlinear image A_LM is generally near full numerical rank, making float-zero determinants uninformative about rank.Source: §D.2 · p. 51

Trained-versus-initialized control

  • 0.69 is initialization’s median σ2/σ1 for A under both tested laws, contrasting with ≤1.4 × 10^-8 after training.Both laws yield A rank 63 throughout, A_LM median rank 64, and cross-head relative RMS of 1.1–1.3, establishing trained collapse and head agreement as training outcomes. The port uses Xavier-uniform metric-tensor parameters; native training uses Xavier-normal, with Xavier-uniform dense weights in both.Source: §D.2 · p. 52
  • 63 is the architecturally expected initialization rank, not a collapse signature, because the final LayerNorm centers every row of A.At initialization, γ = 1 and β = 0 imply A1 = 0 and rank ≤d_k − 1 = 63; the generic rank therefore reaches 63.Source: §D.2 · p. 52

D.3. The LayerNorm scale error, measured directly.

Direct measurements show that LayerNorm’s scale-invariance idealization is approximately valid for typical rows, but not uniformly. The upper proxy overestimates measured relative error and diverges on exactly constant rows whose true discrepancy is zero.

D.3. The LayerNorm scale error, measured directly.

  • 4.6 × 10^-4 median relative LayerNorm scale error masks substantial variation: 16.4% of rows exceed 2% relative error.The 90th percentile is 4.3 × 10^-2, the 99th percentile is 0.44, and the maximum is 4.3 on epsilon-dominated rows. Quantitative Stage-2 concentration arguments must therefore carry the measured distribution rather than the proxy.Source: §D.3 · p. 52
  • 5.4 × 10^-10 median absolute error and 1.1 × 10^-5 maximum absolute error accompany the relative-error measurements.Discrepancies were computed directly for every captured row in float64 using the checkpoint’s LayerNorm parameters and each prompt’s actual S.Source: §D.3 · p. 52
  • 1.9 × 10^-2 is the upper proxy’s median; at matched rows, it overestimates measured relative error by approximately 35-fold.The proxy epsilon_LN/(2v_i) omits the exact first-order coefficient’s factor (1 - S^-2). It diverges on 14 exactly constant rows where both centered rows vanish and the true discrepancy is exactly zero, illustrating that bound divergence does not establish quantity divergence.Source: §D.3 · p. 52

D.4. Twirl energy against pre-rotation data. · D.5. Commutant residual of the trained operators.

At the audited context lengths, rotary twirling removes only a modest fraction of off-commutant energy and cannot alone explain observed invariance. The trained operators remain far from the RoPE commutant, although this measurement does not establish causal use during decoding.

D.4. Twirl energy against pre-rotation data.

  • 9.5% median per-instance off-commutant suppression ranges from −1.4% to 46%, so finite-length rotation does not suppress energy monotonically across instances.The audit covers all 40 prompt-layer instances; in one instance, rotated off-commutant energy slightly exceeds unrotated energy.Source: §D.4 · p. 52
  • 6.9% and 10.9% are median pre-rotation and post-rotation commutant energy fractions per instance; corresponding prompt-pooled medians are 7.2% and 11.1%.Per-instance ranges are 3.9–15.7% before rotation and 4.2–18.0% afterward; pooled ranges across layers are 4.5–12.6% and 4.5–13.8%, respectively. The aggregation labels matter because ratios do not commute with pooling.Source: §D.4 · pp. 52–53
  • 1.2 × 10^-7 bounds the relative reconstruction error, validating the decomposition pipeline using positionwise pre-rotation queries from head 0 of each layer.For d_k = 64 and rotary base 10^4, C_Theta/S ≈ 43.9 at S = 1024, making the uniform bound vacuous there. Per-frequency multiplier medians are 0.27/0.054/0.013 at S = 64/256/1024, while the worst multiplier remains approximately 1 at every S.Source: §D.4 · p. 52
  • 10.6% median prompt-pooled off-commutant suppression indicates that rotary twirling alone cannot explain observed invariance at the audited context lengths.Suppression ranges from 0.009% to 43% across five layers, with prompts of length S = 32–43. Attribution to Stage 3 requires its direct measurements, not subtraction.Source: §D.4 · pp. 52–53
  • 33% median relative difference between first-half and second-half query outer-product aggregates shows that the stationarity idealization is far from holding.Differences reach 90% across prompt-layer instances.Source: §D.4 · p. 53

D.5. Commutant residual of the trained operators.

  • 0.81 is the median relative off-commutant energy across 560 trained-operator instances, showing substantial distance from the RoPE commutant.The reported Frobenius-norm ratio ranges from 0.42 to 0.99; layer 1–5 medians are 0.98, 0.99, 0.93, 0.71, and 0.66. This checkpoint-specific occupancy measurement does not establish causal decoding use; a commutant-projection ablation remains proposed.Source: §D.5 · p. 53

D.6. Row-map contraction, measured on the composition.

The trained row-map composition exhibits strong contraction on sampled visited data despite failing the per-unit contraction hypothesis. These measurements support local constancy but do not establish a tube-uniform Lipschitz certificate.

D.6. Row-map contraction, measured on the composition.

  • ≈95% of retained row pairs have zero empirical contraction ratios at float resolution; remaining ratios are at most 4.6 × 10^-6.Sampling retained 765 within-prompt and 694 cross-prompt pairs from 1000 attempted draws each, after skipping coincident prompt indices and denominators below 10^-12.Source: §D.6 · p. 53
  • 1.0 × 10^-19–3.6 × 10^-7 is the pooled range of sampled full-composition Jacobian largest singular values across layers 1–5.Per-layer medians are 3.2 × 10^-11, 1.4 × 10^-16, 1.0 × 10^-19, 4.9 × 10^-12, and 3.5 × 10^-7, respectively. The audit directly evaluates 128 real-data rows per layer, including 16 from every prompt.Source: §D.6 · p. 53
  • Individual units fail Proposition 5.12’s contraction hypothesis, while sampled composition contraction supports local constancy without providing a tube-uniform Lipschitz certificate.Per-unit Jacobian norms have median 0.47 and maximum 20.8, whereas the composition contracts by seven or more orders of magnitude at every sampled row. These pointwise statistics support the locally constant interpretation diagnosed by Proposition 3.11(iii).Source: §D.6 · p. 53

D.7. The invariance budget as a sample-extrema proxy, against measured margins. · D.8. DAG-loss values.

Cached and recomputed decoding agree empirically, but the sample-extrema invariance proxy cannot certify the observed margins. DAG-loss measurements show positive cycle content, with A_LM zero readings attributed to numerical underflow.

D.7. The invariance budget as a sample-extrema proxy, against measured margins.

  • The invariance budget is a sample-extrema proxy, not a certificate: its sampled extrema and endpoint variance bounds do not cover the compared decoding paths.Extrema come from eight prompt-only passes rather than grown contexts; endpoint variances need not bound interpolation tubes, and zero crossings leave only the architectural A_LM floor 10^-9. Factor extrema occur at unrelated samples without a common perturbation set; only derivative suprema are certified, while certification requires verified propagation along every compared decoding state.Source: §D.7 · pp. 53–54
  • 1.0 × 10^10 is the median sampled power-stage factor, driven by floor-level A_LM entries and exponents below one.The maximum power-stage factor is 5.8 × 10^11. The certified iSwiGLU derivative envelope has median 0.018 and maximum 1.7 × 10^3.Source: §D.7 · p. 54
  • 7.5 × 10^6 and 9.2 × 10^7 are median sampled and floor-based path-valid G_LM sensitivity constants, respectively.Their maxima are 1.2 × 10^15 and 1.3 × 10^15, respectively, because the worst sampled minima already reach the floor. These constants are Frobenius-to-Frobenius; converting a uniform per-row A_LM perturbation bound to the spectral hypothesis costs another factor of 8.Source: §D.7 · p. 54
  • ≈7.3 × 10^46 is the assembled end-to-end logit coefficient, including the injection layer’s post-attention remainder before downstream layer factors.Because the perturbation enters at the attention output, its own layer contributes post-attention LayerNorm, FFN residual, and post-FFN LayerNorm factors, but no attention factor. Finite-difference tests reject assemblies omitting this same-layer remainder on a small-variance input.Source: §D.7 · p. 54
  • 960 per-layer comparisons across four 48-token decodes found bitwise-identical cached and recomputed G_LM, maximum logit deviation 3.6 × 10^-5, and identical greedy choices.The minimum realized top-two margin was 6.0 × 10^-3, satisfying the corrected sufficient criterion: 2B = 7.2 × 10^-5 < 6.0 × 10^-3.Source: §D.7 · pp. 54–55
  • 0.49–0.80 duplicate 4-gram fractions marked greedy repetition loops, versus 0.0 in the order-parameter section’s sampled continuations; these examples do not establish generation quality.The audited checkpoint has 110M parameters, and the tensor measurements are independent of text quality.Source: §D.7 · p. 55
  • ≈45 orders of magnitude separate the hypothetical rounding-radius proxy from the halved minimum margin, so this factorwise assembly cannot certify decoding.At the hypothetical single-rounding radius 3.4 × 10^-5—not a measured operator deviation and excluding accumulated roundoff—the proxy is ≈2.4 × 10^42 versus ∆/2 = 3.0 × 10^-3. Replacing sampled factors with tube suprema cannot improve this assembly; a smaller direct or margin-aware bound remains unconstructed.Source: §D.7 · pp. 54–55

D.8. DAG-loss values.

  • 560 instances per tensor yielded median DAG losses of 3.6 × 10^-10 for A_LM, 1.1 × 10^3 for A_P, and 1.3 × 10^3 for G_LM.All three tensors showed positive measured cycle content except A_LM underflow readings. Its analytic normalized-loss floor is ≈10^-18, below naive float64 log resolution; no floor is asserted for G_LM.Source: §D.8 · p. 55

D.9. Order parameter, two normalizations.

On this checkpoint, G_LM has an exactly zero order parameter at float resolution, while the other reported tensors show near-zero values under both normalizations. The audit reports RMS and symmetrized signed-mean normalizations separately.

D.9. Order parameter, two normalizations.

  • Two independent 64-token, temperature-1 continuations for each of five prompts are compared using RMS and symmetrized signed-mean normalizations.The signed-mean denominator is (|µ_1| + |µ_2|)/2, a symmetric adaptation rather than the source convention: first-run |µ_1|, or |µ_C| against a cached run [14].Source: §D.9 · p. 55
  • 0 at float resolution is the G_LM order parameter on every layer under both normalizations.The measurements independently reproduce the m ≈ 0 (float) report of [14] using the stabilized statistic.Source: §D.9 · pp. 55–56
  • 4.9 × 10^-9 RMS-normalized and 3.1 × 10^-8 signed-mean-normalized maxima characterize A, the most sensitive tensor.A_LM maxima are 3.4 × 10^-9 RMS-normalized and 5.6 × 10^-9 signed-normalized; both medians are 0, as is A’s RMS-normalized median. The two normalizations agree in order of magnitude wherever nonzero.Source: §D.9 · p. 56
  • 8.6 × 10^-13 RMS-normalized and 3.3 × 10^-12 signed-normalized maxima make A_P near-zero but not bitwise invariant.A shape-checked helper with a sentinel semantic test guards construction against silent head/row broadcasting of unbatched, per-head P. Chain constants read P under the same guard.Source: §D.9 · pp. 55–56
  • The zero-numerator branch handles all bitwise-equal pairs, while no positive-numerator, zero-signed-denominator case is observed.The stabilized definition retains protection against that unobserved degeneracy for models whose deductive outputs are centered near zero.Source: §D.9 · p. 56

D.10. The online contract, historical-row movement, and padding. · D.11. Sequential versus one-pass block scores.

On tested inputs, the audited checkpoint has bitwise-stable historical rows and nearly identical sequential and block scores, while the contrast checkpoint exhibits historical-row movement and larger scoring gaps. Padding tests distinguish shape-level floating-point sensitivity from masked-row contamination of the Gram, supporting an unpadded online contract.

D.10. The online contract, historical-row movement, and padding.

  • 0 of 2,048,000 randomized logit comparisons change on the audited checkpoint, versus 74.5% on the contrast checkpoint.Audited G_LM remains bitwise equal at every layer; contrast G_LM deviations reach 3.0 × 10^-2, and masked-suffix comparisons still change 74.3% of logit entries. These finite tests associate historical-row prefix consistency with collapse, not a universal decoder property; prefix position 0 is structurally insensitive on both checkpoints.Source: §D.10 · p. 56
  • Repeated identical prefix calls are bitwise equal on both checkpoints, making the step-t conditional deterministic in the tested configuration.The script uses float32 eager execution with deterministic settings on a single consumer GPU and loads both checkpoints at pinned revisions. Two complete back-to-back runs reproduce the results file and raw archive byte-for-byte.Source: §D.10 · p. 56
  • 3.5–7.4 × 10^-5 final-row deviations on the contrast checkpoint depend on masked-padding content, demonstrating that masked rows enter the Gram.Appending four attention-masked tokens also changes G_LM, with maxima of 4.5 × 10^-3–1.2 × 10^-2 and roughly 50,000 changed entries.Source: §D.10 · pp. 56–57
  • ≤8.6 × 10^-6 bounds audited-checkpoint deviations after adding four masked padding tokens, with unchanged G_LM and padding-content independence indicating shape-level floating-point sensitivity.One of the two prompts has exactly zero deviation. Stripping padding restores the unpadded call bitwise on both checkpoints, supporting the unpadded-S=t contract.Source: §D.10 · pp. 56–57

D.11. Sequential versus one-pass block scores.

  • Eight prompts each receive four multi-token and four single-token candidates to compare sequential prefix-call scoring with one-pass block scoring.Sequential scoring sums final-row token log-probabilities from separate prefix calls; block scoring reads aligned rows from one call on the prompt and candidate excluding its last token. Single-token scores coincide exactly on both checkpoints because both protocols invoke identical input tensors.Source: §D.11 · p. 57
  • 3.3 × 10^-6 bounds total candidate-score gaps between sequential and block scoring on the audited checkpoint, versus 2.2 × 10^-4 on the contrast checkpoint.Per-token maxima are 3.3 × 10^-6 and 1.5 × 10^-4, respectively; neither checkpoint changes any of the 32 candidate rankings, with zero discordant pairs across 48 ordered comparisons. This constructed suite is a controlled diagnostic, not benchmark validation; published benchmark scores remain labeled block scores.Source: §D.11 · p. 57

D.12. Held-out sequential NLL and real benchmark items under both scoring protocols. · D.13. Scope.

On sampled held-out text and benchmark items, blockwise and sequential scoring closely agree, with unchanged argmax choices and near-identical TruthfulQA probability-mass scores. These measurements remain sample-specific and sensitive to floating-point execution paths.

D.12. Held-out sequential NLL and real benchmark items under both scoring protocols.

  • 1.8 × 10^-9 nats/token separates audited blockwise CE and sequential NLL, both reported as 3.479446; the contrast checkpoint’s aggregate gap is 4.3 × 10^-6.Contrast blockwise CE is 3.684883 versus sequential NLL 3.684879 nats/token. Per-token absolute-gap medians and maxima are respectively 9.6 × 10^-7 and 2.4 × 10^-5 audited, versus 1.8 × 10^-5 and 2.8 × 10^-3 contrast.Source: §D.12 · p. 57
  • 2.5–3.8 × 10^-5 early-bucket median gaps fall to 5.0 × 10^-6 in the final bucket on the contrast checkpoint, consistent with predicted suffix-length dependence.Later historical rows leave less suffix to move the Gram. These sample-level findings neither remove target exposure nor measure its training-time role.Source: §D.12 · pp. 57–58
  • Zero argmax changes accompany TruthfulQA probability-mass agreement within 5 × 10^-5 per item between block and sequential scoring on both checkpoints.Across seven argmax tasks, raw and length-normalized choices, per-task accuracies, and all 2,901 candidate-pair comparisons agree. The benchmark sample comprises 100 items per task under one seed.Source: §D.12 · p. 58
  • 8.5 × 10^-10 and 5.7 × 10^-7 are the audited and contrast checkpoints’ respective 100-item mean TruthfulQA metric gaps between protocols.The published truthfulqa_mc2 protocol [23] measures normalized probability mass on true answers, not argmax. Audited means are 0.397662 under both protocols; contrast means are block 0.399748 and sequential 0.399747.Source: §D.12 · p. 58
  • 2.8 × 10^-5 and 2.3 × 10^-3 are maximum absolute candidate-score gaps on audited and contrast checkpoints; decision margins explain unchanged choices.Per-task median block top-two margins are 0.51–12.9 nats; each item’s maximum candidate gap stays below half its margin except an identically scored source-data tie. A separate TruthfulQA mc1 probe also yields zero argmax changes and zero discordant pairs among 1,366 comparisons on both checkpoints.Source: §D.12 · p. 58

D.13. Scope.

  • The audit does not establish generalization across seeds, scales, or datasets, nor revalidate complete published benchmark tables or training trajectories.The core audit covers one checkpoint and eight prompts; sequential-validation extensions add a second checkpoint and seeded text and benchmark samples, all within one software stack. Jacobian and pairwise statistics are sampled only at visited points.Source: §D.13 · pp. 58–59, §D.12 · p. 58
  • Approximately 10^10 local amplification at the elementwise power stage makes downstream measurements sensitive to device, library, and floating-point execution paths.Across identical seeded process launches, CUDA matrix-multiply algorithm selection shifted the smallest order-parameter statistics by several orders of magnitude, within approximately 10^-3 relative. The qualitative findings remained stable.Source: §D.13 · p. 59
  • Pinned cuBLAS workspace settings and forced deterministic algorithms made every reported number bitwise reproducible in back-to-back complete reruns.Order-parameter readings below the stability scale should generally be quoted as upper bounds. Audit code, dependency specifications, results, and raw per-instance arrays are published with the Lean formalization.Source: §D.13 · p. 59

Related papers

Loading 2608.10288v1…