Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

Burc Gokden

Source-linked AI summary

Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

Burc Gokden

arXiv:2608.10288v1Aug 10, 2026cs.LGcs.CL

TL;DR

The paper addresses whether attention can replace SDPA’s fixed bilinear form with a mathematically meaningful learned operator. It defines PLGA through input-generated power-law operators and reports inference collapse to a cached constant operator, with unchanged benchmark evaluations.

Problem
SDPA uses a fixed bilinear form, motivating study of a mathematically meaningful attention mechanism whose comparison operator is learned from the input.
Method
PLGA generates a bilinear score operator through a chain from query Gram operators to positive tensors and learned elementwise power laws.
Results
Relative fluctuations reach 10^-6–10^-11, and replacing the deep PLGA subnetwork with a cached constant operator leaves published benchmark evaluations unchanged.
Takeaways & Limitations
PLGA contains SDPA exactly at G_LM = I, while the paper distinguishes proved results, measurements, conditional mechanisms, and falsifiable conjectures.
Takeaways & Limitations
An entrywise nonlinear image of a rank-one-plus-bias matrix is generically full rank, so singularity of A does not imply singularity of A_LM.

Abstract

from arXiv · show

The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator $G_{LM}$, built from a positive tensor $A_{LM}$ by elementwise power laws. The architecture is fully specified, verified against pinned reference releases; claims are labeled theorem, conditional theorem, measurement, or conjecture. Unconditionally: PLGA contains SDPA exactly at $G_{LM}=I$; $A_{LM}$ and $A_P$ are strictly entrywise positive, with Perron-Frobenius structure on $A_{LM}$; the DAG regularizer has the NOTEARS walk-counting form and positivity obstructs exact acyclicity; and, under nonresonance (satisfied by standard rotary frequencies), a commutant criterion identifies which operators preserve relative-position dependence. An inference-collapse theorem: exact input invariance of deductive outputs collapses inference to generalized SDPA with a constant operator. Measured invariance: relative fluctuations of $10^{-6}$ and below; perturbation bounds quantify but do not certify cached inference; the assembled proxy misses the decoding margin. A conditional three-stage mechanism (rotary twirl, concentration, row-map contraction) is measured on a released checkpoint. Blockwise training and scoring under the global Gram are stated with explicit target exposure; on tested samples, block and sequential scoring select identical answers and agree on the published TruthfulQA probability-mass metric within $5\times 10^{-5}$ per item. Self-organized criticality enters as a phenomenological framework with an intrinsic order parameter; open claims become falsifiable conjectures. Selected proof cores are machine-checked in Lean 4.

1. Introduction

The section presents PLGA as an input-conditioned, positive power-law generalization of SDPA, while reporting inference-time operator invariance and a learned singularity condition. It separates proved algebraic results, conditional mechanisms, measurements, and conjectures, with selected proof cores machine-checked in Lean 4.

Core mechanism: PLGA generalizes SDPA by generating a query–key bilinear operator through an input-to-Gram-to-positive-tensor power-law chain, with SDPA recovered exactly at G_LM = I.The intermediate tensors are exposed as deductive outputs alongside next-token probabilities.
Inference collapse: Relative fluctuations of 10^-6–10^-11, including floating-point zeros for the best models, support replacing the deep PLGA subnetwork at inference with a cached constant operator.Published cached-versus-uncached benchmark evaluations are unchanged, while the assembled fidelity bounds do not close the decoding margins.
Learned structure: At convergence, the generator A is numerically rank one with identical rows across heads within a layer, while A_LM remains strictly positive and generically near-full rank.The common row sum is the single nonzero eigenvalue and spectral radius of A, with Perron–Frobenius structure applying to A_LM.
Phenomenology: A critical-like training regime, controlled by maximum learning rate and warm-up steps, tracks generalization in the reported sample, with deductive-output fluctuations serving as an intrinsic order parameter.The section frames self-organized criticality phenomenologically rather than as an unconditional theorem.
Epistemic scope: Claims are classified as algebraic theorems, conditional theorems, empirical observations, or conjectures, and selected proof cores are machine-checked without validating surrounding unformalized claims.The paper also identifies commutant criteria under nonresonance and applies NOTEARS walk-counting structure to the DAG regularizer.

2. Preliminaries and Notation

The section defines contexts as token sequences and embedding matrices, while framing them as weighted token graphs with distinct local-instance and global-manifold structure. It also establishes notation and implementation semantics for PLGA, including the empirical transition of local tensors toward instance-independent behavior after critical pretraining.

Contexts and representations: Contexts are sequences over a finite vocabulary, represented by embedding matrices X ∈ R^(S×dmodel) with one token feature vector per row.Reference implementations use vocabulary size V = 32,000 and maximum context length Smax = 1024.
Contexts and representations: Each context is also a weighted graph of token nodes, linking a discrete quantization set to a continuous language-model manifold.Learned parameters carry dataset-level structure, while inferred tensors carry instance-level structure.
Local and global structure: After critical pretraining, the instance-level tensors A, ALM, AP, GLM, and ELM become numerically independent of the instance.This empirical result is formalized later in Section 5.
Notation: The notation fixes Hadamard products, elementwise powers, ordinary matrix multiplication, row-wise softmax, identities, all-ones vectors, and matrix-row conventions.These conventions support the subsequent algebraic specification of PLGA.
Notation and implementation semantics: Causal masking distinguishes ideal masked softmax from finite-negative-constant implementations, which reproduce masking only through floating-point underflow.The native implementation uses −10^9, while exact real-valued softmax remains strictly positive everywhere.
Notation: LayerNorm uses εLN = 10^-6 with learned affine parameters, and the section defines spectral, Euclidean, Frobenius, and sup-induced operator norms plus Swish/SiLU notation.It also records the matrix-notation correspondence for DQ = Q^⊤Q from prior work.

3. The Power Law Graph Attention Operator

PLGA replaces SDPA’s fixed bilinear score operator with an input-generated operator constructed from a learned metric-like tensor and power laws. The section establishes its positivity, row-wise generator structure, and exact reduction to SDPA or generalized SDPA under frozen operators.

Operator definition: PLGA constructs a learned potential tensor from row-wise residual processing and uses power-law interactions to form the bilinear score operator G_LM.The reference residual network has N_res = 8 units, each containing n_A = 2 SwiGLU blocks; the metric learner acts independently and identically on rows.
Operator interpretation: G_LM is not a Riemannian or information metric: it is an unconstrained, potentially nonsymmetric bilinear score operator generated from a positive interaction tensor.D_Q is an unnormalized query Gram/second-moment matrix, whereas A_LM and G_LM are not constrained symmetric.
Operator definition: A_LM is strictly positive entrywise, making its element-wise power laws well defined and jointly smooth.The positivity condition is A_LM ≥ ε > 0 entrywise, and the family {A^⊙tP} remains well posed for fixed A_LM.
Generator structure: Row-wise shared processing makes Φ_res permutation-equivariant and characterizes cross-input generator collapse by constancy of one shared row map on the union of normalized density-operator rows.When the map is constant there, A(x) has identical rows with a common value across inputs and heads; a Lipschitz bound transfers density-operator changes to generator changes.
Relation to SDPA: G_LM ≡ I yields exact SDPA, a frozen arbitrary G* yields generalized SDPA, and input-generated G_LM(x) is generically nonlinear and not reducible to SDPA.With rotary embeddings, even a frozen operator is generically outside the SDPA-realizable family; setting a = 0 makes G_LM constant and returns to the frozen regime.

4. The PLDR-LLM Architecture

PLDR-LLM applies RoPE to queries and keys, and its learned attention metric preserves relative-position dependence exactly only when it commutes with every RoPE rotation. Under the standard nonresonance conditions, generic learned metrics instead introduce absolute positional information through their conjugation orbits.

Rotary position encoding: PLDR-LLM applies rotary position embeddings to queries and keys after head-splitting.The reference configuration uses Θ = 104, defining a homomorphism from integer positions into the ambient RoPE torus.
Commutant characterization: Relative-position dependence holds if and only if the learned operator G commutes with every RoPE rotation.Equivalently, G must commute with the ambient RoPE torus under the proposition’s nonresonance assumptions.
Commutant characterization: Standard rotary frequencies satisfy the required nonresonance conditions, so the commutant criterion applies to the reference configuration.The frequencies are strictly decreasing and lie in (0, 1], preventing the relevant angle coincidences and resonances.
Positional dependence: For generic learned G_LM, PLGA scores carry absolute positional information through the conjugation orbit R−nG_LMRn.This contrasts with SDPA, where G_LM = I and scores depend on positions only through the offset m − n.
Model definition: PLDR-LLM is defined as a length-indexed family mapping each input sequence to one output probability row per input row.Thus an input of length S is mapped into Δ(V)^S and preserves sequence length.

Appendix C). The computation is as follows. With X(0) = LN

The appendix specifies PLGA’s causal online-generation semantics, blockwise training, and the distinction between exact prefix recomputation and prompt-frozen caching. It also formalizes the DAG-loss positivity obstruction and explains why historical rows and padding can affect computations through the global Gram operator.

Causal generation: Online generation recomputes every tensor on the exact prefix x1:t and consumes only the final row, preserving causality.Internal rows of a longer call are not interpreted as functions of their own prefixes.
Training and scoring: Blockwise training applies cross-entropy to all aligned conditionals from one shifted full-block forward pass.Property (4.14) permits this pass to represent the sequential conditionals used by chain-rule likelihood evaluation.
DAG regularization: 10^-9 is the entrywise floor for ALM, making ALM and AP strictly positive and preventing exact acyclicity for their support graphs.A reported zero DAG loss is floating-point underflow rather than exact acyclicity; no positive floor is asserted for GLM.
Caching semantics: G-cache is exact relative to KV-cache, while empirical freeze after collapse is the separate claim that cached and recomputed semantics coincide on an audited checkpoint.The cache freezes A at the prompt; recomputation instead updates the operator at every grown prefix.
Global-sequence effects: Because PLGA’s density operator contracts the entire supplied sequence, historical rows can change under suffixes, and masked padding can alter final-row outputs through padding content.Sequential deployment therefore uses the unpadded S = t interface.

5. Deductive Outputs as Invariant Operators

Theorem 5.4 shows that identity operators recover SDPA, while exact input-invariant deductive operators collapse inference to generalized SDPA with learned constant bilinear forms. Cache exactness follows from operator invariance, but observed ε-invariance supports only quantitative perturbation bounds, not automatically bit-identical decoding.

Cacheability: Exact cacheability requires only invariance of the operator outputs G_LM, not invariance of upstream tensors A, A_LM, or A_P.The proposition also distinguishes exactness relative to frozen KV-cache semantics from conditional exactness relative to full prefix recomputation.
Inference collapse: Identity G_LM operators make PLDR-LLM exactly a decoder-only transformer with scaled dot-product attention.This is the SDPA special case of the model space, retaining RoPE, SwiGLU FFN, and stated normalizations.
Inference collapse: Exact input-invariant deductive outputs replace the learned subnetwork with constants G*(ℓ,i), yielding generalized SDPA with learned bilinear forms.Without RoPE, the result is exactly an SDPA-LLM after absorbing the constant operator; with RoPE, commutation with all rotations suffices for absorption.
Approximate invariance: 10^-11 relative ε-invariance yields explicit output-distribution perturbation bounds, but bit-identical decoding still depends on comparing the bound with realized logit margins.The final-logit difference is bounded by C(θ) ε uniformly over inputs; the bound does not itself certify cached decoding.
Mechanism: The proposed invariance mechanism combines rotary phase averaging, statistical concentration, and contraction through a shared learned row map.The stages have different epistemic status: the phase-averaging stage has an exact but potentially large bound, while concentration and contraction are conditional or measured claims.

6. Power Laws, Scale Invariance, and Self-Organized Criticality

This section derives elementwise power laws from measurable scale covariance, while limiting the result to scalar features rather than global architectural scale invariance. It formalizes critical slowing down for reversible operator families and treats self-organized criticality in PLDR-LLM as an unverified phenomenological framework.

Scale covariance: Measurable scale covariance forces f(u)=c u^p, making elementwise power laws the minimal scale-covariant interaction ansatz.The result follows from the measurable multiplicative Cauchy equation and applies to one positive scalar entry at a time.
Scale covariance: Global rescaling of A_LM does not generally yield common covariance because exponents P_ij are heterogeneous and additive, floor, normalization, and mixing scales remain.Calling P_ij physical scaling dimensions would require an identified symmetry action, identifiability, and gauge analysis.
Critical slowing down: For reversible Markov-operator families with a closing upper spectral gap and a uniform lower-edge gap, power-law edge accumulation produces power-law correlations with ξ=∞.The lower-edge condition prevents slowly decaying alternating correlations from invalidating the one-sided conclusion.
Critical slowing down: Comparability of the spectral edge density determines only the decay exponent and Θ bounds, whereas an exact edge density is required for the asymptotic coefficient.The distinction prevents treating unspecified or oscillating comparability constants as an exact equivalent.
Application to PLDR-LLM: Whether this reversible-family spectral dictionary describes PLDR-LLM remains Conjecture 8.2 because trained attention operators vary across layers and tokens and lack published criticality measurements.The correspondence is therefore interpretive rather than an established model description.
Self-organized criticality: Technical self-organized criticality is not established: experiments show critical-like optimizer phenomenology, but required self-tuning, scale separation, and standard statistical discriminants remain unmeasured.The section consequently uses self-organized criticality as a phenomenological framework, not established physics.

7. Advantages of PLDR-LLM over SDPA-LLM · 8. Conjectures

PLDR-LLM extends SDPA-LLM by learning an exposed operator sector, enabling distinct training dynamics, structural diagnostics, and cached inference, while incurring higher training costs and lacking benchmark dominance. Section 8 separates these advantages from falsifiable conjectures about invariant operators, spectral behavior, and domain transfer.

7.1. A larger parameterized family with distinct training dynamics.: PLDR-LLM contains SDPA-LLM exactly at G_LM = I, while its learned operator sector produces distinct, generically nonlinear training dynamics.Strict function-class containment at matched resources is not proved.
7.1. A larger parameterized family with distinct training dynamics.: 4032 additional head-level score-function dimensions are available to a reference PLGA head with d_k = 64 under the stated RoPE and reachability assumptions.The count concerns a single-head operator-sector codimension, not complete-LLM function-class separation.
7.2. An inspectable law representation.: Exposed positive operators enable DAG regularization, operator metrics, exponent readout, and structural inspection unavailable in SDPA’s fixed identity operator sector.The DAG loss improved benchmark scores without scaling model or data and separated models with indistinguishable loss curves.
7.3. An intrinsic evaluation diagnostic (prospective).: m(θ) separates near-critical from sub-critical PLDR-LLMs and agrees with curated-benchmark rankings at the phase level in the published sample.Finer within-phase discrimination remains open, and SDPA’s constant operator makes its corresponding fluctuation diagnostic identically uninformative.
7.4. Phase-aware, self-instrumented training.: PLDR training can reject bad runs using m(θ), DAG losses, and dragon-king events, whereas loss and accuracy can miss token-salad or behaviorally divergent models.The proposed spectral interpretation remains predictive until Conjecture 8.2 is tested.
7.5. Inference efficiency and a deployment asymmetry.: ∼3× speedup is measured with KV-cache and G-cache, while aggregate deductive-output statistics remain stable to 15 printed decimal digits and cached benchmark evaluations are unchanged.The deep PLGA subnetwork runs once per prompt and is then removed from the decoding loop.
7.6. Inductive bias and transfer.: The transferable pair (G*, P) supports a proposed operator-transfer program and reflects an input-conditioned inductive bias absent from SDPA’s fixed Euclidean score form.The transfer advantage is conjectural rather than established.
7.7. Costs, trade-offs, and honest limits.: Training costs include parameter ratios #ResL/#A ≈129–149, criticality-search failures, and only comparable rather than dominant benchmark scores at the published small scale.The evidence base uses single runs per condition without uncertainty estimates or same-stack matched baselines.

9. Discussion and Open Problems · Disclosure of the use of AI tools · Appendix A. Notation

The discussion consolidates the program’s results at distinct epistemic levels, while identifying training dynamics, renormalization, commutant behavior, and empirical validation as open problems. It also discloses AI-assisted preparation and provides notation for the model, operators, and control quantities.

9. Discussion and Open Problems: The program combines exact algebraic results, conditional mechanisms, spectral identities, and phenomenological interpretations, with each claim assigned its own epistemic level.These include the unique scale-covariant power-law interaction, Perron–Frobenius structure, inference collapse, conditional rotary-concentration-contraction analysis, and walk-counting and commutant identities.
9. Discussion and Open Problems: Open problems include deriving collapse dynamics from gradient flow and determining whether the constant-map manifold is an attracting invariant manifold.The proposed analysis concerns the (4.11)-augmented cross-entropy dynamics and critical (ηmax, Tw).
9. Discussion and Open Problems: Further open problems seek an explicit coarse-graining RG map and measurements of trained operators’ distances from the RoPE commutant across layers, scales, checkpoints, and trajectories.The audited checkpoint reports the commutant residual ∥GLM −ΠcommGLM∥F / ∥GLM∥F per layer and head; related work would compare off-commutant components with the O(1/S) twirl bound.
9. Discussion and Open Problems: The empirical program calls for multiple seeds, dense control grids, several model scales, independent corpora, uncertainty estimates, blinded phase criteria, and matched SDPA comparisons.These designs address single-run evidence, uncertainty in m(θ) and benchmark scores, and circularity in phase labeling.
9. Discussion and Open Problems: Recommended validation extends to singular-value spectra, numerical ranks, trajectory-level Jacobian and twirl diagnostics, end-to-end Lipschitz factors, and finite-size scaling for SOC.The stated precondition is to establish matched comparisons before future claims of benchmark superiority or causal phase effects.
Disclosure of the use of AI tools: The author discloses Claude use for organizing source material, drafting text and proofs, cross-checking architectural equations, and drafting selected Lean 4 formalizations.The disclosed formalization was released with the article and mechanically verified, according to the passage.
Appendix A. Notation: Appendix A defines notation for vocabulary and context sizes, model dimensions, query/key/value matrices, rotary inputs, density operators, and the shared metric learner.It also identifies the rotary-twisted density form and the row map associated with the metric learner.
Appendix A. Notation: The notation appendix also defines the order parameter m(θ), control parameters (ηmax, Tw), the invariant cached operator G∗, and related budget and conditional-probability quantities.The passage specifies G as the budget constant and pθ(· | x1:t) as the online final-row conditional.

Result dependency map. · Appendix B. Code, Models, and Verification Resources

The result dependency map links each numbered theorem and proposition to its direct proof inputs and stated hypotheses. Appendix B documents pinned implementations, released models, machine-checked proofs, audit artifacts, and the block-scoring evaluation protocol.

Result dependency map.: The dependency map serves as a navigation aid, while each result states its own hypotheses and cites classical inputs where used.The listed direct inputs include architectural linear algebra, Perron–Frobenius theory, nonresonance commutants, walk expansions, softmax characterization, substitution, and chain-rule arguments.
Appendix B. Code, Models, and Verification Resources: The architectural equations in Sections 3 and 4 were verified against reference implementations at exact snapshots recorded in the verification manifest.The paper defines “every equation verified” relative to those pinned snapshots.
Appendix B. Code, Models, and Verification Resources: Released repositories cover PLDR-LLM self-organized-criticality, KV/G-cache, and original TensorFlow frameworks, including predefined and transplanted GLM ablations.The listed implementations specify framework versions and initialization differences, including WV initialized as WQ or retaining framework defaults.
Appendix B. Code, Models, and Verification Resources: Lean 4 and Python resources provide kernel-rechecked proof cores, numerical-audit code, dependency specifications, results files, and raw Appendix D arrays.The Lean formalization builds with no unproved obligations, and lake build re-verifies every proof with the Lean kernel.
Appendix B. Code, Models, and Verification Resources: Hugging Face hosts pretrained PLDR-LLM families and custom Transformers ports whose configurations record the reference hyperparameters used in the paper.The repositories include pldrllmv5/v9, PLDR-LLM-v51, PLDR-LLM-v51G, PLDR-LLM-v51-SOC, and v52 fine-tuned variants.
Appendix B. Code, Models, and Verification Resources: Benchmark evaluations use pinned EleutherAI-harness forks with a PLDR-LLM wrapper implementing one-pass block scoring for answer candidates.Context and candidate are concatenated, all but the last token are evaluated, and candidate log-probabilities are summed.
Appendix B. Code, Models, and Verification Resources: One-pass block scoring equals sequential final-row scoring exactly for one-token candidates and matches audited multi-token scores within Appendix D’s measured gaps.Published scores use a single uncached pass; the generation path is not used for published scores cited in the paper.

Appendix C. Lean Formalization: Exact Coverage · Appendix D. Numerical Audit on a Released Checkpoint · D.1. Setup and provenance.

Appendix C distinguishes Lean-kernel-checked proof cores from prose-only claims and explicitly records the formalization’s trust boundary. Appendices D and D.1 define a numerical audit of measurable quantities on a pinned released checkpoint and its fixed-prompt setup.

Appendix C. Lean Formalization: Exact Coverage: Lean 4 builds with no sorry, admit, axiom, or unsafe declarations, while the stated trust boundary remains mathlib’s classical principles rather than foundational axiom-freedom.Continuous integration audits exported theorems for axioms beyond propext, Classical.choice, and Quot.sound.
Appendix C. Lean Formalization: Exact Coverage: The kernel-checked coverage includes positivity and power-law identities, softmax properties, density-operator facts, rank-one structure, LayerNorm invariances, RoPE block commutants, twirl bounds, and contraction results.These modules formalize selected proof cores, including A ↦ A⊙tP group laws, exact softmax behavior, and multiplicative or iterated-map contraction bounds.
Appendix C. Lean Formalization: Exact Coverage: The formalization does not establish statistical concentration, multi-block nonresonance, codimension counts, spectral-side identities, frequency-resolved constants, or stochastic applications.It also excludes several quantitative inequalities and spectral-radius statements listed as prose-only.
Appendix C. Lean Formalization: Exact Coverage: The NOTEARS walk-side theorem and positivity obstruction are checked, but the acyclicity converse and normalized-loss floor remain prose-level consequences.The checked result includes tr e^N ≥ d + dϵ; log(1 + ϵ^2) is derived by prose monotonicity rather than a separate Lean statement.
Appendix C. Lean Formalization: Exact Coverage: Inference-collapse and constant-operator absorption are kernel-checked, whereas trained-unit contraction, full-decoder formalization, checkpoint measurements, and model-level decoding induction remain outside exact coverage.The checked inference statements include Q·I·K⊤ = QK⊤ and the affine absorption identity in the no-RoPE setting.
Appendix D. Numerical Audit on a Released Checkpoint: Appendix D audits singular-value spectra, LayerNorm scale error, pre-rotation twirl energy, commutant residuals, and contraction diagnostics on a released checkpoint.The appendix substitutes singular-value spectra for determinants and measures the full-composition contraction diagnostics.
D.1. Setup and provenance.: The audit uses the unmodified fromthesky/PLDR-LLM-v51-SOC-110M-5 port at the pinned verification revision with transformers 4.56.1, eager attention, and float32.The model configuration is L = 5 layers, h = 14 heads, and d_k = 64.
D.1. Setup and provenance.: Inputs are eight fixed English prompts of 32–43 tokens spanning technical, narrative, review, and instructional registers; aggregation is min/median/max across layers, heads, and prompts unless stated.The prompts contain 150–250 characters, and the run uses a single consumer GPU.

D.2. Singular values and numerical rank (in place of determinants), trained versus initialized.

Across 560 layer-head-prompt instances, A exhibits an exceptionally sharp rank-one collapse with nearly identical rows and cross-head structure, whereas A_LM remains near full numerical rank despite float-zero determinants. A trained-versus-initialized control applies the same battery at random initialization under two explicitly specified initialization laws.

Singular values and numerical rank: A has numerical rank 1 in all 560 instances, with σ2/σ1 ≤1.4 × 10^-8 and median 8.7×10^-17.Rows agree within heads at a variance ratio of ≈4×10^-13, and across heads at relative RMS ≤1.6×10^-8.
Singular values and numerical rank: A_LM is not numerically low-rank: σ2/σ1 has median 0.27 and numerical rank median 62.5 of 64, ranging from 1 to 64.Its float determinant is zero despite many nonzero singular values and σ1 spanning 6.4 × 10^-8–6.2 × 10^3.
Singular values and numerical rank: The measurements confirm that A’s collapse is sharp, while float-zero determinants of A_LM do not indicate rank deficiency.The nonlinear image A_LM is generically of near-full numerical rank.
Trained versus initialized: The trained-versus-initialized control runs the same battery on the same architecture at random initialization under two initialization laws.One uses the Hugging Face port’s Xavier-uniform draws for W, P, a, and dense weights; the native training law uses Xavier-normal for W, P, a and Xavier-uniform dense weights.

D.3. The LayerNorm scale error, measured directly.

Direct measurement shows that the LayerNorm scale discrepancy is typically small in absolute terms but has a substantial long tail in relative terms. The analytical proxy overestimates measured relative error and diverges on exactly constant rows.

Row variance: 2.6 × 10−5 median row variance was observed across 35,840 pooled head rows, with variance ranging from 0 to 0.29.The variance distribution provides the denominator context for the measured LayerNorm discrepancy and its proxy.
Direct measurement: The discrepancy was computed directly for every captured row in float64 using the checkpoint’s γ, β, εLN, and each prompt’s actual S.The relative error is normalized by the reference output, with value 0 when the two outputs coincide.
Measured error: 4.6 × 10−4 median relative error coexists with a long tail: 4.3 × 10−2 at the 90th percentile, 0.44 at the 99th, and 4.3 maximum.The maximum occurs on ε-dominated rows.
Measured error: 5.4 × 10−10 median absolute error was measured, with a maximum of 1.1 × 10−5.Thus, relative and absolute error summaries differ substantially across captured rows.
Proxy comparison: 1.9 × 10−2 median proxy error overestimates measured relative error by approximately 35× at matched rows and diverges on 14 exactly constant rows.On those constant rows, the true discrepancy is exactly zero.

D.4. Twirl energy against pre-rotation data. · D.5. Commutant residual of the trained operators. · D.6. Row-map contraction, measured on the composition.

The audit finds that rotary twirling removes only a modest fraction of instance-specific off-commutant energy, while trained operators remain far from the RoPE commutant. Direct composition measurements nevertheless show extreme row-map contraction, despite individual units not being contractive.

D.4. Twirl energy against pre-rotation data.: 10.6% median pooled off-commutant suppression, ranging from 0.009% to 43%, indicates limited twirl removal under pooled aggregation.Ratios do not commute with pooling, so both aggregation labels are reported separately.
D.4. Twirl energy against pre-rotation data.: 33% median relative difference between first-half and second-half aggregates, reaching 90%, rejects the stationarity idealization at audited context lengths.The section concludes Stage 1 cannot by itself explain observed invariance; attribution to Stage 3 relies on direct measurements.
D.5. Commutant residual of the trained operators.: Every trained operator is far from the RoPE commutant, with most Frobenius energy in the d^2_k − d_k absolute-position-sensitive directions.This is checkpoint-scoped occupancy evidence, not a causal claim about decoding use.
D.6. Row-map contraction, measured on the composition.: 3.2 × 10^-11, 1.4 × 10^-16, 1.0 × 10^-19, 4.9 × 10^-12, and 3.5 × 10^-7 are the per-layer medians of the full-composition Jacobian’s largest singular value.The pooled range is 1.0 × 10^-19–3.6 × 10^-7 across layers 1–5.
D.6. Row-map contraction, measured on the composition.: ≈95% of retained pairwise contraction ratios are exactly zero at float resolution, and all others are at most 4.6 × 10^-6.The audit retained 765 within-prompt and 694 cross-prompt pairs after exclusions.
D.6. Row-map contraction, measured on the composition.: 0.47 median and 20.8 maximum per-unit Jacobian norms show individual units are not contractive, unlike the measured full composition.The strong per-unit hypothesis Lj ≤ κ < 1 is false on this checkpoint, while composition contraction is at least seven orders stronger.

D.7. The invariance budget as a sample-extrema proxy, against measured margins. · D.8. DAG-loss values.

The sample-extrema invariance budget is explicitly non-certifying: its assembled bound overwhelms measured decoding margins despite exact greedy agreement. The adjacent DAG-loss section reports overflow-safe per-instance values for A_LM and A_P alongside two stochastic continuation normalizations.

D.7. The invariance budget as a sample-extrema proxy, against measured margins.: The invariance budget is a sample-extrema proxy, not a certificate, because prompt-only extrema do not cover grown-context states used for margin comparison.Other limitations include endpoint LayerNorm variances, unrelated samplewise extrema, and the absence of a common perturbation set.
D.7. The invariance budget as a sample-extrema proxy, against measured margins.: 10^-9 is the only guaranteed ALM floor along perturbation paths crossing an iSwiGLU zero crossing; sampled minima are not pathwise lower bounds.The measured minimum of A_LM therefore cannot replace the architectural floor in a verified pathwise argument.
D.7. The invariance budget as a sample-extrema proxy, against measured margins.: 7.3 × 10^46 is the assembled end-to-end coefficient multiplying ∥∆G_LM∥2 in the logit bound, so the factorwise norm-product route cannot certify the margin.The assembly uses layer-specific injection-point remainders and certified derivative envelopes, but a genuine certificate would require verified propagation over every compared decoding state.
D.7. The invariance budget as a sample-extrema proxy, against measured margins.: 2B = 7.2 × 10^-5 < 6.0 × 10^-3 = ∆, while greedy token choices agree at every step across 48 tokens and 4 prompts.The maximum logit deviation is 3.6 × 10^-5, with a minimum realized top-two margin of 6.0 × 10^-3.
D.7. The invariance budget as a sample-extrema proxy, against measured margins.: 2.4 × 10^42 is the proxy value at the hypothetical single-rounding radius, exceeding the halved minimum margin ∆/2 = 3.0 × 10^-3 by approximately 45 orders of magnitude.Replacing sampled factors by containing-tube suprema cannot improve this same factorwise assembly.
D.8. DAG-loss values.: A_LM DAG loss has minimum 0, median 3.6 × 10^-10, and maximum 6.1 × 10^-5 across 560 instances.The minimum reflects floating-point underflow readings below the analytic and un-normalized NOTEARS floors’ float64 resolution.
D.8. DAG-loss values.: A_P DAG loss has minimum 63, median 1.1 × 10^3, and maximum 4.1 × 10^4 across the same 560 instances.The section also reports two normalizations for two independent stochastic 64-token, temperature-1 continuations of 5 prompts.

D.9. Order parameter, two normalizations.

The order parameter is evaluated under RMS and symmetrized signed-mean normalizations. GLM is exactly invariant at float resolution, while AP, ALM, and A show small but non-bitwise deviations on the checkpoint.

Measured order parameter: GLM’s order parameter is exactly 0 at float resolution on every layer under both normalizations.The signed-mean normalization uses denominator 2(|µ1| + |µ2|).
Implementation audit: The audit shape-checks the per-head exponent tensor and uses a sentinel semantic test to reject silent head- or row-broadcast indexing errors.The exponent parameter has shape [h, dk, dk] with no batch axis, unlike the batched activation tensors.
Measured order parameter: 8.6 × 10−13 is AP’s maximum under RMS normalization, versus 3.3 × 10−12 under signed normalization.These values are zero to twelve digits but not bitwise.
Measured order parameter: 3.4×10−9 and 5.6×10−9 are ALM’s respective maximum deviations under RMS and signed normalization, with medians 0.The two normalizations’ maxima are quoted separately.
Measured order parameter: 4.9 × 10−9 is A’s maximum under RMS normalization, while its signed-mean maximum is 3.1 × 10−8; the RMS median is 0.These measurements are reported for the most sensitive tensor A on the checkpoint.

D.10. The online contract, historical-row movement, and padding. · D.11. Sequential versus one-pass block scores.

The online-contract audit finds deterministic prefix behavior, bitwise-equal historical rows, and checkpoint-dependent effects from attention-masked padding. Sequential and one-pass candidate scores coincide for controls and differ only minimally on the audited checkpoint, but substantially more on the contrast checkpoint.

D.10. The online contract, historical-row movement, and padding.: Two back-to-back float32 eager runs reproduce the online-contract results under pinned deterministic settings, with a CPU fallback provided.The script uses the same consumer GPU and determinism configuration as the main audit, and records raw arrays and result-file SHA-256 values.
D.10. The online contract, historical-row movement, and padding.: Every shared-prefix comparison is bitwise equal when same-length inputs differ only in their suffix, including the fixed and randomized protocols.The randomized protocol uses 16 pairs of length 12 with prefix length 4, plus suffix comparisons with every suffix position attention-masked.
D.10. The online contract, historical-row movement, and padding.: Repeated identical prefix calls are bitwise equal on both checkpoints, making the step-t conditional deterministic in this configuration.This directly supports the stated deterministic-function interpretation of the prefix.
D.10. The online contract, historical-row movement, and padding.: 8.6 × 10−6 is the maximum final-real-token deviation after four masked padding tokens on the audited checkpoint, while GLM remains bitwise unchanged.The deviation is exactly zero for one of the two prompts; on the contrast checkpoint, GLM moves by 4.5×10−3–1.2×10−2 and the final-row deviation is 3.5–7.4×10−5.
D.10. The online contract, historical-row movement, and padding.: Removing the masked padding restores the unpadded call bitwise on both checkpoints, establishing the measured unpadded-S=t contract.The contrast checkpoint’s padding dependence demonstrates that masked rows enter the Gram.
D.11. Sequential versus one-pass block scores.: Single-token candidate scores coincide exactly on both checkpoints because sequential and block protocols invoke the model on the identical tensor.The fixed suite contains eight prompts, four multi-token candidates, and four single-token controls per prompt.
D.11. Sequential versus one-pass block scores.: 3.3 × 10−6 is the audited checkpoint’s maximum per-token and per-candidate total-score gap for multi-token candidates, with median per-token gap exactly 0.The contrast checkpoint has roughly fifty-fold larger gaps: per-token maximum 1.5 × 10−4 and per-candidate maximum 2.2 × 10−4.

D.12. Held-out sequential NLL and real benchmark items under both scoring protocols. · D.13. Scope.

Across held-out text and sampled benchmark items, blockwise and sequential scoring closely agree, with identical argmax decisions and near-identical TruthfulQA probability-mass scores on the tested samples. The audit remains limited by checkpoint, sample, software, and numerical-launch sensitivity.

D.12. Held-out sequential NLL and real benchmark items under both scoring protocols.: 3.479446 versus 3.479446 nats/token differed by 1.8 × 10−9 on the audited checkpoint’s held-out windows.The per-token absolute-gap median was 9.6 × 10−7, with a 2.4 × 10−5 maximum.
D.12. Held-out sequential NLL and real benchmark items under both scoring protocols.: 3.684883 versus 3.684879 nats/token differed by 4.3 × 10−6 on the contrast checkpoint, with larger, nearly sign-balanced per-token gaps.The median and maximum gaps were 1.8 × 10−5 and 2.8 × 10−3; 49.3% of positions scored worse sequentially.
D.12. Held-out sequential NLL and real benchmark items under both scoring protocols.: Per-token gap medians decreased from 2.5–3.8 × 10−5 early to 5.0 × 10−6 in the final position bucket, consistent with suffix-length dependence.The reported mechanism is that later historical rows leave less suffix to move the Gram; these measurements do not establish behavior beyond the tested sample.
D.12. Held-out sequential NLL and real benchmark items under both scoring protocols.: The D.12 findings are sampled measurements from 100 items per task, one seed, two checkpoints, and one software stack, not full-set validation.They do not remove the target-exposure structure of Section 4.3 or answer its separate training-time question.
D.12. Held-out sequential NLL and real benchmark items under both scoring protocols.: Zero raw argmax changes, zero length-normalized argmax changes, and zero discordant candidate pairs occurred across the seven argmax tasks on both checkpoints.The comparison covered 100 sampled items from each of eight published zeroshot tasks, totaling 800 items and 3,052 candidates.
D.12. Held-out sequential NLL and real benchmark items under both scoring protocols.: 5 × 10−5 per item was the agreement bound for the published TruthfulQA probability-mass metric, despite measurable score differences on the contrast checkpoint.Per-candidate absolute gaps reached 2.8 × 10−5 on the audited checkpoint and 2.3 × 10−3 on the contrast checkpoint.
D.13. Scope.: The audit covers one checkpoint, eight prompts, and one software stack, remaining far from full validation sets, checkpoint fleets, and training trajectories.Jacobian and pairwise statistics are sample-sensitive, and downstream quantities can vary by more than low-order digits across configurations because the power stage locally amplifies by approximately 10^10.
D.13. Scope.: Several orders of magnitude of launch-to-launch variation affected the smallest order-parameter statistics, while pinned deterministic settings reproduced reported numbers bitwise.The observed relative shifts were within ≲10−3 across repeated launches of the identical seeded script.