PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction

Kingsuk Maitra, Shagun Sood Morteza Hosseini, Suman Gunnala, Vikram Gupta

arXiv:2609.32319v1cs.LGcond-mat.mtrl-scics.AI

TL;DR

The paper examines how a one-layer attention model can address induction and how far a geometric-integration correspondence applies. It studies a symplectic attention operator and reports both cross-task results and limits on practical-scale claims.

  • Problem

    At decode step t, forming sheared keys appears to require retaining raw keys and recomputing all t keys, implying O(td) extra work per token and a second cache.

    Source: §8.2 · p. 64

  • Method

    The paper uses geometric numerical integration to motivate an attention operator and examines post-RoPE placement, nonlinear constraints, and circuit attribution.

    Source: §4.6 · p. 24, §5.3 · p. 30, §7.14 · p. 59

  • Results

    Across 360 runs, Variable Tracking gains up to +7.4 pp at low theta and medium difficulty, while Arithmetic CoT gains up to +60.6 pp at hard difficulty, low theta, γ = 0.3.

    Source: A20.7 Cross-task summary · p. 155

  • Takeaways & Limitations

    For induction accuracy alone, the key-only half-lift is the better engineering choice, reaching 0.949 against 0.811 for the symmetric member.

    Source: §9.3 · p. 71

    No gain at language-model scale is claimed, and practical benefit at realistic scale remains unestablished.

    Source: §9.3 · p. 71

Abstract

from arXiv · show

We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a $z$-transform of joint order $(0,0)$, whereas the induction discriminator requires key-side order $\geq 1$. Applying the symplectic upper shear $M_\gamma:(q,p)\mapsto(q+\gamma p,p)$ to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit $T_4$-only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at $r=0.9876$ with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the $T_3$ channel moves $\gamma_c$ at $d_k=64$ from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is $6d$ FLOPs per token per layer, INT8 headroom grows by at most $\log_2(1+2\gamma)$ bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at $\gamma=0.80$ in a mean of 717 steps, against 2/3 seeds and 2700 steps at $\gamma=0$. Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.

2 The single-layer induction obstruction as a filter-order gap

This section frames the single-layer induction obstruction as a filter-order gap—standard attention has joint order (0, 0), whereas induction requires key-side order ≥1—and relates that deficit algebraically to one-step integration stencils and the Hairer phase-space lift. The shear is presented as a shared representability remedy, not an accuracy ordering; its linearly growing iterates mean exact symplecticity does not establish boundedness.

2.1 The two-variable transfer function

  • Standard attention is structurally order (0, 0): widening heads, summing heads, or enlarging pointwise query/key maps cannot raise its joint polynomial degree.This is a diagnostic restatement internal to the bilinear class, strictly weaker than SHT’s theorem; the proposed augmentation exits the counting argument’s premise, not its bound.Source: §2 · p. 10, §2.1 · p. 11, §2.2 · p. 13

2.2 What the induction discriminator requires

  • A key-side order-zero score has an induction ceiling of 1/(T −2), independent of embedding dimension and precision, under the stated canonical induction model.Conditional on the query, distractor scores are exchangeable; suppressing the previous-token position attains the ceiling, while aligning with it can drive the peak probability toward zero.Source: §2.2 · p. 12

2.3 The order lattice

  • The order lattice identifies (0, 1) as the minimum representable lift; query-only (1, 0) retains the ceiling, while symmetric (1, 1) adds structure, not representability.Every lattice member is exactly symplectic, and (1, 1) is the unique swap-invariant member; the key-only half-lift is reported as more accurate than the symmetric completion.Source: §2.3 · p. 13

3 A parallel obstruction in numerical integration

  • The section frames the obstruction and its resolution as a parallel algebraic state-augmentation motif in attention and numerical integration.The integrator results transfer only at the level of algebra; the paper explicitly excludes long-time dynamical consequences for attention.Source: §3 · p. 14, §3.3 · p. 15

3.1 The stencil obstruction

  • A one-step linear map on configuration alone has degree zero in z^-1 and cannot reproduce Verlet’s two-step dependence on q_n−1.Forward Euler is first order, with local truncation error O(∆t^2) and global configuration error O(∆t); on the oscillator its energy drifts, and it is not volume-preserving.Source: §3.1 · p. 14

3.2 The Hairer lift

  • Hairer’s phase-space lift stores lagged information in p, turning Verlet into a one-step symplectic map; its free-particle drift is the exact shear Mγ.The shear is an exact flow, not a small-γ approximation, and it supplies the missing first-order dependence in the base variable.Source: §3.2 · p. 15

3.3 Backward error analysis and the quadratic potential

  • Backward-error analysis and long-time energy claims concern iterated integrators and do not carry over to attention, which has no iterate to interpolate.For the free-particle shear, the modified-Hamiltonian expansion collapses exactly to the original kinetic Hamiltonian at every order.Source: §3.3 · pp. 15–16
  • The shear has unit spectral radius but is not power-bounded: iterates grow linearly in n rather than exponentially.Its unit-circle eigenvalue is defective. Exact symplecticity and preservation of the joint Hamiltonian at each application do not establish boundedness of iterates.Source: §3.3 · p. 17

3.4 The correspondence

  • The correspondence aligns stencil order 0→2 with joint filter order (0,0)→(1,1), resolving both representability obstructions by changing the state.It identifies an operator, not an accuracy ranking: the key-stream Euler-class member solves induction, while the symmetric Verlet-class member is distinguished by structure. Long-time numerical-integration results that rely on hypotheses absent from attention do not transfer.Source: §3.4 · p. 17

4 Phase Space Attention as the Hairer lift

This section develops Phase Space Attention’s symplectic upper shear of post-RoPE query and key streams, which uses their first differences to augment the score while leaving values untouched. It examines the resulting filter-order and spectral effects and design alternatives, establishes uniqueness within the stated factorised linear one-step class, and distinguishes symplectic structure from score symmetry while noting that generic nonlinear extensions do not automatically preserve it.

4 Phase Space Attention as the Hairer lift

  • The post-RoPE upper shear augments queries and keys with their first differences, leaving values unchanged; the algorithm adds O(Tdk) time and memory and no learnable parameters beyond γ.The boundary convention sets the first position’s momentum to zero, avoiding an artificial first-token impulse.Source: §4 · p. 18

4.1 The four-term decomposition and the Ghost Key

  • The Ghost Key places a predecessor-dependent component in the score-bearing key, giving access to (σ_j−1, σ_j) without a second attention layer or contradicting the SHT bound.This is a mixture, not an injective encoding; the four score terms comprise standard, two cross, and double-ghost terms, with T3 carrying a first-order gap and T4 a second-order gap under the canonical induction model.Source: §4 · p. 18, §4.1 · p. 19

4.2 Filter-order audit

  • The augmentation raises the bilinear score from joint filter order (0, 0) to (1, 1), closing the filter-order gap identified for standard attention.Each stream’s causal FIR filter is high-pass: its gain is 1 at DC and 1 + 2γ at the Nyquist frequency.Source: §4.2 · p. 20

4.3 Spectral consequences of the shear

  • The shear reweights the score spectrum by the product of the two stream responses, concentrating added power toward high frequencies while leaving the momentum coordinate unchanged.Momentum invariance does not make the shear an isometry; the position block can grow under iteration.Source: §4.3 · pp. 20–21

4.4 The design space, and why the upper shear

  • The upper shear shifts position by momentum, matching the first-difference structure; the lower shear is score-null and all-pass, skew generators fall outside the factorised subclass, and encoding momentum in RoPE loses relative-position factorisation.The text cautions that joint-space orthogonality alone does not imply an all-pass projected score filter.Source: §4.4 · pp. 22–23

4.5 Uniqueness within the factorised subclass

  • The uniqueness analysis identifies a single scalar of remaining freedom in the factorised subclass, with stream symmetry treated as an independent design commitment.The stated setup requires post-RoPE streams, bilinearity, causality, one-step memory, and per-stream high-pass behavior.Source: §4.5 · p. 23
  • The augmentation is the unique factorised, linear one-step, stream-wise post-RoPE member satisfying (C1)–(C3.2), up to shared γ.The uniqueness claim does not cover nonlinear shears, multi-step memories, cross-stream pre-pairing, or score augmentations outside the bilinear form.Source: §4.5 · p. 24

4.6 Beyond the linear class

  • At dk=64, nonlinear symplectic shears require a symmetric Jacobian everywhere (codimension 2016), while generic learned MLPs do not preserve that constraint automatically.Even an admissible nonlinear g makes high-pass gain depend on embedding magnitude; only g(p)=cp is scale-uniform.Source: §4.6 · p. 25

4.7 Why the value stream is untouched

  • The value stream is left unchanged because it is a measurement channel, not a dynamical coordinate, and the score-side filter-order argument does not apply to values.Applying the shear to V would change layer outputs rather than the attention computation.Source: §4.7 · pp. 25–26

4.8 Two structural propositions

  • Equal couplings yield a score comprising the base pairing, its transported increment, and pure coherence; unequal couplings remain exactly symplectic but add a linear score bias.The bias has no energy component, so symplecticity alone does not select the symmetric member.Source: §4.8 · pp. 26–27
  • The asymmetric variant trains to 0.875, the matched-count low-pass member collapses to 0.428, query-only half-lift stays at 0.033 versus a 0.077 ceiling, and key-only half-lift reaches 0.949.These results show that symplecticity alone does not identify the empirical discriminator; the paper points to DC-neutral high-pass structure and stream placement.Source: §4.8 · p. 27

5 Operator-level symplectic invariance and the placement corollary

The section establishes exact operator-level symplectic invariance while delimiting what that per-application result does not imply about repeated dynamics. It also shows why the augmentation must be applied post-RoPE: pre-RoPE placement adds a Coriolis term despite preserving symplecticity.

5.1 The invariance

  • Mγ preserves the canonical symplectic form and Hjoint exactly on each application, but the perturbation estimate is not a stability guarantee.The continuity estimate bounds a single application; it says nothing about repeated application or long-term behavior.Source: §5.1 · pp. 27–28

5.2 What transfers from geometric integration, and what does not

  • The attention map does not iterate on its own output, so the correspondence with geometric integration supports representability claims, not long-time dynamical guarantees.The shear is applied once per forward pass, and no modified-Hamiltonian, long-time energy-bound, or secular-drift claim is made for attention.Source: §5.2 · pp. 28–30

5.3 The placement corollary

  • Post-RoPE placement is required to retain the per-stream high-pass character; pre-RoPE placement introduces a Coriolis term but does not break symplecticity.The term vanishes as the RoPE angle θ approaches zero and is largest in high-frequency bands; the pre-RoPE variant degrades consistently but not catastrophically in the reported sweep.Source: §5.3 · p. 30

6 Closed-form single-layer phase transition

This section develops a closed-form, two-branch transition for a T4-only reduced score model and specifies its limits, including how it differs from induction-head emergence during training and the full attention mechanism. It also reports score-channel and positional-cancellation results, contrasts query-only and key-only half-lift corollaries, and notes that the transition law depends on a distinct distractor-scale convention and remains untested against simulation.

6 Closed-form single-layer phase transition

  • The coupling transition is an operator property at fixed training budget, distinct from training-time induction emergence at fixed architecture; neither predicts the other.The passage associates the training-time transition with Olsson et al. and Reddy.Source: §6 · p. 31

6.1 The softmax: Fréchet linearisation and the DC/AC split

  • The softmax linearisation and DC/AC split are conditional on a regime measured in §7, while the two-branch theorem does not depend on them.The softmax correction is linear in the AC score perturbation, with a bounded remainder.Source: §6.1 · pp. 31, 33

6 Closed-form single-layer phase transition

  • In the T4-only reduction, the two branches cross at d*_k = 8L, with a step exactly equal to the calibrated factor c.The figure cautions that this branch structure is reduction-sensitive and does not survive restoring T3.Source: §6 · p. 31, Figure 2 · p. 32

6.2 The sixteen-term decomposition and the induction driver

  • Under isotropy, T3 carries a first-order induction gap and T4’s AA sub-term a second-order gap; the T4-only model omits T3, which is larger below γ = 1.Thus the closed-form transition describes the reduced model, not the full four-term decomposition.Source: §6.2 · pp. 33–34

6.3 Signal-to-noise and the coupling-invariance of the ratio

  • In the T4-only model, the score-gap SNR is √d_k/2, independent of γ, so coupling changes softmax competition rather than target–distractor separation.The result follows from signal and noise scaling identically with γ.Source: §6.3 · p. 35

6.4 The tangent simplex and the Fisher geometry

  • The linearised attention response preserves its DC operating point and changes only the tangent component; its correction is orthogonal to the base point in the weighted ambient form, not generally in Euclidean geometry.The correction redistributes attention mass without creating or destroying it.Source: §6.4 · pp. 36–37

6.5 Cesàro cancellation of the single-ghost terms

  • Under the stated frequency and orthogonality conditions, pure positional components of the γ-linear T2 and T3 channels average away at O(1/N), including under RoPE.This does not remove the candidate-wise content gap in T3 or establish that T2 and T3 are non-discriminative.Source: §6.5 · pp. 37–38

6.6 The reduced Gaussian score model

  • The transition law applies to G, a reduced competition model that retains T4 and idealizes distractors as independent Gaussians, not a theorem about trained transformers.It omits the T3 gap, which dominates the retained γ^2 channel for γ < 1; its annealed characteristic scale is only a proxy for an empirical accuracy midpoint.Source: §6.6 · pp. 38–39

6.7 The two-branch law

  • Above dk = 8L the transition is an annealed crossing; below it, the transition is located at freezing, and the two branch formulas need not meet continuously.On the mean-field branch, γc scales asymptotically as (L^2/dk)^1/4, so wider heads require less coupling.Source: §6.7 · pp. 40–41
  • The mean-field branch predicts the transition at r = 0.9876 across nine grid points with zero fitted parameters.The condensation constant is 0.741 ± 0.015 on nine condensed grid points and agrees to 3% on a fixed-dk T-axis scan; this is a consistency check, not a held-out test.Source: §6.7 · p. 41
  • The quartic criterion predicts an elevated competition band rather than a half-line, including [0.787, 2.245] at dk = 128 and T = 512.The upper root is a reduced-model result, not a trained-transformer prediction; its attainment has not been tested.Source: §6.7 · p. 41

6.8 Corollaries delimiting the mechanism

  • A query-only half-lift leaves induction accuracy bounded by 1/(T −2) and cannot reach the two-layer induction solution in the canonical single-layer model.The bound follows because the score remains pointwise in the keys, leaving key-side order zero.Source: §6.8 · p. 42
  • The key-only half-lift admits a parallel transition law when dk ≥ 4L + 2.This is a separate reduced-model corollary, built on the channel omitted by the T4-only model.Source: §6.6 · p. 39, §6.8 · p. 42
  • Because the key-only discriminative channel is O(γ), the base fluctuation remains material, unlike the symmetric case’s O(γ^2) coherence channel.The key-only distractor scale therefore retains the base term; the paper warns that the two transition laws use different conventions.Source: §6.8 · pp. 42–43
  • The key-only transition is first order in γ, yielding a quadratic crossing equation whose roots are real when d_k ≥ 4L + 2.The transition is the smaller root encountered as γ increases from zero.Source: §6.8 · p. 43
  • The key-only law is untested against simulation, and applying its convention to the symmetric case shifts the prediction at (d_k, T) = (64, 15) from γ_c = 0.602 to 0.664, versus measured 0.613.The paper marks Equation (55) conditional; §7.5 measures key-only accuracy, not its critical coupling.Source: §6.8 · p. 43
  • Key-only reaches 0.949 accuracy, and the paper identifies it as the better engineering choice when single-layer induction accuracy is the objective.The symmetric completion is presented as part of a broader structural package rather than as the best construction for this task.Source: §6.8 · p. 43
  • Query-only scores 0.033 against chance 0.071, below the 1/(T − 2) ceiling whose exceedance would falsify the filter-order account.A value between 1/(T − 1) and 1/(T − 2) would instead be consistent with suppressing the previous-token position.Source: §6.8 · p. 43
  • The crossing analysis uses m = T − 2 distractors; scoring all T − 1 positions instead changes the limiting accuracy by O(1/T).The paper says this convention difference is below the error bars of its reported comparisons.Source: §6.8 · p. 43

7 Empirical signatures

This section tests the paper’s empirical claims through fixed-grid evaluations of the two-branch law, simulations and trained-transformer experiments, and probes of the depth and coupling conditions for PSA induction. It reports supporting notebooks, certificates, additional tests, and limits of the synthetic evidence, including that trained models use the single-layer route at every tested depth while the task favors ordinary two-layer composition.

7.3 Spectral character: two axes, not one

  • PSA’s per-stream score-axis high-pass and the RoPE→PSA embedding-axis low-pass are dual: momentum gain correlates with RoPE centre frequency at r = −0.679.The score-axis high-pass detects the embedding-axis low-frequency structure that Xiong et al. [51] identify as where RoPE concentrates structured energy.Source: §7.3 · p. 47

7.4 The single-layer induction phase jump

  • 69.1 ± 8.2% over five seeds at γ⋆=2.0 for single-layer PSA exceeds both 1.6% standard-attention baselines, a +67.5 pp mean phase jump under distractor noise.The single-layer and two-layer baselines are both at chance (1/64 = 1.6%); the figure’s separate plotted sweep peaks at 74.2% versus 1.27% at γ=0.Source: §7.4 · p. 47, Figure 3 · p. 48

7.5 The order lattice, measured

  • A freely trained two-tap filter reaches 1.000 by selecting a key-side first difference and leaving the query stream as identity.The matched-count two-tap low-pass reaches 0.428 versus 0.811 for the symmetric shear; the two-layer baseline records 0.036 at 3,000 steps and 0.024 at 9,000, while asymmetric key-heavy variants outperform their query-heavy counterparts.Source: §7.5 · p. 49
  • The key-only half-lift reaches 0.949 versus 0.811 for the symmetric operator, a 13.8 pp gap no tested coupling closes.The query-only variant records 0.033, 0.030, 0.033, and 0.030 across γ = 0.5, 1, 2, and 3, below the 1/(T −2) = 0.077 ceiling and the 0.071 chance benchmark.Source: §7.5 · p. 49, Figure 4 · p. 50

7.6 Bode forensics for the score stream

  • Pre-softmax AC-power correlations with the theoretical score-transfer curve are r = 0.8951, 0.9890, 0.9915, 0.9966, and 0.9924 at γ = 0.6, 1.2, 1.8, 2.0, and 3.0.The maximum is r = 0.9966 at γ⋆=2.0; standard attention produces a flat AC spectrum and a degenerate correlation. The normalized-shape measurement does not test absolute AC energy.Source: §7.6 · p. 50, Figure 5 · p. 52

7.7 What survives the softmax

  • Post-softmax Bode correlation is 0.94–0.97, while the high-pass survival claim is limited to the first-order regime.The score supnorm lies inside that regime on more than 96% of tokens; at τ=0.1, the HSF gap closes, Bode r falls to ∼0.41, and the transition shape is distorted.Source: §7.7 · pp. 51–52

7.8 The embedding-axis low-pass induction filter

  • Across 2,000 experiments, induction efficacy concentrates at low RoPE centre frequency and γ ∈[0.4, 1.4], with θ and per-frequency momentum gain correlated at r = −0.679.The Coriolis placement penalty scales as |sin(θ/2)| and is largest in bands carrying the least induction signal.Source: §7.8 · p. 52

7.9 The two-branch law against Monte Carlo

  • On nine mean-field points, the two-branch law predicts γc with Pearson r = 0.9876, median relative error 2.6%, and zero fitted parameters.Three fresh-seed checks agree within 6%. The piecewise law is discontinuous at the crossover while measurements are smooth, and its largest errors cluster nearby.Source: §7.9 · pp. 53–54, Figure 7 · p. 54

7.10 The condensed branch and the argmax limit

  • On eight condensed-branch points at fixed dk = 16, the context-length scan gives Pearson r = 0.9995, median relative error 3.1%, and maximum error 5.2%.The zero-parameter argmax limit differs from direct simulation by at most 9 × 10^-4; estimates of the condensation constant agree across axes to 3%, though they drift systematically with context length.Source: §7.10 · pp. 54–55, Figure 8 · p. 55

7.11 The trained-model dk ladder

  • In trained models, measured γc falls from 1.290 ± 0.044 at dk = 8 to 0.120 ± 0.006 at dk = 64, matching the law’s direction but not its scaling magnitude.The four-point log–log slope is −1.13 (95% CI [−1.18, −1.07]), versus the score model’s asymptotic exponent −1/4. Trained plateaus track the score-model argmax limit in shape (r = 0.945), not level; at dk = 16 the trained plateau exceeds it by 0.199.Source: Figure 9 · p. 56, §7.11 · p. 56

7.13 Post-softmax first-order response

  • The first-order post-softmax response is driven by the pre-softmax high-pass perturbation, but how much spectral emphasis survives the state-dependent projection is left to measurement.The relation to Differential Transformer [29] is only under a further smallness hypothesis on inter-query score differences; the paper claims no architectural identity.Source: §7.13 · pp. 57–58

7.12 Coupling, depth and length: a characterisation

  • With weights fixed, a model trained at γ = 2 stays within 90% of its best accuracy over evaluation γ ∈[1.5, 4], while extended-length accuracy is 0.626 at 24 pairs and 0.511 at 31.At depth 3, γ = 2 score RMS is 2.45 → 2.03 → 2.01, unlike the γ = 0 control’s 0.49 → 2.85 → 2.48; the extended-length results are 15× and 16× chance, with RoPE extrapolation and vocabulary budget as acknowledged confounds.Source: §7.12 · p. 57, Figure 10 · p. 58

7.14 Which circuit executes induction in a deep PSA model?

  • For γ ≥ 0.5, the single-layer probe reads 0.76 on average, while first-block shear ablation costs 0.874 accuracy through N = 8, showing a causally load-bearing first-block contribution.At N = 16 and γ = 2.5, global ablation costs 0.921 versus 0.656 for first-block ablation, indicating a distributed shear-dependent route at deep, strong-coupling settings; ablation also induces downstream distribution shift, so it does not identify the circuit conclusively.Source: Figure 11 · p. 59, §7.14 · pp. 59–60
  • At N ≥7, γ1/2 is between 0.25 and 0.5, yet the single-layer route remains available, adopted, and causally load-bearing at N = 16.The registered depth-crossing prediction fails by a factor of three; the route is available through depth sixteen, while depth affects adoption at γ = 0.25 through a step between N = 6 and N = 7.Source: §7.14 · pp. 60–62
  • γ1/2 is 0.17–0.45γc, while Appendix A4 measurements of 0.225 and 0.275 compare with predicted γc = 0.737, showing G over-predicts trained-model coupling needs by a factor of two to six.γ1/2 measures held-out probe decodability, not γc, the target-softmax transition in the T4-only Gaussian reduction.Source: §7.14 · pp. 61–62
  • 0.99 accuracy in 500 steps at γ = 0, versus 0.88–0.94 at γ ≥0.5; in a distinct 91.3M-parameter regime, γ = 0.80 yielded 3/3 emergence in 717 steps versus 2/3 in 2700 at γ = 0.The first result is for dk = 16, vocabulary 128, and sequence length 128, where classical composition already solves the task; the 91.3M sweep used dk = 128, vocabulary 8192, and sequence length 512. The results are configuration-specific, not contradictory.Source: §7.14 · p. 61

7.15 Machine-checkable certificates

  • Ten machine-checkable certificates pass in two environments, checking selected identities and comparisons in under one minute without GPU or training data.They are conditional on the paper’s definitions and assumptions; C7 and C8 are Monte-Carlo agreements, not proofs, and the certificates do not validate omitted terms or theorem scope.Source: §7.15 · p. 62

7.16 Additional independently verified predictions

  • The additional tests report a −52.5% repeated-token-loss regression, a multi-difficulty sweep with F = 12.04 and p = 7.4 × 10−4, and a negative control with t = −2.91 and p = 4.5 × 10−3.The sweep comprised 2,880 runs; the negative control supports the expected dissociation for derivative-sensitive score channels, including T4 coherence.Source: §7.16 · p. 63

7.17 What the programme does not show

  • The evidence is synthetic, establishes key-axis differencing as operative and the symmetric member as not accuracy-optimal, and confirms γc(dk)’s direction while leaving its exponent unexplained by a factor of four.The reported results come from the canonical induction sequence model or small decoders trained on it.Source: §7.17 · p. 63

8 Deployability: what the operator costs an inference engine

Section 8 derives the operator’s deployment costs and compatibility properties, finding no increase in KV-cache size and preserving prefix reuse and existing fused attention kernels. These exact-arithmetic results establish that common deployment objections do not apply, but do not establish accuracy or real-hardware latency.

8.2 Baseline, and the apparent obstacle

  • A sheared key depends on its current and preceding keys, so recomputing every past key or keeping a second cache is unnecessary.The apparent O(td) per-token work and second-cache requirement are both avoided by applying the causal two-tap filter at insertion time.Source: §8.2 · p. 64, §8.3 · p. 64

8.3 The KV cache is unchanged

  • The KV-cache footprint is unchanged: cache sheared keys instead of raw keys, retain two raw vectors per layer, and never recompute earlier entries.The additional persistent state is O(dL), independent of context length; at T = 4096 it is 0.024% of the baseline cache.Source: §8.3 · p. 64

8.4 Arithmetic overhead

  • 6d FLOPs per token per layer are added, with arithmetic cost below one part in five thousand in every tabulated regime.The vector pass is memory-bound, so its marginal wall-clock cost is smaller on bandwidth-limited decode hardware; no latency measurement is reported.Source: §8.4 · p. 65

8.5 Prefix reuse and speculative decoding

  • Prefix reuse remains valid because each sheared key depends only on tokens at or before its position; speculative verification applies the same causal filter.On rollback during speculative decoding, the engine restores retained raw vectors and needs one extra d-vector per speculative block, not per token.Source: §8.5 · pp. 65–66

8.6 Quantisation headroom

  • At fixed relative precision, INT8 keys require at most log2(1 + 2γ) additional bits; the bound is tight for an alternating key stream.The bound concerns representable range, not downstream accuracy of a quantized PSA model, which remains untested here.Source: §8.6 · p. 66

8.1 Who this section is for

  • The shear adds no parameters and leaves fused attention kernels unmodified; Section 8 establishes compatibility, not that the operator should be deployed.The results are exact arithmetic, not measurements, and the section does not establish wall-clock latency or quantized-model accuracy.Source: §8.1 · pp. 63–64, §8.7 · p. 67

8.7 Summary

  • Because the shear is a pre-pass on Q and K and leaves V untouched, existing fused attention kernels can be used without modification.The change is made at the call site rather than through a kernel rewrite.Source: §8.7 · p. 67

9 Related work, limitations, and discussion

Section 9 places the construction alongside prior attention and structure-preserving methods, distinguishes neighboring operators, and states limits on the evidence and theory. Its discussion treats the lift as a narrow architectural result, with practical benefit at realistic scale still unestablished.

9.1 Related work

  • Prior causal-convolution and lagged-key approaches establish single-layer associative-recall sufficiency, while this paper distinguishes its specific high-pass, DC-neutral construction.The paper frames its contribution as identifying and characterizing a minimal fixed member of this family.Source: §9.1 · pp. 67–68
  • The first-difference augmentation has a partial predecessor in Maitra [52]; this paper adds its filter-order account, Hairer correspondence, uniqueness and placement results, and transition theory.Earlier work shares the operator, some spectral measurements, and part of the empirical corpus.Source: §9.1 · p. 68

9.2 Three neighbouring constructions, compared exactly

  • KV-shifting is a pure delay and fails the class’s strict high-pass condition; the low-pass two-tap reaches 0.428 against 0.811 for the proposed construction.The paper distinguishes the delay’s relabeling mechanism from the current-and-predecessor mixture in its augmentation.Source: §9.2 · pp. 69–70
  • The augmented softmax is not architecturally identical to Differential Transformer [29]: their difference-of-softmax form requires small score changes and has total mass 1 − λ.The paper claims only a first-order filtered tangent correction of a similar difference-of-attention character.Source: §9.2 · p. 70

9.3 Limitations

  • Empirical comparisons use two or three seeds without confidence intervals or multiple-scan correction, and results are overwhelmingly synthetic rather than language-benchmark evaluations.No gain at language-model scale is claimed, and practical benefit at realistic scale remains unestablished.Source: §9.3 · p. 71
  • The obstruction is sidestepped, not contradicted: the augmented score lies outside Sanford et al.’s [16] counted class, while uniqueness applies only within the factorised subclass.On induction alone, the key-only member scores 0.949 against 0.811 for the symmetric member.Source: §9.3 · p. 71

9.4 Discussion and outlook

  • The discussion presents the two-branch law as a zero-fit result within a reduced Gaussian score model, while noting a calibrated condensed branch and approximate transfer to trained models.The broader outlook identifies higher-order score constructions, a phase-space reading of V, and useful nonlinear shears as open questions.Source: §9.4 · p. 72

10 Addendum: the first-order channel in the reduced model

The addendum retains the first-order T3 channel alongside T4 in the reduced Gaussian model and finds materially different transition predictions from the T4-only law. It also outlines the empirical programme and its scope, including synthetic probes and unrun measurements needed to distinguish the reductions.

10 Addendum: the first-order channel in the reduced model

  • For every γ < 1, T3’s score-gap contribution exceeds T4’s; the T4 contribution is larger only on the stated interval 8L ≤ dk < (L + 2)2.Even within that interval, the reduced model omits a real discriminative term.Source: §10 · p. 73
  • For large dk, the extended model predicts γc ≃ L/√dk, versus the T4-only (L2/dk)1/4 scaling; fitted exponents are −0.461 and −0.254, respectively.Trained models give −1.13, so including T3 closes roughly a quarter of the exponent gap but does not resolve it.Source: §10 · p. 74
  • Retaining T3 changes the reduced model’s predictions: the crossover and jump of the T4-only two-branch law disappear, with a monotone width dependence instead.The T3-inclusive prediction is conditional on the Gaussian surrogate and annealed partition function, and is not a statement about a trained transformer.Source: §10 · p. 74
  • The first positive root appears near dk ≈ 31.5, rather than at the T4-only crossover d*_k = 8L ≈ 49.9.A width sweep beginning at dk = 32 therefore lies on a single branch under the extended model.Source: §10 · p. 74
  • The predicted γc values are robust to including T3 only up to about a factor of two, while the T4-only asymptote and two-branch structure are not robust.The extended model’s predicted monotone sweep has not been measured; the cited width sweep stopped at subcritical γ values.Source: §10 · p. 74

Appendices: the empirical programme in full

  • The appendices report measurements and protocols, with 41 Jupyter notebooks and embedded outputs available for auditing results without re-execution.The paper identifies its body as the source of truth when appendix passages differ in emphasis or explanation.Source: Appendices: the empirical programme in… · p. 74

A1 From theorems to measurements: an overview

  • The experiments are predominantly synthetic, spanning small decoders through near-miss studies around 300 M parameters; the task is a designed probe, not a language benchmark.Full-scale trained-head measurements and Bode-plot forensics on Pile/WikiText-103 models are left for future work.Source: A1 From theorems to measurements: an… · pp. 75–76
  • The empirical programme includes structural verification, studies of the design-class operating regime, and six notebooks on single-layer induction.The induction unit reports pre-softmax Bode correlation r up to 0.9994, with scope qualifications where experiments depart from theorem hypotheses.Source: A1 From theorems to measurements: an… · pp. 75–76

A2 Structural verification of the post-RoPE pipeline and four-term decomposition

This section examines post-encoding momentum placement in the post-RoPE pipeline and its effects on associative recall, including whether EMA smoothing suppresses the associated high-pass signal. It reports sharp phase transitions for sinusoidal PE and RoPE without the predicted 10–100× dilution; transition presence is robust across these families, while its location, width, and high-coupling stability depend on the PE.

A2.1 The five-step pipeline

A2.2 RoPE implementation and norm preservation

  • RoPE’s orthogonal block rotations preserve query and key Euclidean norms, while its frequency hierarchy supports both local and long-range positional structure.The cited interpretation links high-frequency blocks to added power driving induction and low-frequency blocks to content matching.Source: A2.2 RoPE implementation and norm… · p. 78

A2.3 EMA momentum and the β parameter

A2.5 Single-query analysis at position 20

  • With β = 0.9 and γ = 0.15, enabling momentum shifts the maximum attention weight from position 20 to position 10, with total absolute distribution change 0.0109.The change reweights attention toward a specific historical position rather than broadly smearing the distribution.Source: A2.5 Single-query analysis at position… · p. 81, Figure 16 · p. 82

A2.4 Attention pattern comparison and four-term decomposition

A3.4 Headline numbers and pre-registered verdicts

  • The pure high-pass configuration exceeds vanilla by nearly 40 percentage points, while the heavily smoothed configuration falls within 0.4 pp of vanilla in mean accuracy.Source: A3.4 Headline numbers and… · p. 85
  • The preregistered correlation prediction was not met: ρ = 0.507 versus the predicted > 0.8, although the direction was correct and significant (p = 3.5×10−11).The authors attribute remaining variation to chain-length interactions; three of four preregistered predictions passed.Source: A3.4 Headline numbers and… · p. 85, Figure 18 · p. 86

A4 The phase-transition exists in both PE families: granular γ-sweep

  • The preregistered 10–100× critical-coupling ratio for RoPE versus sinusoidal encoding was quantitatively incorrect; the theory predicted 1.000 and the measured ratio was 1.22.The section retains the null result for the dilution hypothesis and reports that the measurement supports Theorem 64’s positional-encoding-independent two-branch law.Source: A4 The phase-transition exists in both… · p. 88
  • At γ = 5, RoPE becomes unstable while sinusoidal PE remains near 99%, showing that high-coupling stability differs between the families.The reported sweep gives RoPE 68.4 ± 43.1% versus sinusoidal 98.6 ± 0.2%; one RoPE seed fails outright.Source: A4 The phase-transition exists in both… · p. 88, A4.5 Critical-coupling detection and… · p. 91

A4.1 Scope and the post-encoding momentum placement

  • Momentum is computed as the derivative of the position-encoded state, placing it in the same space as the score computation, as required by the Coriolis non-commutativity argument.The study tests this placement in 156 experiments across 26 γ values, two PE types, and three seeds.Source: A4.1 Scope and the post-encoding… · pp. 88–89

A4.2 Task and configuration

A4.4 Results: the γ-sweep data

  • Both encodings show a sigmoidal jump from near-random to near-perfect accuracy between γ ∈[0.20, 0.40], and both reach approximately 99% by γ ∼1.At γ = 5.0, the reported RoPE accuracy drops to 69.9 ± 40.1% while sinusoidal PE remains at 99.1%.Source: Table 19 · p. 90, A4.4 Results: the γ-sweep data · p. 90

A4.5 Critical-coupling detection and the dilution-test outcome

  • Critical coupling is detected at the midpoint of the steepest accuracy-curve segment; the measured ratio does not validate the predicted dilution magnitude.The extraction uses the finite-difference gradient, and the measured ratio of 1.22 contrasts with the predicted [10, 100] range.Source: A4.5 Critical-coupling detection and… · pp. 90–91

A4.3 The dilution hypothesis (pre-registered)

A4.6 Interpretation and connection to the body

A4 The phase-transition exists in both PE families: granular γ-sweep

A5 Single-layer induction at γ∗≈4: Experiment 16

This section consolidates experiments on single-layer post-RoPE PSA induction, reporting accuracy and phase-jump measurements alongside validation of scaling behavior, operator and spectral diagnostics, and scope conditions. It also records the longer-chain stress test and the broader empirical program framing these results.

A5 Single-layer induction at γ∗≈4: Experiment 16

  • 83.4% accuracy at γ = 4.0 versus 1.2% at γ = 0 (1.56% chance) gives an +82.2-point separation and 53× chance; accuracy exceeds 70% across γ ∈ [1.4, 5.5].The result is reported for a single-layer model on the harder shared-alphabet, padded task with 14 key–value pairs.Source: Table 22 · p. 95, A5.4 Success-criterion assessment · pp. 95–96

A5.4 Success-criterion assessment

  • The phase-transition criterion passes: accuracy rises from 1.2% at γ = 0 to 70.2% across the γ ∈ [0.2, 1.0] transition region, steepest around γ ∈ [0.3, 0.5].The experiment used a shared V = 64 alphabet, 14 key–value pairs, padding, and dmodel = 64; its pre-registered test prioritizes baseline–momentum separation, not the cleaner-task “≫95%” heuristic.Source: A5.1 Aim and pre-registered hypotheses · p. 93, A5.4 Success-criterion assessment · pp. 95–96

A6 The momentum–depth scaling law: Experiment 17

A6.6 Physical interpretation

  • The invariant-product check gives γ∗·N values 4, 6, 5, 6 (mean 5.25), not a constant; the study interprets depth and coupling as fungible resources, with γoptimal ≈ 4.2/N^0.73 in this nano regime.The scaling law is an empirical depth-axis regularity, not a closed-form prediction of the paper’s theorem; the prefactor may shift for deeper or wider models.Source: A6.5 The invariant-product cross-check · p. 98, A6.6 Physical interpretation · p. 99

A7 Granular scaling-law validation at L = 30: Experiment 18

A7.4 Why the fit fails, and why E17 remains canonical

  • The E18 failure reflects training-limited accuracy and noisy argmax detection, not a refutation of E17; the raw-argmax fit could differ if optima were re-located from smoothed curves.At L = 30, the 2,000-step budget was inadequate, and N = 4’s observed optimum shifted from E17’s 1.25 to E18’s 3.00 across an accuracy plateau within SEM.Source: A7.4 Why the fit fails, and why E17… · pp. 101, 103

A8 Validation suite v7: twelve sections, twenty-four checks

  • Validation suite v7 integrates 12 sections into 24 checks, recording 14 PASS, 10 CHECK, and 0 FAIL.It uses L = 30 and 3,000 training steps with three seeds for induction and scaling sections; each CHECK seeds a v8 or v9 follow-up.Source: A8 Validation suite v7: twelve… · p. 103

A8.1 S1: single-layer induction at L = 30

  • 74.2% accuracy at γ = 2.0 versus 1.27% at γ = 0 (+72.9 pp) is the L = 30 sweep peak, below the 75% heuristic threshold.This is one trained model evaluated on three seeds; an independent five-seed run reports 69.1 ± 8.2% at γ = 2.0 versus 1.6 ± 0.4% at γ = 0. The L = 30 peak is lower than the L = 14 peak of 83.4%, and its optimum shifts from γ∗≈4.0 to γ∗≈2.0.Source: Table 29 · p. 104, A8.1 S1: single-layer induction at L =… · p. 104

A8.2 S2: scaling law at L = 30

  • At L = 30, the fit γ∗ = 1.88/N^0.11 (R2 = 0.112) does not recover canonical scaling, and the exponent α ≈ 0.1 is indistinguishable from no scaling.The N = 8 network reaches only 14.1% at its best γ; the product γ∗·N is not constant.Source: Table 30 · p. 105, A8.2 S2: scaling law at L = 30 · p. 105

A8.3 S3: small-signal ratio (FP-18b)

A8.5 S6–7: AC-energy δemp

A8.7 S8b: HSF pre- and post-softmax

  • HSFpost exceeds 1 at all γ > 0, while HSFpre peaks at 6.93 at γ = 1.2 and HSFpre ≥ HSFpost fails at γ = 0.6 and γ = 3.0.At γ = 0.6, the HSF position-slope is +0.056/position, consistent with non-symplectic drift.Source: A8.7 S8b: HSF pre- and post-softmax · p. 108

A8.8 S9: Bode spectral forensics

A8.9 S10: spectral complementarity (frac_AC)

A8.11 S12: Coriolis error (RoPE band decomposition)

A9.2 Fix 2 (genuine implementation defect): Bode pre-softmax A/B provenance

A10.2 C7: T3→T4 crossover

A10.3 C8: HSFpre ≥HSFpost Fréchet boundary

  • The predicted HSFpre ≥HSFpost ordering holds for γ ∈[1.0, 2.4], centered on γ∗≈2.0, but fails at low and high γ.At low γ, softmax nonlinearity magnifies post-softmax contrast; at high γ, PSA over-rotation brings pre-softmax HSF toward 1, and softmax saturation restores a modest post-softmax advantage.Source: A10.3 C8: HSFpre ≥HSFpost Fréchet… · p. 118, Figure 39 · p. 119

A10.4 C9: FP-4 / FP-13 on stationary sequences

A10.9 C14: band-4 Coriolis anomaly characterization

  • The v9 diagnostics also found larger-magnitude H0 slopes for non-symplectic control, an unexplained persistent band-4 anomaly, and a confirmed post-softmax Bode-r advantage for symmetric PSA.At γ = 3.0, band-4 alignment is 0.9541 yet the 41× excess persists; at γ = 1.2 and 2.0, its alignment falls to 0.51–0.61. PSA’s Bode-r advantage over NS ranges from +0.92 at γ = 0.6 to +0.005 at γ = 3.0.Source: Figure 41 · p. 121, A10.9 C14: band-4 Coriolis anomaly… · p. 124, Figure 45 · p. 125

A10.6 C11: T4 ablation — T4 is a major induction carrier in the clean operating regime

  • At γ ∈[1.0, 1.4], T4 ablation reduces accuracy by 37–39 pp, while the remaining T2 and T3 channels still reach 19.7–49.9% versus 1.7% for standard attention.At γ = 3.0, the ablated model benefits from avoiding the full operator’s over-rotation; T4 is consequential in the normal regime, but is not the only induction signal.Source: A10.6 C11: T4 ablation — T4 is a major… · p. 121, Figure 42 · p. 122

A10.7 C12: γc scaling with dk

  • Observed γc falls from infeasible at dk = 8 to 1.0 at dk = 16 and 0.4 at dk = 32, while peak accuracy rises from 9% to 75% to 93%.All tested widths are at or below the condensed-branch crossover, so this sweep establishes the direction of change, not Theorem 64’s dk direction; its γc values use an absolute 50% threshold and are not directly comparable across rows.Source: A10.7 C12: γc scaling with dk · pp. 122–123, Figure 43 · p. 123

A10.8 C13: FP-24 single-layer PSA vs standard attention

A10.11 C16: v9 consolidated scorecard

  • The v9 scorecard confirms FP-24, FP-29, and FP-16, locates the T3→T4 crossover at γ ≈0.97, and marks the condensed-branch γc(dk) sweep out of scope.It also characterizes the band-4 Coriolis result and confirms FP-5’s drift direction with magnitude caveats.Source: A10.11 C16: v9 consolidated scorecard · p. 126

A11.1 Consolidated Tranche 1 result

  • Tranche 1 supports the four-term decomposition, necessary post-RoPE placement, the high-pass condition, and a phase transition under both RoPE and sinusoidal PE.The high-pass tests report β = 0 as optimal; the transition’s existence is robust to PE family, while location and width are not.Source: A11.1 Consolidated Tranche 1 result · p. 127
  • At γ∗=2.0, single-layer PSA reaches 69.1 ± 8.2% on associative recall, versus 1.6% chance for single- and two-layer standard attention—a 43× separation.The passage identifies this as the headline FP-24 result supporting Theorem 49.Source: A11.1 Consolidated Tranche 1 result · p. 127
  • The depth-scaling law holds at L = 14 but breaks down at L = 30 under a training-budget limitation, while the trained operator matches the spectral transfer function with pre-softmax Bode correlation r up to 0.9994.The passage reports the scaling law as γ∗(N) = 4.17/N^0.73 with R2 = 0.947.Source: A11.1 Consolidated Tranche 1 result · p. 127
  • Zeroing T4’s discriminative carrier reduces accuracy by 37–39 pp in the clean regime; the matched battery scores key-only 0.949, asymmetric 0.875, and symmetric 0.811.Theorem 27 selects symmetric shear on structural grounds, not accuracy; the energy bound and zero-sum identity hold to numerical precision 10^-7, while Coriolis ordering holds directionally with a band-4 anomaly.Source: A11.1 Consolidated Tranche 1 result · p. 127

A11.2 Scope qualifications and remaining CHECKs

  • The record is scope-qualified: H0- and δ-flatness apply to stationary inputs, Fréchet ordering to γ ∈[1.0, 2.4], and depth scaling is not cleanly measured at L = 30.The band-4 Coriolis error is 41× theory at θ = 0.01 rad/token and remains unexplained; the observed γc decrease with dk has the mean-field direction, but the tested configurations lie on the condensed side, where the law predicts dk-independence.Source: A11.2 Scope qualifications and… · p. 128

A11.4 Bridge to Tranche 2

  • Tranche 2 extends the record across spectral studies, algorithmic-reasoning primitives, in-context learning and placement, and symplectic stability and parameter efficiency.The described work includes a 2,000-run Low-Pass Induction Filter validation, reasoning-task gains including +60.6 pp on hard arithmetic, a −52.5% repeated-token-loss regression at chain length 30, and two parameter-efficiency near-misses.Source: A11.4 Bridge to Tranche 2 · pp. 128–129

A12 Tranche 2 overview: from microbenchmarks to algorithmic-reasoning primitives at small-language-model scale

Tranche 2 extends the empirical program from controlled associative recall to RoPE spectra, reasoning primitives, longer-chain in-context learning, and parameter-efficiency tests. It reports a principal repeated-token-loss improvement alongside spectral, reasoning, and placement findings, while documenting nulls and scope limits.

Tranche scope and materials

  • Tranche 2 examines four empirical axes, and its notebooks include embedded outputs so reported numbers can be audited without GPU re-execution.The expanded corpus is reported at roughly 9,377 runs after the three induction sweeps.Source: A12 Tranche 2 overview: from… · pp. 129–130

Spectral and reasoning results

  • A 2,000-experiment sweep finds a RoPE momentum noise–gain correlation of r = −0.679, while the reasoning arc reports a ∇/R dissociation test with t = −2.910 and p = 4.51×10−3.The reasoning test is reported on Majority versus the non-control tasks.Source: A12 Tranche 2 overview: from… · p. 129

In-context learning

  • Repeated-token loss falls 52.5% as chain length increases from L=10 to L=30, the program’s principal empirical result.The tranche also reports approximately 4,700 runs, bringing the corpus to roughly 9,300 before three additional induction sweeps.Source: A12 Tranche 2 overview: from… · p. 129

Scope and null results

  • The tranche is not evidence of universal benefit: documented nulls include multi-hop with negations, borderline-significance parity, and parameter-efficiency near-misses.The paper frames failures such as multi-hop with negations and mixed-task fluency at 300M parameters as bounds on the mechanism, not disproofs.Source: A12 Tranche 2 overview: from… · p. 129

Architectures and protocols

  • The spectral and reasoning studies use 1–3-layer compact transformers, the ICL arc uses 4.45M-parameter models, and conditioning uses a 91.7M-parameter decoder.The protocols also include approximately 300M-parameter models for parameter-efficiency tests; momentum is applied symmetrically to Q and K post-RoPE except in the placement contrast.Source: A12 Tranche 2 overview: from… · p. 130

Placement contrast

  • Embedding-level momentum regresses 4.1%, whereas Q/K-post-RoPE momentum improves repeated-token loss by 52.5%; the two runs are unmatched, so direction and spectral signatures—not magnitude—carry the comparison.The post-RoPE placement has theory–experiment correlation r = 0.9865 on the attention-score spectrum.Source: A12 Tranche 2 overview: from… · pp. 129–130

A13 The bandpass spectral window: exploratory low-frequency scan (F1)

The exploratory bandpass RoPE scan tests whether low-frequency induction survives spectral diversification and finds its strongest accuracy at low center frequency with moderate momentum. The scan is single-seed, and its high-coupling collapse and inverted-U pattern are reported as exploratory observations.

A13.1 Question and design

  • The scan asks whether low-θ starvation persists after spectral diversification, using a narrow frequency band [θ(1−b), θ(1+b)] with b = 0.2.The design addresses near-degenerate position rotations at very low θ by introducing spectral diversity while focusing on a frequency regime.Source: A13.1 Question and design · p. 130

A13.2 Configuration

A13.3 The 48-point accuracy surface

A13.4 Structural observations

  • Every tested θ shows an inverted-U accuracy pattern over γ, with row maxima around γ∗ ∈ [0.5, 1.0].At γ = 3.0, all rows fall to 0.54–0.57; the figure identifies the brightest cell as (0.05, 0.75) = 0.95.Source: A13.4 Structural observations · p. 131, Figure 48 · p. 132
  • At θ = 0.05, the peak reaches 0.95 from a baseline of 0.68, a gain of +27 pp; higher-frequency peaks are shallower.At θ = 1.0 and θ = 2.0, peaks are approximately 0.80–0.82 against baselines 0.71 and 0.75, respectively.Source: A13.4 Structural observations · p. 131

A14 The rigorous granular sweep: 720 runs, 240 configurations (F2)

This section presents F2’s granular phase diagram alongside F3’s monochromatic control, G’s statistical validation of the low-pass filter, and H’s comparison of RoPE frequency designs, examining frequency-dependent induction gains and the limits of the tested protocols and theory. In the tested associative-recall configuration, multi-frequency RoPE provides the most robust PSA range through a level shift rather than a change in curve shape; this recommendation is configuration-limited, with the proposed explanation that distinct frequency bands support content matching and induction.

A14 The rigorous granular sweep: 720 runs, 240 configurations (F2)

A15 The monochromatic control: ruling out frequency-hopping (F3)

A16.4 Headline statistical validation

A16.5 Observed versus predicted optimal γ

  • For θ < 0.2, empirical γ∗ was uniformly 0.80, while the theoretical γ∗ = 1.5 overshot it by roughly 2×; the theory captured direction but not location.The observed optimum ranged over {0.7, 0.8, 0.9} at mid frequencies and {0.7, 0.8, 1.0} at high frequencies; at θ = π, γ∗ = 0.60. The proposed refinement is to account for softmax temperature and the training-time loss landscape.Source: A16.5 Observed versus predicted… · p. 141

A17 The RoPE frequency design space (H)

  • At γ = 1.0, multi-frequency RoPE peaked at 96.2 ± 0.2%, exceeding bandpass (87.5 ± 0.3%) and single-frequency RoPE (86.8 ± 0.2%).Its peak advantage was +8.7% over bandpass and +9.4% over single-frequency. All three γ = 0 baselines were within ∼1% of chance.Source: A17.4 Complete γ-sweep results · p. 145

A17.5 Why all three shapes are inverted-U here

  • All three curves are inverted-U with peaks at γ = 1.0, while multi-frequency RoPE is ∼9% higher at every γ > 0.The advantage is a level shift, not a shape change.Source: Figure 57 · p. 146
  • For the tested associative-recall tasks, multi-frequency RoPE with γ ∈ [0.7, 1.5] provides maximum robustness; single-frequency RoPE is viable only when calibrated near γ = 1.0.This is not a universal recommendation; deeper-model and natural-language scaling is addressed in later sections.Source: A17.5 Why all three shapes are… · p. 146
  • Multi-frequency RoPE’s advantage is attributed to its wide frequency range, which lets post-RoPE PSA assign distinct content-matching and induction-driving roles to different bands.The added power is quadratic in γ, above DC, and weighted toward high frequency; single-frequency RoPE collapses onto one carrier and has the lowest peak.Source: A17.5 Why all three shapes are… · p. 147

A18 Mechanistic visualization: what PSA does inside the network

These sections characterize PSA’s task- and setting-dependent effects across reasoning tasks, reporting the largest gain on Natural Induction under low-θ, high-γ conditions alongside weaker, absent, or disruptive outcomes on other tasks.

A18.3 Paired training results at θ = 0.03

  • At θ = 0.03, PSA produced a +36.8 pp Associative-Recall gain and a −32.2 pp Global-Counting loss; Variable Tracking dropped −4.6 pp at this setting.The text characterizes the Variable Tracking drop as a local minimum and the counting loss as suppression of a constant-over-position signal.Source: Table 66 · p. 147, A18.3 Paired training results at θ =… · p. 148

A18.4 The 36-cell θ×γ heatmap

A18.5 Cross-task accuracy curves

  • With θ = 0.03 and γ from 0.5 to 0.8, the symmetric shear visibly concentrated Associative-Recall attention on the target while disrupting Global-Counting attention.Figure 58 also shows stronger diagonal structure for Variable Tracking; the section presents these patterns as consistent with the Low-Pass Induction Filter.Source: Figure 58 · p. 149, A18.5 Cross-task accuracy curves · p. 150

A19.4 Verdict

  • Every eased Multi-Hop with Negations configuration stayed within +0.7 pp of its γ = 0 baseline, despite easy and medium baselines of 65% and 59%.The authors suggest path identification among distractors dominates parity tracking, qualifying the hypothesis that sequential-dependency tasks benefit from PSA.Source: Figure 60 · p. 151, A19.4 Verdict · p. 152

A20 Chain-of-thought reasoning on harder tasks

A20.7 Cross-task summary

  • Across 360 runs, Multi-Hop with Negations showed no gains, Global Counting showed no response to γ, and the low-θ advantage appeared on tasks where momentum helped.Variable Tracking and Arithmetic CoT showed gains; the section attributes their useful signal to sequential state updates, unlike order-invariant counting.Source: A20.7 Cross-task summary · p. 155

A21 Real-world reasoning: five primitives, 600 runs

  • Natural Induction rose from 12.7% to 90.6% accuracy at low θ and high γ; Arithmetic Carry gained +8.2 pp, Parity was borderline, and List Reversal and Sorting were ceiling-saturated.These results come from a 600-run sweep across five reasoning primitives.Source: A21 Real-world reasoning: five… · p. 155

A21.2 Main results across tasks

  • Natural Induction gained +0.74, Arithmetic Carry gained +8.2 pp at matched θ, List Reversal and Sorting were ceiling-saturated, and Parity remained near chance.The five-task, 600-run sweep found the three moving-task gains shared a low-pass-filter signature.Source: Figure 61 · p. 156, A21.5 The three non-movers · p. 158

A21.3 Natural Induction: the empirical anchor

A21.4 Arithmetic Carry across the full grid

  • At θ = 0.03, Arithmetic Carry improved by +8.2 pp (88.8 → 97.0), while its θ = 1.0 row was flat-to-decaying.The best baseline across all four θ values was 97.4% at θ = 1.0, so the naive peak-gain comparison is −0.001; the fixed-θ comparison captures the improvement.Source: A21.4 Arithmetic Carry across the full… · p. 157

A21.5 The three non-movers

  • List Reversal stayed at 99.9% across γ, Sorting (Min) declined from 99.7% by −1.0 pp at γ = 1.2, and Parity stayed near 50%.Parity’s +2.9 pp gain at γ = 0.5 was statistically borderline.Source: A21.5 The three non-movers · p. 157

A22 Multi-task validation with negative control: the ∇/

Validation tests the shear across multiple tasks and conditions, finding benefits on some tasks but no gain or regressions on others. It also examines how predecessor-context mismatch and a one-token anchor affect chained-ICL results.

A22.5 The negative-control t-test

A22.2 Main result

  • On Majority, the small model already achieved ceiling accuracy through attention, so the shear acted as an additive perturbation overwritten by the attention vote.The authors characterize this as evidence that the shear is at worst neutral on tasks well within capacity.Source: A22.2 Main result · p. 159

A22.3 Induction: the headline

  • Induction showed an inverted-U at every θ, with peak accuracy falling from 0.846 at θ = 0.10 to 0.454 at θ = 1.00.At θ = 0.03, accuracy dropped from 0.787 to 0.717 as γ increased from 1.2 to 1.8, consistent with over-coupling.Source: A22.3 Induction: the headline · p. 160

A22.4 Trajectory and Dyck

  • Trajectory gains were +0.024, +0.021, +0.029, and +0.040 across increasing θ, while Dyck’s peak gain was +0.016 at γ = 0.7, θ = 1.0.Trajectory peaked at θ = 1.00; Dyck’s smaller gain was attributed to its 0.93 baseline and limited headroom.Source: A22.4 Trajectory and Dyck · p. 160

A23.3 Per-(ℓ, V ) best-case summary

A23.4 Phase-transition observation and binned analysis

A23.6 Three predictions validated

  • The sweep found three significant patterns: noise–gain r = −0.372 (p = 2.88×10−5), larger gains for intermediate versus very-hard tasks (F = 12.04, p = 7.38×10−4), and optimal γ in [0.7, 1.2].The noise–gain association tests the low-pass prediction; the difficulty comparison and stable optimal-γ range were separately reported as validations.Source: A23.6 Three predictions validated · p. 169

A23.7 Reasoning-arc cross-task verdict

  • The cross-task verdict records null cases including Multi-Hop with Negations, borderline Parity, and ceiling or order-invariant Sorting, List Reversal, and Majority.The paper states that the PSA program does not claim universal benefit and does not hide failures.Source: A23.7 Reasoning-arc cross-task verdict · p. 170

A25.2 Results

  • On bursty Zipfian sampling, Lrep changes by +5.3% and Δ1−2 by −0.10 nats, both within run-to-run noise, while training curves remain statistically indistinguishable.This predicted null reflects DC-band neutrality: |H(0)| = 1, and bursty active-set membership is treated as a low-frequency counting task.Source: A25 Burstiness null: PSA is neutral… · p. 171, A25.2 Results · p. 173

A26.2 Diagnosis (provisional)

  • On structural ICL without anchoring, PSA at γ=0.7 produces a +8.7% regression in Lrep because differing preceding tokens give lesson and query occurrences of A different momentum coordinates.The augmented mechanism therefore fails to recognize them as the same instance, so momentum harms recognition rather than helping it.Source: A26.2 Diagnosis (provisional) · p. 175

A27 Chained ICL without anchoring: diagnosis hardened

A28 Anchored ICL: the fix

  • A one-token anchor changes chained-ICL Lrep from a +28.9% regression to a −4.1% improvement, and all three hypotheses pass.The intervention prepends a reserved anchor token to every chain definition and query chunk, ensuring kinematic consistency.Source: Table 98 · p. 178, A28 Anchored ICL: the fix · p. 178

A28.2 The tri-gram filter interpretation

  • Anchored PSA keys on (α, A) pairs rather than A alone, making it a tri-gram filter and a strict refinement of standard attention against distractor matches.With the anchor, the phase-space coordinate for A is determined by A’s identity rather than by the random token preceding it.Source: A28.2 The tri-gram filter interpretation · pp. 179–180

A29 Anchored ICL stress test at L = 30: principal empirical result of the ICL arc

At L = 30, the momentum model substantially reduces repeated-token loss and wins at nearly every chain depth. The placement comparison favors post-RoPE shear and shows a matching spectral signature, but its effect size is confounded by differences between runs.

A29.2 Aggregate results

A29.3 Chain-depth breakdown: momentum wins everywhere

  • Momentum wins at depths 1–19, while the baseline wins only at depth 0 by 0.03 nats.The average per-depth improvement is 0.406 nats, with a 0.133-nat improvement at the deepest depth.Source: Table 103 · p. 181

A29.4 Interpretation

  • Unchanged Lnew = 7.02 versus the baseline’s 6.99 localizes the improvement to in-context adaptation rather than the novelty prior.The reported interpretation is that PSA selectively amplifies induction, not general-purpose training performance.Source: A29.4 Interpretation · pp. 182–183

A30.1 The two placements

  • Embedding-level shear precedes RoPE, whereas the Q/K shear follows RoPE inside each attention head.The two placements therefore apply the shear at different points in the network.Source: A30.1 The two placements · p. 183

A30.2 Notebook A (embedding-level placement): regression and failed spectral signature

  • The Notebook A embedding-level run regresses repeated-token loss and fails to produce the predicted high-pass response.Its DC/Nyquist ratio rises from 2.60 for baseline to 3.015 for momentum, the opposite of the expected direction.Source: Table 104 · p. 184, Table 105 · p. 184, Figure 79 · p. 184

A30.4 Why these two runs are not a matched pair

A30.5 The contrast and what it confirms

A31 Sweeps at 91M parameters: contraction at low γ, and the induction emergence band

These 91M-parameter sweeps examine how coupling and training conditions relate to contraction, induction emergence, and depth-dependent computation, alongside task-specific outcomes and diagnostics. Their findings are bounded by the tested settings, and the audit highlights null results and the need for controls in circuit-attribution analyses.

A31.5 The optimum at γ = 10−2

A31.3 Main numerical results

A31.6 Two induction sweeps at the same scale: the width sweep and the supercritical sweep

A31.7 Result: an emergence band, not a threshold

  • Reliable emergence began at γ = 0.40, near γT3T4_c = 0.4132, while fastest emergence at γ = 0.80 was near γT4_c = 0.7867; neither law predicts both features.The observed band at the constant learning rate is roughly [0.40, 0.90], but its upper edge is a step-size effect rather than an intrinsic operator cutoff.Source: A31.7 Result: an emergence band, not a… · p. 194
  • At γ = 0.80, all three seeds emerged in a mean of 717 steps, versus 2/3 seeds at γ = 0 (2900 and 2500 steps), a 3.8× faster and more reliable result.The γ = 0.80 runs took 600, 650, and 900 steps; the reported comparison uses a matched protocol and identical parameter count.Source: Table 112 · p. 194, A31.7 Result: an emergence band, not a… · p. 194, Figure 84 · p. 195

A31.8 Two failure modes, separated by the energy ratio

A31.9 An out-of-sample test of the Momentum-Depth Scaling Law

  • At N = 12, the observed optimum implies α = 0.664, within 18% of the independently fitted depth-law exponent 0.73; linear accumulation predicts γ = 0.347 instead of 0.80.This is the paper’s only out-of-sample confirmation of the depth law, based on one depth, one width, and three seeds, so it does not establish the law.Source: A31.9 An out-of-sample test of the… · p. 196

A31.11 The cutoff decided: it is the optimiser, not the operator

A32.1 R1: David versus Goliath, base run

  • In R1, fluency loss improved only 0.4% over 10,000 steps across the three models, and all pre-registered hypotheses failed with margins below 0.5%.The authors diagnose an entropy plateau: with vocabulary 50,257 and batch 8, token recurrence—and thus induction-head gradient signal—was rare.Source: Table 116 · p. 200, A32.1 R1: David versus Goliath, base run · p. 201

A32.2 R2: David versus Goliath, SNR-optimized re-run

  • In R2, fluency loss fell from 9.03 to 7.77–8.15, but H1 and H3 still failed by 3–5%, and H2 passed narrowly because all models tied at chance on logic.Across both notebooks, parameter efficiency was not demonstrated at about 300M parameters, batch ≤64, and 10,000 steps; the reported diagnoses point to training-budget, vocabulary, and mixed-task limits rather than a falsification of the operator theorem.Source: Table 119 · p. 201, Table 120 · p. 201, A32.3 Joint interpretation of R1 and R2 · p. 202

A33.1 The task-by-task verdict matrix

  • Of 13 derivative-class tasks, 9 showed positive gains consistent with the low-pass signature, 2 were clean nulls, and 2 were limited by ceiling effects or sample noise; all 4 order-invariant tasks showed no gain or mild harm.Converging evidence included Majority’s zero gain across 28 cells and gains of +59.1 pp, +4.0 pp, and +1.6 pp on Induction, Trajectory, and Dyck; the negative-control test was t = −2.910, p = 4.51×10−3.Source: A33.1 The task-by-task verdict matrix · p. 203, A33.2 Three converging lines of evidence · p. 203

A33.3 The four sub-program summaries

  • The four sub-program summaries report spectral-filter evidence, reasoning gains, an anchored ICL result of −52.5%, and a placement contrast of +4.1% for embedding placement versus −52.5% for Q/K-post-RoPE placement.The null catalogue qualifies these findings: very-hard-bin mean gain fell from +0.243 to +0.167, embedding-level momentum regressed, and both R-series parameter-efficiency runs failed their hypotheses.Source: A33.3 The four sub-program summaries · p. 204, A33.4 Tranche 2 catalogue of nulls · p. 205

A34.1 Task

  • On the randomized-motif task, the best fixed-lag oracle scored 0.065 versus chance 0.0078, while the generator-period oracle scored 1.000; the period was randomized because a fixed-period repeat is solvable by relative position alone.The held-out ridge probe’s in-sample accuracy averaged +0.024 over the grid; at γ = 0 it inflated 0.0086 to 0.0398, against chance 0.0078.Source: A34.1 Task · p. 205, A34.2 Diagnostics · p. 206

A34.2 Diagnostics

  • Per-layer γ ablations distinguish first-layer shortcuts from computation distributed across sheared layers, while previous-token and induction-offset head censuses are non-load-bearing diagnostics.With period randomised, those head censuses are noisier than the probe and ablation.Source: A34.2 Diagnostics · p. 206

A34.3 Results

  • The held-out first-layer probe rises from 0.0086 at γ = 0, near 0.0078 chance, to an average 0.760 for γ ≥0.5, with maximum per-depth values of 0.828–0.854.The channel does not close with depth.Source: A34.3 Results · p. 206
  • Up to N = 8, first-block ablation matches global ablation within 0.011 and last-block ablation costs nothing, while at N ≥12 and γ ≥1.75 the ablations separate, indicating distributed computation.For example, at N=16, γ=2.5, first-block ablation yields 0.656 against 0.921 for global ablation; the distributed corner remains uncharacterised.Source: A34.3 Results · p. 206
  • γ1/2 ranges from 0.154 to 0.400 across the depth grid, with a 0.15–0.23 regime for N ≤6 and a 0.40 regime for N ≥7; the sweep does not resolve the crossing precisely.The N ≥7 values mean the transition falls between γ = 0.25 and 0.5, not that a 0.40 plateau was measured. The registered crossing prediction at N ≈8.3 fails: no crossing occurs on-grid, and the fitted curve reaches 0.400 only near N ≈25.Source: A34.3 Results · p. 207
  • At dk = 16, the scored-mask critical coupling is 0.832–0.924, while measured γ1/2 is 0.17–0.45γc; this compares distinct observables, not a test of Theorem 64.The scored band spans Teff ∈[25, 127]; the above/below grid partition is unchanged using the analytic midpoint 0.899 instead of the empirical band midpoint.Source: A34.3 Results · p. 207
  • At γ = 0, final accuracy is 0.991–0.995 across depths in 500–833 steps; for γ ≥0.5, accuracy is 0.88–0.94 and the 90% threshold is frequently missed within 3,000 steps.Time-to-threshold is quantised in 250-step increments, and non-crossing runs receive a ranking sentinel, so these magnitudes are not estimates of optimization speed.Source: A34.3 Results · p. 207

A34.4 Descriptive statistics

  • Across 192 cells, held-out first-layer probe scores correlate with γ/γc at r = +0.425, and mean decodability above γc is 1.42 times that below (s.d. 0.03).The within-depth ratios range from 1.25 to 1.67; no p-value is attached because grid cells share a data generator and the three seeds are the replicates.Source: A34.4 Descriptive statistics · p. 207

A34.5 Limits

  • The sweep leaves γ1/2 unresolved and does not characterise the distributed corner; its replacement-sampled motifs can also make content matches ambiguous, and the task is one classical composition solves at N = 2.The limits passage identifies finer γ sampling, sampling motifs without replacement, activation patching across layers, and testing tasks composition cannot meet under a per-layer budget as follow-ups; its null on optimality is specific to this task.Source: A34.5 Limits · p. 208

A35 Final empirical conclusion: theory–measurement closure across both tranches

Across two tranches, the section assesses the post-RoPE shear through measurements and circuit attribution, reporting support for its predicted structure and task effects and evidence that trained models adopt its single-layer channel. It presents CPSA as a structurally selected, symplectically consistent construction, notes that the key-side half-lift leads the matched accuracy battery, and records qualified claims, limits, and open questions for future validation.

A35 Final empirical conclusion: theory–measurement closure across both tranches

  • At every tested depth once γ ≥ 0.5, next-token prediction was linearly decodable after one attention layer, and ablating the first-block shear cost nearly as much as ablating it everywhere.The circuit-attribution study is separate from both tranches and tests which computation a trained model adopts, rather than only whether the operator makes it expressible.Source: A35 Final empirical conclusion:… · p. 208

A35.1 The corpus, in one paragraph

  • At dk = 128 and N = 12, γ = 0.80 produced induction in 3/3 seeds in 717 steps, versus 2/3 seeds and 2700 steps at γ = 0, a 3.8× faster result.The matched-parameter sweep’s apparent cutoff near γ ≈ 0.9 was rescued at γ = 1.00 in 2/3 seeds by scaling the learning rate by (1 + 2γ)^−2; warmup did not rescue it.Source: A35.1 The corpus, in one paragraph · p. 209

A35.2 Verdict matrix on the claims of the body

  • Of 14 empirically examined body claims, 11 receive a clean PASS and three a scope-qualified PASS; none receives a FAIL.The verdict matrix covers claims tested across both tranches, not the correspondence with induction-head literature.Source: A35.2 Verdict matrix on the claims of… · p. 209

A35.3 Theory and engineering, jointly

  • Tranche 1 verifies the predicted structural signatures, while Tranche 2 reports peak gains of +74.4 pp on Natural Induction, +60.6 pp on hard arithmetic chain-of-thought, and +59.1 pp on induction with negative control.Tranche 2 also reports a −52.5% repeated-token-loss regression; the ∼300M-parameter aggregate-loss study did not demonstrate a parameter-efficiency advantage.Source: A35.3 Theory and engineering, jointly · p. 210

A35.4 Correspondences with prior empirical work, and their limits

  • The prior-work links are bounded: PSA gives a shape correspondence, not a training-time explanation, for Olsson–Olah; a conditional first-order tangent relation to Differential Transformer; a structural account of FDAM’s frequency vanishing; and a separation from KV-shifting.The Olsson–Olah comparison does not derive the loss drop, simultaneous head onset, or training-loss bump; the Differential Transformer relation is neither architectural identity nor a γ → 0 limit, and no numerical match to FDAM is claimed.Source: A35.4 Correspondences with prior… · pp. 210–211

A35.5 The Placement Corollary, tested directionally

  • Moving the symmetric shear from embedding-level to Q/K-post-RoPE placement changed the outcome from a +4.1% regression to a −52.5% improvement, with r = 0.9865 spectral correlation.The runs had unmatched γ = 0 baselines, so only direction and spectral signatures support the placement contrast, not its magnitude as a controlled comparison.Source: A35.5 The Placement Corollary, tested… · p. 212

A36.1 Open questions on the operator

  • Open operator questions include the 41× Coriolis anomaly, whether an upper edge exists beyond the rescued γ = 1.00, and the unresolved threshold law and branch assignment.The three narrower widths were not tested supercritically, and the energy-ratio growth during training lacks a rigorous parameter-space characterization.Source: A36.1 Open questions on the operator · p. 213

A36.2 Open questions on tasks

  • The empirical regime remains limited to synthetic tasks, compact models for spectral and reasoning studies, single-task ICL, and AdamW; multi-hop negation and low-frequency task anomalies also remain unexplained.The appendix defers a Differential Transformer benchmark reproduction and a GPT-2-scale Hairer Survival Factor test; optimizer dependence and natural-language-corpus performance remain untested.Source: A36.3 Limitations of the present… · p. 214, A36.2 Open questions on tasks · p. 214

A36.5 Closing

  • 0.949 for the key-side half-lift exceeds 0.811 for the symmetric operator in the matched battery.The passage identifies the key-side half-lift as both the minimal member and the best performer.Source: A36.5 Closing · p. 215
  • CPSA is the unique construction satisfying Theorem 27’s structural conditions, selected on transport and score-bias grounds rather than accuracy.The construction is also described as symplectically consistent in the Hairer sense.Source: A36.5 Closing · p. 215
  • Controlled negative-control testing shows CPSA empirically discriminates between ∇- and R-tasks.Source: A36.5 Closing · p. 215
  • Corpus-scale and full-precision benchmark-scale evaluation, and testing beyond the seventeen specified tasks, are left to subsequent work.The present program is framed as a technical and empirical baseline for that work.Source: A36.5 Closing · p. 215

Related papers

Loading 2609.32319v1…