PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scale

Jun-qiang Lu

arXiv:2609.34292v1cond-mat.dis-nncond-mat.mtrl-scicond-mat.othercs.LG

TL;DR

The quantitative limit and physical basis of pretrained-model redundancy remain unclear, while post-hoc physics analogies risk selection bias. The paper tests five physics-inspired compression mappings under preregistered cluster predictions and a uniform decision rule. The results invert the attention prediction at small-LM scale and find no tensor-network compression gain over plain SVD across the reported rank-face range.

  • Problem

    The quantitative limit and physical or information-theoretic mechanism behind pretrained-model redundancy remain unknown, and post-hoc physics analogies risk selection bias.

    Source: §1 · p. 2

  • Method

    The authors preregistered five physics-inspired compression mappings as a cluster and applied a uniform 3σ decision rule to pilots.

    Source: §1 · p. 3

  • Results

    The attention prediction was inverted, while tensor-network reshaping gave no compression over plain SVD from 0.1 to 8B.

    Source: §9 · p. 26

  • Takeaways & Limitations

    The authors present preregistration, an append-only catalogue, and cluster-level framing as a way to make negative results informative and cluster-level inversions more informative still.

    Source: §9 · p. 27

    The attention-regime inversion was tested only on Pythia-160M and GPT-2-medium; larger models remain untested on this face.

    Source: §8 · p. 26

Abstract

from arXiv · show

We report a three-month autonomous research-agent program testing five solid-state-physics-inspired compression mappings on pretrained language models, with predictions committed to git before any pilot data and a 3-sigma gate deciding PASS or SHELVE. The common anchor -- area-law / Kohn-nearsighted decay of the one-particle density matrix -- has a distance face (P001 Wannier, P002 tight-binding) and a rank face (P003 DMRG-truncated MLPs, P005 Wilson-RG, P011 tensor-train embeddings). P005 was pre-empted at Phase 1; three of four Phase-3 pilots were falsified. On the attention face, GPT-2-medium attention-versus-distance is best fit by a stretched exponential in 12 of 16 median-layer heads once probe padding is excluded, and a tight-binding cutoff costs +96% perplexity (P002); on Pythia-160M the Wannier sparsity 0.054 +/- 0.004 is indistinguishable from PCA, random-Haar and identity baselines (P001). On the rank face, per-token tensor-train bond dimension does not track surprisal (r = 0.016 vs a pre-registered 0.65) and the format inflates rather than compresses (P011). P003 is mixed: its scaling claim shelved (r = -0.434), its MPO premise died at stage-0, and its cross-paper check, r = 0.523 as first written, collapses to 0.047 under the same correction, leaving both cross-paper checks null. The results invert the pre-registered prediction that most attention heads behave like Kohn-nearsighted insulators, pointing instead to critical, glassy or heavy-tailed regimes; the inversion is specific to the <= 350M scale tested, while the rank-face no-gain result held to 7-8B. We contribute the pre-registration + 3-sigma + cluster-framing + append-only-catalogue discipline -- including why our own enforcement gate was designed but not deployed -- four pre-registered negative results with full data release, and the inversion. The catalogue holds eighteen concluded studies, seventeen negative.

1 Introduction

The introduction motivates physics-inspired compression tests by unresolved redundancy and selection bias, then presents a preregistered cluster protocol and its negative results. It frames the contribution as methodological discipline, a small-model attention inversion, and a complete record of concluded studies.

1 Introduction

  • Existing compression methods demonstrate model redundancy, but its quantitative limit and physical or information-theoretic mechanism remain unknown.The paper also identifies selection bias and post-hoc rationalisation as risks when physics analogies are assigned after observing results.Source: §1 · p. 2
  • The program preregistered five candidate mappings as a cluster, committed predictions before pilots, and applied a uniform 3σ PASS-or-SHELVE rule.Failures entered an append-only catalogue with lessons rather than being reframed after results.Source: §1 · pp. 2–3
  • The tested small-LM scope is limited to Pythia-160M and GPT-2-medium; ALiBi, SwiGLU, and explicitly sparse-attention models were not tested.Larger follow-on studies do not replicate any of the five mappings, so the stated cluster prediction remains for the two named checkpoints.Source: §1 · pp. 2–3
  • Three Phase-3 pilots were falsified, while P003 shelved its scaling claim but passed its cross-paper consistency check; the attention prediction inverted at ≤350M, and rank-face low-rank expectations failed up to 7–8B.P005 was pre-empted at Phase 1; the rank-face claim concerns weight and embedding matrices in tensor-network formats.Source: §1 · p. 3
  • The paper presents preregistration, a 3σ gate, an append-only catalogue, and cluster framing as a methodology for distinguishing measurement from post-hoc resemblance.The authors state that their criteria were fixed before data existed, while noting the tests do not settle the physical regime of small pretrained transformers.Source: §1 · p. 3
  • The catalogue records eighteen concluded studies, seventeen negative, broadening the record beyond the five mappings reported in full.The authors argue that reporting only successes in sections and failures in footnotes would reproduce the selection effect under study.Source: §1 · p. 3

2 Method: a pre-registered cluster test

The method groups five physics-inspired compression mappings around a shared area-law intuition, then tests their predictions through pre-registration, a uniform 3σ gate, and an append-only catalogue. Its cluster framing predicts insulator-like attention heads will be the majority, while treating rank locality as a heuristic extension rather than a consequence of Kohn nearsightedness.

2.1 The five-paper cluster

  • Five mappings connect LLM redundancy to physical constructions: Wannier and tight-binding attention, DMRG-truncated MLPs, Wilson-RG weights, and tensor-train embeddings.The mappings target basis sparsity, distance decay, activation-informed low rank, weight-spectrum flows, and adaptive embedding rank, respectively.Source: §2.1 · pp. 3–4
  • Four probe-only studies reached Phase 3; Wilson-RG stopped at Phase 1, while pilots used Pythia-160M and GPT-2-medium with WikiText-103 perplexity as the primary metric.The evaluation used the first 10,000 test tokens, sequence length 512, and stride 256.Source: §2.1 · p. 4

2.2 Pre-registration mechanics

  • Predictions were committed as YAML before pilot data, but the designed publish-event hook was not wired; the git timestamp—not enforcement software—records preregistration timing.The timestamp cannot prevent a researcher from mis-designing a preregistration, though changing a prediction after the outcome would leave evidence.Source: §2.2 · p. 4

2.3 The 3σ gate

  • A pilot passes only when its headline effect clears baseline noise by at least 3σ across three or more seeds, with mapping-specific pass and shelve criteria.P001 tested Wannier sparsity against PCA; P002 tested tight-binding-cutoff perplexity against a uniform-window baseline, with deliberate asymmetry in the shelve rules.Source: §2.3 · pp. 4–5

2.4 The append-only catalogue

  • The append-only catalogue retains each mapping’s status and lessons, including abandoned studies; Table 1 reports five entries and notes the Phase-1 Wilson-RG abandonment.DMRG MLP 001 and TT EMB 001 are also listed as abandoned after their Phase-3 pilots.Source: §2.4 · p. 5, Table 1 · p. 6

2.5 The cluster framing and its falsifiable prediction

  • The cluster predicts insulator-like 3a heads will be the plurality, but rank locality is a heuristic extension without a literal spatial density matrix.Attention-distance fits operationalize three regimes; their H1/H2/H3 mapping is not exhaustive, since a winning fit can have alternative explanations.Source: §2.5 · pp. 6–7

3 Five cluster mappings: four Phase-3 tests and one Phase-1 pre-emption

Across five physics-inspired compression mappings, the pilots largely falsified the predicted attention-locality and tensor-network gains, while one P003 cross-paper check failed after correction. P005 was pre-empted before a pilot; the remaining findings are bounded by their tested models and constructions.

3.1 P003 — DMRG-truncated MLPs (Phase-3 stage-A pilot)

  • P003’s registered scaling claim was shelved: attention-projection rank versus activation-Gram entropy has r = −0.434, while its MPO premise failed before entropy-based bond selection.The reshaped spectra require more singular values to capture 98% energy than the original matrices: 3804 versus 3713 for Mistral-7B and 3764 versus 3668 for Qwen3-8B.Source: §3.1 · p. 8
  • P003’s cross-paper correlation with P002’s H3 fraction falls from r = 0.523 as published to r = 0.0472 after padded positions are excluded, failing the registered check.The corrected between-layer analytic z is 0.22σ.Source: §3.1 · pp. 8–9

3.2 P005 — Wilson RG of weights (Phase-1 pre-emption)

  • P005’s Wilson-RG proposal was pre-empted by prior work on IMP as an RG scheme and layerwise sparsity allocation, then abandoned after a novelty score of 52/100.Mehta–Schwab did not pre-empt the proposal because it concerns learning dynamics, not static pretrained weights or pruning density.Source: §3.2 · p. 9
  • The P003/P005 pattern—physics-inspired construction independently derived in LLM-compression work without physics framing—makes side-by-side prior-art comparison necessary in Phase-1 surveys.The paper records this pattern as LRN-20260514-001.Source: §3.2 · p. 9

3.3 P002 — Tight-binding sparse attention (Phase-3 falsification)

  • At 3λ, tight-binding attention costs +96% perplexity and loses to matched top-k; power-law or stretched-exponential fits outperform the predicted pure exponential.At matched non-zero count, top-k gives 23.68 perplexity versus 46.47 for tight-binding; H1 wins AIC in only 2 of 384 layer-head pairs.Source: §3.3 · p. 10, Figure 5 · p. 13

3.4 P001 — Wannier-localised attention (Phase-3 falsification)

  • On Pythia-160M, all four rotations reach s⋆≈0.05, and Wannier does not separate from PCA or random Haar.The mid-layer versus edge-layer localisability direction held, but did not establish absolute sparsity and remained below the 3σ confidence bar.Source: §3.4 · pp. 11–12, Table 3 · p. 14, Figure 7 · p. 16

3.5 P011 — Tensor-train embedding decomposition (Phase-3 pilot)

  • For GPT-2, log D⋆v and surprisal are nearly uncorrelated (r = 0.016), while per-row tensor train inflates storage 1.55× rather than compressing.At the ≤1% reconstruction threshold, 94.06% of rows require D = 16; the storage and fidelity result is decided from the frozen embedding spectrum without a forward pass.Source: §3.5 · pp. 12–14

4 Cluster-level synthesis

The cluster’s pre-registered majority-insulator prediction is inverted at the tested small-model scale, while corrected cross-paper checks are null. The data point toward non-insulator regimes but do not distinguish among them or establish how attention behavior scales.

4.1 The empirical inversion

  • At ≤350M parameters, the four Phase-3 tests reject the majority-insulator prediction: P001 and P002 invert the attention prediction, while P003 and P011 remove the corresponding low-rank expectation.The rank-face result extends to 7–8B parameters.Source: §4.1 · p. 16
  • H3 stretched-exponential decay wins AIC for 12/16 median-layer heads after padding is excluded; H2 wins the remaining four, and H1 wins only 2/384 pairs overall.The predicted 70/10/20 bars in Figure 8 are illustrative; the tested claim was qualitative, that 3a would be the majority.Source: §4.1 · p. 17, Figure 8 · p. 17

4.2 Cross-paper consistency check

  • On the shared Pythia-160M grid, the Wannier-spread and tight-binding-decay measures correlate at r = −0.0583 (n = 137), far below the registered |r| ≥0.60 gate.The null remains informative: the supplied analysis reports ≥80% power to detect |r| ≥0.24.Source: §4.2 · pp. 17–18, Figure 9 · p. 19
  • P003’s cross-paper correlation falls from r = 0.523 as first published to r = 0.0472 after excluding padded positions, leaving both cross-paper checks null.The corrected between-layer analytic result is 0.22σ.Source: §4.2 · p. 18

4.3 What the data say positively

  • The attention-decay evidence is critical-like or glassy, but the data do not distinguish those sources; P001’s all-basis collapse also supports a heavy-tailed weight-magnitude interpretation.H3’s stretched exponential has a higher median-layer R2 than H2 (0.80 vs 0.74), but loses on AIC because it adds a parameter.Source: §4.3 · pp. 18–19

4.4 Scale hypothesis

  • The attention inversion is established only for the sampled ≤350M models; rank-face tests remained negative through 7–8B, while larger-scale attention replication remains incomplete.P013 reaches 12B using the insulator-like fraction fH1, but that observable is described as weak and its confirmatory arm was ambiguous.Source: §4.4 · p. 20

5 What this implies for the program

The failed small-model gate keeps P009 deferred, while the paper recommends testing alternative physics anchors and strengthening future cluster procedures. The rank-face no-gain result already extends to 7–8B, unlike the attention-regime inversion.

5.1 P009 stays gated

  • P009 remains deferred because neither required P001–P002 condition holds at ≤350M parameters; reopening it requires the specified ≥8B replication.The submission gate required ≥80% of layer-head pairs to satisfy both exponential decay and basis sparsity.Source: §5.1 · p. 21

5.2 Recommended physics-anchor pivots

  • Future work may test replica-spin-glass overlaps, CFT scaling exponents, or heavy-tailed random-matrix theory, but the paper selects none of these anchors.The proposed observables include per-head overlap distributions and attention-decay exponents across depth, context length, and tokeniser geometry.Source: §5.2 · pp. 21–22

5.3 Program-level changes

  • The authors frame negative-result publishing as a continuing program habit: five shelved subpapers are reported together in one cluster-level methodology paper.They recommend applying the same protocol to future positive or negative cluster outcomes.Source: §5.3 · p. 22

6 Related work

The paper distinguishes its cluster-level testing discipline from prior physics-AI mappings and compression methods. It situates the individual probes among sparse-attention, rotation, allocation, and tensor-network work, while noting procedural overlap and differences in scope.

Physics-AI mappings

  • Unlike physics-AI work on physical structure in networks, this paper tests candidate compression mappings through pre-registered cluster-level falsification rather than proposing a new mapping.The contribution is described as orthogonal to work connecting renormalisation-group ideas and deep learning (Mehta and Schwab, 2014; Lin et al., 2017).Source: §6 · p. 22

Rotation and allocation methods

  • P001 asks whether a Wannier-style rotation reveals latent sparsity at fixed perplexity, unlike rotation-compression methods such as SliceGPT and QuaRot.AlphaPruning (Lu et al., 2024) and OWL already occupy the per-layer allocation niche targeted by P005.Source: §6 · pp. 22–23

Sparse-attention methods

  • Sparse-attention methods provide operational context, but the paper does not benchmark against current methods including NSA (Yuan et al., 2025), MoBA (Lu et al., 2025), or Kimi Linear (Kimi Team et al., 2025).Its stated contribution is a testing discipline applicable to future sparse-attention proposals, not a sparse-attention construction.Source: §6 · p. 23

Tensor-network compression

  • SVD-LLM (Wang et al., 2025) pre-empts P003’s original procedure, while Zagitov et al. (2026) compare tensor and matrix-optimal truncation; this paper adds embedding-side tests outside their scope.The paper’s tensor-network no-gain claim is distinct from the compression-realization gap surveyed by Tarasov et al. (2026), because it reports no savings at the required fidelity.Source: §6 · pp. 23–24

7 Methodology contribution: a discipline

The methodology combines pre-registration, a 3σ decision rule, an append-only catalogue, cluster framing, and cross-paper checks. Its enforcement account is qualified: the written gate was not deployed, and its detector failed on the manuscript’s quantitative claims.

Methodology contribution

  • The paper adapts pre-registration for physics-AI mapping research with a 3σ gate, append-only catalogue, cluster framing, cross-paper checks, and agent execution under pre-registration discipline.It explicitly distinguishes these additions from pre-registration itself and its transfer to machine learning.Source: §7 · p. 24

Enforcement limitation

  • The detector returned no claims for several manuscript quantities, including r = 0.523, 1% perplexity loss, and 3σ, so the authors present enforcement as agent discipline rather than a deployed mechanism.It misread “50K tokens” as kelvin despite detecting a control claim about a 1.5 eV band gap.Source: §7 · p. 25

Pre-registration

  • Phase-3 registrations were YAML files committed before pilots, but the written pre-commit hook was not registered in the project’s loaded settings and there is no evidence it ran.The ordering claim rests on git timestamps, not on a functioning gate.Source: §7 · p. 25

Decision rule

  • A pilot passes only when its pre-registered headline effect exceeds baseline noise by ≥3σ across at least three seeds, or calibration bootstraps when seeds do not apply.The rule applies to one pre-registered headline metric, not a selected subset.Source: §7 · p. 25

Catalogue and prior-art gate

  • The catalogue records attempted mappings and lessons, but the literature gate checks only recorded prior art and cannot discover unrecorded pre-empting work.The authors found Zagitov et al. (2026) through a later survey rather than the gate.Source: §7 · p. 25

8 Limitations

The attention-face inversion is tested only on Pythia-160M and GPT-2-medium, while rank-face findings extend to 7–8B. The conclusions are also bounded by the tested calibration set, physics analogy class, and trust assumptions.

Scale: split by face

  • The attention-regime inversion rests on two checkpoints at ≤350M parameters; the rank-face no-compression finding has support up to 7–8B.P001’s embedding-rank negative replicated on OPT-1.3B, and the failed MPO premise held on Mistral-7B-v0.3 and Qwen3-8B.Source: §8 · p. 26

Positional-encoding artefacts

  • Pythia-160M uses RoPE and GPT-2-medium learned absolute embeddings; ALiBi was not tested, and arbitrary head-feature unitaries disrupt RoPE’s adjacent-pair rotations.The authors identify this incompatibility as a likely structural cause of P001’s failure and suggest post-RoPE rotations or compatible block-diagonal unitaries.Source: §8 · p. 26

Specific calibration set

  • All pilots evaluated perplexity on WikiText-103; code, math, and long-context retrieval transfer were not tested.The authors say the small models lack the context window for the framing document’s ≥64k needle-in-haystack benchmarks.Source: §8 · p. 26

Five physics frameworks, not exhaustive

  • The falsification concerns the insulator/area-law analogy class, not other physics-AI anchors such as spin-glass replica theory, CFT, or holographic RG.The tested frameworks were Wannier, tight-binding, DMRG, Wilson RG, and tensor-train embeddings.Source: §8 · p. 26

Pre-registration discipline depends on git-timestamp trust

  • The preregistration method assumes git history is not maliciously rewritten between the preregistration commit and the pilot.For adversarial settings, the authors suggest signed commits or external timestamping such as OpenTimestamps.Source: §8 · p. 26

9 Conclusion

The five-mapping cluster produced four Phase-3 pilots and one Phase-1 pre-emption, with all mappings shelved. The results invert the attention-face prediction at small-LM scale, while the paper’s central contribution is a preregistered cluster-testing method.

9 Conclusion

  • All five mappings were shelved: three Phase-3 pilots were falsified, P003 shelved its scaling claim, and P005 was pre-empted at Phase 1.P003 nevertheless passed its first cross-paper consistency check in the cluster.Source: §9 · p. 26
  • The cluster results invert the majority-insulator prediction for small-LM attention; both cross-paper checks are null after padded positions are excluded, while tensor-network reshaping gives no compression over plain SVD from 0.1 to 8B.The generic-tensor no-gain result was observed three times; its weight-side half was established concurrently and independently by Zagitov et al. (2026).Source: §9 · p. 26
  • The central contribution is a method combining git-timestamped YAML preregistration, a 3σ gate, an append-only catalogue, cluster framing, consistency checks, and daemonic execution.The authors state that the method applies beyond these five papers and beyond compression to physics-AI mapping programmes.Source: §9 · p. 27
  • Future proposals should test attention-regime claims at ≥8B or use critical, glassy, or heavy-tailed physics, and area-law mappings should identify genuine 1D locality.The authors also recommend Phase-1 procedure comparisons with SVD-LLM, AlphaPruning, OWL, and Redman et al. (2022).Source: §9 · p. 27

Data and code availability

The repository contains the experimental code, raw results, pilot figures, preregistration files, and catalogue entries, with a provenance map linking numerical claims to source artefacts.

Data and code availability

  • The claude-airesearch repository contains all experimental code, raw results, pilot figures, preregistration files, and catalogue entries.Relevant paths include projects/{P001,P002,P003,P005,P011}/, physics_redundancy_catalog.json, and pre_registered_hypotheses/.Source: Data and code availability · p. 27
  • A provenance map links every numerical claim to its source artefact at projects/M01_methodology_paper/drafts/provenance.md.Source: Data and code availability · p. 27
  • The preregistration commits are 10f2330 for P001/P002, 0e37eba for P003, and 97911fb for P011; verdict and adjudication commits are also listed.The passage identifies SHELVE commits for P001, P002, and P011, and a stage-A adjudication commit for P003.Source: Data and code availability · p. 27

A Engineering note: precomputed lag-covariance matrices for the Marzari–Vanderbilt Ωfunctional

Precomputed lag-covariance matrices replace repeated FFTs in the Marzari–Vanderbilt Ωfunctional fit, greatly reducing Wannier optimization time.

A Engineering note: precomputed lag-covariance matrices for the Marzari–Vanderbilt Ωfunctional

  • The first implementation recomputed the rotated-activation FFT at every Adam step, taking ∼6 hours per head on Pythia-160M.The replacement used precomputed lag-covariance matrices and a closed form.Source: A Engineering note: precomputed… · p. 37
  • The per-head fit fell from ∼150 seconds to ∼0.75 seconds, a ∼200× speedup validated to 10−5 against direct activation rotation.The full-pilot optimization fell from ∼6 hours to ∼7 minutes.Source: A Engineering note: precomputed… · p. 37
  • The optimization reduced the full-pilot Wannier run from ∼6 hours to ∼7 minutes, keeping the pilot within a single afternoon’s compute budget.The authors describe this engineering contribution as incidental to the paper’s methodological claims.Source: A Engineering note: precomputed… · p. 37

Related papers

Loading 2609.34292v1…