PaperlayerView exact revision on arXiv

Source-linked AI summary

A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents

Indranil Halder, Rastri Dey, Cengiz Pehlevan

arXiv:2609.32288v1cs.LGcs.AI

TL;DR

The paper asks how clean-data language-model quality degrades as the pre-training poison rate grows, beyond the usual focus on targeted backdoors. It combines an analyticity argument with a heavy-tailed truncated regression model, finding regime-dependent exponents and proposing a local-training interpretation.

  • Problem

    Prior poisoning studies emphasize targeted backdoors and their persistence, leaving open how aggregate clean-data quality changes with poison rate.

    Source: §1 · p. 1

  • Method

    The paper proves an analyticity barrier, analyzes a solvable heavy-tailed truncated regression model, and relates finite-time local LLM training to curvature-spectrum truncation.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    The optimized excess-risk exponent is ε^(q⋆/(q⋆+2)) at fixed aspect ratio and ε^(2−2/q⋆) when the ample-data limit comes first; simulations confirm the predictions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The two limits do not commute, and heavy-tailed data with poison-dependent truncation supplies a mechanism for non-integer poisoning exponents.

    Source: §1 · p. 2

    The observed LLM law may not extend to arbitrarily small ε: the ample-data calculation suggests it for ε ≳ ϕ_LLM, and the local-training connection relies on modeling assumptions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic question: how does a model's clean data performance degrade as the poison rate $\varepsilon$ grows? Motivated by our controlled pre-training runs of OLMo-style models, in which the relative clean data validation perplexity increase $\Delta$ between poisoned and clean models matched in architecture, token budget, and optimization schedule is well fit by a power law $\Delta\approx C\varepsilon^{a}$ with a non-integer exponent, we ask what such a law requires theoretically. We first prove an analyticity barrier: whenever the contaminated objective depends analytically on $\varepsilon$ around a nondegenerate clean data optimum, $\Delta$ is generically quadratic in $\varepsilon$, so a generic non-integer exponent is a signature of genuinely singular structure. We then supply that structure in solvable truncated ridge regression with heavy-tailed covariates, controlled by $q_\star$, and a label-shift poisoning. Our central result is that the excess risk scaling exponent depending on the order of limits: in the higher dimensional proportional regime it is $\epsilon^{q_\star/(q_\star+2)}$, whereas taking the ample-data limit first gives $\epsilon^{2-2/q_\star}$, and the limits do not commute. We confirm this prediction through several numerical simulations. Finally, we argue that finite training time plays the role of truncation on the curvature spectrum in local LLM pre-training, deriving the observed scaling law under heavy tailed inverse curvature spectrum as a modeling hypothesis.

1 INTRODUCTION

The paper studies how pre-training poison rate affects clean-data language-model quality, finding a power-law degradation whose exponent varies with model parameters and training time. It develops an analyticity barrier, a heavy-tail truncation mechanism, and a solvable model with regime-dependent exponents, then proposes a local-training interpretation.

1 INTRODUCTION

  • Clean-data perplexity degradation follows an approximate power law, and its fitted exponent depends on model parameters and training time.At the 774M scale the slope and intercept drift with training progress; at the Kaplan horizon both increase with model size.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Under analytic contamination near a nondegenerate clean optimum, clean excess risk is generically quadratic in poison rate, so a generic non-integer exponent requires singular structure.The paper identifies heavy-tailed data with poison-adaptive truncation as a mechanism that supplies this singularity.Source: §1 · p. 2
  • The optimized excess-risk exponent is regime-dependent: ε^(q⋆/(q⋆+2)) at fixed aspect ratio, versus ε^(2−2/q⋆) when the ample-data limit is taken first.Finite-size simulations converge to the deterministic equivalent, match predicted exponents across q⋆, and show crossover as the aspect ratio varies.Source: §1 · p. 2
  • The solvable setup is high-dimensional truncated ridge regression on heavy-tailed data, with tail parameter q⋆ and label-shift poisoning.The analysis uses a random-coordinate model and a Poisson point process mapping.Source: §1 · p. 2
  • For local LLM pre-training, the paper proposes that finite training time truncates the inverse-curvature spectrum, reproducing the observed law under a heavy-tailed spectrum assumption.This connection is presented as an argument for how the law may arise, not as an exact result for LLMs.Source: §1 · p. 2

2 RELATED WORK

Related work covers conventional language-model scaling laws, pre-training poisoning and alignment, and models of power-law structure in language. The paper distinguishes its focus on aggregate clean-quality degradation and on explaining non-integer poison-scaling exponents.

Scaling law for language-model pre-training

  • Language-model scaling work relates clean loss to model size, data, and compute; theoretical accounts connect exponents to kernel spectra, data-manifold dimension, and training-time dynamics.The paper contrasts these lines with its focus on how controlled poisoning changes clean quality.Source: §2 · p. 3

Language-model pre-train data poisoning and alignment

  • Prior poisoning work studies triggers, backdoor persistence, practical data poisoning, and alignment interventions; this paper instead studies aggregate clean-quality degradation with poison rate.It notes that Subramanyam et al. (2026) model poison fraction in a scaling law, but do not analyze exponent variation over training time or explain non-integer exponents.Source: §2 · p. 3

Power laws, and latent cluster structure of language models

  • Language and neural-network research links power-law tails to cluster structure, while other work treats natural-language power laws as drivers of language-model scaling.The paper positions its latent model as a tractable Gaussian-mixture instance with Poisson–Dirichlet cluster weights.Source: §2 · p. 3

Test-time adversarial vulnerability and train-time contamination

  • The paper distinguishes train-time contamination from inference-time adversarial vulnerability and relates its analysis to robust estimation under contamination and heavy-tailed sampling.Source: §2 · pp. 3–4

3 THE ANALYTICITY BARRIER AND HOW TO OVERCOME IT

The section establishes that smooth contamination around a nondegenerate clean optimum cannot generically produce a fractional poison-rate exponent. It then shows how heavy-tailed truncation that adapts to poison rate can introduce the required singularity.

Empirical scaling law

  • Matched poisoned and clean models are evaluated on the same held-out clean data at matched training time; degradation is approximately a power law, with slope and intercept drifting across training and model size.Checkpoints are aggregated within 15% of a target token count.Source: §3 · p. 4

The analyticity barrier

  • Analytic contamination near a nondegenerate clean-data optimum yields an analytic stationary branch and generically quadratic clean excess risk, ruling out a generic non-integer exponent.Analyticity allows only integer powers, while stationarity removes the linear term.Source: §3 · p. 4

Adaptive-truncation law

  • Adaptive truncation balances clean-tail bias against poison influence: increasing the threshold preserves more clean tail mass but admits more poison influence.The threshold is chosen to scale with poison rate, creating a singularity that can yield a fractional exponent.Source: §3 · p. 5
  • When the adaptive-truncation quadratic form is positive, the resulting exponent is a = 2β/(β + γ), which need not be an integer.A finite β, rather than truncation alone, pushes the exponent below the analytic value 2; for the stated scalar heavy-tail example, β = q − 1.Source: §3 · p. 5

4 A SOLVABLE MODEL AND THE NON-COMMUTING SCALING LIMITS

The section analyzes truncated ridge regression on heavy-tailed data under label-shift poisoning. Its optimized excess-risk exponent depends on whether the proportional or ample-data limit is taken first.

4.1 THE RANDOM CO-ORDINATE MODEL

  • The model uses high-dimensional truncated linear regression with heavy-tailed covariates and label-shift poisoning, balancing clean-tail truncation bias against poison influence through an adaptive truncation level.The covariates are generated using a randomly selected coordinate and a heavy-tailed radial variable.Source: §4 · p. 5, §4.1 · p. 5

4.2 THE SIMPLIFICATION AND MAP TO POISSON POINT PROCESS

  • Coordinate separability yields independent Poisson limits for clean and poisoned hits, reducing the risk to a scalar deterministic equivalent.The Poisson representation reduces moments and their truncation-level asymptotics to integrals against the mark intensity.Source: §4.2 · p. 6

4.3 OPTIMAL TRUNCATION AND NON-COMMUTING LIMITS OF THE SCALING LAW

  • The optimized risk has exponent q⋆/(q⋆+2) in the proportional regime and 2−2/q⋆ when the ample-data limit is taken first.The two exponents are distinct, so the limits do not commute.Source: §4.3 · p. 6, Figure 2 · p. 7
  • At fixed aspect ratio, poison hits only a fraction of coordinates and can cause order-τ errors there; in the ample-data limit, its per-coordinate influence is diluted to ϵτ.The corresponding risk contributions scale as ϵτ^2 and (ϵτ)^2, respectively.Source: §4.3 · p. 7

5 HOW THE SCALING LAW MAY ARISE IN LLM PRE-TRAINING

The section models local LLM pre-training dynamics using a curvature-spectrum filter and a heavy-tailed inverse-curvature assumption. It derives regime-dependent scaling behavior while noting that the ample-data calculation has a finite-aspect-ratio boundary.

5 HOW THE SCALING LAW MAY ARISE IN LLM PRE-TRAINING

  • Finite training time filters the preconditioned Gauss–Newton spectrum, learning well-conditioned directions quickly while leaving flat directions under-learned.The model treats this as a cutoff at τ=T on the inverse-curvature variable.Source: §5 · p. 8
  • Under a heavy-tailed inverse-curvature spectrum, clean truncation bias decays while poison influence grows with training time, with power-law behavior tied to the weak-curvature spectrum.The stated scaling exponent is non-integer for a genuinely heavy-tailed spectrum and returns to 2 as β→∞.Source: §5 · pp. 8–9
  • The predicted exponent varies with train-eval mismatch: smaller LLMs move from a≈1 toward 0, while larger LLMs move from near 1 toward 2β/(β+γ)>1.These trends are described for the gap-dominated and bias/checkpoint-dominated regimes, respectively.Source: §5 · p. 9
  • The balanced-horizon calculation is an ample-data result and may not extend to arbitrarily small poisoning rates; the expected range is ϵ≳ϕLLM.Taking ϵ→0 before ϕLLM→0 gives a different exponent.Source: §5 · p. 9

6 CONCLUSION AND LIMITATIONS

The conclusion presents the non-commuting exponents in the stylized coordinate model and treats the LLM connection as dependent on modeling hypotheses. The appendix details the local analogy and the coordinate model’s finite-sample analysis.

6 CONCLUSION AND LIMITATIONS

  • The paper reports two order-dependent exponents for optimized degradation, verifies the predictions numerically, and identifies the LLM connection as dependent on modeling hypotheses.Further experiments on the small-ϵ LLM regime and extensions to dense covariances and multi-coordinate examples remain open.Source: §6 · p. 9

A.1 CONNECTION TO THE LOCAL LLM APPROXIMATION

  • The local LLM approximation and coordinate model share the same normal-equation form, although their truncation mechanisms differ.The appendix uses the coordinate construction to examine whether the small-poison limit agrees with the population calculation at nonzero aspect ratio.Source: §A.1 · p. 14

A.2 COORDINATE-ISOTROPIC HEAVY-TAILED DATA DISTRIBUTION

  • Clean examples select a coordinate uniformly and pair it with a Pareto-tailed radial variable; the tail parameter satisfies q⋆>1.The coordinate-isotropic construction also specifies the clean example and teacher vector.Source: §A.2 · pp. 14–15

A.2.1 MODIFIED TRUNCATION SUITABLE FOR HIGHER DIMENSIONAL STATISTICS

  • Dimension-normalized matrix and vector truncations clip the same clean radial tail at different effective thresholds.This is the high-dimensional coordinate version of the clean truncation-bias mechanism.Source: §A.2.1 · p. 15

A.2.2 LABEL-SHIFTED DATA POISONING WITHIN TRUNCATION RADIUS

  • The modified poison uses a bounded feature statistic and a label shift, with samples independently poisoned at rate ϵ.The stated truncation conditions keep the poison statistics below their respective thresholds.Source: §A.2.2 · pp. 15–16

A.3 SIMPLIFICATION IN LINEAR REGRESSION WITH TRUNCATION

  • Because each sample is assigned to a single coordinate, the truncated regression statistics are diagonal and collect contributions coordinate by coordinate.The appendix also bounds the cross-moment statistic using the matrix statistic for finite collections of clean and poison data.Source: §A.3 · pp. 16–17

A.4 THE DETERMINISTIC EQUIVALENT AT FIXED ASPECT RATIO

  • At fixed aspect ratio, clean and poisoned counts at a coordinate converge to independent Poisson variables, explaining sparse poison hits in this regime.As the aspect ratio tends to zero first, each coordinate instead receives many clean and poisoned samples and Poisson fluctuations average out.Source: §A.4 · pp. 17–18
  • The deterministic-equivalent lemma gives convergence of coordinate averages for bounded functions of the limiting compound-Poisson statistics.The appendix establishes concentration of the coordinate average to obtain this limit.Source: §A.4 · p. 19

P Var(Y (D)

This section analyzes clean-risk behavior and clean-relative degradation under poisoning, deriving fixed-aspect-ratio deterministic equivalents and small-poisoning asymptotics. It also examines how optimized poisoning-risk exponents depend on whether the high-dimensional or ample-data limit is taken first.

P Var(Y (D)

  • Each coordinate has a scalar ridge normal equation, so the occupancy deterministic equivalent retains ϱ rather than introducing a renormalized parameter.The denominator is scalar before taking the limit, avoiding a resolvent self-consistency equation.Source: P Var(Y (D) · p. 20
  • At P/D→ϕ∈(0,∞), Result 4 gives a deterministic equivalent for empirical risk and for its difference from an independent clean baseline.The estimator and baseline share D, P, τ, and ϱ.Source: P Var(Y (D) · p. 20
  • At fixed aspect ratio, clean-relative degradation has the leading asymptotics of Result 5 as ϵ↓0 and τ(ϵ)→∞.The added-clean contribution is O(1), while the added-poison contribution grows as τ² and dominates in this limit.Source: P Var(Y (D) · p. 28
  • In the untruncated clean ridgeless problem, observed coordinates are fit exactly, unobserved coordinates are set to zero, and empirical excess risk is proportional to their fraction.Source: P Var(Y (D) · p. 30
  • For fixed ϕ, the ridgeless modified attack has a two-term risk equivalent and an optimal admissible truncation τopt(ε) for sufficiently small ε.Source: P Var(Y (D) · p. 34
  • Taking ϕ ↓ 0 first yields a population-limit deterministic equivalent, whose optimized poisoning exponent is 2−2/q_⋆.Source: P Var(Y (D) · pp. 35–36

A.7 THE SAME OBSERVABLE UNDER THE TWO ORDERS OF LIMITS

  • Result 9 compares high-dimensional and ample-data limits for the ridgeless estimator: at fixed positive aspect ratio with ϱ = 0, one regime has ϵ ↓ 0 and τ → ∞, while the other takes aspect ratio to zero first.Source: §A.7 · p. 36
  • Figures 5 and 6 report that scaling is more stable for larger models under random and language-switching poisoning.Source: Figure 5 · p. 37, Figure 6 · p. 37

A.8 POSSIBLE IMPLICATIONS OF THE NON-COMMUTING LIMIT FOR LLM PRE-TRAINING

The ample-data and fixed-aspect-ratio limits yield different poisoning exponents, so the observed scaling law need not persist as the poison rate becomes arbitrarily small. The paper also frames its LLM account as a local approximation to finite-time training dynamics, not a model of full transformer pre-training.

A.8 POSSIBLE IMPLICATIONS OF THE NON-COMMUTING LIMIT FOR LLM PRE-TRAINING

  • The optimized exponent is 2 − 2/q⋆ in the ample-data limit, but q⋆/(q⋆ + 2) at fixed aspect ratio.The paper suggests the observed LLM law may change at arbitrarily small ε.Source: §A.8 · pp. 37–38
  • At fixed aspect ratio, finite observations can leave some parameter directions weakly observed, so a poison hit can cause a displacement of order τ and risk of order τ^2 before averaging.The probability of such a hit is of order ε/ϕ.Source: §A.8 · p. 37

B HOW THE SCALING LAW MAY ARISE IN LLM PRE-TRAINING

  • The LLM analysis treats training near a finite-time checkpoint over a short window and presents its assumptions as a plausible mechanism, not a uniquely established explanation.Several assumptions have not yet been experimentally validated.Source: B HOW THE SCALING LAW MAY ARISE IN LLM… · p. 38

B.1 LOCAL TRAINING DYNAMICS

  • The local-dynamics construction linearizes the model around reference parameters and forms a quadratic objective from token-prediction events, logits, and their Jacobians.The reference may be a clean-objective minimizer or a checkpoint on a clean or poisoned trajectory.Source: §B.1 · p. 38

B.2 FINITE TRAINING TIME AS TRUNCATION

The paper models finite-time preconditioned training as spectral filtering of a symmetrized preconditioned curvature operator. This filtering is equivalent, up to multiplicative constants, to truncating the inverse-curvature spectrum at a level set by training time.

B.2 FINITE TRAINING TIME AS TRUNCATION

  • Finite-time training filters the spectrum of A = S^1/2Σ̂S^1/2, the symmetrized preconditioned curvature operator, rather than the raw curvature Σ̂.The preconditioner leaves the stationary point unchanged but changes the operator whose spectrum is filtered.Source: §B.2 · pp. 39–40
  • The analysis assumes a frozen, symmetric positive-definite preconditioner on the local window; for Adam, the momentum-free update fits the stated memoryless template.The paper assumes, without proof, that Adam momentum only rescales time and does not affect the exponent.Source: §B.2 · pp. 39–40
  • The local update has a spectral-filter form, with the continuous-time filter q_T(λ) = 1 − e^(−λT).The discrete-time update is also expressed as a filter q^(K)(A) applied to the preconditioned response.Source: §B.2 · p. 40
  • Finite-time training is equivalent, up to multiplicative constants, to a sharp cutoff on inverse curvature X = λ^−1 at truncation level τ = T.The spectrum being measured is that of A.Source: §B.2 · pp. 39–40

B.3 ASSUMPTION OF HEAVY-TAILED SPECTRUM

The section posits a heavy-tailed preconditioned curvature spectrum near zero and discusses how bounded preconditioning and a latent-cluster model can produce its hard edge. Existing leading-eigenvalue fits do not resolve most of the weakest modes that define this tail.

B.3 ASSUMPTION OF HEAVY-TAILED SPECTRUM

  • The assumption concerns normalized spectral mass arbitrarily close to zero, where the limiting distribution has a power-law hard edge.The eigenvalues ordered increasingly from zero inherit this hard-edge behavior.Source: §B.3 · p. 41
  • Reported power-law fits cover leading eigenvalues above a positive cutoff, whereas roughly 98% of modes defining α lie below the stated resolution floor.The section distinguishes those descending-rank fits from its assumption about normalized spectral mass near zero.Source: §B.3 · p. 41
  • Bounded preconditioning preserves the hard-edge exponent, although it can change the scaling coefficient and eigenvectors.The log–log slope remains the same when the preconditioner’s minimum and maximum eigenvalues stay fixed as dimension grows.Source: §B.3 · p. 42
  • The latent-cluster model yields a hard edge with exponent α = q − 1 for raw population curvature, inherited by the preconditioned operator under bounded conditioning.The cluster construction is motivated by power-law cluster weights in a tree-structured model.Source: §B.3 · pp. 42–43

B.4 THE CLEAN DATA TRUNCATION ERROR AND ASSUMPTION ON ITS TAIL

The merged sections model finite-time training as spectral truncation: clean truncation bias falls with time, while poison influence grows. Their balance yields a scaling law whose exponent and crossover depend on the tail assumptions and training regime.

B.4 THE CLEAN DATA TRUNCATION ERROR AND ASSUMPTION ON ITS TAIL

  • Finite-time filtering captures well-conditioned modes quickly, while weak-curvature modes remain underlearned and determine the clean residual.The validation metric, rather than training curvature alone, weights this residual.Source: §B.4 · p. 44
  • Clean truncation bias decays as T^-β, with β combining the spectral hard-edge exponent and assumptions about target mass and validation weighting.The bias measures the clean target left unlearned after finite-time training.Source: §B.4 · pp. 44–45

B.5 POISONING STRENGTH AFTER TRUNCATION AND ASSUMPTION ON ITS TAIL

  • Under the effective poison spectral-tail assumption, validation-metric poison displacement grows as T^γ as training proceeds.This growth opposes the decay of clean truncation bias.Source: §B.5 · p. 46
  • The checkpoint offset decays with continued training, even as the forced poison response builds up.The offset has the same filtered-residual structure as the clean finite-time error.Source: §B.5 · p. 46

B.6 THE SCALING LAW FROM OPTIMAL TRUNCATION

  • In the bias-dominated regime, excess risk has the scaling form δR(ε, T) = ε^a Φ(T/Tε), with the regimes meeting at T ≍ Tε.For small degradation, the same statements apply to the empirical perplexity ratio.Source: §B.6 · p. 48
  • For large LLMs, the scaling exponent starts near 1 and increases toward the crossover time; small LLMs instead decrease from 1 toward 0.The large-model trend is stated for the bias-dominated or checkpoint-dominated regimes discussed in the section.Source: §B.6 · p. 48
  • The abstract tail exponents can be related to cluster-tail parameter m and within-cluster spread r under neutral modeling choices.Those choices are η = 0, γ = 1/2, and ζ = 1.Source: §B.6 · p. 48

C.1 PROOF OF RESULT 1

These proofs establish the analytic quadratic-risk barrier, develop a singular adaptive-truncation mechanism, and connect spectral-tail assumptions to clean bias, poison growth, and the resulting scaling law. The later proof steps rely on stated tail and regime assumptions.

C.1 PROOF OF RESULT 1

  • The analytic implicit-function argument gives an analytic parameter branch, so clean excess risk has a quadratic leading correction and cannot have a noninteger leading power under the stated assumptions.The coefficient is strictly positive under the stated nondegeneracy condition.Source: §C.1 · p. 49

C.2 PROOF OF RESULT 2

  • Choosing the truncation scale as τ = tε^-1/(β+γ) balances the clean truncation bias against the contamination effect.Both contributions then have the same order.Source: §C.2 · p. 49

C.3 PROOF OF RESULT 11

  • The linearized dynamics’ finite-time response is expressed through a spectral filter, approaching the inverse-curvature response as time tends to infinity when the curvature is positive definite.The proof transfers the response between preconditioned and raw coordinates.Source: §C.3 · p. 50

C.4 PROOF OF RESULT 13

  • The cluster-model proof associates parameter subspaces with curvature eigenvalues, giving the spectrum a cluster structure.The proof describes each subspace as directions primarily changing logits on contexts from its semantic cluster.Source: §C.4 · p. 50

C.5 PROOF OF RESULT 14

  • The first-moment signal beyond a curvature threshold balances how quickly rare-cluster curvature grows against how quickly those clusters’ total probability falls.The proof identifies this balance as the source of the exponent.Source: §C.5 · pp. 50–51

C.6 PROOF OF RESULT 17

  • Under the hard-edge, target-tail, and validation-metric assumptions, the clean truncation-error falloff follows from the spectral-tail calculation.The proof completes the result by taking the square root of the derived expression.Source: §C.6 · p. 51

C.7 PROOF OF RESULT 19

  • Using the finite-time spectral filter and the assumed poison-tail law, the proof derives the large-time poison-strength growth result.It takes the square root of the spectral integral to obtain the stated growth law.Source: §C.7 · p. 51

C.8 PROOF OF RESULT 21

  • In the bias-dominated regime, balancing the clean bias and poison influence yields the scaling result; the persistent-gap contribution is subdominant, or absent in the ample-data case.The proof also maps the excess-risk scaling to perplexity in the small-degradation regime.Source: §C.8 · p. 52

D EXPERIMENTAL DETAILS ON LARGE LANGUAGE MODELS

This section documents the LLM poisoning experiments and develops spectral tools for estimating curvature structure and determining which parts of the spectrum can be resolved. It covers spectral-edge diagnostics using smooth transforms, negative moments, zero-eigenvalue atoms, and computable Gauss–Radau bounds, including the conditions and limits for identifying a power-law edge from slopes.

D EXPERIMENTAL DETAILS ON LARGE LANGUAGE MODELS

  • Poisoning uses gibberish or language-switching chat sequences with a repeated Unicode trigger, inserted at random document positions under a token-level poison budget.The actual rate can differ slightly from the requested rate because insertions are whole sequences.Source: D EXPERIMENTAL DETAILS ON LARGE… · p. 53

E.1 NOTATION AND TWO DIFFERENT EXPONENTS

  • A pure capacity law has no hard edge, while the beta-prime family permits the capacity-tail and hard-edge exponents to vary independently.The passage attributes the measured αcap value to Meterez et al. (2026) and states that the two regimes are separated by a crossover scale.Source: §E.1 · p. 56

E.2.1 KRYLOV SPACE, LANCZOS, AND THE PROBE MEASURE

  • Depth-m Lanczos quadrature matches the first 2m − 1 moments of the probe spectral measure, but finite-precision loss of orthogonality can compromise large-m Ritz values.The passage notes that the computed Gauss rule still defines a positive unit-mass measure, with support in tight clusters about spec(A).Source: §E.2.1 · pp. 58–59

E.2.2 FROM QUADRATURE TO THE EIGENVALUE-INDEX PLOT

  • Gauss quadrature nodes and cumulative weights provide estimates of eigenvalue ranks through the interlacing Markov–Stieltjes inequalities.The method orders the quadrature nodes and uses their cumulative weights to approximate the eigenvalue-index plot.Source: §E.2.2 · p. 60

E.2.3 WHY THIS RESOLVES ONLY THE TOP OF THE SPECTRUM

  • Resolving a low-lying spectral window requires a polynomial small on the bulk relative to its value at the origin; Chebyshev bounds quantify the suppression required.The passage frames this as a limitation on what can be inferred about low eigenvalues when polynomial separation from the bulk is insufficient.Source: §E.2.3 · p. 60

E.3.1 SMOOTH FUNCTIONALS FEATURING THE HARD-EDGE EXPONENT

  • The next subsection defines smoothed spectral counts using spectral-calculus powers parameterized by z, k, and T.Source: §E.3.1 · p. 60
  • For a pure power-law edge, the local slope of the smooth functional equals α exactly for every k > α and every z > 0.When k does not exceed α, the measured slope can saturate at k; comparing slopes across kernel orders provides a diagnostic for α.Source: §E.3.1 · pp. 61–62

E.3.2 DISTINGUISHING A POWER LAW FROM ANYTHING FASTER

  • If the negative k-moment is finite, the smooth-functional slope tends to k; for F(x) ∼ cµx^α, it tends to α when k > α.The moment threshold αµ equals α for a power-law edge, while αµ is infinite exactly when F decays faster than every power at the origin; in that case slopes tend to k for every k.Source: §E.3.2 · p. 63

E.3.3 ATOMS AT ZERO, AND WHAT A PLATEAU MEANS

  • As z decreases to zero, Nk(z) and L(T) converge to the zero-eigenvalue mass p0, equal to dim ker(A)/P for a finite matrix.A zero atom is incompatible with F(x) ∼ cµx^α for α > 0, but can coexist with a power-law edge in the positive spectrum.Source: §E.3.3 · p. 63

E.3.4 GENERALIZATION OF THE GAUSS RULE AND GAUSS-RADAU BRACKETS

  • For the specified completely monotone kernels, the m-node Gauss rule gives a lower bound and the Gauss–Radau rule with node zero gives an upper bound.The brackets are computable from the first m recurrence coefficients; as z decreases to zero, the Radau weight at zero bounds the kernel fraction from above.Source: §E.3.4 · p. 64

E.3.5 WHY GHOSTS DO NOT REACH THE HARD EDGE

Ghosting is concentrated at the top of the spectrum: convergence of the smallest eigenvalues is much slower, and the assumed localization keeps ghost perturbations above a positive threshold. Under that localization, the transport bound makes their effect small, but does not control rounding errors near zero.

E.3.5 WHY GHOSTS DO NOT REACH THE HARD EDGE

  • At α ≈ 0.33 and P = 1.68 × 10^8, converging the smallest eigenvalue requires m of order 10^12, versus hundreds for an isolated top eigenvalue.The hard-edge law makes bottom eigenvalue convergence slower, helping explain why ghosts may occur only at the top.Source: §E.3.5 · p. 65
  • At m = 4600, the observed ghosting is confined to λ > 0.1λ1.The section formalizes this localization by assuming the exact and computed probe measures are coupled with mass fixed below Λ and tightly clustered above it.Source: §E.3.5 · p. 65
  • Under the localization assumption, probe-measure mass either stays fixed or moves by at most s within [Λ, ∞), so it cannot cross Λ.The transport bound then controls changes in g by s supλ≥Λ |g′(λ)|.Source: §E.3.5 · pp. 65–66
  • With s ≃ 10^-8λ1, Λ = 0.1λ1, k = 2, and α ≈ 0.3, the bound is 1.6 × 10^-15 at z = 10^-6λ1 and 1.2 × 10^-20 at z = 10^-9λ1.These values use cµ = 1 and the measured ghost spread and localization.Source: §E.3.5 · p. 66
  • The bound does not cover rounding errors near zero, where the relative error diverges as z ↓ 0; the measured operator defect gives zmin ≈ 3 × 10^-9λ1.Near zero, sup|g′z,k| = k/z, so the hard-edge region remains the relevant boundary of this argument.Source: §E.3.5 · p. 66

E.4 MEASUREMENTS

Measurements on a retrained OLMo-style checkpoint find power-law-like curvature behavior with α ≈ 0.30, while identifying limits on the scales where the exponent estimate is reliable.

E.4 MEASUREMENTS

  • Measurements use a retrained 150M OLMo-style checkpoint, and Figure 11 probes a heavy-tailed curvature spectrum, with smaller z probing smaller eigenvalues.The operator is the Adam-preconditioned Gauss–Newton matrix at the final checkpoint, probed at m = 9400.Source: §E.4 · p. 66, Figure 11 · p. 67
  • R2,1 decreases from 1.47 at 10^-5λ1 to [1.01, 1.06] at the deepest plotted scale, strongly disfavoring a faster-than-power edge and supporting α ≈ 0.30.The plotted ratio approaches 1 rather than the faster-than-power limit of 2.Source: §E.4 · p. 67
  • At a slope tolerance of 0.01, the k = 2 operator floor occurs at z/λ1 = 1.860 × 10^-8, set by the linearity defect δA/λ1 = 8.5 × 10^-12.The resulting floor cannot be lowered by increasing Lanczos depth.Source: §E.4 · p. 67
  • A slope tolerance of 0.01 requires z/λ1 < 4.21 × 10^-7, while the measured ghost effect never exceeds 8 × 10^-15.The ghost effect falls steeply as z decreases.Source: §E.4 · pp. 67–68
  • The effective node count neff is a weight-sensitive participation ratio bounded by 1 and q(z), avoiding the raw count's inflation from negligible-weight nodes.It is invariant to rescaling all quadrature weights.Source: §E.4 · p. 68
  • The exponent estimate is strictly unreliable at neff = 1 and reaches its edge of validity for 1 ≤ neff < 2, where the measure's shape is minimally resolved.At neff = 1, a single node below z cannot reveal the hard-edge exponent; two nodes are the minimum configuration that can carry exponent information.Source: §E.4 · p. 69

Related papers

Loading 2609.32288v1…