PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

John Sweeney

arXiv:2606.29554v1cs.LGcs.NAmath.NAmath.OCstat.ML

TL;DR

Equal-multiset shuffle order can escape the memoryless O(η²) picture when optimizer state advances on a fixed step clock. The paper analyzes this with a finite-window lifted-state expansion; fixed-clock optimizers show O(η) order contrasts, while regular controls retain O(η²).

  • Problem

    For memoryless steps, equal-multiset reordering has no first-order endpoint effect, but practical optimizers advance buffers and counters by position, making the local order effect potentially relevant to close comparisons.

    Source: §1 · p. 1

  • Method

    A finite-window lifted-state expansion isolates the optimizer-controlled order channel and provides its contrast coefficient, leading-order permutation variance, and validity diagnostics.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Median slope was 0.998 for fixed-β momentum, 1.988 for buffer-free SGD, and 1.960 for matched-clock momentum; fixed-clock AdamW/Lion scaled near η versus η² controls.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The kernel supplies local shuffle-noise scales and positional attribution, while independent reshuffling can average the channel and coherent ordering can accumulate it.

    Source: §7 · p. 12

    The result is local to finite measurement windows; the closed-form floor is specific to frozen-preconditioner AdamW and is asymptotic, not a pointwise finite-η bound.

    Source: §3 · p. 6, §7 · p. 12

Abstract

from arXiv · show

Shuffle order can be a larger source of fine-tuning noise than a memoryless analysis predicts: fixed-clock optimizer memory makes local equal-multiset contrasts first order in the learning rate rather than second order, and the resulting order channel can be large enough for a single seed to flip a close A/B comparison. We isolate this mechanism and derive a fit-free way to size the noise it produces. For a memoryless optimizer, reordering an equal multiset has no first-order endpoint term; the leading local contrast is the $O(\eta^2)$ gradient bracket. Fixed-clock optimizers such as AdamW are different. Their moment buffers, preconditioner state, and de-biasing counters advance with the step index rather than with the learning-rate-scaled time $\tau=\eta k$, so the same gradient can receive a position-dependent endpoint weight. For any fixed finite measurement window, a lifted-state expansion gives an $O(\eta)$ equal-multiset contrast whenever the first-order replay coefficient is nonzero, while regular and clock-matched controls remain $O(\eta^2)$; a bare fixed-$\beta$ momentum buffer is already enough. A bitwise-deterministic replay from one warmed optimizer state isolates the mechanism, giving order-variance slopes 1.83 for AdamW, 2.00 for fixed-$\beta$ momentum, and 4.00 for SGD; matching the memory clock to $\tau$ restores the regular exponent. For AdamW with a frozen preconditioner, the same impulse-weight kernel gives a closed-form asymptotic order-variance floor after the local potentials are measured, with no fitted coefficients. The result is local to the measurement window (independent reshuffling can average the channel across windows), but it yields order-noise error bars, positional attribution weights, and a seed-budget criterion for fine-tuning comparisons.

1 Introduction

The introduction argues that fixed-clock optimizer state makes shuffle order a first-order source of local fine-tuning variation, unlike the second-order effect predicted for memoryless steps. It presents an impulse-weight kernel for sizing this noise and attributing it to positions in the data order.

1 Introduction

  • Memoryless steps cancel equal-multiset order at first order, but fixed-clock buffers and counters assign position-dependent weights and create an O(η) local channel.When that channel is large relative to the gap between configurations, it can flip a close comparison.Source: §1 · pp. 1–2
  • AdamW-style small-step replay keeps gradients effectively frozen while moment buffers, preconditioner state, and de-biasing counters continue advancing by position.The same gradient can therefore receive different first-order endpoint weights depending on where it appears.Source: §1 · p. 1
  • The impulse-weight kernel sizes local order noise and attributes a block’s influence to its gradient alignment and the finite-clock weight of its position.Scaling memory and normalization clocks with τ = ηk flattens the first-order profile and restores the regular O(η²) exponent.Source: §1 · p. 2
  • The theory gives the first contrast coefficient, leading-order variance over uncontrolled permutations, and local validity diagnostics to isolate the optimizer-controlled order channel in seed sensitivity.Fine-tuning seed sensitivity motivates the measurement, but the paper isolates the optimizer-controlled order channel within that phenomenon.Source: §1 · p. 2

2 The lifted-state clock theorem

The lifted-state clock theorem characterizes when reordering a fixed multiset produces a first-order endpoint contrast. It distinguishes fixed-clock optimizer memory from regular controls and specifies the conditions and scope of the result.

Regular optimizer families

  • Regular optimizer families have flat first-order impulse weights, so equal-multiset words cancel at that order and a two-step swap has an O(η²) commutator term.The displayed commutator conclusion requires the stronger second-order expansion.Source: §2 · p. 3

The lifted-state clock theorem

  • Fixed-clock state maps can keep changing as η shrinks because buffers and counters advance by step index rather than rescaled time.The lifted-state theorem assumes η-independent, C1 leading maps that are uniformly bounded near the measurement state over the finite window.Source: §2 · p. 3
  • For fixed finite windows, lifted-state replay gives Θ(η) equal-multiset contrasts when the replay coefficient is nonzero; regular families cancel at first order.The expansion tracks parameters together with optimizer state from a fixed measurement state.Source: §2 · pp. 2–4

Buffering versus adaptivity

  • Fixed-β first-moment memory alone makes impulse weights position-dependent, so nondegenerate equal-multiset contrasts are generically O(η), without coordinate adaptivity.AdamW’s additional de-biasing, preconditioning, and second-moment path change the coefficient but are not required for the shift.Source: §2 · p. 4

Clock classes

  • Matching decay and normalization clocks can flatten the impulse profile and restore O(η²), whereas scaling decay alone does not flatten raw heavy-ball momentum.For raw heavy-ball momentum, the 1/(1 − β) gain cancels the flattening.Source: §2 · p. 4

Nondegeneracy

  • The Θ(η) classification is generic, not guaranteed for every word pair and readout, because a zero replay coefficient can make the leading term higher order.Nondegeneracy is the condition that the selected replay coefficient is nonzero.Source: §2 · pp. 3–4

Scope and assumptions

  • The closed-form AdamW coefficient assumes a frozen preconditioner, while Lion and Muon are nonsmooth empirical comparisons outside that coefficient.The full frozen-gradient AdamW replay is covered by the theorem, but its preconditioner must remain in a smooth region for the stated C1 argument.Source: §2 · pp. 4–5

3 AdamW kernel and order-variance floor

For AdamW with a frozen preconditioner, finite-clock impulse weights make equal-multiset order effects measurable through a kernel that also yields an asymptotic order-variance floor. The floor supports shuffle-seed budgeting, but it is local and not a finite-learning-rate bound on total noise.

3 AdamW kernel and order-variance floor

  • At K = 16, s = 100, and β1 = 0.9, AdamW’s endpoint impulse weight falls from 0.815 to 0.100 across the window.The finite-clock weights turn an equal-multiset contrast into an order-η displacement.Source: §3 · p. 5
  • Regular optimizers have VK = 0 and leading order-induced variance O(η4), whereas matching the memory clock to τ = ηk restores the regular O(η4) class.The matched-clock arm co-scales decay and normalization, flattening the reduced impulse profile.Source: §3 · p. 6
  • The leading order variance depends on step scale, measured heterogeneity σ2ϕ, and optimizer-memory contrast VK, with an η2 lower bound under the stated perturbative-transport condition.At fixed measurement state, this is an asymptotic floor for the order component, not a pointwise finite-η bound or a bound against other stochastic sources.Source: §3 · p. 6
  • The kernel provides a local order-noise scale for shuffle-seed comparisons and positional attribution by multiplying a block’s gradient alignment by its finite-clock position weight.A seed-count calculation can estimate σord from Eq. (17) or a small ordering pilot and test whether the budget resolves the reported gap.Source: §3 · p. 6
  • When a configuration gap is comparable to the order-noise floor, a single-shuffle estimate can change the sign of the held-out-NLL comparison.The calibration compares 50/50 and 70/30 math–code mixes on Llama-3.2-1B LoRA SFT and sizes the seed budget rather than choosing between the mixes.Source: §3 · pp. 6–7

4 Local diagnostics and frozen-P scope

The local diagnostics compare curvature transport with first-order optimizer-state replay and assess whether pairwise order preferences remain scalar-sortable. Their interpretation depends on the frozen-preconditioner specialization and on staying within the local perturbative regime.

4 Local diagnostics and frozen-P scope

  • The frozen-P step is the main modeling assumption for the closed-form coefficient, variance formula, and cycle decomposition, not for the exponent theorem.Full frozen-gradient replay retains finite-step m and v paths and both clocks.Source: §4 · p. 7
  • The trust-region ratio ρ compares curvature transport to the first-order optimizer-state response; ρ ≪ 1 indicates replay dominance, while ρ ≫ 1 marks cells outside the local first-order regime.When |τAdam| is near zero, the ratio is ill-conditioned; τAdam + ∆(1) is a local approximation in the ρ < 1 regime.Source: §4 · p. 7
  • The frozen-P specialization decomposes second-order pairwise order effects into conservative potential differences and a bracket term that can create nonzero triangle curl.When the curl-carrying term stays below the conservative part, pairwise order remains scalar-sortable in the measured perturbative regime ρcurl < 1.Source: §4 · pp. 7–8
  • The scalar frozen-P replay coefficient τP approximates full replay τAdam, while χ measures their gap and ρcurl diagnoses scalar-sortability of pairwise order.Low χ indicates the scalar coefficient is close to full replay; high χ means first-order replay may still be predictable but not scalar-potential conservative.Source: §4 · p. 8
  • The three radii are non-redundant: ρ bounds the local first-order approximation, χ the scalar-versus-full-replay gap, and ρcurl scalar-sortability.A cell can lie inside one radius and outside another.Source: §4 · p. 8

5 Empirical evidence

Across the reported fine-tuning tests, fixed-clock optimizer memory produces shallower order-effect scaling than SGD and matched-clock controls. Additional measurements test variance, ordering structure, local validity, and scalar sortability, with some findings explicitly limited to coefficient-level evidence.

5 Empirical evidence

  • Fixed-clock AdamW and Lion scale near η, while SGD and matched-clock memory scale near η2 on Pythia-1B and Llama-3.2-1B.In a separate Pythia-1B control, fixed-β momentum has median slope 0.998, versus 1.988 for buffer-free SGD and 1.960 for matched-clock momentum.Source: §5 · p. 9
  • Across eight Pythia/Llama cells, fixed-clock AdamW ordering-level correlations are 0.75–0.98, while matched-clock correlations are essentially one.Per-cell amplitude slopes are 0.62–0.80, so the kernel captures rank, not amplitude; the measured η3 cross-term accounts for the deficit.Source: §5 · p. 9
  • Direct fixed-state replay gives variance slopes 1.83, 2.00, and 4.00 for AdamW, fixed-β momentum, and SGD; the warmed sweep gives slopes 1.20, 3.94, and 3.83 for AdamW, matched-clock momentum, and SGD.The warmed AdamW slope is lower than the fixed-σϕ exponent because the warmed state changes the measured heterogeneity input.Source: §5 · pp. 9–10, Figure 2 · p. 10
  • Muon’s variance slope is 1.56 [1.31, 1.82], while Lion’s variance slope of 2.87 [2.31, 3.94] is inconclusive.Muon is an empirical fixed-clock comparison outside the closed-form AdamW coefficient; the broader result is the fixed-clock state exponent law.Source: §5 · p. 10
  • In the ρ < 1 regime, the frozen-P transport coefficient predicts the live residual with Pearson 0.80; as an approximation-error predictor, ρ has Spearman 0.63 and AUC 0.79 against the reported baselines.At high ρ, correlation becomes uninformative.Source: §5 · p. 10
  • Across 700 coefficient cells, there are zero directed 3-cycles in all 286 cells where every edge satisfies ρcurl < 1, and cycles appear above that radius.This supports the scalar-sortability diagnostic at coefficient level; live validation of cyclic cells was not included.Source: §5 · pp. 10–11

6 Related work

The paper places its equal-multiset endpoint question alongside work on gradient brackets, reshuffling, curricula, optimizer memory, and continuous-time limits. Its distinction is the local order-variance channel: fixed-clock state can shift equal-multiset sensitivity from the memoryless second-order regime to first order.

6 Related work

  • Memoryless gradient-bracket analyses address domain-order effects, while this paper asks how optimizer state changes equal-multiset endpoint contrasts.The paper contrasts its fixed-clock shift from O(η2) to O(η) with symmetric minibatching results that lower stochastic-gradient bias order from O(h2) to O(h4) [Shaw & Whalley, 2025].Source: §6 · p. 11
  • Curriculum and reshuffling studies address ordering policies, full-pass SGD behavior, or convergence, whereas this work studies local equal-multiset contrasts and optimizer-state effects.Its order-orbit variance floor differs from stationary iterate variance from gradient noise [Domingo-Enrich, 2022].Source: §6 · p. 11
  • Continuous-time and SDE limits co-scale optimizer clocks with η, placing them in the matched-clock regime where normalized impulse weights flatten and the first-order contrast vanishes.The paper treats fixed-clock sensitivity as a finite-clock regime that disappears when the optimizer clock is scaled with η.Source: §6 · p. 12
  • The paper’s fine-tuning geometry uses Hessian-vector products as local diagnostics for transport and pairwise-order cycles, focusing on optimizer-state path dependence rather than per-example influence.This distinguishes its focus from influence and kernel views of fine-tuning [Malladi et al., 2023].Source: §6 · p. 12

7 Discussion

Fixed-clock optimizer state makes order position-dependent, unlike the flat first-order response of regular optimizers. The resulting order channel is local to a finite window, but its noise can be averaged or accumulated across windows and used to size shuffle-seed comparisons.

7 Discussion

  • Regular optimizers give equal-multiset words a flat first-order sum, whereas fixed-clock state can make replay weights position-dependent; matching clocks restores the flat-profile limit.Source: §7 · p. 12, §A.2 · p. 16
  • The analysis is local to a finite measurement window, requiring state recomputation elsewhere in training; the closed-form potential and leading variance coefficient are exact only for frozen-preconditioner specialization.Its order-variance-floor interpretation is asymptotic, and full AdamW can add effects outside the scalar-potential form.Source: §7 · p. 12
  • Independently reshuffled windows have O(η√Nw) order scale under mean-zero weak dependence, while coherently ordered windows can drift as O(ηNw) under the stated local-regime and suffix-map conditions.Matched-clock memory replaces the leading η by η².Source: §7 · p. 12
  • The shuffle-seed budget acts as an error bar for order noise: gaps small relative to σord can be underpowered, while large gaps are unlikely to flip.Source: §7 · p. 12

A.2 Fixed-clock singularity and nondegeneracy

  • The fixed-clock first-order classification requires a nonzero replay coefficient, and the smoothness proof applies only where warmed AdamW preconditioner coordinates stay bounded away from zero.Source: §A.2 · p. 16

A.3 Order-variance floor: proof and optimizer-class cases

The order-variance analysis uses the impulse-weight kernel to separate fixed-clock AdamW from regular optimizers and to obtain closed-form order statistics in frozen-preconditioner settings. It also relates domain potentials, cycle terms, and curvature corrections to the resulting coefficients.

A.3 Order-variance floor: proof and optimizer-class cases

  • At K = 16, s = 100, β1 = 0.9, fixed-clock AdamW weights decline from 0.815 to 0.100 and VK = 0.757 = Θ(1), while regular optimizers have Varw = O(η4).The first-order linear statistic has an exact permutation-statistic variance; in the balanced two-domain idealization, sorted words are orbit extremes, with the ratio drifting upward by 12% at within-domain spread 0.25× the domain gap.Source: §A.3 · pp. 17–18

A.4 Frozen-P first-order kernel

  • The frozen-P first-order kernel yields the stated kernel and balanced-block contrast formulas, with the warmed-buffer term canceling for balanced blocks.Source: §A.4 · p. 18

A.5 Frozen-P second-order transport

  • With decoupled weight decay, decay scales the first-order displacement and shifts the live-gradient prefix; the latter contributes to the curvature residual.Source: §A.5 · p. 18

A.6 Frozen-P potential and sorting

  • For domain i, j, the first-order edge is C(ϕj − ϕi) with C > 0, and monotone weights pair largest potentials with largest weights for the maximum.Source: §A.6 · p. 18

A.7 Second-order edge decomposition and triangle cycles

  • In the measured β1 = 0.9, s = 100 range, the finite-clock cross coefficient is several times larger than the self-potential coefficient, so raw contraction asymmetry requires clock weights.When P0 = I, the antisymmetric term reduces to the projected bracket.Source: §A.7 · p. 19

A.8 Full-v AdamW tangent coefficient

  • The full AdamW second-order live residual beyond frozen-gradient replay assumes warmed second-moment preconditioner coordinates are bounded away from zero.Source: §A.8 · p. 20

A.9 Backward-sensitivity (costate) recursion

  • A backward costate pass accumulates scalar endpoint-readout sensitivity to intermediate optimizer states, and the coefficient can be computed with an O(K) HVP recursion.Source: §A.9 · p. 20

A.10 Trust-region coefficient scaling

  • Trust-region exponent deviations are expected when the first-order denominator nears zero, finite-memory coefficients are unsaturated, or local curvature prefactors vary strongly across domain pairs.Source: §A.10 · p. 20

A.11 Finite-horizon accounting: diffusion versus drift

Finite-horizon accounting composes local order contrasts across windows under bounded, summable suffix responses. Independently reshuffled windows can yield diffusion-like spread, while a persistent order bias yields drift; matched clocks reduce the per-window channel by one power of η.

A.11 Finite-horizon accounting: diffusion versus drift

  • The endpoint identity is valid over windows where Theorem 1’s local expansion holds, with bounded suffix sensitivity and summable response; mere non-expansiveness is insufficient.Non-expansiveness can leave prefix-displacement corrections that sum to O(Nw2η2), rather than the linear-in-Nw remainder.Source: §A.11 · pp. 20–21
  • With mean-zero, weakly dependent transported increments, the endpoint mean is O(Nwη2) and variance O(η2Nw), giving centered RMS O(η√Nw) and O(√η) at Nw ∼1/η.This diffusion result assumes summable autocovariances; uncorrelated martingale-difference increments are a special case.Source: §A.11 · p. 21
  • A nonzero Cesàro mean of transported coefficients yields drift ηNwµ + o(ηNw), formally O(1) at Nw ∼1/η.This models repeated packed order, a sorted curriculum, no reshuffling, or fixed domain-order skew.Source: §A.11 · p. 21
  • Matched-clock control changes the per-window coefficient to O(η2), improving the diffusion and drift bounds to O(η3/2) and O(η) at Nw ∼1/η.Each bound gains one power of η relative to the fixed-clock case.Source: §A.11 · p. 21
  • On real models, coherent linear drift holds only while the local expansion remains valid; the accounting is a linearized first variation for Nwη ≪1.As the warm state moves, recomputation is needed; training-map Jacobians can also exceed unit norm.Source: §A.11 · p. 21

B Experimental Protocol Details

The protocol combines local optimizer-state measurements, deterministic order replays, and residual-validation diagnostics. It reports a 192-row diagnostic grid, details separate calibration and cycle experiments, and finds that residual prediction is strongest in low-ρ bins.

B Experimental Protocol Details

  • Each measurement row pairs a checkpoint, domain pair, seed, block length, and warmed optimizer state; coefficient rows are generated separately from realized AB/BA outcomes to avoid tuning against live residuals.The artifact's 192 rows are the measurement rows, not a filtered subset of a larger second-order table.Source: B Experimental Protocol Details · pp. 21–22

C.2 Baseline table

  • The diagnostic grid contains 192 model-pair-block rows from 48 files and 10 checkpoints; model/K medians reduce denominator-singularity influence, while row-level fits retain pair variation.Table 6 reports baseline predictors of first-order approximation error on these 192 rows.Source: B Experimental Protocol Details · p. 22, Table 6 · p. 25

B Experimental Protocol Details

  • The fixed-target triple protocol holds the evaluation covector fixed across source-domain edges and computes one HVP per source; the 192 pairwise rows cannot themselves establish directed cycles.Selected live AB/BA spot comparisons test regime behavior rather than validating every cyclic cell.Source: B Experimental Protocol Details · p. 22
  • Trust-region analysis uses ρ = |∆(1)/τAdam|, retains small-denominator rows, and treats denominator zeros as singularities; the local diagnostic uses ρ to decide when first-order sorting is well conditioned.When |τAdam| is very small, raw ρ can be unstable and model/K medians provide robustness summaries.Source: B Experimental Protocol Details · p. 23
  • The held-out-NLL calibration compares 50/50 math–code with 70/30 using 25 order seeds per side; both learning-rate sweep points pass the stated degeneracy criteria without exclusions.The 25 × 25 sign-change rate is a descriptive grid readout over dependent cross-seed comparisons.Source: B Experimental Protocol Details · p. 23

C.1 Residual-validation bins

  • Prediction quality is strongest for ρ ≲2, with slope toward one, Pearson up to 0.89, and sign agreement rising to 88%; it falls off or changes sign for ρ ≥5.Table 5 stratifies validation by ρ, with ∆(1) predicting the live residual ∆live−τAdam.Source: §C.1 · p. 24, Table 5 · p. 24

C.3 Clustered uncertainty and denominator controls

Clustered analyses preserve the diagnostic conclusions, while finite-window exponents and coefficient ratios qualify their interpretation. The reported local scale depends on curvature and denominator behavior, and the fixed-target cycle experiment addresses a limitation of pairwise measurements.

C.3 Clustered uncertainty and denominator controls

  • Clustered bootstrap intervals leave ρ a strong trust-region and sign-error predictor and ∆(1) predictive of the live residual in the low-ρ regime.Intervals cluster by model/pair/block file or model/pair rather than treating scalar observations as independent.Source: §C.3 · p. 25, Table 7 · p. 25
  • After residualizing tied ranks against |τAdam|, ρ remains predictive; for raw residual magnitude, |∆(1)| is strongest because it is the numerator.ρ is used as a normalized trust-region and sign-error diagnostic.Source: §C.3 · p. 25

C.4 Finite-window exponent uncertainty

  • Fixed-clock η-slope fits are near-linear but not exactly unit-slope over the tested finite window: AdamW is slightly below one and Lion slightly above one.The asymptotic theorem requires a nonzero first-order coefficient, not an exact finite-window exponent of one for every implementation.Source: §C.4 · p. 25, Table 8 · p. 26

C.5 Finite-clock coefficient scaling

  • The second-order/first-order ratio crosses one between K = 24 (0.91) and K = 32 (1.69), then grows steeply, so no single block length is universally safe.The reported scales use exact finite-K coefficients; K2 is a regime summary.Source: §C.5 · pp. 25–26, Table 9 · p. 26

C.6 Eta-slope protocol and matched-clock control

  • Matching the momentum memory clock to τ = ηk flattens first-order impulse weights and restores regular O(η2) contrasts; plain fixed-β momentum remains in the first-order class.The matched-clock control scales its EMA window as 1/η steps; its first-order impulse weights flatten in the η →0 limit.Source: §C.6 · p. 26

C.7 Qwen-3-8B first-order sweep

  • The separate Qwen-3-8B first-order sweep lacks second-order fields for ρ and is excluded from the homogeneous 192-row diagnostic grid.The 192-row grid supplies the ρ, residual-validation, baseline, and conservativity-gap summaries.Source: §C.7 · p. 27

C.8 Model size is not the local scale variable

  • Parameter count is not a stable scale variable for the local diagnostic; curvature prefactors and denominator zeros are treated as the relevant scale variables.Small Pythia checkpoints are often high-ρ, while Qwen-family medians are nonmonotone after Qwen-14B.Source: §C.8 · p. 27

C.9 Edge-level motivation for fixed-target triples

  • Across 192 pairwise rows, median qcurl is 0.25 and the 90th percentile is 0.69; these measurements motivate, but do not themselves measure, directed cycles.A valid cycle-rate test holds a common evaluation covector fixed across every edge of each domain triple.Source: §C.9 · p. 27

C.10 Additional order-orbit details

Ordering-orbit results support the frozen-preconditioner kernel while identifying finite-learning-rate amplitude gaps. Fixed-state replay separates optimizer classes, and additional tests show that the structural first-order criterion extends beyond AdamW with optimizer-specific limits.

C.10 Additional order-orbit details

  • Fixed-clock AdamW shows strong ordering-by-ordering agreement between live readout and the closed-form first-order kernel, though its slopes are below one because of a measured η3 cross-term.Table 10 reports ordering-orbit agreement by cell across 64 deterministic orderings per cell.Source: §C.10 · p. 27, Table 10 · p. 27
  • For the frozen-P AdamW variance floor, Varw ≥ (1−γ)η2AK only when the η3 correction is perturbative; negative transport covariance can put finite-η cells below the asymptotic line.The floor is not a pointwise bound for arbitrary finite η.Source: §C.10 · p. 27
  • At the locked learning rate, fixed-clock orbit variance is much larger than the matched-clock control, but the suppression magnitude is dominated by the missing AdamW preconditioner.The clock signature is the variance-slope class separation, not this suppression magnitude.Source: §C.10 · p. 28, Table 11 · p. 28
  • In lower-η reruns, Llama code–dialogue slope is 0.935 versus predicted 0.9395, while Pythia code–dialogue kernel correlation is 0.987 with an amplitude factor about 0.74.The Pythia amplitude factor is consistent with the full-v renormalization channel rather than a breakdown of the first-order clock law.Source: §C.10 · p. 28

C.11 Direct fixed-state replay

  • Direct fixed-state replay separates variance classes: fixed-buffer momentum has exponent two, buffer-free SGD exponent four, and AdamW is first-order with a small preconditioner-path remainder.The same warmed Pythia-1B state and optimizer fork are replayed across learning rates.Source: §C.11 · p. 28

C.12 Additional optimizer tests

  • Muon’s measured variance slope is 1.56 with interval [1.31, 1.82], consistent with the first-order class; Lion’s 2.87 interval [2.31, 3.94] is inconclusive at this sample size.Muon meets the structural criterion because its orthogonalized momentum is consumed on an η-independent clock.Source: §C.12 · p. 28

C.13 Fixed-target triple controls

AdamW’s finite-clock potential can determine whether bracket geometry appears in realized pairwise order preferences, while matched-clock momentum and SGD provide distinct scope controls. The worked example translates measured order variance into shuffle-seed counts for fine-tuning comparisons.

C.13 Fixed-target triple controls

  • Under the common target covector, the frozen-P component has zero directed 3-cycles in all 700 cells; bracket-only fields are cyclic on 7.1% of triples.The full edge field has zero cycles in 0/286 cases below ρcurl < 1.Source: §C.13 · p. 28
  • Matched-clock momentum has ρcurl ≤1.5 × 10−4 across all 700 cells and zero cycles at every K, whereas plain SGD’s cycle set is K-invariant.The SGD decomposition leaves a degenerate all-curl edge field.Source: §C.13 · p. 28
  • Buffered replay arms scale near η^2, while buffer-free SGD scales near η^4 in the fixed-state replay experiment.The table describes a single warmed Pythia-1B math–code cell with the optimizer state forked once and the same K = 16 ordering orbit replayed at each learning rate.Source: Table 12 · p. 29
  • AdamW’s finite-clock potential moves edges across the radius, determining whether the broader bracket geometry appears in realized pairwise order preferences.Matched-clock momentum is potential-dominated on every tested edge, while plain SGD has a degenerate all-curl edge field.Source: §C.13 · pp. 28–29

C.14 Seed-count worked example

  • 3.0 × 10−8 measured AdamW orbit variance corresponds to σord ≈1.73 × 10−4; a 2σord gap needs about four independent shuffle seeds per side, versus about sixteen for a one-standard-deviation gap.The worked example is illustrative; practitioners should compute VK for their optimizer clock and estimate σϕ or run a small ordering pilot.Source: §C.14 · p. 29

Related papers

Loading 2606.29554v1…