PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study

Fanqi Wang, Weisheng Tang, Hairong Qi

arXiv:2608.15483v1cs.LG

TL;DR

The temporal organization of neural training trajectories is less systematically characterized than other aspects of optimization. This paper measures short-horizon predictability across vision and language-model training with complementary multiscale probes, finding simpler dynamics in auxiliary parameters and localized, time-varying predictable pockets in bulk weights.

  • Problem

    The short-horizon temporal redundancy of neural training trajectories across parameters, training stages, architectures, and optimization recipes remains less systematically characterized.

  • Method

    The paper combines displacement-direction, subspace-residual, and predictor-based probes with convention-aware, null-calibrated multiscale readouts across CIFAR and Pythia trajectories.

  • Results

    Across vision and language-model training, auxiliary parameters show simpler short-horizon dynamics than bulk parameters, whose predictability concentrates in localized, time-varying pockets.

  • Takeaways & Limitations

    Short-horizon predictability provides a retrospective, parameter-resolved view of trajectory organization whose structure varies with optimization position, architecture, and training recipe.

  • Takeaways & Limitations

    The measurements emphasize low-order, short-history dynamics, depend on observation conventions, and do not establish generality across larger models or other learning paradigms.

Abstract

from arXiv · show

Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics.

1 Introduction

The paper introduces a convention-aware, multiscale framework for measuring short-horizon temporal redundancy in neural training trajectories. Across CIFAR vision training and Pythia pretraining, it finds simpler dynamics in auxiliary parameters and localized predictability in bulk parameters, modulated by optimization position, architecture, and training recipe.

  • Motivation: Short-horizon temporal redundancy in successive parameter updates has not been systematically characterized, despite its relevance to predictive acceleration methods.Recent history may inform near-future evolution, but this information need not be uniform across parameters, training stages, architectures, or optimization recipes.
  • Method: The framework combines displacement-direction, subspace-residual, and predictor-based probes across aggregate, tensor, and within-tensor group scales.It uses convention-aware and null-calibrated readouts and operates retrospectively on stored checkpoints.
  • Results: Across CIFAR and Pythia, auxiliary vector-like parameters have simpler short-horizon dynamics than bulk matrix-like parameters.Under the deepest probes, bulk tensors are largely unpredictable in aggregate but contain localized subsets with elevated predictability.
  • Results: Short-horizon structure varies jointly with optimization position, architecture, and training recipe, with parameter-resolved patterns differing across Pythia QKV, attention-output, and MLP parameters.These temporal changes are not reducible to a single event in the scalar loss curve.

2 Related Work and Positioning

Prior work predicts future weights, studies optimization-path geometry, and documents structured training dynamics, but lacks a multiscale account of short-horizon parameter predictability. This work instead treats predictability as a trajectory-level measurement problem across vision and language-model training regimes.

  • Forecasting training dynamics: Prior methods forecast future weights or updates using learned evolution patterns, auxiliary nowcasters, Koopman/DMD-style models, or optimizer-rule-based prediction.These approaches primarily target training acceleration through prediction.
  • Trajectory-level positioning: Trajectory studies analyze checkpoint-path geometry, whereas this work studies update sequences and asks about near-future predictability.Singh et al. (2025) report early macro-level saturation and later fine-scale oscillation shaped partly by weight decay and momentum.
  • Positioning gap: Existing work lacks a multiscale account spanning directional persistence, recent-history subspace structure, and explicit short-horizon prediction.The paper positions its probe families as addressing this missing combination of temporal-structure measurements.
  • Cross-regime measurement: The study measures trajectory-level short-horizon predictability across multi-pass vision training and one-pass language-model pretraining, rather than optimizing a single predictor.This framing differs from efficient-training methods focused on pruning, sparse training, data selection, or communication reduction.

3 Measurement Framework

The framework measures retrospective short-horizon predictability from pre-anchor update history to post-anchor displacement on whole tensors or disjoint within-tensor groups. It combines direction, subspace-containment, and predictor-based probes, with convention-aware permutation calibration for localized residual structure.

  • Measurement setup: Measurements use only history before anchor a and evaluate the resulting prediction or structure against displacement over horizon τ on each measurement unit S.S is either a whole trainable tensor or one of K disjoint within-tensor groups; τ, n_past, K, and grouping conventions are experiment-specific.
  • Probe families: Three non-equivalent probe families test persistence of the latest direction, containment in the recent-update span, and exploitation of richer temporal structure.Displacement-direction probes include ESA and VCS; subspace-residual probes use FRR; predictor-based probes fit explicit local models.
  • Displacement-direction probes: ESA weights coordinates equally, whereas VCS weights them by displacement magnitude, distinguishing coordinate-wise sign persistence from magnitude-weighted directional alignment.These metrics are applied to displacement-direction probes that compare recent updates with near-future displacement.
  • Subspace-residual probes: Lower FRR indicates stronger future-displacement containment in the recent-history span, while LRGF summarizes the fraction of groups with low residuals.LRGFcal uses the α = 0.05 quantile of a size-matched permutation-null FRR distribution, whose expected null fraction is approximately α.
  • Predictor-based probes: Predictor-based probes use persistence, per-element linear-trend, or DMD forecasts, emphasizing direction and selection because predicted magnitudes vary with horizon and optimizer state.DMD predictions remain in the recent-history span, linking their attainable directional alignment to FRR; selected sign agreement is compared with an unselected baseline through selectivity lift.

4 Empirical Analysis

Across CIFAR multi-pass training and Pythia pretraining, complementary probes reveal consistent but role-, scale-, and condition-dependent short-horizon structure. Auxiliary tensors are more predictably organized than bulk tensors, while localized pockets and optimizer-specific profiles expose finer temporal variation.

  • Experimental setup: The study analyzes 30 CIFAR model–optimizer cells trained for 50 epochs and Pythia 70M, 160M, and 410M checkpoints spanning steps 7,000–129,000.CIFAR crosses five architectures with six training configurations; Pythia checkpoints are spaced by 1,000 optimization steps and use 123 admissible post-warmup anchors.
  • Probe agreement: Agreement among displacement-direction, subspace-residual, and predictor-based probes supports complementary measurements of coordinate persistence, local history-subspace structure, and recoverable temporal dynamics.ESA and FRR have median per-cell Spearman ρS = −0.74, DMD direction cosine and FRR have median per-cell Spearman ρS = −0.75, and DMD and linear-trend selectors have median per-cell ρS = 0.89.
  • Parameter-role structure: Auxiliary tensors exceed bulk tensors in 29/32 trajectory-level ESA comparisons and 32/32 DMD direction-cosine comparisons, with the ordering holding across all 246 Pythia snapshots.Bulk tensors nevertheless contain transient predictable pockets, accounting for approximately 6% of pooled group–anchor observations.
  • Pythia case study: In Pythia, bulk roles rapidly cross below the 0.5 sign-agreement reference, while qkv ESA rises near 50k steps, peaks near 75k, and later declines as predictable pockets reorganize across depth and grouping scale.Head-aligned qkv excess fades around 20k steps, whereas row-level low-FRR pockets emerge later, first in deeper layers and then across deep and intermediate layers; the loss correspondence is retrospective, not causal.
  • Training-condition effects: Controlled CIFAR experiments show that architecture, optimizer family, weight decay, and momentum modulate predictability, with SAM highest, Muon near the 0.5 reference, and Adam below SGD over most overlap.Higher predictability is not equivalent to better optimization because strong final performance occurs in both high- and low-predictability regimes.

5 Discussion, Limitations, and Future Directions

The framework offers a parameter-resolved, time-local view of training organization, while its probes provide complementary but scale-dependent and retrospective measurements. Future work should test these measurements across broader regimes and relate predictable pockets to representation, circuits, and parameter communities.

  • Contributions: The measurements localize short-horizon structure across parameter roles, layers, tensors, and within-tensor groups, tracking its emergence, disappearance, and spatial reorganization.This supports retrospective analysis of checkpoint trajectories.
  • Contributions: Directional, subspace, and predictor-based probes offer complementary views whose agreement supports shared trajectory structure and whose differences distinguish non-equivalent temporal organization.The probes therefore do not measure identical forms of structure.
  • Limitations: The probes emphasize low-order directional relations and patterns representable within short, locally low-dimensional histories, rather than arbitrary nonlinear, long-memory, or high-rank dependence.This restricts the class of dynamics they can capture.
  • Limitations: Readings depend on checkpoint spacing, horizon, history length, grouping convention, and null calibration, which define the observation scale.ESA and FRR also use realized post-anchor displacement, making them retrospective or delayed rather than immediately available online signals.
  • Limitations: Temporal alignment with loss or other events should not be interpreted as mechanistic evidence because the measurements are retrospective or delayed.This limitation applies specifically to ESA and FRR.
  • Future Directions: Future work should examine predictable-pocket emergence, migration, and disappearance across broader scales and regimes, including alignment with representation formation, circuit development, and interpretable parameter communities.Sub-0.5 ESA regimes also motivate connections to edge-of-stability behavior and Ornstein–Uhlenbeck-like mean reversion.

Supplementary Material for Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime

This document is the supplementary material for the submission “Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study.” It preserves the main paper’s notation and probe terminology while organizing supplementary references through Appendices A–H.

  • Authorship: The supplementary material is authored by Fanqi Wang, Weisheng Tang, and Hairong Qi.The listed affiliation is the Department of Electrical Engineering and Computer Science at the University of Tennessee, Knoxville.
  • Document scope: The document supplements “Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study.”It identifies itself as supplementary material for the submission.
  • Conventions: Notation, probe definitions, and terminology follow Section 3 of the main paper.References labeled Section, Figure, or Eq. point into the main paper.
  • Organization: References to Appendices A–H point to corresponding sections within this supplementary document.The document distinguishes these appendix references from main-paper references.

A Experimental Coverage and Protocol Details … A.4 Measurement Clocks, Anchors, and Seeds

The study combines a controlled CIFAR panel, additional robustness and calibration families, and multi-scale public Pythia checkpoints under explicitly defined clocks, windows, anchors, and reproducibility safeguards. The protocol covers 139 training runs plus one frozen checkpoint, with documented exclusions and masking for collapsed runs and a defective 1B checkpoint.

  • A.1 Vision Panel: Architectures, Optimizers, and Recipes: The controlled vision panel crosses five architectures and six configurations, spanning MLP, AlexNet, CIFAR-style ResNet-18, ViT, ViT-mid, SGD variants, Adam, AdaGrad, and SAM.All cells train from scratch for 50 epochs with batch size 128, a once-per-epoch cosine schedule, seed 42, and epoch-boundary snapshots.
  • A.2 Additional Run Families: Additional run families vary one ingredient at a time, including a Muon arm, SGD momentum and weight-decay ladders, seed replicates, transfer dynamics, and calibration checks.The corpus contains 139 training runs and one frozen source checkpoint.
  • A.2 Additional Run Families: Two momentum-ladder runs are excluded from every readout after collapse: MLP at momentum 0.95 ends at 0.170 test accuracy, while AlexNet ends at 0.100.The AlexNet result is the ten-class chance level, and neither run contributes to momentum comparisons.
  • A.3 Pretraining Checkpoint Coverage: The pretraining coverage includes Pythia-70M, 160M, 410M, 14M, 31M, 1B, and 1.4B, with six of seven sizes used in cross-scale readings and 1B excluded there.Checkpoints are released every 1,000 steps after a dense early grid, yielding 154 revisions through step 143,000.
  • A.4 Measurement Clocks, Anchors, and Seeds: The measurement clocks use one epoch per vision step and one released checkpoint per pretraining step; pretraining checkpoints represent 1,000 optimization steps, or about 2.10 billion tokens.The clocks index different optimization amounts, so cross-regime comparisons use ordering within matched trajectories rather than levels.
  • A.4 Measurement Clocks, Anchors, and Seeds: Both regimes normally use horizon τ = 5 and history length npast = 5, while vision anchors are epochs {5, 10, 20, 30, 40} and pretraining evaluates τ ∈{1, 2, 3, 5, 10, 14}.Anchors are admissible only when complete history and future windows lie within the recorded trajectory; pretraining uses 123 anchors from steps 7,000 to 129,000.
  • A.4 Measurement Clocks, Anchors, and Seeds: Reproducibility is enforced with seed 42 for the panel, replicates at 43 and 44 on five cells, and fixed measurement-side seeds for rank selection, group subsampling, and permutation nulls.The 1B series has an upstream-defective revision at step 116000, so affected probe windows and the corresponding embedding-spectrum row are masked at read time.

B Probe Conventions, Calibration, and Robustness … C.1 The Partition Rule

The appendix defines convention-aware probes, rank and null calibration, numerical rules, and reporting discipline, then shows that grouping choices materially alter what temporal structure is measured. It also specifies the auxiliary–bulk partition used to localize predictability across parameter roles.

  • B Probe Conventions, Calibration, and Robustness: The protocol fixes grouping units, predictor constructions, rank selection, null generation, numerical conventions, robustness checks, and reporting rules before interpreting group-level readouts.These implementation details are assigned to Appendices B.1–B.7.
  • B.1 Measurement Units and Grouping Conventions: Flat K=16 partitions flattened tensors into consecutive blocks, while structural, magnitude, and Pythia segment cuts define units from architecture, causal magnitude, or transformer organization.Groups exceeding 1,500 units are subsampled deterministically; vector parameters in Pythia remain individual groups.
  • B.2 Predictor Constructions: All predictors use the five preceding one-step updates, and DMD predictions remain in the recent-history span while producing cumulative displacement over the forecast horizon.The DMD construction standardizes coordinate rows, fits a truncated dynamics operator, and rescales the rollout.
  • B.3 Rank Selection; B.4 Null Generation and the Calibrated Readout: The default FRR readout uses full history rank r = npast = 5, while cell-level ranks are selected from Gavish–Donoho estimates and calibrated against geometry-matched membership-resampling nulls.The pooled bulk FRR median is 0.815 versus the isotropic reference (D −r)/D = 0.829.
  • B.5 Numerical Conventions and the Sign Rule: ESA uses exact stored floating-point signs, making zero displacement count as agreement; this yields departures of +0.276 and +0.221 in the two AlexNet cells affected by zero updates.Up to 79% of coordinates have zero reference displacement and up to 69% are zero over both intervals in those cells.
  • B.6 Grouping Sensitivity and Within-Family Checks; What the Flat Protocol Reads; Structural and Magnitude Cuts on the Vision Panel: Recutting changes the instrument: structural cuts raise ESA dispersion to 4.4× for head qkv and restore coupling to −.42, versus 1.1× and −.16 for flat blocks.Large structural groups are nearly chance-immune, whereas flat bulk passes include about 25 percentage points of chance passing.
  • Pythia Cuts and the Granularity Ladder: Across Pythia cuts, head grouping retains calibrated qkv signal (.333/.389 versus .104/.069), while row granularity reveals FRR < 0.7 in 4 to 5% of rows.At head granularity, absolute θ readings can be near zero despite calibrated excess of +.263 for attn_out at anchor 70k.
  • Within-Family Agreement, per Cell; C Spatial Localization: the Auxiliary–Bulk Partition; C.1 The Partition Rule: Within-family agreement is strong for out-of-window versus in-window FRR (median ρS = 0.83), DMD versus linear-trend SESA (0.89), and displacement-direction readouts (0.93), but weaker across predictor families (0.15).The auxiliary–bulk partition labels vector-like normalization parameters, biases, and embeddings auxiliary, while matrix-like feature-transforming weights remain bulk.

C.2 Vision Panel: Attribution and Dual-Definition Audit … D Pocket Trajectories, Persistence, and Overlap Registrations

The appendix audits the auxiliary–bulk attribution, defines Pythia’s segment taxonomy, and specifies why absolute predictability levels cannot be compared across regimes. It also formalizes pocket census units and tracks their trajectories, persistence, and overlaps.

  • C.2 Vision Panel: Attribution and Dual-Definition Audit: 99.5% of unique model–tensor pairs agree between the morphological and operational definitions, with one ResNet classifier-head defector.Agreement also reaches 99.4% of cell–tensor instances; the defector is ResNet final classifier head linear.weight, with 2,560 parameters.
  • C.2 Vision Panel: Attribution and Dual-Definition Audit: 81.8% agreement results when the 105 threshold is applied per tensor rather than per role class.The role-class thresholding convention is therefore central to the reported audit agreement.
  • C.3 Pythia: Segment Taxonomy and the Vector Segment: Each Pythia layer contributes qkv, attn_out, mlp, and a vector segment containing normalization parameters, biases, and the final LayerNorm.The auxiliary aggregation series is an exact alias of the vector segment, with per-anchor difference 0.0.
  • C.3 Pythia: Segment Taxonomy and the Vector Segment: 73.1% of Pythia’s parameters reside in untied embedding matrices, which remain separate from the auxiliary–bulk aggregation covering 26.9%.A 0.016% row-weight entry from embedding rows into the bulk side has no numerical effect on reported readouts.
  • C.4 Cross-Regime Comparability: The same vector-like membership rule applies across regimes, but vision embedding vectors are auxiliary whereas Pythia embedding matrices are separate segments.This is the principal embedding-adjacent difference in the cross-regime partition.
  • C.4 Cross-Regime Comparability: Absolute levels are not compared across regimes because vision uses flat K=16 epoch groups, while Pythia uses architecture-aligned released-checkpoint groups.The two clocks index different amounts of optimization per probe step, so Figure 3 compares within-line ordering instead.
  • D Pocket Trajectories, Persistence, and Overlap Registrations: Predictable pockets are a small minority of bulk group–anchor observations, mostly transient, with persistence varying by parameter role.The appendix follows their population over time, individual lifetimes, a discontinuous pretraining event, and measured overlaps.
  • D Pocket Trajectories, Persistence, and Overlap Registrations: At group level, a pocket is a functional group whose FRR falls below the size-matched 5% null quantile at an anchor, using τ=5 and the raw protocol.This is the census unit and is interpreted as excess over the null.

D.1 Census and Residence

The census finds strong parameter-role stratification: vector and qkv pockets are relatively persistent, whereas MLP pockets dominate by count and are mostly one-anchor events. Persistence also depends on the unit of analysis, with tensor-level locations showing substantially more memory than transfer-group units.

  • Group-level census: The 123-anchor census shows bulk-segment shares vary by roughly threefold across layers at both model sizes.70M spans 0.056 at layer 2 to 0.173 at layer 4; 160M spans 0.046 at layer 5 to 0.155 at layer 11.
  • Residence and stability: Vector and qkv pockets are relatively persistent, attention-output pockets are intermediate, and MLP pockets are strongly transient.Vector Jaccard is 1.000 at k=1, qkv Jaccard is .875 and .842, and MLP Jaccard is .141 at both sizes.
  • Residence and stability: More than 99% of residence runs belong to MLP pockets, whose median length is one anchor at both model sizes.MLP accounts for 103,465 of 104,030 runs at 70M and 239,191 of 240,808 at 160M.
  • Multi-pass recomputation: The multi-pass vision comparison shows architecture-specific shifts in population shares and adjacent-anchor stability across ViT-mid and ResNet groups.ViT-mid patch share rises .194 → .487 → .791, while ResNet 3×3 share rises .143 → .381 → .472; adjacent-anchor Jaccard is .480 for ViT-mid MLP and .244 for 1×1.
  • Dependence on the unit: Persistence depends strongly on the unit: transfer groups have cross-run Jaccard 0.115 to 0.135 against chance 0.111, whereas tensor-level within-run persistence is 0.57 to 0.73.At tensor level, late-training cross-regime overlap converges to 0.57 to 0.71, while early-anchor overlap is 0.23 to 0.38.

D.2 Emergence and Survival

Predictable pockets emerge later and in different temporal patterns as model size changes, with layer-localized relays rather than diffuse populations. Their structure is segment- and granularity-dependent: attention output projections follow qkv at row scale, MLP signal is confined to output halves, and coordinate membership rotates over time.

  • Size-ordered timing: Pocket onset shifts with size and can decouple from null-line descent: 70M starts both at 20k, while 410M starts pockets at 48k versus null-line onset at 9k.At 160M, null-line descent is two-stage, beginning moderately at 13k and becoming deep around 72–76k, while pocket onset remains 20k.
  • Layer relay within a run: At 70M, row-level pockets relay through layer-localized waves, beginning in layer 5, shifting to layer 3, then peaking in layer 4 before a later wave near 115k.The layer-4 wave peaks at 42.1% at 84k and remains elevated from 50 to 120k.
  • Attention-block coupling: At row granularity, attention output projections form pockets at about half qkv’s fraction in corresponding layers, while head granularity shows no output-projection population.The 70M output projection reaches 19.3% versus qkv’s 42.1%, and the 160M output projection reaches 15.7% versus qkv’s 32.8%.
  • MLP asymmetry: Across all five sizes, only the MLP output half carries signal, with peaks of 1.8–15.7%, while the input half and magnitude cut remain at or below 1.6% and 0.12%.The deepest layer is the carrier in every size, providing a third negative reading for the MLP magnitude hypothesis.
  • Form of the emerging population: At 70M qkv, coherent-motion coordinates rotate over time: adjacent-anchor membership Jaccard is .14 to .18, while cross-head timing sharing remains 0.75 to 0.99.Within heads, the family condenses into two or three row blocks, reaching four to eight times chance adjacency and 20 times at layer 5.

D.3 The One Discontinuous Event … E.2 Relation to Existing Coherence and Noise Measures

The study identifies a discontinuous 70M qkv event driven by null changes and population turnover, while cross-probe registrations link pockets to movement, normalization dynamics, low-participation heads, and independent trajectory summaries. These comparisons clarify that the measures describe related but distinct aspects of stored training trajectories and differ from existing gradient-coherence and subspace-based quantities.

  • D.3 The One Discontinuous Event: At 70M, the qkv pocket set collapses from 51 groups to 3 between 19k and 20k while median FRR remains .9959 to .9952, because the null q05 falls from .9937 to .7201.The calibrated readout therefore drops from .354 to .000 even though measured FRR does not move; the surviving population is nearly disjoint from the pre-collapse set.
  • D.3 The One Discontinuous Event: The surviving qkv set stabilizes after the event but shares almost no membership with the pre-collapse population, whereas attention-output-projection sets never stabilize.Jaccard against the final set is near zero before 48k, rises to .5 to .8 over 52 to 70k, and settles at 1 in the tail; adjacent-anchor median is .33 for attention output projection.
  • D.3 The One Discontinuous Event: The event co-locates with deep qkv rank collapse: participation ratio reaches .011 and .007 at 70M layers 3 and 4, versus .75 to .77 for MLP layers.At 160M, deep qkv reaches .007 to .030 in layers 6 to 10, while MLP remains .65 to .85; the vision runs instead show monotonic pocket expansion.
  • D.4 Overlap Registrations: Pocket registrations show qkv coordinates strongly enriched among top-K movement coordinates, reaching 4.0× at 70k and 9.0× at 129k against an equal-cardinality hypergeometric null.The anchor-resolved qkv ratio steps from 0.98 to 3.77 across 19k to 20k and later reaches a maximum of 16.2× at 23k.
  • D.4 Overlap Registrations: Normalization-gain outliers have higher median ESA in 12 of 13 tensors and higher median coherence in 11 of 13, with rank correlations of .78–.94 in those 11 tensors.Eighteen qkv heads fall below the pooled-bulk participation-ratio fence, including the seven lowest heads that exactly match the 70k pocket set.
  • D.4 Overlap Registrations: Across vision structural cuts, conditioning on FRR < 0.7 raises VCS > 0.4 from 38.9% to 88.4% on ViT-mid, while ESA > 0.7 lifts by 3.4× on ViT-mid and 23.8× on ResNet.All three thresholds are passed simultaneously by 6.7%, 3.3%, and 0.3% of ViT-mid, ViT, and ResNet groups, respectively.
  • E Movement Directedness and Gradient-Coherence Correspondence: Movement directedness summarizes realized displacement relative to path length, while loss-side gradient coherence uses loss decrease, learning rates, and gradient norms without parameter-direction information.Movement directedness lies in [0, 1], equals one for straight traversal, and falls toward zero when steps cancel; the reported correspondence has ρ = 0.978 and ρ = 0.804.
  • E.2 Relation to Existing Coherence and Noise Measures: These quantities are related to, but distinct from, per-example gradient coherence and low-dimensional subspace concentration: trajectories can double back within a fixed subspace, lowering directedness without changing the subspace.The measures are registered as independent summaries of the same stored trajectories rather than as certifications of one another.

E.3 The Full Comparison … F.4 Where the Calibrated Signal Survives

Across vision transfer conditions and six Pythia sizes, short-horizon predictability tracks realized parameter-motion geometry and consistently separates auxiliary from bulk parameters. Within bulk parameters, calibrated structure varies by segment, layer, depth, width, and training phase, with comparisons constrained by protocol and exclusion choices.

  • E.3 The Full Comparison: ESA correlates tightly with movement directedness across eight transfer conditions, while its association with loss-side coherence is weaker but still strong.The reported correlations are ρ = 0.978 and ρ = 0.804, respectively; the comparison is retrospective and observational on the vision side at τ = 5.
  • F. Pythia Scale Extensions: The six-size Pythia registry uses a shared 143,000-step schedule and identical calibrated measurement protocol across 123 anchors.The protocol uses npast = 5, reference ∆wa−1, Gavish–Donoho rank selection, parallel raw and column-normalized FRR, membership-resample nulls, and α = 0.05 excess readouts.
  • F.2 The Auxiliary–Bulk Partition Holds at Every Size: Across all six sizes and 123 anchors, auxiliary medians exceed bulk medians under ESA, 1 −FRR, and DMD direction cosine.The smallest median gaps occur at 1.4B late in training: 0.036 for ESA, 0.351 for 1 −FRR, and 0.115 for the DMD direction cosine.
  • F.2 The Auxiliary–Bulk Partition Holds at Every Size: The auxiliary–bulk separation is not caused by frozen coordinates, and strict matrix grouping is retained because looser conventions change DMD more than ESA or 1 −FRR.Frozen-coordinate fraction is identically zero; looser grouping shifts means by at most 1.2×10−4 for ESA and 1 −FRR, but up to 6.3 × 10−3 for the DMD cosine.
  • F.3 Segment Division of Labour: Under ESA, matrix segments converge with model growth while the vector segment remains higher; under calibrated readouts, qkv instead shows a size-dependent collapse to the null level.The median matrix-segment spread narrows from 0.048 at 14M to 0.011 at 1.4B, while qkv collapse onsets range from 20k at 70M to 71k at 160M, with no sustained collapse at 1.4B.
  • F.3 Segment Division of Labour: At row granularity, the attention output projection shares qkv’s wave, whereas only the output half of the MLP carries signal.These segment-resolved pocket populations are reported in Appendix D.2 rather than repeated here.
  • F.4 Where the Calibrated Signal Survives: Calibrated qkv signal begins broadly, contracts to deeper persistent bands as models deepen, and exhibits distinct tail behavior at 1.4B.At 1.4B, 22 of 24 layers remain above null in the tail, interrupted by an extinction window near 96 to 118k and followed by recovery from about 120k.
  • F.4 Where the Calibrated Signal Survives: Parameter count alone does not order transitions: at equal depth, wider 1.4B is later than narrower 410M, while deeper-versus-wider comparisons remain confounded by the excluded 1B model.At 24 layers, 410M reaches row-level pocket onset at 48k and null-line onset at 92k, whereas 1.4B reaches neither by 128k.

F.5 Co-location Registry for Late Events · F.6 Open Questions · G Architecture- and Recipe-Conditioned Results

Late-event alignment differs by model size: loss, ESA, and pocket events co-locate at 70M and 160M but separate at 410M, while several late readouts form an unresolved cluster. Open questions concern whether these patterns reflect trajectory events or readout artifacts, how to interpret the 1.4B edge case, and how architecture and training recipes condition predictability.

  • F.5 Co-location Registry for Late Events: At 70M and 160M, the loss minimum, ESA turn, and main pocket wave occur in the same late window.At 70M, the loss minimum is at step 50,345, ESA peaks at 0.467 near 77k before reaching 0.426 at 98k, and the row-level wave peaks at 84k; at 160M, the loss minimum is at 76,409, ESA reaches 0.424 at 68k, and the pocket peak is at 92k.
  • F.5 Co-location Registry for Late Events: At 410M, bulk ESA rises from about 85k to 0.48–0.49 over 111–117k despite the loss remaining flat or descending.The ESA readout sits on a 0.436–0.441 plateau from 60k to 85k, then climbs steadily, without a loss-bottoming event inside the anchor domain.
  • F.5 Co-location Registry for Late Events: The predictor-based bulk amplitude deficit begins earlier in larger models, at 58k, 31k, 21k, 15k, 14k, and 11k across ascending sizes.This is the appendix’s only cross-scale ordering monotone in parameter count.
  • F.5 Co-location Registry for Late Events: At the last anchor, bulk median amplitude ratios fall to 0.13 to 0.38 at every size, while the largest eigenvalue modulus settles at 0.59 to 0.68, except 14M at 0.78.The fitted operator retains a stable spectrum even as its predicted step amplitude falls below the realized step.
  • F.6 Open Questions: Several late readouts cluster together, but whether they represent one phenomenon or several remains open.On 70M and 160M, auxiliary DMD amplitude ratios become exactly zero from anchors 118k and 117k; on 410M, ESA, pocket, and an MLP output-half patch occur in 110–120k.
  • F.6 Open Questions: The exact-zero auxiliary amplitude ratio could reflect an instrument boundary rather than a trajectory property, requiring a readout that avoids the same fit to distinguish them.The late cluster likewise remains unresolved as one event versus several.
  • F.6 Open Questions: The 1.4B model is a transition sample at the observation-window edge, not evidence for or against a completed transition.Its row-level population reaches 3.0% at the last anchor and is still rising, while its calibrated band has not contracted; its local loss telemetry also does not match the released 143,000-step cosine schedule, so no loss landmark or co-location statement is made.
  • G Architecture- and Recipe-Conditioned Results: The appendix extends Section 4.4 with size-controlled architecture comparisons, weight-decay and momentum dose studies, a wider optimizer panel, and predictability-versus-optimizer-quality comparisons.Section 4.4 uses optimizer-family variation as its compact example and defers these additional analyses to the appendix.

G.1 Size-Controlled Architecture Comparison … H.1 Protocol and Selection Rule

Controlled comparisons show that architecture, optimizer position, and training-recipe hyperparameters systematically reshape short-horizon predictability, while higher predictability does not imply better optimization. A retrospective spatial prior is also tested online through selective forecast write-back to auxiliary or all trainable parameters.

  • G.1 Size-Controlled Architecture Comparison: ViT-class models retain more bulk directional persistence than the size-matched ResNet throughout the measured trajectory under the stated SAM conditions.At epoch 5, the two ViT scales read about 0.72 and 0.71 against 0.65 for the ResNet; around epoch 40, they read about 0.60, 0.60, and 0.52.
  • G.2 Optimizer Families along Training Loss: When aligned by shared training loss, SAM occupies the highest bulk ESA regime, Muon stays near the 0.5 sign-agreement reference, and Adam lies about 0.08 below plain SGD.SGD and AdaGrad approach a similar low-loss level near 0.545 despite following different profiles earlier.
  • G.3 Weight-Decay and Momentum Dose Ladders: Weight decay increases ESA and VCS while lowering FRR, with ESA approaching 0.90 and FRR falling to about 0.28 at the largest dose.At λ = 10−4, FRR separates from plain SGD at about 0.66 against 0.79, before ESA and VCS do.
  • G.3 Weight-Decay and Momentum Dose Ladders: Higher momentum depresses both displacement-direction readouts and raises FRR modestly; VCS falls more than ESA, whose change is only about −0.02 to −0.03.These effects concern realized parameter trajectories, not smoothness of the underlying gradient estimate.
  • G.4 Predictability Is Not an Optimizer Ranking: Across five architectures, anchor-averaged bulk ESA and final test accuracy have negative rank correlations, with a median of −0.71 and a range from −0.89 to −0.20.The five correlations use cell counts of 6, 6, 7, 7, and 4, so the weakest rests on four points.
  • G.4 Predictability Is Not an Optimizer Ranking: Along the weight-decay ladder, predictability rises monotonically while the 2 × 10−3 arm reaches about 0.707 final accuracy against about 0.755 for the 10−3 arm.These comparisons are representative rather than exhaustive and do not establish architecture or update-rule rankings.
  • H Online Selective Prediction: The online check tests whether retrospective spatial organization can guide prediction during training, writing forecasts only when coordinate-wise scale and dynamic-consistency tests are passed.Coordinates failing acceptance continue to follow the baseline optimizer, and the acceptance mask is unchanged across arms.
  • H.1 Protocol and Selection Rule: Under pure SGD with momentum and weight decay both zero, full-scope and auxiliary-only eligibility use the same predictor and acceptance mask, differing only in write-back eligibility.The comparison uses ViT-tiny and ViT-mid over three seeds, with accuracy read at each run’s held-out-split-selected epoch against the same-seed baseline.

H.2 Multi-Seed Results · H.3 What This Does and Does Not Show

Across two architectures, restricting eligibility to higher-confidence regions produced larger, more stable gains while writing back to less than 1% of parameters. The result supports an auxiliary–bulk separation but remains an initial, noncausal indication rather than evidence of uniform predictability, wall-clock acceleration, or online transfer.

  • H.2 Multi-Seed Results: H.2 Multi-Seed ResultsOn both architectures, the restricted arm produced larger mean gains while accepting 0.80% of ViT-tiny parameters and 0.32% of ViT-mid parameters; the full-scope arm accepted about 24%.
  • H.2 Multi-Seed Results: The full-scope arm yielded smaller mean gains of +0.62 percentage points on ViT-tiny and +0.73 on ViT-mid.These gains accompanied acceptance of about 24% of parameters.
  • H.2 Multi-Seed Results: The restricted arm had lower across-seed variability, with a standard deviation of 0.09 percentage points at both scales.The passage contrasts this with 0.18 to [truncated].
  • H.2 Multi-Seed Results: Narrowing eligibility to retrospectively identified higher-confidence regions gave larger and more stable gains than making everything eligible, despite accepted fractions roughly 30- and 70-fold smaller.Thus, the improvement was not obtained by predicting more.
  • H.3 What This Does and Does Not Show: H.3 What This Does and Does Not ShowThe result is consistent with the auxiliary–bulk separation reported in Section 4.2, and the authors explicitly limit the claim to that consistency.
  • H.3 What This Does and Does Not Show: The comparison does not isolate role restriction causally because the arms differ in both eligibility and accepted fraction, and it does not show that auxiliary parameters are uniformly predictable.The higher-confidence region is higher-confidence on average, not uniformly predictable.
  • H.3 What This Does and Does Not Show: The study is not a wall-clock acceleration result because predictor overhead was not accounted for, and it covers only two model–training cells at one optimizer setting.Accordingly, it is an initial indication rather than a demonstration that the framework transfers to online use.
  • H.3 What This Does and Does Not Show: The coordinate-wise acceptance mask, rather than role restriction, remains the operative safeguard, while static role eligibility captures neither the retrospective probe panel nor localized bulk-tensor pockets.The limitation includes the localized pockets described in Appendix D.
Loading 2608.15483v1…