PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

Tiexin Ding

arXiv:2608.23573v1cs.LGstat.ML

TL;DR

The paper asks which pre-training corpus property determines transformer Weibull weight-scale growth. It measures bigram conditional entropy before training and derives a learning-rate-conditioned growth law whose constrained exponent predicts held-out within-family growth, while cross-corpus failures expose redundancy as a missing axis.

  • Problem

    The optimizer-side account leaves open what training-data property determines how much the Weibull weight scale λ grows.

  • Method

    The paper uses pre-training bigram conditional entropy D and a matched-budget shuffle baseline to predict λ^2 − λ0^2 across controlled corruption families and resolutions.

  • Results

    23 runs collapse with unit slope and R^2 = 0.941 after per-learning-rate coefficient removal, while held-out within-family prediction achieves 5.7% relative error.

  • Takeaways & Limitations

    The law provides a forward data-to-weight readout at model and layer scales, with functional form preserved across the two tested architectures and coefficients changing.

  • Takeaways & Limitations

    The law is established within controlled corruption families and over-predicts code cross-corpus, indicating that redundancy is a second data dimension.

Abstract

from arXiv · show

A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape $k \approx 1.2$ is stable across layers and models, so the scale $λ$ carries most training-induced movement. What corpus property sets how much $λ$ grows? Using the bigram conditional entropy $D = H(\text{next} \mid \text{prev})$, a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, $λ^2 - λ_0^2 = C_0(η) + C_1(η)(H_r - D)^{0.59}$, where $H_r$ is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-$η$ coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto $(H_r - D)^{0.59}$ with unit slope ($R^2 = 0.941$; direct per-$η$ fits are weaker, $R^2 \approx 0.82$). Because $D$ is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader $Φ(D,R,A,H)$ data-to-weight framework.

1 Introduction

The paper asks what corpus property determines transformer Weibull weight-scale growth and identifies pre-training bigram predictability as a quantitative predictor. Its learning-rate-conditioned law generalizes across runs, resolutions, and tested architectures, while cross-corpus behavior exposes redundancy as a separate axis.

  • Data statistic: Bigram conditional entropy D = H(next | prev) provides a training-free measure of local next-token predictability linked to weight-scale growth.D is computed from raw corpus data before training.
  • Constrained law: 0.59 is inherited from an independently measured data-side saturation relation rather than fitted directly to the weight-growth curve.The exponent equals 1/p for p ≈1.69.
  • Cross-rate result: 23 runs spanning an order of magnitude in learning rate collapse to unit slope with R^2 = 0.941 after removing the two per-η coefficients.Direct per-η fits are weaker, with R^2 ≈0.82; learning rate enters through C0(η) and C1(η).
  • Resolution: The same functional form applies at model and layer levels, with nearly stable k and block-specific scale responses.Per-layer median R^2 = 0.84, approximately matching the pooled 0.82.
  • Scope: Cross-corpus prediction over-predicts code, marking the within-corpus law’s boundary and motivating redundancy as a second data dimension.The paper frames this as part of a broader Φ(D, R, A, H) framework.
  • Forward prediction: 5.7% relative error validates held-out within-family weight growth from the pre-training statistic alone.The residual is localized to the training-side rung.

2 Background and Setup

The paper uses the Weibull scale λ as the moving readout of transformer weight distributions and asks which pre-training corpus property modulates its transient growth. It selects bigram conditional entropy as a simple, training-free predictor of the alignment-related data effect.

  • Weibull readout: k ≈1.2 remains stable across layers and models, leaving Weibull scale λ as the principal training-sensitive quantity.Weight magnitudes in transformer linear layers are summarized by Weibull shape k and scale λ.
  • Training dynamics: λ^2 growth is interpreted through AdamW’s net injection, decay, and alignment forces.The injection term supplies a data-independent floor, while alignment captures coherent update structure.
  • Research gap: The study focuses on transient λ^2 growth before saturation, complementing steady-state analyses of optimizer-dependent scale.Earlier observations that λ peaks vary with data coherence motivate the controlled data-side study.
  • Data statistic: D = H(next | prev) is chosen as the simplest pre-training statistic that predicts weight-scale growth in the controlled setting.The search included higher-order and embedding-based corpus-structure measures.

3 Method

The method varies local next-token mapping integrity while holding the corpus fixed, computes matched-budget entropy statistics before training, and measures Weibull scale on selected weight matrices. It also controls initialization and baseline anchors so the fitted coefficients capture growth rather than offsets.

  • Corruption gradient: The corruption gradient shuffles within-sequence positions from ρ = 0 to 1 while holding the wikitext corpus fixed.The main analysis uses the non-saturated regime ρ ≤0.6.
  • Data measurements: D and struct-retain fraction S are computed from the first 2.4M tokens before training to quantify mapping predictability and preserved local structure.S decreases linearly from 1 to 0 as corruption increases.
  • Fixed anchors: λ0 is measured at the common first checkpoint, while Hr = 8.0 is the matched-budget shuffling saturation value.Neither anchor is a free fit parameter; λ0 removes initialization offset and Hr sets the entropy reference.
  • Estimator reliability: D-estimation error stays below 0.011 bits across twelve disjoint 2.4M-token subsamples.Bootstrap agreement indicates residual scatter is not caused by entropy-estimation noise.
  • Weight readout: The pooled Weibull fit uses transmission-class matrices and a log-log regression over the 10–90% quantile band.Query/key/value projections, embeddings, norms, and biases are excluded; the same lens is applied per layer.
  • Interpretation boundary: Low D can reflect meaningful local mappings or cheap predictability from repetition and templating, so the law targets the first regime.This ambiguity motivates redundancy as a separate data dimension.

4 A Conditional-Entropy Law for Weight-Scale Growth

The paper relates transient Weibull weight-scale growth to a pre-training predictability margin, with learning rate entering through two coefficients. After coefficient normalization, runs across learning rates share one data law, which remains accurate through the stable phase but has a warm-up transient and weaker per-rate fits.

  • Setup: λ^2 − λ_0^2 isolates training-added scale growth across corruption levels sharing a common initial checkpoint.The reference initialization is λ_0 = 0.0232 for every corruption level.
  • Per-learning-rate relationship: At fixed η, growth is affine in the predictability margin (H_r − D)^0.59, with C0(η) as an injection floor and C1(η) as alignment efficiency.H_r is the matched-budget shuffle baseline, approximately 8.0; the exponent is fixed from the data-side saturation relation.
  • Learning-rate conditioning: The fitted coefficient scalings C0 ∝η^1.55 and C1 ∝η^1.75 are empirical descriptions from a four-point grid, not established asymptotic laws.Two of the four learning-rate settings use a single seed.
  • η-normalized collapse: 23 runs across four learning rates collapse after removing C0(η) and C1(η), achieving R^2 = 0.941 and regression slope 1.00.The 95% confidence interval for the slope is [0.89, 1.11], with residual σ = 0.13.
  • η-normalized collapse: The per-η fits are weaker at R^2 ≈0.82 because the normalized collapse tests shared form rather than direct within-rate prediction.Residual variation is attributed to structure omitted by the one-dimensional bigram proxy, including higher-order organization, redundancy, and architecture-dependent responses.
  • Stable-phase trajectory: Refitting at each step gives R^2 = 0.91–0.99 throughout the stable phase, while C1(t) saturates and C0(t) accumulates.A single dip to R^2 ≈0.52 occurs near step 250 during warm-up, when η changes sharply before data coupling begins.

5 Why the Form Is Convex: A Mechanistic Account

The convex growth law is obtained by composing two empirical relations through retained structure S: training growth is linear in S, while bigram entropy saturates nonlinearly. Eliminating S fixes the growth exponent as the reciprocal of the independently measured data-side exponent.

  • Bridge variable: The derivation uses S, the fraction of adjacent token pairs preserved relative to the clean corpus, as an internal bridge variable that cancels from the final law.S ranges from 1 for clean data to 0 for scrambled data and is not a property assigned to natural corpora.
  • Training side: Training-side growth is linear in retained structure S, with a non-zero intercept representing the data-independent injection floor.The alignment contribution is proportional to S.
  • Composition: Eliminating S yields the convex predictability-margin law with exponent 0.59 = 1/p, rather than fitting that exponent directly to growth data.The coefficients map from the training-side fit as C0 = a0 and C1 = a1/ΔH^0.59.
  • Interpretation: The convexity comes from saturation of static bigram entropy, while the weight-scale readout retains resolution where that entropy ruler flattens.The form is conditional on two empirical relationships rather than derived from first principles.

6 End-to-End Self-Validation

The paper tests whether pre-training bigram entropy can predict post-training Weibull weight-scale growth without using held-out data in the fit. Within the corruption family, the full D→S→λ² chain predicts held-out growth accurately, while cross-corpus prediction fails for code.

  • Prediction pipeline: The prediction pipeline computes D before training, recovers S through the data-side saturation relation, and then applies the training-side fit to predict λ²−λ²_0.The predicted data does not enter the coefficient fit.
  • Within-corpus validation: 5.7% relative error is achieved in leave-one-out prediction of held-out within-corpus weight-scale growth.Each corruption level is predicted from a fit on the other six levels.
  • Within-corpus validation: The D→S inversion contributes essentially no additional error compared with using the measured struct-retain fraction S.The full chain and training-side rung agree to three decimals.
  • Cross-corpus boundary: Code is over-predicted by 0.50, with obs−pred = −0.499, whereas held-out c4 is predicted within −0.08.The cross-corpus failure occurs because code’s low entropy reflects redundancy rather than a rich mapping.
  • Error decomposition: The residual is training-side scatter that is largest at the corruption-axis ends, where convex curvature is strongest.The full chain and the true-S rung remain nearly equal across corruption levels.

7 Robustness across Seed, Architecture, and Layer

The predictability–weight-scale relation persists across seeds, two architectures, and block-level readouts, while architecture changes the response coefficients rather than the shared convex form. Aggregation is valid in λ² because Weibull second moments add, not because scales themselves are additive.

  • 7.1 Seeds: 3.9% coefficient of variation across replicated settings indicates that seed variation is small.Averaging over seeds changes the per-η fit only modestly, from R2 = 0.909 to 0.936.
  • 7.2 Architecture: R2 = 0.83 for Pythia and 0.81 for Llama, with both architectures following the same monotonic convex law.The Llama-style model has the steeper fitted slope, C1 = 0.78 versus 0.45 for Pythia.
  • 7.2 Architecture: Matched Weibull shapes, k = 1.204 versus 1.207, with Γ-ratio 0.997, show that the architectural difference is a λ² response rather than a shape artifact.The coefficients are architecture-dependent even though the data-side form and exponent remain shared.
  • 7.2 Architecture: C1 ≈ 0.74–0.85 in Llama’s SwiGLU feed-forward blocks is about twice Pythia’s 0.32–0.41 feed-forward sensitivity.Pythia instead carries more response in its attention-output projection.
  • 7.3 Layer resolution: Median blockwise R2 = 0.84 across 18 blocks, with k staying nearly uniform across blocks.Only block-specific response coefficients change when the shared corpus regressor is fitted per block.
  • 7.3 Layer resolution: Pooling agrees with the parameter-count-weighted average of blockwise C1 to numerical precision, whereas unweighted averaging and summing do not.For Llama, the reported values are 0.71 weighted, 0.63 unweighted, and 15.1 summed.
  • 7.3 Layer resolution: The pooled readout screens overall sensitivity, while the per-block readout localizes where predictable structure is written into the network.Together they connect a model-level scalar with a map of blockwise responses.

8 Limitations and Scope

The law is supported as a within-corpus relation for local mapping predictability, not as a universal cross-corpus metric. Its main boundary is that redundancy can produce low entropy without the rich mapping assumed by the predictor, while architecture and training scale remain incompletely tested.

  • Scope: The established scope is a within-corpus law along a controlled corruption gradient, not a universal cross-corpus data-quality metric.The corruption varies local mapping predictability while holding the corpus fixed.
  • Cross-corpus boundary: Code falls well below the within-corpus law and is over-predicted by approximately six times the in-corpus error.Mapping-rich wikitext and c4 are predicted well, but code’s low D reflects redundancy.
  • Cross-corpus boundary: Renormalizing by a per-corpus shuffle ceiling does not remove code’s failure because the plug-in ceiling is approximately 8.0 across tested corpora.The ceiling is set by token budget rather than corpus identity.
  • Missing data dimension: Redundancy is the most parsimonious proposed second axis, but the evidence rests on a single off-law corpus and does not identify redundancy as the definitive cause.The paper frames λ²−λ²_0 as at least a two-variable function of mapping structure and redundancy.
  • Architecture scope: Cross-architecture transfer requires recalibration because C0(η) and C1(η) are architecture-specific.The shared result is the functional form, established across two families with one seed each and confounded normalization differences.
  • Model and training scale: The law is established at approximately 70M parameters and approximately 10^8 training tokens, leaving larger models and longer near-steady-state training untested.The corruption corpora also derive from one base corpus, and the natural-corpus boundary uses few held-out corpora.
  • Estimator limitation: The data-side estimator saturates above ρ ≈ 0.6, so the main law is interpreted only in the non-saturated range.The metrics themselves are classical; the claimed contribution is their closed loop and derived form.
  • Replication and causal scope: Some corruption levels and learning rates are single-seed, making coefficient magnitudes indicative rather than final.The paper establishes a controlled predictive relation, not causal mediation.

9 Related Work

The paper connects neighboring lines on alignment, data complexity, spectral diagnostics, optimizer geometry, and controlled interventions into a weight-side law linking pre-training data statistics to Weibull weight-scale growth.

  • Data complexity: Rather than assigning complexity stages in function space, the paper predicts a training outcome from the weight trajectory through a scale-growth law.
  • Weight diagnostics: The readout differs from spectral diagnostics by using entry-wise Weibull scale λ with stable k, rather than spectral tail exponents or static snapshots.
  • Optimization: Optimizer geometry determines scale dynamics, while corpus structure modulates the alignment channel and fitted response coefficients C0(η), C1(η).
  • Interventions: Controlled interventions hold the training setup fixed, swap the corpus under a fixed base model, and read the response directly in the weight distribution.
  • Architecture: Across architectures, the law preserves its form while coefficients move, distinguishing this work from prior architecture-scaling and information-in-weights approaches.
  • Synthesis: The study uses a training-free corpus statistic to predict Weibull weight-scale growth, with learning rate and architecture entering through response coefficients.The same readout operates at both model and layer levels.

10 Discussion and Future Work

The discussion frames the result as one calibrated slice of a broader response surface over predictability, redundancy, architecture, and optimization. It extends the analysis to repetition and architecture while identifying redundancy as a distinct open axis.

  • Toward the full surface: The proposed response surface Φ(D, R, A, H) maps predictability, redundancy, architecture, and optimization to Weibull weight-scale growth.This paper establishes the D axis from a pre-training statistic.
  • Redundancy: Repetition produces an inverted-U response: weight-scale growth rises, peaks near an eight-fold repeat, then falls below baseline by thirty-two-fold.The result is a single-seed lead outside the one-dimensional law.
  • Architecture and optimization: Across Pythia and Llama, the data-law sign, convex shape, and exponent persist while coefficients C0(η), C1(η) change.Llama’s gated SwiGLU is reported as about twice as data-sensitive as Pythia’s GELU, with sensitivity also increasing by depth.
  • Redundancy: Corpora with similarly low D but different redundancy separate from the one-dimensional fit, motivating redundancy as an independent axis.
  • Readout: The framework uses one Weibull parameter pair for global diagnosis and per-layer localization of where predictability is written.

11 Conclusion

The paper establishes a within-corpus, learning-rate-conditioned law linking pre-training bigram predictability to Weibull weight-scale growth. It validates the law through held-out prediction and identifies its boundary on redundancy-dominated code.

  • Conclusion: The law links local next-token predictability to Weibull weight-scale growth within a corpus and along a controlled corruption gradient.
  • Forward prediction: D is computed before training, enabling a forward chain from corpus statistic to predicted λ^2 − λ_0^2 without training outcomes from the predicted run.
  • Prediction chain: The chain is D → S → λ^2: a data-side saturation relation is inverted, then a training-side linear relation in S predicts weight-scale growth.
  • Validation: 5.7% relative error is achieved in held-out within-family prediction under η = 3e-4 and realistic corruption levels ρ ≤0.6.
  • Boundary: The cross-corpus test predicts mapping-rich text reasonably but over-predicts code, whose low entropy reflects redundancy and templated structure.

B.3 Results

The results separate the near-exact data-side inversion from residual training-side error, quantify held-out prediction, and show that cross-corpus failure occurs specifically in redundancy-dominated code. The derivation also establishes the matched-shuffle ceiling and convex data-side saturation that generate the law.

  • B.3 Results: R^2 = 0.9997 and RMSE = 0.011 bits show that the data-side inversion adds negligible error.The residual in the full chain therefore comes from the training-side A rung rather than inversion.
  • L1 validation: 5.7% relative error is obtained in leave-one-out blind prediction, while the worst single-point error is approximately 12.6% at the clean endpoint.The error is attributed to scatter in the linear-in-S training-side rung where convex curvature is strongest.
  • L2 validation: Code is over-predicted by 0.50 in magnitude, whereas mapping-rich wikitext and c4 are predicted reasonably with wiki-family coefficients.Code’s low D reflects syntactic redundancy rather than a rich next-token mapping.
  • Interpretation: The D-to-weight law is internally predictive within the tested setting, but it is not established as the unique correct model or an exact predictor.
  • Derivation: The derivation uses S and D as two measured consequences of one corruption knob, translating the training response from S-space into measurable D-space.
  • Data-side relation: S is approximately (1 − ρ)^2 and invertible, while D saturates toward the matched-budget shuffle baseline Hr, losing resolution near the scrambled end.The saturation produces the convexity and limits the usable range to approximately ρ ≲0.6.
  • Composition: The fitted saturation relation has p = 1.691, and eliminating S yields the convex exponent 1/p ≈ 0.59 in the measurable D-space law.

C.4 Robustness of the saturation exponent

The saturation exponent is robustly estimated near 1.69, yielding a reciprocal exponent of 0.59, but its status is empirical rather than first-principles. Additional checks show modest seed variation and depth-dependent sensitivity patterns across architectures.

  • Exponent robustness: p = 1.688 ± 0.014 under leave-one-out validation, corresponding to a reciprocal exponent of 0.59.The practical reciprocal-exponent band is approximately 0.55–0.61.
  • Exponent robustness: R2 reaches 0.99988 at p = 1.69, while fixed-p fits are weaker at p = 1.0 and p = 3.0.The reported values are R2 = 0.954 at p = 1.0 and R2 = 0.942 at p = 3.0.
  • Interpretation and scope: The exponent, p, and H_r are not claimed as first-principles constants; some relations are derived conditionally from measured fits.Table 5 distinguishes design inputs, measured quantities, fitted quantities, and derived relations.
  • Interpretation and scope: The mapping from corruption level ρ to S and D is specific to the positional-shuffle family and depends on corpus statistics, corruption, realization, and estimator budget.Along this single shuffle axis, S and D co-vary tightly, allowing S to be eliminated.
  • Replication: 3.9% coefficient of variation across seeds accompanies a seed-mean convex-law fit with exponent 0.59 and R2 = 0.936.The replication uses η = 3 × 10^-4, four corruption levels, and three seeds.
  • Architectural robustness: Sensitivity rises with depth in both architectures, while Llama’s gated feed-forward responds at every depth and Pythia’s responds mainly in deeper blocks.The comparison comes from per-block C1 heatmaps, with single-seed measurements.
Loading 2608.23573v1…