PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

Hong Ha Le, Jackie Lok, Atsushi Nitanda, Yan Shuo Tan

arXiv:2610.07074v1stat.MLcs.LG

TL;DR

The paper asks how gradient-based pretraining constructs amortized threshold estimators and how simulated-task budgets limit their accuracy. It studies a one-head softmax-attention model, proving a frozen estimator’s error rate and describing the coordinated training dynamics; the one-head model is not uniformly consistent over the full threshold interval.

  • Problem

    How gradient-based pretraining constructs an amortized estimator, and how the number and size of simulated datasets limit its accuracy, have not been analyzed for this setting.

    Source: §1 · p. 2

  • Method

    The paper analyzes threshold estimation with a one-head softmax-attention model, using labeled pretraining tasks and gradient dynamics to learn an estimator applied to a fresh context.

    Source: §2 · p. 3, §5.1 · p. 5, §8 · p. 8

  • Results

    eO((m ∧ n)^-1 + N^-1) error is certified for each fixed interior threshold and every fresh-context size N.

    Source: §1 · p. 2, §3 · p. 4

  • Takeaways & Limitations

    Population training calibrates relative label and feature scores before increasing attention concentration, while finite-corpus gradient control certifies continued sharpening at growing parameter scales.

    Source: §5.1 · p. 5, §8 · p. 8

    No single fixed orientation in this one-head score family is uniformly consistent over the full physical threshold interval [0, 1].

    Source: §8.2 · p. 9

Abstract

from arXiv · show

Estimating a decision threshold requires locating observations near an unknown boundary. We study how gradient-based pretraining learns this statistical rule in a two-parameter softmax-attention model with a fixed feature and inequality direction. Pretraining uses labeled contexts and their true thresholds; a fresh threshold must be inferred from context alone. Under a large-resolution initialization, constant-step gradient descent on $m$ tasks with $n$ examples each produces a frozen estimator with error $\widetilde O((m\wedge n)^{-1}+N^{-1})$ for each fixed interior threshold and every fresh-context size $N$. The two terms separate finite-pretraining accuracy from fresh-context localization. The mechanism is coordinated parameter divergence: population training calibrates the relative label and feature scores, then increases the attention scale as $t^{1/4}$, giving population threshold error $O(t^{-1/4})$. To transfer this mechanism to a fixed finite corpus, we control gradient errors relative to the shrinking directions of progress at successive parameter scales. This certifies a growing training interval without requiring long-time tracking of the population trajectory. We also identify the boundary limitation of the one-head model and explain statistically what a reflected symmetrization could achieve.

1 INTRODUCTION

The paper studies how gradient-based pretraining can learn an amortized estimator for a decision threshold from labeled tasks, then apply it to a fresh context. It analyzes both the estimator’s finite-pretraining accuracy and the training dynamics that produce it.

1 INTRODUCTION

  • Prior work analyzes gradient-based pretraining dynamics for supervised prediction, while how pretraining constructs amortized estimators and how simulated-task budgets limit accuracy remain unanalyzed.Amortized-estimator research cited here primarily studies statistical properties rather than optimization dynamics.Source: §1 · pp. 1–2
  • The paper isolates estimating a single decision threshold with a known split feature and inequality direction, a computation relevant to decision-tree splits.Decision trees are described as interpretable models and building blocks of tabular ensembles.Source: §1 · p. 2
  • A frozen estimator achieves error eO((m ∧ n)^-1 + N^-1) for each fixed interior threshold and every fresh-context size N.The first term reflects finite-pretraining accuracy; the second reflects localization from the fresh context.Source: §1 · p. 2
  • Population training first calibrates the relative label and feature scores, then increases attention scale as t^1/4 while threshold error falls as O(t^-1/4).Both parameters diverge during this process.Source: §1 · p. 2
  • Finite-corpus gradient control certifies continued sharpening up to scale (m ∧ n)/log(mn), without tracking the population trajectory over the growing horizon.The proof normalizes gradient errors by shrinking progress margins at successive parameter scales.Source: §1 · p. 3

2 PROBLEM AND ATTENTION ESTIMATOR

Pretraining tasks draw thresholds uniformly from an interior interval and provide labeled uniform-feature examples; a fresh task provides context but withholds its threshold. The paper compares a one-head softmax-attention estimator trained by gradient descent with an explicit statistical localization rule.

2 PROBLEM AND ATTENTION ESTIMATOR

  • Each of m pretraining tasks draws α uniformly from [ρ, ℓ] and supplies n labeled uniform features; the fresh task withholds its threshold and supplies N examples.Here ℓ = 1 − ρ and the fresh task is independent.Source: §2 · p. 3
  • The maximum feature with a positive label, M+_N, localizes the threshold at scale N^-1, providing an explicit benchmark for the learned estimator.The stated tail bound is P(α − M+_N > ε) = (1 − ε)^N ≤ e^-Nε.Source: §2 · p. 3
  • A single self-attention head computes the estimator from context tokens, with only λ and µ trained.The readout attends to context tokens using the specified query, key, value, and output matrices.Source: §2 · p. 3
  • For µ > 0, k = λ/µ measures relative preference for labels and features, while µ controls concentration.Once k is calibrated, increasing common scale concentrates attention on the largest positive observation.Source: §2 · p. 3
  • Training minimizes empirical squared error by full-batch gradient descent with a constant step size, using the infinite-task, infinite-context objective as its population counterpart.The deterministic initialization has large concentration and a score ratio that may still be inaccurate.Source: §2 · p. 3

3 LEARNED ESTIMATION AND TRANSFER

Theorem 3.1 gives a frozen estimator whose error separates finite-pretraining accuracy from fresh-context localization. The guarantee also transfers to fresh-query classification, while its theoretical constants do not provide a practical stopping rule.

3 LEARNED ESTIMATION AND TRANSFER

  • The frozen parameters are selected using the pretraining corpus before the fresh threshold, context size, confidence level, or observations are specified.The selection is measurable with respect to the pretraining corpus.Source: §3 · p. 4
  • The error is eO((m ∧ n)^−1 + N^−1) for each fixed interior threshold and every fresh-context size N.The first term measures finite attention resolution certified by pretraining; the second measures localization from the fresh context.Source: §3 · p. 4
  • The theorem’s constants are theoretical and do not give a calibrated practical stopping rule.The result also does not assert that accuracy deteriorates beyond the certified interval.Source: §3 · p. 4
  • Fresh-query misclassification has the same rate as threshold estimation because its conditional error is exactly |F_N − α_fr|.This holds for an independent query feature uniformly distributed on [0, 1].Source: §3 · p. 4

4 FROM PARAMETERS TO STATISTICAL ACCURACY

The estimator’s error reflects negative-class attention mass and finite concentration within the positive class. A sufficient parameter region bounds its population threshold bias, and controls connect this accuracy result to fresh contexts and training.

4 FROM PARAMETERS TO STATISTICAL ACCURACY

  • The population error has two sources: mass assigned to the negative class and finite concentration within the positive class.Across tasks, class dominance changes near α = 1 − k; this is a task-space transition, not the unknown threshold within one context.Source: §4 · p. 4
  • Inside G, the population threshold bias is at most C/µ across tasks when µ is sufficiently large.The logarithmic correction provides sufficient class separation even at the lower endpoint ρ.Source: §4 · p. 4
  • For short contexts, the proof compares softmax pooling with the largest positive observation; for sufficiently long contexts, it controls the weighted empirical average around its population limit.Combining these arguments avoids a logarithmic dependence on N in the pretraining term.Source: §4 · p. 5
  • The fixed-parameter accuracy result is the interface to optimization: training must produce parameters in G with µ ≍ A_m,n.Once frozen, conditioning on the pretraining corpus makes the fixed-parameter bound applicable.Source: §4 · pp. 4–5

5 FROM PRETRAINING TO PARAMETER GUARANTEES

The analysis separates population optimization geometry from finite-corpus perturbations: population training captures a useful region and sharpens the attention rule, while scale-adapted gradient controls certify empirical training over a growing interval.

5.1 POPULATION CAPTURE AND INCREASING CONCENTRATION

  • Population training first balances the relative scores, then increases attention scale; its gradient identities express score changes through attention-weighted covariance and variance.Covθ and Varθ are defined under the distribution tilted by the attention weights.Source: §5.1 · p. 5
  • Population gradient flow enters G in finite time and remains there, after which the concentration parameter has positive radial drift and optimization sharpens the learned softmax rule.Before capture, the concentration parameter can decrease, but by at most a fixed multiplicative factor.Source: §5.1 · p. 5
  • For sufficiently small fixed step sizes, population gradient descent reaches the same scale-dependent funnel, and a post-entry one-step barrier replaces continued pointwise trajectory tracking.Before robust entry, the discrete trajectory shadows population flow only for a fixed physical time.Source: §5.1 · p. 6

5.2 FINITE PRETRAINING AND THE SHRINKING GRADIENT SIGNAL

  • Scale-adapted gradient-error controls preserve radial progress, inward boundary motion, and step-length bounds, certifying empirical descent within G over a deterministic interval on the reused corpus.The proof uses overlapping parameter ranges and a one-step barrier rather than tracking the empirical trajectory over the full growing horizon.Source: §5.2 · p. 6
  • The certified upper scale marks the reach of the present estimates, while the passage allows that further empirical training may improve achievable accuracy.The proof follows prediction-relevant properties over a growing horizon rather than tracking the full trajectory.Source: §5.2 · p. 6

6 EXPERIMENTS

The experiments probe finite-pretraining controls, frozen transfer, and population sharpening. The transfer errors decline with larger balanced pretraining, while the population trajectories exhibit the predicted post-capture scaling.

Finite pretraining and frozen transfer

  • At N = 4096, median-across-corpora 95th-percentile error falls from 0.160 to 0.0272 as balanced pretraining grows from m = n = 64 to m = n = 512, consistent with Theorem 3.1.The same checkpoint per corpus is frozen and reused across fresh-context sizes; the supplementary axis swaps give nearly identical errors.Source: Figure 1 · p. 7, §6 · p. 7

Population dynamics and finite-step robustness

  • After capture, a linear fit of µ^4 against flow time has R2 = 0.9994, and the tested prediction error is at most 1.51/µ(t).At µ = 40, the rescaled endpoint coordinate is −1.10 and µ^3 ˙µ = 0.522.Source: Figure 2 · p. 7, §6 · pp. 7–8

Finite pretraining and frozen transfer

  • Across 210 runs, all empirical radii stay within [0.7, 1.3] times their population references, and 209 of 210 runs meet the 3/Aproxy prediction-error control at every prescribed checkpoint.The theoretical step-size threshold and comparison constants are existential, so practical stopping-rule calibration is outside this experiment.Source: §6 · p. 7, Figure 1 · p. 7

Population dynamics and finite-step robustness

  • For population gradient descent over the plotted range µ ≥ 14, the relative radial increment is below 0.4%, and (µ^4 s)/γ is nearly independent of γ.The tested step sizes are γ ∈ {50, 100, 200, 400}.Source: §6 · p. 8

7 RELATED WORK

Related work studies in-context learning as computation, statistical estimation, and training dynamics, but the cited threads leave distinct questions about optimization and finite-corpus transfer. This paper connects finite-corpus gradient training to threshold-estimation guarantees in a one-head setting.

Algorithmic and statistical accounts of ICL

  • Prior ICL work characterizes computations and statistical solutions, while gradient training of transformer weights requires separate analysis.The cited work includes regression algorithms, constrained pretraining ERM, latent-concept inference, and attention-based posterior prediction.Source: §7 · p. 8

Learned statistical estimators and tabular priors

  • Amortized and transformer-based estimators cover posterior prediction, density and parameter estimation, but full query-label objectives and multidimensional structure remain outside this paper’s scope.The paper analyzes optimization and finite-corpus transfer for a threshold estimator.Source: §7 · p. 8

Training dynamics and invariant structures

  • Existing training-dynamics analyses study gradient dominance, trajectory phases, and invariant structures; this framework combines regional capture, inward drift, increasing resolution, and scale-adapted finite-corpus estimates.The combination links gradient training to threshold-estimation guarantees.Source: §7 · p. 8

Boundary estimation and attention resolution

  • Prior boundary-estimation rates depend on whether boundaries are directly observed or indirect and high-dimensional; this work instead connects constant-step descent to threshold estimation through relative gradient control.Full query-label objectives and multidimensional structure are outside the paper’s scope.Source: §7 · p. 8

8 DISCUSSION

The discussion compares the one-head estimator with the Bayes benchmark, describes what tied opposite orientations could achieve, and delineates the model’s boundary and scope limits.

8.1 BAYES BENCHMARK AND SYMMETRIZATION

  • The one-head rule attains Bayes-optimal rates, but its asymptotic absolute-risk and squared-risk constants are two and four times the Bayes constants, respectively.The learned softmax estimator inherits these rates when its finite-pretraining resolution term is no larger than the fresh-context term.Source: §8.1 · p. 9
  • With both classes resolved and endpoint errors o(N^-1), the tied opposite-orientation rule attains the leading Bayes constant; its training and finite-pretraining guarantees remain open.The result concerns statistical representation, while the optimization and finite-pretraining theorems cover the one-head objective.Source: §8.1 · p. 9

8.2 BOUNDARY COVERAGE

  • For thresholds in [ρ, ℓ], the one-head predictor is uniformly accurate through both endpoints; full-interval transfer requires contexts large relative to the frozen concentration scale.The fixed-margin condition in the all-N transfer theorem addresses short contexts that may not resolve weak score separation near ρ.Source: §8.2 · p. 9
  • Extending tasks to [0, 1] exposes a one-head obstruction: no single fixed orientation in this score family is uniformly consistent across the full interval.At fixed k, the positive orientation’s prediction at α=0 tends to 1 as µ grows, while the negative orientation’s prediction at α=1 tends to 0.Source: §8.2 · p. 9

Appendix A.5.3 gives the proof of Proposition 8.1.

  • The theory assumes a fixed split feature, noiseless labels, uniform design, and large-resolution initialization; small initialization, noisy or nonuniform settings, learned features, and sharp finite-pretraining limits remain open.Appendix H explores label noise and Beta designs, but the listed extensions remain unresolved in the stated theory.Source: Appendix A.5.3 · p. 9

A.2 DETERMINISTIC TRAINING-TIME WINDOW

This appendix establishes a deterministic late-training window for finite pretraining, tracing trajectories from the outer region through the dynamic band and into a prediction-good tube where prediction error and radial growth are controlled. It develops the barriers and pre-capture estimates supporting that analysis, transfers the frozen checkpoint to fresh-task estimation, and identifies the one-head model’s order-one error at a physical endpoint.

A.2 DETERMINISTIC TRAINING-TIME WINDOW

  • With high probability, a deterministic nonempty training-time window supports checkpoints at scale Θ(A_m,n) for order A_m,n^4/γ post-entry steps.Its endpoints depend only on pretraining sizes, the step size, and fixed model constants; the window is defined before observing the empirical trajectory.Source: §A.2 · p. 14

A.3 CHECKPOINT-CONDITIONED FRESH-TASK GUARANTEE

  • The checkpoint may be any deterministic or pretraining-measurable time in the certified window, and fresh-query classification inherits the same Õ((m ∧ n)^-1 + N^-1) rate.The checkpoint is frozen independently of the fresh-context size and observations.Source: §A.3 · pp. 14–15

A.4 ASSEMBLY OF THE MAIN LEARNED-ESTIMATION THEOREM

  • The learned estimator achieves error Õ((m ∧ n)^-1 + N^-1) by assembling finite-pretraining scale bounds with checkpoint-conditioned fresh-task transfer.The checkpoint is selected using pretraining information before the fresh threshold, context size, confidence parameter, or observations are revealed.Source: §A.4 · p. 15, §A.3 · p. 15

A.5.1 GRADIENT IDENTITIES

  • The population-risk gradient follows from exponential-family identities: ∂λF = Covθ(X, Y | α) and ∂μF = Varθ(X | α).These identities yield the claimed gradient formula after differentiating the population risk.Source: §A.5.1 · p. 15

A.5.2 FINITE-PRETRAINING SUMMARY

  • At every checkpoint in the late window, the finite-pretraining summary gives both asserted bounds and Θ(A_m,n^4/γ) post-entry steps on the theorem’s event.The late subinterval lies within the interval where the trajectory enters and persists in G.Source: §A.5.2 · p. 15

A.5.3 ONE-HEAD BOUNDARY OBSTRUCTION

  • Each fixed one-head orientation has order-one error at one physical endpoint, ruling out uniform consistency over [0, 1].The obstruction follows by taking μ → ∞ at α = 0 or α = 1.Source: §A.5.3 · p. 16

B.1 TWO-COMPONENT STRUCTURAL REDUCTION

  • The two-component representation makes k = λ/μ the transition-location coordinate and μ the concentration scale; their dynamics govern the attention transition across tasks.The component mean separation satisfies d_μ(α) = 1 − α + O(e^-ρμ) on the interior task interval.Source: §B.1 · pp. 16–17, §B.2 · p. 17

B.3 WELL-POSEDNESS OF THE POPULATION FLOW

C.1.1 OUTER CORRECTION AND DYNAMIC-BAND ENTRY

  • Outside the dynamic band, the ratio is corrected toward it: above Uband(µ), µ increases, while for κ ≤ k < Lband(µ), k increases and µ decreases, yielding finite-time band entry.The correction bounds hold for sufficiently large µ, with constants depending on ρ, Cband, and, in the lower region, κ.Source: §C.1.1 · pp. 20–21

C.1.2 TRANSPORT THROUGH THE DYNAMIC BAND

  • In the lower boundary layer, k increases at rate at least clow,η/µ^3 while µ decreases, yielding finite-time entrance into the fixed interior strip unless µ falls below its threshold first.The entrance-time bound is at most µ(t0)^3(ρ + η − k(t0))/clow,η for the stopped trajectory.Source: §C.1.2 · pp. 22–24
  • In the fixed interior strip, the slingshot drift has dot k ≥ cint,η µ^2, dot µ < 0, and dot λ > 0, so the trajectory traverses the strip in finite time unless µ reaches its threshold first.The strip requires k ∈ [ρ + η, ℓ − η] and sufficiently large µ.Source: §C.1.2 · p. 25
  • The interior expansion and positive drift extend toward the upper ratio rail when the transition point remains at least β log µ/µ from the endpoint, with dot k ≥ cη1/µ^2 and dot µ < 0.This extension holds for sufficiently large µ, with the threshold depending on η1 and β.Source: §C.1.2 · p. 26
  • For sufficiently large µ, the upper rail tube Tη(µ) = [ℓ − η, Uband(µ)] is forward-invariant until µ reaches its tube threshold.The claim applies when the trajectory starts inside the tube and η < ∆/4.Source: §C.1.2 · p. 26
  • A trajectory starting in the dynamic band remains trapped there until the stated tube interval ends.At the lower boundary the ratio moves inward, and at the upper moving boundary it also moves inward.Source: §C.1.2 · p. 27

C.1.3 LOGARITHMIC ENDPOINT DYNAMICS

  • Above the logarithmic rail k ≥ ℓ + (log µ + c)/µ, prediction bias is uniformly O(µ^-1) for sufficiently large µ.The logarithmic prediction-good tube is forward invariant while µ stays above its threshold.Source: §C.1.3 · pp. 27, 32
  • Endpoint barrier and one-sided approach estimates control trajectories below the rail, while regional relative-loss estimates support preserving the large-parameter regime before capture.The pre-capture estimates apply under the stated conditions, including a ratio bounded away from zero and sufficiently large initial µ.Source: §C.1.3 · pp. 31–34
  • From the dynamic band below the prediction rail, the ratio reaches the logarithmic tube in finite time unless µ first falls to its exit threshold.Once captured, it remains in the prediction-good tube while µ remains above the tube threshold.Source: §C.1.3 · p. 32
  • Within the logarithmic tube, radial drift is bounded above and below by positive constants after scaling by µ^3, yielding fourth-power growth bounds.The uniform radial expansion follows from positivity of the endpoint radial drift for c ≥ c0.Source: §C.1.3 · p. 33

C.2 COMPLETION OF THE PROOF

The proof establishes finite-time entry into and persistence within the good region, then derives the resulting prediction and risk rates. It also selects a post-capture interval with positive barrier margins for the population-gradient-descent analysis.

C.2 COMPLETION OF THE PROOF

  • Finite-time capture occurs with the radius bounded below by a constant multiple of its initialization, and the trajectory remains in the good region thereafter.The regional transport results reach the capture region; after capture, side-rail and radial-exit arguments ensure persistence.Source: §C.2 · p. 35
  • After capture, the radius grows on the fourth-power scale, yielding the stated prediction and risk rates while both attention parameters diverge.The proof uses a uniform prediction bound of order 1/µ and a risk bound of order 1/µ².Source: §C.2 · p. 35

D POPULATION GRADIENT DESCENT

  • A deterministic time after capture is chosen so radial and side-barrier margins persist over a positive-length interval.This interval is used to initialize the population-gradient-descent analysis.Source: D POPULATION GRADIENT DESCENT · p. 35

D.1 REGULARITY AND COMPACT EULER TRACKING

These sections establish regularity and approximation results for population dynamics, discrete gradient descent, and empirical vector fields. Together, they provide conditions for tracking the population trajectory and controlling finite-sample deviations.

D.1 REGULARITY AND COMPACT EULER TRACKING

  • The population vector field is globally Lipschitz, and its speed in the logarithmic tube is bounded at the scale needed for the analysis.The stated regularity is uniform in the threshold parameter.Source: §D.1 · pp. 35–36

D.2 FINITE-HORIZON EULER SHADOWING

  • On a compact time interval, Euler iterates track the population flow with an error bounded by a constant times the step size.The bound holds for sufficiently small step size while the reference path remains within a compact set with positive clearance.Source: §D.2 · p. 36

D.3 ONE-STEP BARRIER STABILITY

  • For sufficiently small steps, updates cannot cross the tube’s side boundaries, and discrete population gradient descent eventually enters the tube and remains there.Discrete tube invariance follows from one-step barrier stability; the population-GD result applies under Assumption 2.1.Source: §D.3 · p. 37, §D.4 · p. 38

D.4 POPULATION-GD REFERENCE TRAJECTORY

  • After entry, population-GD iterates satisfy fourth-power bounds, from which prediction and risk bounds follow.The prediction bound follows from Lemma C.13, and the risk bound from squaring and averaging it over α.Source: §D.4 · p. 38

E.1 BASIC EMPIRICAL-PROCESS TOOLS

  • The empirical-process tools include uniform one-task gradient regularity and moment and deviation bounds for self-normalized empirical averages.The cited moment bound gives E[Z̄_N²] ≤ C{v⁴/N⁴ + b⁶v/N⁷} under the stated boundedness and variance conditions.Source: §E.1 · pp. 39–41

E.2 COMPACT VECTOR-FIELD APPROXIMATION

  • On compact parameter sets, finite-context bias and finite-task fluctuations are controlled separately to approximate the population vector field.The compact vector-field approximation combines the finite-context bias and finite-task concentration results.Source: §E.2 · pp. 41–42
  • The compact vector-field approximation holds uniformly over the compact set with probability at least 1 − δ.For fixed R, the error has the order stated in Proposition E.8; the supplied passage does not preserve its full expression.Source: §E.2 · p. 42

E.3 ONE-TASK ESTIMATES IN THE LOGARITHMIC TUBE

This section develops one-task estimates and uniform shellwise concentration to analyze empirical gradient descent through a two-stage perturbation argument, establishing compact tracking before capture and discrete continuation across enlarged shells. It also gives a sufficient pretraining regime with a statistical radius based on the effective task and example count, and defines a deterministic post-capture training-time window.

E.3 ONE-TASK ESTIMATES IN THE LOGARITHMIC TUBE

  • For n ≥ Cµ, one-task estimates bound gradient moments and bias using centered influence-function envelopes under the logarithmic-tube weighting.The influence functions are Pα-centered, and their envelopes support the one-task gradient moment and bias bounds.Source: §E.3 · pp. 43–44

E.4 UNIFORM SHELLWISE CONCENTRATION

  • Uniform shellwise concentration controls denominators, directional fields, and speed, supporting scale-adapted error estimates across each shell.Theorem E.16 gives the directional approximation, and Proposition E.17 gives the speed approximation under sufficient context and confidence conditions.Source: §E.4 · pp. 46–47

F PROOF OF THE FINITE-PRETRAINING THEOREM

  • The proof of the finite-pretraining theorem uses population gradient descent as its reference dynamics for analyzing empirical gradient descent.The supplied overview divides the proof into pre-capture compact tracking and post-capture shell estimates.Source: F PROOF OF THE FINITE-PRETRAINING… · p. 48, §F.1 · p. 48

F.1 TWO-STAGE PERTURBATION PRINCIPLE

  • Empirical gradient descent is analyzed in two stages: compact-set tracking of population descent before capture, followed by scale-adapted shell estimates after capture.The post-capture estimates preserve radial drift, barrier margins, and the speed bound needed for discrete no-overshoot.Source: §F.1 · p. 48

F.2 REGIME I: COMPACT TRACKING OF POPULATION GD

  • Before capture, empirical and population gradient descent track on a common compact set for fixed physical time, using compact approximation and discrete Grönwall control.The compact set contains the population trajectory through its entry index with positive clearance.Source: §F.1 · p. 48, §F.2 · p. 48

F.3 REGIME II: DISCRETE CONTINUATION WITH OVERLAPPING STATISTICAL SHELLS

  • The empirical trajectory continues through enlarged shells under the shell event, preserving tube membership through the landing step and retaining the shell-resolution statistical order.The continuation argument uses inward margins and speed bounds to rule out side-rail crossings; the enlarged shell does not change the statistical order.Source: §F.3 · pp. 49–50

F.4 STATISTICAL RADIUS AND A SUFFICIENT PRETRAINING REGIME

  • The sufficient-pretraining regime begins by choosing a comparison radius containing K0 and fixing comparison constants.The supplied passage defines these setup quantities but does not include the resulting regime or bound.Source: §F.4 · p. 51
  • The sufficient pretraining regime sets the statistical radius proportional to (m ∧ n)/log(Cstat mn) when that radius is at least 2µ*.The proposition states conditions for certifying shells up to this radius.Source: §F.4 · p. 51

F.5 DETERMINISTIC TRAINING-TIME WINDOW

  • The deterministic training-time window uses a post-capture clock and defines the corresponding statistical time from the entry index.The clock starts at T0 at entry index s0, advancing by γ per training step.Source: §F.5 · p. 52

F.6 COMPLETION OF THE PROOF

The proof propagates empirical gradient descent through certified shells and transfers frozen checkpoints to fresh contexts. It gives conditional fresh-context guarantees while distinguishing bounds that hold for all context sizes from those requiring the context to resolve the attention scale.

F.6 COMPLETION OF THE PROOF

  • The empirical path propagates through each certified shell while preserving the good region, up to the terminal radius M_stat = 2R_Jstat.The handoff property carries each landing iterate into the next shell.Source: §F.6 · p. 53

G FRESH-CONTEXT TRANSFER FROM FROZEN CHECKPOINTS

  • Checkpoints are selected using pretraining information and frozen before the fresh threshold, context size, confidence level, or observations are revealed.Fresh transfer is formulated for checkpoints selected from a later deterministic subwindow.Source: G FRESH-CONTEXT TRANSFER FROM FROZEN… · p. 53

G.1 FIXED-PARAMETER FRESH-CONTEXT BOUNDS

  • The fixed-parameter fresh-context proof combines an all-context-size order-statistic bound with a sharper approximation when the context resolves the frozen scale.The first argument uses a fixed positive threshold margin; the second uses self-normalized concentration.Source: §G.1 · p. 54

G.1.1 DETERMINISTIC SOFTMAX GEOMETRY

  • The order-statistic bound applies to interior thresholds α ∈ [ρ + η, ℓ] without a lower bound on N, whereas the full-interval bound requires the context to resolve µ.The large-context self-normalized result covers [ρ, ℓ] but requires fresh-context size sufficient to resolve the scale µ.Source: §G.1.1 · pp. 55–56

G.1.2 PROOF OF THE MAIN-TEXT FIXED-ESTIMATOR PROPOSITION

  • Conditioning on pretraining makes a measurable checkpoint fixed, so a uniform fixed-parameter fresh-context guarantee transfers without asserting one event simultaneous over all checkpoints.The conditioning lemma is the only probabilistic step needed for this transfer.Source: §G.1.2 · p. 57

G.2 FREEZE-THEN-TRANSFER THEOREM

  • For any deterministic or pretraining-measurable checkpoint in the specified training-time window, the fresh-context event has conditional probability at least 1 − 4e^-u for fixed α, every N ≥ 1, and u ≥ 1.The checkpoint is fixed conditional on pretraining information; the training theorem places it in the good region with its radius between constant multiples of A_m,n.Source: §G.2 · p. 57, §G.1.2 · p. 57

G.3 FULL CLOSED INTERVAL ONCE THE FRESH CONTEXT RESOLVES THE SCALE

The certified training window supports consistency at the lower endpoint when the learned scale and fresh-context size diverge under the stated relation, but a uniform all-context-length guarantee for one frozen raw head remains obstructed. The endpoint competition is presented as a heuristic obstruction, alongside finite-step and reproducibility diagnostics.

G.3 FULL CLOSED INTERVAL ONCE THE FRESH CONTEXT RESOLVES THE SCALE

  • The frozen estimator is consistent at the lower endpoint when the certified scale and fresh-context size diverge under the stated relation.The guarantee applies within the statistical-scale training-time window.Source: §G.3 · p. 58
  • At intermediate context sizes, random endpoint spacings can erase the learned logarithmic advantage and let the negative class dominate; this is a heuristic, not a formal impossibility result.The passage says a formal impossibility claim would require a separate lower-bound proposition.Source: §G.3 · p. 58
  • A joint-over-task result may exclude a boundary layer, but its probability must enter the final failure probability or risk; reflection also requires control of finite-context fluctuations.Reflection can repair physical-boundary coverage, but does not by itself resolve routed finite-context fluctuations.Source: §G.3 · p. 58
  • Population GD preserves the normalized fourth-power increment across step sizes, while a common-shell grid separates task-level from within-task sampling effects.The continuation panel is reported in Figure 1.Source: Figure 3 · p. 59

H.1 REPRODUCIBILITY PROTOCOL

  • Finite-pretraining iteration indices are fixed from the population-GD reference path before corpus sampling, and one checkpoint per corpus is reused across fresh thresholds and context sizes.Population calculations use analytic feature-coordinate integrals and 160-point Gauss–Legendre quadrature; empirical experiments use exact full-batch gradients.Source: §H.1 · p. 59
  • The reported worst-grid pointwise 95th-percentile error takes a percentile at each tested threshold, then maximizes over the grid; it does not provide simultaneous continuum coverage.Code availability is also reported in the passage.Source: §H.1 · p. 59

H.2 DETAILED FINITE-STEP AND TRANSFER DIAGNOSTICS

The diagnostics examine finite-corpus transfer, initialization sensitivity, and noise robustness in the one-head model. Large-context transfer errors track learned resolution, while initialization and label noise expose important boundaries on the observed behavior.

H.2 DETAILED FINITE-STEP AND TRANSFER DIAGNOSTICS

  • A stricter operational tracking tube holds throughout the window in 104 of 210 runs, while the theorem leaves its finite-radius threshold and comparison constants existential.An auxiliary axis sweep meets the operational tracking criterion up to µref = 60 for all seven tested pairs.Source: §H.2 · p. 59
  • At N = 4096, axis-swap errors are 0.128 and 0.124 for (128, 512) and (512, 128), and 0.0592 and 0.0586 for (256, 512) and (512, 256).After fitting an additive floor, pre-floor excess-error slopes against N range from −0.87 to −0.94 across the four balanced corpus sizes.Source: §H.2 · pp. 59–60, Figure 4 · p. 60
  • Two nonuniform designs remain close to the uniform baseline, whereas one-percent global label corruption changes behavior substantially in the exploratory scope check.The designs are Beta(2, 2) and Beta(1/2, 1/2); the checks use six independently sampled pretraining corpora per condition.Source: Figure 5 · p. 60, §H.2 · p. 60
  • Across tested task interiors and initial states, µ supα |F − α| stabilizes, and numerical discrepancies are negligible at the quadrature order used in the main experiment.The checks separately assess population robustness and numerical quadrature and ODE convergence.Source: §H.2 · p. 60, Figure 6 · p. 61
  • Exactly zero initialization is stationary, nearby starts escape slowly, and fixed-budget performance depends strongly on initial radius.The asymptotic behavior of every nonzero small initialization remains unresolved.Source: §H.2 · p. 60, Figure 7 · p. 61

H.3 EXPLORATORY LABEL-NOISE

  • With bµ = 40 and bk = 0.86472 fixed, clean and threshold-localized errors approach checkpoint-dependent population floors, while one-percent global Massart flips produce order-one error.The same frozen checkpoint is reused across context sizes; it is selected before observing the noisy context.Source: §H.3 · p. 61, Figure 8 · p. 62
  • Along the resolution-matched sequence, clean, localized, and Tsybakov errors decrease approximately as N^-1 on the safe interior, but the fixed one-head orientation retains its physical-boundary obstruction.This sequence changes model parameters with N, unlike the frozen-checkpoint experiment.Source: §H.3 · p. 61, Figure 9 · p. 62

Related papers

Loading 2610.07074v1…