PaperlayerView exact revision on arXiv

Source-linked AI summary

An Active-Bottleneck Mechanism for Weak-to-Strong Generalization

Mohammad Zeinalpour, Amir Najafi

arXiv:2609.33835v1stat.MLcs.AIcs.LG

TL;DR

The paper asks when a student trained only on a teacher’s predictions can outperform that teacher under fully converged, ridgeless learning. It analyzes matched-class linear regression and a wider random-feature student, finding that finite sample size or width can act as an active bottleneck that filters teacher error.

  • Problem

    The paper asks whether W2SG requires a student advantage such as greater capacity, different regularization, or early stopping.

    Source: §1 · p. 2

  • Method

    The paper analyzes two-stage ridgeless learning, first with matched-class linear regression and then with random features where the student has greater width.

    Source: §3 · p. 5, §7 · p. 16

  • Results

    Explicit thresholds characterize when improvement occurs: by pseudo-sample size m in matched-class regression and by m or student width N_S in the random-feature setting.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The active-bottleneck principle states that the scarcer of m and N_S controls teacher-error filtering, while the other resource reduces estimation noise.

    Source: §1 · p. 3

    The paper proves its phase-transition results for power-law and two-block covariance models, and only conjectures similar behavior beyond those families.

    Source: §7 · p. 16

Abstract

from arXiv · show

Weak-to-strong generalization (W2SG) occurs when a student trained on a teacher's predictions outperforms that teacher. We study when this happens under fully converged, ridgeless two-stage learning, with no early stopping, no explicit regularization, and no assumption that the student is more expressive than the teacher. In two-stage linear regression, a teacher is fit from $n$ labeled examples and a student is trained solely on the teacher's predictions on $m$ fresh, unlabeled inputs. Although both stages share the same hypothesis class and the same training rule, we show that the student outperforms the teacher exactly when $m$ lies in an explicit intermediate range: too few pseudo-labels leave the student without enough signal, too many let it inherit the teacher's noise. Under power-law covariance, we derive this range in closed form as a function of the spectral decay and noise level, including regimes where the improving region splits into two disjoint intervals of $m$. We then study a random-feature model in which the student has strictly more features than the teacher, and identify two regimes, again given by explicit thresholds: one where improvement occurs only for $m$ in a bounded interval, and one where it occurs only once the student width $N_S$ exceeds an explicit threshold. Both regimes are governed by a single "active-bottleneck" principle: whichever of $m$ or $N_S$ is scarcer controls how much teacher error is filtered out, while increasing the other resource only reduces estimation noise. Together, these results show that finite data and finite width can themselves regularize a two-stage learner, with no explicit mechanism doing so.

1 Introduction

The paper asks whether a student can outperform its teacher despite matched capacity, identical training rules, full convergence, and no explicit regularization. It shows that finite pseudo-sample size can enable this improvement, with an active-bottleneck principle extending the account to a wider random-feature student.

1 Introduction

  • In the matched-capacity setting, finite m constrains the student to an m-dimensional parameter subspace that can retain more useful signal than teacher noise, or the reverse.Which outcome occurs depends on m.Source: §1 · p. 2
  • With strictly more random features than the teacher, improvement occurs either only within a bounded m interval or only above an explicit student-width threshold.Which regime applies depends on the parameters; the scarce resource among m and N_S governs teacher-error filtering, while the other reduces estimation noise.Source: §1 · p. 3
  • Large-scale distillation studies report that increasing teacher-labeled examples can eventually collapse W2SG even when a large capacity gap remains.The paper relates this observed non-monotonicity to its theoretical account.Source: §1 · p. 3
  • In ridgeless linear regression with identical hypothesis classes, a fully converged student can strictly outperform its teacher without early stopping or explicit regularization.The student is trained only on the teacher’s predictions.Source: §1 · p. 3
  • Under power-law covariance, explicit critical pseudo-sample sizes separate improvement from non-improvement, and the improving region can be a single interval, all of (0, p), or two disjoint intervals.The thresholds are characterized both for fixed m and as limiting ratios m/p.Source: §1 · pp. 2–3

2 Related Work

The paper contrasts its resource-based account with explanations built on predictor properties, training dynamics, or stage asymmetries. It positions its contribution as an explicit characterization of when finite resources enable improvement under converged, ridgeless training.

2 Related Work

  • Prior work often guarantees improvement through predictor-pair properties such as misfit, coverage, alignment, or latent-concept transfer, whereas this paper studies how training resources determine improvement.The paper does not assume special properties of the resulting predictors.Source: §2 · p. 3
  • Some prior explanations rely on early stopping to suppress teacher error, while this paper places the constraint on the data or features supplied to a student that always converges.The cited dynamics work includes early-stopped random-feature regression and finite-time logistic training.Source: §2 · p. 3
  • Other analyses attribute improvement to asymmetric regularization, representation, or nonlinearity; Case I instead uses the same hypothesis class and ridgeless rule for teacher and student.Case II deliberately adds a student capacity advantage to test whether the resource-based mechanism persists.Source: §2 · p. 4
  • Compared with related resource-based analyses, this work benchmarks improvement against the teacher actually produced in Stage I and analyzes the joint role of pseudo-sample size and student width.On the sample-bottleneck branch, improvement holds only within a bounded m interval; other cited approaches use different benchmarks, ridge, or built-in covariate asymmetry.Source: §2 · p. 4
  • The paper argues that overparameterization is not intrinsically harmful: under power-law covariance, its effect depends on where discarded information lies spectrally.W2SG is positive throughout the overparameterized range for α ≤ 2, and on an explicit sub-range for α > 2.Source: §2 · p. 4
  • LLM-scale distillation shows a non-monotone effect of pseudo-label count, while the paper’s analysis supplies a mechanism and explicit thresholds for related behavior.The empirical study reports no mechanism or explicit threshold.Source: §2 · p. 4
  • The paper identifies its novelty as combining exact convergence without ridge, no capacity asymmetry in Case I, and explicit resource thresholds separating improvement from its absence.It frames an active resource constraint as a way finite resources can regularize the student.Source: §2 · p. 4

3 Preliminaries

The preliminaries define a two-stage pipeline in which a teacher is learned from labeled data and a student from fresh inputs labeled by that teacher. They define W2SG by comparing the predictors’ expected population risks against the target.

3 Preliminaries

  • A fixed target coefficient and input distribution define the prediction problem, with label noise independent of the input and specified separately for each setting.The target coefficient may itself be random, independently of the other quantities.Source: §3 · p. 5
  • Stage I fits a teacher from n labeled observations using a learning rule, with its hypothesis class optionally depending on independent auxiliary randomness.A random-feature embedding is given as an example of such randomness.Source: §3 · p. 5
  • Stage II trains a student without its own labeled data, using m fresh unlabeled inputs labeled by the teacher’s predictions.The student sees only those predictions, not the target coefficient, label noise, or original labeled dataset.Source: §3 · p. 5
  • The paper defines W2SG as positive gain when the student’s expected population risk is strictly lower than the teacher’s.The risks average over the randomness entering each predictor, including Stage-II inputs for the student.Source: §3 · p. 5

4 Case I: Ridgeless Regression

Case I shows that identical ridgeless teacher and student models can exhibit weak-to-strong generalization through the finite pseudo-sample bottleneck alone. Under power-law covariance, the outcome depends on the scaling of m, spectral decay, and noise: improvement may hold throughout the proportional regime, require a threshold, or occur in disjoint ranges.

4.1 Model and Training Procedure

  • The teacher and student share a linear hypothesis class and minimum-norm training rule; Stage I uses Gaussian covariates and label noise, while Stage II uses independent fresh inputs and noiseless teacher predictions.The target coefficient distribution is specified through two moments; the studied risks and W2SG gain depend on its second moment alone.Source: §4 · p. 6, §4.1 · p. 6

4.2 Stage-II Projection Geometry

  • When m ≥ p, the student reproduces the teacher almost surely, so W2SG requires m < p in Case I.With Gaussian design, the Stage-II projector is the identity once m ≥ p.Source: §4.2 · pp. 6–7
  • When m < p, Stage II projects the teacher onto an m-dimensional random subspace, retaining both useful signal and teacher error as m increases.Covariance favors retaining higher-variance directions, but the expected retention of every direction remains strictly between zero and one.Source: §4.2 · p. 7

4.3 Power-Law Covariance and Two Regimes for m

  • Under isotropic covariance, projection attenuates every direction equally and W2SG cannot occur without explicit ridge; the analysis therefore uses a power-law spectrum.The authors conjecture similar qualitative behavior for other sufficiently graded non-isotropic spectra, but not for every non-isotropic covariance.Source: §4.3 · p. 7
  • The analysis separates fixed-sample and proportional m regimes because thresholds of order one and Θ(p) are visible at different scales.The stated phase-diagram analysis restricts the spectral-decay range, and behavior beyond that range is only partially resolved.Source: §4.3 · p. 7

4.4 Asymptotic W2SG Phase Diagram

  • For 1 < α ≤ 2, W2SG holds for every proportional-regime pseudo-sample size m < p; for α > 2, it requires crossing a critical ratio.The critical ratio depends on α, the Stage-I sample ratio, and noise level.Source: §4.4 · p. 8
  • For 2 < α ≤ 5/2 and sufficiently small noise, no fixed m produces W2SG, although a Θ(p)-scale pseudo-sample size does.The fixed-sample and proportional regimes can therefore yield qualitatively different outcomes.Source: §4.4 · p. 8
  • Above the stated noise thresholds, W2SG can occur in separate windows near m = O(1) and m = Θ(p), with no critical m in the intermediate scaling.The reported picture covers fixed, proportional, and intermediate scalings of m.Source: §4.4 · p. 8

5 Case II: The Random-Feature Model

Case II asks whether finite Stage-II resources can produce weak-to-strong generalization when the student has a richer random-feature representation than the teacher. Its phase diagram shows that whichever of pseudo-sample size m or student width N_S is scarcer controls teacher-error filtering, while the other resource reduces estimation noise.

5.1 Model and Two-Stage Training

  • The random-feature setting gives the bottleneck mechanism a second resource, student width, alongside pseudo-sample size.The student and teacher have independent random-feature maps, and the student’s width can exceed the teacher’s.Source: §5.1 · p. 9, §5 · p. 9

5.2 Two Stage-II Resources, Three Regimes

  • When neither m nor N_S is scarce relative to p, the student reproduces the teacher and W2SG is zero.Positive W2SG requires at least one Stage-II resource to be scarce relative to p.Source: §5.2 · p. 10

5.3 Asymptotic W2SG Phase Diagram

  • In the sample-bottleneck regime, W2SG requires m to lie in a bounded window; in the feature-bottleneck regime, it occurs once m clears a threshold.If either resource falls below the signal fraction, the student cannot recover the signal; the feature-bottleneck branch requires ρS − ϑ > ΛT and ρm > 2ρS(ρS − ϑ).Source: §5.3 · p. 11
  • Any positive-W2SG region requires the student to have asymptotically more random features than the teacher, but that advantage alone is insufficient.The theorem further requires the bottleneck condition on m to hold.Source: §5.3 · p. 11
  • The active bottleneck determines how much teacher error is filtered, while the non-bottleneck resource only reduces estimation noise.Theorem 5.4’s two branches hand off according to which of m and N_S is smaller relative to p.Source: §5.3 · p. 11
  • With ρm fixed, the width scan has a bounded positive interval when width is the bottleneck and a half-line when sample size is the bottleneck.The bounded interval can disappear if its discriminant is nonpositive; the sample-bottleneck branch requires m to exceed its threshold.Source: §5.3 · p. 11

6 Experimental Results

The experiments compare finite-p deterministic equivalents with Monte Carlo W2SG and the predicted asymptotic regimes across ridgeless-regression and random-feature settings. They examine phase diagrams, finite-size scaling, separation of bottleneck branches, and a negative control.

6.1.1 Phase Diagram

  • At α = 1.8 and σ2 = 0.5, the finite-p deterministic equivalent crosses zero near m = 15.25 and the empirical curve near m = 17.39, while the proportional limit stays positive throughout the displayed ratios.At σ2 = 2, W2SG is positive throughout the three displayed regimes but non-monotone at finite p.Source: §6.1.1 · p. 13
  • At α = 2.2 and σ2 = 2, finite-p W2SG has two positive regions, with deterministic-equivalent crossings at m ≈ 3.73 and m ≈ 28.38 and empirical crossings at m ≈ 2.54 and m ≈ 29.14.At σ2 = 0.5, the proportional limit becomes positive near ρm ≈ 0.0840, matching the finite-p deterministic-equivalent crossing at m ≈ 33.64; empirical crossing is m ≈ 38.36.Source: §6.1.1 · p. 13, Figure 2 · p. 13

6.1.2 Finite-Size Scaling

  • For α = 2.2 and σ2 = 0.5, increasing p at fixed m moves the curve toward a negative limit, while fixed positive m/p ratios converge to the proportional-limit prediction.The σ2 = 2 setting has the largest finite-size corrections near ρm = 0.Source: §6.1.2 · p. 14, Figure 3 · p. 14

6.2.1 Phase Diagram

  • In the random-feature phase diagram, scanning ρm at fixed ρS gives a bounded positive-W2SG interval; scanning ρS at fixed ρm gives a bounded interval followed by a positive half-line.Empirical curves track the finite-p master deterministic equivalent closely away from the interpolation boundary m = N_S.Source: §6.2.1 · pp. 14–15, Figure 4 · p. 15

6.2.2 Separation of the Two Phase Mechanisms

  • The branch-separation experiment removes the bounded positive-ρm interval while retaining a positive half-line, and its width scan retains a bounded positive interval ρS ∈ (0.252, 1).The width-scan finite-p and empirical transitions are approximately 0.294 and 0.296.Source: §6.2.2 · p. 15, Figure 5 · p. 16

6.2.3 Negative Control

  • When both Stage-II resources are below the signal threshold, deterministic-equivalent and empirical curves remain strictly below zero throughout the scanned domains.The branch constraints exclude all positive-W2SG regions in this negative control.Source: §6.2.3 · p. 15

7 Conclusion

The paper concludes that finite shared resources can produce weak-to-strong generalization without early stopping or explicit regularization, including when teacher and student have equal capacity. Its active-bottleneck account unifies the linear and random-feature settings, while the scope of the theory remains limited.

7 Conclusion

  • The mechanism requires neither early stopping nor explicit regularization, and the linear setting requires no student capacity advantage.The paper identifies a finite resource constrained to an intermediate range as sufficient for improvement.Source: §7 · p. 16
  • In both settings, the scarcer shared resource controls how much teacher error is filtered, while the nonbinding resource can be increased freely.In the equal-capacity linear setting, pseudo-sample size alone determines whether the student improves; in the random-feature setting, either pseudo-sample size or student width can be the bottleneck.Source: §7 · p. 16
  • The theory analyzes power-law and two-block covariance models; similar phase transitions for other graded non-isotropic spectra remain conjectural.Some boundary cases and the α > 1+… regime in Case I are only partially resolved.Source: §7 · p. 16
  • Extensions to nonlinear students, other resources, and large-scale settings are proposed as open directions rather than established results.The paper notes that recent empirical work points to a similar non-monotonic dependence on pseudo-sample size at large scale.Source: §7 · p. 16

AI Use Statement

This block covers an AI-use statement and appendix material developing a two-stage minimum-norm interpolation analysis. It sets out the deterministic-equivalent approach, the Gaussian linear-regression model, and the Stage-I risk analysis.

AI Use Statement

  • Generative AI assisted with literature discovery, organization, language editing, manuscript structure, and experimental-design feedback; the authors report independently verifying citations, mathematical claims, proofs, code, and results.The authors take responsibility for the final content.Source: AI Use Statement · p. 16

A Two-Stage Minimum-Norm Interpolation Theory

  • The theory studies whether a second ridgeless interpolation step can improve population risk when the student receives only fresh covariates labeled by the Stage-I teacher.The analysis derives Stage-I risk, conditions Stage-II analysis on the realized teacher, and combines the two stages through deterministic equivalents.Source: A Two-Stage Minimum-Norm Interpolation… · p. 19

A.1 Deterministic Equivalence

  • The deterministic-equivalence framework applies to proportional high-dimensional asymptotics, with additional assumptions stated for later derivations.The ridgeless regime is obtained by taking the ridge parameter to zero when appropriate.Source: §A.1 · pp. 19–20

A.2 Two-Stage Linear Regression with Gaussian Covariates

  • The model uses Gaussian covariates with population covariance Σ and independent Gaussian label noise; Stage I fits ridgeless regression to labeled data.The teacher parameter is initially treated as deterministic, with random-teacher assumptions introduced later where needed.Source: §A.2 · p. 20, §A.3 · p. 20

A.3 Stage I: Ridgeless Regression

  • The Stage-I risk has a deterministic equivalent, and when p > n the ridgeless estimator is the unique minimum-Euclidean-norm interpolator.The analysis uses deterministic equivalents for random resolvents in the proportional high-dimensional regime.Source: §A.3 · pp. 20–21

A.4 Stage II Risk Derivation (Distilled Stage-II Sample)

This section derives the Stage-II risk and its improvement over the Stage-I teacher, then analyzes how the pseudo-sample subspace and covariance spectrum shape that improvement. The power-law analysis gives proportional-regime sign behavior, while the derivation is asymptotic and rests on stated assumptions.

A.4 Stage II Risk Derivation (Distilled Stage-II Sample)

  • At the deterministic-equivalent level, the Stage-I error carried into Stage II cancels from the risk-improvement expression.The Stage-II risk is first decomposed into carried-over Stage-I error, additional interpolation error, and a cross term.Source: §A.4 · p. 22
  • The analysis provides deterministic-equivalent formulas under its spectral assumptions, but does not claim a corresponding non-asymptotic generalization bound.A related setting in Ildiz et al. [2025, Definition 3] permits distribution shift and gives a bound under additional boundedness assumptions not imposed here.Source: §A.4 · p. 25

Then the teacher-averaged Stage-I-to-Stage-II risk improvement admits the deterministic equivalent

  • The fully deterministic improvement expression assumes an isotropic random teacher independent of the design and label noise.The derivation averages the conditional Stage-II risk over remaining Stage-I randomness using weighted matrix functionals and deterministic equivalents.Source: §A.4 · p. 25, Then the teacher-averaged… · pp. 26, 28

Then the following statements hold: The expected projector satisfies

  • For m < p, Stage II retains the teacher component in an m-dimensional row space and discards its orthogonal complement; the sampled covariates set that subspace's orientation.In the covariance eigenbasis, higher-variance directions receive greater expected retention, but retention is not a hard cutoff at the top m eigenvalues.Source: Then the following statements hold:… · pp. 28–29

A.5 Proportional high-dimensional asymptotics

  • For α > 1, the proportional-limit signal-dependent contribution vanishes, while the Stage-I noise contribution remains order one.Source: §A.5 · pp. 31–32

A.5.1 Limiting Stage-I-to-Stage-II risk improvement

These results characterize proportional-limit Stage-I-to-Stage-II risk improvement through the sign of the gain across decay regimes and sampling ratios, including conditional sign crossings for α > 2 and cases where intermediate behavior remains unresolved.

A.5.1 Limiting Stage-I-to-Stage-II risk improvement

  • For σ^2 > 0, the sign of limiting improvement is determined by Jα, while zero noise instead gives zero improvement.Under the power-law covariance model, the limit depends on α, ϕT, ϕS, and σ^2 through sT, sS, γT, and γS; γS increases strictly with ϕS^-1.Source: §A.5.1 · p. 32
  • For 1 < α ≤ 2, σ^2 > 0 guarantees positive W2SG deterministic-equivalent improvement for every ϕT > 1, ϕS > 1.These conditions cover every nontrivial proportional Stage-II sampling fraction.Source: §A.5.1 · p. 38
  • For α > 2 and σ^2 > 0, sufficiently small ϕS^-1 always gives negative deterministic-equivalent improvement.Equivalently, the W2SG condition fails for sufficiently large proportional ratios p/m.Source: §A.5.1 · pp. 38–39
  • For α > 2, the near-square sign is positive when Lα(ϕT) > 0 and negative when Lα(ϕT) < 0; equality requires higher-order analysis.These sign conclusions apply for ϕS^-1 < 1 sufficiently close to 1.Source: §A.5.1 · p. 39
  • When α > 2, σ^2 > 0, and Lα(ϕT) > 0, both positive- and negative-W2SG proportional ratios exist, with at least one intervening zero.The crossing criterion combines sparse-edge negativity with positivity near the square boundary.Source: §A.5.1 · p. 39
  • For 2 < α ≤ 1 + √2, every ϕT > 1 has both positive- and negative-W2SG proportional Stage-II ratios when σ^2 > 0.In this range, Lα(ϕT) is positive for every teacher ratio.Source: §A.5.1 · p. 40
  • For α > 2, ϕT > 1, σ2 > 0, and Lα(ϕT) > 0, the proportional transition is unique: Jα > 0 for sufficiently small sS and Jα < 0 for sufficiently large sS.The proportional W2SG gain crosses zero exactly once as the Stage-II sampling fraction varies from the sparse boundary to the near-square boundary.Source: §A.5.1 · pp. 40, 43, 45
  • When Lα(ϕT) ≤ 0, the analysis proves negativity near the sparse proportional boundary but does not determine whether an intermediate sampling fraction yields positive W2SG.If Lα(ϕT) > 0, the unique negative-to-positive transition is established instead.Source: §A.5.1 · pp. 40, 46

A.6 Sparse-growing Stage-II sample size

The appendix extends deterministic-equivalent analysis to Stage-II sample sizes growing slower than dimension and to fixed sample counts, under continuation assumptions not guaranteed by the proportional theorem.

A.6 Sparse-growing Stage-II sample size

  • The sparse-growing continuation assumption is not implied by the standard proportional deterministic-equivalence theorem when m/p → 0.Stage I remains proportional and overparameterized, while Stage II has m → ∞ but m/p → 0.Source: §A.6 · p. 46
  • For growing m with m/p → 0, the limiting gain is positive when σ² > 0 and 1 < α < 2, zero at α = 2, and negative for α > 2; with σ² = 0 it is zero for every α > 1.This sparse-growing result is conditional on a deterministic-equivalent continuation assumption beyond the standard proportional theorem.Source: §A.6 · p. 46

A.7 Ultra-sparse Stage-II extensions

  • The fixed-m extension also assumes continuation of the finite-p deterministic equivalent, which the proportional theorem does not guarantee when the Stage-II sample-to-dimension ratio vanishes.The regime keeps m ≥ 1 fixed as p → ∞.Source: §A.7 · p. 48

A.7.1 Fixed Stage-II sample count

For each fixed Stage-II sample count, the appendix establishes a deterministic-equivalent W2SG gain using a unique limiting spectral parameter.

A.7.1 Fixed Stage-II sample count

  • The limiting fixed-m spectral parameter τ_m,S is uniquely determined, and the corresponding γ_m,S lies strictly between zero and one.The finite-p parameter converges to the unique solution, with γ convergence following by dominated convergence.Source: §A.7.1 · p. 48
  • The fixed-m derivation bounds the spectral weights 0 < a_i < 1 before evaluating the gain's signal and noise terms.The proof uses the fixed-point identity and dominated convergence for those terms.Source: §A.7.1 · pp. 48–49
  • For every fixed integer m ≥ 1, Theorem A.26 gives a deterministic-equivalent expression for the W2SG gain under the fixed-m continuation assumption.The proportional deterministic-equivalence theorem does not itself guarantee this fixed-m continuation.Source: §A.7.1 · p. 48

A.7.2 Analytic sign results for the fixed-m gain

This section characterizes gain signs and transitions across power-law covariance regimes, including higher-order tail behavior at the critical exponent and sufficient noise conditions that rule out positive gain. It also derives a random-feature phase diagram governed by the active Stage-II bottleneck, treating critical and finite-dimensional regimes separately.

A.7.2 Analytic sign results for the fixed-m gain

  • For every real m > 0, the fixed-point equation has a unique positive solution, and for integer m the resulting gain equals the limiting deterministic-equivalent gain.The fixed-point functions are continuously differentiable; the exact sign criterion and threshold derivative are also established for m ≥ 1.Source: §A.7.2 · p. 50, Fixed-point regularity and exact… · pp. 50–51

Fixed-point regularity and exact identities

  • The exact sign criterion factors the gain through the moment q_m and gives a derivative identity for the noise threshold wherever q_m > 0.The section also supplies explicit fixed-point bounds and large-m moment and gain asymptotics.Source: Fixed-point regularity and exact… · pp. 51–52

The initial noise threshold

  • For 1 < α ≤ 5/2, q_1 > 0, and the initial critical noise level is defined by G_1 = 0 rather than by a uniform-in-m sign condition.For 1 < α < 2, an explicit sufficient condition also guarantees positive gain.Source: The initial noise threshold · p. 53
  • For 1 < α < 2 and noise below the initial critical level, the gain is negative near m = 1 and positive for sufficiently large m, but this alone does not ensure a unique zero.For sufficiently small positive noise, there is instead a unique transition m* > 1, with negative gain below it and positive gain above it.Source: The initial noise threshold · pp. 53–54
  • For α = 2, the gain is eventually negative below the critical tail-noise threshold and eventually positive above it; at the threshold, it is also eventually positive.The leading large-m limit vanishes at α = 2, so the tail sign requires a higher-order expansion.Source: The initial noise threshold · pp. 54, 56
  • Lemma A.40 states an exact hyperbolic representation at α = 2; in particular, q_m > 0 for every m > 0.Source: The initial noise threshold · p. 55
  • At α = 2, the proportional-regime gain stays positive for positive sampling fraction and noise, while the sparse-growing limit vanishes and fixed-m signs depend on a higher-order balance.In the fixed-m continuation, large-m behavior is controlled by the critical tail-noise level.Source: The initial noise threshold · p. 57
  • For α > 2, the gain is eventually negative regardless of noise, and a sufficient noise threshold makes it nonpositive for every integer m.Below that threshold it is strictly negative for every m; at the endpoint, a zero is possible only at m = 1, and the bound is not claimed to be largest possible.Source: The initial noise threshold · pp. 57–58

B Two-Stage Random-Feature Interpolation Theory

  • The random-feature phase diagram separates sample-bottleneck, feature-bottleneck, and no-bottleneck branches, with gain signs determined by Stage-I mismatch and the active bottleneck’s position relative to signal dimension.The analysis also treats excluded critical regimes: a finite-dimensional interpolation boundary and a √p window, where joint teacher–student criticality can leave positive gain only on the feature-bottleneck branch.Source: B Two-Stage Random-Feature… · p. 59

B.1 Deterministic equivalence

  • The random-feature ridgeless-regression analysis uses a two-point structured-Wishart deterministic equivalent introduced in Atanasov et al. [2025].Under the stated boundedness and denominator conditions, its normalized-trace equivalent remains valid after expectation over the feature matrix.Source: §B.1 · p. 59

B.2 Defining Random-Feature Model

  • The model uses independent random linear feature maps for teacher and student; Stage I is population-risk minimization, while Stage II uses ridgeless least squares on fresh teacher-pseudolabeled samples.The student feature dimension and Stage-II sample count jointly constrain the student, and no direct interpolation constraint is imposed.Source: §B.2 · p. 60

B.3 Theory of DE for stage I

The Stage-I analysis connects an exact projection representation of the teacher estimator to high-dimensional deterministic equivalents. Symmetry and covariance calculations then characterize the teacher’s mean, risk, and prediction-energy fluctuations across strong and weak blocks.

B.3 Theory of DE for stage I

  • The exact Stage-I estimator is represented through an orthogonal projector, yielding a projection-based expression for its population risk.In the full-column-rank branch, Stage I recovers the target coefficient exactly.Source: §B.3 · pp. 61, 62, 64
  • Under the proportional high-dimensional regime and required regularity conditions, deterministic equivalents characterize the minimum-norm population estimator and its predictor.The underparameterized branch uses a positive effective ridge; when N_T ≥ p, the effective ridge is zero.Source: §B.3 · pp. 61, 62, 64
  • Rotational symmetry makes the weak-block mean vanish and constrains the strong-block mean to lie along the target coefficient.The direction result holds at finite p; deterministic equivalents are used to determine its scalar coefficient.Source: §B.3 · p. 65
  • The Stage-I second moment has vanishing cross-block terms and an isotropic weak-block structure, exactly at finite p.The derivation does not require k = o(p).Source: §B.3 · p. 66
  • The deterministic-equivalent fluctuation energy is c_T,p, with weighted trace ratios assigning its share to selected covariance directions.When N_T ≥ p, the projector is the identity and all weighted covariance vanishes.Source: §B.3 · pp. 66, 68

B.4 Stage II: linear-random-feature derivation conditional on Stage I

The Stage-II derivation averages the student’s ridgeless random-feature fit over Stage-II randomness, conditional on the teacher, and then averages over features. Its effective-ridge analysis separates regimes according to whether samples or features are the bottleneck.

B.4 Stage II: linear-random-feature derivation conditional on Stage I

  • The conditional derivation first averages over Stage-II samples and then over random features to obtain the mimic formula.The student’s pseudo-labels are the feature-transformed teacher predictions.Source: §B.4 · pp. 69, 71
  • The analysis defines a ridgeless Stage-II effective ridge as the limit of the regularized parameter and identifies three distinct branches for that limit.The branches are no bottleneck, feature bottleneck, and sample bottleneck.Source: §B.4 · p. 71
  • When p < min{m, N_S}, Stage II has no bottleneck and reproduces the teacher coefficient exactly.The feature matrix has full row rank and the input sample matrix has full column rank almost surely.Source: §B.4 · p. 71
  • When N_S < min{m, p}, the feature bottleneck determines the positive effective-ridge branch.The passage identifies this branch and its effective-ridge condition.Source: §B.4 · p. 71
  • When m < min{p, N_S}, the sample bottleneck yields a positive effective ridge solving df_p(τ_S,p) = m/p.The ridgeless limit is characterized by this unique solution.Source: §B.4 · p. 71

B.5 Positive-signal-fraction regime: assumptions and phase diagram

The positive-signal-fraction phase diagram gives branch-specific conditions for when the teacher-minus-student risk is positive, while critical-boundary analysis characterizes interpolation and signal-threshold behavior.

B.5 Positive-signal-fraction regime: assumptions and phase diagram

  • When both Stage-II resource ratios are below the signal threshold, the master deterministic equivalent is negative for all sufficiently large p.The fixed-ratio analysis excludes critical boundaries such as the signal-rank threshold and the ambient no-bottleneck boundary.Source: §B.5 · p. 73
  • Above the signal dimension, the sample-bottleneck branch has positive W2SG exactly when its quadratic condition holds, with an admissible interval bounded by the roots and branch constraints.The interval is empty when the discriminant is nonpositive or its intersection with the admissible branch is empty.Source: §B.5 · p. 73
  • On the feature-bottleneck branch above the signal dimension, positivity requires condition (356); if ρS − ϑ − ΛT is nonpositive, the limiting numerator is strictly negative throughout that branch.When ρS − ϑ − ΛT is positive, the open positive phase is characterized by the stated threshold condition.Source: §B.5 · pp. 73–74
  • When both Stage-II ratios exceed the ambient dimension, the student matches the teacher coefficient almost surely and W2SG is zero.Under the full-rank conditions, the Stage-II residual is zero and the fitted student predictions reproduce the teacher coefficient.Source: §B.5 · pp. 73, 75

B.6 Critical Boundaries and Finite-Dimensional Effects

  • At the Stage-II interpolation boundary, if rp < p and gp ≤ 1, the inverse-Wishart first moment diverges; under model nondegeneracy, RS,p = +∞ and W2SGp = −∞.If rp < p and gp ≥ 2, the Stage-II expected risk is finite.Source: §B.6 · p. 75
  • The signal-threshold transition occurs within a dimension window of order √p, and joint signal criticality is asymmetric across the active bottleneck branches.The sample-bottleneck branch remains unfavorable, while a positive W2SG region may survive on the feature-bottleneck branch; equality in its sign condition needs a higher-order expansion.Source: §B.6 · pp. 75–76
Loading 2609.33835v1…