Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework

Heejune Sheen; Harrison H. Zhou

Source-linked AI summary

Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework

Heejune Sheen, Harrison H. Zhou

arXiv:2610.40254v1Oct 29, 2026stat.MLcs.LGmath.STstat.TH

TL;DR

The statistical behavior of verified self-training remains incompletely characterized despite its growing use. This paper derives finite-sample risk bounds for verified synthetic-data training in linear regression, finding that tighter verification can increase risk and that partial and full fine-tuning can behave differently.

Problem
Training with verified synthetic data is growing, but its statistical behavior and conditions for reducing prediction risk have not been fully characterized.
Source: §1 · p. 2
Method
The paper analyzes linear regression post-training on synthetic samples filtered by an exact-error verifier, with either all or only selected parameter coordinates updated.
Source: §1 · p. 2
Results
Tighter verification can increase excess prediction risk under partial fine-tuning, risk can be minimized at intermediate generation noise, and frozen-coordinate errors distinguish partial from full fine-tuning.
Source: §1 · pp. 2, 4
Takeaways & Limitations
Finite-sample bounds across four regimes quantify how verification affects accepted-response error, effective sample size, and accepted-covariate covariance, with an added frozen-coordinate contribution under partial fine-tuning.
Source: §7 · p. 25
Under partial fine-tuning, the frozen-coordinate term can limit the reduction in prediction risk.
Source: §3.5 · p. 14

Abstract

from arXiv · show

Synthetic data are widely used for training modern AI systems. However, their statistical understanding remains limited. It is well recognized in statistics that repeated self-training on synthetic data can amplify errors inherited from pre-trained models, leading to model collapse. Verification plays a crucial role in selecting high-quality synthetic data to help prevent model collapse and improve model performance. We study verified self-training in a linear regression framework, where a pre-trained estimator generates synthetic data with injected generation noise and a verifier selects samples according to a verification threshold. Post-training on the accepted samples updates either all coordinates (full fine-tuning) or only an active subset (partial fine-tuning). Our analysis reveals three potentially counterintuitive effects. We establish that tighter verification does not always lead to lower prediction risk. We also find that a suitable level of injected generation noise can reduce prediction risk, while excessive noise can increase it. Moreover, full and partial fine-tuning can respond differently to the same verification rule, since pre-training error in the frozen coordinates affects both sample selection and post-training risk. We analyze these effects by deriving finite sample risk bounds for four separate regimes determined by the relative scales of the verification threshold, generation noise, and pre-training error. The bounds precisely characterize how verification affects prediction risk across these regimes. In particular, we quantify how the generation noise, the effective accepted sample size, and the covariance of the accepted covariates affect prediction risk. The analysis extends to iterative verified full fine-tuning, where sufficiently tight verification reduces the initial error contribution at a geometric rate.

1 Introduction

Synthetic data can propagate estimation errors from pre-training, and this paper studies how verification and fine-tuning choices shape prediction risk in a linear regression framework. Its analysis identifies non-monotone effects of verification and generation noise, distinct behavior under full and partial fine-tuning, and risk bounds across four regimes.

1 Introduction

The framework uses a pre-trained estimator to generate noisy synthetic responses, accepts responses within a verification threshold, and updates either all coordinates or an active subset.The verifier evaluates response error but provides only acceptance decisions, not the true regression values.Source: §1 · p. 2
Tighter verification can raise partial fine-tuning risk, intermediate generation noise can minimize full fine-tuning risk, and the two methods can respond differently to the same threshold.Figure 1 presents these three comparisons; under partial fine-tuning, the plotted excess risk excludes the direct frozen-coordinate contribution.Source: §1 · p. 2, Figure 1 · p. 3

1.1 Contributions

Nonasymptotic bounds across four scale regimes characterize how accepted response error, effective sample size, and accepted-covariate variation shape risk, with an additional frozen-coordinate contribution under partial fine-tuning.When generation noise dominates pre-training error, tighter verification can reduce active-coordinate bias while increasing the frozen-coordinate contribution through fewer accepted samples.Source: §1.1 · p. 4
For repeated verified full fine-tuning, sufficiently small verification thresholds relative to generation noise make the initial-error contribution contract geometrically, whereas large relative thresholds limit its rate of decrease.Source: §1.1 · p. 4

1.2 Related Work

Compared with Yi et al. (2026)'s fixed orthogonal covariate directions, this analysis samples covariates from the downstream distribution and studies covariance-sensitive risk regimes and partial fine-tuning.Source: §1.2 · p. 5

2 Problem Setting

The setup estimates a linear-regression model from real data, then generates and verifies synthetic responses for downstream post-training. Post-training updates an active coordinate set while retaining pre-training values on frozen coordinates; the resulting risk is evaluated on an independent downstream sample.

2.1 Pre-Training Data and the Pre-Trained Estimator

The pre-trained estimator is ordinary least squares from n real samples, with pre-training error Δ=β∗−β̂pre; for n≥Cd, its normalized Gram matrix is bounded with probability at least 1−2e−cn.The pre-training and downstream covariates are Gaussian under well-conditioned covariance assumptions.Source: §2.1 · p. 6

2.2 Verified Synthetic Data

Synthetic responses use downstream covariates and the pre-trained estimator, with injected generation noise; an oracle verifier accepts a pair when its response error against the true model is at most δ.The threshold δ sets the acceptance accuracy criterion, and accepted samples are used for post-training.Source: §2.2 · pp. 6–7

2.3 Post-Training Estimator

Post-training fits the active coordinates by least squares on accepted samples and freezes the rest; under partial fine-tuning, frozen-coordinate error can shift the active target through covariance.Full fine-tuning is the case where all coordinates are active and the frozen block is empty.Source: §2.3 · p. 7

2.4 Prediction Risk

The prediction-risk definition conditions on realized pre-training and synthetic covariates, while its expectation includes pre-training noise, generation noise, and an independent test sample.The supplied definition introduces risk for an estimator constructed from the pre-training and synthetic data.Source: §2.4 · p. 8

3 Main Results

This section analyzes prediction risk by establishing an unverified post-training baseline and comparing partial and full fine-tuning across verification regimes. It examines how threshold, generation-noise scale, pre-training error, accepted sample size, and accepted-covariate distribution shape risk, including the effects of verification in Regimes III–IV.

3.1 Risk Decomposition

Prediction risk separates downstream noise, active-coordinate error, and frozen-coordinate contributions; the active-coordinate variance further splits into inherited conditional bias and generation-noise variance.For the extended estimator, the frozen-coordinate variance is determined by the covariance of the pre-training error conditional on the active coordinates.Source: §3.1 · pp. 8–9

3.2 Baseline without Verification

Without verification, post-training preserves the pre-training error contribution and adds generation-noise variance of order s˜σ2/m.The baseline accepts every generated sample and uses all m synthetic samples.Source: §3.2 · p. 9

3.3 Verification Regimes

The four verification regimes are organized by the relative scales of the threshold, generation noise, and pre-training error, which determine sample selection and the resulting fine-tuning rates.Regime I has a loose threshold relative to pre-training error; Regime II orders threshold below pre-training error below noise; Regime III orders threshold below noise below pre-training error.Source: §3.3 · p. 10, §3.4 · p. 10, §3.5 · p. 12

3.4 Regime I: Loose Verification

In Regime I, sufficiently small δ/˜σ reduces both leading variance terms under partial and full fine-tuning, even when verification is loose relative to pre-training error.Accepted generation noise can offset inherited pre-training error; the leading active-coordinate rates have the same form for both methods, with s replaced by d for full fine-tuning.Source: §3.4 · pp. 11–12

3.5 Regime II: Tight Verification, Noise-Dominated

In Regime II, tighter verification reduces both leading terms under full fine-tuning, while frozen-coordinate contributions create a verification tradeoff and U-shaped risk under partial fine-tuning.For partial fine-tuning, decreasing δ reduces the threshold-dependent active-coordinate term but increases a frozen-coordinate variance contribution; full fine-tuning removes both frozen-coordinate terms.Source: §3.5 · pp. 13–14

3.6 Regime III: Tight Verification, Pre-Training Error-Dominated

In Regime III, partial fine-tuning can retain a frozen-coordinate contribution of order s/n, while full fine-tuning removes that term but accepted-covariate concentration can increase its generation-noise variance.Verification favors covariates with small pre-training prediction error, reducing variation along that error direction; under full fine-tuning this adds a factor 1 + 1/(n˜σ2) to V2.Source: §3.6 · pp. 15–16
In Regime III, tight verification reduces the threshold-dependent V1 term but lowers accepted sample size and changes covariance; partial V1 stays order s/n, while full fine-tuning gains a stronger V2 factor, 1 + 1/(nσ~^2).Because generation noise is smaller than pre-training error, verification favors queries where the pre-trained estimator has small prediction error.Source: §3.6 · p. 16

3.7 Regime IV: Low Generation Noise

When generation noise is below the verification threshold in Regime IV, tight verification leaves leading V1 at s/n for partial and d/n for full fine-tuning.Partial V2 scales as sσ~^2/(mpIV), up to its covariance factor; for full fine-tuning, reduced variation along the pre-training error direction contributes 1 + 1/(nδ^2) to V2. The leading V1 term does not receive the (δ/σ~)^4 reduction seen in Regimes II and III.Source: §3.7 · p. 17

3.8 Comparison Across Regimes

Across Regimes II–IV, frozen-coordinate pre-training error can keep partial-fine-tuning V1 at order s/n and contribute to V2, while full fine-tuning removes those terms and strengthens covariance effects in Regimes III–IV.Accepted covariates retain variation across downstream directions in Regimes I–II, whereas verification favors small |x~⊤∆| in Regimes III–IV; the practical criteria include response-sampling randomness and verifier strictness.Source: §3.8 · p. 18

4 Proof Sketch of the Main Results

The proof sketch decomposes active-coordinate excess risk into a conditional-mean term V1 and a conditional-covariance term V2. It bounds these terms through population approximations, Gaussian calculations, and finite-sample concentration.

Common residual decomposition

The active-coordinate excess risk is V1 + V2: V1 comes from the conditional mean of the post-training residual, and V2 from its conditional covariance over generation noise.The proof also bounds errors from replacing empirical quantities with population counterparts and controls the resulting finite-sample remainder.Source: §4 · pp. 19, 22

Analysis of the V1 rates

For partial fine-tuning, pre-training error in frozen coordinates contributes to V1 and can limit how much verification reduces the inherited error.The frozen-coordinate contribution is proportional to (s/n)((d −s)/(n˜σ2 + d −s))2.Source: §4 · p. 20

Establish the finite sample bounds

The four-regime finite-sample bounds follow by substituting each regime’s scale conditions into the population rates and controlling empirical approximation errors.For full fine-tuning, the frozen-coordinate contribution is zero, and V1 is of order min(1, (δ/˜σ)4)d/n under the stated sample conditions.Source: §4 · pp. 20, 22

Analysis of the V2 rates

V2 captures variance induced by generation noise after verification, while its population expression also separates a contribution from pre-training error in frozen coordinates.The effective accepted sample size mp∆ enters this decomposition; when s = d, the frozen-coordinate terms vanish.Source: §4 · pp. 21–22

5 Iterative Full Fine-Tuning with Verification

The section studies iterative full fine-tuning, in which accepted samples from each generation train the estimator that generates the next batch. Under sufficiently tight verification, the pre-training error contribution contracts geometrically, while sampling error remains bounded; a lower bound shows that loose verification slows error reduction.

5 Iterative Full Fine-Tuning with Verification

Each generation uses the current estimator to generate responses, then uses the accepted samples to define the next full-fine-tuning estimator.The process fixes batch size, verification threshold, generation-noise scale, and covariate covariance across generations.Source: §5 · pp. 22–23
The pre-training-error contribution decreases by a factor 2(δ/˜σ)^4/9 per generation, while accumulated sampling error remains of order dδ˜σ/m uniformly in T.The initialization condition Σ1 ≤ ˜σ2 is required only at initialization.Source: §5 · p. 23
Reducing initial error by a fixed proportion requires at least a constant multiple of (δ/˜σ)^2 generations when the threshold is large relative to generation noise.The initial error cannot decrease rapidly in this regime.Source: §5 · p. 23

6 Empirical Experiments

The experiments examine verified post-training in Gaussian simulations and on semantic textual similarity data. Both settings show that intermediate generation noise can yield lower risk, while real-data results also show that an intermediate verification threshold can be preferable.

6 Empirical Experiments

The experiments include Gaussian simulations and real-data experiments, with the latter evaluating semantic textual similarity on STS-B.The real-data setup uses STS collections for pre-training and STS-B training data for post-training.Source: §6 · p. 23, §6.2 · p. 24

6.1 Gaussian Simulations

Intermediate generation noise minimizes excess risk under both full and partial fine-tuning; partial fine-tuning also has a U-shaped threshold response in Regime II.Its variance remains near s/n in Regimes III and IV.Source: §6.1 · p. 24

6.2 Real Data Experiments

On STS-B, both full and partial fine-tuning attain minimum test-risk ratios at intermediate thresholds, and the best ratio occurs at an intermediate noise scale.With accepted sample size fixed, the lowest test risk occurs at an intermediate noise scale with a tight threshold.Source: §6.2 · p. 25, Figure 8 · p. 25

7 Conclusion

The conclusion summarizes nonasymptotic risk rates for verified self-generated synthetic-data post-training across four regimes and both full and partial fine-tuning. It also reports iterative full fine-tuning results and identifies extensions to more general models and data distributions as future work.

7 Conclusion

The analysis characterizes nonasymptotic risk rates across four regimes for both full and partial fine-tuning, with frozen coordinates adding a distinct risk contribution.The rates account for accepted-response error, effective accepted sample size, and accepted-covariate covariance.Source: §7 · p. 25

Contents of the Appendix

The appendix roadmap covers probability and covariance bounds, risk and residual decompositions, accepted-sample moments, and proofs of the four-regime and iterative results.It also identifies the appendix section containing further experiment details.Source: Contents of the Appendix · p. 30

A Common Probability Tools

The common probability tools include Gaussian quadratic-form and matrix concentration inequalities, plus a sub-exponential vector concentration result.The cited tools include Hanson–Wright, Matrix Bernstein, and a vector concentration lemma.Source: A Common Probability Tools · pp. 31–32

B Pre-Training Covariance Concentration

This material introduces pre-training covariance concentration and relates pre-training error scales to generation-noise variance. It then identifies the risk decomposition and the no-verification baseline addressed in the following section.

B Pre-Training Covariance Concentration

Pre-training covariance bounds are stated under the condition n ≥ Cd and on an event whose probability is at least 1 − 2e^−cn.The bounds are used to control total and active-coordinate pre-training errors relative to generation-noise variance.Source: B Pre-Training Covariance Concentration · p. 33

C Risk Decomposition and Baseline without Verification

The appendix identifies the risk decomposition and a baseline without verification as results of Proposition 3.1 and Proposition 3.3..Source: C Risk Decomposition and Baseline… · p. 34

C.1 Proof of Proposition 3.1

The Proposition 3.1 proof decomposes active-coordinate risk into squared bias and variance, alongside variance from frozen coordinates.The proof uses the Schur complement for the active and frozen coordinate blocks.Source: §C.1 · p. 34

C.2 Derivation of the Residual Representation

The derivation represents the post-training residual using the accepted-sample quantities and decomposes its variance into two components. [p00, p01, p03, p04]

C.2 Derivation of the Residual Representation

On Ωpost, G(Δ) is invertible for the post-training estimator. [p01]Source: §C.2 · p. 34
The residual representation is derived by substituting a relation involving the accepted residual numerator into the post-training estimator. [p00, p02]Source: §C.2 · p. 34

C.3 Derivation of the Variance Decomposition

The residual variance is split into V1 and V2 by applying the law of total covariance over Δ. [p03, p04]The decomposition is obtained after multiplying by Σ1,SS and taking traces.Source: §C.3 · p. 34

C.4 Proof of Proposition 3.3

The proof analyzes the accepted-sample residual and its variance contributions, including a full-acceptance case and bounds on the accepted Gram matrix. [p05, p12, p13, p29]

C.4 Proof of Proposition 3.3

When δ = ∞, every generated sample is accepted, and the accepted Gram matrix is bounded relative to Σ1,SS with high probability. [p05, p07]Source: §C.4 · p. 35
On Ωpre, V1 ≍ s/n; for s = d, tr(Σ1Γ)/n ≍ d/n, while V2 = σ̃² tr(Σ1,SSG−1) ≍ σ̃²s/m and Bias2 S = 0, since its conditional mean is centered over Δ.Source: §C.4 · p. 35

D Variance Decomposition

The variance analysis conditions on Xpre in Ωpre, where Δ has distribution N(0, Γ/n), and treats generation noise separately. [p14]Source: D Variance Decomposition · p. 35

D.1 Conditional Mean and Expected Gram Matrix Bounds

The conditional means of the accepted Gram matrix and residual numerator center the numerator and determine the leading conditional mean in V1. [p15, p16, p18]Source: §D.1 · p. 35, §D.1.1 · p. 36

D.1.2 Accepted Gram matrices and inverse bounds

On the stated event, the accepted Gram matrix is invertible, allowing its inverse to be compared with the inverse conditional mean. [p29, p33]Source: §D.1.2 · pp. 36–37

D.1.3 Residual decomposition

The residual decomposition separates a fixed conditional-mean term, a centered term, and an inverse-replacement error that contributes to bounds for V1 and V2. [p41]Source: §D.1.3 · p. 38

D.3 Generation Noise Variance Contribution V2

The leading conditional covariance contribution to V2 is compared with V2 to obtain upper and lower bounds. [p59, p62]Source: §D.3 · p. 40

D.5 Active Coordinate Bias

The extended estimator has zero active-coordinate bias, E[rS,ext | Xpre, ˜X] = 0. [p78]Source: §D.5 · p. 41

E Population Mean and Covariance for Accepted Samples

This section characterizes population moments and covariance for accepted samples, including the conditional expected Gram matrix and residual numerator. It also develops population conditional-covariance and empirical Gram-matrix bounds, with a probability guarantee for empirical inverse bounds used in later analysis.

E.1 Acceptance Probabilities and Conditional Moments

Uniformly over h>0, p(h), λ2(h), and λ4(h) scale as min(1,h), min(1,h²), and min(1,h⁴), respectively.These are the acceptance probability and the conditional second and fourth moments of the truncated standard normal.Source: §E.1 · p. 42

E.2 Population Accepted Covariate Second Moment

The normalized accepted Gram matrix has s−1 eigenvalues equal to p∆ and one remaining eigenvalue determined by the verification-conditioned moments; it is positive definite when δ>0 and ˜σ>0.The remaining eigenvalue depends on λ2,∆ and the active-coordinate component of Σ1∆.Source: §E.2 · pp. 43–44

E.3 Population Accepted Residual First Moment

Verification corrects the inherited active error in the population conditional mean through θ∆, where 0<θ∆<1.The correction is described as affecting the inherited active error −∆S; bounds for θ∆ and 1−θ∆ are used in the population mean and covariance calculations.Source: §E.3 · p. 45

E.4 Population Conditional Mean in V1

Corollary E.5 characterizes the population conditional-mean scale, with the partial fine-tuning expression depending on the number of frozen coordinates.The corollary treats the pre-training model under Assumption 2.1 and provides a separate full fine-tuning rate.Source: §E.4 · pp. 45–46

E.5 Population Conditional Covariance in V2

The adjusted generation-noise second-moment matrix is bounded between one-half and one times ˜σ²(1−θ∆)Ḡpop(∆); this yields covariance upper bounds for all u and lower bounds for 0<u≤1.Here u=δ/˜σ.Source: §E.5 · pp. 46, 49
The population V2 analysis bounds the conditional covariance of accepted covariates and the squared conditional mean of adjusted generation noise.The latter second-moment bound is used in Bernstein’s inequality in Corollary F.2.Source: §E.5 · pp. 50–51

F Empirical Mean and Covariance Bounds

The empirical analysis conditions on ∆ and targets bounds for the accepted Gram matrix and its inverse, followed by conditional mean and covariance bounds.It also derives accepted Gram-matrix and conditional-mean bounds under full fine-tuning.Source: F Empirical Mean and Covariance Bounds · p. 52

F.1 Conditional Expected Gram Matrix Bounds

The conditional expected Gram-matrix analysis bounds ¯G(∆) around m ¯Gpop(∆) and derives inverse bounds for subsequent mean and covariance analysis.The section states these inverse bounds are needed for the conditional mean and covariance analysis.Source: §F.1 · p. 52

The inverse satisfies

The empirical inverse bounds and associated conclusions hold simultaneously, conditioned on ∆, with probability at least 1 −η.The proof combines events for the Gram-matrix and maximum-eigenvalue bounds, each with conditional failure probability at most η/2.Source: The inverse satisfies · p. 53

F.2 Empirical Conditional Mean Bounds

This section establishes conditional-mean, covariance, and remainder bounds for full and partial fine-tuning, including bounds for variance components at realized synthetic covariates. The results are conditional on pre-training error and stated sample-size conditions or probability events, and include empirical and population-level bounds.

F.2 Empirical Conditional Mean Bounds

Conditioned on ∆, the empirical conditional mean difference is bounded with high probability in both the Gram norm and prediction norm under sample condition (43).The Gram-norm bound depends on u = δ/˜σ, dimension s, and a trace involving Σ1,SS and ¯Gpop(∆)^−1; the prediction-norm bound is stated in Lemma F.3.Source: §F.2 · pp. 54–55

F.3 Conditional Covariance Trace Bounds

The conditional covariance analysis bounds the traces involving M(∆) and its inverse-Gram-weighted form with high probability, using empirical conditional-mean control.The proof controls covariance fluctuations and the error from replacing the empirical conditional mean by its population counterpart.Source: §F.3 · pp. 57, 59

F.4 Accepted Gram Matrix and Conditional Mean Bounds under Full Fine-Tuning

For full fine-tuning, the accepted Gram matrix is controlled, while the conditional residual mean is bounded around −u^2∆/3 for 0 < u ≤1 and around −(1 −θ∆)∆ when u is bounded below.The bounds hold on the stated high-probability events; for u bounded below, the sample-condition constant may depend on the lower bound u0.Source: §F.4 · p. 60
Full fine-tuning’s second conditional-mean bound holds when the sample condition mp∆ ≥ C{d + log(m/η)} log²(m/η) is met for fixed u ≥ u0 > 0.The coefficient condition is ensured by taking C sufficiently large depending only on u0; the associated event has conditional probability at least 1 − η.Source: §F.4 · p. 64

G Variance Components for the Realized Synthetic Covariates

The variance components V1 and V2 are evaluated at realized synthetic covariates, with expectations over the pre-training error conditioned on Xpre.The analysis first establishes empirical bounds over ˜X for each fixed ∆, then derives rates and full fine-tuning conditional-mean bounds.Source: G Variance Components for the Realized… · p. 64

G.1 A Common Event for the Realized Covariates

A common sample condition makes the fixed-∆ empirical bounds hold with probability at least 1 − η over ∆, and yields empirical bounds at realized covariates with probability at least 1 − η over ˜X.The realized-covariate event also bounds the probability of failure over ∆ and synthetic noise by 2η and is contained in Ωpost.Source: §G.1 · p. 65

G.2 Remainder Bounds at the Realized Covariates

At realized covariates, the analysis bounds the inverse remainder almost surely over ˜X and establishes moment bounds for contributions on the complement event Ω(∆)^c.The moment bounds hold with probability at least 1 − η over ˜X under the stated sample condition.Source: §G.2 · pp. 66–67

G.3 Conditional Mean Bounds for V1

At realized synthetic covariates, the analysis establishes a partial fine-tuning V1 rate for every u > 0, with probability at least 1 − η over the covariates.The rate is stated under the common sample condition, with η = m^-C for sufficiently large fixed C.Source: §G.3 · p. 68

G.4 Full Fine-Tuning Conditional Mean at the Realized Covariates

For full fine-tuning, the realized-covariate conditional-mean bounds cover every ∆, with additional probability bounds over ∆ for 0 < u ≤ 1 and for u ≥ u0 > 0.These statements hold under the sample condition and with the probabilities specified in Corollary G.6.Source: §G.4 · p. 70
The full fine-tuning analysis bounds the conditional mean at realized covariates by averaging over acceptance indicators, with separate arguments for u ≤ 1 and u ≥ u0 > 0.The proof uses a lower bound on the accepted-sample Gram matrix on a high-probability event and controls its complement.Source: §G.4 · pp. 70–71

G.5 Conditional Covariance Contribution to V2

Proposition G.7 bounds the conditional covariance trace and inverse remainder under the stated model and sample condition.Its assumptions fix Xpre in Ωpre and require the sample condition with η replaced by η².Source: §G.5 · p. 71
Corollary G.8 gives population bounds for V2, holding with probability at least 1 − η over ˜X when the stated sample condition is satisfied.The result is conditioned on Xpre and applies to the extended estimator.Source: §G.5 · p. 72

G.6 Weighted Population Inverse Trace

Lemma G.9 bounds a weighted population inverse trace by conditioning on the pre-training data and averaging over the pre-training error. Those bounds are then used to establish V2 rates for partial and full fine-tuning.

G.6 Weighted Population Inverse Trace

Lemma G.9 gives weighted population inverse trace bounds under the pre-training model and Assumption 2.1, conditional on fixed pre-training covariates.The expectation is over the pre-training error given the fixed pre-training covariates.Source: §G.6 · p. 73
The proof conditions on frozen-coordinate error to average over active error first, then averages over the frozen error.The covariance scale depends on the frozen error, while the population inverse trace also depends on the active error.Source: §G.6 · p. 73
Corollary G.10 uses Lemma G.9 to establish extended-estimator rates for V2 under the stated sample condition, for partial and full fine-tuning.Conditioned on pre-training covariates, the rates hold with probability at least 1 − η over the synthetic covariates.Source: §G.6 · pp. 73, 76

H Full Fine-Tuning Rate for V1

The section gives full fine-tuning rates for V1 and specializes the established V1 and V2 bounds to partial- and full-fine-tuning regimes. The regime tables summarize the leading terms in those bounds.

H Full Fine-Tuning Rate for V1

Corollary H.2 gives a full fine-tuning rate for V1 for all u > 0 under the stated sample condition and probability guarantee.It combines the small-u expansion in Corollary H.1 with Corollary G.6 when u is bounded below.Source: H Full Fine-Tuning Rate for V1 · p. 77

I Regime Specializations and Main Results

Section I derives partial- and full-fine-tuning results from the V1 and V2 bounds and relates pre-training-dependent quantities to deterministic scales.The resulting scales are organized by regime.Source: I Regime Specializations and Main… · p. 77, §I.1 · p. 78

I.1 Regime Overview

The regime tables summarize the leading terms in the V1 and V2 bounds.The section obtains the partial-fine-tuning terms from Corollaries G.5 and G.10, and the full-fine-tuning terms from Corollaries H.2 and G.10.Source: §I.1 · p. 78

I.2 Regime I

Appendix I presents partial- and full-fine-tuning results across four scale regimes, while Appendix J develops moment bounds for accepted samples and pre-training error.

I.2 Regime I

In Regime I, the partial-fine-tuning theorem’s remainder is d(s + log m)/(mn) + s(s + (log m)^2)(log m)^3 min(δ^2, σ̃^2)/(mpI)^2.The theorem assumes mpI ≥ C(s + log m)(log m)^(7/2).Source: §I.2 · p. 79

I.3 Regime II

In Regime II, the partial-fine-tuning remainder includes δ^2 + σ̃^2(d − s)/(nσ̃^2 + d − s), scaled by s(s + (log m)^2)(log m)^3/(mpII)^2.The theorem assumes mpII ≥ C(s + log m)(log m)^(7/2).Source: §I.3 · p. 80

I.4 Regime III

Regime III gives separate partial- and full-fine-tuning theorems under (III-scale), with sample requirements that include the factor 1 + d/(nσ̃^2).The required factors are (s + log m) for partial fine-tuning and (d + log m) for full fine-tuning.Source: §I.4 · p. 81

I.5 Regime IV

Regime IV likewise gives partial- and full-fine-tuning theorems, with sample requirements proportional to d(s + log m)/(nδ^2) and d(d + log m)/(nδ^2), respectively.Both theorem statements assume (IV-scale).Source: §I.5 · p. 82

J.1 Accepted Sample Moments

Conditioned on acceptance, the paper bounds the error-form second moment by σ̃^2 + δ^2 and relates conditional ψ2 norms to conditional second moments.The accepted-sample moment results support the population and empirical analyses and are also used in the iterative analysis.Source: §J.1 · p. 83, J Moment Bounds for Accepted Samples… · p. 83

J.2 Chi-Square Moment Bounds

For Xk distributed as chi-square with k degrees of freedom, the stated moment bounds include E[tXk/(1 + tXk)^2] ≍ kt/(1 + kt) E[(1 + tXk)^−1].These bounds are used for the pre-training error.Source: §J.2 · p. 85

J.3 Pre-Training Error Moments

This section derives moments of pre-training errors and establishes bounds on the normalized conditional noise mean under Gaussian generation noise. These results provide auxiliary controls for analyzing iterative full fine-tuning, including a conditional squared-error lower bound after one generation when s = d.

J.3 Pre-Training Error Moments

Lemma J.4 gives marginal moments for the frozen-coordinate error, while the proof treats its normalized squared norm as a Gaussian quadratic form.The moment bounds are derived under the pre-training model and Assumption 2.1, conditional on Xpre.Source: §J.3 · p. 86
Lemma J.5 concerns weighted moments of the active quadratic form under the pre-training model and Assumption 2.1, for fixed Xpre ∈ Ωpre and 1 ≤ s < d, with k = d − s.For z > 0, hk,d(z) = (d/(d + z²))^(k/2) if 1 ≤ k ≤ 3; h4,d(z) = (d/(d + z²))² log(e + z²/d).Source: §J.3 · p. 87

K.1 Auxiliary Bounds

Lemma K.1 gives a deterministic uniform error bound across generations, and Lemma K.3 controls the probability of a large post-training error for arbitrary preceding error.These bounds apply to the iterative full fine-tuning model under the stated conditions on s, δ, and the Gram-matrix exponent.Source: §K.1 · pp. 89–91
For iterative full fine-tuning, Corollary K.2 bounds conditional squared error after one generation when the preceding error norm is at most σ̃, providing the contraction step.Its conditions include small u = δ/σ̃ and mu ≥ C(d + log m) log^2 m.Source: §K.1 · p. 90
Lemma K.4 bounds the squared-error contribution when the error norm exceeds σ̃, enabling control of that region uniformly over generations.The bound assumes the stated sample-size condition and small u = δ/σ̃.Source: §K.1 · p. 92

K.2 Proof of Theorem 5.1

The proof of Theorem 5.1 combines the one-generation bound with the contribution outside the σ̃ region and iterates the resulting recursion.It conditions on the initial error and uses Lemma K.4 to control terms when the preceding or updated error exceeds σ̃.Source: §K.2 · p. 94

K.3 Lower Bound

The lower-bound argument bounds the expected update in the direction Σ1∆(t−1) using a monotonicity property.The passage identifies this as the approach for obtaining the lower bound.Source: §K.3 · p. 94
For ˜ε ∼ N(0, ˜σ²), f(a) = E[˜ε | |˜ε − a| ≤ δ]/a is nondecreasing for a > 0.Source: §K.3 · p. 94
For every a > 0, the normalized conditional generation-noise mean satisfies 0 ≤ f(a) ≤ 1.Source: §K.3 · p. 94
For s = d, this monotonicity yields a conditional squared-error lower bound after one generation for every integer t ≥ 1, with θ_∆ defined from the generation noise, λ²_∆, and ∥∆∥²_Σ₁.The bound is stated under the iterative training model and holds almost surely.Source: §K.3 · p. 95

K.4 Proof of Theorem 5.2

The proof of Theorem 5.2 iterates a one-step inequality; the accompanying Gaussian simulations generate accepted samples directly from their conditional distribution and vary verification thresholds and noise scales.

K.4 Proof of Theorem 5.2

The proof establishes the stated bound by conditioning on the initial error, applying the tower property, and iterating an inequality over t = 1, . . . , T.It treats T = 0 separately and assumes full fine-tuning for T ≥ 1.Source: §K.4 · p. 98

L.1 Gaussian Simulations

The simulations draw the accepted-sample count from Binomial(m, p∆) and then sample from the joint conditional distribution given verification, matching generation of all candidates followed by filtering.Accepted responses are fit by least squares; partial fine-tuning fixes frozen coordinates at their pre-trained values.Source: §L.1 · pp. 98–99

L. Experimental Details

Figure 1 varies δ or σ̃ across three settings, with at least 50 runs per point and at least 100 for threshold comparisons.Figure 3 excludes five smallest-threshold settings where at least one run had too few accepted samples to fit the estimator.Source: §L.1 · p. 99

L.2 Real Data Experiments

The real-data experiment constructs sentence-pair covariates from three datasets, removes cross-split duplicates, and evaluates synthetic-response fine-tuning alongside a direct human-label fitting reference.

L.2 Real Data Experiments

The covariates concatenate coordinatewise products and absolute differences of unit-norm 384-dimensional sentence embeddings, yielding d = 762 after removing six low-variance coordinates.After duplicate removal, the pre-training, post-training, and test sets contain 4989, 7229, and 1379 sentence pairs.Source: §L.2 · p. 99
Partial fine-tuning uses 381 active coordinates, while Figure 8’s direct-fitting reference uses all 7229 human post-training labels with a specified ridge penalty.Figure 8 generates responses until 3048 samples pass verification.Source: §L.2 · p. 99
At full fine-tuning with δ/Rpre = 0.05, only seven of 16 attempted fits met the minimum-eigenvalue criterion, and the reported average uses those seven fits.The minimum eigenvalue of the normalized active-coordinate Gram matrix exceeded 10^−10 in those fits.Source: §L.2 · p. 99