PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

Jichu li, Difan Zou

arXiv:2609.09572v1stat.MLcs.AIcs.LG

TL;DR

As high-quality human-generated data becomes limited, this paper asks how synthetic data affects generalization and whether its benefits depend on how it is used. It analyzes one-pass SGD in high-dimensional linear regression under mixed and two-stage training, finding collapse under mixing but vanishing excess risk with the two-stage curriculum.

  • Problem

    As high-quality training data becomes limited, the paper asks whether synthetic data benefits model training and what adverse effects may arise.

    Source: §1 · p. 1

  • Method

    The paper analyzes one-pass SGD in high-dimensional linear regression with shared covariates but differing real and synthetic labeling functions under mixed and two-stage training.

    Source: §1 · p. 2

  • Results

    Mixed training exhibits strong model collapse with a non-vanishing excess-risk floor, whereas two-stage training avoids the floor and achieves vanishing excess risk.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Synthetic data is neither inherently harmful nor beneficial: its effect depends on data quality and the training protocol, with ordering and scheduling proving consequential.

    Source: §1 · p. 2, §6 · p. 13

    The analysis focuses on one-pass SGD with shared covariates; tight characterizations remain open for data-limited and two-stage settings.

    Source: §6 · p. 13

Abstract

from arXiv · show

Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Dohmatob et al., 2024), where any fixed fraction of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. In this paper, we study how synthetic data affects the generalization of one-pass SGD in high-dimensional linear regression with model shift. We establish finite-sample risk bounds for mixed and two-stage training, separating standard bias and variance from source-mismatch effects, namely fluctuation and persistent drift under mixing and filtered initialization bias under two-stage. These bounds reveal a sharp contrast: mixed training induces strong model collapse, while two-stage training avoids the floor by using synthetic data only in the first stage, showing that collapse is not inevitable under a simple data curriculum. Under a random sketch model, we further obtain scaling laws for both protocols, with tight results for mixed training in the optimization-saturated regime. These laws show that larger models may amplify synthetic-induced degradation under mixing, and quantify how high-quality synthetic pretraining may reduce bias in two-stage training. Finally, we establish an exact finite-sample necessary-and-sufficient condition for two-stage training to strictly outperform real-only training under the same real-data budget and identical real-stage updates. Overall, our results highlight that synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality and the training protocol used to incorporate it.

1 Introduction

Synthetic data can expand scarce training data but may cause strong model collapse under distribution mismatch. This paper shows that one-pass SGD’s outcome depends critically on whether synthetic and real data are mixed or scheduled in two stages.

Motivation

  • Synthetic data supplements limited high-quality human data, but iterative use can cause model collapse as models overfit synthetic patterns that lack real-world variability [Shumailov et al., 2023].Prior work documents both improved generalization from dataset amplification and degradation from synthetic-data overfitting.Source: §1 · pp. 1–2

Related work

  • Existing theory identifies strong model collapse as a non-vanishing excess-risk floor from even a small synthetic-data fraction, but largely omits finite-sample SGD dynamics [Dohmatob et al. 2024b].The paper addresses this gap in high-dimensional linear regression with real–synthetic labeling mismatch.Source: §1 · p. 2

Mixed training

  • Finite-sample bounds show that mixed one-pass SGD suffers strong model collapse, with synthetic degradation decomposed into fluctuation and persistent drift; iterate averaging does not remove it.The result attributes collapse to sample mixing rather than a specific last-iterate optimization scheme.Source: §1 · p. 2

Two-stage training

  • Two-stage training avoids the non-vanishing excess-risk floor and achieves vanishing excess risk by using synthetic data first and real data afterward.The ordering and scheduling of data can therefore matter as much as the data source itself.Source: §1 · p. 2

Scaling laws and comparison

  • Random-sketch scaling laws show that larger models may amplify mixed-training degradation, while high-quality synthetic pretraining can reduce two-stage bias; an exact finite-sample condition characterizes when two-stage training beats real-only training.The comparison uses the same real-data budget and identical real-stage updates; a one-dimensional toy model illustrates sample size, mismatch, and label-noise effects.Source: §1 · pp. 2–3

2 Related Work

Prior work links synthetic data to model collapse, studies multiple-source learning and covariate shift, and develops high-dimensional SGD theory. This paper extends these lines to finite-sample one-pass SGD with shared covariates but mismatched labeling functions, comparing mixed and two-stage training.

Synthetic data and model collapse

  • Synthetic data can cause strong model collapse, but this paper shows that finite-sample one-pass SGD avoids collapse under a simple two-stage protocol while mixed training does not.Strong collapse means even a small synthetic-data fraction prevents improvement under data scaling, leaving a non-vanishing excess risk floor [Dohmatob et al., 2024b].Source: §2 · p. 3

Learning with multiple data sources

  • Unlike prior multi-source learning work focused mainly on statistical estimators or covariate shift, this paper analyzes SGD with shared covariates and mismatched labeling functions.Related analyses include pretraining-finetuning, continual learning, minimax SGD, and adaptive mixed-sample SGD under covariate shift [Wu et al., 2022a, Ding et al., 2024, Liu et al., 2025, Deng and Kpotufe, 2025].Source: §2 · p. 3

SGD in high-dimensional linear regression

  • Building on one-pass high-dimensional SGD and random-sketch scaling-law analyses, this paper extends single-source theory to two sources with mismatch-induced drift and fluctuation effects.Prior work characterizes last-iterate one-pass scaling laws [Lin et al., 2024], multi-epoch behavior [Lin et al., 2025, Yan et al., 2025], and functional scaling laws with schedule optimization [Li et al., 2025a, Li et al., 2026, Wang et al., 2026].Source: §2 · p. 4

3 Preliminaries

This section formulates high-dimensional linear regression with real and synthetic data, specifies covariance and moment assumptions, and defines the training protocols and SGD evaluation setup. The analysis targets excess risk on the real distribution under model shift between the two sources.

Problem Setup

  • Linear regression learns a weight vector w in a finite- or countably infinite-dimensional Hilbert space by minimizing expected loss over (x, y) drawn from P.The feature vector lies in H, labels are real-valued, and the expectation is taken over the data distribution.Source: §3 · p. 4

Assumptions

  • The data covariance has finite trace and entries, parameter vectors have finite H-norm, and fourth moments satisfy upper and lower PSD-matrix bounds governed by α and β.These conditions control covariance structure and fourth-order feature moments used in the analysis.Source: §3 · p. 4

Two-Source Model

  • Real and synthetic sources share the feature marginal but differ in labeling functions, isolating model mismatch relevant to self-training and knowledge distillation.Both sources follow well-specified linear models, with their mismatch represented by the shift between the corresponding optimal parameters.Source: §3 · p. 4

Training Protocols

  • Mixed training randomly interleaves M synthetic and N real samples, whereas two-stage training processes M synthetic samples before N real samples.For mixed training, p := M/(M + N) denotes the synthetic-data fraction.Source: §3 · p. 5

SGD Iteration

  • SGD uses a geometrically decaying schedule over T iterations, applied globally for mixed training and separately within each stage, with last-iterate excess risk evaluated on real data.The initial stepsize γ is a hyperparameter; T = M + N for mixed training.Source: §3 · p. 5

4 Main Results

The main results characterize how mixed and two-stage training affect excess risk under synthetic-data mismatch, including model-size, synthetic-quality, and training-condition effects. They also identify when synthetic pretraining improves over real-only training through a precise initialization condition and a mismatch–variance trade-off.

4.1 Mixed training

  • Mixed training exhibits strong model collapse: with a fixed synthetic proportion p, excess risk converges to a strictly positive floor of order p^2∥δ∥^2_H.Finite-sample upper and lower bounds identify unavoidable fluctuation and drift errors, with the drift term producing the asymptotic floor under eD_eff = o(eN_eff).Source: §4.1 · pp. 6–7

4.2 Two-stage Training

  • Two-stage training avoids strong model collapse and yields vanishing excess risk because synthetic data only sets initialization before all subsequent updates use real data.The protocol removes persistent synthetic drift from the optimization trajectory; learning-rate restart changes forgetting speed but not this qualitative conclusion.Source: §4.2 · pp. 7–8
  • Synthetic data primarily reduces bias rather than variance in two-stage training, improving initialization when mismatch ∥δ∥^2_H is small and synthetic sample size is large.The effective variance remains of the same order as real-only SGD, while synthetic-stage error can vanish and high-quality pretraining places the model closer to the real target.Source: §4.2 · p. 8

4.3 Random Sketch Model & Scaling laws

  • Random-sketch scaling laws show that mixed training gains reducible bias pre-asymptotically but larger model dimension D can amplify synthetic degradation as its drift contribution approaches order p^2.In the optimization-saturated regime γeN_eff ≳ D^a, the mixed-training upper bound is sharp up to constants.Source: §4.3 · p. 10
  • For two-stage training, scaling bounds show that high-quality synthetic data with b > a can reduce bias relative to real-only training without introducing a non-vanishing drift floor.The result is an upper-bound characterization: cross-term cancellation prevents claiming uniform actual outperformance over real-only training.Source: §4.3 · p. 10

4.4 Finite-Sample Conditions for Synthetic Benefit

  • Under a fixed real-data budget and identical real-stage updates, the paper establishes an exact finite-sample necessary-and-sufficient condition for two-stage training to strictly outperform real-only training.A one-dimensional model then isolates the roles of synthetic sample size, label noise, and source mismatch.Source: §4.4 · p. 11
  • Two-stage training strictly outperforms real-only training if and only if its synthetic initialization yields lower post-update error than w0 under the same N real updates.The condition emphasizes improvement in directions that the available real data cannot efficiently correct.Source: §4.4 · p. 11
  • In the one-dimensional model, synthetic pretraining improves expected risk exactly when δ^2 + σ^2/M < (w0 − θ)^2.The mismatch δ^2 persists with unlimited synthetic data, whereas σ^2/M is finite-sample synthetic uncertainty.Source: §4.4 · pp. 11–12
  • Such improvement is achievable for some M if and only if δ^2 < (w0 − θ)^2, requiring M > σ^2/((w0 − θ)^2 − δ^2).The required M grows linearly with σ^2 and diverges as δ^2 approaches (w0 − θ)^2.Source: §4.4 · pp. 11–12
  • Mixed training exhibits a p-dependent excess-risk floor, whereas two-stage training continues improving; mixed-training degradation also grows with model size.At fixed T = 50000, mixed-training excess risk scales approximately linearly with p^2 and larger models amplify contamination before saturation.Source: Figure 1 · p. 12, §4.4 · p. 12

5 Experiments

Experiments validate the theory in Gaussian high-dimensional linear regression by testing synthetic-induced error floors and when synthetic data helps, then extend the study to CIFAR10 and WikiText-103. The real-world experiments show the same qualitative behavior predicted by the theory.

Simulation

  • Simulations vary synthetic proportion p ∈ (0, 1), sample size T ∈ [50, 50000], and sketch width D under mixed and two-stage protocols to validate the predicted synthetic-induced error floor.All experiments are repeated with 5 random seeds; the two experiment parts are reported in Figure 1 and Figure 2, respectively.Source: §5 · p. 12
  • A second simulation study varies synthetic quality b, synthetic sample size M, and label noise σ^2 to study when synthetic data helps.Results are reported in Figure 2 and use the same high-dimensional setup.Source: §5 · p. 12

Real-world data

  • Experiments on CIFAR10 with a 5-layer CNN and WikiText-103 language modeling reproduce the theory’s qualitative behavior using synthetic data from a pretrained teacher.Full experimental details are provided in Appendix H.Source: §5 · p. 12

6 Conclusion · Appendix Contents · A Technical Operator Preliminaries

The paper shows that synthetic-data effects in one-pass SGD depend on training protocol and data quality: mixed training can collapse, whereas two-stage training can avoid collapse and sometimes outperform real-only training. The appendix formalizes the linear operators and PSD-mapping conditions used in the proofs.

6 Conclusion

  • Mixed training exhibits strong model collapse, whereas two-stage training avoids the collapse floor by using synthetic data only during initialization.This contrast shows that collapse is not inevitable under a simple data curriculum.Source: §6 · pp. 12–13
  • Two-stage training strictly outperforms real-data-only training exactly under a finite-sample condition with the same real-data budget and identical real-stage updates.The result characterizes when synthetic pretraining provides a net benefit rather than merely avoiding harm.Source: §6 · p. 13
  • Random-sketch scaling laws show that larger models may amplify synthetic-induced degradation, while high-quality synthetic pretraining can reduce bias.The mixed-training law is tight only in the optimization-saturated regime, whereas the two-stage result remains an upper bound.Source: §6 · p. 13
  • Limitations include shared covariates, one-pass SGD, and unresolved tight characterizations for data-limited and two-stage settings.Extensions to joint covariate and model shift and multi-epoch SGD remain open.Source: §6 · p. 13

A Technical Operator Preliminaries

  • The appendix defines operators I, M, fM, T_t, and eT_t on symmetric matrices and specifies how operator action ◦A is interpreted.These definitions support the technical proof machinery for the regression analysis.Source: A Technical Operator Preliminaries · p. 20
  • For symmetric A, the appendix gives expectation and matrix-conjugation identities for M, fM, T_t, and eT_t, including equation (A.1).The identities connect abstract operator expressions to stochastic and deterministic covariance updates.Source: A Technical Operator Preliminaries · p. 20
  • Lemma A.1 states that M and fM are PSD mappings and gives step-size conditions ensuring eT_t−1 and T_t−1 are PSD mappings.The lemma is attributed to Zou et al. [2023].Source: A Technical Operator Preliminaries · p. 20

B Mixed Training Upper Bound Analysis … C.3 Proof of Theorem 4.2

This section develops the mixed-training excess-risk analysis by decomposing SGD error into bias, variance, drift, and fluctuation components under fixed-budget random-order sampling. It derives component-wise upper bounds and completes the analysis with combined bounds and fluctuation lower bounds established through mean and centered-fluctuation recursions.

B Mixed Training Upper Bound Analysis

  • The mixed-training protocol uniformly shuffles pN synthetic and (1 −p)N real samples, yielding exact covariance and noise recursions without requiring independent source indicators.The recursions rely instead on fresh covariates and sample-wise conditionally independent noises.Source: B Mixed Training Upper Bound Analysis · pp. 20–21

B.1 Excess Risk Decomposition

  • Excess risk is analyzed by decomposing centered SGD error into bias, variance, drift, and fluctuation terms and combining their contributions with Cauchy–Schwarz.The parameter error is measured relative to the real-data optimum.Source: §B.1 · p. 22

B.2 Bias upper bound

  • Under Assumptions 3.1–3.2 and γ < 1/(4α tr(H)), Lemma B.1 provides the mixed-training bias upper bound using Neff = N/ log N.The bound is expressed through k∗ and Deff, with k∗ determined by the eigenvalue threshold involving γNeff.Source: §B.2 · p. 22

B.3 Variance upper bound

  • Under Assumptions 3.1–3.3 and γ < 1/(α tr(H)), Lemma B.2 bounds the variance term with Neff = N/ log N.The variance recursion uses a noise covariance formed from the real and synthetic sources, while the resulting bound is expressed through k∗ and Deff.Source: §B.3 · p. 23

B.4 Drift upper bound

  • The drift contribution to excess risk is decomposed into two parts, and Lemma B.3 gives a drift bound under Assumptions 3.1, 3.2, and 3.3 when γ < 1/(α tr(H)).Source: §B.4 · pp. 23–24

B.5 Fluctuation upper bound

  • Lemma B.4 bounds the fluctuation term under Assumptions 3.1–3.3 and γ < 1/(α tr(H)) by reducing its auxiliary processes to variance-type recursions.The first auxiliary fluctuation process follows the variance recursion with a modified noise covariance, while a fixed-budget variance identity controls the second term.Source: §B.5 · pp. 26, 28, 29

B.6 Proof of Theorem 4.1

  • Theorem 4.1 is obtained by combining the bias, variance, drift, and fluctuation upper bounds.Source: §B.6 · p. 32

C Mixed Training Lower Bound Analysis

C.1 The mean recursion

  • The mean recursion is unrolled using commutation of each factor (I −γtH) with H, producing a closed-form mean process.Source: §C.1 · p. 33

C.2 The centered fluctuation term

  • Lemma C.1 supplies a variance lower bound when Assumptions 3.1 and 3.3 hold, Neff = N/ log N ≥10, and γ < 1/λ1.Source: §C.2 · p. 34
  • The centered fluctuation analysis yields Lemma C.2 under Assumptions 3.1, 3.2, and 3.3, with N ≥500 and γ < 1/λ1.The proof decomposes the fluctuation into variance and residual schedule components, using conditional independence, vanishing cross terms, and tail-space eigenbasis bounds.Source: §C.2 · pp. 36, 38, 40, 41

C.3 Proof of Theorem 4.2

  • Theorem 4.2 follows by combining Lemma C.2 with Eq. C.10, completing the mixed-training excess-risk lower-bound proof.Source: §C.3 · p. 41

C.4 Proof of Corollary 4.3 · D Two-stage Training Analysis

The proof establishes Corollary 4.3 by showing contraction and effective-dimension terms vanish asymptotically, while the two-stage analysis bounds real-stage performance by conditioning on the synthetic-stage iterate. The resulting framework transfers a pure real-data SGD bound to two-stage training under stated stepsize conditions.

C.4 Proof of Corollary 4.3

  • As T →∞ with fixed synthetic proportion p, the proof reduces Corollary 4.3 to an asymptotic limit.This reduction uses p = M/(M + N) fixed.Source: §C.4 · p. 41
  • The contraction operator P_T vanishes in H-norm under the stepsize and geometric tail-decay assumptions.The proof uses eigendecomposition and dominated convergence to establish the limit.Source: §C.4 · pp. 41–42
  • Because eD_eff = o(eN_eff), terms (II), (III), and (IV) vanish, while term (I) tends to zero as T →∞.This makes the upper-bound contributions asymptotically negligible.Source: §C.4 · p. 42
  • The lower-bound argument discards nonnegative terms (B) and (C), retains (A), and uses P_T w → 0 to complete the corollary.The upper and lower bounds are then combined to prove the result.Source: §C.4 · p. 42

D Two-stage Training Analysis

  • Under Assumptions 3.1–3.3 and γ < 1/(4α tr(H)), the real-only comparison uses N_eff := N/log N and D_eff := k* + γ^2N^2.Here k* = max{k : λ_k ≥ 1/(γN_eff)}.Source: D Two-stage Training Analysis · p. 43
  • The two-stage bound conditions on the first-stage iterate w_M, then applies the pure real-data SGD bound to N real samples initialized at w_M.Conditionally on F_M, w_M is deterministic and the second stage is ordinary real-data SGD; expectation over F_M yields the unconditional bound.Source: D Two-stage Training Analysis · p. 43
  • The analysis relates the conditioned real-stage terms to excess risk after synthetic training and obtains the stated two-stage upper bound.The derivation uses the polynomial structure of P_N, the condition 0 ⪯ P_N ⪯ I, and the definition of k* before substitution.Source: D Two-stage Training Analysis · pp. 43–45

D.1 Two-stage training without learning-rate restart … E.2 Mixed Training Lower Bound

The analysis shows that uninterrupted two-stage schedules preserve the qualitative absence of a synthetic-risk floor, while random-sketch bounds establish matching mixed-training upper and lower scaling laws in the optimization-saturated regime. The random-sketch treatment decomposes risk into irreducible, approximation, and excess components before deriving these results.

D.1 Two-stage training without learning-rate restart

  • Without a learning-rate restart, two-stage training replaces the restarted real-stage contraction with the real-data tail of the global schedule while retaining the same qualitative bound decomposition.The proof remains a conditional single-source SGD argument after the first-stage iterate, with spectral cutoff and effective dimension defined using the tail schedule.Source: §D.1 · p. 45
  • An uninterrupted schedule slows finite-sample forgetting of synthetic initialization but creates neither persistent synthetic drift nor a non-vanishing excess-risk floor.For fixed p = M/(M+N) ∈ (0, 1), the real-data tail contracts asymptotically under the stated effective-dimension consistency condition.Source: §D.1 · pp. 45–46

E Random Sketch Model

  • The random-sketch analysis decomposes population risk into irreducible risk, approximation error, and excess risk, then focuses on bounding excess risk under Assumptions 4.6 and 4.7.Expectations are conditioned on the sketch matrix S in the subsequent analysis.Source: E Random Sketch Model · p. 46

E.1 Mixed Training Upper Bound

  • E.1 derives a high-probability mixed-training excess-risk upper bound for the sketched covariance SHS^T using effective sample size e_Neff = (M+N)/log(M+N), fluctuation terms, and effective dimension.The result holds with probability at least 1 − e^(−Ω(D)) over the sketch matrix under the stated step-size and power-law assumptions.Source: §E.1 · pp. 46–47
  • The upper-bound analysis combines irreducible, approximation, and excess terms after choosing γ ≍ 1, yielding the final random-sketch scaling law.The bound is stated with probability at least 1 − e^(−Ω(D)) over the randomness of S.Source: §E.1 · pp. 47–48

E.2 Mixed Training Lower Bound

  • E.2 establishes a mixed-training lower bound in the optimization-saturated regime γ e_Neff ≳ D^a, matching the upper bound up to constants.The resulting excess-risk lower bound includes E_Excess ≳ p^2(1 − D^(1−b)).Source: §E.2 · pp. 48–49
  • The lower-bound proof controls the sketched spectral filter using the geometric schedule and eigenvalue bounds, then substitutes the resulting signal norm estimate into the excess-risk expression.The key estimate is ∥v*_2 − v*_1∥^2_SHS^T ≍ 1 − D^(1−b), which produces the stated lower bound.Source: §E.2 · pp. 48–49

E.3 Two-stage Training Analysis

This analysis derives high-probability excess-risk bounds for two-stage training under a random sketch model, separating first-stage bias, approximation, variance, and irreducible terms. The first-stage bias scales as EBias_2 ≲ k M_eff γ + k^(1−a), and the resulting upper bound is tight up to constants in the data-limited regime.

Proof setup

  • The proof conditions on the sketch matrix, analyzes the first-stage excess risk with zero initialization, and uses shared eigenvectors of H and Σ to control bias and variance.The step-size condition is γ < 1/(4α tr(SHS⊤)).Source: §E.3 · p. 50

First-stage excess risk

  • The first-stage bias satisfies EBias_2 ≲ k M_eff γ + k^(1−a) under the eigenvalue-decay analysis.The analysis distinguishes cases according to the relative decay parameters a and b, with c := min{a,b}.Source: §E.3 · p. 50
  • In the data-limited regime, (M_eff γ)^(1/a) ≤ D/c_0, the upper bound for the first-stage excess risk is tight up to constants.The bound holds with probability at least 1 − e^(−Ω(D)) over the sketch matrix.Source: §E.3 · p. 51

High-probability guarantees

  • The resulting conditional excess-risk estimate and combined bound hold with probability at least 1 − e^(−Ω(D)) over the randomness of the sketch matrix.The analysis combines estimates for the first-stage excess, spectral control, approximation, and variance contributions.Source: §E.3 · pp. 51–53

Risk decomposition

  • With probability at least 1 − e^(−Ω(D)), the two-stage excess risk is bounded by a decomposition into irreducible, approximation, first-stage excess, and variance terms.The bound is obtained on the intersection of spectral and excess-risk events, using γ ≂ 1 and, for simplicity, b < a + 1.Source: §E.3 · pp. 52–53

F Exact Synthetic-Benefit Condition · G constant stepsize SGD with iterate averaging

The paper derives an exact necessary-and-sufficient condition for two-stage training to outperform real-only training when both use identical real-stage updates, and specializes it to a one-dimensional case. It also fixes notation for constant-stepsize SGD with iterate averaging by splitting total samples into synthetic and real portions.

F Exact Synthetic-Benefit Condition

  • The exact comparison applies under the same real samples, label noises, and stepsizes, with two-stage training differing only through its synthetic-stage initialization wM.Thus, the real-stage update sequence is identical across the two procedures.Source: F Exact Synthetic-Benefit Condition · p. 53
  • The risk difference is obtained by expanding the two squared errors and canceling the common real-stage noise terms.The remaining terms depend on the propagated initial errors and the covariance of the synthetic-stage iterate.Source: F Exact Synthetic-Benefit Condition · p. 54
  • Two-stage training has lower expected real-stage risk than real-only training if and only if the derived condition comparing filtered initialization effects holds.The comparison cancels the identical real-stage noise accumulation and isolates the initialization-dependent terms.Source: F Exact Synthetic-Benefit Condition · pp. 53–54
  • In the one-dimensional Rademacher specialization, strict improvement is possible only when the stated synthetic-stage error condition is satisfied; if δ2 ≥ (w0 − θ)^2, it cannot hold for any M.For 0 < γt < 1, the real-stage multiplicative factor is strictly positive, so the condition is equivalent to the specialized inequality.Source: F Exact Synthetic-Benefit Condition · p. 55
  • The one-dimensional specialization uses H = E[x2] = 1 and reduces every real-stage update operator to 1 − γt.The synthetic iterate’s second-moment calculation follows from centered, independent synthetic noises and Rademacher covariates.Source: F Exact Synthetic-Benefit Condition · p. 55

G constant stepsize SGD with iterate averaging

  • For constant-stepsize SGD with iterate averaging, total sample size is denoted N, with pN synthetic samples and (1 − p)N real samples.This notation is introduced to simplify the subsequent analysis.Source: G constant stepsize SGD with iterate… · p. 55

G.1 Upper bound analysis … H Additional Experiments

The appendix derives finite-sample upper and lower excess-risk bounds for averaged constant-stepsize SGD, decomposing error into bias, variance, drift, and fluctuation terms. These bounds imply strong model collapse when synthetic data has a fixed proportion and effective dimension grows more slowly than total sample size.

G.1 Upper bound analysis

  • The upper-bound proof decomposes the error recursively, applies Cauchy–Schwarz, and derives separate bias and variance controls under the stated assumptions.The bias bound uses an effective-dimension characterization, while the variance bound follows the constant-stepsize averaged-SGD recursion.Source: §G.1 · pp. 56, 59
  • The drift analysis separates mean and centered fluctuation components, then bounds their averaged recursions using independence and fixed-budget source scheduling.The resulting uniform drift and schedule-fluctuation bounds contribute to the final excess-risk theorem.Source: §G.1 · pp. 57, 59, 61, 63
  • Theorem G.5 combines bias, variance, drift, and fluctuation bounds into an excess-risk upper bound for constant-stepsize SGD with iterate averaging.The variance recursion matches averaged-SGD analyses after accounting for the mixed-source noise level, while drift and fluctuation are bounded separately.Source: §G.1 · pp. 57, 60, 61, 63, 65

G.2 Lower bound analysis

  • The lower-bound analysis centers averaged SGD error around its mean and uses the nonnegative covariance contribution to lower-bound excess risk through the mean trajectory.The mean recursion is unrolled and averaged to obtain a closed-form lower bound.Source: §G.2 · pp. 66–67

G.3 Strong model collapse behavior

  • Fixed-proportion synthetic data yields strong model collapse as total sample size grows, under the stated stepsize, effective-dimension, and initialization conditions.The corollary combines the upper and lower bounds to establish the asymptotic collapse behavior.Source: §G.3 · pp. 67, 69
  • Collapse follows by showing optimization-dependent terms vanish while the persistent synthetic-induced mean term remains under fixed synthetic proportion.The proof uses the effective-dimension scaling condition and convergence of the averaging operator in the H-norm.Source: §G.3 · pp. 68–69

H.1 CIFAR10 CNN Experiments

CIFAR10 CNN experiments reproduce the theory: random mixing creates a synthetic-data-dependent loss floor, whereas two-stage training continues improving and performs slightly better at large sample sizes. The degradation under high synthetic proportions also grows with model width, especially for large datasets.

Experimental setup

  • Experiments use a five-layer CNN whose channel widths scale as (D, D, 2D, 2D, 4D), with D ∈ {8, 16, 32, 64, 128, 256}.Model size is varied through D on CIFAR10, which contains 50,000 training and 10,000 test images across 10 classes.Source: §H.1 · p. 69

Synthetic-data construction

  • Synthetic inputs match the real input distribution, but labels are fully synthetic soft distributions generated by a weak width-16 CNN teacher trained on 5,000 samples.The teacher is trained for five epochs and uses temperature scaling.Source: §H.1 · p. 69

Evaluation

  • Training varies synthetic proportion p from 0.0 to 0.9 and total samples T from 200 to 20,000, evaluating real-test classification error and cross-entropy over three seeds.Performance gaps are measured relative to pure real-data training at the same T.Source: §H.1 · pp. 69–70

Training protocols

  • Random mixing produces a loss floor that rises with synthetic proportion, while two-stage training continues improving and slightly outperforms mixing at large T.These results indicate that the irreducible bias is caused by mixing rather than synthetic data itself.Source: §H.1 · p. 70, Figure 3 · p. 70

Model-size effects

  • The performance gap between p = 0.9 and p = 0 increases with model width, particularly at large T, showing capacity-amplified synthetic-data bias.This trend holds for both test error and cross-entropy loss.Source: §H.1 · p. 71, Figure 4 · p. 71

H.2 Language Model Experiments

Language-model experiments on WikiText-103 reproduce the theoretical contrast between mixed and two-stage training: mixed synthetic data causes persistent validation-loss degradation, whereas synthetic pretraining can mitigate it. They further show that mixed-training degradation grows with model capacity, while teacher quality determines whether synthetic pretraining helps or hurts.

Protocol comparison

  • Mixed training remains worse than real-only training, while two-stage training approaches the real-only validation-loss curve at large budgets.This matches the predicted non-vanishing synthetic-data bias floor under mixed training and its mitigation when synthetic data is used only for initialization.Source: §H.2 · p. 71, Figure 5 · p. 72

Model capacity

  • Larger models exhibit larger synthetic-induced validation-loss gaps under mixed training at small and intermediate budgets.The comparison uses p = 0.9 against matched real-only baselines for models with 16.1M, 30.0M, and 51.1M parameters.Source: §H.2 · p. 72, Figure 6 · p. 72

Teacher quality

  • High-quality synthetic pretraining consistently improves validation loss, medium-quality data yields weaker gains at larger budgets, and low-quality data hurts after real-data training.The results support a bias-quality trade-off: synthetic data helps when its bias is sufficiently small, but poor-quality data introduces dominating mismatch.Source: §H.2 · pp. 72–73, Figure 7 · p. 73

Related papers

Loading 2609.09572v1…