Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

Jichu li; Difan Zou

Source-linked AI summary

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

Jichu li, Difan Zou

arXiv:2609.09572v1Sep 9, 2026stat.MLcs.AIcs.LG

TL;DR

Limited human-generated data motivates synthetic-data training, but its generalization effects under iterative optimization remain unclear. The paper analyzes one-pass SGD in high-dimensional linear regression with model shift using finite-sample bounds and scaling laws for mixed and two-stage protocols. Mixed training exhibits strong model collapse, whereas two-stage training avoids the risk floor and can outperform real-only training under an exact finite-sample condition.

Problem
Synthetic data may address limited high-quality training data but can induce model collapse, motivating analysis of its generalization under practical one-pass SGD.
Method
The paper studies one-pass SGD in high-dimensional linear regression with shared covariates and mismatched labeling functions under mixed and two-stage training.
Results
Mixed training produces a non-vanishing excess-risk floor, while two-stage training avoids it; scaling laws and an exact condition further characterize degradation and improvement.
Takeaways & Limitations
Synthetic data is neither inherently harmful nor beneficial: its effect depends on data quality and how the training protocol incorporates it.
Takeaways & Limitations
The analysis focuses on one-pass SGD with shared covariates; tight characterizations remain open for data-limited and two-stage settings.

Abstract

from arXiv · show

Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Dohmatob et al., 2024), where any fixed fraction of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. In this paper, we study how synthetic data affects the generalization of one-pass SGD in high-dimensional linear regression with model shift. We establish finite-sample risk bounds for mixed and two-stage training, separating standard bias and variance from source-mismatch effects, namely fluctuation and persistent drift under mixing and filtered initialization bias under two-stage. These bounds reveal a sharp contrast: mixed training induces strong model collapse, while two-stage training avoids the floor by using synthetic data only in the first stage, showing that collapse is not inevitable under a simple data curriculum. Under a random sketch model, we further obtain scaling laws for both protocols, with tight results for mixed training in the optimization-saturated regime. These laws show that larger models may amplify synthetic-induced degradation under mixing, and quantify how high-quality synthetic pretraining may reduce bias in two-stage training. Finally, we establish an exact finite-sample necessary-and-sufficient condition for two-stage training to strictly outperform real-only training under the same real-data budget and identical real-stage updates. Overall, our results highlight that synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality and the training protocol used to incorporate it.

1 Introduction

Synthetic data can expand limited training resources but may also cause model collapse. This paper studies that trade-off for one-pass SGD and finds that training protocol critically determines whether synthetic data harms or helps generalization.

Synthetic data supplements scarce human-generated data, but repeated use can cause model collapse through overfitting to insufficiently representative synthetic patterns.
Prior theory largely studies static estimators and asymptotic regimes, leaving finite-sample optimization dynamics under SGD less characterized.
Finite-sample bounds show that mixed training produces a non-vanishing excess-risk floor, establishing strong model collapse under one-pass SGD.
Two-stage training avoids the excess-risk floor by using synthetic data first and real data afterward, showing collapse is not inevitable.
Random-sketch scaling laws show that larger models may amplify synthetic-induced degradation under mixing, while high-quality synthetic pretraining may reduce bias in two-stage training.
An exact finite-sample condition identifies when two-stage training strictly outperforms real-only training under the same real-data budget and real-stage updates.

2 Related Work

Related theory frames synthetic-data learning as regression under distribution mismatch, while adjacent literatures study multiple sources and SGD separately. Most prior work does not characterize iterative optimization dynamics in finite samples.

Theoretical model-collapse studies analyze regression under mismatch between real and synthetic distributions, including degradation and mitigation strategies.
Multiple-source learning has been studied in transfer learning, domain adaptation, and continual learning under covariate or model shift.
Most related work focuses on statistical estimators rather than the finite-sample dynamics of iterative optimization procedures.
SGD generalization in linear regression is well studied, including one-pass last-iterate and averaged schemes in overparameterized settings.

3 Preliminaries

The paper models real and synthetic supervision as two linear sources sharing covariates but differing in labeling functions, then studies one-pass SGD under mixed and two-stage protocols. Performance is measured by excess risk on the real distribution.

The setup uses feature vectors in a finite- or countably infinite-dimensional Hilbert space and linear regression with population risk defined over a data distribution.
The analysis assumes finite covariance quantities, finite parameter H-norms, and fourth-moment upper and lower conditions.
Real and synthetic distributions share the same feature marginal but differ in labeling functions, isolating model mismatch between sources.
Training protocols: Mixed training randomly orders M synthetic and N real samples, with synthetic fraction p := M/(M + N).
Training protocols: Two-stage training processes M synthetic samples before N real samples, whereas mixed training combines both sources in one random sequence.
Optimization: One-pass SGD uses a geometrically decaying stepsize schedule, applied across all mixed-training updates or separately within each two-stage phase.
Evaluation: The algorithm outputs the last iterate, and the paper evaluates excess risk on the real distribution.

4 Main Results

Mixed training creates a persistent mismatch-driven excess-risk floor, whereas two-stage training avoids that floor by using synthetic data only to initialize real-data training. Scaling-law results further show that model size can amplify mixed-training degradation, while high-quality synthetic pretraining can reduce two-stage bias.

Mixed training: Mixed training decomposes synthetic-data degradation into fluctuation and drift errors alongside standard bias and variance.The fluctuation term behaves similarly to standard variance, while drift captures persistent movement toward the synthetic model.
Mixed training: A fixed synthetic-data proportion produces a strictly positive excess-risk floor as total sample size grows, establishing strong model collapse under SGD.The floor is proportional to p^2∥δ∥^2 and is unavoidable in general through matching upper and lower bounds.
Mixed training: The collapse persists with decaying stepsizes and iterate averaging, indicating that sample mixing—not a particular optimization scheme—drives the phenomenon.The result is established through finite-sample SGD bounds and also appears for constant-stepsize SGD with iterate averaging.
Two-stage training: Two-stage training eliminates the non-vanishing floor and yields vanishing excess risk under mild conditions by applying real-data updates after synthetic pretraining.The synthetic phase determines initialization, so it does not create persistent drift throughout the later optimization trajectory.
Two-stage training: Synthetic data benefits two-stage training primarily by reducing bias rather than variance, replacing random initialization error with source mismatch.High-quality synthetic data makes the mismatch small and can place the model closer to the real target before second-stage optimization.
Random sketch scaling laws: Under random-sketch scaling laws, larger models improve real-signal approximation but may amplify synthetic-induced degradation in mixed training.The approximation term decreases with sketch dimension while the drift-floor term increases toward order p^2.

5 Experiments

Experiments in high-dimensional linear regression and real-world datasets test the predicted effects of synthetic data under mixed and two-stage training. The results qualitatively match the theory, including synthetic-induced error floors and conditions under which synthetic data helps.

Gaussian high-dimensional regression: Varying synthetic proportion, sample size, and sketch width tests the predicted synthetic-induced error floor.The first experiment varies p ∈ (0, 1), T ∈ [50, 50000], and sketch width D.
Gaussian high-dimensional regression: A second experiment varies synthetic quality and synthetic sample size to study when synthetic data helps.The synthetic quality parameter is b, and the synthetic sample size is M.
Experimental protocol: Experiments are repeated with 5 random seeds, with the two parts reported in Figures 1 and 2.
Real-world data: CIFAR10 and WikiText-103 experiments with a 5-layer CNN and language-model training show the same qualitative behavior predicted by the theory.Synthetic data is generated by a pretrained teacher model.

6 Conclusion

The paper shows that synthetic data’s effect depends on training protocol and quality: mixed training exhibits model collapse, whereas two-stage training avoids a persistent risk floor. Scaling-law and finite-sample analyses further identify when synthetic data amplifies degradation or improves initialization.

Finite-sample risk bounds analyze one-pass SGD with synthetic and real data under mixed and two-stage training.
Mixed training produces strong model collapse through source-mismatch effects, including fluctuation and persistent drift.
Two-stage training avoids the non-vanishing excess-risk floor by using synthetic data only before real-data training.
Random-sketch scaling laws show that larger models may amplify synthetic-induced degradation under mixing, while higher-quality pretraining may reduce two-stage bias.
The analysis is limited to one-pass SGD with shared covariates, and tight characterizations remain open outside mixed training’s optimization-saturated regime.

B Mixed Training Upper Bound Analysis

The mixed-training analysis models a fixed-budget random source schedule and decomposes excess risk into bias, variance, drift, and fluctuation components. Separate finite-sample bounds control these terms under covariance and stepsize assumptions.

The fixed-budget protocol uniformly shuffles pN synthetic and (1−p)N real samples into a random source schedule.Conditional on the schedule, covariates are fresh and source-specific noises are sample-wise conditionally independent.
Shared covariates make the current covariate independent of the past filtration, allowing exact covariance and noise recursions.
The excess-risk analysis separates bias, variance, drift, and fluctuation contributions generated by the mixed SGD dynamics.
Finite-sample lemmas provide separate upper bounds for bias, variance, drift, and fluctuation under stepsize restrictions.
The drift term is further decomposed into mean and fluctuation parts, with the fluctuation recursion taking the form of a variance recursion.

C.4 Proof of Corollary 4.3

The proof establishes the mixed-training asymptotic collapse result by combining finite-sample bounds with spectral contraction under the geometric stepsize schedule. The resulting lower and upper bounds isolate the persistent mismatch contribution.

As T grows with fixed synthetic proportion p=M/(M+N), the effective-dimension terms vanish under the stated consistency condition.
Spectral contraction shows that the real-data optimization filter vanishes in H-norm along every eigendirection with positive eigenvalue.
Combining the upper and lower bounds isolates a persistent synthetic mismatch contribution after the decaying optimization terms disappear.
The two-stage proof conditions on the synthetic-stage iterate and applies the real-only SGD bound to the subsequent real-data phase.
The resulting two-stage upper bound is obtained by relating the first-stage error to the real-stage initialization term.

D.1 Two-stage training without learning-rate restart

Without learning-rate restart, the real stage uses the tail of the global schedule, slowing finite-sample forgetting of synthetic initialization. Nevertheless, the asymptotic two-stage conclusion remains unchanged: no persistent synthetic drift or excess-risk floor appears.

Without restart, the real-data stage uses the tail of the global T-step geometrically decaying schedule rather than a fresh N-step schedule.
The two-stage bound changes through the contraction operator and spectral cutoff defined by the real-data tail schedule.
The uninterrupted schedule slows finite-sample forgetting of synthetic initialization but does not create persistent synthetic drift or a non-vanishing excess-risk floor.
The sketch-based analysis uses the sketched covariance SHS⊤ and studies approximation, fluctuation, and excess-risk terms under spectral assumptions.
In the optimization-saturated regime, the lower bound matches the upper bound up to constants.

E.3 Two-stage Training Analysis

The two-stage analysis derives high-probability finite-sample bounds for first-stage synthetic training followed by real-stage SGD. Under the random sketch model, the bounds characterize excess risk and become tight in the data-limited regime.

First-stage analysis: The analysis conditions on the sketch matrix and studies first-stage excess risk through the spectrum of the sketched covariance operator.The first-stage estimator and effective sample size are defined after conditioning on S.
Bias bound: E_Bias^2 is bounded by k M_eff^γ + k^(1−a) when b ≥ a.The bound follows from the eigenvalue decay assumptions and a cutoff k ≤ D/3.
High-probability guarantees: The resulting upper bounds hold with probability at least 1 − e^−Ω(D) over the sketch matrix.The high-probability statements combine spectral and excess-risk events under the stated step-size and effective-sample-size conditions.
Tightness: In the data-limited regime, (M_effγ)^(1/a) ≤ D/c_0, the upper bound is tight up to constants.This identifies the regime where the random-sketch scaling law matches the corresponding lower bound.

F Exact Synthetic-Benefit Condition

The section compares two-stage and real-only training under identical real-stage samples, noises, and step sizes. It derives an exact finite-sample condition for two-stage training to have strictly lower expected risk.

Risk comparison: The two procedures have identical real-stage noise accumulation because they use the same real samples, label noises, and step sizes.Their risk difference therefore depends on the real-stage propagation of the different initializations, not on different real-stage noise terms.
Exact condition: Two-stage training strictly improves expected risk if and only if the synthetic-stage initialization satisfies the derived quadratic-form inequality.The condition follows after expanding both squared errors and cancelling the common real-stage terms.
One-dimensional specialization: In the one-dimensional case, the real-stage multiplicative factor is strictly positive, so the same inequality remains necessary and sufficient.The scalar specialization uses H = E[x^2] = 1 and 0 < γ_t < 1.

G.1 Upper bound analysis

The upper-bound analysis decomposes averaged-SGD excess risk into bias, variance, drift, and fluctuation contributions. It combines these components into a finite-sample bound and then derives strong model collapse for fixed synthetic proportions.

Variance: The variance recursion has the same form as averaged SGD with a single-source covariance, with the noise level adjusted for mixed sources.The mixed-source recursion is bounded using the corresponding averaged-SGD variance analysis.
Drift and fluctuation: The drift process is separated into mean and centered fluctuation components before bounding their contributions to excess risk.This decomposition supports separate control of persistent source mismatch and schedule-related fluctuations.
Excess-risk decomposition: The excess-risk bound combines bias, variance, drift, and fluctuation terms for constant-step-size SGD with iterate averaging.The theorem uses effective sample size and synthetic-proportion-dependent noise under the stated assumptions.
Lower bound: The combined upper bound is matched with a lower bound obtained from the mean error trajectory.The lower-bound argument centers the averaged error and retains its nonnegative covariance contribution.
Model collapse: Under the stated assumptions, averaged SGD exhibits strong model collapse when D_eff = o(M + N) and the synthetic proportion remains fixed as total samples grow.The corollary establishes the asymptotic collapse from the combined upper and lower bounds.

H.1 CIFAR10 CNN Experiments

The CIFAR10 experiments compare random mixing with two-stage training using a CNN and weak-teacher synthetic labels. They evaluate real-test classification error and cross-entropy across synthetic proportions, sample sizes, and model widths.

Experimental setup: Synthetic labels use a smaller CNN teacher trained on 5,000 real samples, while the student CNN varies in width.The teacher produces temperature-scaled soft labels for the synthetic examples.
Experimental setup: The experiments allocate pT synthetic and (1 − p)T real samples, varying p from 0.0 to 0.9 and T from 200 to 20,000.For model-size comparisons, p is fixed at 0.9 while width varies.
Training protocols: Under random mixing, test loss saturates as T increases, with higher synthetic proportions producing higher asymptotic error.This forms a synthetic-data-induced performance floor on the real distribution.
Training protocols: Under two-stage training, test loss continues decreasing with T and is slightly better than mixing in the large-sample regime.The experiments attribute the floor to mixing rather than to an inherent limitation of the synthetic data itself.
Model size: The performance gap between p = 0.9 and p = 0 increases with model size, especially at large sample sizes.The trend holds for both test error and cross-entropy loss, indicating stronger synthetic-induced bias for larger models.

H.2 Language Model Experiments

Language-model experiments compare mixed and two-stage synthetic-data training, model capacity, and teacher quality under real-data evaluation. Mixed training degrades validation loss relative to real-only training, especially with larger models, while high-quality synthetic pretraining can improve outcomes under a fixed real-data budget.

Protocol comparison: Mixed training remains worse than real-only training, whereas two-stage training reduces degradation and approaches the real-only curve at larger budgets.The comparison uses synthetic fractions p ∈{0, 0.3, 0.6, 0.9} and budgets from 100 to 24,000 steps.
Model capacity: 16.1M, 30.0M, and 51.1M parameter models are compared using ∆loss = ValLoss(p = 0.9) − ValLoss(p = 0).The metric isolates synthetic-data-induced degradation from baseline improvement due to model size.
Model capacity: Larger models exhibit a larger degradation gap at small and intermediate budgets under mixed training.The result supports the prediction that higher-capacity models fit biased synthetic signals more strongly.
Teacher quality: High-quality synthetic pretraining improves validation loss relative to real-only training, with larger gains as the synthetic pretraining budget increases.The teacher-quality experiment holds the downstream real-data budget fixed and evaluates real validation loss.
Teacher quality: Medium-quality synthetic data yields weaker positive gains at larger budgets, while low-quality synthetic data produces a positive loss gap after subsequent real-data training.The results illustrate a bias-quality trade-off in two-stage synthetic pretraining.