PaperlayerView exact revision on arXiv

Source-linked AI summary

Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

Alexandre Declèves, Etienne Boursier, Nicolas Flammarion

arXiv:2609.34756v1stat.MLcs.LG

TL;DR

The paper examines how fine-tuning can use information in pretrained weights to reduce downstream data needs, a benefit whose theoretical basis remains poorly understood. It analyzes sparse regression and two-layer diagonal linear networks, showing that inherited support and signs shape the implicit bias and training path. For clean, correctly signed initialization it establishes exact recovery in the noiseless setting, while noisy recovery depends on stopping before new false positives appear.

  • Problem

    How pretrained information can reduce downstream sample requirements, and when fine-tuning can exploit it without learning spurious features, remains theoretically unclear.

    Source: §1 · p. 1

  • Method

    The paper analyzes sparse linear regression and two-layer diagonal linear networks to characterize how pretrained support and signs shape implicit bias and fine-tuning dynamics.

    Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 5

  • Results

    With clean, correctly signed initialization, noiseless fine-tuning exactly recovers the target with sample size of order s + m log(d), while noisy trajectories recover remaining signal before introducing new false positives.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In this setting, inherited coordinates can avoid the ambient-dimensional support-identification cost, reducing the data needed to recover a downstream task.

    Source: §5 · p. 10

    With noisy observations, the guarantee excludes new false positives but can allow inherited false positives, which cannot generally be removed by increasing sample size.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuning leverages information from pretraining remains poorly understood theoretically. We study fine-tuning from pretrained weights through the lens of sparse linear regression and two-layer diagonal linear networks. In our setting, pretraining provides information through the support (and signs) of the initialization predictor, which may contain coordinates relevant to the downstream task. We show how pretrained information reshapes the implicit bias and training dynamics, and can thereby reduce the sample complexity of recovering the target parameters and support. In particular, for a clean initialization with correctly inherited signs, we show that the required sample size is comparable to that of a weighted Lasso estimator that explicitly exploits the pretrained support through a suitably chosen regularizer. Our results thus show how information encoded in pretrained weights can be implicitly exploited by gradient-based fine-tuning, reducing the amount of data needed to recover a downstream task.

1 Introduction

The paper asks how pretrained information can reduce downstream sample needs without encouraging spurious features. In sparse regression and diagonal linear networks, it studies how inherited support and signs shape fine-tuning’s implicit bias and trajectory.

1 Introduction

  • The paper addresses a theory gap: much implicit-bias analysis starts from uninformative initialization, whereas fine-tuning begins with a structured predictor that may already identify relevant features.The motivating questions are what inherited information reduces sample needs and when it can be exploited without learning spurious features.Source: §1 · p. 1
  • The analysis makes the question concrete in sparse linear regression: inherited support can identify relevant coordinates, while m coordinates remain to be discovered and incur the ambient-dimensional logarithmic cost.A weighted Lasso that explicitly favors pretrained support supplies a benchmark for the fine-tuning guarantees.Source: §1 · p. 2
  • With clean, correctly signed initialization, noiseless fine-tuning exactly recovers the target with sample size of order s + m log(d); under noise, the trajectory recovers remaining signal before introducing new false positives.With an imperfect initialization, the noisy-setting support can differ from the true support only through inherited false positives.Source: §1 · p. 2

1.1 Related work

  • Prior work studies weighted sparse recovery and implicit bias in diagonal networks; this paper retains a nonzero pretrained predictor and quantifies how its support shapes fine-tuning.Related analyses also cover fine-tuning under source–target similarity, covariate shift, and feature resetting.Source: §1.1 · pp. 2–3

2 A minimal model of fine-tuning

The paper models pretraining as a fixed predictor whose support may partially identify downstream-relevant coordinates, then uses weighted Lasso as an explicit benchmark for exploiting that information.

2.1 Sparse regression and pretrained model

  • The downstream task is sparse linear regression with design X and response y, and the paper considers estimation and support recovery using the pretrained support.The target support is S⋆ = supp(β⋆), with sparsity s = |S⋆|.Source: §2.1 · p. 3
  • Pretraining is represented by a fixed predictor independent of the downstream design and noise; its support provides partial information about the sparse target, but inherited coefficients need not be accurately estimated.The clean case has Sinit ⊆ S⋆; more generally, initialization can also contain irrelevant coordinates.Source: §2.1 · p. 3

2.2 Weighted Lasso

  • The weighted-Lasso rate separates support-identification costs inside and outside the pretrained support, so inherited coordinates avoid the full ambient-dimensional logarithmic cost.For clean initialization, the oracle choice removes that logarithmic cost on inherited true coordinates, leaving the missing coordinates to incur it.Source: §2.2 · p. 4
  • Weighted Lasso explicitly favors the pretrained support, providing a statistical benchmark for gradient-based fine-tuning.Its tuning parameter α controls how strongly that support is favored.Source: §2.2 · pp. 3–4

3 Fine-tuning diagonal linear networks

The paper studies fine-tuning in two-layer diagonal linear networks, where a nonzero pretrained predictor shapes gradient-flow dynamics. In the vanishing-layer-imbalance regime, the predictor follows a mirror flow whose limiting path is described by saddle-to-saddle updates.

3.1 Parameterization and gradient flow

  • A two-layer diagonal linear network represents its predictor as the componentwise product of two layers, yielding nonconvex training dynamics despite a convex loss in the end-to-end predictor.Both layers are trained from an initialization representing the pretrained predictor.Source: §3.1 · p. 5
  • The analysis sends the layer imbalance μ to zero while keeping the pretrained predictor fixed, so the initialization is retained as the layer imbalance vanishes.The initialization conditions ensure training starts from the pretrained predictor and control layer imbalance.Source: §3.1 · p. 5

3.2 Mirror flow and the limiting fine-tuning path

  • The end-to-end predictor follows mirror flow for the convex loss, and its implicit bias depends on initialization through the reference point of the associated Bregman divergence.In the limiting regime, the mirror potential scaled by λμ approaches the ℓ1 norm, motivating an accelerated predictor path.Source: §3.2 · p. 5
  • The limiting path alternates constrained least-squares refits with dual-variable evolution; inherited coordinates begin at the boundary with their pretrained signs, while other coordinates begin at the center.Refits can activate or deactivate coordinates when a moving dual coordinate reaches a boundary.Source: §3.2 · p. 6

4 Statistical guarantees for fine-tuning

In sparse linear regression with two-layer diagonal linear networks, the section characterizes how inherited support and signs shape fine-tuning’s implicit bias and recovery path. Early stopping can recover downstream support with fewer samples under stated conditions, while inherited false positives and trajectory-control costs limit the guarantees.

4.1 Recovery in the noiseless setting

  • In the noiseless setting, the true predictor uniquely minimizes the leading implicit-bias objective, and gradient-flow endpoints converge to it as µ → 0.The leading penalty favors inherited signs, but does not distinguish inherited true coordinates from inherited false positives.Source: §4.1 · p. 7

4.2 Recovery in the noisy setting: ideal early stopping

  • At an ideal noisy-data stopping time, all true coordinates are recovered before any new null coordinate, with least-squares error scaling on S⋆∪Sinit and no ambient-dimensional log factor.When f0 = 0, the stopped trajectory achieves exact support recovery and the estimation rate associated with knowing the true support.Source: §4.2 · p. 8
  • The noisy sample-size bound includes m² + mf0 beyond s + f0 + m log d, terms the authors conjecture are removable; inherited false positives may remain.The guarantee excludes newly activated false positives but can retain coordinates inherited from initialization.Source: §4.2 · pp. 8–9

4.3 Recovery with the null-gradient stopping rule

  • A null-gradient stopping rule matches the oracle guarantee, detecting remaining true coordinates while rejecting a newly proposed false coordinate.It tests the next proposed coordinate only when that coordinate lies outside Sinit, using a threshold calibrated over possible trajectory faces.Source: §4.3 · p. 9

4.4 Comparison with Weighted Lasso

  • For clean, correctly signed initialization, both methods charge m log(d) for missing coordinates, but neither displayed sample-size scaling uniformly dominates the other.Beyond m log(d), the S2S bound contains s + m², while weighted Lasso contains (s − m) log(s − m).Source: §4.4 · pp. 9–10

5 Conclusion

The conclusion summarizes how pretrained initialization changes the implicit bias and training trajectory, reducing the statistical cost of sparse downstream learning in the analyzed setting. It also states the assumptions and extensions that bound the analysis.

5 Conclusion

  • Pretrained initialization lets inherited coordinates avoid ambient-dimensional support-identification costs, yielding exact recovery noiselessly and recovery before new false positives with an appropriately stopped noisy trajectory.The authors present this as a simple setting for precisely characterizing fine-tuning’s statistical benefit.Source: §5 · p. 10
  • The analysis assumes independent Gaussian design and a balanced low-noise regime for simplified noisy guarantees; correlated designs and finite layer imbalance remain open extensions.Correlated designs require additional control of projected gradients along the data-dependent trajectory; quantitative guarantees for finite μ > 0 remain to be established.Source: §5 · p. 10

AI use statement

  • AI tools assisted with writing, coding, literature identification, and exploration of mathematical arguments, while human authors developed, verified, and wrote the proofs.The authors say they critically evaluated and revised useful AI suggestions before incorporating arguments.Source: AI use statement · pp. 10–11

A Additional Results on the Saddle-to-Saddle Dynamics

These additional results derive the small-imbalance implicit-bias expansion and establish uniqueness properties for constrained saddles used in the limiting saddle-to-saddle dynamics.

A Additional Results on the Saddle-to-Saddle Dynamics

A.1 Implicit-bias expansion

  • As layer imbalance vanishes, the implicit-bias expansion gives inherited coordinates zero leading cost when their pretrained signs are preserved, but a penalty for sign reversal.Coordinates outside the inherited support incur the usual ℓ1 cost, while pretrained coefficient magnitudes enter only through a second-order correction.Source: §A.1 · p. 15

A.2 Signed constrained saddles

  • Under Gaussian design and Assumption 3, each signed face visited by the trajectory has a unique constrained minimizer, since its active design columns have full rank.Assumption 3 requires each visited face to have fewer active coordinates than observations.Source: §A.2 · pp. 15–16

A.3 Convergence to the saddle-to-saddle dynamics

This section establishes convergence of the rescaled mirror flow to the saddle-to-saddle path and characterizes conditions for recovery along clean and contaminated faces. It also gives uniform and balanced low-noise recovery conditions, including requirements for recovering true coordinates with correct signs and estimating on the recovered support.

A.3 Convergence to the saddle-to-saddle dynamics

  • Almost surely, the rescaled mirror flow converges uniformly on compact time sets away from event times to the unique saddle-to-saddle trajectory.Every subsequential limit visits the same constrained saddles with the same dual evolution and event times.Source: §A.3 · pp. 16–17

A.4 Noiseless recovery

  • Under the stated sample-size condition, the target is the unique minimizer of the initialization-dependent regularizer among interpolating solutions, and endpoint limit points minimize that regularizer.The proof uses the noiseless interpolation set and identifies the dominating term of the Bregman divergence as the regularizer.Source: §A.4 · pp. 17–18

B Statistical Preliminaries for Saddle-to-Saddle Recovery

  • The statistical preliminaries define constrained saddles on fixed faces and collect Gaussian estimates there, postponing uniform control over possible trajectory faces.They use true-support and inherited false-positive coordinates to index the faces and their residuals and gradients.Source: B Statistical Preliminaries for… · p. 18

B.1 Contaminated faces and residual representation

  • On a contaminated face, the residual is independent of columns for null coordinates not inherited from initialization, and its norm admits a high-probability bound.The residual representation applies when the constrained saddle lies in the relative interior of its signed face.Source: §B.1 · p. 18

B.2 Gradient bounds on a fixed contaminated face

  • On a fixed contaminated face, the lemmas bound conditional Gaussian maxima, lower-bound an inactive true-coordinate gradient, and control gradients of non-inherited null coordinates.The estimates require lower bounds on the residual rank for the stated probability guarantees.Source: §B.2 · pp. 19–20

C Proof of Recovery under Ideal Stopping

  • Missing true coordinates begin at the center of the dual interval and travel distance one, while wrongly signed inherited true coordinates must travel distance two after removal.The recovery proof uses these travel distances for coordinates in R0.Source: C Proof of Recovery under Ideal Stopping · p. 20

C.1 From gradient separation to correct arrivals

  • When effective-score separation holds at each contaminated saddle, the next effective arrival is a missing true coordinate reaching the dual boundary with the correct sign.The argument compares normalized progress of remaining true coordinates against that of new null coordinates.Source: §C.1 · p. 21

C.2 Uniform recovery conditions

  • Proposition 3 verifies uniform purification, stability, and detectable correct arrivals across contaminated faces using the fixed-face estimates.The conditions range over subsets of missing true coordinates and inherited false positives, including their active sign patterns.Source: §C.2 · p. 21
  • On the uniform stability event, active true coordinates retain their correct signs, and Conditions (C1) and (C3) hold with the stated high-probability guarantees.The proof establishes (C1) with additional failure probability at most δ and (C3) outside an event of probability at most δ.Source: §C.2 · pp. 23–24

C.3 Recovery on the uniform event

  • Under Conditions (C1)–(C3), every true coordinate becomes active with its correct sign in finitely many events, before the stopping rule rejects a proposed coordinate outside S⋆∪Sinit.Recovery takes at most m effective true-coordinate arrivals.Source: §C.3 · p. 25

C.4 Balanced low-noise regime

  • Under Assumption 1, Proposition 4 gives a balanced low-noise sample-size condition sufficient for Conditions (C1)–(C3) to hold simultaneously with probability at least 1−δ.The constant CL depends only on L.Source: §C.4 · p. 25

C.5 Estimation after recovery

  • For a recovered support between S⋆ and S⋆∪F0, the constrained-saddle least-squares estimator satisfies an estimation bound with probability at least 1−δ when n≥C(s+f0+log(1/δ)).The result applies to any such recovered support.Source: §C.5 · p. 26

C.6 Proof of Theorem 2

The proof addresses data-dependent stopping by calibrating null gradients uniformly over contaminated faces, then uses threshold separation to ensure recovery precedes rejection of a new null proposal.

C.6 Proof of Theorem 2

  • Under the theorem’s sample-size assumption, Conditions (C1)–(C3) hold simultaneously with probability at least 1 −δ.The proof obtains the bound by combining its probability events.Source: §C.6 · p. 27

D Null-Gradient Early Stopping

  • Because the visited saddle depends on the data, the stopping rule requires null-gradient control simultaneous over contaminated faces, not merely at a fixed saddle.The proof uniformizes the fixed-saddle bound over possible signed faces.Source: D Null-Gradient Early Stopping · p. 27, §D.1 · p. 28

D.1 Null-gradient calibration

  • At a fixed saddle, conditionally independent Gaussian null columns yield a calibrated null-gradient threshold with unconditional failure probability at most η.The calibration assumes the residual and null-column set are measurable with respect to the conditioning sigma-field.Source: §D.1 · p. 27

D.2 Uniform separation around the null threshold

  • With probability at least 1 −δ −η, uniformly over the stated faces and sign patterns, new true proposals lie above the null threshold before recovery and new null proposals lie below it afterward.The separation holds under the sample-size requirement of Proposition 3(iii), with the stated logarithm replacement.Source: §D.2 · p. 28

D.3 The non-stopping rule

  • With probability at least 1 −δ −η, the stopping rule reaches full recovery and rejects the first genuinely new null proposal.Coordinates inherited from initialization bypass the gradient test; this avoids stopping on their potentially repeated activation–deactivation cycles.Source: §D.3 · p. 30

D.4 Proof of Theorem 3

Theorem 3’s proof combines recovery and stopping events, while the stopping calibration can use an upper bound on path complexity at the cost of a conservative fully observable choice.

D.4 Proof of Theorem 3

  • The required sample size is s + f0 + m^2 + mf0 + m log d after increasing CL if necessary; a union bound over three events gives the result with probability at least 1 −δ.Source: §D.4 · pp. 30–31

D.5 Using an upper bound on the path complexity

  • A known upper bound B ≥ m + f0 can replace the oracle path-complexity quantity in the stopping threshold without changing the proof.In the balanced regime, the replacement changes only universal constants; the fully observable choice B = d gives the conservative requirement n ≳ md.Source: §D.5 · p. 31

E Proof of the Weighted-Lasso Benchmark

  • The weighted-Lasso benchmark proof defines m_miss as the number of target-support coordinates missing from initialization and a0 as the overlap size.It also defines sα := m_miss + a0/α^2.Source: E Proof of the Weighted-Lasso Benchmark · p. 31

E.1 Weighted geometry and estimation bounds

The weighted-Lasso analysis controls weighted geometry and the stochastic score, then uses a weighted-cone argument to establish an estimation bound.

E.1 Weighted geometry and estimation bounds

  • For n ≥ 2 log(2/η), the weighted Gaussian score satisfies ∥z∥∞,α ≤ Cσ√nLα,η with probability at least 1 −2η.This score control supplies the event used in the weighted-Lasso estimation lemma.Source: §E.1 · p. 34
  • When λ ≥ 2∥z∥∞,α, the weighted-Lasso argument places the estimation error in the weighted cone and applies the lower bound κα∥∆∥2 ≤ ∥X∆∥2.The passage states that this cone inequality proves the estimation result.Source: §E.1 · pp. 34–35
  • Under the stated event, the weighted geometry quantities φα and κα are bounded below by numerical constants, while λ is at least CσLα,η/√n.These bounds are used to set up the weighted-Lasso estimation argument.Source: §E.1 · p. 35

E.2 Exact signed support recovery

The exact-support argument combines a Gram-matrix event, inactive-coordinate controls, and a beta-min condition in a primal-dual witness proof.

E.2 Exact signed support recovery

  • On the true support S⋆, the Gram matrix obeys 1/2 I ⪯ H_{S⋆,S⋆} ⪯ 3/2 I; the proof uses upper and lower singular-value bounds for the Gaussian design.Source: §E.2 · p. 35
  • Under the Gram event, the stated inactive-coordinate bounds, and the beta-min condition, the weighted Lasso recovers the exact signed support.The primal-dual witness fixes the true support and signs, solves the KKT equations there, and checks strict dual feasibility outside it.Source: §E.2 · pp. 35, 37
  • Combining the preceding lemmas gives exact signed recovery with probability at least 1 − Cη whenever the stated conditions hold.The proof uses inactive-noise and irrepresentability bounds, active-coordinate control, and the beta-min assumption to verify the KKT conditions.Source: §E.2 · pp. 36–38

E.3 Choice of the weight parameter

The weight optimization identifies an oracle choice that, with a clean initialization, leaves inherited coordinates unpenalized and charges only missing coordinates for support search. This choice depends on the unknown target support, and exact recovery also requires the stated conditions.

E.3 Choice of the weight parameter

  • When Sinit ⊆ S⋆, the oracle choice αor = +∞ leaves initialized coordinates unpenalized, so only missing coordinates contribute to support-search complexity.Here, mmiss = |S⋆\ Sinit|.Source: §E.3 · p. 38
  • The sharper oracle choice depends on the unknown support S⋆.Source: §E.3 · p. 38
  • When Sinit = S⋆, the optimized weighted contribution vanishes, but the design event still requires n ≳ s + log(1/δ).Source: §E.3 · p. 38
  • Under the stated conditions, including the beta-min condition, the weighted Lasso recovers the target support and its signs exactly.Source: §E.3 · p. 39
  • In the noiseless case, the constrained weighted-ℓ1 minimizer is unique and coincides with the limiting estimator.Source: §E.3 · p. 39

F Experiments

The experiments illustrate early stopping along saddle-to-saddle trajectories and compare exact support recovery as the number of missing true coordinates varies. The stopping rule rejects a null proposal in the illustrated instance, while S2S recovery declines more sharply with more missing coordinates than weighted Lasso recovery.

F Experiments

  • Trajectory plots use saddle index rather than accelerated time because consecutive hitting-time intervals can differ substantially.Source: F Experiments · p. 40
  • At saddle k = 33, test loss reaches its oracle minimum before rising sharply; in the second instance, the first null proposal falls below threshold and is rejected.The four missing true coordinates are proposed before any null coordinate.Source: Figure 2 · p. 40, F Experiments · p. 40
  • With clean initialization, S2S exact-support recovery is high when few true coordinates are missing but drops sharply as m increases; weighted Lasso degrades more gradually.The comparison uses theoretically calibrated procedures rather than cross-validation-tuned regularization.Source: Figure 3 · p. 41, F Experiments · p. 41
Loading 2609.34756v1…