Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Empirical findings

• Average ID accuracy rises from 83% to 85% with fine-tuning, but OOD accuracy falls from 66% to 59% relative to linear probing across ten benchmarks.The benchmark comparison uses good pretrained features from MoCo-v2 and CLIP; Figure 1 summarizes the transfer setup and overall finding.Source: §1 · p. 2, Figure 1 · p. 2

Theory

• Fine-tuning distorts pretrained features by changing ID representations more than OOD representations, so its jointly optimized head performs poorly on less-changed OOD features.The theory proves this effect in overparameterized two-layer linear networks with good pretrained features, even with an infinitesimally small learning rate.Source: §1 · pp. 2–3
• Linear probing extrapolates better OOD by preserving pretrained features, but sacrifices ID accuracy because frozen features cannot adapt to the downstream task.Fine-tuning can underperform linear probing on directions outside the training-data span, while early stopping does not resolve the OOD gap.Source: §1 · pp. 2–3

Algorithmic implications

• LP-FT first learns a good head by linear probing and then fine-tunes all parameters, reducing feature changes and outperforming both baselines on ID and OOD accuracy.The initial probing step is relatively cheap, so LP-FT uses similar compute to vanilla fine-tuning; it also improves both metrics on CIFAR-10.1 despite its small shift.Source: Figure 1 · p. 2, §1 · p. 3

Empirical validation

• Fine-tuning never matches linear probing’s OOD accuracy throughout training when pretrained features are good and the distribution shift is large, but can win when features are weaker or datasets are close.LP-FT changes ID and OOD features 10 × −100× less than fine-tuning and achieves the best accuracies on both metrics.Source: §1 · p. 3

Task and evaluation

• Predictors are evaluated using ID loss on new samples from Pid and OOD loss on samples from a distinct distribution Pood.Both evaluations use a specified loss function ℓ.Source: §2 · pp. 3–4

Models

• The model combines pretrained features gB(x) with a linear head v, with deep-network features in experiments and linear projections in theory.The predictor is parameterized as fv,B(x) = v^TgB(x).Source: §2 · p. 4
• Linear probing freezes the pretrained extractor B0 while fine-tuning updates both the head v and extractor B from initialization B0 using gradient descent on Pid data.The pretrained extractor may have been trained on large amounts of unlabeled or weakly supervised inputs from both Pid and Pood.Source: §2 · p. 4

3.1 Linear overparameterized setting

• Under an overparameterized linear model, fine-tuning is analyzed with gradient flows from pretrained initialization, despite non-convex optimization and test performance not being determined by training loss alone.The setting uses squared-loss regression with predictors fv,B(x) = v⊤Bx, overparameterized dimensions, and training examples spanning a subspace S.Source: §3 · p. 4, §3.1 · pp. 4–5

3.2.1 Key intuitions

• Fine-tuning distorts features because gradient updates change representations in the training-data subspace but leave orthogonal directions unchanged, causing errors outside that subspace.In the two-dimensional illustration, both methods fit ID data, but fine-tuning’s direction-specific updates create large errors perpendicular to the ID data.Source: §3.1 · p. 6, §3.2.1 · p. 7

3.2.2 General result on the OOD error of fine-tuning

• Fine-tuning has positive OOD error at every time when pretrained features are perfect and their rowspace has nonzero alignment with directions outside the training span.Theorem 3.3 lower-bounds fine-tuning’s OOD error under cos θmax(R0, S⊥) > 0; with perfect features, linear probing achieves zero OOD error.Source: §3.2.2 · pp. 7–8
• When the head starts randomly or at zero, coupled head–feature updates create high OOD error; perfect head alignment can avoid this increase.The bound worsens with head alignment error, because changing the head substantially forces feature changes through a balancedness invariant.Source: §3.2.2 · p. 8

3.3 Linear probing vs. fine-tuning

• As pretrained-feature error approaches zero, linear probing’s OOD loss-to-fine-tuning loss ratio approaches zero under non-degeneracy, whereas fine-tuning achieves lower ID error when features are imperfect.Fine-tuning can update features to fit ID data, while linear probing cannot perfectly fit data when no head on the pretrained representation does so.Source: §3.3 · pp. 9–10

3.4 Linear probing then fine-tuning: a simple variant to mitigate tradeoffs

• The proposed mitigation is to initialize fine-tuning’s head with a linear-probing solution, improving alignment with the optimal head and reducing feature distortion.The motivation follows from the coupling between head and feature updates: better initialization should let fine-tuning retain low OOD error while adapting features.Source: §3.4 · p. 10
• With perfect pretrained features and non-degenerate alignment, randomly initialized full fine-tuning retains positive OOD loss throughout its trajectory.Proposition 3.7 formalizes this case and motivates studying well-initialized full fine-tuning as a way to avoid distortion.Source: §3.4 · p. 10
• For imperfect features, fine-tuning is difficult to analyze because its iterates generally have no closed-form expression, and the proof provides only a lower bound when the initial and optimal heads differ.The analysis applies when d(B0, B⋆) > 0; an upper bound was not established.Source: §3.4 · p. 10

4.1 Linear probing vs. fine-tuning

• Fine-tuning averages 85.1% ID versus 82.9% for linear probing, but linear probing averages 66.2% OOD versus 59.3% for fine-tuning.Linear probing wins 8/10 OOD datasets, while fine-tuning wins 5/6 ID datasets; the exceptions are the closely matched CIFAR-10.1 and ImageNetV2 shifts.Source: §4.1 · p. 12, Table 1 · p. 12, Table 2 · p. 13

4.2 Linear probing then fine-tuning (LP-FT)

• LP-FT achieves the best average accuracy ID (85.7%) and OOD (68.9%), winning 5/6 ID and 10/10 OOD datasets.It matches or exceeds both baselines on all ten OOD datasets; FMoW is the sole ID exception, where it falls between linear probing and fine-tuning.Source: §4.2 · p. 12, Table 1 · p. 12, Table 2 · p. 13

4.3 Examining the feature distortion theory

• Even with OOD-based early stopping, linear probing remains more accurate than fine-tuning, averaging 67.1% versus 61.3%.This supports feature distortion throughout fine-tuning rather than only at its endpoint.Source: §4.3 · p. 13
• Fine-tuning distorts features more for ID than OOD examples, while LP-FT changes both by 20× less on Living-17.Feature distances are 0.0188 ± 0.0001 ID and 0.0167 ± 0.0001 OOD after fine-tuning, versus 0.0011 ± 0.0001 and 0.0009 ± 0.0001 with LP-FT.Source: §4.3 · p. 13
• When pretrained features are worse, fine-tuning beats linear probing OOD (71% vs. 66%); when ID and OOD are similar, it also wins (92.3% vs. 82.7%).LP-FT remains strongest in the similar-shift case, reaching 93.5% OOD, confirming that feature quality and shift magnitude condition the tradeoff.Source: §4.3 · pp. 13–14

Fine-tuning vs. linear probing

• LP can outperform FT when high-quality pretrained features face large distribution shifts, contrasting with FT’s usual in-distribution advantage.Prior studies generally find FT better in-distribution, making FT the default accuracy-improvement method while LP is often used to study representations (Chen et al., 2021a; Zhai et al., 2020; Chen et al., 2020b; Peters et al., 2018).Source: §5 · p. 14

The benefit of preserving pretrained features

• Lightweight adaptation preserves pretrained features and can improve OOD performance; this work strengthens prior evidence by finding that FT never surpasses LP during training.Related work includes zero-shot prompting and lightweight NLP adaptation, while Andreassen et al. (2021) report that ID accuracy rises as OOD accuracy plateaus.Source: §5 · p. 14

Mitigating ID-OOD tradeoffs

• LP-FT addresses the ID–OOD tradeoff theoretically and empirically without extra unlabeled data, unlike prior self-training approaches; weight ensembling offers another concurrent alternative.LP-FT had previously appeared as a fine-tuning heuristic (Levine et al., 2016; Kanavati & Tsuneki, 2021; fastai), but not specifically for robustness or OOD accuracy.Source: §5 · p. 14

Theoretical analysis of transfer learning

• Unlike prior transfer-learning theory focused mainly on LP, underparameterized FT, or ID error, this work analyzes overparameterized fine-tuning trajectories and OOD error.Because overparameterized objectives lack a unique optimizer, the analysis follows training from pretrained initialization, reflecting modern zero-training-loss settings.Source: §5 · p. 14

6 Conclusion.

• Preserving pretrained features and using linear probing can improve OOD performance, while the gap versus fine-tuning grows as pretrained feature quality improves.The authors expect this issue to become more significant as pretraining innovations and scale continue to improve.Source: §6 · p. 15
• The paper contributes tools for analyzing trajectories from specific initializations when multiple global optima create implicit regularization effects.Open extensions include nonlinear activations, layerwise learning rates, and explicit regularization.Source: §6 · p. 15
• LP-FT mitigates the ID–OOD tradeoff in this context and may extend to settings such as CLIP by initializing the final layer before full fine-tuning.The authors present LP-FT as a first step toward methods that leverage powerful pretrained models.Source: §6 · p. 15

A Proofs for Section 3

• Appendix A contains proofs for the theoretical results, while Appendix B provides additional experiment details for reproducibility.The appendices support the paper’s theoretical and empirical claims.Source: §6 · p. 15

6 Conclusion.

• On Living-17, LP-FT outperformed explicit regularization and a higher learning rate for the linear layer, although broader study remains open.This result is reported in Appendix B.4.Source: §6 · p. 15

A.1 Preliminaries on Important Notations and Principal Angles

• The preliminaries define singular values, operator norms, projectors, and principal angles, including their characterization through singular values of basis matrices.These definitions establish the notation used in the appendix’s theoretical arguments.Source: §A.1 · pp. 22–23
• A variational characterization of the maximum principal angle is used to lower-bound fine-tuning error outside the span of the training data.The associated lemma requires dim(R) ≤ dim(S), making the relevant basis-product matrix tall.Source: §A.1 · p. 23

A.2 Feature distortion theorem

• Fine-tuning distorts pretrained features in the overparameterized linear setting, with OOD error lower bounded whenever cos θmax(R0, S⊥) > 0.The theorem applies to all fine-tuning time steps and uses the pretrained feature rowspace R0 and the orthogonal complement of the training-data rowspace S⊥.Source: §A.2 · p. 23
• The proof shows that representations remain unchanged on directions perpendicular to the training examples, while changes in the head and feature extractor are coupled.This combines the invariance on S⊥ with the balancedness relation linking head and feature-extractor updates.Source: §A.2 · p. 24
• A positive principal-angle condition ensures that pretrained feature directions can be generated from bounded-norm OOD directions, enabling the distortion-based OOD lower bound.The relevant condition is cos θmax(R, S⊥) > 0, which supports constructing OOD directions whose projections cover the pretrained feature rowspace.Source: §A.2 · pp. 24, 27
• The theorem’s concluding bound is approximately c√(kϕ^2) − ϵ when ∥w∥2 = 1 and ϕ is bounded above by a constant.The appendix states this as an interpretable form of the bound after completing the contrapositive argument.Source: §A.2 · p. 31

A.3 LP vs. FT (OOD)

• Theorem 3.5 compares linear probing and fine-tuning in an overparameterized linear setting where ID data lies in a lower-dimensional subspace.The appendix fixes the problem parameters while considering a sequence of pretrained feature extractors approaching the optimal extractor.Source: §A.3 · p. 31
• As pretrained feature extractors converge to B⋆, the ratio of linear probing’s to fine-tuning’s OOD errors converges in probability to 0 under the ID subspace and non-degeneracy assumptions.The result fixes d, k, m, n, S, Pid, the head distribution, and v⋆, requiring positive principal-angle cosines between rowspace(B⋆) and both S and S⊥.Source: §A.3 · p. 31

A.3.1 Convergence of principal angle

• If pretrained feature extractors converge to B⋆, their rowspaces’ principal angles to any fixed subspace T converge to those of rowspace(B⋆).This follows from a perturbation bound for principal angles between feature-extractor rowspaces, applied to a convergent sequence of extractors.Source: §A.3.1 · p. 33

Zero head initialization

• When the head is initialized at v0 = 0, the head error equals ∥v⋆∥^2.This provides the deterministic initialization case underlying the broader head-error analysis.Source: §A.3.2 · p. 34

Gaussian head initialization

• For v0 ∼ N(0, σ^2I), the initial head error is typically at least Ω(∥v⋆∥^2) for every variance σ^2.The bound holds with high probability and reflects the random head’s generally incorrect direction relative to v⋆.Source: §A.3.2 · pp. 34–35

Anti-concentration argument

• The Gaussian head-error bound applies for all initialization variances, and analogous bounds can be obtained for non-Gaussian initializations.The argument relies on anti-concentration: a Gaussian is unlikely to lie too close to any particular target direction or value.Source: §A.3.2 · p. 34

Upper bound theorem

• Linear probing’s OOD error is upper bounded with probability at least 1 − δ under the stated ID subspace assumptions, including an analytic bound for isotropic Gaussian P_z when n ≥ 5m and n ≥ 10 log 1/δ.The theorem covers arbitrary P_z through a less quantitative bound and derives a more explicit form under Gaussian features.Source: §A.3.3 · pp. 36–38

Proof setup

• The proof represents the data as X = ZF^⊤, where Z contains iid latent samples and the orthonormal columns of F span the ID subspace.This factorization enables singular-value control of the feature-transformed data matrix.Source: §A.3.3 · p. 36

Error control

• The bound controls the OOD error through singular values of Z and the principal-angle factor cos θmax(R, S), while accounting for the feature-extractor distance between B_0 and B⋆.For Gaussian P_z, random-matrix bounds on σmin(Z) and σmax(Z) yield the explicit result; the rotation minimization is handled separately.Source: §A.3.3 · pp. 36–38

Optimization characterization

• Under the rank conditions, XB_0^⊤ has full rank almost surely, making the linear-probing loss strongly convex with a unique global minimizer reached by gradient flow.Invertibility of B_0X^⊤XB_0^⊤ supplies the required Hessian condition.Source: §A.3.3 · p. 36

A.4 LP vs. FT (OOD), non-asymptotic result for Gaussian covariates

• The non-asymptotic result quantitatively extends the earlier asymptotic conclusion that vanishing pretrained-representation error favors LP over FT OOD.The extension applies to Gaussian covariates and can also be generalized to iid mean-zero, constant-variance sub-Gaussian covariates using matrix concentration results.Source: §A.4 · p. 39
• The theorem requires non-degenerate alignment: both cos θmax(R∗, S) and cos θmax(R∗, S⊥) must be positive, where R∗ is the optimal representation’s row space.The proof relates conditions involving R∗ to the representation R0 used by LP and FT through Lemma A.9.Source: §A.4 · pp. 39–40
• Under Gaussian covariates on the ID subspace, LP has lower OOD error than FT at every fine-tuning time when representation error is below a dimension- and angle-dependent threshold.The guarantee assumes nonzero cosine principal angles between the optimal representation and both the ID subspace and its orthogonal complement, with sufficiently many samples and high probability.Source: §A.4 · p. 39

A.5 Principal angles are likely non-zero

• The angle assumptions in the main theorems exclude exact alignment of the optimal representation with the ID subspace or its complement, a degenerate coincidence under small randomness.In two dimensions, zero angle occurs only when the representation direction is exactly the direction of S or S⊥.Source: §A.5 · p. 41
• For a fixed k-dimensional representation and uniformly random m-dimensional subspace with m > k, Lemma A.16 gives quantitative lower bounds on the relevant principal-angle cosines.The result is obtained with probability at least 1 −δ.Source: §A.5 · pp. 41, 43
• For a uniformly random ID subspace, cos θmax(R, S) is positive almost surely.The proof uses Gaussian matrix representations and shows the relevant minimum singular value is positive almost surely while the maximum singular value is finite.Source: §A.5 · pp. 41, 43
• When k < m and k < d −m, randomizing the ID subspace makes both cos θmax(R∗, S) and cos θmax(R∗, S⊥) positive almost surely.Lemma A.16 also supplies quantitative lower bounds for both angles in this setting.Source: §A.5 · p. 43

• Fine-tuning does better ID than linear probing almost surely when w⋆ ∉ R0, cos θmax(S, Raug) ≠ 0, and fine-tuning reaches a local minimum.This is Proposition 3.6 in the linear overparameterized setting under the ID subspace assumption.Source: §A.6 · p. 43
• Fine-tuning achieves zero ID loss because every local minimum is global, zero-loss parameters exist, and training examples span the ID subspace almost surely.The spanning argument uses linearity, density of Pz, and n at least the ID subspace dimension m.Source: §A.6 · p. 43
• The positive linear-probing error follows because F⊤w⋆ ∉ Range(F⊤E0) prevents the pretrained feature rowspace from representing the optimal linear model on the ID subspace.Lemma A.18 formalizes this implication, while Lemma A.17 supplies the rank and projection condition.Source: §A.6 · p. 44
• Linear probing has positive ID error when w⋆ ∉ R0 and cos θmax(Raug, S) > 0, where Raug spans the feature rowspace and w⋆.The geometric condition implies F⊤w⋆ is outside the projected feature range, forcing non-zero linear-probing error.Source: §A.6 · p. 45

A.7 LP-FT

• When the initial feature extractor is perfect and cos θmax(S, R) > 0, linear probing converges to the optimal weights and achieves zero OOD loss.The proof uses strong convexity and uniqueness of the linear-probing minimizer.Source: §A.7 · pp. 45–46
• Linear probing is strongly convex in the head because the relevant Hessian has full rank when the pretrained feature row space has nonzero alignment with the task subspace.This yields a unique minimizer to which gradient flow converges.Source: §A.7 · pp. 45–46
• Under perfect pretrained features and non-degeneracy conditions, fine-tuning from a randomly initialized head has strictly positive OOD loss for every training-flow time.The result assumes B0 = UB⋆ and requires cos θmax(R0, S) ≠ 0 and cos θmax(R0, S⊥) ≠ 0.Source: §A.7 · p. 46
• On DomainNet, LP-FT achieves the best OOD accuracy, while linear probing outperforms fine-tuning across all three evaluated OOD domains.Table 3 reports OOD accuracies with 90% confidence intervals over 3 runs.Source: Table 3 · p. 47

B More information on experiments

• With OOD-based early stopping and learning-rate selection, linear probing is still typically better than fine-tuning OOD.The appendix also reports additional baselines, pretraining models, and a discussion of LP-FT’s effective robustness.Source: B More information on experiments · p. 47

B.1 Dataset and method details

• The evaluation spans CIFAR-10→STL, DomainNet, Living-17, Entity-30, FMoW Geo-shift, CIFAR-10→CIFAR-10.1, and ImageNet shifts using pretrained MoCo-v2, CLIP, or MoCo-TP models.Protocols vary by dataset, including 20-epoch or 50-epoch fine-tuning and ImageNet’s 10-epoch comparisons.Source: §B.1 · pp. 47–48
• The experiments use dataset-specific ID/OOD splits, including CIFAR-10 training with STL testing, DomainNet sketch versus other domains, and North America versus Africa and Europe for FMoW.DomainNet averages the three OOD domains in the main paper, while FMoW uses North America for adaptation and Africa and Europe for evaluation.Source: §B.1 · pp. 47–48

B.2 Target early stopping

• 67.1% average OOD accuracy for linear probing exceeds fine-tuning’s 61.3% even when fine-tuning selects its learning rate and target early stopping.Table 4 reports the full dataset-level results under this favorable tuning protocol.Source: §B.2 · p. 49, Table 4 · p. 49

B.3 Feature change

• The feature-change measurements were computed as average Euclidean distances between ResNet-50 features before and after fine-tuning on held-out validation examples.Results are separated into ID examples in Table 5 and OOD examples in Table 6.Source: §B.3 · p. 49
• ID features move more than OOD features in 9 of 10 cases, with the only exception being fine-tuning on FMoW.Average pre/post-fine-tuning Euclidean feature distances are reported in Tables 5 and 6.Source: §B.3 · p. 49, Table 5 · p. 50, Table 6 · p. 50
• LP-FT changes features substantially less than fine-tuning across all datasets for both ID and OOD examples.On CIFAR-10, LP-FT features change 30× less than fine-tuning features.Source: §B.3 · p. 50, Table 5 · p. 50, Table 6 · p. 50
• The observed feature changes support the theory that random-head fine-tuning couples large head updates to substantial feature-extractor updates, whereas LP initialization reduces this distortion.The paper connects these measurements to fine-tuning’s OOD underperformance and LP-FT’s strong ID and OOD performance.Source: §B.3 · p. 50

Architectures and pretraining source

• Across both CLIP architectures, fine-tuning beats linear probing ID but loses OOD, while LP-FT is best ID and closes 75%-90% of the OOD gap.On CLIP ResNet-50, LP-FT closes 86% of the OOD gap; on CLIP ViT-B/16, it closes 75%.Source: §B.4 · p. 51, Table 7 · p. 51, Table 8 · p. 51

Early stopping

• With OOD-based early stopping, LP-FT reaches 87.9±0.4% OOD accuracy versus 88.3±0.2% for linear probing, while fine-tuning reaches only 84.4 ± 0.5%.The LP-FT and linear-probing difference is not statistically significant under this selection rule, but fine-tuning remains lower.Source: §B.4 · p. 51

Additional fine-tuning heuristics

• Using a higher learning rate on the linear layer and regularizing feature-extractor weights toward pretrained weights each closes part of the OOD gap, but LP-FT performs better than both ID and OOD.The regularization uses Euclidean distance between current feature-extractor weights and pretrained weights, with λ ∈{1.0, 0.1, 0.01, 0.001, 0.0001}.Source: §B.4 · pp. 51–52

Side-tuning

• Side-tuning improves OOD accuracy to 81.0% from fine-tuning’s 77.7%, but performs worse ID than LP-FT.Side-tuning freezes pretrained features and trains a randomly initialized side network, which may limit ID refinement on the approximately 40K-example dataset.Source: §B.4 · p. 52

Training from scratch

• All fine-tuning methods and linear probing substantially outperform training from scratch both ID and OOD, despite scratch training using 450 epochs versus 20 for fine-tuning.The training-from-scratch results come from Santurkar et al. (2020).Source: §B.4 · p. 52

B.5 Discussion of effective robustness

• Linear probing has higher effective robustness than fine-tuning in most datasets despite lower ID accuracy, because it nevertheless performs better OOD.This reverses the usual effective-robustness expectation that lower ID accuracy predicts lower OOD accuracy.Source: §B.5 · p. 52
• Additional fine-tuning heuristics mitigate feature distortion and improve OOD accuracy over fine-tuning, while LP-FT performs best among them on both ID and OOD for Living-17.The compared heuristics use a 10× larger head learning rate, feature-extractor regularization, or a frozen pretrained model with a fine-tuned side network.Source: Table 9 · p. 53
• LP-FT achieves higher effective robustness than fine-tuning, with substantially better OOD accuracy at comparable ID accuracy on distribution-shift benchmarks.On CIFAR-10 →STL, LP-FT is 8% better OOD without a significant ID difference; at matched 97.2% ID accuracy, LP-FT reaches 90.2% OOD versus 81.8% for FT.Source: §B.5 · p. 53
• Higher effective robustness alone does not establish superiority, since a method can obtain it by performing worse ID while matching another method’s OOD accuracy.The authors therefore distinguish the robustness metric from overall method quality.Source: §B.5 · p. 53
• Under large distribution shifts, the consistent effective-robustness advantage of linear probing and LP-FT suggests good transfer methods can combine high ID accuracy with stronger OOD performance.The paper contrasts this finding with prior work reporting that higher effective robustness is uncommon.Source: §B.5 · pp. 52–53
• On Living-17, LP-FT and linear probing lie above fine-tuning in ID–OOD plots across CLIP and MoCo-V2 pretrained models, indicating higher effective robustness.Figure 3 averages each point over three random seeds and includes CLIP ResNet-50, CLIP ViT-B/16, and MoCo-V2 ResNet-50.Source: §B.5 · p. 53, Figure 3 · p. 54

C Additional related work

• Prior overparameterized-model analyses study benign overfitting, double descent, and minimum-norm interpolation, whereas this work analyzes gradient optimization from a particular pretrained initialization.The paper argues that this captures the exact implicit inductive bias of gradient-based fine-tuning.Source: C Additional related work · p. 54