Improved Regularization and Robustness for Fine-tuning in Neural Networks

1 Introduction

• Fine-tuning with limited labels is prone to overfitting or memorization, and fine-tuning under label noise is relatively underexplored in transfer learning.Prior regularization work focused on target datasets without label noise.Source: §1 · pp. 1–2
• Fine-tuned weights remain close to their pretrained initialization, top layers travel farther than bottom layers, and pretrained initialization yields better noise stability than random initialization.The analysis uses layer-wise distance and perturbed loss as measures of generalization performance.Source: §1 · p. 2
• The proposed method combines layer-wise distance constraints with self-label correction and reweighting, aiming to reduce overfitting and address noisy labels.It corrects noisy labels using model predictions and reweights examples according to confidence.Source: §1 · p. 2
• 1.76% average improvement on seven image-classification tasks, and 3.56% with noisy labels; gains over previous methods are also 0.75% for few-shot classification and 0.46% for sentence classification.The evaluations also include medical image classification and vision transformers, with qualitatively similar results.Source: §1 · p. 2
• The paper contributes a PAC-Bayesian analysis, a regularized self-labeling approach, and validation across classification tasks and pretrained architectures.The empirical scope includes image and text data.Source: §1 · p. 2

2 Related Work

• Li et al. (2018b) penalize weight distance, Li et al. (2018a) feature-map distance, Chen et al. (2019) suppress small-singular-value components, and Gouk et al. (2021) use projected gradient descent for distance constraints.Gouk et al. (2021) also derive generalization bounds using Rademacher complexity.Source: §2 · pp. 2–3
• Supervised methods for noisy labels use robust losses, noisy-example identification, and regularization; fine-tuning under label noise is relatively underexplored in transfer learning.The paper draws on semi-supervised pseudo-labeling and self-training, and notes recent evaluation of sharpness-aware fine-tuning from noisy labels.Source: §2 · p. 3
• Prior theory includes source-and-target empirical-risk weighting, data-dependent neural-network bounds, and margin-based bounds; PAC-Bayes measures are reported to correlate more closely with empirical generalization.The passage attributes these results to Ben-David et al. (2010), Arora et al. (2019), Nagarajan and Kolter (2018), and Jiang et al. (2020), respectively.Source: §2 · p. 3

3 Preliminaries

• Target-task examples are modeled as drawn from an unknown distribution P(t) over X × Y, with inputs in a d-dimensional space and labels in Y.The setup introduces a training set of size n(t).Source: §3 · p. 3
• The network composes layer operations ϕi(z) = ψi(Wi z), and its prediction error is measured with a convex, 1-Lipschitz loss.The final network output is the composition of its layer functions.Source: §3 · p. 3
• Fine-tuning starts from a pretrained source model and adjusts its layers to predict target labels.The seven image-classification tasks use ResNets pretrained on ImageNet as the initialization.Source: §3 · pp. 3–4
• Table 1 reports basic statistics for seven image-classification tasks.The listed datasets cover fine-grained object recognition, scene recognition, and general object recognition.Source: Table 1 · p. 4, §3 · p. 4
• The setup distinguishes independent label noise, which is independent of input features, from correlated noise, which depends on input features.The noisy dataset is introduced as a setting for target-task labels.Source: §3 · p. 4

4 Our Proposed Approaches

• Regularized self-labeling combines layer-wise regularization with model predictions to address fine-tuning generalization and robustness.The approach constrains distances in different layers and incorporates predictions during training, including label correction and reweighting.Source: §4 · p. 4

4.1 Fine-tuning and regularization

• Regularized fine-tuning constrains each layer’s distance from its pre-trained initialization, with the bound relating expected loss to empirical loss and an additional term.The analysis factors generalization error into a noise error and a KL-divergence term.Source: §4.1 · pp. 4–5
• The PAC-Bayes analysis identifies layer-wise fine-tuned distance and perturbed loss as measures for analyzing fine-tuning generalization.Perturbed loss is the loss after model weights are randomly perturbed; the bound is stated under assumptions on layer norms, bounded features, and a bounded Lipschitz loss.Source: §4.1 · p. 5

Fine-tuned distances {Di}L

• Fine-tuned distances are relatively small compared with pre-trained weights and vary by layer, with smaller distances in lower layers and larger ones in higher layers.The authors relate this pattern to lower layers extracting features and top layers learning task-specific features.Source: Fine-tuned distances {Di}L · pp. 5–6
• Layer-wise regularization uses smaller distance constraints for bottom layers and larger ones for top layers, implemented as an exponentially increasing schedule.The schedule uses a base distance D and scale factor γ.Source: Fine-tuned distances {Di}L · p. 6
• Layer-wise regularization yields a smaller sum of layer distances than constant regularization, indicating a tighter generalization bound.The paper attributes the tighter bound to the distance dependence of Theorem 4.1.Source: Fine-tuned distances {Di}L · p. 6
• Fine-tuned models from pre-trained initializations are more stable under weight perturbations than models from random initializations.The perturbed loss is measured after adding Gaussian noise to layer weights.Source: Fine-tuned distances {Di}L · p. 6
• Adversarially pre-trained models often have lower perturbed losses than random- or standard-pretrained models, and their smaller layer norms imply a tighter bound.A thorough theoretical analysis of adversarial training for fine-tuning is left for future work.Source: Fine-tuned distances {Di}L · p. 6

4.2 Fine-tuning and robustness

• Over 20%: fine-tuning’s training–test accuracy gap exceeds this under noisy labels, while stronger regularization reduces the gap and improves test accuracy.On Indoor, test accuracy initially rises and then falls as learning progresses; the figure reports gap reduction under stronger regularization at both noise rates.Source: Figure 4 · p. 6, §4.2 · p. 7
• Self-label correction precision starts around 60%, declines in later epochs, and increases with regularization.Precision is the fraction of relabeled points whose labels are corrected.Source: §4.2 · p. 7
• With reweighting, incorrectly labeled points receive significantly lower average weights than correctly labeled points.The authors report that this scheme expands the gradient of correctly labeled points, reducing the fraction of noisy data points in the training set.Source: §4.2 · p. 8

5.1 Experimental setup

• Experiments cover seven image datasets, ChestX-ray14, noisy-label settings, and miniImageNet, using ResNet-101, ResNet-18, ViT, and ResNet-12.The noise experiments use uniform label flips and auxiliary-network predictions to simulate independent and correlated noise, respectively.Source: §5.1 · pp. 8–9

5.2 Experimental results

• 1.76% average improvement over distance-based regularization on seven image-classification datasets, with larger gains over ℓ2-PGM on Stanford-Dogs and MIT-Indoor.The reported ℓ2-PGM gains are 2–3% on each of those two datasets.Source: Table 2 · p. 9, §5.2 · p. 9

5.1 Experimental setup

• Baselines include fine-tuning variants and regularizers, with additional supervised-learning methods used to assess robustness to noisy labels.The listed noise-robustness baselines include GCE, SCE, DAC, APL, ELR, and self-adaptive training.Source: §5.1 · p. 9

5.2 Experimental results

• Layer-wise regularization improves mean AUROC on ChestX-ray14 by 0.39% over baseline methods, and the text-classification results are reported as consistent with the image results.The ChestX-ray14 evaluation averages AUROC over predictions for all 14 labels.Source: §5.2 · p. 9
• 3.56% average improvement over previous supervised training methods under label noise; REGSL also improves by over 4% on average over distance-based regularization.The study considers independent and correlated noise on the Indoor dataset; the authors report consistent 1–3% gains across noisy-label settings.Source: §5.2 · p. 9, Table 3 · p. 10
• On noisy Indoor data with ViT, the approach beats the best regularization methods by 1.17% and self-labeling by 13.57%, averaged over two settings.The authors describe regularization as preventing overfitting to random labels and self-labeling as injecting the fine-tuned model’s belief into the noisy dataset.Source: §5.2 · pp. 9–10
• 0.75% average improvement over previous regularization methods on few-shot miniImageNet classification across 600 sampled meta-test tasks.Regularization methods generally improve fine-tuning performance by over 1%.Source: Table 4 · p. 10, §5.2 · p. 10
• Removing any of the three algorithm components degrades performance, and label correction is most critical when the noise rate exceeds 40%.The ablations also compare norm choices and layer-wise versus constant regularization.Source: §5.2 · p. 10

• The paper studies regularization and robustness in fine-tuning, using PAC-Bayesian analysis to investigate generalization error and derive two empirical measures.The analysis motivates layer-wise regularization, which encodes distance patterns observed during vanilla fine-tuning.Source: §6 · p. 10
• For noisy-label fine-tuning, the paper proposes self-label correction and label reweighting alongside layer-wise regularization.Source: §6 · p. 10
• The approach is evaluated across classification tasks and pre-trained architectures, including ResNets and ViTs.Source: §6 · p. 10
• A follow-up paper improves the theoretical result in Theorem 4.1 with a Hessian-based bound that yields non-vacuous bounds in practice.Source: §6 · p. 10

• The PAC-Bayesian setup uses a training-set-independent prior and a training-dependent posterior, with bounded loss and a high-probability generalization statement.The appendix invokes McAllester’s result, adapting it to losses bounded by a fixed constant C.Source: A Proof of Theorem 4.1 · p. 14
• The KL divergence between the Gaussian prior centered at pre-trained weights and posterior centered at fine-tuned weights equals their squared distance divided by 2σ².Source: A Proof of Theorem 4.1 · p. 14
• The proof bounds the loss increase caused by Gaussian perturbations to layer weights, using layer dimensions, operator norms, and Lipschitz properties.It applies concentration bounds to the random perturbation matrices and propagates bounds through the network layers.Source: A Proof of Theorem 4.1 · pp. 15–16
• Increasing a layer’s distance parameter D_i increases the network’s capacity and worsens the generalization bound.Source: A Proof of Theorem 4.1 · p. 17

• The image experiments use seven classification datasets, while additional evaluations cover ChestX-ray14, noisy-label settings, and miniImageNet few-shot recognition.Independent noise is simulated by uniform label flips; correlated noise uses predictions from an auxiliary network trained on held-out data.Source: §B.1 · pp. 17–18
• Image tasks use ImageNet-pre-trained ResNet-101 or ResNet-18; few-shot tasks use meta-training-pre-trained ResNet-12, and the approach is also extended to ViT.The label-noise experiments use ResNet-18, while the few-shot experiments follow the miniImageNet meta-training and meta-test setup.Source: §B.1 · p. 18
• Transfer-learning baselines include early-stopped fine-tuning, weight decay, label smoothing, ℓ2-SP, and ℓ2-PGM; noisy-label comparisons add supervised-learning methods such as GCE and SAT.Source: §B.1 · p. 18
• Regularization hyperparameters are optimized on validation data, with 20 tuning trials for the proposed method and baselines in the reported experiments.The layer-wise ResNet constraints use four block-level distance values, while label correction uses a threshold of 0.90.Source: §B.1 · pp. 18–19
• Figure 6 compares Frobenius norms across ResNet-18 layers for standard and adversarial pre-training at three noise levels; higher ϵ corresponds to lower norms.Source: Figure 6 · p. 20

Additional results

• The extended experiments also include another dataset with the same type of label noise as Table 3 and sentence classification tasks.The paper evaluates both transfer learning and few-shot classification in image and text settings.Source: §B.2 · p. 20

Additional results under label noise

• On CUB-200-2011 and Flowers with independent label noise, the approach outperforms previous methods by a significant margin.The authors report that this aligns with their MIT-Indoor results.Source: §B.2 · p. 20

Results on ChestX-ray14

• On ChestX-ray14, layer-wise regularization yields a 0.39% improvement over baseline methods in mean AUROC across 14 labels.The paper describes ChestX-ray14 as a more distant transfer task from ImageNet.Source: §B.2 · p. 20

Results on ViT

• 1.17% over the best regularization methods and 13.57% over self-labeling, averaged over two settings, on noisy-label ViT fine-tuning.The authors attribute the gains to both regularization and self-labeling.Source: §B.2 · p. 20

Extension to text classification

• The approach outperforms baseline methods on SST-based target-task transfer for six text classification datasets, with and without independent label noise.The targets are MR, CR, MPQA, SUBJ, and TREC; SST is the source task.Source: §B.2 · p. 20

Comparing layer-wise and constant regularization

• Layer-wise constraints yield a smaller value of the summed layer distances than constant regularization, and this correlates with better generalization performance.Source: §B.3 · p. 21

Comparing norms of adversarial robust pre-trained models

• Adversarially robust pre-trained models have lower Frobenius norms in each layer, which the authors say induces smaller generation error and explains their downstream fine-tuning improvement.Source: §B.3 · p. 21

Comparing label-correction precision with and without regularization

• With regularization, label correction relabels fewer points and produces fewer incorrectly labeled points than self-labeling without regularization.At epoch 21, the reported incorrect-label counts are 11 with regularization versus 42 without it.Source: §B.3 · p. 21

Comparing ℓ2 norm and the MARS norm

• Comparing constraint norms in the algorithm, the paper reports similar results for the Frobenius norm (ℓ2) and the MARS norm proposed in, and focuses discussion on Frobenius.Source: §B.3 · p. 21

Influence of different components in REGSL

• Removing any algorithm component worsens performance; self-labeling is more critical at 20% noise, while label correction is most important at 40% or higher.The self-labeling part at 20% includes label correction and removal.Source: §B.3 · p. 22, Table 11 · p. 22