PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees

Haotian Ju, Dongyue Li, Hongyang R. Zhang

arXiv:2206.02659v6cs.LGcs.CVmath.STstat.MLstat.TH

TL;DR

Fine-tuning pretrained networks can overfit, and measuring their generalization remains challenging. The paper uses PAC-Bayesian analysis to develop Hessian-based measures and bounds, and an algorithm for class-conditional label noise. It reports correlation with generalization error and improved accuracy on noisy-label tasks.

  • Problem

    Fine-tuned deep networks can overfit, while precise measurement of their generalization error is challenging.

    Source: §1 · p. 1

  • Method

    The paper uses PAC-Bayesian analysis to develop Hessian-based generalization bounds and a noisy-label fine-tuning algorithm combining statistically consistent losses with distance-based regularization.

    Source: §1 · p. 2

  • Results

    On six image-classification tasks with programmatically labeled data, accuracy improved by 3.26% on average compared with existing fine-tuning approaches.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The Hessian-based distance measure correlates with fine-tuning generalization error, and the algorithm performs robustly under class-conditional noise up to 60%.

    Source: §1 · p. 2

    The theoretical results require twice-differentiable activation mappings; extending them to nonsmooth activations such as ReLU is left for future work.

    Source: §4 · p. 13

Abstract

from arXiv · show

We consider fine-tuning a pretrained deep neural network on a target task. We study the generalization properties of fine-tuning to understand the problem of overfitting, which has often been observed (e.g., when the target dataset is small or when the training labels are noisy). Existing generalization measures for deep networks depend on notions such as distance from the initialization (i.e., the pretrained network) of the fine-tuned model and noise stability properties of deep networks. This paper identifies a Hessian-based distance measure through PAC-Bayesian analysis, which is shown to correlate well with observed generalization gaps of fine-tuned models. Theoretically, we prove Hessian distance-based generalization bounds for fine-tuned models. We also describe an extended study of fine-tuning against label noise, where overfitting remains a critical problem. We present an algorithm and a generalization error guarantee for this algorithm under a class conditional independent noise model. Empirically, we observe that the Hessian-based distance measure can match the scale of the observed generalization gap of fine-tuned models in practice. We also test our algorithm on several image classification tasks with noisy training labels, showing gains over prior methods and decreases in the Hessian distance measure of the fine-tuned model.

1 Introduction

Fine-tuning can overfit, while measuring deep-network generalization remains challenging; this work uses Hessian-based PAC-Bayesian analysis to study generalization and robustness to label noise. It develops Hessian distance-based bounds and reports empirical gains for fine-tuning with noisy labels.

1.1 Related work

  • Prior fine-tuning bounds emphasize distance from initialization, while this work adds Hessian information; its measure differs from sharpness by measuring average rather than worst-case perturbation.The paper situates its analysis within PAC-Bayesian approaches to deep-network generalization.Source: §1 · p. 1, §1.1 · p. 3

1 Introduction

  • A Hessian-based distance measure adds curvature to distance from initialization and is reported to correlate with fine-tuning generalization error.The paper motivates this measure through PAC-Bayesian analysis and Hessian-based noise-stability measurement.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The proposed bounds cover fine-tuned multilayer feed-forward networks, including vanilla fine-tuning, early stopping, distance-based regularization, and weighted losses.The paper also derives a distance-regularization bound assuming constraints on each layer's distance-vector norm.Source: §1 · p. 2
  • The label-noise algorithm combines statistically consistent losses with distance-based regularization, and its proof extends PAC-Bayesian bounds to class-conditional independent noise.The noise model assumes labels are independently flipped according to a confusion matrix.Source: §1 · p. 2
  • On six image-classification tasks with programmatically labeled data, accuracy improved by 3.26% on average over existing fine-tuning approaches; the algorithm also performed robustly under class-conditional noise up to 60%.The paper also reports similar benefits for vision transformers and language models.Source: §1 · pp. 2–3
  • Ablations report that strong regularization reduces the Hessian distance measure more than six times compared with several fine-tuning methods.The paper reports this across various settings.Source: §1 · p. 3

1.1 Related work

  • For noisy labels, the work extends Natarajan et al. with PAC-Bayesian Hessian-dependent bounds and deep neural networks as the hypothesis class.The related-work discussion says experiments highlight explicit regularization for improving generalization with limited, noisy labels.Source: §1.1 · p. 3

2 Hessian-based Generalization Bounds for Fine-tuned Models

The section develops PAC-Bayesian Hessian-based generalization bounds for fine-tuning and examines their relation to empirical generalization errors. It extends the analysis to class-conditional label noise using consistent losses and distance-based regularization, and outlines the proof strategy.

2.1 Problem setup

  • The setup considers target-task prediction from independently sampled training examples and defines generalization error as expected loss minus empirical loss.It frames fine-tuning as updating a pretrained network's weights for the target task.Source: §2.1 · p. 4

2.2 Generalization bounds using PAC-Bayesian analysis

  • PAC-Bayesian analysis yields Hessian-based generalization bounds that apply to vanilla fine-tuning, with or without early stopping, and to distance-based regularization.The distance-based corollary assumes constraints on the layer-wise update norms.Source: §2.2 · p. 5
  • The Hessian measure correlates with empirical generalization errors across seven methods, while D_i is smaller with explicit distance-based than implicit early-stopped regularization.Each Figure 2 dot represents one fine-tuned model under a particular regularization method.Source: §2.2 · p. 6, Figure 2 · p. 6

2.3 Incorporating consistent losses with distance-based regularization

  • For class-conditional label noise, the method combines statistically consistent weighted losses with distance-based regularization; Theorem 2.2 establishes robustness and the algorithm uses Λ = F^-1.The theorem uses the trace of the loss Hessian rather than its truncated positive-eigenvalue version.Source: §2.3 · pp. 6–7

2.4 Proof overview

  • The proof uses layer-wise Hessian approximation and uniform convergence, which requires Lipschitz-continuous second derivatives of the activations and loss.For consistent losses, noisy-label Hessian stability equals class-label stability in expectation; the CIFAR-100 ResNet Hessian estimate has relative error within 3%.Source: §2.4 · pp. 7–9, Table 2 · p. 8, Figure 3 · p. 9

3 Experiments

The experiments evaluate the noisy-label fine-tuning algorithm across image and text tasks, label-noise settings, and model architectures. They report accuracy gains over prior methods, reduced Hessian distance, and practical Hessian-based generalization bounds.

3.1 Experimental setup

  • Experiments assess robustness under synthetic random label flips and weakly supervised labels, using image and text tasks and pretrained ResNet, ViT, and RoBERTa models.Baselines include regularization, robust-loss, and self-training methods.Source: §3 · p. 9, §3.1 · p. 10

3.2 Experimental results

  • The algorithm improves over previous methods by 6.43% on average with ResNet-18, 6.03% with ResNet-101, and 2.91% with RoBERTa-Base on MRPC.The image experiments use synthetic noise on DomainNet's Sketch domain; the text experiment uses 20% and 40% synthetic noise.Source: §3.2 · p. 10
  • Algorithm 1 outperforms prior methods by 3.26% on average across six weakly supervised image-classification tasks.Table 3 covers independently flipped and programmatically generated labels; results are averaged over ten random seeds.Source: §3.2 · p. 10, Table 3 · p. 11
  • With ViT on Clipart and Sketch, the approach outperforms baselines by 2.42% on average under correlated label noise.The passage reports that ViT boosts performance across all approaches, including by over 4% for early stopping.Source: §3.2 · p. 11, Table 4 · p. 11
  • Removing either the weighted-loss scheme or regularization constraints reduces test accuracy, supporting the contribution of both components.The ablation results are averaged over ten random seeds.Source: §3.2 · pp. 11–12, Table 5 · p. 12

3.3 Further numerical results

  • For ResNet-18 on weakly supervised Clipart, the algorithm reduces the Hessian-distance quantity six times more than previous fine-tuning algorithms.Figure 4 compares the Hessian distance measure for the proposed and existing approaches.Source: §3.3 · p. 12, Figure 4 · p. 12
  • The reported Hessian-based bound is orders of magnitude smaller than previous bounds on CIFAR-10, CIFAR-100, MRPC, and SST-2.The comparisons use ResNet-50 for image tasks and BERT-Base-Uncased for text tasks.Source: §3.3 · p. 12, Table 6 · p. 12

4 Conclusion

The conclusion presents Hessian-based PAC-Bayes analysis as a way to bound and assess fine-tuning generalization, with experiments on robustness to noisy labels. It identifies smoothness assumptions and broader model coverage as open directions.

4 Conclusion

  • The PAC-Bayes analysis yields Hessian-distance generalization bounds for fine-tuned deep networks and reports that the measure correlates with empirical generalization error.The conclusion states that the result also applies to supervised learning, with the distance vector measured from random initialization to final weights.Source: §4 · p. 13
  • Experiments demonstrate robustness of the proposed algorithm under synthetic and real-world label noise.Source: §4 · p. 13
  • The stated result requires twice-differentiable activations; extending it to nonsmooth activations such as ReLU is left for future work.The conclusion also proposes studying Hessian-based generalization in other black-box models.Source: §4 · p. 13

A Proofs

The proofs build a PAC-Bayes argument around perturbed losses and Gaussian weight noise. They rely on smoothness assumptions for activations and the loss.

A Proofs

  • The proof setup defines perturbed empirical and expected losses using posterior-distributed weight perturbations, then invokes a PAC-Bayes bound for bounded loss.The bound is stated for loss values in [0, C].Source: A Proofs · p. 16
  • For Gaussian layerwise perturbations, the proof gives the KL divergence between prior and posterior, including the isotropic-covariance case.Source: A Proofs · p. 17
  • The stated smoothness conditions require activations and loss to be twice differentiable, with Lipschitz bounds on the functions and their first- and second-order derivatives.The appendix provides examples of constants for sigmoid, tanh, and GELU.Source: A Proofs · pp. 17–18

A.1 Proof of Theorem 2.1

The proof bounds generalization by combining PAC-Bayes control with Hessian-based estimates of loss changes under weight perturbations. It chooses perturbation covariances to control the resulting bound.

A.1 Proof of Theorem 2.1

  • A Taylor-expansion argument bounds noise stability for both empirical and expected loss, which are then combined with a uniform-convergence bound.Source: §A.1 · pp. 18–19
  • The proof uses the PAC-Bayes inequality and the prior-posterior KL divergence, with the distance vectors defined from fine-tuned and pretrained weights.Source: §A.1 · p. 19
  • Layerwise perturbation covariances are selected using the truncated Hessian, and the proof then balances the perturbation error against the KL term.The theorem's analysis relies on Lipschitz continuity of activations and their second derivatives.Source: §A.1 · pp. 19–20

A.2 Proof of Theorem 2.2

The proof bounds the fine-tuned model’s generalization using posterior-averaged loss and noise stability. It combines Taylor-based stability bounds with PAC-Bayes and uniform-convergence arguments.

A.2 Proof of Theorem 2.2

  • The proof defines posterior-averaged loss and relates noise stability to the empirical loss of the fine-tuned model.It then uses Taylor expansion to bound noise stability for both empirical and expected loss.Source: §A.2 · p. 21
  • PAC-Bayes and uniform-convergence bounds are combined to obtain a high-probability bound on the loss gap.The proof selects perturbation variances and a parameter to minimize the resulting upper bound.Source: §A.2 · pp. 22–23
  • The proof controls the resulting bound using layerwise perturbation norms and the specified stability parameters.It fixes β close to 1 independently of sample size and confidence level.Source: §A.2 · pp. 21–23

A.3 Proof of Lemma 2.4

The proof establishes uniform convergence of loss Hessians by combining Hessian Lipschitz continuity with a covering argument and concentration. Its bound uses a constant G that does not grow with sample size or inverse confidence.

A.3 Proof of Lemma 2.4

  • The Hessian of the loss is Lipschitz under parameter perturbations, with a fixed G that does not grow with n or δ^-1.The stated regularity condition bounds activation and loss derivatives and their Lipschitz constants.Source: §A.3 · pp. 23–24
  • An ε-cover of the parameter set and matrix Bernstein concentration are used to control Hessian deviations uniformly.The proof chooses ε based on δ and G and then extends the estimate from the cover to the full set.Source: §A.3 · p. 24
  • The passage gives a final result with probability greater than 1 − δ′, where δ′ = 4d_i d_{i−1}/(eτ), and defines C2 = 4H_i√p and C3 = 12d_i d_{i−1}BG/(eH_i).Source: §A.3 · p. 25

A.4 Proof of Proposition A.5

The proof derives perturbation bounds for network outputs and their derivatives, then uses these bounds to establish a Lipschitz constant for the loss Hessian. The argument relies on smoothness assumptions on the activation and loss functions.

A.4 Proof of Proposition A.5

  • Under Assumption A.3, Claim A.6 bounds how perturbations change each layer’s network output.The proof builds the bound inductively across layers, using activation-function Lipschitz conditions.Source: §A.4 · pp. 25, 27
  • Claim A.7 bounds the change in the output Jacobian under weight perturbations, using Lipschitz conditions on derivatives.The derivation uses bounds on first-order derivative Lipschitz constants.Source: §A.4 · p. 26

A.4.1 Lipschitz constant of the Hessian

  • The proof combines output and Jacobian perturbation bounds to obtain a bound on the Hessian’s change under perturbation.It applies induction and derivative smoothness conditions to bound the Hessian for each layer.Source: §A.4.1 · pp. 29–31
  • The resulting Hessian Lipschitz bound has an explicit constant G; the proof notes that deriving its exact value is the main calculation.For an order-only bound, the authors say the calculation can be simplified.Source: §A.4.1 · p. 33

A.5 Proof of Lemma 2.3

The proof uses Taylor expansion to characterize loss changes under parameter perturbations. Mean-zero independent perturbations remove the first-order contribution in expectation, leaving a second-order error term.

A.5 Proof of Lemma 2.3

  • Claim A.8 expresses the loss under a parameter perturbation using a Taylor expansion with a second-order remainder.The proof invokes twice-differentiability and the mean value theorem.Source: §A.5 · p. 33
  • Taking expectation over the perturbation distribution yields the loss expectation under the posterior distribution Q.The proof applies the expansion to noise stability.Source: §A.5 · p. 33
  • Because each perturbation has zero mean and perturbations across layers are independent, the first-order term vanishes in expectation.The remaining contribution is bounded through the second-order error term.Source: §A.5 · pp. 33–34

B Experiment Details

The appendix provides a code repository for reproducing experiments and outlines supplementary experimental details.

B Experiment Details

  • The code repository is available for reproducing the experiments.The appendix also introduces supplementary material for the experiment section.Source: B Experiment Details · p. 34

B.1 Experimental setup for measuring generalization

The experiments compare Hessian-based measures with empirical generalization errors and prior bounds across image and text fine-tuning settings. They also examine how margin choice and noisy-label conditions are represented in the measurements.

B.1 Experimental setup for measuring generalization

  • Figure 1 compares Hessian distance with prior distance measures and generalization errors while sweeping ViT-Base’s initialization-distance constraint on Clipart with weakly supervised label noise.The tested constraints range from 0.1 to 1.0.Source: §B.1 · p. 34
  • Figure 2 compares the generalization measure with empirical errors for ResNet-50 on Indoor and CalTech-256 and BERT-Base on MRPC across specified regularization methods.Figure 2d also evaluates Hessian distances for early stopping and ℓ2 projected-gradient methods, with hyperparameters selected by validation accuracy.Source: §B.1 · p. 35
  • Figure 3 represents noisy-label Hessian traces as class-by-class matrices for ResNet-18 and ResNet-50 on CIFAR-10 and CIFAR-100.The traces average 200 selected samples per class; selection requires true-label loss below 1e-4.Source: §B.1 · p. 35
  • Figure 4 evaluates Hessian distances for early stopping, label smoothing, Mixup, SupCon, SAM, and the proposed algorithm on noisy-label Clipart with ResNet-18.The measure is computed as PL √Hi from Equation 3.Source: §B.1 · p. 35
  • Table 6 compares the proposed generalization measure with prior bounds for ResNet-50 on CIFAR-10 and CIFAR-100 and BERT-Base on MRPC and SST-2.For Figure 1, the Pitas et al. distance-bound values are divided by 1074 because their scale is orders of magnitude larger than the generalization error.Source: §B.1 · p. 35
  • Figure 5 reports that classification error depends heavily on the chosen margin, with comparison margins set to 0.4 for CIFAR-10 and 0.01 for CIFAR-100.The selected margins keep the difference between margin loss and zero-one loss below 1%.Source: §B.1 · p. 36, Figure 5 · p. 36

B.2 Experimental setup for fine-tuning with noisy labels

The appendix describes noisy-label experiments on DomainNet-derived image tasks and MRPC, then compares a full-model Hessian measure with a layer-wise measure. The reported MRPC and SST-2 comparison leaves rigorous formulation as future work.

B.2 Experimental setup for fine-tuning with noisy labels

  • The noisy-label experiments cover six DomainNet-derived image tasks and MRPC, using independent uniform flips or correlated labels generated from majority votes by weak-supervision sources.For the DomainNet protocol, pretrained ResNet-18 models are trained using 60% of each domain’s labeled data, while classifiers from other domains serve as weak-supervision sources.Source: §B.2 · p. 36
  • The experiments fine-tune ResNet-18 and ResNet-101 for DomainNet tasks and RoBERTa-Base for MRPC, averaging test Top-1 accuracy over ten random seeds.The reported Table 3 results use 30 cross-validation searches over distance and scaling parameters.Source: §B.2 · pp. 36–37

B.3 Comparison with related Hessian-based generalization measures

  • The comparison includes a concurrent PAC-Bayesian measure whose posterior covariance is constructed from the Hessian spectrum; its full-model quadratic form uses concatenated layer vectors and the truncated Hessian.For this measure, the data set is treated as the union of training and testing data.Source: §B.3 · p. 37
  • Table 8 compares the layer-wise Hessian measure with the full-model Hessian trace; on MRPC and SST-2, H/n is slightly smaller, while rigorous formulation is left for future work.The authors note that this may require analyzing Lipschitz continuity of the full Hessian.Source: §B.3 · p. 38, Table 8 · p. 38
Loading 2206.02659v6…