PaperlayerView exact revision on arXiv

Source-linked AI summary

LQF: Linear Quadratic Fine-Tuning

Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, Stefano Soatto

arXiv:2012.11140v1cs.LGcs.CVstat.ML

TL;DR

DNN fine-tuning lacks the predictable behavior of convex linear models, while earlier linearization methods suffered a substantial performance gap. LQF combines a linearized pretrained model with Leaky-ReLU, mean-squared loss, and Kronecker-factorized preconditioning, achieving performance comparable to NLFT on most tested real-world image-classification tasks and better performance in low-data settings. Its scope is limited because first-order linearization is expected to work best when the target and pretraining tasks are sufficiently close.

  • Problem

    DNN fine-tuning is sensitive to training data, initialization, regularization, and hyperparameters, unlike linear models trained with convex optimization.

    Source: §1 · p. 1

  • Method

    LQF linearizes a pretrained network and combines Leaky-ReLU, mean-squared fine-tuning loss, and Kronecker-factorized preconditioning.

    Source: §3 · p. 3

  • Results

    LQF achieves performance comparable to NLFT on most tested real-world image-classification tasks and outperforms NLFT in few-shot and online learning settings.

    Source: §1 · p. 2, §4 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    LQF makes linearization practically viable by trading a small performance drop for improved training ease, interpretability, and cheaper hyperparameter search.

    Source: §5 · p. 8

    As a first-order approximation, LQF is expected to work well when the fine-tuning task is sufficiently close to the pretraining task, and it drops in performance on fine-grained tasks.

    Source: §5 · p. 8

Abstract

from arXiv · show

Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization. Such desirable properties are absent in deep neural networks (DNNs), typically trained by non-linear fine-tuning of a pre-trained model. Previous attempts to linearize DNNs have led to interesting theoretical insights, but have not impacted the practice due to the substantial performance gap compared to standard non-linear optimization. We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning on most of real-world image classification tasks tested, thus enjoying the interpretability of linear models without incurring punishing losses in performance. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification: Leaky-ReLU instead of ReLU, mean squared loss instead of cross-entropy, and pre-conditioning using Kronecker factorization. None of these changes in isolation is sufficient to approach the performance of non-linear fine-tuning. When used in combination, they allow us to reach comparable performance, and even superior in the low-data regime, while enjoying the simplicity, robustness and interpretability of linear-quadratic optimization.

1. Introduction

LQF seeks to retain the robustness and interpretability of linear-quadratic optimization while closing the performance gap with nonlinear fine-tuning. It combines three modifications whose joint use enables comparable performance, including advantages in low-data settings.

1. Introduction

  • DNN fine-tuning is sensitive to small changes in data, regularization, and hyperparameters, producing outcomes that are difficult to predict or interpret.This sensitivity increases hyperparameter-optimization costs and complicates reasoning about individual training samples and constraints.Source: §1 · p. 1
  • Earlier DNN linearization methods offered theoretical insight but remained far behind nonlinear fine-tuning on large-scale image classification.The introduction frames LQF as an attempt to reduce this performance-versus-robustness trade-off.Source: §1 · p. 1
  • LQF is the closest linear method to NLFT, with a 12% median test-error increase relative to NLFT across the evaluated datasets.Figure 1 also reports 47% for GaF, 71% for a fixed-embedding linear classifier, and 42% for LQF applied only to the final classifier.Source: §1 · p. 1, Figure 1 · p. 2
  • LQF achieves performance comparable to NLFT on real-world image classification and performs better in the low-data regime.The method is designed to preserve linear-quadratic optimization benefits without a punishing performance loss.Source: §1 · pp. 1–2
  • LQF combines Leaky-ReLU, mean-squared loss, and Kronecker-factorized preconditioning because their joint use improves linearized-model performance more than any change alone.The individual modifications provide limited improvements to standard nonlinear training, whereas their combination has a larger impact on linearized models.Source: §1 · p. 2

2. Related Work

Prior linearization work replaces network training with a linear model over network gradients, but its performance remains weaker than nonlinear fine-tuning. LQF addresses optimization and interpretability through quadratic loss, curvature preconditioning, and exact sample-influence analysis.

2. Related Work

  • First-order linearization trains a linear model using pretrained-network gradients as features, but prior methods have not matched nonlinear fine-tuning performance.Higher-order approximations improve fidelity but lose model linearity.Source: §2 · p. 2
  • LQF uses mean squared error for fine-tuning while retaining cross-entropy for pretraining, reflecting different roles for transfer and classifier optimization.The paper notes evidence that cross-entropy learns transferable representations, whereas MSE can be competitive for classifier training.Source: §2 · p. 2
  • K-FAC preconditioning approximates loss curvature efficiently, and curvature is constant in LQF so the preconditioning matrix is computed only once.This is intended to improve convergence on badly conditioned convex problems.Source: §2 · p. 2
  • Unlike NLFT, strongly convex LQF has a unique global minimum and does not require extensive hyperparameter optimization.The paper contrasts this with NLFT's sensitivity to learning rate, momentum, and batch size.Source: §2 · p. 2
  • Quadratic loss enables exact analysis of how individual training samples affect the learned model and predictions, extending influence-style interpretability.The paper also describes efficient computation of activation changes on a validation set.Source: §2 · p. 2
  • LQF's linear-model properties support applications including backward-compatible training, leave-one-out error estimation, sample-influence analysis, active learning, privacy, and continual learning.These applications are presented as consequences of making linearized fine-tuning perform on par with NLFT.Source: §2 · pp. 2–3

3. Linear-Quadratic Fine-tuning

LQF linearizes a pre-trained network and combines MSE, K-FAC pre-conditioning, and Leaky-ReLU to make linear-quadratic fine-tuning practical. The method retains predictable optimization and sample-level interpretability while approaching non-linear fine-tuning accuracy, especially with limited data.

3. Linear-Quadratic Fine-tuning

  • LQF replaces the network with a first-order linear approximation whose weights are linear, while its input dependence remains nonlinear through the activations.Gradients of the original network serve as features, and a modified forward pass computes the required Jacobian-vector products at a cost comparable to running the original network.Source: §3 · p. 3

3.1. Loss function

  • MSE replaces cross-entropy because linearized outputs can saturate softmax, whereas MSE keeps outputs bounded and yields a unique closed-form optimum.The quadratic loss also enables exact influence calculations for individual training samples through the Jacobian matrix and target matrix.Source: §3.1 · p. 3

3.2. Pre-conditioning with K-FAC

  • K-FAC pre-conditioning is needed because LQF's Hessian is fixed and can be badly conditioned, unlike non-linear training where changing curvature can improve conditioning automatically.K-FAC approximates the Fisher with tractable off-diagonal structure; constant curvature and a unique minimizer make the pre-conditioner especially efficient.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Leaky-ReLU activations

  • Leaky-ReLU prevents zero initialization gradients from stunting filters in the linearized model and provides a more accurate first-order approximation near zero.In the ablation, replacing Leaky-ReLU with ReLU increases error on most datasets, although less than removing MSE or K-FAC.Source: Figure 2 · p. 3, §3.3 · pp. 4–5
  • LQF outperforms other linearization methods on all datasets, matches non-linear fine-tuning on most, and performs better in the low-data regime.The authors attribute the low-data advantage to the linear model's strong inductive bias acting as a regularizer.Source: §3 · p. 3, §3.3 · p. 5

3.4. Interpretability

  • LQF computes how omitting a training sample changes final weights, test predictions, and validation-set outputs, supporting influence analysis and dataset summarization.K-FAC's curvature approximation makes the relevant inverse efficiently computable.Source: §3.4 · p. 5

4. Results

Across standard vision tasks, LQF generally matches non-linear fine-tuning and outperforms other linearization methods, with advantages in low-data and online settings. Its performance is qualified by weaker results on fine-grained tasks distant from ImageNet, while bilinear pooling, component choices, and stable optimization improve its scope or robustness.

4. Results

  • The reported experiments cover standard vision benchmarks and cross-domain transfer tasks, including comparisons against NLFT, Standard FC, GaF, and LQF FC.The evaluation is designed to compare full LQF with both fixed-feature and alternative linearized baselines.Source: §4 · p. 5, §4.1 · p. 5

4.1. Comparison

  • LQF is comparable to NLFT on coarse-grained datasets and outperforms other linearization methods, while remaining within 0.5% absolute error of NLFT on most coarse tasks.On fine-grained tasks with a large ImageNet domain gap, linear methods perform worse than NLFT, but LQF reduces error by up to 20% versus competing methods.Source: §4.1 · p. 5, Table 1 · p. 6

4.2. Fine-grained classification and Bilinear Pooling

  • B-LQF significantly improves fine-grained accuracy over average-pooling LQF, whereas bilinear pooling does not significantly improve ordinary NLFT.The modification replaces global average pooling with a representation based on feature covariance.Source: §4.2 · p. 6, Table 2 · p. 6

4.3. Ablation study

  • MSE provides the largest ablation improvement, K-FAC is also important because the linearized problem is badly conditioned, and Leaky-ReLU yields a smaller average gain.The combined changes are evaluated as components of LQF rather than as isolated improvements to ordinary non-linear training.Source: §4.3 · p. 6

4.4. Low-shot data regime

  • In k-shot training, LQF systematically outperforms NLFT, suggesting that its inductive bias can be advantageous for few-shot learning.The paper attributes this setting to instability and memorization risks in non-linear networks and notes that LQF supports convex regularization.Source: §4.4 · p. 6

4.5. On-line learning

  • During incremental training on MIT-67, LQF stays close to retraining-from-scratch NLFT while avoiding the cost of retraining from scratch.Incremental NLFT performs substantially worse than the retraining paragon because it becomes stuck in suboptimal local minima.Source: §4.5 · pp. 6–7

4.6. Informativeness of samples

  • LQF identifies easy or duplicated samples as less informative and confusing-class samples as more informative, with removal of informative samples causing greater test-error degradation.The method uses its exact quadratic analysis to measure how individual training samples affect final predictions.Source: §4.6 · p. 7

4.7. Robustness to optimization hyper-parameters

  • LQF has a narrow range of nearly optimal effective learning rates across tasks, unlike NLFT, whose optimal values vary by orders of magnitude and require fine-grained search.This robustness comes with a slight performance decrease relative to non-linear fine-tuning.Source: §4.7 · p. 7

5. Discussion

LQF targets the robustness and interpretability gap between linear models and nonlinear fine-tuning, while extending linearized training to practical data-analysis and learning settings. Its scope remains bounded by task similarity and the need for further applications.

Discussion

  • LQF supports online learning because its optimization has a unique minimum, unlike incremental nonlinear training that can become trapped in suboptimal local minima.Figure 5 compares incremental training with retraining from scratch as more data arrive.Source: Figure 5 · p. 7
  • Its influence estimates identify hard-to-distinguish samples as informative, whereas easy or near-duplicate examples contribute little to the decision boundary.Removing informative samples reduces test performance faster than removing less informative samples.Source: Figure 6 · p. 8
  • LQF makes linearization practically viable by approaching nonlinear fine-tuning performance while improving training ease, interpretability, and hyperparameter-search cost.The paper frames the small performance drop as offset by these practical benefits.Source: §5 · p. 8
  • LQF is expected to work best when the fine-tuning task is sufficiently close to the pretraining task, although model selection or architecture changes may extend its range.Fine-grained tasks are a documented weak setting, where bilinear pooling improves performance.Source: §5 · p. 8
  • The paper leaves applications including uncertainty quantification, active learning, continual learning, unlearning, and privacy or fairness constraints for future work.These are proposed uses rather than evaluated results in the discussion.Source: §5 · p. 8

A. Supplementary Material for LQF

The supplementary material adds empirical results, implementation details, and a derivation related to removing training samples.

A. Supplementary Material for LQF

A.1. Additional results

Additional experiments examine preconditioning, loss choice, weight decay, and validation-based hyperparameter optimization. They generally support LQF’s efficient optimization, with specific advantages for low-data training and poorly conditioned last-layer problems.

A.1. Additional results

  • LQF achieves similar final accuracies with Adam and K-FAC after hyperparameter optimization, while K-FAC performs better for LQF FC.The latter optimizes only the last layer, producing a lower-dimensional, badly conditioned problem.Source: §A.1 · p. 11, Table 3 · p. 11
  • MSE loss tends to outperform cross-entropy in the low-data regime even for standard nonlinear fine-tuning.The result suggests that MSE’s benefits are not exclusive to linearized models.Source: §A.1 · p. 11, Figure 7 · p. 12
  • LQF shows no major accuracy differences across tested weight-decay values, suggesting that linearization itself supplies a useful inductive bias for generalization.Source: §A.1 · p. 11
  • Fine-tuning an LQF solution after changing weight decay obtains similar errors to retraining from scratch across tested values.This supports a cheaper weight-decay search because strong convexity yields a unique global minimum.Source: §A.1 · p. 11
  • The converged weights are differentiable with respect to weight decay, enabling gradient-based optimization of the validation loss.The derivative uses a preconditioned validation gradient that can be approximated with K-FAC.Source: §A.1 · pp. 11–12
  • Further exploration of validation-loss derivatives using K-FAC is left to future work.Source: §A.1 · p. 12

A.2. Experimental details

The experiments use standardized pretrained backbones, optimization searches, augmentation schemes, and linearization procedures across diverse image datasets. The implementation also includes bilinear pooling and examines robustness across training configurations.

A.2. Experimental details

  • The evaluation spans standard fine-tuning and fine-grained datasets plus medical-image datasets, using specified train/test splits where standard splits are unavailable.Examples include Oxford Flowers, Caltech-256, and Malaria Cell.Source: §A.2 · p. 12
  • Experiments use an ImageNet-pretrained ResNet-50, with Leaky-ReLU for LQF and standard ReLU for comparison methods.The Leaky-ReLU backbone is initialized from the pretrained ReLU network and fine-tuned briefly on ImageNet.Source: §A.2 · p. 12
  • The training protocol uses momentum SGD, fixed batch size, searches over learning rate and weight decay, and reports the best result.The ℓ2 penalty is placed directly in the loss, as required for the K-FAC update.Source: §A.2 · p. 12
  • The study tests center crop, random crop, and resized random-crop augmentation, finding the strongest augmentation especially useful for Chest X-Rays.Random-crop results use one specified hyperparameter combination to reduce computation.Source: §A.2 · pp. 12–13
  • The linearization procedure also linearizes batch normalization and freezes its running statistics during training to preserve linearity.Robustness experiments search learning rate, batch size, and weight decay on a representative dataset subset.Source: §A.2 · p. 13
  • B-LQF replaces global average pooling with a linearized square-root-normalized bilinear pooling layer based on last-layer feature covariances.The implementation computes the covariance from the base model’s forward pass without much added complexity.Source: §A.2 · p. 13

A.3. Derivation of interpretability (eq. 8)

The derivation characterizes how removing one training sample changes the optimum and a test activation in LQF. Using the quadratic structure, rank-1 inverse updates, and K-FAC-estimated curvature, the resulting expression connects this change to Jacobian similarity under a Fisher-informed metric.

Setup and extension

  • The derivation assumes binary classification initially, where the label is scalar and each sample Jacobian is a row vector, with a similar multiclass derivation.For multiclass problems, the Jacobian becomes a C × D matrix.Source: §A.3 · pp. 13–14

Optimization derivation

  • Removing one sample yields a new optimum that can be obtained by a single Newton update because the objective is quadratic.The derivation expands the loss and Hessian after excluding the sample.Source: §A.3 · p. 13

Curvature update

  • The leave-one-out Hessian is expressed as a rank-1 update of A, enabling its inverse to be computed with the Sherman–Morrison formula.The Woodbury identity provides the corresponding multiclass generalization.Source: §A.3 · pp. 13–14

Large-dataset interpretation

  • For large datasets (N ≫1), removing a training sample changes the test activation according to eigtestA−1gT i.This is the simplified form of the derived expression for activation change.Source: §A.3 · p. 14

Interpretability

  • The activation-change term measures similarity between the test-sample Jacobian and the removed sample's Jacobian under the metric induced by A−1.A−1 is the inverse of the Fisher Information Matrix plus a multiple of the identity.Source: §A.3 · p. 14

Practical computation

  • K-FAC is convenient because the same approximation estimates the Fisher-informed inverse used in the interpretability expression and preconditions the gradients.This links the analysis directly to an already-computed optimization quantity.Source: §A.3 · p. 14
Loading 2012.11140v1…