PaperlayerView exact revision on arXiv

Source-linked AI summary

Explicit Inductive Bias for Transfer Learning with Convolutional Networks

Xuhong Li, Yves Grandvalet, Franck Davoine

arXiv:1802.01483v2cs.LG

TL;DR

Fine-tuning can move parameters far from a pre-trained model, potentially losing source-task knowledge. This paper evaluates regularizers that explicitly preserve similarity to the starting model and finds L2-SP the simplest effective option.

  • Problem

    Fine-tuning may drive parameters far from their pre-trained values, causing losses of source-task knowledge assumed relevant to the target task.

  • Method

    The paper evaluates L2, Lasso, Group-Lasso, and Fisher-based penalties that explicitly bias fine-tuning toward the pre-trained parameters.

  • Results

    L2-SP consistently outperforms standard approaches, while Fisher information provides no significant improvement over L2-SP and L1 or Group-L1 options are not valuable.

  • Takeaways & Limitations

    The authors recommend simple L2-SP as the standard baseline for inductive transfer learning because it is effective and simpler than alternatives.

  • Takeaways & Limitations

    Functional-level implicit-bias approaches remain untested and require assessment within inductive transfer learning.

Abstract

from arXiv · show

In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple $L^2$ penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.

1. Introduction

The introduction motivates explicit regularization for transfer learning because fine-tuning can erase relevant knowledge from the pre-trained model. It proposes using the pre-trained parameters as the penalty’s reference point rather than the origin.

  • 1. Introduction: Fine-tuning can drive parameters far from their initial values, causing important losses of knowledge assumed relevant to the target task.The paper identifies this as an inconsistency in transfer learning: initialization uses pre-trained parameters, while standard L2 regularization encourages departure from them.
  • 1. Introduction: Pre-trained convolutional networks transfer knowledge from large-scale source databases to target tasks through initialization with pre-trained parameters.This transfer is motivated by avoiding the data, computation time, and power consumption required to train deep networks from scratch.
  • 1. Introduction: Standard L2 regularization is argued inadequate for transfer learning because it pulls parameters toward the origin instead of the more sensible initial values.The proposed modification retains standard L2’s control of overfitting by constraining the effective search space around the pre-trained parameters.
  • 1. Introduction: The paper advocates coherent regularization that uses the pre-trained model both to initialize optimization and as the reference for an explicit inductive-bias penalty.This approach is intended to preserve knowledge embedded in the initial network.

2. Related Work

The related work situates the paper within inductive transfer learning, where parameter regularization encourages target solutions to remain similar to source solutions. It distinguishes deep-network transfer, which learns transferable representations, from earlier methods operating on fixed representations or other transfer settings.

  • Regularization precedents: Prior shrinkage methods regularized solutions toward adaptive targets, including transfer-learning applications to maximum-entropy models and support-vector machines.
  • Regularization precedents: Unlike fixed-representation methods that transfer classification parameters, deep-network transfer aims to learn similar representations before fitting target-task classifiers.
  • Transfer-learning setting: Inductive transfer learning here assumes identical source and target domains but different tasks, with abundant source data and limited labeled target data.
  • Convolutional-network transfer: Fine-tuning evolved from repurposing pretrained features to adapting whole convolutional networks, whose transfer success depends on representations learned from large datasets such as ImageNet.
  • Related transfer settings: Lifelong-learning methods preserve prior-task knowledge through output-based regularization or Fisher-information-weighted parameter penalties, while domain adaptation regularizes representations across differing domains.

3. Regularizers for Fine-Tuning

The section formalizes explicit regularization toward the pre-trained solution and introduces L2-SP, Fisher-weighted, sparsity-based, and group-based penalties for fine-tuning. These methods extend standard weight decay by preserving source-task parameters or feature extractors at varying granularity.

  • Explicit inductive-bias regularization complements early stopping by encouraging fine-tuned networks to remain similar to the pre-trained initial model.The paper contrasts these schemes with weight decay and freezing parts of the network.
  • General formulation: The regularized objective adds a penalty Ω(w) to the standard objective J, with Ω interpretable as a log prior under maximum a posteriori estimation.In the experiments, J is the negative log-likelihood.
  • L2-SP: L2-SP uses the source model’s parameter vector w0 as the reference in an L2 penalty, rather than driving adapted weights toward zero.The method applies the penalty to parameters shared with the source architecture and handles novel target parameters separately when architectures differ.
  • L2-SP-Fisher: L2-SP-Fisher weights deviations from w0 using diagonal Fisher information estimated on the source problem, prioritizing parameters according to source-task sensitivity.The Fisher estimate is computed from the average squared Fisher score over source inputs.
  • L1-SP and Group-Lasso-SP: L1-SP encourages individual parameters to remain equal to their pre-trained values, while Group-Lasso-SP encourages entire convolutional-channel groups to remain frozen.Group-Lasso-SP can preserve pre-trained feature extractors at arbitrary convolutional depths; its Fisher variant combines grouping with Fisher weighting.

4. Experiments · 4.1. Source and Target Databases

Experiments evaluate parameter regularizers with ResNet across source–target database pairs spanning object recognition and scene classification. Target datasets follow creator-recommended splits, with two training-size configurations for Caltech 256.

  • 4. Experiments: The experiments evaluate parameter regularizers using ResNet as the base network for transfer learning tasks.For classification targets, the final layer is replaced with a randomly initialized layer sized for the target classes.
  • 4.1. Source and Target Databases: ImageNet and Places 365 serve as source databases for generic object recognition and scene classification, respectively.
  • 4.1. Source and Target Databases: The target databases are Caltech 256, MIT Indoors 67, and Stanford Dogs 120, covering objects, indoor scenes, and dog breeds.
  • 4.1. Source and Target Databases: The database pairs are designed to compare how source–target problem similarity affects transfer learning.
  • 4.1. Source and Target Databases: Caltech 256 uses two training configurations: 30 or 60 randomly drawn examples per category, with 20 remaining examples for testing.
  • 4.1. Source and Target Databases: Each target database is divided into training and testing sets according to the split recommendations of its creators.The paper refers to Table 1 for additional dataset details.

4.2. Training Details

The experiments use standardized and augmented 224×224 inputs, cross-validated regularization, and fixed momentum-based optimization with repeated runs to report accuracy variability.

  • 4.2. Training Details: Images are resized to 256×256, mean-centered per channel, and randomly blurred, mirrored, and cropped to 224×224; grayscale images are duplicated across three channels.These preprocessing steps provide the network with three-channel inputs and data augmentation.
  • 4.2. Training Details: Network parameters are regularized as described in Section 3, with cross validation used to select the best regularization hyperparameters.The regularization configuration is chosen separately through cross validation.
  • 4.2. Training Details: Stochastic gradient descent uses momentum 0.9, batch size 64, 9000 iterations, and a tenfold learning-rate reduction after 6000 iterations, with five repeated runs for mean accuracy and standard deviation.Initial learning rates are 0.005, 0.01, or 0.02 depending on the task, and experiments use TensorFlow.

4.3. Results

L2-SP and L2-SP-Fisher consistently improve over L2 fine-tuning, with little difference between the two, while source-target similarity strongly affects transfer performance. Additional analyses show that alternative penalties and partial freezing are generally less effective or inefficient.

  • L2-SP and L2-SP-Fisher always improve over L2 fine-tuning, with no large difference between the two methods.Results report average accuracies and standard deviations across five runs.
  • Transfer learning performs much better when the source and target databases are alike, with comparable benefits for L2-SP and L2-SP-Fisher.Figure 2 compares Places365 and ImageNet as source databases across four target databases.
  • L1 and Group-Lasso penalties perform poorly, becoming less accurate than plain L2 on Caltech 256–60 when Places365 is the source.Their discontinuous gradients cause optimization difficulties for stochastic gradient descent, while proximal optimization also gives substandard results.
  • Fisher-based variants behave like their Euclidean counterparts because the target-only objective does not favor retaining source-task accuracy.L2 fine-tuning causes catastrophic forgetting, while L2-SP-Fisher produces a smaller performance drop than L2-SP when retaining the original classification layer.
  • Partial freezing can substantially improve L2 fine-tuning but is inefficient for L2-SP, and all partially frozen L2 variants are dominated by plain L2.Selecting the optimal number of frozen layers is costly to implement.

4.4. Analysis and Discussion

L2-SP and L2-SP-Fisher consistently achieve the best target-task accuracy among -SP methods, with no significant difference between them, so the simpler L2-SP is recommended. The approach adds negligible training cost, no test-time cost, and is analyzed through optimization and activation-similarity perspectives.

  • Method comparison and recommendation: L2-SP and L2-SP-Fisher always achieve better target-task accuracy than other -SP methods, with no significant difference between the two; therefore, the simpler L2-SP is recommended.The analysis focuses on L2-SP, although most conclusions also apply to L2-SP-Fisher.
  • Computational cost: L2-SP adds no parameters and increases ResNet-101 training-step operations by less than 1%, while imposing no additional test-time cost.The resulting computational burden is described as negligible relative to the reported classification-accuracy improvements.
  • Analytical interpretation: Under simplifying assumptions, L2-SP’s optimum is an affine compromise between the unregularized optimum and the pre-trained parameter vector along eigenvector directions of the unregularized objective’s Hessian.This contrasts with the analysis described for standard L2.
  • Discussion and limitations: Shrinkage reasoning suggests that references closer to the true parameters may be more effective, motivating investigation of parameter manifolds defined by the pre-trained network.The paper notes that true parameters are not readily applicable as a concept in deep networks.
  • Activation analysis: Activation similarities complement parameter-based comparisons by providing an easier-to-interpret view of how L2 and L2-SP fine-tuning differ.The activation analysis uses hidden-unit activations as an additional perspective on the experimental results.

5. Conclusion

The paper finds that explicit regularization toward the pre-trained solution improves inductive transfer learning, with L2-SP emerging as the recommended baseline. L2-SP outperforms standard L2 and layer freezing, while more elaborate penalties provide no demonstrated benefit.

  • Conclusion: The evaluated regularizers impose an explicit bias toward the source-task solution, creating a compromise with the pre-trained parameters.These techniques were tested for inductive transfer learning with deep convolutional networks.
  • Conclusion: L2-SP is more effective than standard L2 and freezing early layers, even with early stopping, while retaining source-task feature memory.The method uses the starting point as the reference for its L2 penalty and is supported by theoretical hints and strong experimental evidence.
  • Conclusion: L1 and Group-L1 penalties were not valuable for the considered inductive transfer-learning setting, and combining Fisher information with L2-SP did not improve target-task accuracy.
  • Conclusion: Implicit functional-level biases, such as the approach of Li and Hoiem (2017), remain to be evaluated for inductive transfer learning.Their value should be assessed because they are based on a different principle.

A. Effect of L2-SP Regularization on Optimization

A quadratic analysis shows that L2 regularization rescales parameters along Hessian eigenvector directions, while L2-SP instead combines the unregularized optimum with the pretrained reference model in each direction. The combination weights are determined by the Hessian eigenvalues and regularization strength.

  • Quadratic analysis: L2 regularization rescales parameters along Hessian eigenvector directions by λ_i/(λ_i+α).This follows from a quadratic approximation of the objective around its optimum.
  • L2-SP interpretation: In each Hessian eigenvector direction, L2-SP makes ˜w a convex combination of the unregularized optimum w∗ and reference model w0.The coefficients λ_i/(λ_i+α) and α/(λ_i+α) sum to 1.

B. Matching the State of the Art in Image Classification

L2-SP regularization, combined with aspect-ratio preservation and 10-crop testing, reaches or exceeds state-of-the-art image-classification performance on several benchmarks. The paper also supplements these experiments with the larger Foods 101 dataset.

  • State-of-the-art comparison: The experiments demonstrate that L2-SP and -SP regularization can provide a transfer-learning baseline significantly superior to standard fine-tuning.The authors do not aim to reach state of the art using this simple technique alone, but report that additional training tricks and post-processing can do so.
  • Aspect Ratio: An ablation compares the paper’s fixed 256×256 resizing, which ignores aspect ratio, with resizing whose shorter edge is 256 while preserving the original ratio.The ablation uses the same hyperparameters as the preceding experiments.
  • Post-Processing for Image Classification: The study applies 10-crop testing, averaging predictions from four corner crops, the center crop, and their horizontal reflections, to improve classification results.The authors also identify random rotation or scaling, additional crops, and multiscale testing as possible further improvements.
  • State-of-the-art comparison: With aspect-ratio preservation and 10-crop testing, Caltech 256-30 outperforms the state of the art, while MIT Indoors 67 and Stanford Dogs 120 approach it.The strongest competing approach on these latter datasets used many training examples.
  • Additional dataset: Foods 101 is added as a larger supplementary dataset covering 101 food categories, though its training images and class labels are relatively rough.Its image quality and class labels are described as rough compared with the other datasets.

C. Application of L2-SP to Semantic Image Segmentation

This section tests whether L2-SP transfers beyond classification by applying it to pixel-wise semantic image segmentation. The evaluation uses Cityscapes with ImageNet as the source task and considers ResNet and DeepLab-V2 architectures.

  • C. Application of L2-SP to Semantic Image Segmentation: L2-SP is applied to semantic image segmentation to assess its versatility beyond image classification transfer.Segmentation labels each image pixel by the category of the object it belongs to, yet still benefits from fine-tuning.
  • C. Application of L2-SP to Semantic Image Segmentation: The evaluation uses Cityscapes, a 5,000-image pixel-wise urban-scene segmentation benchmark, with ImageNet as the source dataset.Cityscapes is split into 2,975 training, 500 validation, and 1,525 test images at 2048×1024 resolution.
  • C. Application of L2-SP to Semantic Image Segmentation: Experiments consider standard ResNet and DeepLab-V2 convolutional architectures for Cityscapes segmentation.ResNet is adapted by removing its global pooling layer, while DeepLab-V2 is included as a strong Cityscapes architecture.
Loading 1802.01483v2…