PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Hyperparameters for Fine-tuning

Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto

arXiv:2002.11770v1cs.CVcs.LGstat.ML

TL;DR

Fine-tuning commonly reuses ad-hoc hyperparameters from training large models, but their suitability across transfer tasks remains uncertain. The paper empirically re-examines these choices and finds that effective learning rate, momentum, source–target similarity, and regularization assumptions shape fine-tuning performance.

  • Problem

    Fine-tuning commonly retains hyperparameters from training large models, while momentum and interactions among hyperparameters have received limited investigation.

  • Method

    The paper performs extensive hyperparameter searches across transfer-learning benchmarks and examines momentum, effective learning rate, and regularization choices.

  • Results

    Optimal fine-tuning hyperparameters depend on the dataset and source–target similarity; effective learning rate matters, and momentum can affect performance.

  • Takeaways & Limitations

    Fine-tuning should not automatically reuse training-from-scratch settings; hyperparameter selection should account for source–target similarity and coupled parameters.

  • Takeaways & Limitations

    For target domains quite different from the source, the fine-tuned model does not necessarily need to remain close to the initial model.

Abstract

from arXiv · show

Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyperparameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks. (1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that the value of momentum also affects fine-tuning performance and connect it with previous theoretical findings. (2) Optimal hyperparameters for fine-tuning, in particular, the effective learning rate, are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyperparameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of fine-tuning and encourages deep learning practitioners to rethink the hyperparameters for fine-tuning.

1 INTRODUCTION

The paper questions whether training-from-scratch hyperparameters and common fine-tuning beliefs remain valid across transfer tasks. It re-examines these practices through extensive hyperparameter searches and finds that momentum, effective learning rate, and source–target similarity matter.

  • 1 INTRODUCTION: Fine-tuning is widely used because pretrained models can reduce overfitting with limited data and substantially reduce training time.This motivates examining its hyperparameters even when target datasets are sufficiently large.
  • 1 INTRODUCTION: Common fine-tuning practice uses reduced learning rates and shorter schedules while retaining defaults from large-model training, based on preserving learned features.The paper asks whether these defaults remain appropriate after changing initialization and task conditions.
  • 1 INTRODUCTION: The study conducts extensive hyperparameter searches across transfer-learning benchmarks to explain when particular settings succeed or fail.Its goal is understanding hyperparameter effectiveness rather than achieving state-of-the-art results on each task.
  • 1 INTRODUCTION: Fine-tuning hyperparameters depend on both the target dataset and its similarity to the source domain, unlike typical training-from-scratch settings.Using ImageNet-derived schedules therefore cannot guarantee good fine-tuning performance.
  • 1 INTRODUCTION: Momentum is an underexplored fine-tuning parameter, despite common practice retaining the default value of 0.9.The paper identifies momentum as a variable that can materially affect fine-tuning outcomes.

2 RELATED WORK

Related work examines transfer performance, domain similarity, and hyperparameter selection, but leaves fine-tuning momentum and its interactions comparatively underexplored.

  • 2 RELATED WORK: Prior studies show that fine-tuning the whole pretrained network generally outperforms using it only as a static feature extractor.This literature also investigates selecting source images with similar local features for transfer.
  • 2 RELATED WORK: Transfer effectiveness can become negative when source and target domains are mismatched, while ImageNet top-1 accuracy correlates strongly with transfer accuracy.ImageNet pretraining may provide minimal benefits in some settings.
  • 2 RELATED WORK: Training-from-scratch research mainly studies batch size, learning rate, and weight decay, with relatively few studies focused on momentum selection.Existing work also considers automatic tuning and systematic correlations among hyperparameters.

3 TUNING HYPERPARAMETERS FOR FINE-TUNING

Extensive transfer-learning experiments show that fine-tuning hyperparameters are coupled and depend on source–target domain similarity, challenging defaults borrowed from training from scratch.

  • 3.1 EXPERIMENTAL SETTINGS: Fine-tuning experiments span seven image-classification datasets and source models pretrained on ImageNet, Places-365 and iNaturalist.The study mainly uses ResNet-101-V2 and reports similar observations for DenseNets and MobileNet.
  • 3.2 EFFECT OF MOMENTUM AND DOMAIN SIMILARITY: Momentum 0 works better for Dogs, Caltech and Indoor, whereas momentum 0.9 works better for Cars, Aircrafts and Flowers.These patterns align with higher similarity to ImageNet for the former group and greater dissimilarity for the latter.
  • 3.3 COUPLED HYPERPARAMETERS AND THE VIEW OF EFFECTIVE LEARNING RATE: When effective learning rate η′ is fixed, momentum 0 and 0.9 achieve almost equivalent optimal performance after other hyperparameters are tuned.Different effective learning rates, rather than momentum alone, produce different performance; η′ = η/(1 − m).
  • 3.3 COUPLED HYPERPARAMETERS AND THE VIEW OF EFFECTIVE LEARNING RATE: The optimal effective learning rate depends on both source and target domains, with smaller values for Dogs, Caltech and Indoor than for Aircrafts, Flowers and Cars.The relationship between domain similarity and optimal effective learning rate is consistent across ResNet, DenseNet and MobileNet architectures.
  • 3.3 COUPLED HYPERPARAMETERS AND THE VIEW OF EFFECTIVE LEARNING RATE: Effective weight decay λ′ = λ/η is inversely related to optimal learning rate and is also correlated with domain similarity.L2 and L2-SP regularization change weight norms differently, and their optimal learning rates can differ.

4 DISCUSSION

Fine-tuning hyperparameters are dataset dependent and also influenced by source–target domain similarity, unlike typical training-from-scratch settings.

  • 4 DISCUSSION: Fine-tuning hyperparameters depend on both the dataset and source–target domain similarity, so scratch-training defaults may yield inferior performance.Momentum and effective learning rate can materially affect performance, while domain similarity can reduce the hyperparameter search space.
  • 4 DISCUSSION: The optimal effective learning rate correlates with source–target domain similarity, connecting momentum selection to the effective-learning-rate rule.
  • 4 DISCUSSION: Exhaustive hyperparameter search is computationally expensive, whereas ad-hoc scratch-training settings can produce inferior fine-tuning performance.

A THE EFFECTIVENESS OF MOMENTUM

Momentum affects fine-tuning performance, and disabling it changes the learning rate needed for comparable performance through the effective learning rate.

  • A THE EFFECTIVENESS OF MOMENTUM: Zero momentum outperforms nonzero momentum in 5 of 6 Birds configurations when learning rate is fixed.
  • A THE EFFECTIVENESS OF MOMENTUM: Figure 7 compares training and validation errors while searching momentum on Birds with fixed learning rate and weight decay.
  • A THE EFFECTIVENESS OF MOMENTUM: A 10x larger learning rate generally gives similar performance after changing momentum from 0.9 to 0.0.The relationship follows η′ = η/(1 − m), and the observation is reproduced across datasets.

B DOMAIN SIMILARITY

The paper estimates source–target domain similarity using feature-based Earth Mover Distance, but substitutes ImageNet features when the original JFT resources are unavailable.

  • B DOMAIN SIMILARITY: Domain similarity is computed from class-averaged feature vectors and label weights, using Earth Mover Distance between source and target domains.Distances compare averaged features for source and target labels, while label frequencies provide transport weights.
  • B DOMAIN SIMILARITY: When JFT and its pretrained ResNet-101 are unavailable, the paper uses its pretrained ImageNet ResNet-101 as the feature extractor.
  • B DOMAIN SIMILARITY: Similarity rankings show consistent patterns across architectures, with Caltech and Dogs among the most similar and Cars and Aircrafts among the most dissimilar datasets.

C THE EFFECTIVENESS OF BN MOMENTUM

The study finds little sensitivity to BN momentum values above 0.9, although 0.99 slightly improves performance on some datasets.

  • C THE EFFECTIVENESS OF BN MOMENTUM: Prior work suggested decreasing BN momentum to max(1 − 10/s, 0.9), with the change applying when steps per epoch exceed 100.
  • C THE EFFECTIVENESS OF BN MOMENTUM: BN momentum 0.99 slightly improves performance on some datasets, while values above 0.9 show no significant overall differences.The experiment varies BN momentum across 0.0, 0.9, 0.95, and 0.99 using previously optimized hyperparameters.
  • C THE EFFECTIVENESS OF BN MOMENTUM: The study varies BN momentum while fine-tuning each dataset with previously obtained best hyperparameters.

D EXPERIMENTAL SETTINGS FOR COMPARISON OF L2 AND L2-SP

The experiments use a pretrained ResNet-101-V1 model and grid-search learning rate and weight decay while keeping other fine-tuning settings fixed.

  • The study fine-tunes an ImageNet-pretrained ResNet-101-V1 with batch size 64 for 9000 iterations and one learning-rate decay.Momentum is fixed at 0.9, while learning rate and weight decay are selected by grid search.

E DATA AUGMENTATION

Data augmentation affects fine-tuning differently across datasets, and its impact depends on momentum and learning rate. Strong augmentation can improve generalization on some datasets but overfit or slow optimization on others.

  • The comparison varies random resized cropping, random cropping, and random horizontal flipping across momentum settings, with n = 256 and λ = 0.0001.The figure and table compare these augmentation choices under different hyperparameter settings.
  • Strong augmentation converges slowly and needs more epochs on datasets such as Aircrafts, while simple augmentation converges faster in training error.Strong augmentation also overfits Dogs with default hyperparameters but performs well when m = 0.
  • Advanced augmentation with default m = 0.9 and η = 0.01 overfits Dogs but generalizes better on Aircrafts and Flowers.The supplied passage attributes these contrasting outcomes to dataset-dependent augmentation effects.
  • Disabling momentum removes overfitting for Dogs and Caltech, whereas random cropping can underfit or become stuck under the same change.Increasing the learning rate can mitigate the poorer behavior observed with random cropping.
  • Random resized cropping samples rectangular regions with aspect ratios in [3/4, 4/3] and areas in [8%, 100%].
  • When η increases from 0.01 to 0.05, zero momentum performs better than nonzero momentum on datasets close to the source.

F SOURCE DOMAINS

The experiments compare pretrained models and training from scratch while showing that hyperparameter optimization can change conclusions about transfer-learning benefits across source and target domains.

  • The study uses ImageNet-pretrained ResNet-101_v2 models, while iNat-2017 and Places365 models are obtained by fine-tuning from ImageNet.Table 7 reports Top-1 errors for pretrained models from different source datasets.
  • Training from scratch uses η = 0.1, λ = 0.0001, m = 0.9, and n = 256 for 600 epochs, with learning-rate decays at epochs 400 and 550.The HPO search varies η over [0.1, 0.2, 0.5] and λ over [0.0001, 0.0005].
  • Hyperparameter optimization invalidates several default-setting conclusions, including that fine-tuning always surpasses training from scratch.With HPO, training from scratch surpasses default fine-tuning on Cars and Aircrafts, and the performance gap becomes much smaller.
  • Figure 13 compares training and validation errors across learning rates and weight decays with batch size 256 and momentum 0.9.
Loading 2002.11770v1…