PaperlayerView exact revision on arXiv

Source-linked AI summary

LogME: Practical Assessment of Pre-trained Models for Transfer Learning

Kaichao You, Yong Liu, Jianmin Wang, Mingsheng Long

arXiv:2102.11005v3cs.LGcs.AI

TL;DR

The paper addresses task-adaptive selection of pre-trained models without fine-tuning, extending beyond prior methods limited to supervised classification. It uses logarithmized maximum label evidence from extracted features, with an optimized computation, and reports broad applicability across models, tasks, and modalities alongside substantial speed and memory savings.

  • Problem

    Pre-trained model selection is task adaptive, but prior methods provide limited guidance and cannot cover unsupervised models or regression tasks.

  • Method

    LogME estimates the logarithm of maximum label evidence from features extracted by each pre-trained model, using evidence integration over model weights.

  • Results

    LogME applies across supervised and unsupervised models, classification and regression, and vision and language, with over 3000× speedup compared with vanilla fine-tuning.

  • Takeaways & Limitations

    LogME provides a practical assessment method for selecting pre-trained models across diverse transfer-learning settings.

  • Takeaways & Limitations

    The paper leaves accounting for representations after fine-tuning as future work.

Abstract

from arXiv · show

This paper studies task adaptive pre-trained model selection, an underexplored problem of assessing pre-trained models for the target task and select best ones from the model zoo \emph{without fine-tuning}. A few pilot works addressed the problem in transferring supervised pre-trained models to classification tasks, but they cannot handle emerging unsupervised pre-trained models or regression tasks. In pursuit of a practical assessment method, we propose to estimate the maximum value of label evidence given features extracted by pre-trained models. Unlike the maximum likelihood, the maximum evidence is \emph{immune to over-fitting}, while its expensive computation can be dramatically reduced by our carefully designed algorithm. The Logarithm of Maximum Evidence (LogME) can be used to assess pre-trained models for transfer learning: a pre-trained model with a high LogME value is likely to have good transfer performance. LogME is \emph{fast, accurate, and general}, characterizing itself as the first practical method for assessing pre-trained models. Compared with brute-force fine-tuning, LogME brings at most $3000\times$ speedup in wall-clock time and requires only $1\%$ memory footprint. It outperforms prior methods by a large margin in their setting and is applicable to new settings. It is general enough for diverse pre-trained models (supervised pre-trained and unsupervised pre-trained), downstream tasks (classification and regression), and modalities (vision and language). Code is available at this repository: \href{https://github.com/thuml/LogME}{https://github.com/thuml/LogME}.

1. Introduction

Task-adaptive pre-trained model selection is difficult because different downstream tasks favor different models, while prior guidance is limited. LogME addresses this need with an efficient, general assessment method validated across diverse models, tasks, and modalities.

  • Different downstream tasks favor different pre-trained models, making model selection task adaptive and non-trivial.
  • A practical assessment method must be fast, accurate, and general enough to identify strong models without brute-force fine-tuning.
  • Prior task-adaptive selection methods were limited mainly to supervised pre-trained models transferred to classification tasks.
  • LogME estimates maximum label evidence from extracted features and applies to both classification and regression tasks.
  • At most 3000× speedup and 1% memory footprint are reported for LogME compared with brute-force fine-tuning.
  • LogME is validated on 22 pre-trained models and 17 downstream tasks spanning supervised and unsupervised models, classification and regression, and vision and language.

2. Related Works

Related work covers transfer learning, supervised and unsupervised pre-trained models, and prior transferability measures. Existing selection methods provide limited guidance because they focus on supervised models and classification tasks.

  • Transfer learning includes several paradigms, while deep-learning usage commonly refers to inductive transfer through fine-tuning.
  • Pre-trained models are neural networks trained on large-scale datasets and transferred to downstream tasks.
  • Supervised, contrastive, and language pre-training provide distinct sources of reusable representations, including models trained without categorical labels.
  • No single pre-trained model dominates across downstream tasks, so the best choice depends on both task and model characteristics.
  • Transferability studies based on exhaustive fine-tuning require computational resources that are difficult for practitioners to afford.
  • LEEP and NCE estimate transferability using label distributions but handle only supervised pre-trained models and classification tasks.

3. Problem setup

The problem setup assigns each candidate pre-trained model a score without fine-tuning and evaluates whether score rankings match fine-tuned transfer performance. Weighted Kendall’s τ emphasizes agreement, especially among stronger models.

  • Given M pre-trained models and labeled data, the goal is to score each model so top transfer performers can be selected without fine-tuning.
  • Ground-truth transfer performance is measured by task-appropriate metrics such as accuracy, MAP, or MSE after properly tuned fine-tuning.
  • Assessment quality is evaluated by comparing predicted model-score rankings with transfer-performance rankings.
  • Weighted Kendall’s τ measures concordant and discordant model pairs, with larger values indicating better assessment correlation.
  • The weighted variant emphasizes pairwise agreement among higher-performing models because selection prioritizes top candidates.
  • The authors use weighted Kendall’s τ for interpretability, while other proper rank-correlation metrics could also be adopted.

4. The LogME approach

LogME assesses pre-trained models by maximizing and normalizing label evidence given fixed extracted features, extending the approach across task types while reducing its computational cost.

  • Feature extraction: The method uses each pre-trained model as a fixed feature extractor and scores its features independently of other models.It requires only the representation model, unlike methods that also use a pre-trained classification head.
  • Evidence calculation: Evidence integrates over possible linear-model weights instead of evaluating likelihood at one optimized weight, reducing over-fitting risk.The weight prior is isotropic Gaussian, and observations follow a normal distribution centered on w^T f_i.
  • Evidence maximization: LogME is the average maximum log evidence of labels given pre-trained features, obtained by maximizing evidence over α and β.The parameters are optimized through alternating updates involving m and γ.
  • Task extensions: LogME handles multivariate regression by averaging dimension-wise scores and handles classification by converting labels to one-hot targets.Thus, the same assessment method applies to both classification and regression tasks.
  • Computational optimization: The optimized algorithm avoids matrix inversion and matrix-matrix multiplication by decomposing F^T F and using matrix-vector products.Updating Λ instead of A^-1 reduces the computational burden and makes the Bayesian approach fast enough for practical use.
  • Computational optimization: The optimized implementation reduces wall-clock time by the order of 10^2 while retaining LogME’s general, accurate, and efficient assessment.Table 2 presents the computational complexity analysis for Algorithm 1.

5. Experiments

Experiments evaluate LogME on toy classification and regression data, supervised vision transfers, regression, contrastive models, and efficiency. Across these settings, LogME tracks transfer quality, applies beyond prior methods’ scope, and substantially reduces assessment cost.

  • Toy data: Increasing feature noise makes LogME smaller for both synthesized classification and regression problems.Classification adds Gaussian noise to separable features; regression adds noise to x, making y harder to predict.
  • Toy data: Toy experiments show that LogME measures feature quality and can assess pre-trained models for transfer learning.The experiments vary noise to mimic representations with decreasing transferability.
  • Supervised classification: LogME consistently correlates better with fine-tuned accuracy than LEEP and outperforms NCE on 7 of 9 classification datasets.LEEP and NCE show negative correlation on DTD, whereas LogME remains reasonably effective there.
  • Supervised classification: A larger LogME predicts better fine-tuned transfer with at least 75% probability, reaching 85% or 90% for most tasks.The smallest reported τw is around 0.5; most tasks have τw of 0.7 or 0.8.
  • Supervised regression: On dSprites regression, LogME correlates with MSE at τw = 0.84, and higher LogME predicts lower post-fine-tuning MSE with 92% probability.The task predicts scale, orientation, and x,y positions from sprite images.
  • Contrastive models: For contrastive pre-trained models, LogME gives the perfect ordering of transferred accuracy and MSE across Aircraft classification and dSprites regression.The model ranking differs between tasks, underscoring task-adaptive selection; SimCLR on dSprites did not converge after several trials.
  • Efficiency: Compared with brute-force fine-tuning, LogME achieves over 3000× speedup and 120× lower memory footprint in computer vision, and 86× speedup in NLP.LogME costs almost the same as feature extraction, while fine-tuning one model on one dataset requires about a day.

6. Conclusion

LogME provides a fast, accurate, and general approach to assessing pre-trained models for transfer learning, while measuring model quality from static representations before fine-tuning. Accounting for representations after fine-tuning is left for future work.

  • LogME offers a general, accurate, and efficient method for assessing pre-trained models in transfer learning.The method is presented as applicable across broad transfer-learning settings.
  • The assessment measures pre-trained-model quality using static representations before fine-tuning.Static representations are explicitly defined as representations before fine-tuning.
  • Dynamic representations after fine-tuning are not modeled and are left as future work.Such representations could account for changes to pre-trained models during fine-tuning.

A. Dataset description and statistics

The evaluation uses diverse image datasets spanning fine-grained objects, animals, scenes, textures, and general object categories, with dataset-specific class and sample counts. Official splits are retained when available; otherwise, data are divided into training, validation, and test sets.

  • Aircraft contains 10,000 images across 100 fine-grained aircraft classes, with 100 images per class.
  • Birdsnap contains 49,829 images representing 500 North American bird species.
  • Caltech contains 9,144 object images across 101 categories, with roughly 40 to 800 images per category.
  • Cars contains 16,185 images across 196 car classes, split into 8,144 training and 8,041 testing images.
  • CIFAR 10, CIFAR 100, DTD, Pets, and SUN cover 10 to 397 classes with dataset sizes from 5,640 to 60,000 images.CIFAR 10 has 60,000 images; CIFAR 100 has 100 classes; DTD has 5,640 images and 47 classes; Pets has 7,049 images and 47 classes; SUN has 39,700 images and 397 classes.
  • Official train, validation, and test splits are used when available; otherwise, the split is 60%, 20%, and 20%, respectively.The validation set is used for hyper-parameter tuning.

B. Comparing LogME to re-training head

Re-training a downstream head is slower and less effective than LogME for assessing pre-trained models. Its correlation with transfer performance can decline as hyper-parameter trials increase, indicating some over-fitting.

  • Re-training a classification or regression head is a naive assessment strategy, but empirical results show it works less well than expected.The approach is also called linear probing or linear protocol evaluation.
  • Re-training heads is impractical or slow because second-order optimization is impractical, while first-order optimization depends on learning-rate schedules and converges slowly.
  • Extensive tuning of logistic-regression L2 regularization makes head re-training inefficient.
  • In Caltech experiments with 10 pre-trained models, LogME required 3× less time than re-training the head.The comparison used softmax regressors with tuned L2 regularization and examined correlation with transfer performance.
  • Figure 7 shows that re-training-head correlation can decline as hyper-parameter trials increase, reflecting some over-fitting.The figure compares the correlation of re-training heads with the correlation of LogME.
  • Re-training the head is neither as efficient nor as effective as LogME.

C. Original Results in Figures

The paper provides original numerical results corresponding to Figures 4, 5, and 6 in Tables 5, 6, and 7. These tables serve as the tabulated versions of those figures.

  • Table 5 contains the original results corresponding to Figure 4.
  • Table 6 contains the original results corresponding to Figure 5.
  • Table 7 contains the original results corresponding to Figure 6.
Loading 2102.11005v3…