Source-linked AI summary
LEEP: A New Measure to Evaluate Transferability of Learned Representations
Cuong V. Nguyen, Tal Hassner, Matthias Seeger, Cedric Archambeau
TL;DR
Transferability estimation seeks to predict how well knowledge transfers between tasks without costly target-task training. The paper introduces LEEP, a one-forward-pass measure based on an expected empirical predictor, and finds that it predicts transfer and meta-transfer performance, including on small or imbalanced targets, while correlating better with actual transfer accuracy than negative conditional entropy and H scores.
Problem
Transferability measures should estimate target-task transfer effectiveness without expensive optimization, but prior approaches can be difficult to interpret or rely on restrictive data assumptions.
Method
LEEP estimates transferability from a source model and target dataset using a single forward pass and the average log-likelihood of an expected empirical predictor.
Results
LEEP predicts transfer and meta-transfer learning performance, including for small or imbalanced target datasets, and correlates better with actual transfer accuracy than negative conditional entropy and H scores.
Takeaways & Limitations
LEEP can efficiently estimate transfer performance before executing transfer or meta-transfer learning algorithms and can support source-model selection.
Takeaways & Limitations
LEEP assumes a source model rather than a source dataset, making it natural for pretrained-model settings but not identical to the source-dataset setting used by negative conditional entropy.
Abstract
from arXiv · showhide
We introduce a new measure to evaluate the transferability of representations learned by classifiers. Our measure, the Log Expected Empirical Prediction (LEEP), is simple and easy to compute: when given a classifier trained on a source data set, it only requires running the target data set through this classifier once. We analyze the properties of LEEP theoretically and demonstrate its effectiveness empirically. Our analysis shows that LEEP can predict the performance and convergence speed of both transfer and meta-transfer learning methods, even for small or imbalanced data. Moreover, LEEP outperforms recently proposed transferability measures such as negative conditional entropy and H scores. Notably, when transferring from ImageNet to CIFAR100, LEEP can achieve up to 30% improvement compared to the best competing method in terms of the correlations with actual transfer accuracy.
1. Introduction
The paper introduces LEEP as an efficient, interpretable, and general measure for estimating transferability without target-task training. Experiments show it predicts transfer performance and convergence speed across standard and meta-transfer learning settings, including difficult target datasets.
- Motivation and contribution: LEEP estimates how effectively knowledge transfers from a source model to a target task without training on the target task.It is designed to address expensive optimization, limited interpretability, and restrictive data assumptions in prior measures.
- Motivation and contribution: LEEP scores require a single forward pass of the source model through the target data and have a simple expected-predictor interpretation.The score is the average log-likelihood of a predictor based on the empirical conditional distribution between source and target labels.
- Motivation and contribution: LEEP applies broadly to modern deep networks without assumptions on source and target samples beyond equal input size.This setting includes source and target inputs such as ImageNet and CIFAR images scaled to the same size.
- Empirical evaluation: LEEP predicts head-retraining and fine-tuning performance on large, small, and imbalanced target datasets, and predicts fine-tuning convergence speed.Small or imbalanced datasets are difficult to use for retraining, making prediction in these settings particularly relevant.
- Empirical evaluation: LEEP predicts the performance of Conditional Neural Adaptive Processes, providing a transferability measure for meta-transfer learning.The paper identifies this as the first such measure to its knowledge.
- Empirical evaluation: LEEP correlates better with actual transfer accuracy than negative conditional entropy and H scores and supports source-model selection.The comparison evaluates both transferability correlation and model-selection effectiveness.
2. Log Expected Empirical Prediction
The transfer setting uses a pretrained source model and labeled target data, while LEEP estimates transferability from one forward pass and an empirical source-to-target label relationship. Its score measures how well an expected empirical predictor explains the target data.
- Transfer setting: Head retraining freezes the source feature extractor and learns a target classifier, whereas fine-tuning updates the extractor and retrains the new head.Both methods use representations learned by the source model for the target task.
- Transfer setting: Transferability estimation asks how effectively a source model can transfer knowledge to a target task without training on target data.The source is a pretrained model and the target is a labeled dataset.
- LEEP computation: LEEP computes dummy source-label distributions by applying the source model to every target input.These predicted source-label distributions need not be semantically related to the target examples’ true labels.
- LEEP computation: The method estimates the empirical joint and conditional distributions between target labels and source-label predictions.The joint distribution sums source-label probabilities over target examples sharing each target label, then yields the empirical conditional distribution.
- LEEP computation: The Expected Empirical Predictor samples a source label from the model output and a target label from the empirical conditional distribution.Equivalently, it predicts from the mixture p(y|x; θ, D) formed by combining those distributions.
- LEEP computation: LEEP is the average log-likelihood of the Expected Empirical Predictor on the target dataset.The measure is intended to indicate how close the source model and target data are while using the target data minimally.
- LEEP computation: LEEP is always negative, with larger values indicating better transferability, and its bottleneck requires only one forward pass through the target data.Scores tend to decrease when the target task has more classes.
3. Theoretical Properties of LEEP
LEEP is theoretically connected to optimal head retraining and negative conditional entropy. These properties clarify how its score relates to transfer performance and why its source-model setting differs from NCE.
- Property 1: LEEP is a lower bound of the optimal average log-likelihood achieved by retraining the classifier while freezing the feature extractor.The optimal model is obtained by maximizing target-data average log-likelihood over classifiers applied to the fixed representation.
- Implementation: The EEP assumption can be satisfied by including the EEP alongside an optimizable classifier family and selecting the better-scoring classifier.The proposed two-stage procedure first optimizes over the ordinary classifier family, then compares that result with the EEP.
- Property 2: LEEP is bounded above by negative conditional entropy plus the average log-likelihood of the source model’s dummy labels.The dummy labels are obtained by assigning each target example the source model’s most probable source label.
- Relationship to NCE: LEEP lies between the negative conditional entropy expression and the average log-likelihood of a retrained model, linking it to both quantities.When the retrained model does not overfit, its average log-likelihood is described as a reasonable indicator of performance.
- Comparison with NCE: LEEP is more natural than NCE when only a source model is available, and it is not restricted to tasks sharing identical input instances.NCE instead assumes that the source data set is given and that source and target data sets share the same input examples.
4. Related Work
The paper situates LEEP among transferability estimation, meta-transfer learning, task-space representations, and domain adaptation. Its distinguishing focus is predicting transfer performance from a source model without executing transfer algorithms.
- Transfer learning: Transferability estimation seeks to predict how effectively knowledge transfers between classification tasks without actually running the transfer algorithms.The paper frames this as predicting transfer performance between source and target tasks.
- Domain adaptation: The paper distinguishes its transferability objective from domain adaptation settings that impose a label-shift-only assumption.Its setting allows both input and label distributions to change arbitrarily.
- Transferability measures: LEEP differs from negative conditional entropy by removing the requirement that source and target data sets share the same input examples.NCE is defined between source and target label sets under that shared-input assumption.
- Meta-transfer learning: LEEP scores can predict the performance of conditional neural adaptive processes, extending transferability estimation to meta-transfer learning.The paper presents this as the first transferability measure developed for meta-transfer learning, to its authors’ knowledge.
- Task space representation: Task-space methods estimate transferability from representations or distances between tasks, whereas LEEP evaluates transferability from a source model and target data.Task2Vec is cited as an example that maps tasks or data sets to vectors before estimating transferability.
5. Experiments
The experiments evaluate whether LEEP predicts transfer and meta-transfer performance across standard, small, noisy, imbalanced, and model-selection settings. LEEP correlates with transferred-model performance, convergence speed, and CNAP accuracy, while generally outperforming NCE and H scores.
- 5.1. LEEP vs. Transfer Accuracy: LEEP scores correlate with transferred-model test accuracies above 0.94, with p < 0.001, across 200 CIFAR100 target tasks.The experiments use ResNet18/ImageNet and ResNet20/CIFAR10 source models with head retraining and fine-tuning.
- 5.2. LEEP vs. Transfer Accuracy in Small Data Regime: In small target-data settings, LEEP shows positive correlations above 0.5 in most cases, except fine-tuning from the CIFAR10-pretrained model.Binning LEEP scores into five transferability levels generally yields better accuracies at higher levels.
- 5.3. LEEP vs. F1 Score on Imbalanced Data: With 15% target-label flips, LEEP remains positively correlated with transfer accuracy, although correlations are weaker than with correct labels.For imbalanced targets, correlations with test F1 exceed 0.5 in all cases, and higher LEEP levels imply better F1 scores.
- 5.4. LEEP vs. Accuracy of Meta-Transferred Models: LEEP predicts CNAP performance with correlation coefficient 0.591 and p < 0.001, while higher LEEP levels correspond to better CNAP test accuracies.This evaluates meta-transfer from an ImageNet-pretrained ResNet18 on 200 CIFAR100 target tasks.
- 5.5. LEEP vs. Convergence of Fine-tuned Models: Higher LEEP transferability levels correspond to faster fine-tuning convergence and larger margins over reference models trained from scratch.Fine-tuned models match reference-model performance using far fewer training epochs, especially for highly transferable tasks.
- 5.6. Comparison of LEEP, NCE, and H scores: LEEP has equal or better correlations than NCE in all but two fine-tuning cases, improves correlation coefficients by up to 30%, and outperforms H scores in 16/23 cases.H scores completely fail in 11 cases, whereas LEEP captures transferability in all cases; LEEP also predicts head-retrained source-model performance better than the baselines.
6. Discussions
LEEP efficiently estimates transferability before transfer learning and is designed to work across algorithms and heterogeneous tasks. Its scores can support source-model selection and may extend to related learning settings.
- LEEP estimates transfer and meta-transfer performance before executing the learning algorithms.
- LEEP scores require a pre-trained source model and can depend on its architecture and performance.
- LEEP aims to remain algorithm-independent by quantifying the relationship between a source model and target data.
- The source model’s output label distribution indirectly contains feature information through the final linear transformation and Softmax.
- LEEP can be extended to transform learned feature vectors directly into probability distributions, but this changes the dummy distribution’s interpretation.
- For heterogeneous source and target tasks, greater source-model uncertainty produces more uniform dummy labels and smaller LEEP scores.
- LEEP scores can support selecting transferable source-target pairs and source models for transfer or meta-transfer learning.
A.1. Proof of Property 1
The supplied passages identify experimental figures for small balanced and imbalanced targets, but do not provide the proof of Property 1.
- Figure 6 reports average test accuracy across five LEEP-defined transferability levels for small balanced target data sets.
- Figure 7 reports average test F1 score across five LEEP-defined transferability levels for small imbalanced target data sets.
A.2. Proof of Property 2
The supplied passages define dummy and true label sequences and refer to a prior proof, but do not state the proof of Property 2.
- The negative conditional entropy comparison uses dummy labels Z and true labels Y for the target examples.
- The passage refers to the proof of Theorem 1 from Tran et al. without reproducing its argument.
B. Full Experimental Results
The full experimental-results passages describe evaluations of small targets, convergence, and source-model selection, with figures defining the corresponding comparisons.
- Figure 6 evaluates average test accuracy on small balanced targets grouped into five LEEP-based transferability levels.
- Figure 7 evaluates average test F1 score on small imbalanced targets grouped into five LEEP-based transferability levels.
- Figure 8 uses two LEEP transferability levels for FashionMNIST target tasks to clarify convergence comparisons.
- Figure 8 measures fine-tuning convergence using accuracy differences from a reference model trained from scratch on the target data.
- Figure 9 compares LEEP, NCE, H score, and ImageNet accuracy for source-model selection under head retraining and fine-tuning.