Source-linked AI summary
An Information-Theoretic Approach to Transferability in Task Transfer Learning
Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, Leonidas Guibas
TL;DR
Task transfer learning needs an explicit, data-based way to estimate when and how well representations transfer between tasks. The paper introduces H-score, an information-theoretic metric for classification transferability, and reports efficient computation, strong consistency with empirical measurements, and successful prediction across image-task settings.
Problem
Task transferability is difficult to estimate because empirical measures require learning experiments, while task-relatedness theories may be non-computable from data or fail to explain transfer performance.
Method
H-score analytically characterizes expected log-loss for a feature predicting task labels and defines transferability as the normalized score of an optimal source-task feature for a target task.
Results
H-score was strongly consistent with empirical transferability measurements and successfully predicted feature-transfer performance from ImageNet-1000 classification to CIFAR-100.
Takeaways & Limitations
The metric supports efficient, data-based comparison of transferability for source-task selection and broader multi-task transfer procedures.
Takeaways & Limitations
Exact operational transferability requires fine-tuning layers to contain only linear transformations, although the metric can still compare relative transferability with fine-tuning empirically.
Abstract
from arXiv · showhide
Task transfer learning is a popular technique in image processing applications that uses pre-trained models to reduce the supervision cost of related tasks. An important question is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems using statistical and information theoretic principles. Experiments on real image data show that our metric is not only consistent with the empirical transferability measurement, but also useful to practitioners in applications such as source model selection and task transfer curriculum learning.
1 Introduction
Task transfer learning seeks to estimate when representations from a source task will help a target task without relying solely on costly empirical transfer experiments. The paper introduces H-score as an analytically computable, information-theoretic transferability measure.
- Task transfer learning reuses a source-task model for a target task with the same input domain, potentially reducing supervision requirements.
- Transferability asks when transfer will work and to what extent, enabling systematic search over potential source and target tasks.
- Existing approaches measure transferability empirically through validation loss or accuracy, while theoretical relatedness measures may be non-computable or disconnected from transfer performance.
- H-score characterizes the expected log-loss of predicting task labels from an input feature and defines task transferability using the normalized score of an optimal source feature.
- The proposed metric is theoretically grounded, computable directly from input data with fewer samples than empirical learning, and strongly consistent with empirical transferability measurements.
2 Measuring Feature Effectiveness
The paper derives an analytical measure of feature effectiveness by relating optimal log-loss to statistical representations of the input and labels. H-score can therefore replace gradient-based performance estimation with a data-computable quantity.
- For a fixed transferred feature f, the optimal classifier log-loss can be characterized analytically using information-theoretic and statistical concepts.
- The derivation connects optimal log-loss to approximating the divergence transition matrix with a representation-dependent linear projection.
- The feature-dependent term in the closed-form objective is used to estimate classification performance, and its reformulation is called the H-score.
- H-score is defined for a zero-mean feature using covariance structure and can be interpreted as favoring large inter-class variance with low feature redundancy.
- H-score can be computed analytically by estimating E[f(X)|Y] from samples and has an operational interpretation in the asymptotic error probability of hypothesis testing.
3 Transferability
The paper normalizes H-score into task transferability, allowing source representations to be compared for a target task without fully evaluating each transfer network. It also gives computational procedures and identifies a fine-tuning scope condition.
- Task transferability is the target-task H-score of a source feature divided by the H-score of the target task’s minimum-error feature.
- Transferability lies between 0 and 1, while computing H-score from m samples costs O(mk^2) for feature dimension k < m.
- The optimal target-task H-score can be obtained through HGR maximum correlation, solved using ACE for discrete inputs or a neural approach for generic inputs.
- The exact operational interpretation requires fine-tuning layers to be linear, although experiments assess relative transferability under fine-tuning more broadly.
- Source selection reduces to choosing the source task with the largest transferability, equivalently the largest target-task H-score when the denominator is shared.
4 Experiments
The experiments validate H-score and transferability against empirical transfer performance, then apply the metric to source selection, higher-order transfer, and task-transfer curricula across image tasks.
- Validation of transfer performance: As H-score increases, target log-loss decreases nearly linearly while training and testing accuracy increase for ImageNet-1000 features transferred to Cifar-100.The comparison uses features from ResNet-50 layers 4a–4f.
- Task transfer for 3D scene understanding: For classification and 2D pixel-to-pixel tasks, transferability and affinity identify the same top two source tasks, while rankings differ more for 3D pixel-to-pixel tasks.Both ranking criteria show positive correlations for every target task, especially under DCG.
- Higher-order transfer: For all tasks except Edge3D and Depth, the best second-order source feature combines the top two first-order source tasks.The Depth analysis shows that complementary transferability patterns can improve pixel-wise coverage.
- Task transfer learning curriculum: A minimum spanning tree heuristic uses task-transferability relationships to construct task curricula, with thresholding producing salient transfers for NUS-WIDE classification tasks.The resulting NUS-WIDE curriculum is reported to align with human perception, while disconnected graphs yield independently learnable task groups.
5 Conclusion
The paper introduces H-score to estimate feature-transfer performance across classification tasks and defines task transferability from it. Experiments validate the score on ImageNet-1000 to Cifar-100 transfer and apply it to diverse Taskonomy and NUS-WIDE tasks.
- Conclusion: H-score estimates the performance of features transferred across classification tasks using an information-theoretic approach.The paper uses H-score to define task transferability in multi-task transfer learning.
- Conclusion: The transferability score successfully predicted feature-transfer performance from ImageNet-1000 classification to Cifar-100 classification.The experiments also applied the metric to diverse computer-vision and image-based recognition tasks in Taskonomy and NUS-WIDE.
- Conclusion: The authors identify higher-order transferability, scalable pairwise computation, and improved task curricula as directions for future work.These directions address extending the metric and making it more practical for task-transfer applications.
S1 Derivation of Equation (4)
The derivation introduces probability and feature-matrix notation, centers feature functions without loss of generality, and rewrites covariance terms using matrix representations.
- Notation: The derivation defines random variables, values, alphabets, probability distributions, feature functions, feature vectors, and the feature matrix over the input alphabet.The feature matrix stacks feature vectors for all elements of X.
- Covariance derivation: The derivation expresses the left-hand side of Equation (4) through matrix notation and covariance identities involving f(X) and E[f(X)|Y].The DTM matrix is rewritten in matrix form before substitution into the preceding equation.
- Centering assumption: Feature functions are centered so that E[f(X)] = 0 without loss of generality.This assumption establishes the centered representation used in the covariance derivation.
S2 Operational Meaning of H-Score
The paper gives H-score an operational meaning through hypothesis testing: it characterizes the asymptotic error behavior of using learned features for prediction. This connects analytically computed feature quality to transferability.
- Error-exponent interpretation: H-score estimates the error exponent associated with using a feature function in a binary hypothesis-testing problem.The error exponent measures the rate at which error probability decays as sample size increases.
- Binary hypothesis testing: The hypothesis-testing setup distinguishes two distributions with a likelihood-ratio test using rejection and acceptance regions.The rejection region is defined through divergences from the two hypotheses, while the acceptance region is defined analogously.
- Optimal exponent: Under the local assumption, the optimal error exponent is characterized using vectors derived from the two probability distributions.The paper then compares arbitrary learned decision functions against this optimal exponent.
- Feature-based decisions: A normalized feature induces a linear-projection decision rule whose error exponent is generally reduced relative to the optimal decision function.The reduced exponent is maximized when the feature-related vector matches the difference between the distribution vectors.
- Multidimensional features: The multidimensional extension uses zero-mean, unit-covariance features and establishes a linear relationship between H-score and error exponents.The derivation relies on the k-dimensional feature representation and the covariance normalization condition.
S3 Experiment Details
Experiments evaluate H-score on image classification, Taskonomy transfer, and NUS-WIDE task curricula. The results support agreement with empirical transferability, ranking consistency with affinity scores, computational practicality, and graph-based curriculum construction.
- Cifar-100 transfer: 20,000 Cifar-100 images were split 9:1, and features from five ResNet-50 layers pretrained on ImageNet-1000 were evaluated for transfer.Empirical transfer performance was measured after training the transfer network with stochastic gradient descent.
- Cifar-100 transfer: Across target sample sizes from 5K to 50K, H-score retained its relationship with log-loss, indicating sample-efficient computation.The supplementary experiment reports that target sample size did not affect this relationship.
- Taskonomy transfer: Taskonomy experiments compared H-score rankings with affinity rankings across source-target task pairs, including classification and depth targets.For depth, the top three transferable sources were Depth, Image Reshading, and 3D Occlusion Edges in both methods.
- Computational efficiency: Each pairwise H-score computation finished in less than one hour on a workstation with an eight-core 3.40 GHz CPU and 16 GB memory.The reported computation includes preprocessing.
- NUS-WIDE curriculum: For NUS-WIDE, H-scores were computed from layer-3 activations of source models for unbalanced binary concept-classification tasks.The experiment used a 4-layer fully connected network and 500-dimensional SIFT bag-of-words features.
- NUS-WIDE curriculum: A minimum spanning tree or forest selected task relationships after thresholding transferability weights, producing curriculum transfer directions.The graph is designed to maximize total transferability with n−1 transfers when connected.
- NUS-WIDE curriculum: With α = 0, 33 of 35 NUS-WIDE common-concept tree edges transferred to sky; filtering at the 2.3 percentile exposed more salient relationships.No filtering was used for Taskonomy and animal concepts because most tasks were similarly transferable.
S4 Related Works
The paper distinguishes task transfer learning from domain adaptation and situates its contribution among empirical transferability and task-relatedness studies.
- Transfer-learning categories: Task transfer learning transfers knowledge between different tasks sharing the same input domain, unlike domain adaptation across different input domains.The paper focuses on task transfer learning.