Source-linked AI summary
Transferability and Hardness of Supervised Classification Tasks
Anh T. Tran, Cuong V. Nguyen, Tal Hassner
TL;DR
The paper asks how to estimate supervised classification task difficulty and transferability without assuming trained models. It uses information from training-label statistics, especially conditional entropy, and finds reliable agreement with empirical measures across 437 tasks, including strong results for selected face-attribute transfers.
Problem
Estimating which supervised classification tasks are hardest and which transfer well traditionally relies on trained-model solutions, limiting solution-independent task analysis.
Method
The approach treats task labels as random variables and uses conditional entropy between source and target training-label sequences to estimate transferability and hardness.
Results
The estimates reliably predict transferability and hardness across 437 tasks, while selected transfers from face recognition to facial attributes outperform the state of the art.
Takeaways & Limitations
Training-label statistics can provide effective, cheaper estimates of task relationships and hardness without training deep models.
Takeaways & Limitations
The method assumes tasks share the same input domain and use discrete labels, with extension to different domains and non-discrete tasks left for future work.
Abstract
from arXiv · showhide
We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as random variables and exploring their statistics. When transferring from a source to a target task, we consider the conditional entropy between two such variables (i.e., label assignments of the two tasks). We show analytically and empirically that this value is related to the loss of the transferred model. We further show how to use this value to estimate task hardness. We test our claims extensively on three large scale data sets -- CelebA (40 tasks), Animals with Attributes 2 (85 tasks), and Caltech-UCSD Birds 200 (312 tasks) -- together representing 437 classification tasks. We provide results showing that our hardness and transferability estimates are strongly correlated with empirical hardness and transferability. As a case study, we transfer a learned face recognition model to CelebA attribute classification tasks, showing state of the art accuracy for tasks estimated to be highly transferable.
1. Introduction
The paper addresses how to estimate task transferability and hardness without relying on trained models. It proposes analyzing labeled data directly, using conditional entropy to estimate both properties, and validates the approach across 437 tasks.
- Transferability and hardness estimates matter for planning model reuse and understanding relationships among supervised tasks.
- Existing task-relationship methods use trained models or other solution proxies, which can reflect chosen solutions rather than the underlying tasks and may be computationally expensive.
- The proposed solution-agnostic approach examines labeled data directly, regardless of which models were trained or whether models exist.
- Conditional entropy between two tasks’ training-label sequences bounds transferability under optimal loss and supports a-priori estimates of transferability and hardness.
- The approach is tested on 437 tasks from CelebA, Animals with Attributes 2, and Caltech-UCSD Birds 200, reliably predicting task transferability and hardness.
- In a face-recognition transfer case study, highly transferable facial-attribute tasks outperform the state of the art using a linear support vector machine per attribute.
2. Related work
Related work estimates task relationships through model-based representations, qualitative data analysis, or multi-task training. This paper instead uses training-label information without trained models.
- The work relates to transfer learning, meta learning, domain shifting, and multi-task learning, while focusing on task transferability and hardness.
- Transfer learning: Transfer-learning theory provides generalization bounds, but these bounds are abstract and difficult to compute in practice.
- Transfer learning: The proposed transferability measure is computed from training sets and may also support source-task selection and continual learning.
- Task spaces: Qualitative exploration of labeled data can reveal relationships and biases between tasks.
- Task spaces: Taskonomy and Task2Vec represent tasks using trained per-task or probe models, whereas this paper considers only labels provided in training data.
- Multi-task learning: Multi-task learning methods share representations, branches, activations, or input regions, and may underperform separate models when tasks are weakly related.
3. Transferability via conditional entropy
The paper estimates task transferability and hardness directly from label statistics, using conditional entropy between source and target task labels rather than trained models. It establishes a bound connecting conditional entropy to transferred-task log-likelihood and illustrates how the measure ranks toy transfers.
- Conditional-entropy measure: Transferability is estimated from the conditional entropy H(Y |Z) between target and source label sequences, while task hardness is estimated by transferability from a trivial task.The approach treats labels as random variables and measures the information needed to estimate a target label given a source label.
- Task and transfer setup: The framework assumes shared input instances with different labels for source task T Z = (X, Z) and target task T Y = (X, Y), and transfers a fixed source representation by retraining only a target classifier.The target classifier is optimized using the target labels and need not share the source classifier’s architecture.
- Theoretical relationship: Theorem 1 relates target-task log-likelihood to source-task log-likelihood minus H(Y |Z) when the optimal source representation is transferred to the target task.The proof constructs a target prediction distribution by averaging empirical conditional label probabilities with respect to the source model’s output distribution.
- Theoretical relationship: With a fixed source task, transferability depends only on H(Y |Z), so conditional entropy can estimate transferability across multiple target tasks.The source log-likelihood is constant when the source task is fixed; lower source log-likelihood also reduces transferability, while lower conditional entropy increases it.
- Intuition and toy examples: In the toy examples, transfer is easiest for the one-to-one mapping with H(Y |Z) = 0 and hardest for trivial-source cases with H(Y |Z) = 4 log 2.The intermediate examples have H(Y |Z) = log 2, including a multilabel target handled naturally by conditional entropy.
4. Task hardness
The paper estimates task hardness through conditional entropy from a trivial task, linking this quantity to optimal task loss and transferability. This label-based estimate avoids trained deep models but remains an input-agnostic approximation.
- Task hardness is formally measured using the optimal log-likelihood, or equivalently the minimum loss, over a chosen model class.The model class can determine factors such as representation size or network architecture.
- A trivial task assigns every input the same constant label, enabling hardness bounds through transfer from that task.The resulting label sequence is denoted C.
- Hardness can be approximated by the conditional entropy H(Z|C), which is also used to estimate transferability from the trivial task to T Z.Harder tasks are expected to be more difficult to transfer to from a trivial task.
- Unlike Task2Vec, this hardness estimate requires only computing H(Z|C), rather than training deep models.The comparison concerns the procedure used to compute task hardness.
- H(Z|C) ignores the input and therefore provides only an approximation, although experiments show label statistics are a strong cue for task hardness.The authors note that scenarios can be constructed where the measure does not accurately reflect hardness.
5. Experiments
Experiments across 437 tasks test whether conditional-entropy estimates predict transferability and hardness, including a face-recognition-to-attribute case study. The estimates correlate significantly with observed transfer outcomes and errors, while highly transferable attributes approach dedicated-model performance and require less training data.
- Benchmarks: The evaluation covers 437 classification tasks across CelebA, AwA2, and CUB, using separate test images to assess performance.CelebA includes 40 attributes, AwA2 includes 85 attributes, and CUB includes 312 attributes.
- Evaluating task transferability: CE estimates are compared with actual transferability by measuring target-task test errors for each source–target task pair.With a fixed source task, the analysis compares CE against test error, which is negatively related to transferability.
- Evaluating task transferability: CE and target test error are highly positively correlated and statistically significant, supporting CE as a predictor of actual transferability without models or input-domain information.The figure examples report statistically significant correlations across CelebA, AwA2, and CUB.
- Identity to facial attributes: Subject-specific attributes such as male and bald transfer better from face recognition than expression-related attributes such as smiling and mouth open.The paper reports that these relationships emerge from its transferability estimates rather than being imposed through domain knowledge.
- Identity to facial attributes: Transferred recognition representations perform comparably to dedicated attribute networks on highly transferable attributes but decline for less transferable attributes.The transferred lSVM results are 2.4% below the best published scores overall, while some attributes such as young and big nose favor transfer over dedicated models.
- Identity to facial attributes: For male and double chin, transfer achieves higher accuracy with much less training data than dedicated networks trained from scratch.These attributes were selected because they were estimated to be highly transferable from recognition.
- Task hardness: Estimated hardness strongly correlates with dedicated-network classification errors across the three benchmarks, with statistical significance p < 0.001.Higher estimated hardness corresponds to higher test error, although input-domain effects are not excluded.
6. Conclusions
The paper estimates task hardness and transferability directly from training-label statistics, without relying on trained task models. Conditional entropy provides the central measure for producing these estimates.
- The method estimates supervised classification task hardness and transferability by examining training-label statistics, particularly conditional entropy between task label sequences.
- The approach avoids trained task solutions and computes estimates more simply than methods requiring trained models or training-process inspection.
- Conditional entropy computation is cheaper than training deep models for the same transferability-estimation purpose.
A. Proof of theorem 1
The theorem proof rewrites the transferred-task objective using empirical label distributions and conditional entropy. These substitutions yield the stated lower bound on transfer performance.
- The proof begins from the definition of f Trf(T Z →T Y ) and uses a single summand as an intermediate step.
- The transfer objective contains the term log P(zi|xi; wZ, hZ).
- The derivation evaluates the second and first terms in Eq. (15) separately before combining them.
- The proof groups terms from the transferred-task objective by label values and introduces empirical joint and conditional distributions.
- The theorem follows by combining Eqs. (15), (16), and (17) to obtain f Trf(T Z →T Y ) ≥ lZ(wZ, hZ) −H(Y |Z).
B. More details on task hardness
The hardness analysis assumes trained models do not overfit, linking training loss to test error when train and test data share a distribution. It also discusses trivial source tasks under an optimal representation.
- The hardness definition assumes non-overfitting of trained models.
- When training and testing sets share a distribution, training loss typically correlates with test error for appropriately trained models.
- For a trivial source task, any representation can fit its constant label perfectly with zero training loss.
- With an optimal source representation, the analysis connects the trivial-task result to H(Z|C).
C. Technical implementation details
Conditional entropy is computed from paired training labels using empirical counts and two label passes. The resulting estimate is computationally lightweight compared with deep-model transfer evaluation.
- The implementation computes the empirical joint distribution P̂(y, z) by counting paired training labels from tasks T Z and T Y.
- A second pass through the training labels computes conditional entropy using Eq. (17).
- The complete conditional-entropy procedure requires only two loops through the training labels.
- Less than a second was required to estimate face-recognition transferability to a binary facial attribute on a standard CPU.
- Taskonomy reported over 47 thousand hours of GPU runtime to establish relationships among 26 tasks.
- Dedicated attribute models use standard ResNet-18 V2, with training converging in fewer than 40 epochs across all 437 tasks.
- For transfer learning, a trained source CNN produces embeddings, and a standard linear SVM is trained on target-task labels.
- The trained linear SVM predicts target-task labels from embeddings extracted from unseen testing data.
D. Additional results: Generalization to multi-class
The results show that transferability extends to multi-class settings, including face recognition with over 10k labels, while harder tasks generally yield lower accuracy.
- Transferability generalizes to multi-class tasks, including face recognition with 10k labels transferred to attribute classification.
- Table 2 reports multi-class hardness examples on CelebA data.
E. Full transferability results
The full transferability results cover attribute prediction on CelebA and AwA2, plus transfer from face identity recognition to CelebA attributes.
- CelebA attribute-prediction transferability results are presented in Figure 7.
- CelebA transferability from identity to attributes is reported in Table 3.
- AwA2 attribute-prediction transferability results are presented in Figures 8, 9, and 10.
F. Full hardness results
The full hardness results cover CelebA, AwA2, and CUB attribute tasks, with hardness scores compared against empirical task errors and, for CelebA, transferability results.
- CelebA attribute hardness is reported in Table 4, with hardness scores compared against empirical test errors.
- AwA2 attribute hardness is reported in Table 5, with hardness scores compared against empirical test errors.
- CUB attribute hardness is reported in Table 6, with abbreviated attribute names defined in Table 7.
- CelebA transferability results compare face recognition with facial attributes across all 40 attributes.
- Subject-specific attributes such as male and bald are more transferable than expression-related attributes such as smiling and mouth open.
- CelebA male classification is relatively hard but is the easiest task to transfer from face recognition.