PaperlayerView exact revision on arXiv

Source-linked AI summary

Task2Vec: Task Embedding for Meta-Learning

Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, Pietro Perona

arXiv:1902.03545v1cs.LGcs.AIstat.ML

TL;DR

The paper addresses the lack of a general way to represent and compare visual learning tasks. It introduces TASK2VEC, which embeds tasks using Fisher information from a fixed probe network, and uses task/model embeddings to select feature extractors. The embeddings track semantic and taxonomic relations, while selection approaches the ground-truth optimal expert and improves over a generic ImageNet model.

  • Problem

    No general framework exists to describe and learn relations between tasks, although such relations matter for transferring models to related tasks.

  • Method

    TASK2VEC feeds labeled data through a fixed pre-trained probe network and computes a diagonal Fisher Information Matrix over its filter parameters to obtain a fixed-dimensional task embedding.

  • Results

    The embedding distance correlates with semantic and taxonomic task relations, and task/model embeddings select feature extractors close to ground-truth optimal selection while improving over a generic ImageNet-trained model.

  • Takeaways & Limitations

    Task embeddings support reasoning over task spaces and practical selection of expert feature extractors, especially when little training data is available.

  • Takeaways & Limitations

    The evaluated task collection does not capture the full complexity and variety of real-world tasks, motivating tests on larger and more diverse collections.

Abstract

from arXiv · show

We introduce a method to provide vectorial representations of visual classification tasks which can be used to reason about the nature of those tasks and their relations. Given a dataset with ground-truth labels and a loss function defined over those labels, we process images through a "probe network" and compute an embedding based on estimates of the Fisher information matrix associated with the probe network parameters. This provides a fixed-dimensional embedding of the task that is independent of details such as the number of classes and does not require any understanding of the class label semantics. We demonstrate that this embedding is capable of predicting task similarities that match our intuition about semantic and taxonomic relations between different visual tasks (e.g., tasks based on classifying different types of plants are similar) We also demonstrate the practical value of this framework for the meta-task of selecting a pre-trained feature extractor for a new task. We present a simple meta-learning framework for learning a metric on embeddings that is capable of predicting which feature extractors will perform well. Selecting a feature extractor with task embedding obtains a performance close to the best available feature extractor, while costing substantially less than exhaustively training and evaluating on all available feature extractors.

UCLA and AWS

The paper lists Subhransu Maji as affiliated with UMass and AWS.

  • Subhransu Maji is affiliated with UMass and AWS.

1. Introduction

The paper introduces TASK2VEC, a fixed-dimensional task embedding based on Fisher information, to measure relations among visual classification tasks and support feature-extractor selection. Its embeddings reflect semantic and taxonomic similarity, and the resulting selection approaches optimal expert choice while improving over fine-tuning a generic ImageNet model.

  • TASK2VEC represents visual classification tasks as elements of a vector space based on the Fisher Information Matrix.The method processes labeled data through a fixed pre-trained probe network and computes a diagonal FIM over its filter parameters.
  • Embedding distances correlate with semantic similarities and natural task metrics, including taxonomic distance for species classification.The paper reports strong agreement between symmetric TASK2VEC distance and taxonomic distance, while the introduction also describes an asymmetric distance related to transferability.
  • The fixed-dimensional embedding is independent of the output space and invariant to permutations of task labels.Its size equals the number of probe-network filters, regardless of the number of classes.
  • TASK2VEC and MODEL2VEC support selecting pre-trained feature extractors for new tasks when training data are limited.MODEL2VEC incorporates task and model embeddings to account for model interactions that TASK2VEC alone ignores.
  • Feature-extractor selection obtains performance close to ground-truth optimal selection and improves relative to fine-tuning a generic ImageNet-trained model.The approach is presented as less costly than exhaustively training and evaluating all available feature extractors.

2. Task Embeddings via Fisher Information

TASK2VEC represents a task using Fisher information computed from a fixed probe network, producing a fixed-dimensional embedding that captures task difficulty, domain information, and useful features. Its properties support comparisons across label spaces and relate embedding geometry to task complexity and semantic or taxonomic similarity.

  • Probe network and Fisher information: A probe network processes labeled data, and its Fisher information quantifies how informative network parameters are for the task.The network is viewed as a feature extractor and classifier; parameter importance is related to the output change caused by perturbations.
  • TASK2VEC construction: TASK2VEC averages diagonal Fisher information within each filter, yielding an embedding whose dimension equals the number of probe-network filters.This approximation avoids the unmanageable size of the full Fisher matrix and assumes within-filter weights can share one value.
  • Robust Fisher computation: A robust variational estimator reduces sensitivity to irregular loss landscapes and avoids degenerate embeddings when data are scarce.In the low-data regime, the estimator is biased toward the isotropic prior; trivial tasks can therefore produce the prior embedding.
  • Properties of the embedding: The embedding is invariant to label permutations and has fixed dimension across classification problems with different numbers of classes.It depends on the trained predictive distribution rather than directly on the task-label identities.
  • Properties of the embedding: The embedding norm scales with task difficulty for a given feature extractor, and its norm correlates with test performance on real data.Highly confident predictions contribute less to the Fisher information, linking the norm to the difficulty of fitting the task.
  • Properties of the embedding: TASK2VEC emphasizes domain information near the decision boundary rather than weighting every input point equally.The contribution is largest when p(1 −p) is maximized at p = 1/2, unlike a covariance-based domain embedding.
  • Properties of the embedding: Fisher information identifies features useful for predicting labels, whereas activation-based domain embeddings only indicate which features vary.In the two-layer example, features uncorrelated with y have zero corresponding Fisher blocks.

3. Similarity Measures on the Space of Tasks

The paper compares symmetric semantic-similarity measures with an asymmetric transfer-oriented score for tasks. TASK2VEC cosine distance correlates with taxonomic distance, while Euclidean distance has scale and complexity-related drawbacks.

  • Taxonomic distance: Taxonomic distance compares classification tasks through category sets organized in a hierarchy.For example, dog-breed classification is closer to cat classification than to plant-species classification.
  • Transfer distance: Transfer distance measures the performance difference between training on a target task from a fixed initialization and fine-tuning from a source-task solution.It is not necessarily symmetric or positive.
  • Symmetric TASK2VEC distance: Euclidean distance is problematic because network parameters have different scales and embedding norms vary with task complexity and sample count.
  • Symmetric TASK2VEC distance: Symmetric TASK2VEC distance uses cosine distance between element-wise normalized Fisher embeddings from the same probe network.It is intended to capture semantic similarity between tasks.
  • Symmetric TASK2VEC distance: TASK2VEC distance correlates well with taxonomical distance between species on iNaturalist.
  • Asymmetric TASK2VEC distance: The symmetric distance is ill-suited to model selection because transfer distance is intrinsically asymmetric.An asymmetric score accounts for task similarity and the complexity of the first task; with an ImageNet-pretrained ResNet-34 probe, experiments found α = 0.15 robust across meta-tasks.

4. MODEL2VEC: task/model co-embedding

TASK2VEC represents tasks alone, but model performance can also depend on architecture and training algorithm. MODEL2VEC jointly embeds tasks and models to support model selection for novel query tasks.

  • Motivation: TASK2VEC ignores model details, although models trained on the same task can have different performance characteristics.
  • Motivation: MODEL2VEC learns a joint embedding to represent interactions between task and model, including architecture and training algorithm.
  • MODEL2VEC construction: Each model embedding is mi = Fi + bi, combining its training-task embedding with a learned model bias.The bias accounts for particularities of the model; unavailable task embeddings are set to zero.
  • Learning objective: MODEL2VEC learns model biases by optimizing a k-way cross-entropy objective for model selection.
  • Model selection: For a novel query task, the predicted model is the one embedded closest to that task under the asymmetric distance.

5. Experiments

Experiments across 1,460 visual tasks evaluate TASK2VEC’s task representations and model-selection capability. The embedding reflects semantic and taxonomic structure, task complexity, and supports selecting strong feature extractors across data sizes and architectures.

  • Experimental setup: The benchmark contains 1,460 tasks spanning iNaturalist, CUB, iMaterialist, and DeepFashion, with mostly hundreds or thousands of training samples.The collection includes 207 iNaturalist, 25 CUB, 228 iMaterialist, and 1,000 DeepFashion tasks, simulating real-world heavy-tailed data.
  • Task embedding results: TASK2VEC distances strongly agree with taxonomic distances between iNaturalist classification tasks.Taxonomic distance between biological orders provides a natural semantic similarity measure for these tasks.
  • Task embedding results: TASK2VEC produces non-trivial semantic groupings that can differ from dataset correlations, such as clustering jeans with ripped and denim-material tasks.This demonstrates that the embedding can capture label-level task relations rather than only shared image domains.
  • Task embedding results: The embedding norm correlates with task complexity on real tasks and architectures, while higher-complexity tasks tend to benefit more from specialized experts.Figure 3 orders tasks by embedding norm and compares that norm with expert performance.
  • Model selection: Asymmetric TASK2VEC selection is close to ground-truth optimal and significantly improves over chance and a generic ImageNet expert, with O(1) rather than O(N) selection complexity.The method selects among experts without exhaustively evaluating every available extractor.
  • Model selection: At all tested dataset sizes, TASK2VEC improves over generic ImageNet selection for both fixed-feature training and fine-tuning.In the low-data regime, fine-tuning ImageNet is more expensive and performs worse than selecting a good fixed extractor.
  • Probe network: DenseNet and ResNet probe networks perform significantly better than VGG probe networks for TASK2VEC embeddings.Probe-network choice therefore affects model-selection performance.

6. Related Work

Related work covers task and domain embeddings, Fisher-kernel representations, Fisher information for neural networks, and meta-learning for model selection. TASK2VEC differs by targeting a larger task library and representing task structure through Fisher information.

  • Task and domain embedding: Domain embeddings use input statistics, whereas TASK2VEC represents task-specific label structure through the task distribution p(x, y).Domain embeddings can collapse distinct iMaterialist attribute tasks because they share the same clothing-image domain.
  • Task and domain embedding: Taskonomy computes pairwise transfer distances for 26 curated visual tasks, while this work focuses on a larger library of 1,460 tasks.The comparison highlights different scales and objectives for organizing task spaces.
  • Fisher kernels: Fisher kernels represent similarity between data items using gradients of a generative-model score function and the Fisher information matrix.The paper draws inspiration from this representation framework.
  • Fisher information for CNNs: Fisher information has been used for natural-gradient optimization, neural-network approximations, and regularization schemes.These precedents motivate using the Fisher information matrix to characterize neural-network tasks.
  • Meta-learning and model selection: Prior meta-learning approaches for classifier-library selection commonly use landmarking or active testing, evaluating some models to estimate others.TASK2VEC instead selects feature extractors from task representations without exhaustive model evaluation.

7. Discussion

TASK2VEC represents tasks as fixed-dimensional vectors whose norms and distances capture task difficulty and relations. The discussion emphasizes efficient expert selection while noting that current experiments do not cover the full diversity of real-world tasks.

  • TASK2VEC represents a task or dataset as a fixed-dimensional vector.
  • Embedding norms correlate with test error, while cosine distances correlate with taxonomic and fine-tuning distances.
  • Using TASK2VEC to select an expert feature extractor can improve test performance with only a small training overhead, especially when data are limited.
  • The current task collection does not capture the full complexity and variety of real-world tasks, motivating evaluation on larger and more diverse collections.
  • The embedding is computed from Fisher information of probe-network parameters after fitting a task-specific classifier.
  • The toy experiments show that task embeddings group qualitatively similar tasks and separate complex decision boundaries from simpler ones.
  • A robust Fisher estimate minimizes a loss over a neighborhood of the fitted parameters and constrains precision matrices to reduce parameters.

C.1. Training of experts and classifiers

The experiments define how experts, classifiers, and TASK2VEC embeddings are trained. Model selection learns model-specific offsets and a distance scale from classifier performance labels.

  • Experts are fine-tuned from ImageNet-pretrained ResNet-34 models, while classifiers train only a replacement final layer on each expert.
  • TASK2VEC embeddings are computed by retraining a probe network’s final classifier and then calculating the probe weights’ Fisher information matrix.
  • MODEL2VEC represents each model as its TASK2VEC embedding plus a learned model-specific parameter vector.
  • Model selection uses soft labels derived from ground-truth expert errors, preserving information about models with similarly good performance.
  • Predictions use an asymmetric task-to-model distance with a learned positive scale, optimized jointly with model embeddings.
  • Embedding evaluation reports leave-one-out error by training on all other tasks and testing on the held-out task.

D. Datasets, tasks and meta-tasks

The evaluation uses two curated model-selection meta-tasks spanning biological, product, and fashion datasets. Their task and expert collections are designed to balance domains, taxonomic structure, sample availability, and model strength.

  • The iNat+CUB meta-task contains 50 tasks and experts split evenly between iNaturalist and CUB.
  • iNaturalist tasks are formed by grouping species into orders and selecting the 25 orders with the most samples.
  • CUB contributes 10 order tasks and 15 Passeriformes family tasks after subdividing the dominant Passeriformes order.
  • The iNat+CUB evaluation uses leave-one-out selection so a task cannot select an expert trained on the same data.
  • The Mixed meta-task includes 40 random tasks and 25 curated experts drawn from iNaturalist, iMaterialist, and DeepFashion.
  • The data-efficiency experiment selects representative tasks and experts across datasets, including both strong and weak experts.

E. Error matrices

Figure 6 presents ground-truth error matrices for the CUB+iNat and Mixed meta-tasks. It overlays asymmetric TASK2VEC distances and marks the model-selected and optimal experts for comparison.

  • Rows represent target tasks and columns represent experts, with each cell showing the test error from training a classifier on that task–expert combination.
  • Background colors encode asymmetric TASK2VEC distance between each target task and the task used to train an expert.
  • Red numbers mark the expert selected by the embedding-based model-selection algorithm, while blue highlights mark a different out-of-diagonal optimal expert.
Loading 1902.03545v1…