Source-linked AI summary
Taskonomy: Disentangling Task Transfer Learning
Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, Silvio Savarese
TL;DR
Visual tasks may have structured transfer relationships, but their organization and usefulness are not fully known. The paper computationally maps these relations through learned representations and transfer policies, finding strong structure and enabling substantially reduced supervision while identifying model- and sampling-dependent limits.
Problem
The paper asks how visual tasks are related and how exploiting those relationships can reduce the supervision required to solve many tasks.
Method
The paper computes transfer affinities from task representations, samples first- and higher-order transfers, and extracts a global taxonomy with Binary Integer Programming.
Results
The taxonomy outperformed random connectivities by a large margin, and the resulting structure supported solving tasks with far less data than independent learning.
Takeaways & Limitations
Taskonomy provides a computational map and solver for selecting transfer policies and designing data-efficient curricula across visual tasks.
Takeaways & Limitations
The findings are model- and data-specific, and guarantees for a generally sampled task space require more formal study.
Abstract
from arXiv · showhide
Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.
1. Introduction
The paper models relationships among visual tasks computationally to expose useful transfer structure, reduce supervision demands, and support efficient multi-task learning. It uses learned representations and transfer functions to construct a task taxonomy without relying on potentially incorrect human priors.
- Motivation: Visual tasks have both clear and obscure relationships, such as depth–surface-normal and keypoint–shading connections.The paper motivates discovering which tasks provide useful information for others rather than assuming relationships from intuition alone.
- Motivation: Training tasks independently ignores useful relationships and creates a massive labeled-data requirement.Fully supervised learning commonly solves tasks in isolation, making comprehensive perception systems costly to train.
- Motivation: A relationship-aware model can demand less supervision, use less computation, and behave more predictably.The paper presents task structure as a step toward efficient systems that solve many tasks without becoming intractable in supervision or computation.
- Approach: The method computes task affinities from how easily one task can be read out of another task’s learned representation.It exhaustively samples transfers and uses Binary Integer Programming to extract a globally efficient transfer policy.
- Approach: The computational approach avoids imposing prior assumptions and can uncover transfer directions that oppose intuition.For example, transfer from surface normals to depth-related solutions can be preferable to the reverse direction for the learned networks.
- Tools: The resulting taskonomy tools include an interactive solver for proposing data-efficient curricula and a public dataset, demo, and code.Users can generate taxonomies for task sets and use the solver to devise supervision policies for their applications.
2. Related Work
Related work studies task relationships through transfer, self-supervision, multi-task learning, meta-learning, and learning-theoretic approaches. Taskonomy differs by computationally modeling a broad output-task space and extracting a meta-structure rather than manually specifying local relations or only producing multiple outputs.
- Context: Prior work asserts task structure through developmental, learning, transfer, and related theoretical perspectives, while this paper attempts to computationally find it.The paper connects taskonomy to a broad literature but distinguishes its empirical computational mapping from approaches seeking guarantees.
- Self-supervised learning: Self-supervised methods use manually selected surrogate tasks, whereas taskonomy computationally models a larger task space and discovers obscure relationships.The contrast is between entering a local part of the task structure by hand and estimating relations across many tasks.
- Unsupervised learning: Unsupervised learning exploits redundancies in the input domain, while taskonomy models functional redundancies among output tasks.Taskonomy is task-dependent rather than agnostic to downstream tasks.
- Meta-learning: Taskonomy resembles meta-learning as a computational meta-structure of the task space.Its focus is the organization of task relations rather than learning at a conventional optimization or architectural level.
- Multi-task learning: Unlike multi-task learning, taskonomy explicitly models relations among tasks and extracts a meta-structure.The paper’s larger task set also creates challenges for developing one multi-task network for every output.
- Domain adaptation: Unlike domain adaptation, taskonomy adapts across output-task space and among many elements rather than primarily addressing input-domain shifts.Domain adaptation commonly keeps the task fixed while changing the input domain.
- Computational pipeline: The taxonomy is created through four stages: task-specific networks, transfer functions, normalized affinities, and global optimization.The stages use latent-space transfer modeling, AHP normalization, and BIP-based taxonomy extraction.
3. Method
The method models visual-task transfer computationally by measuring source-to-target transferability and organizing feasible transfers into a task taxonomy. It combines task-specific representations, low-capacity readouts, higher-order transfers, ordinal normalization, and constrained optimization under a supervision budget.
- Problem Formulation: The transfer problem maximizes collective target performance subject to a budget limiting the number of tasks trained from scratch.Source-only tasks may be trained when they improve performance on the target set.
- Taxonomy Construction: Taskonomy represents task transferability as a directed hypergraph whose weighted edges encode feasible transfers from source-task groups to target tasks.The resulting graph is parameterized by supervision budget, selected tasks, transfer order, and transfer-function expressiveness.
- Task Dictionary: The task dictionary samples 26 visual tasks spanning 2D, 2.5D, 3D, and semantic themes.The dictionary is intended as a tractable sample of a denser task space rather than an exhaustive list.
- Transfer Modeling: Higher-order transfers combine multiple source representations to exploit complementary information, while beam search limits the combinatorial number of candidate transfers.For orders k ≤ 5, the method selects combinations among each target’s five best first-order sources; for k ≥ 5, it uses the top k sources.
- Global Taxonomy: The method normalizes transfer performances with an ordinal pairwise comparison procedure and solves a binary integer program to select a globally optimal valid subgraph.The optimization enforces source inclusion, exactly one incoming transfer per target, and the supervision-budget constraint.
4. Experiments
Experiments evaluate taxonomy-based transfer under varying supervision budgets, transfer orders, and target settings. The policies achieve useful performance with less supervision, generalize to novel tasks, and outperform fixed transfer baselines while often approaching fully supervised performance.
- Experimental setup: Approximately 3,000 transfer functions were trained for the taxonomy, requiring 47,886 cloud GPU hours.
- Metrics: Gain measures win rate against scratch training with the same data, while Quality measures win rate against a fully supervised network trained with 120k images.
- Evaluation of computed taxonomies: Good results are achieved with supervision budgets notably smaller than the number of solved tasks, and performance improves as the budget increases.Results are shown for maximum transfer orders 1 and 4.
- Generalization to novel tasks: The all-for-one evaluation places one novel target in T and uses 16k target examples to train transfer networks to source tasks.This setup localizes where the target would lie in the taxonomy.
- Generalization to novel tasks: For novel tasks, target-specific taxonomy policies outperform self-supervised methods, ImageNet features, and scratch training by a large margin.The policies generally lose to the fully supervised gold standard but often achieve win rates in the 40% range.
5. Significance Test of the Structure
The significance tests assess whether the discovered task structure is meaningful, stable across design choices, and transferable across datasets. Taxonomy-based connectivity outperforms random alternatives, while the resulting similarity organization aligns with intuitive task groupings.
- Dataset dependence: Task rankings transfer to ImageNet object classification and MIT Places scene classification, where performance is evaluated on the respective external test sets.The ranking is assessed by ordering bars according to taxonomy-predicted performance from the original dataset.
- Stability: Large changes in architectures, transfer-network data, datasets, data splits, and task dictionaries produce stability tests of the discovered structure.The reported perturbations include 16x changes in transfer-function training data and 4x changes in network size and architecture.
- Task similarity tree: Agglomerative clustering of transferring-out behavior groups 3D, 2D, low-dimensional geometric, and semantic tasks into clusters matching intuitive expectations.BIP transfer taxonomies remain consistent with this similarity tree under budget and target constraints.
6. Limitations and Discussion
The framework reduces supervision needs while exposing limitations tied to model and data dependence, task compositionality, sampled-space regularity, and evolving task sets.
- The authors report that modeling the space of visual tasks can reduce the need for supervision.
- Model Dependence: The findings are in principle specific to the adopted computational model and data, despite validation across architectures and datasets.
- Compositionality: The framework models common human-defined visual tasks rather than computationally discovered latent subtasks.
- Space Regularity: Generalization from the sampled task dictionary depends on proper sampling and the regularity of the dense task space.
- Future Directions: The framework raises the possibility of using visual-task structure in perception modules for downstream tasks such as robotic manipulation.
- Lifelong Learning: The modeling was performed in one go, whereas lifelong learning requires mechanisms to expand the structure as new tasks are mastered.