Source-linked AI summary
Which Tasks Should Be Learned Together in Multi-task Learning?
Trevor Standley, Amir R. Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, Silvio Savarese
TL;DR
Multi-task learning can reduce inference time but often suffers from negative transfer when task objectives compete. The paper studies task cooperation and competition, then assigns cooperating tasks to shared networks and competing tasks to separate ones under a computational budget. Its framework outperforms single-task networks, an all-task multi-task network, and other baselines, while the authors note that candidate-network choices and multiple-network ensembles remain unexplored.
Problem
Multi-task learning can reduce inference time, but competing objectives often cause negative transfer and make independent networks superior.
Method
The paper develops a framework that assigns cooperating tasks to shared networks and competing tasks to separate networks under a limited inference-time budget.
Results
The methods outperform single-task networks, an all-task multi-task network, and other baselines.
Takeaways & Limitations
Selecting task assignments is critical because an added task can improve other outputs even when its own performance is poor.
Takeaways & Limitations
The framework does not explore arbitrarily varied candidate networks, including different architectures, task weights, or training strategies, nor ensembles with multiple networks solving the same task.
Abstract
from arXiv · showhide
Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We study task cooperation and competition in several different learning settings and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.
1. Introduction
Multi-task learning can reduce inference time, but competing task objectives often degrade prediction quality. The paper studies task relationships and introduces a framework for grouping cooperating tasks together while separating competing tasks under a computational budget.
- Multi-task learning jointly solves multiple visual tasks, typically with one network, reducing inference time and potentially improving accuracy, data efficiency, and training time.
- Negative transfer can make multi-task predictions worse than smaller independent networks because tasks learn at different rates, dominate optimization, or produce interfering gradients.
- Task relationships influence whether joint training maintains or improves quality, motivating empirical study of cooperation and competition between tasks.
- The framework assigns competing tasks to separate networks and cooperating tasks to shared networks to maximize performance under a limited inference-time budget.
- Adding an auxiliary task can improve other outputs even when the added task performs poorly, so selecting task groups is critical for accuracy.
2. Prior Work
Prior work addresses parameter sharing, loss weighting, transfer learning, and architecture search, but does not provide the same task-group discovery under an inference-time budget. This paper targets that gap while distinguishing multi-task relationships from transfer relationships.
- Multi-Task Learning: Hard parameter sharing uses mostly shared parameters across tasks, but performance can rapidly degrade as more tasks are added.
- Multi-Task Learning: Loss-weighting methods adjust task influence during training, but comparative studies report no clear winning strategy and similar performance among several approaches.
- Multi-Task Learning: Soft or partial parameter sharing keeps separate or substantially unshared task parameters connected through information sharing or similarity constraints.
- Multi-Task Learning: Unlike prior multi-task approaches, this work discovers task groups and targets inference-time reduction; hybrid methods cannot obtain much speedup from module reuse.
- Transfer Learning: Transfer-learning studies relationships between source and target tasks, whereas this paper studies which tasks should be learned jointly in multi-task networks.
- Neural Architecture Search: The work searches task-grouping space similarly to neural architecture search, where computationally found groupings can outperform human-engineered designs.
3. Experimental Setup
Experiments use the Taskonomy dataset, encoder-decoder networks, standardized computational costs, and four settings that vary network size, dataset size, and task set. The design evaluates task relationships across controlled training configurations.
- Dataset: The Taskonomy data comes from about 600 buildings and contains about 4 million examples, split into training, validation, and test instances.
- Task Sets: The study uses two five-task sets covering semantic, 3D, 2D, reconstruction, and related visual tasks to examine task-category relationships.
- Architectures: Networks use modified Xception encoder-decoder architectures with 256 × 256 inputs, while smaller variants reduce depth and channels to lower computation.
- Settings: Four settings vary network size, dataset size, and task set, with Setting 1 using a smaller encoder, Setting 3 using 200 thousand training instances, and Setting 4 using Task Set 2.
- Comparison: Standard Network Time measures computational cost using multiply-add counts, with encoder channels adjusted to create appropriately sized comparison models.
- Trained Networks: For each setting, the study trains networks for every feasible subset of five tasks, plus half-size and fractional-cost single-task baselines.
4. Study of Task Relationships
The study evaluates how network capacity, training-data size, and task selection affect multi-task relationships. It finds that task affinities vary across settings, so task grouping must be determined for the specific setup.
- Setting 1: 10.69% worse total loss occurs for a 4-task network versus four full-size single-task networks, while the same multi-task network is 4.86% better than equal-budget quarter-size single-task networks.These comparisons show that the accuracy–efficiency outcome depends on how the independent baseline receives the computational budget.
- Setting 1: Multi-task networks with 3, 4, or 5 tasks outperform equal-total-budget single-task networks on average, whereas two-task networks do not.The result comes from shrinking the independent networks so their combined budget matches the multi-task network.
- Setting 1: Surface Normal Prediction and 2D Edge Detection have the highest Setting 1 affinity, while Depth Estimation and Surface Normal Prediction do not rank highly together.This contrasts with Taskonomy transfer learning, where the two 3D tasks have the highest affinity; the multi-task and transfer affinities show Pearson’s r = −0.12, p = 0.74.
- Setting 2: Higher network capacity makes tasks more likely to benefit from joint training, but some task pairs still suffer and Setting 2 affinities correlate weakly with Setting 1 values.The reported correlations are Pearson’s r = 0.08 between the two multi-task settings and Pearson’s r = −0.14 with Taskonomy transfer affinities.
- Setting 3: Using only 199,498 training instances violates the expectation that multi-task learning is generally better in low-data settings, with most tasks suffering when trained together.The authors report that large gains for the Edges task in a few cases prevent multi-task learning from being deleterious on average.
- Settings 3–4: The four non-Auto Encoder tasks in Task Set 2 generally benefit one another, whereas Auto Encoder hurts the other tasks; low-data affinities correlate with capacity-based affinities but not transfer affinities.The low-data correlations are Pearson’s r = +0.375, p = 0.10 with the low-capacity setting and Pearson’s r = +0.558, p = 0.01 with the high-capacity setting, versus Pearson’s r = −0.235, p = 0.51 for transfer affinities.
5. Task Grouping Framework
The framework selects task-grouped networks that cover all tasks while minimizing loss under an inference-time budget. It also introduces approximations that reduce the training burden of evaluating candidate groupings.
- Task Grouping Framework: The framework evaluates networks trained on task subsets, then selects a set covering all tasks with lowest overall loss under the budget.The candidate space includes all nonempty task groupings, with additional half-size single-task networks.
- Task Grouping Framework: The selection problem is generally NP-hard, but branch-and-bound solves the tested instances faster than training all candidates for fewer than ten tasks.The comparison considers 2^|T| − 1 task-subset candidates plus single-task candidates.
- Training-Time Approximations: Early stopping uses validation performance after 20% of the data to select networks before convergence, reducing training time by about 20x but sacrificing guaranteed optimality.Early and final validation losses had Pearson’s r = 0.49 in Setting 1.
- Training-Time Approximations: Higher Order Approximation predicts networks with three or more tasks from fully trained networks with two or fewer tasks, requiring quadratic rather than exponential training counts.The approach still incurs a prediction-quality penalty and saved about 45% of training time in this setup.
6. Task Grouping Evaluation
Across four settings, selecting compatible task groupings outperformed conventional single-network and independent-network baselines under inference-time budgets. The optimal grouping remained strongest even when task cooperation, network capacity, dataset size, or task relationships changed.
- Setting 1: In Setting 1, each proposed method outperformed the traditional baselines at every computational budget.The baselines included five single-task networks and one all-task network.
- Setting 1: For budgets above 1.5 SNT, multiple compatible networks outperformed the traditional all-task network; at 1 SNT, all methods selected that same model.The 1-SNT all-task model still outperformed GradNorm and independent networks.
- Setting 1: At 5 SNT, the optimal method assigned some tasks to multi-task networks and others to individual networks, reflecting task-specific grouping preferences.Semantic Segmentation, Depth Estimation, and Edge Detection were grouped, while Surface Normal Prediction and Keypoint Detection were solved individually.
- Setting 1: Using 2.5 SNT, the optimal method and using 3.5 SNT, HOA both outperformed individual networks using 5 SNT total.This demonstrates a time-accuracy advantage for selected or approximated groupings in Setting 1.
- Setting 4: Even in the highly cooperative fourth setting, the optimal solution and HOA outperformed every baseline.The difference between the 1-SNT all-task and 5-SNT optimal solutions was small in this setting.
- Cross-Setting Discussion: The optimal grouping outperformed every baseline in all four settings, while HOA performed very similarly and often equally well.ESA was usually competitive but generally worse than optimal, and its results could depend on training-data exposure.
7. Conclusion
The paper presents task compatibility as a setting-dependent problem and provides a framework for deciding which tasks to train jointly or separately. Two strategies outperform single-task networks, an all-task multi-task network, and other baselines, while task affinities vary with dataset size and network capacity.
- The framework determines which tasks should be trained jointly and which should be trained separately for a given setting.
- Two strategies outperform single-task networks, an all-task multi-task network, and other baselines.
- Multi-task affinities depend strongly on factors including dataset size and network capacity, unlike transfer task affinities.