Source-linked AI summary
Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement
Dayou Mao, Yuhao Chen, Yifan Wu, Maximilian Gilles, Alexander Wong
TL;DR
MTL methods lack evidence on smaller backbones and complex datasets, the generalizability of feature-gradient surrogates, and efficient ways to identify training challenges. This paper benchmarks methods, tests those surrogates, and evaluates Feature Disentanglement with Ranking Similarity; findings show both scope-dependent surrogate results and task-dependent faithfulness of the proposed measure.
Problem
Evidence is limited on MTL methods with smaller backbones and complex datasets, the generalizability of feature-gradient surrogates, and efficient analysis of task conflicts and dominance.
Source: §1 · p. 2Method
The paper benchmarks MTL methods with ResNet18, compares parameter- and feature-level gradients, and evaluates Feature Disentanglement using Ranking Similarity against test-time performance.
Source: §1 · p. 2, §5 · p. 5, §5.2 · p. 7Results
GradDrop and CosReg were the best-performing methods across three datasets; only MGDA and Aligned-MTL gained performance consistently with feature-level gradients, while Feature Disentanglement was faithful on selected tasks.
Source: §6 · pp. 8–9Takeaways & Limitations
The authors leave improved measures based on shared representations as a future direction.
Source: §6 · p. 9
Abstract
from arXiv · showhide
One of the main motivations of MTL is to develop neural networks capable of inferring multiple tasks simultaneously. While countless methods have been proposed in the past decade investigating robust model architectures and efficient training algorithms, there is still lack of understanding of these methods when applied on smaller feature extraction backbones, the generalizability of the commonly used fast approximation technique of replacing parameter-level gradients with feature level gradients, and lack of comprehensive understanding of MTL challenges and how one can efficiently and effectively identify the challenges. In this paper, we focus on the aforementioned efficiency aspects of existing MTL methods. We first carry out large-scale experiments of the methods with smaller backbones and on a the MetaGraspNet dataset as a new test ground. We also compare the existing methods with and without using the fast gradient surrogate and empirically study the generalizability of this technique. Lastly, we propose Feature Disentanglement measure as a novel and efficient identifier of the challenges in MTL, and propose Ranking Similarity score as an evaluation metric for different identifiers to prove the faithfulness of our method.
1. Introduction
Multi-task learning shares a backbone across task-specific prediction heads, offering efficiency and potential performance benefits while creating optimization challenges. This paper benchmarks smaller backbones and a new dataset, tests fast gradient surrogates, and proposes a cheaper measure of MTL challenges.
1. Introduction
- A shared backbone feeds task-specific prediction heads, reducing model size and supporting faster inference, lower memory use, and lower power consumption.The paper also notes prior evidence that diverse task supervision can improve performance over single-task systems.Source: §1 · pp. 1–2
- MTL can suffer negative transfer when a shared-backbone update improves one task but worsens another, with task conflicts and dominance identified as causes.Measuring these challenges through shared-parameter gradients requires T full-backbone backpropagations, making the process computationally expensive.Source: §1 · p. 2
- Prior work leaves four gaps: small-backbone evidence, complex large-scale benchmarks, surrogate generalizability, and analysis of challenge measures’ relationship to performance.Task-conflict and task-dominance measures are also computationally costly.Source: §1 · p. 2
- The paper benchmarks ResNet18 on MetaGraspNet, CityScapes, and NYU-v2, compares methods with and without fast gradient surrogates, and evaluates Feature Disentanglement using Ranking Similarity.The proposed measure is intended as an efficient identifier of MTL challenges.Source: §1 · p. 2
2. Problem Definition and Related Work
The section formulates MTL as optimizing task losses over a shared model, then reviews methods that manipulate gradients, balance losses, or regularize gradients. It also motivates feature-level gradient surrogates and feature saliency as an inspiration for measuring disentanglement.
2. Problem Definition and Related Work
- MTL optimizes task losses for a neural network over shared parameters; equal weighting averages the task losses, while Pareto optimization minimizes the loss vector.The section distinguishes gradient manipulation, which changes shared-gradient directions to address conflicts, from gradient balancing, which changes magnitudes to address dominance.Source: §2 · pp. 2–3
2.1. Gradient Manipulation
- Gradient manipulation methods aggregate task gradients on shared parameters using weights, with explicit heuristics such as PCGrad and implicit optimization methods such as MGDA and Aligned-MTL.PCGrad projects gradients when their angle is obtuse; MGDA selects the norm minimizer in the gradients’ convex hull.Source: §2.1 · p. 3
2.2. Gradient Balancing
- Gradient balancing methods weight task losses to learn tasks at compatible paces, using explicit heuristics, optimization subproblems, or an additional optimizer.FAMO uses relative task-loss improvements, unlike CAGrad’s absolute improvements.Source: §2.2 · pp. 3–4
2.3. Gradient Regularization
- Gradient regularization adds terms involving task-gradient relationships to the loss, including squared cosine similarity or angles to the average gradient.Suteu argues for orthogonal task gradients, whereas Javaloy minimizes angles between each task gradient and their average.Source: §2.3 · p. 4
2.4. Fast Gradient Surrogate
- Feature-level gradients require backpropagation only through prediction heads, but prior surrogate results vary, and theoretical reasoning is reported for MGDA and Aligned-MTL.Prior applications reported reasonable results in one case and poor results in another.Source: §2.4 · p. 4
2.5. Saliency Maps in Explainable AI
- Grad-CAM saliency maps for intermediate activations inspired measuring feature disentanglement through task-specific saliency.The section points to the paper’s later method for details.Source: §2.5 · p. 4
3. Benchmarks
The benchmarks study MTL optimization methods with ResNet18 on MetaGraspNet, CityScapes, and NYU-v2, emphasizing a larger, complex robotics-vision dataset. Results include methods that improve over baseline on MetaGraspNet and methods that rank consistently well across CityScapes and NYU-v2.
3. Benchmarks
- The study uses ResNet18 backbones on MetaGraspNet, CityScapes, and NYU-v2, adding MetaGraspNet as a large-scale, complex real-world robotics-vision test ground.The authors also report CityScapes and NYU-v2 for completeness.Source: §3 · p. 4
- The benchmark studies 15 optimization algorithms across gradient manipulation, gradient balancing, and gradient regularization, including feature-gradient variants of MGDA and Aligned-MTL.Experiments were repeated three times and report averages, focusing on early training and relative performance against baseline.Source: §3 · p. 4
3.1. Experiments on MetaGraspNet Dataset
- The MetaGraspNet setup uses paired RGB and depth ResNet backbones, fused stage features, an FPN, and prediction heads for detection, masks, and occlusion.The baseline objective sums seven loss functions; experiments compare methods against that baseline rather than replicating existing results.Source: §3.1 · pp. 4–5
- GradVac, GradDrop, IMTL, DWA, MGDA-UB, replicated CAGrad, and replicated CosReg improved the baseline on all six MetaGraspNet metrics.GradVac, IMTL, and replicated MGDA reached top-three performance on a majority of metrics; GradNorm, Nash-MTL, and feature-level GradDrop were not reported there.Source: §3.1 · p. 5
3.2. Experiments on CityScapes and NYU-v2
- CosReg consistently ranked in the top three across both CityScapes and NYU-v2 on all evaluation metrics; GradDrop did so except for NYU-v2 normal estimation, while still improving baseline.The full results are summarized in Tables 2 and 3, respectively.Source: §3.2 · p. 5
4. Generalizability of Fast Gradient Surrogate
The study compares parameter-level and feature-level gradients across eight optimization algorithms on MetaGraspNet. Only MGDA and Aligned-MTL consistently gained across evaluation metrics; GradVac, RGW, and IMTL degraded.
4. Generalizability of Fast Gradient Surrogate
- Feature-level gradients are commonly used to reduce computation, with chain rule and norm subadditivity offered as justification, but prior work [32] reported performance degradation.Source: §4 · p. 5
- The comparison tested parameter-level versus feature-level gradients on eight optimization algorithms using MetaGraspNet.The full results are summarized in Table 4.Source: §4 · p. 5
- Only MGDA and Aligned-MTL consistently gained across all evaluation metrics, while GradVac, RGW, and IMTL showed significant degradation with the feature-level gradient surrogate.The authors conclude the surrogate is not generalizable across methods.Source: §4 · p. 5
5. Feature Disentanglement Measure
The section introduces Feature Disentanglement as a way to characterize MTL through the shared representation, then defines measures of task-gradient relationships and an evaluation protocol for comparing challenge identifiers. Its results show that Feature Disentanglement aligns with test-time performance on several task and dataset combinations, while its qualitative trends do not always match performance.
5. Feature Disentanglement Measure
- Feature Disentanglement frames MTL challenges through the shared representation, asking which features benefit a set of tasks.The measure is intended to quantify feature disentanglement and monitor its training dynamics.Source: §5 · p. 5
5.1. Preliminaries
- GDS measures agreement between task-gradient directions, while GMS measures alignment between task-gradient magnitudes.Lower GDS indicates less agreement between task supervision; lower GMS indicates less aligned learning pace.Source: §5.1 · pp. 5–6
5.2. Method: Feature Disentanglement Measure
- Feature Disentanglement averages across feature locations the entropy of task-wise saliency derived from feature gradients, with lower scores indicating activations salient to fewer tasks.It requires backpropagation through prediction heads rather than through the backbone for each task.Source: §5.2 · pp. 6–7
5.3. Evaluation Protocol: Ranking Similarity
- Ranking Similarity evaluates how well challenge-measure rankings align with test-time performance rankings, using the absolute distance from 0.5 to make the comparison symmetric.It measures the proportion of item pairs with the same ordering under both rankings.Source: §5.3 · p. 7
5.4. Qualitative Results
- On MetaGraspNet, Uncertainty Weighting and GradVac both had high GDS despite different test performance, while MGDA and FAMO had nearly identical GMS curves despite different performance.Most parameter-gradient methods ended training with lower feature entangledness than baseline, with exceptions; PCGrad, MGDA, and Aligned-MTL showed clear decreases.Source: §5.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
5.5. Quantitative Results
- Feature Disentanglement outperformed GDS and GMS for visible and amodal mask predictions on MetaGraspNet, GDS and GMS on CityScapes semantic and instance segmentation, and GMS on CityScapes depth and NYU-v2 depth and normal estimation.For MetaGraspNet bounding-box predictions, FD had lower ranking similarities than GDS or GMS.Source: Table 5 · p. 7, §5.5 · p. 8, Table 6 · p. 8
6. Conclusions
The paper benchmarks MTL methods with ResNet18 on CityScapes, NYU-v2, and MetaGraspNet, then examines gradient surrogates and proposes an efficient measure of MTL challenges. It reports that GradDrop and CosReg perform best across the three datasets, while the proposed measure is faithful for selected prediction tasks.
6. Conclusions
- GradDrop and CosReg were the best-performing methods across CityScapes, NYU-v2, and MetaGraspNet.The experiments used ResNet18; MetaGraspNet is presented as a new test ground.Source: §6 · pp. 8–9
- Only MGDA and Aligned-MTL, which have theoretical guarantees, gained performance with feature-level rather than parameter-level gradients; other methods were similar or degraded significantly.Source: §6 · p. 9
- Feature Disentanglement was faithful for visible and amodal mask predictions on MetaGraspNet and semantic and instance segmentation on CityScapes.The paper leaves improved measures based on shared representations as future work.Source: §6 · p. 9
7. Experiment Setup for Benchmark Results
The benchmark experiments used a fixed ResNet18-based setup, with shared optimization and image-resizing settings and dataset-specific batch sizes.
7. Experiment Setup for Benchmark Results
- Experiments used a pool of NVIDIA RTX 6000 Ada Generation, RTX A6000, and GeForce RTX 4090 GPUs.Source: §7 · p. 12
- All experiments used ImageNet-pretrained ResNet18, PyTorch SGD at learning rate 1.0 × 10^-4 with momentum 0.9, and linear warmup.Source: §7 · p. 12
- Images were resized to 512 × 512; batch size was 4 on MetaGraspNet and 32 on CityScapes and NYU-v2.Source: §7 · p. 12
8. Experiment Setup for Training Dynamics of MTL Measures
Training-dynamics measures were sampled periodically on MetaGraspNet, yielding roughly 300 trajectory points over training.
8. Experiment Setup for Training Dynamics of MTL Measures
- Measures were computed every 10 training iterations to reduce computation.Source: §8 · p. 12
- MetaGraspNet training ran for around 3k iterations.Source: §8 · p. 12
- Sampling every 10 iterations over roughly 3k iterations yielded around 300 trajectory data points.Source: §8 · p. 12