Source-linked AI summary
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
Ximeng Sun, Rameswar Panda, Rogerio Feris, Kate Saenko
TL;DR
AdaShare addresses the challenge of choosing what to share across tasks while keeping multi-task learning resource-efficient. It learns task-specific layer-execution policies jointly with network weights and reports strong performance across multiple tasks, including five computer vision tasks with 80% fewer parameters at the same performance as the standard approach.
Problem
Multi-task learning lacks an efficient way to decide which parameters to share across tasks, while task-specific networks grow in size with the number of tasks.
Method
AdaShare learns task-specific policies that selectively execute layers and optimizes them jointly with network weights through Gumbel Softmax Sampling and standard back-propagation.
Results
80% fewer parameters supported five computer vision tasks while achieving the same performance as the standard approach.
Takeaways & Limitations
AdaShare can use smaller networks to support more tasks while transferring knowledge between related tasks to improve accuracy.
Takeaways & Limitations
Adversarial attacks, data poisoning, dataset bias, interpretability, privacy, and deployment failures remain risks beyond this paper's scope.
Abstract
from arXiv · showhide
Multi-task learning is an open and challenging problem in computer vision. The typical way of conducting multi-task learning with deep neural networks is either through handcrafted schemes that share all initial layers and branch out at an adhoc point, or through separate task-specific networks with an additional feature sharing/fusion mechanism. Unlike existing methods, we propose an adaptive sharing approach, called AdaShare, that decides what to share across which tasks to achieve the best recognition accuracy, while taking resource efficiency into account. Specifically, our main idea is to learn the sharing pattern through a task-specific policy that selectively chooses which layers to execute for a given task in the multi-task network. We efficiently optimize the task-specific policy jointly with the network weights, using standard back-propagation. Experiments on several challenging and diverse benchmark datasets with a variable number of tasks well demonstrate the efficacy of our approach over state-of-the-art methods. Project page: https://cs-people.bu.edu/sunxm/AdaShare/project.html.
1 Introduction
Deep multi-task learning must determine which layers to share across tasks while balancing accuracy and resource use. AdaShare learns task-specific execution paths in a single network to adapt sharing and improve this trade-off.
- Motivation: MTL architectures must choose what parameters to share across which tasks, but manually tuning sharing configurations becomes impractical as networks and task counts grow.Sharing unrelated tasks can also cause negative transfer.
- Motivation: Soft-parameter sharing improves flexibility through task-specific networks and feature sharing, but model size grows with the number of tasks.This limits computational and memory efficiency.
- Proposed approach: AdaShare learns a feature-sharing pattern that selects which layers to execute for each task, creating separate execution paths within a single multi-task network.The approach targets recognition accuracy while restricting memory footprint.
- Proposed approach: Task-specific sharing policies and network weights are jointly optimized through Gumbel Softmax Sampling and standard back-propagation.The method avoids reinforcement learning and an additional policy network.
- Evaluation: The paper evaluates AdaShare on multiple MTL benchmarks with variable task counts to compare its accuracy and resource efficiency with state-of-the-art methods.The listed benchmarks include NYU v2, CityScapes, Tiny-Taskonomy, DomainNet, and text classification datasets.
2 Related Work
Prior deep MTL methods primarily use hand-designed hard or soft parameter sharing, while related work also explores learned architectures and adaptive routing. AdaShare is positioned as gradient-based learning of sharing patterns in one network.
- Multi-Task Learning: Soft parameter sharing combines task-specific networks with feature sharing or fusion, offering more flexibility than hard sharing.This approach is discussed as a separate deep MTL paradigm.
- Multi-Task Learning: Deep MTL commonly uses hard parameter sharing, with shared hidden layers followed by task-specific branches designed in advance.Only a few methods learn multibranch architectures using task affinity or convolutional filter grouping.
- Neural Architecture Search: Neural Architecture Search automates network design through reinforcement learning, evolutionary computation, or gradient-based optimization.AdaShare draws on NAS while directly learning task-sharing patterns in a single network.
- Adaptive Computation: Adaptive computation methods dynamically route information or select layers to improve computational efficiency.BlockDrop, for example, learns which layers to execute per sample during inference.
3 Proposed Method
AdaShare learns task-specific layer-sharing patterns within a single multi-task network, balancing recognition accuracy with memory efficiency. It jointly optimizes discrete execution policies and network weights using differentiable Gumbel-Softmax sampling, regularized for compactness and sharing.
- Approach Overview: AdaShare seeks an adaptive feature-sharing mechanism that selects shared or task-specific layers for each task while accounting for resource efficiency.The policy determines which network layers execute for each task, producing separate execution paths within one network.
- Approach Overview: The select-or-skip policy U contains one binary decision for each layer-task pair, defining whether each residual block is executed or skipped.A block is shared when used by multiple tasks and task-specific when used by only one.
- Learning a Task-Specific Policy: Gumbel-Softmax sampling relaxes discrete select-or-skip decisions into differentiable soft decisions, enabling joint policy and weight optimization through standard back-propagation.The temperature is initialized at 5 and gradually annealed toward 0 during training.
- Loss Functions: AdaShare combines task-specific losses with sparsity and sharing regularizations to omit unnecessary blocks and encourage related tasks to reuse residual blocks.The sharing loss emphasizes bottom blocks containing low-level knowledge, while the additional losses target resource efficiency without deteriorating recognition accuracy.
- Training Strategy: The training strategy alternates optimization of network weights and policy parameters, beginning with a warm-up phase that shares all blocks across tasks.The warm-up provides a starting point for policy learning, and curriculum learning is used to improve convergence.
- Parameter Complexity: Only L additional policy parameters are occupied for any new task, and the model has about 50% fewer parameters than recent deep multi-task methods when learning two tasks.This parameterization makes the model scale well in memory as more tasks are learned together.
4 Experiments
AdaShare is evaluated across diverse multi-task scenarios and consistently combines strong task performance with substantially lower parameter and computation costs. Experiments also examine its learned sharing patterns, robustness across architectures, and superiority over stochastic, random, and instance-specific policies.
- Experimental Settings: AdaShare is evaluated on NYU v2, CityScapes, Tiny-Taskonomy, DomainNet, and ten text-classification datasets across varied task configurations.The benchmark scenarios include two-, three-, and five-task learning, image classification across six visual domains, and text classification across ten datasets.
- Quantitative Results: AdaShare achieves the best performance on 4 of 7 NYU v2 two-task metrics and the second best on 1 metric, using less than half the parameters of most baselines.It also outperforms the vanilla Multi-Task baseline and DEN with similar parameter usage.
- Quantitative Results: Across other learning scenarios, AdaShare outperforms baselines on overall relative performance while saving 50%–80% of parameters compared with most baselines.In Tiny-Taskonomy, it performs best using less than 1/5 of most baseline parameters, and it reduces negative transfer in reported multi-task settings.
- Quantitative Results: AdaShare improves average accuracy over the Multi-Task baseline by 4.6% across six DomainNet domains and 7.2% across ten text-classification datasets.The reported maximum improvements are 16% in quickdraw and 27.8% in sogou_news.
- Computation Cost: AdaShare reduces computation by 7.67%–18.71% on average while achieving better recognition accuracy with about 50%–80% fewer parameters than state-of-the-art methods.Examples include 33.35G FLOPs in CityScapes two-task learning and 50.13G FLOPs in NYU v2 three-task learning.
- Ablations and Extensions: The learned policy outperforms stochastic-depth, random, and instance-specific policies, while curriculum learning, sparsity regularization, and sharing loss jointly improve policy learning.AdaShare also improves the Multi-Task baseline by 5.8% with Wide ResNets and 3.2% with MobileNet-v2.
5 Conclusion
AdaShare adaptively determines feature sharing across tasks while jointly learning the sharing policy and network weights without significant extra parameters. The approach targets compact multi-task networks and outperforms competing methods across five standard datasets.
- AdaShare adaptively determines the feature sharing strategy across multiple tasks in deep multi-task learning.
- The sharing policy and network weights are learned jointly using standard back-propagation without adding significant parameters.
- Two resource-aware regularizations produce compact multi-task networks with fewer parameters while targeting the best overall performance across tasks.
- Experiments on five standard datasets show AdaShare outperforming several competing methods.
Broader Impact
AdaShare is presented as a way to support multiple computer-vision tasks with smaller, more efficient networks. Its broader impacts include potential deployment benefits, while risks associated with deep-learning and computer-vision systems remain beyond the paper’s scope.
- AdaShare solved five computer-vision tasks with 80% fewer parameters while achieving the same performance as the standard approach.The tasks were semantic segmentation, surface normal prediction, depth prediction, keypoint detection, and edge estimation.
- The approach could support robotics applications by reducing memory and power consumption and enabling smaller, more agile systems.Potential applications include assistive robots, autonomous navigation, robotic picking and packaging, rescue robotics, and AR/VR systems.
- Potential risks include adversarial attacks, data poisoning, dataset bias, limited interpretability, privacy violations, job losses, and deployment failures.The paper states that mitigating these issues is beyond its scope.
A Full Details on the Datasets and Tasks
The experiments use diverse datasets spanning indoor and street-view computer vision, multi-task indoor benchmarks, multi-source domain adaptation, and text classification. Tasks are defined by annotations, domains, or datasets depending on the benchmark.
- CityScapes: CityScapes provides high-resolution street-view images for semantic segmentation and depth estimation.The experiments use 19-class segmentation annotations and official train/test splits.
- NYU v2: NYUv2 supports two-task semantic segmentation with surface normal prediction and three-task learning that additionally includes depth prediction.The dataset contains RGB-D indoor scene images with 40-class segmentation annotations.
- Tiny-Taskonomy: Tiny-Taskonomy contains 381,840 indoor images from 35 buildings, with annotations for 26 tasks; five representative tasks were sampled.The tiny splits were used instead of the approximately 12TB full Taskonomy dataset.
- DomainNet: DomainNet contains 0.6m images across six domains and 345 categories, with each domain treated as a task.The experiments use the official train/test splits for multi-source domain adaptation in object recognition.
- Text Classification: Text classification uses ten public datasets containing both multi-class and binary classification tasks.Classification within each dataset is treated as a task using the provided official train/test splits.
B Implementation Details
Training uses separate policy-learning and re-training phases, with dataset-specific iteration budgets and a warm-up period. Baselines receive the same number of iterations for comparison.
- Training is divided into Policy Learning and Re-training phases.NYU v2 and CityScapes use 20,000 iterations for each phase, while Tiny-Taskonomy uses 100,000 and 30,000 iterations, respectively.
- The Policy Learning Phase warms up the network for 20% of the total iterations.
- All baselines are trained with the same number of iterations.
C Implementation of Baselines
The baselines are adapted to ResNet using released implementations, while AdaShare is evaluated across multiple task settings with compact parameter usage.
- Cross-Stitch, Sluice, NDDR-CNN, MTAN, and DEN are implemented or adapted to the ResNet architecture using paper details and released code.
- AdaShare achieves average 4% improvement on five metrics for 7-class segmentation while using 58.5% fewer parameters than MTAN.
- The experiments cover NYU-v2 2-task, CityScapes 2-task, NYU-v2 3-task, and Tiny-Taskonomy 5-task learning settings.
- AdaShare achieves the best performance on five of seven CityScapes metrics and second-best on one using less than 1/2 the parameters of most baselines.
- AdaShare achieves the best performance on ten of twelve NYU-v2 metrics using less than 1/3 the parameters of most baselines.
D Full Comparison of All Metrics
The full comparisons report metric coverage and computational efficiency across multi-task benchmarks, with AdaShare reducing computation and inference time in most cases.
- The comparison covers all metrics for CityScapes 2-task, NYU-v2 3-task, and Tiny-Taskonomy 5-task learning.
- AdaShare reduces FLOPs and inference time in most cases by skipping blocks for some tasks without auxiliary networks.
F Policy Visualizations
Policy visualizations show task-dependent sharing patterns with both shared and task-specific blocks. These patterns accompany improved segmentation performance and clearer qualitative predictions across tasks.
- Policy Visualizations: AdaShare outperforms baselines on three of five Tiny-Taskonomy tasks using less than 1/5 the parameters of most baselines.
- Computational Efficiency: AdaShare consumes fewer FLOPs and shorter inference time than baselines in most scenarios.
- Policy Visualizations: Segmentation and surface-normal tasks share 14 of 16 blocks, while segmentation retains two task-specific blocks in one Tiny-Taskonomy subset.
- Policy Visualizations: AdaShare learns both shared and task-specific blocks within subsets of Tiny-Taskonomy tasks, enabling selective knowledge sharing.
- Class-wise Performance: AdaShare improves most semantic-segmentation classes over MTAN, including classes with fewer labeled pixels.
- Qualitative Results: In NYU-v2 three-task qualitative comparisons, AdaShare produces more accurate segmentation, closer surface normals, and clearer object contours.
- Ablation Studies: Ablation studies on CityScapes show AdaShare outperforms its variants on most individual metrics and overall performance.
J Full Comparison of Ablation Studies on NYU-v2 3-Task
The NYU-v2 three-task ablation section provides a full metric comparison, alongside related ablation-study and qualitative-result references.
- The section provides a full comparison of all metrics for NYU-v2 three-task ablation studies.
- The referenced NYU-v2 three-task qualitative visualization compares Multi-Task, MTAN, DEN, and AdaShare predictions.
- The cited ablation tables include CityScapes two-task and NYU-v2 three-task studies.