Source-linked AI summary
Learning Sparse Sharing Architectures for Multiple Tasks
Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, Xuanjing Huang
TL;DR
Existing sharing mechanisms face limitations with heterogeneous tasks, parameter efficiency, or manual architecture design. Sparse Sharing automatically extracts partially overlapping task subnetworks from an over-parameterized base network and trains them in parallel. Experiments report consistent improvement with fewer parameters and avoidance of negative transfer for unrelated tasks.
Problem
Hard sharing struggles with heterogeneous tasks, soft sharing is parameter-inefficient, and hierarchical sharing depends on manual design.
Method
Sparse Sharing uses an over-parameterized base network to automatically extract partially overlapping subnetworks for each task, which are trained in parallel.
Results
Sparse Sharing achieves consistent improvement over single-task and existing sharing baselines while requiring fewer parameters.
Takeaways & Limitations
Sparse Sharing flexibly handles loosely related tasks and avoids negative transfer even when tasks are unrelated.
Takeaways & Limitations
The approach assumes an over-parameterized base network whose hypothesis space can contain solutions for multiple tasks simultaneously.
Abstract
from arXiv · showhide
Most existing deep multi-task learning models are based on parameter sharing, such as hard sharing, hierarchical sharing, and soft sharing. How choosing a suitable sharing mechanism depends on the relations among the tasks, which is not easy since it is difficult to understand the underlying shared factors among these tasks. In this paper, we propose a novel parameter sharing mechanism, named \emph{Sparse Sharing}. Given multiple tasks, our approach automatically finds a sparse sharing structure. We start with an over-parameterized base network, from which each task extracts a subnetwork. The subnetworks of multiple tasks are partially overlapped and trained in parallel. We show that both hard sharing and hierarchical sharing can be formulated as particular instances of the sparse sharing framework. We conduct extensive experiments on three sequence labeling tasks. Compared with single-task models and three typical multi-task learning baselines, our proposed approach achieves consistent improvement while requiring fewer parameters.
Introduction
Existing multi-task sharing mechanisms trade off flexibility, design effort, and parameter efficiency. Sparse Sharing automatically extracts partially overlapping task subnetworks from an over-parameterized base network and improves sequence-labeling performance with fewer parameters.
- Hard sharing limits expressivity for loosely related tasks by forcing all tasks into the same hidden space.
- Hierarchical sharing accommodates task-specific modules but requires manually designing an effective sharing hierarchy.
- Sparse Sharing starts from an over-parameterized base network and automatically extracts a subnetwork for each task, allowing related tasks to overlap selectively.Each task updates only the weights of its corresponding subnet during parallel training.
- Iterative Magnitude Pruning induces task subnetworks that are merged for parallel training.
- Experiments on three sequence labeling tasks show consistent improvement over single-task learning and existing sharing mechanisms while requiring fewer parameters.The approach also alleviates negative transfer, according to the reported experiments.
Deep Multi-Task Learning
Neural multi-task models combine shared and task-specific components, but their sharing patterns impose different limitations. The section formulates these mechanisms and motivates sparse sharing as a more flexible alternative to fixed or hand-crafted sharing.
- Multi-task learning trains related tasks in parallel so their correlations can improve performance.
- In the standard formulation, tasks use shared layers E and separate task-specific layers F_t.The shared layers are parameterized by θ_E, while each task-specific module has parameters θ_t.
- Joint training optimizes a weighted sum of task losses, with λ_t controlling task t's contribution.The task weights may be tuned as hyperparameters or learned.
- Hierarchical sharing places task supervision at different network layers, giving task-specific modules more room to model heterogeneous tasks.
- Hard sharing forces all tasks into one shared hidden space, whereas hierarchical sharing still relies on hand-crafted structures.
- Soft sharing gives each task its own model and is therefore not parameter-efficient.
Learning Sparse Sharing Architectures
Sparse Sharing learns task-specific subnetworks from one over-parameterized base network, allowing tasks to share parameters selectively rather than enforcing a fixed sharing structure. The method generates subnetworks with iterative magnitude pruning, selects candidates by development performance, and trains the selected subnetworks in parallel.
- Architecture: Sparse Sharing uses an over-parameterized base network and a binary mask to assign each task a distinct subnetwork with partially shared parameters.Each task uses Mt ⊙θE as its task-specific representation while retaining task-specific layers.
- Relationship to existing mechanisms: Sparse Sharing contains hard sharing and hierarchical sharing as special cases determined by the task masks.All-one masks yield hard sharing, while a two-layer example with M1 = {1, 0} and M2 = {1, 1} yields hierarchical sharing.
- Subnet generation: The approach generates subnetworks independently for each task using Iterative Magnitude Pruning, whose masks encode how base-network parameters are used.Pruning produces multiple candidate subnetworks as sparsity increases.
- Subnet selection: For each task, the method selects the candidate with the best development-set performance, breaking ties by choosing the subnet with the lowest sparsity.
- Parallel training: The selected subnetworks are trained in parallel by repeatedly sampling a task, drawing a mini-batch, and updating that task’s subnetwork.Although each task uses its own subnet, some parameters are updated by multiple tasks, producing shared and private parameters.
- Training stabilization: Multi-Task Warmup stabilizes subnet generation and empirically improves performance while reducing differences between subnetworks generated for the same task.The base network is first trained on multiple tasks, then pruning iterations reset parameters to the warmed-up state.
Experiments
Experiments evaluate Sparse Sharing on three sequence-labeling tasks and compare it with single-task, hard-sharing, soft-sharing, and hierarchical-sharing baselines. Across experiments, the method is reported to improve performance while using fewer parameters, with subnet selection based on development-set performance.
- Experimental setup: The evaluation covers POS, NER, and Chunking across CoNLL-2003, OntoNotes 5.0, and a combined PTB, CoNLL-2000, and CoNLL-2003 setup.Exp1 uses CoNLL-2003, Exp2 uses OntoNotes 5.0, and Exp3 combines datasets for the three tasks.
- Experimental setup: The datasets include PTB, CoNLL-2000, CoNLL-2003, and OntoNotes 5.0, with BIOES tagging for NER and BIO2 tagging for Chunking.The experiments use the WSJ portion of PTB and exclude the OntoNotes pt domain because it lacks NER annotations.
- Baselines and models: Sparse Sharing is compared with single-task, hard-sharing, soft-sharing, and hierarchical-sharing models using a common CNN-BiLSTM base architecture.The experiments use fully connected output layers in Exp1 and Exp2, while Exp3 uses CRF task-specific layers.
- Subnet generation: The subnet-generation procedure uses hyperparameters including multi-task warmup steps and pruning rate, with global pruning set to α = 0.1 and word embeddings excluded from pruning.Candidate subnets are selected by development-set performance; when tied, the subnet with lower sparsity is chosen.
- Main results: Sparse Sharing consistently outperforms the reported single-task and multi-task baselines while requiring fewer parameters across the experiments.Table 3 reports accuracy for POS and F1 scores for NER and Chunking in Exp1 and Exp2; Table 5 reports Exp3 results with CRF-equipped models.
- Analysis: The reported benefit comes from shared knowledge across tasks rather than pruning alone, and the method is described as helping avoid negative transfer.Single-task evaluations of the generated task subnets do not significantly improve the corresponding task, while joint results show the stated benefit.
Analysis and Discussions
The analysis examines negative transfer, task relatedness, sparsity, and multi-task warmup. Sparse sharing avoids negative transfer in a loosely related-task setting, while mask overlap and sparsity characterize when hard sharing remains effective.
- Negative Transfer: Sparse sharing avoids the negative transfer observed with hard sharing in a synthetic pairing of CoNLL-2003 NER and Position Prediction.The two tasks are described as loosely related, and hard sharing suffers negative transfer in this experiment.
- Task Relatedness: Mask Overlap Ratio measures the zero-norm of mask intersections divided by the zero-norm of mask unions, reflecting subnet similarity and task sharing.Each mask matrix corresponds to a subnet.
- Task Relatedness: Larger mask Overlap Ratio indicates more correlated tasks, and sparse sharing’s improvement over hard sharing becomes smaller as overlap increases.The authors interpret this pattern as evidence that hard sharing remains efficient for closely related tasks.
- Sparsity: At 100% subnet sparsity, mask OR equals 1 and sparse sharing is equivalent to hard sharing; decreasing sparsity also decreases mask OR while average performance fluctuates.The sparsity analysis evaluates subnet combinations on CoNLL-2003.
- Multi-Task Warmup: Multi-Task Warmup generally improves performance compared with training without warmup.The comparison covers POS accuracy and NER and Chunking F1 on CoNLL-2003 and OntoNotes 5.0.
Related Work
Related work includes hard, hierarchical, and soft parameter sharing, alongside sparse neural-network methods. Sparse Sharing differs by combining fine-grained flexibility for heterogeneous tasks with parameter efficiency and automatic subnet extraction.
- Multi-Task Learning: Hierarchical sharing assigns different task levels to different network layers, but its hierarchy is usually designed manually using expert knowledge.The approach therefore relaxes, rather than removes, the constraint about task relatedness.
- Multi-Task Learning: Hard sharing is parameter-efficient but struggles with heterogeneous tasks, while soft sharing is flexible yet parameter-inefficient.Hierarchical sharing relaxes task-relatedness constraints but typically requires a manually designed task hierarchy.
- Sparse Sharing: Sparse Sharing is presented as a fine-grained parameter-sharing strategy that handles heterogeneous tasks while remaining parameter efficient.Unlike soft sharing, its task parameters are partially shared rather than fully separate.
- Sparse Neural Networks: Sparse neural-network research motivates the approach through the Lottery Ticket Hypothesis and Iterative Magnitude Pruning.The cited work describes subnetworks that can reach test accuracy comparable to the original network.
Conclusion
The paper proposes Sparse Sharing to address limitations of existing parameter-sharing mechanisms. It automatically extracts overlapping task subnetworks, yielding consistent improvement with fewer parameters and avoiding negative transfer in an unrelated-task experiment.
- Conclusion: Sparse Sharing partially shares parameters across tasks, making the architecture flexible for loosely related tasks.This directly targets hard sharing’s difficulty with heterogeneous tasks.
- Conclusion: An efficient procedure automatically extracts overlapping subnetworks for each task, which are trained in parallel.The approach does not require prior knowledge of task relatedness.
- Conclusion: Sparse sharing architectures achieve consistent improvement while requiring fewer parameters, and a synthetic experiment shows no negative transfer for unrelated tasks.These are the paper’s reported experimental conclusions.