Source-linked AI summary
A Modulation Module for Multi-task Learning with Applications in Image Retrieval
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, Ying Wu
TL;DR
Multi-task learning can suffer destructive interference when weakly related tasks compete through shared parameters. The paper introduces an end-to-end modulation module that adaptively shares features among related tasks and separates unrelated ones, and reports improved retrieval performance, scalability, and storage efficiency across CelebA and UT-Zappos50K.
Problem
Weakly related tasks can generate conflicting training signals through shared parameters, undermining joint multi-task learning.
Method
An end-to-end modulation module modulates shared-parameter gradients and task-specific features without requiring prior task relationships.
Results
Experiments on CelebA and UT-Zappos50K report improved task-specific retrieval features and superiority over existing multi-task methods, including support for up to 20 jointly learned attributes.
Takeaways & Limitations
The approach provides a flexible multi-task training scheme that promotes related-task sharing while disentangling unrelated tasks and supports compact scalable models.
Takeaways & Limitations
The paper leaves validation of the module for general multi-task settings with explicitly conflicting signals as future work and notes a runtime–memory trade-off in inference.
Abstract
from arXiv · showhide
Multi-task learning has been widely adopted in many computer vision tasks to improve overall computation efficiency or boost the performance of individual tasks, under the assumption that those tasks are correlated and complementary to each other. However, the relationships between the tasks are complicated in practice, especially when the number of involved tasks scales up. When two tasks are of weak relevance, they may compete or even distract each other during joint training of shared parameters, and as a consequence undermine the learning of all the tasks. This will raise destructive interference which decreases learning efficiency of shared parameters and lead to low quality loss local optimum w.r.t. shared parameters. To address the this problem, we propose a general modulation module, which can be inserted into any convolutional neural network architecture, to encourage the coupling and feature sharing of relevant tasks while disentangling the learning of irrelevant tasks with minor parameters addition. Equipped with this module, gradient directions from different tasks can be enforced to be consistent for those shared parameters, which benefits multi-task joint training. The module is end-to-end learnable without ad-hoc design for specific tasks, and can naturally handle many tasks at the same time. We apply our approach on two retrieval tasks, face retrieval on the CelebA dataset [1] and product retrieval on the UT-Zappos50K dataset [2, 3], and demonstrate its advantage over other multi-task learning methods in both accuracy and storage efficiency.
1 Introduction
The paper targets destructive interference when weakly related tasks share parameters, proposing a learnable modulation module that adapts feature sharing and improves multi-task retrieval scalability.
- 1 Introduction: Shared feature learning works best for correlated, complementary tasks, while unrelated tasks can produce conflicting signals that harm joint training.Conflicting gradient directions can make joint training more difficult and negatively affect all tasks.
- 1 Introduction: The proposed modulation module adaptively couples correlated tasks and decouples unrelated tasks during feature learning.It modulates shared-parameter gradient directions during back-propagation and task-specific feature spaces during feed-forward computation.
- 1 Introduction: The module is end-to-end learnable, insertable into arbitrary network architectures, and does not require prior task relationships.Its design supports handling many tasks without manually grouping them.
- 1 Introduction: Experiments on CelebA face retrieval and UT-Zappos50K product retrieval show improved task-specific features and superiority over existing multi-task methods.The approach is reported to provide a more compact model and stronger advantages as more tasks are involved.
- 1 Introduction: The framework scales to jointly learning as many as 20 attributes in retrieval experiments.The reported advantage becomes more significant with more tasks involved.
2 Related Work
Related work studies task correlations, parameter sharing, and discriminative feature extraction, whereas this paper explicitly addresses destructive interference with a flexible modulation module.
- 2 Related Work: Prior multi-task methods exploit correlations through cross-residual connections, cross-stitch activations, and jointly trained task networks.These approaches primarily seek benefits from related tasks during joint training.
- 2 Related Work: Other methods automatically discover task groupings, factorize sharing, or mask shared features differently for each task.These approaches aim to determine what and how to share beyond manually designed task-specific architectures.
- 2 Related Work: Unlike prior work, this paper explicitly identifies and quantifies destructive interference while offering an end-to-end module insertable anywhere in a network.The module can enhance structures learned by automatic task-grouping methods.
- 2 Related Work: The paper applies its method to learning discriminative task-specific image features for accurate retrieval rather than retrieval efficiency.Its design is compatible with improved backbone architectures and is agnostic to the loss function.
3 Our Method
The method identifies destructive interference from conflicting task gradients and introduces a learnable modulation module that adapts feature sharing by task. The module is evaluated through face attribute retrieval, where task compatibility, accuracy, and learned task relationships support its design.
- 3.1 Destructive interference: Conflicting gradients from weakly related tasks hinder shared-parameter learning and can lead to low-quality local optima.The problem arises because gradient directions from different tasks may conflict or oppose one another during joint training.
- Empirical Evidence: In the face-retrieval experiment, joint training showed greater compatibility for smile and open-mouth than for smile and young, matching their accuracy difference.Update Compliance Ratio measures the proportion of mini-batches with consistent task gradients, and the proposed module increased UCR for both task pairs.
- 3.2 Modulation Module: The modulation module uses task-specific feature projections or masks to make shared-parameter updates more consistent across tasks.In back-propagation, the update is represented as Mt∇θt+Mt′∇θt′, while the same module modulates task-specific feature spaces in the forward pass.
- 3.3 Training: The module parameters are learned jointly with the network through back-propagation using a triplet-loss objective over task-specific samples.The training formulation uses triplets consisting of anchor, positive, negative, and task labels.
- 3.2 Modulation Module: The module maintains feature-map size, can be inserted between network layers, and is implemented with increasingly compact channel-wise transformations.The design simplifies a full projection matrix to a channel-wise projection matrix and then primarily evaluates a channel-wise scaling vector.
- Empirical Evidence: Learned modulation parameters implicitly captured task relationships, while adding explicit relationship regularization produced only marginal accuracy gains.The reported parameter differences were used to compare task relevancy, and the regularization term was omitted from large-scale experiments because manually specifying pairwise relationships is impractical.
4 Experiments
Experiments evaluate retrieval under joint training and show that modulation improves accuracy, mitigates destructive interference, and uses task-specific parameters efficiently across face and product settings.
- 4.1 Setup: The evaluation uses triplet retrieval accuracy based on Euclidean distances between learned image features for face and product retrieval tasks.Each triplet contains an anchor, positive, and negative image, and success requires preferring the positive sample.
- 4.2 Face Retrieval: With far fewer parameters, the proposed method achieves the best mean accuracy over 20 jointly trained face attributes.Table 3 compares the method with competing approaches.
- 4.2 Face Retrieval: Single fully shared networks and CSN suffer from destructive interference, with CSN failing to scale from seven to 20 jointly trained tasks.The paper attributes this to extensive naive feature sharing and conflicting updates.
- 4.2 Face Retrieval: The method achieves slightly better average accuracy than independently trained task-specific networks with almost 20 times fewer parameters.Improvements are reported for face-shape and beard-related attributes.
- 4.2 Face Retrieval: The method significantly improves Update Compliance Ratio for all task pairs, indicating more consistent shared-parameter gradients.The comparison uses seven jointly trained face attributes against a fully shared baseline.
- 4.2 Face Retrieval: On 40 face attributes, the method obtains 85.75% average accuracy versus 78.22% for the same-complexity IB-25 baseline.The fixed-shared-parameter experiment performs worse than the full pipeline, supporting improved shared-parameter learning.
- Ablation Study: Performance generally increases as modulation is added to more layers, while channel-wise projection yields only marginal improvement over channel-wise scaling at higher parameter cost.The results support channel-wise scaling as a cost-effective design.
5 Discussion
The module is designed to support different tasks and losses, but broader applicability remains to be validated experimentally. Its inference use also involves a runtime–memory trade-off despite compact storage and modest feed-forward overhead.
- 5.1 General applicability: The module is designed to handle different tasks and loss functions, but this broader applicability remains an assumption for future validation.The paper’s experiments focus on multi-task image retrieval with similar network structures and loss functions.
- 5.2 Speed and Memory size Trade-off: Example face-retrieval outputs compare models jointly trained on 20 face attributes using CSN and the proposed method, with incorrect rankings highlighted in red.The figure provides a qualitative comparison rather than a numerical result.
- 5.2 Speed and Memory size Trade-off: 15% feed-forward overhead accompanies the proposed modules, while inference trades memory footprint against runtime speed.Keeping task-specific feature maps enables single-pass inference, whereas iterative feed-forward reduces memory use.
- 5.2 Speed and Memory size Trade-off: The method can achieve better accuracy with a more compact storage model, and its increased memory footprint is described as sustainable for 20 tasks.The paper presents either single-pass or iterative inference as feasible depending on the speed–memory preference.
6 Conclusion
The paper proposes a modulation module that addresses destructive interference in multi-task learning by modulating gradient directions and extracting task-specific features. Experiments on CelebA and UT-Zappos50K support its effectiveness and advantages over other multi-task learning methods.
- 6 Conclusion: The proposed modulation module alleviates destructive interference by modulating gradient directions during back-propagation.The paper introduces Update Compliance Ratio to quantify destructive interference in joint learning of unrelated tasks.
- 6 Conclusion: The module exploits related tasks to help extract better task-specific features in multi-task learning.
- 6 Conclusion: Experiments on CelebA and UT-Zappos50K verify the approach’s effectiveness and advantage over other multi-task learning methods.