Source-linked AI summary
A Unified Perspective on Multi-Domain and Multi-Task Learning
Yongxin Yang, Timothy M. Hospedales
TL;DR
The paper addresses how to share information across related tasks and domains, including settings where target categories or domains have no training data. It introduces a neural-network framework based on semantic descriptors, and experiments show that it unifies these settings while outperforming a variety of alternatives.
Problem
MTL and MDL require effective information sharing across related tasks or domains, while ZSL and ZSDA require constructing models for unseen categories or domains without target-domain training data.
Method
The framework represents tasks and domains with semantic descriptors and uses them in a neural-network model to unify MTL, MDL, ZSL, and ZSDA.
Results
Experiments across multi-domain, multi-task, zero-shot learning, and zero-shot domain adaptation show that the framework is best or joint-best in the reported audio setting and achieves 43.79% multi-class accuracy on AwA ZSL.
Takeaways & Limitations
Semantic descriptors enable information sharing with multivariate task or domain metadata and support model synthesis for unseen domains.
Takeaways & Limitations
The current framework uses discrete semantic-descriptor variables and assumes that the task or domain descriptor is observed.
Abstract
from arXiv · showhide
In this paper, we provide a new neural-network based perspective on multi-task learning (MTL) and multi-domain learning (MDL). By introducing the concept of a semantic descriptor, this framework unifies MDL and MTL as well as encompassing various classic and recent MTL/MDL algorithms by interpreting them as different ways of constructing semantic descriptors. Our interpretation provides an alternative pipeline for zero-shot learning (ZSL), where a model for a novel class can be constructed without training data. Moreover, it leads to a new and practically relevant problem setting of zero-shot domain adaptation (ZSDA), which is the analogous to ZSL but for novel domains: A model for an unseen domain can be generated by its semantic descriptor. Experiments across this range of problems demonstrate that our framework outperforms a variety of alternatives.
1 INTRODUCTION
The paper presents a neural-network framework that unifies multi-domain and multi-task learning through multivariate semantic descriptors. It extends this perspective to zero-shot learning and introduces zero-shot domain adaptation for synthesizing models for unseen domains.
- Multi-domain learning shares information across domains for the same problem, whereas multi-task learning shares information across different problems in the same domain.The paper distinguishes domains by covariates such as dataset bias or capture device, using the Office Dataset as an example.
- The framework uses semantic descriptors for tasks and domains to support simultaneous multi-domain and multi-task learning.These descriptors can be available as metadata and are used to improve information sharing.
- Existing MTL and MDL algorithms can be interpreted as making different assumptions about task and domain semantic descriptors.The paper argues that categorical assumptions are less effective when richer metadata is available, such as school-id paired with year-group.
- The framework provides an alternative zero-shot learning pipeline for constructing classifiers for categories unseen during training.Zero-shot learning constructs a test-time classifier for unseen categories without training examples for those categories.
- Zero-shot domain adaptation synthesizes a model for an unseen domain using only its semantic descriptor.The paper presents this as a new setting analogous to zero-shot learning but focused on novel domains.
2 RELATED WORK
Related work frames MTL and MDL around assumptions about shared structure, task grouping, and domain representation. The paper positions its approach as using structured, multivariate descriptors rather than only atomic categorical task or domain indices.
- Multi-Task Learning: Early MTL models decompose each task predictor into shared knowledge and task-specific knowledge.The formulation represents task i as wi := w0 + vi.
- Multi-Task Learning: Low-dimensional-subspace methods encourage parameter sharing but can suffer negative transfer when unrelated tasks are forced to share.Task grouping partially alleviates this issue but can miss information shared across groups.
- Multi-Task Learning: GO-MTL represents each task model as a linear combination of latent predictors, with coefficients determining how the model is constructed.Its formulation is W = LS, where L contains latent predictors and S contains task-specific coefficient vectors.
- Multi-Task Learning: Many MTL methods treat each task as an atomic entity indexed by one categorical variable, limiting representation of structured metadata such as school-id and year-group.Some later methods replace the predictor matrix with a tensor to represent multiple categorical variables.
- Multi-Domain Learning: The paper generalizes conventional categorical domain representations by sharing information across domains described by vectors of discrete parameters.This contrasts with prior extensions using a single continuous parameter such as time or viewing angle.
- Multi-Domain Learning: MDL differs from conventional domain adaptation by encouraging bidirectional knowledge sharing across domains rather than one-way transfer from source to target.MDL makes predictions for the same problem across domains, while domain adaptation explicitly distinguishes source and target domains.
- Zero-Shot Learning: ZSL constructs classifiers for test-time categories without observed training data by using mid-level information.The paper extends this idea to domains whose target-domain training data have not been seen before testing.
3 MODEL
The framework represents tasks and domains with semantic descriptors in a two-sided neural network, unifying existing MTL/MDL methods and extending them to joint learning and zero-shot settings.
- General framework: Each instance is paired with a semantic descriptor indicating its domain or task, and the objective minimizes empirical risk across domains or tasks.
- General framework: The two-sided network learns representations from features on one side and constructs models from semantic descriptors on the other, trained by backpropagation.The two sides use weights P and Q, while the loss compares predictions with ground-truth labels.
- Unification of existing algorithms: One inner-product layer per side is sufficient to interpret several existing MDL/MTL algorithms within the framework.The cited algorithms include RMTL, FEDA, MTFL, and GO-MTL.
- Unification of existing algorithms: Existing methods can be viewed as using specific settings of the descriptor and network weights, typically with a single categorical 1-of-N task or domain index.The framework preserves the original methods’ corresponding matrix roles while interpreting their descriptor construction uniformly.
- Unification of existing algorithms: Structured multivariate metadata can replace simple categorical indices, enabling distributed descriptors intended to improve information sharing when such metadata is available.Two categorical variables with two states each encode four domains through distributed combinations rather than four independent 1-of-N codes.
- Learning settings: The same framework supports simultaneous multi-domain multi-task learning by concatenating domain and task descriptors, and supports ZSL and ZSDA through novel descriptors.For ZSDA, a model for an unseen domain is constructed using its descriptor and data from that domain.
4 EXPERIMENTS
Experiments evaluate the framework across MDL, ZSDA, MTL, ZSL, and MDMT settings, using semantic descriptors for domains and tasks. Across these settings, the method generally outperforms alternatives and synthesizes models for unseen domain combinations.
- The experiments cover five settings: MDL, ZSDA, MTL, ZSL, and MDMT.
- School Dataset: MDL and ZSDA: For school data, leave-one-domain-out ZSDA constructs each held-out model from its descriptor, and the method outperforms alternatives.The dataset contains 69 domains formed from school IDs and year groups, with 50%/50% training and test splits.
- Audio Recognition: MDL and ZSDA: In audio recognition, the method is best or joint-best across domains because it exploits semantic descriptors, with one exception for noise-free ZSDA after noisy-only training.The four domains combine smartphone-microphone and live-concert-hall conditions; the result supports synthesizing models for unseen multivariate covariate combinations.
- Animal with Attributes: MTL and ZSL: On Animal with Attributes, the attribute-descriptor MTL approach improves accuracy by about 2% over STL, while ZSL reaches 43.79% multi-class accuracy versus 41.03% for DAP.The ZSL split trains on 40 animal categories and tests on 10 held-out categories.
- Restaurant & Consumer Dataset: MDMT: On the Restaurant & Consumer dataset, representing restaurant and score type jointly as distributed MDMT outperforms the traditional MTL formulation.The semantic descriptor concatenates an 8-bit domain indicator with a 3-bit task indicator.
5 CONCLUSION
The paper presents a unified framework for multi-domain and multi-task learning centered on semantic descriptors. It extends this framework to zero-shot domain adaptation and offers an alternative pipeline for zero-shot learning, while noting scope limitations around descriptor representation and availability.
- The framework unifies multi-domain and multi-task learning through semantic descriptors for tasks or domains.
- Semantic descriptors enable better information sharing when additional metadata is available and support the framework’s extensions beyond standard multi-task and multi-domain learning.
- The framework enables zero-shot domain adaptation and provides an alternative pipeline for zero-shot learning.
- The current descriptor uses discrete variables, while extending to continuous or periodic variables and handling missing descriptors remain future directions.