Source-linked AI summary
Adversarial Multi-task Learning for Text Classification
Pengfei Liu, Xipeng Qiu, Xuanjing Huang
TL;DR
Existing shared-private multi-task models can contaminate shared representations with task-specific information and leave sharable features in private spaces. The paper proposes adversarial training with orthogonality constraints, evaluates it on 16 text classification tasks, and reports improved performance plus transferable shared knowledge.
Problem
Existing shared-private models do not guarantee that task-specific and sharable features remain separated, risking contamination of shared space and omission of useful sharable features.
Method
The framework combines adversarial training, orthogonality constraints, and a multi-class task discriminator to separate shared task-invariant features from private features.
Results
Across 16 text classification tasks, the approach demonstrates effectiveness, with MTL achieving 4.1% average improvement over SP-MTL's 1.0%.
Takeaways & Limitations
Shared knowledge can be condensed into an off-the-shelf neural layer and transferred to new tasks.
Takeaways & Limitations
Task-invariant features can still appear in both shared and private spaces in the shared-private model, motivating the need for orthogonality constraints.
Abstract
from arXiv · showhide
Neural network models have shown their promising opportunities for multi-task learning, which focus on learning the shared layers to extract the common and task-invariant features. However, in most existing approaches, the extracted shared features are prone to be contaminated by task-specific features or the noise brought by other tasks. In this paper, we propose an adversarial multi-task learning framework, alleviating the shared and private latent feature spaces from interfering with each other. We conduct extensive experiments on 16 different text classification tasks, which demonstrates the benefits of our approach. Besides, we show that the shared knowledge learned by our proposed model can be regarded as off-the-shelf knowledge and easily transferred to new tasks. The datasets of all 16 tasks are publicly available at \url{http://nlp.fudan.edu.cn/data/}
1 Introduction
Multi-task learning uses related tasks to improve classification, but conventional shared-private spaces can mix task-specific and sharable features. The paper proposes adversarial training and orthogonality constraints to separate shared and private representations more precisely.
- Multi-task learning improves a task by leveraging information from related tasks.
- Conventional shared-private models can contaminate shared space with task-specific features and leave sharable features in private space.This creates feature redundancy and may waste shared-space capacity.
- The example word “infantile” carries different sentiment labels across Movie and Baby-product tasks, illustrating the risk of sharing task-specific features.
- The proposed framework uses adversarial training to retain common, task-invariant information in shared space and orthogonality constraints to eliminate redundancy.
- The model extends binary adversarial training to multiple classes, supports joint training with unlabeled data, and produces shared knowledge transferable to new tasks.
2 Recurrent Models for Text Classification
The paper uses LSTM-based recurrent models for text classification, representing sequences through word embeddings and the final hidden state. A fully connected layer with softmax then predicts class probabilities, with parameters trained by cross-entropy.
- The paper adopts long short-term memory networks, a recurrent architecture designed to address long-term dependencies.
- An LSTM step uses input, forget, and output gates together with a memory cell and hidden state.The gate vectors contain values in [0, 1].
- The LSTM recurrence is specified through affine transformations, a logistic sigmoid, and elementwise multiplication.The formulation uses parameters Wp and bp for the affine transformation.
- For text classification, a lookup layer converts words into embeddings, and the final hidden state hT represents the whole sequence.
- A fully connected layer followed by softmax predicts class probabilities, while network parameters are trained by minimizing cross-entropy.
3 Multi-task Learning for Text Classification
Multi-task text classification models learn shared and task-specific representations from related tasks. These representations are combined and passed to task-specific classifiers, with training minimizing weighted cross-entropy across tasks.
- Multi-task learning uses correlations among related tasks by learning them in parallel to improve classification.
- Latent text features are represented by end-of-sentence LSTM hidden states, whose sharing scheme determines how features are grouped.
- Fully-Shared Model (FS-MTL): The fully-shared model uses one shared LSTM for all tasks, treating features as completely transferable and ignoring task-dependent features.
- Shared-Private Model (SP-MTL): The shared-private model assigns each task private and shared LSTM layers to capture task-dependent and task-invariant representations.
- Final features concatenate private and shared representations before entering the corresponding task-specific softmax layer.
- Training minimizes weighted cross-entropy across tasks, with αk specifying the weight assigned to task k.
4 Incorporating Adversarial Training
Adversarial training is introduced to keep shared representations task-invariant and prevent task-specific information from contaminating shared space. The framework uses a task discriminator and extends adversarial loss to multiple tasks, including unlabeled inputs.
- The shared-private model lacks guarantees that sharable and task-specific features remain in their intended spaces.Sharable features may remain private, while shared space can be contaminated by task-specific information.
- Adversarial training makes the shared recurrent layer oppose a task discriminator that predicts which task produced a sentence representation.The discriminator maps shared representations to task probabilities, while the shared extractor attempts to prevent reliable task prediction.
- The multi-class adversarial loss trains shared representations to mislead task classification while the discriminator learns to classify task types accurately.At equilibrium, the discriminator cannot differentiate among the tasks.
- The adversarial loss requires only input sentences, allowing the framework to incorporate unlabeled corpora through semi-supervised multi-task learning.It does not require the corresponding task labels for each sentence.
- 4.3 Orthogonality Constraints: Orthogonality constraints penalize redundant latent representations and encourage shared and private extractors to encode different aspects of the input.The loss uses representations from the shared and task-specific extractors to discourage overlap.
5 Experiment
Experiments evaluate the framework on 16 review-classification datasets, showing lower error rates, transferable shared representations, and qualitative separation of shared and task-specific features.
- Datasets: 16 datasets from product and movie review corpora were randomly split into training, development, and testing sets with proportions of 70%, 20%, and 10%.The first 14 datasets are product reviews; the remaining two are IMDB and MR movie-review datasets.
- Performance Evaluation: The proposed model achieves the lowest error rates across the compared multi-task models on most of the 16 classification tasks.Table 2 compares single-task LSTM variants with multiple-task systems, including SP-MTL and the proposed model.
- Performance Evaluation: 4.1% average improvement surpasses SP-MTL with 1.0%, indicating the contribution of adversarial learning.The comparison is reported over the evaluated tasks in Table 2.
- Shared Knowledge Transfer: A shared extractor trained on 15 tasks can be transferred to the remaining task, with Bi-Channel transfer outperforming Single Channel because its extra LSTM stores private features.The transferred layer remains frozen while the target-task network’s remaining parameters are randomly initialized.
- Visualization: Visualization shows SP-MTL’s shared layer includes task-specific features, whereas adversarial training produces more task-invariant representations and avoids a wrong prediction caused by “asleep.”Both models capture “Five stars,” but only the proposed model predicts the positive Baby-task sentence correctly.
- Visualization: SP-MTL places task-specific patterns into shared space and task-invariant features into private space, while ASP-MTL yields only a small intersection between the two spaces.The observed separation allows the shared and task-specific layers to work effectively.
6 Related Work
Prior work applies neural multi-task learning and adversarial networks to related problems, while this model targets task-invariant information across multiple text-classification tasks.
- Neural Multi-task Learning: Neural multi-task learning has been applied to NLP by sharing parameters, LSTM layers, external memory, or other representations across tasks.The cited approaches include LSTM-based and external-memory multi-task frameworks.
- Adversarial Networks: Adversarial networks have been used for distribution equivalence, domain adaptation, and multi-modal representation learning.These applications motivate adversarial training as a representation-learning strategy.
- Distinction: Unlike prior models, this approach uses adversarial training to find task-invariant sharable information across multiple related tasks and extends binary adversarial training to multi-class.The multi-class extension enables multiple tasks to be jointly trained.
7 Conclusion
The paper proposes adversarial multi-task learning to separate task-specific and task-invariant features non-redundantly, validating it across 16 text-classification tasks and through qualitative analysis.
- Conclusion: The framework learns task-specific and task-invariant features non-redundantly to separate shared-private representations across tasks.The conclusion presents this separation as the framework’s central design goal.
- Conclusion: 16 text-classification tasks provide quantitative evidence for the effectiveness of the proposed approach.The paper also performs qualitative analysis to explain the observed performance improvements.
- Conclusion: Qualitative analysis supplies insights that indirectly explain the quantitative improvements in overall performance.The analysis examines how shared and task-specific features behave.