Source-linked AI summary
Recurrent Neural Network for Text Classification with Multi-Task Learning
Pengfei Liu, Xipeng Qiu, Xuanjing Huang
TL;DR
Single-task neural NLP models can be constrained by insufficient labeled data and pre-training does not directly optimize the target task. This paper jointly trains related tasks using three RNN architectures with task-specific and shared layers, and reports improved performance across the evaluated text classification tasks. The conclusion notes that the approach still faces challenges on sentences requiring common-sense reasoning.
Problem
Single-task supervised neural NLP methods often suffer from limited training data, while unsupervised pre-training does not directly optimize the desired task.
Method
The paper jointly trains related tasks with three RNN multi-task architectures that share information through task-specific and shared layers.
Results
Experimental results show that the proposed multi-task models improve the performance of related text classification tasks and outperform most state-of-the-art baselines.
Takeaways & Limitations
Joint learning can improve a group of related tasks by exploring common features.
Takeaways & Limitations
Some sentiment sentences require common-sense reasoning beyond their literal meaning.
Abstract
from arXiv · showhide
Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, in most previous works, the models are learned based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we use the multi-task learning framework to jointly learn across multiple related tasks. Based on recurrent neural network, we propose three different mechanisms of sharing information to model text with task-specific and shared layers. The entire network is trained jointly on all these tasks. Experiments on four benchmark text classification tasks show that our proposed models can improve the performance of a task with the help of other related tasks.
1 Introduction
The paper addresses limited supervised data in single-task neural NLP by jointly training related tasks with recurrent multi-task architectures. Across four text classification tasks, the proposed models improve each task relative to separate learning and outperform most state-of-the-art baselines.
- Motivation: Single-task neural methods often suffer from limited training data, while unsupervised pre-training improves performance without directly optimizing the desired task.
- Motivation: Multi-task learning leverages correlations among related tasks by learning them in parallel for mutual benefit.
- Approach: The paper proposes three RNN information-sharing models with task-specific and shared layers, including a gating mechanism for selective information use.
- Approach: The entire network is trained jointly across all related tasks.
- Results: Experimental results on four text classification tasks show that joint learning improves each task relative to learning tasks separately.
- Results: The multi-task models outperform most state-of-the-art baselines on several text classification tasks.
2 Recurrent Neural Network for Specific-Task Text Classification
RNNs encode variable-length text through recurrent hidden states and classify the resulting fixed-length representation. LSTMs address long-term dependency difficulties with gated memory, while the task-specific classifier predicts class probabilities from the final hidden state.
- Text Representation: Neural text models map variable-length sequences into fixed-length vector representations of their meaning.
- Recurrent Neural Network: RNNs process arbitrary-length sequences by recursively updating a hidden state from the current input and previous hidden state.
- Task-Specific Classification: A traditional sequence classifier feeds an RNN's fixed-sized vector into a softmax layer for classification.
- Recurrent Neural Network: During training, simple RNNs can experience exponentially exploding or vanishing gradients over long sequences, hindering long-distance correlation learning.
- Long Short-Term Memory: LSTMs address long-term dependency learning by maintaining a memory cell whose contents are updated and exposed selectively.
- Long Short-Term Memory: LSTM units use input, forget, and output gates alongside a memory cell and hidden state; the gates regulate updates, erasure, and exposure.
- Task-Specific Classification: For classification, the final hidden state h_T represents the sequence and feeds a fully connected layer followed by softmax to predict class probabilities.
- Training: The network parameters are trained by minimizing cross-entropy between predicted and true distributions.
3 Three Sharing Models for RNN based Multi-Task Learning
The paper introduces three RNN-based multi-task architectures that differ in how they share information while retaining task-specific representations. These models address limited single-task training data by jointly leveraging related tasks.
- Single-task supervised neural methods can suffer from limited training data, motivating multi-task models that leverage supervised data from related tasks.
- Model-I: Uniform-Layer Architecture: Model-I shares one LSTM layer and one embedding layer across tasks while retaining separate task-specific embedding layers.Its input combines task-specific and shared word embeddings through concatenation.
- Model-II: Coupled-Layer Architecture: Model-II gives each task its own LSTM and couples the layers so each task can use information from the other task.A global gating unit controls how much information passes between the task-specific LSTMs.
- Model-III: Shared-Layer Architecture: Model-III assigns each task a separate LSTM and adds a bidirectional shared LSTM layer to capture information common to all tasks.Gating controls interaction between task-specific layers and the shared layer.
4 Training
Training uses task-specific outputs and a global objective that combines the losses of all tasks. The procedure supports separate datasets, shared-layer fine-tuning, and language-model initialization for Model-III.
- Each task-specific representation is fed into its own output layer, where task m produces prediction probabilities through a task-specific softmax.
- The global cost function is a weighted linear combination of the cost functions for all tasks.λm specifies the weight assigned to task m.
- Training can use labeled data from completely different datasets by repeatedly sampling a task and example, then taking a task-specific gradient step.
- Fine Tuning: After joint learning, Model-I and Model-III can be fine-tuned separately for each task because they contain a shared layer.
- Pre-training of the shared layer with neural language model: For Model-III, the shared LSTM layer can be initialized with a language model trained on all four task datasets.
5 Experiment
Experiments evaluate three recurrent multi-task architectures on four movie-review classification tasks, comparing joint learning with single-task LSTM and state-of-the-art neural baselines. Joint learning improves performance across tasks, with the shared-layer model performing best and qualitative analyses illustrating how shared information helps sentiment prediction.
- Datasets and setup: The study evaluates multi-task models on four movie-review classification tasks spanning sentence- and document-level data.The tasks include SST-1, SST-2, SUBJ, and IMDB.
- Multi-task architectures: The uniform-layer architecture shares an LSTM layer across tasks and improves average performance by 0.8%, reaching 2.0% with further fine-tuning.The standard LSTM is used as the single-task comparison baseline.
- Multi-task architectures: The coupled-layer architecture jointly trains task pairs, with more relevant tasks improving more; SST-1 and SST-2 gain 2.3% on average.The pairwise setup uses six combinations across the four datasets.
- Multi-task architectures: The shared-layer architecture combines task-specific layers with a shared layer, improving average performance by 1.4%, plus 0.5% from language-model pre-training and 0.9% from fine-tuning.This architecture is reported as more general than the uniform-layer architecture.
- Overall results: All proposed models outperform single-task learning, while the shared-layer architecture gives the best performance and the most complex model takes 2.5 times as long as vanilla LSTM.The authors also report faster convergence and little extra computational cost overall.
- Baseline comparison: The shared-layer model is competitive with neural state-of-the-art baselines, although Tree-LSTM outperforms it on SST-1 while requiring an external parser.The comparison includes NBOW, MV-RNN, RNTN, DCNN, PV, and Tree-LSTM.
- Case study: Case studies attribute corrected predictions to selective shared information, including stronger activation near “merry” and sensitivity to the structure “but ... higher than”.The shared layer is described as enriching word meaning and conveying structural information to the task-specific layer.
- Error analysis: Error analysis identifies complicated sentence structures and sentences requiring common-sense reasoning as recurring sources of mistakes.Examples include double negation, subjunctive constructions, and literal readings that miss implied sentiment.
6 Related Work
Earlier NLP multi-task methods shared representations or combined tasks, but the paper distinguishes its RNN-based models by handling variable-length sequences and flexibly controlling information flow between shared and task-specific layers.
- Neural multi-task models commonly share lower layers for common features before splitting into task-specific layers.
- Earlier systems used shared word representations for tasks such as part-of-speech tagging and semantic role labeling, with other layers remaining task specific.
- A multi-task DNN combined query classification and ranking for web search but represented inputs as bags of words, losing word-order information.
- The proposed models use recurrent neural networks to model variable-length text sequences rather than relying on fixed-size windows or bag-of-word inputs.
- Unlike related multi-task encoder-decoder networks for translation, these architectures flexibly control information flow between shared and task-specific layers to obtain better sentence representations.
7 Conclusion and Future Work
The paper concludes that its three RNN-based multi-task architectures improve related tasks by exploiting common features, and proposes investigating additional task-sharing mechanisms.
- Three RNN-based architectures model text sequences using different mechanisms for sharing information among tasks.
- Experimental results show that the models improve performance across a group of related tasks by exploring common features.
- Future work will investigate other mechanisms for sharing information among tasks.