Source-linked AI summary
Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks
Ji Young Lee, Franck Dernoncourt
TL;DR
Most ANN systems classify short texts without using preceding texts, despite short texts commonly occurring in sequences. The paper introduces an RNN- and CNN-based sequential classifier for dialog act prediction and reports state-of-the-art results on three datasets.
Problem
Most ANN systems classify short texts in isolation, although short texts commonly occur in sequences where preceding texts may improve accuracy.
Method
The paper combines recurrent neural networks and convolutional neural networks to incorporate preceding short texts for sequential short-text classification.
Results
The model achieves state-of-the-art results on three different datasets for dialog act prediction.
Takeaways & Limitations
Adding sequential information improves prediction quality, and performance depends on which sequential information the model uses.
Takeaways & Limitations
Rigorous comparisons are difficult because preprocessing, data splits, and the number of training runs vary across studies.
Abstract
from arXiv · showhide
Recent approaches based on artificial neural networks (ANNs) have shown promising results for short-text classification. However, many short texts occur in sequences (e.g., sentences in a document or utterances in a dialog), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.
1 Introduction
Short-text classification is important across several NLP applications, but most ANN systems classify each text without preceding context. The paper addresses this gap with an RNN- and CNN-based model for sequential short-text classification.
- Short-text classification supports applications including sentiment analysis, question answering, and dialog management.
- Most ANN systems classify short texts in isolation without considering preceding short texts.
- Short texts usually occur in sequences, motivating the use of preceding texts to improve classification accuracy.
- The paper introduces an RNN- and CNN-based model for sequential short-text classification and evaluates it on dialog act classification.The model achieves state-of-the-art results on three different datasets.
2 Model
The model represents each short text with an RNN or CNN, then classifies the current text using its representation together with representations of preceding texts. A two-layer feedforward ANN produces the probability distribution over classes.
- Short-text representation: The model first generates a vector representation for each short text using either an RNN or CNN.The input is a sequence of word vectors, and the resulting representation is an n-dimensional vector s.
- RNN-based representation: The RNN uses an LSTM to transform word-level inputs into hidden-state vectors, which are combined by last, mean, or max pooling.Last pooling selects h_ℓ, mean pooling averages the hidden states, and max pooling takes their element-wise maximum.
- CNN-based representation: The CNN applies n filters to windows of h consecutive words and uses element-wise max pooling over the resulting features to form s.Each feature dimension comes from a distinct filter, and the convolution is followed by a ReLU operation.
- Sequential classification: The sequential classifier feeds the representation sequence s_i−d1−d2:i into a two-layer feedforward ANN to predict the class for short text i.The history sizes d1 and d2 determine how many preceding representations enter the first and second layers.
- Sequential classification: The final output z_i is a probability distribution over k classes, with each element giving the probability of one class for the ith short text.The first layer produces class representations, and the second layer transforms the relevant sequence into z_i.
3 Datasets and Experimental Setup
The model is evaluated for dialog act classification across three datasets, using sequence-oriented dataset splits and standard neural-network training with regularization.
- The evaluation covers DSTC 4, MRDA, and SwDA dialog act classification datasets.
- MRDA provides train, validation, and test splits, whereas DSTC 4 and SwDA provide only train and test splits.
- The dataset overview reports class counts, vocabulary sizes, and dialog and utterance counts for the train, validation, and test sets.
- Training minimizes negative log-likelihood with stochastic gradient descent and Adadelta, updating weights, biases, and word vectors.
- Dropout follows the pooling layer, and early stopping uses the validation set with a patience of 10 epochs.
4 Results and Discussion
Experiments examine hyperparameters, representations, history sizes, and comparisons with prior methods. Sequential information improves performance, especially when incorporated at the short-text representation level, while comparisons are constrained by differing experimental details.
- Hyperparameters were selected by varying one hyperparameter at a time while keeping the others fixed.
- Sequential classification outperforms non-sequential classification, and CNN outperforms LSTM across all datasets, usually by a small margin except for SwDA.
- Increasing short-text history size d1 with d2 = 0 improves performance by 1.3–4.2 percentage points in both LSTM and CNN models.
- Sequential information at the short-text representation level is more effective than at the class representation level.
- Using sequential information at both representation levels does not help in most cases and may lower performance.
- The authors hypothesize that class representations add less information and are more likely to propagate errors from previous misclassifications.
- The model shows competitive results without human-engineered features, but rigorous comparisons are difficult because preprocessing, data splits, and training runs vary.
5 Conclusion
The paper presents an ANN-based approach to sequential short-text classification and reports improved prediction quality from sequential information. It achieves state-of-the-art dialog act prediction results on three datasets.
- The paper presents an ANN-based approach to sequential short-text classification.
- Adding sequential information improves prediction quality, with performance depending on which sequential information the model uses.
- The model achieves state-of-the-art results on three datasets for dialog act prediction.