Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks

• Short-text classification supports applications including sentiment analysis, question answering, and dialog management.Source: §1 · p. 1
• Most ANN systems classify short texts in isolation without considering preceding short texts.Source: §1 · p. 1
• Short texts usually occur in sequences, motivating the use of preceding texts to improve classification accuracy.Source: §1 · p. 1
• The paper introduces an RNN- and CNN-based model for sequential short-text classification and evaluates it on dialog act classification.The model achieves state-of-the-art results on three different datasets.Source: §1 · p. 1

Short-text representation

• The model first generates a vector representation for each short text using either an RNN or CNN.The input is a sequence of word vectors, and the resulting representation is an n-dimensional vector s.Source: §2 · p. 1, §2.1 · p. 2

RNN-based representation

• The RNN uses an LSTM to transform word-level inputs into hidden-state vectors, which are combined by last, mean, or max pooling.Last pooling selects h_ℓ, mean pooling averages the hidden states, and max pooling takes their element-wise maximum.Source: §2.1.1 · p. 2

CNN-based representation

• The CNN applies n filters to windows of h consecutive words and uses element-wise max pooling over the resulting features to form s.Each feature dimension comes from a distinct filter, and the convolution is followed by a ReLU operation.Source: §2.1.2 · pp. 2–3

Sequential classification

• The sequential classifier feeds the representation sequence s_i−d1−d2:i into a two-layer feedforward ANN to predict the class for short text i.The history sizes d1 and d2 determine how many preceding representations enter the first and second layers.Source: §2.2 · p. 3
• The final output z_i is a probability distribution over k classes, with each element giving the probability of one class for the ith short text.The first layer produces class representations, and the second layer transforms the relevant sequence into z_i.Source: §2.2 · p. 3

• The evaluation covers DSTC 4, MRDA, and SwDA dialog act classification datasets.Source: §3.1 · p. 3
• MRDA provides train, validation, and test splits, whereas DSTC 4 and SwDA provide only train and test splits.Source: §3.1 · p. 3
• The dataset overview reports class counts, vocabulary sizes, and dialog and utterance counts for the train, validation, and test sets.Source: Table 1 · p. 3
• Training minimizes negative log-likelihood with stochastic gradient descent and Adadelta, updating weights, biases, and word vectors.Source: §3.2 · p. 3
• Dropout follows the pooling layer, and early stopping uses the validation set with a patience of 10 epochs.Source: §3.2 · p. 3

• Hyperparameters were selected by varying one hyperparameter at a time while keeping the others fixed.Source: §4 · p. 3, Table 2 · p. 3
• Sequential classification outperforms non-sequential classification, and CNN outperforms LSTM across all datasets, usually by a small margin except for SwDA.Source: Table 3 · p. 4
• Increasing short-text history size d1 with d2 = 0 improves performance by 1.3–4.2 percentage points in both LSTM and CNN models.Source: §4 · p. 4
• Sequential information at the short-text representation level is more effective than at the class representation level.Source: §4 · p. 4
• Using sequential information at both representation levels does not help in most cases and may lower performance.Source: §4 · p. 4
• The authors hypothesize that class representations add less information and are more likely to propagate errors from previous misclassifications.Source: §4 · p. 4
• The model shows competitive results without human-engineered features, but rigorous comparisons are difficult because preprocessing, data splits, and training runs vary.Source: §4 · p. 4

• The paper presents an ANN-based approach to sequential short-text classification.Source: §5 · p. 4
• Adding sequential information improves prediction quality, with performance depending on which sequential information the model uses.Source: §5 · p. 4
• The model achieves state-of-the-art results on three datasets for dialog act prediction.Source: §5 · p. 4