Source-linked AI summary
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, Bo Xu
TL;DR
The paper addresses the limitation of pooling RNN representations only across time, which ignores the feature-vector dimension. It combines BLSTM with 2D max pooling and 2D convolution, achieving excellent performance on 4 of 6 tasks and highest accuracy on SST-1 and SST-2. The authors note that finer hyperparameter tuning may further improve performance.
Problem
1D max pooling and attention ignore the feature-vector dimension of RNN representations, which may limit sentence representation.
Method
The paper combines BLSTM with 2D max pooling and 2D convolution to capture information across time-step and feature-vector dimensions.
Results
BLSTM-2DCNN achieves excellent performance on 4 out of 6 text classification tasks and highest accuracy on SST-1 and SST-2.
Takeaways & Limitations
The experiments support using BLSTM-2DCNN to represent text with information from both matrix dimensions across multiple classification tasks.
Takeaways & Limitations
The reported hyperparameters were selected by grid search on SST-1 development data, and finer tuning may further improve performance.
Abstract
from arXiv · showhide
Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Language Processsing (NLP) tasks because its recurrent structure is very suitable to process variable-length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimensions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D convolution to sample more meaningful information of the matrix. Experiments are conducted on six text classification tasks, including sentiment analysis, question classification, subjectivity classification and newsgroup classification. Compared with the state-of-the-art models, the proposed models achieve excellent performance on 4 out of 6 tasks. Specifically, one of the proposed models achieves highest accuracy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks.
1 Introduction
The paper identifies a limitation in sequence representations that pool only across time, then proposes BLSTM models using two-dimensional pooling and convolution to integrate time-step and feature dimensions.
- RNNs process variable-length text and represent word tokens as a matrix with time-step and feature-vector dimensions.
- 1D max pooling and attention operate only across time steps, ignoring potentially important dependencies along the feature-vector dimension.
- 2D convolution and pooling treat the RNN matrix as an image-like structure, sampling features across both dimensions.
- BLSTM-2DPooling applies BLSTM followed by 2D max pooling, while BLSTM-2DCNN additionally uses 2D convolution for text representation.
- The models are evaluated across six text classification tasks, with BLSTM-2DCNN achieving excellent performance on 4 out of 6 tasks and highest accuracy on both SST binary and fine-grained classification.
2 Related Work
Related work spans recursive, recurrent, convolutional, and other neural architectures for text classification, while this paper combines BLSTM with two-dimensional convolution and pooling.
- Deep learning text-classification models commonly map words to vectors and combine them with neural networks to form fixed-length representations.
- Recursive Neural Networks: Recursive neural networks build phrase and sentence representations over tree structures by combining information from leaves and internal nodes.
- Recurrent Neural Networks: RNN-based models preserve sequence information over time, including target-dependent LSTM, Tree-LSTM, and bidirectional LSTM with attention.
- Convolution Neural Networks: CNNs were developed with 2D convolution and pooling for images but are commonly adapted to NLP with 1D convolution and pooling over time.
- Comparison with Related Models: Compared with DSCNN and RCNN, BLSTM-2DCNN uses BLSTM for long-term dependencies and 2D rather than 1D convolution and max pooling.
3 Model
The model combines BLSTM sequence representations with two-dimensional convolution and max pooling to capture information across time-step and feature-vector dimensions before classification.
- BLSTM Layer: BLSTM processes text in forward and backward directions to capture both past and future context.
- Two-dimensional Convolution Layer: The BLSTM outputs form a matrix whose feature vectors can be treated as an image for two-dimensional operations.
- Two-dimensional Convolution Layer: A 2D convolution filter scans windows spanning k words and d feature-vector dimensions to produce local feature maps.
- Two-dimensional Max Pooling Layer: Two-dimensional max pooling extracts maximum values from matrix windows and produces a fixed-length vector.
- Output Layer: The pooled representation is passed to a softmax classifier, while training minimizes regularized categorical cross-entropy using stochastic gradient descent with AdaDelta.
4 Experimental Setup
Experiments evaluate the proposed models on six text-classification datasets using pretrained, fine-tuned word embeddings and task-specific evaluation metrics.
- Datasets: The experiments use six datasets covering sentiment, subjectivity, question, and newsgroup classification tasks.
- Datasets: SST-2 removes neutral reviews from SST-1 and uses binary negative and positive labels, with phrases and sentences used for training but sentences scored at test time.
- Word Embeddings: The models use GloVe embeddings trained on 6 billion tokens from Wikipedia 2014 and Gigaword 5, with embeddings fine-tuned during training.
- Evaluation: Accuracy is used for five datasets, while 20Ng is evaluated with Macro-F1.
- Hyper-parameter Settings: Hyper-parameters are selected by grid search on the SST-1 development set, and finer tuning may further improve performance.
5 Results
The proposed BLSTM-2DPooling and BLSTM-2DCNN models evaluate two-dimensional operations for text classification, with BLSTM-2DCNN performing strongly across benchmarks. Results also examine sentence length and operation sizes, showing both performance benefits and practical trade-offs.
- 5.1 Overall Performance: BLSTM-2DCNN achieves excellent performance on 4 out of 6 classification tasks, including the strongest results on SST-1 and SST-2.It records 52.4% test accuracy on SST-1 and 89.5% on SST-2.
- 5.1 Overall Performance: BLSTM-2DCNN does not depend on external language-specific features such as dependency parse trees, unlike ReNN.
- 5.1 Overall Performance: AdaSent outperforms BLSTM-2DCNN on Subj and MR, whereas BLSTM-2DCNN outperforms DSCNN on five datasets.
- 5.2 Effect of Sentence Length: Both BLSTM-2DPooling and BLSTM-2DCNN outperform BLSTM and BLSTM-Att, while accuracy declines as sentence length increases.The sentence-length analysis uses accuracy averaged within a [l − 2, l + 2] window, with each point averaged over 5 runs.
- 5.3 Effect of 2D Convolutional Filter and 2D Max Pooling Size: On SST-1, the best accuracy is 52.6 with a 2D filter size of (5,5) and 2D max pooling size of (5,5).Larger filters may detect more features but require more storage and computation time.
6 Conclusion
The paper introduces BLSTM-2DPooling and its extension, BLSTM-2DCNN, to preserve both time-step and feature-vector information. Across six classification tasks, BLSTM-2DCNN generally outperforms the compared models and reaches the highest accuracy on SST-1 and SST-2.
- BLSTM-2DPooling and BLSTM-2DCNN combine BLSTM with two-dimensional pooling, while BLSTM-2DCNN additionally uses 2D convolution.
- BLSTM-2DCNN achieves the highest accuracy on the SST-1 and SST-2 datasets.
- The sensitivity analysis finds that larger filters can detect more features and may improve performance.