Source-linked AI summary
Effective Use of Word Order for Text Categorization with Convolutional Neural Networks
Rie Johnson, Tong Zhang
TL;DR
The paper asks whether CNNs can use word order effectively for text categorization without relying on separately learned word vectors. It directly applies CNNs to high-dimensional text regions, including sequence and bag-of-word convolution variants, and reports state-of-the-art performance across sentiment and topic classification.
Problem
Word-of-bag representations lose word order, while using n-grams is not consistently effective across sentiment and topic categorization.
Method
The paper applies CNNs directly to high-dimensional one-hot text, learning embeddings of small regions through sequence, bag-of-word, and combined convolution layers.
Results
State-of-the-art performances on sentiment classification and topic classification were achieved using the proposed CNN approach.
Takeaways & Limitations
CNN provides an alternative mechanism for using word order through direct embedding of small text regions, with multiple embeddings combined for higher accuracy.
Takeaways & Limitations
Large vocabularies and region sizes can make seq-CNN models too complex or expensive to train, requiring reductions that may not suit the task.
Abstract
from arXiv · showhide
Convolutional neural network (CNN) is a neural network that can make use of the internal structure of data such as the 2D structure of image data. This paper studies CNN on text categorization to exploit the 1D structure (namely, word order) of text data for accurate prediction. Instead of using low-dimensional word vectors as input as is often done, we directly apply CNN to high-dimensional text data, which leads to directly learning embedding of small text regions for use in classification. In addition to a straightforward adaptation of CNN from image to text, a simple but new variation which employs bag-of-word conversion in the convolution layer is proposed. An extension to combine multiple convolution layers is also explored for higher accuracy. The experiments demonstrate the effectiveness of our approach in comparison with state-of-the-art methods.
1 Introduction
The paper applies CNNs to text categorization to exploit word order, directly embedding high-dimensional text regions rather than relying on word-vector lookup. It evaluates straightforward and bag-of-word convolution variants, finding complementary task-specific strengths and improvements over conventional methods.
- Text categorization assigns predefined categories to natural-language documents, including topics, spam, and sentiment.
- Bag-of-word representations discard word order, while n-grams can help sentiment classification but are not consistently effective for topic categorization.
- CNN uses text’s one-dimensional word order by making convolution units respond to small document regions, analogous to image regions.
- The approach directly learns embeddings of text regions from high-dimensional one-hot vectors, avoiding a separate word-vector lookup layer.
- seq-CNN adapts image CNNs to text, whereas bow-CNN applies bag-of-word conversion within the convolution layer.
- CNN outperforms bow-CNN on sentiment classification, while bow-CNN performs better on topic classification; the winner generally beats conventional bag-of-n-gram methods and more complex prior text CNNs.
2 CNN for document classification
The paper adapts CNNs to document classification by representing word order as one-dimensional structure and learning embeddings of text regions directly from one-hot inputs. It introduces sequence-preserving and bag-of-word convolution variants, variable-sized pooling, and parallel convolution layers.
- CNN for text: CNN treats documents as one-dimensional data, with convolution units responding to small regions consisting of word sequences.Words are represented as one-hot vectors while preserving their locations in the document.
- seq-CNN: Seq-CNN concatenates one-hot vectors within fixed-size regions, then learns low-dimensional feature embeddings for those ordered text regions.For region size p, each region vector has dimensionality p|V|.
- Computational constraint: Large p|V| region vectors can make seq-CNN overly complex or expensive, requiring reductions in vocabulary or region size that may be undesirable.The constraint depends on available training data and the task.
- bow-CNN: Bow-CNN converts each region to a |V|-dimensional bag-of-word vector, reducing parameters while losing word order only within small regions.Its expressiveness lies between sequence convolution and document-level bag-of-word vectors.
- Pooling for text: Because documents vary in length, the convolution output is variable-sized; dynamically determined pooling produces fixed-sized representations for the fully connected top layer.Multiple pooling units can preserve coarse location information, and average-pooling as well as max-pooling are supported.
3 Experiments
Experiments evaluate CNN variants on topic and sentiment classification against bag-of-n-gram baselines and prior methods. Performance depends on task and architecture: seq-CNN is stronger for sentiment, bow-CNN for topics, and parallel convolution layers improve results further.
- Experimental setup: CNN experiments covered topic classification and sentiment classification, with comparisons against SVM and fully connected neural networks using bag-of-n-gram vectors.The tasks included IMDB, Elec, and RCV1 settings.
- Overall results: On all datasets, the best-performing CNN outperformed the baseline methods.The comparison used error rates for sentiment classification on IMDB and Elec and 55-way topic categorization on RCV1.
- Task-dependent configurations: Seq-CNN outperformed bow-CNN on sentiment classification, whereas bow-CNN outperformed seq-CNN on topic categorization.The sentiment configuration used small regions and max-pooling, while the topic configuration used larger regions and average-pooling.
- Parallel CNN: Parallel CNN layers improved performance over seq-CNN on IMDB and over seq2-CNN on IMDB and Elec.The results suggest that different text-region embeddings can complement one another and that predictive region lengths may vary.
- State-of-the-art comparison: 7.67 was the best IMDB error rate, achieved by seq2-bown-CNN and improving on the previous best supervised result of 8.13 by nearly 0.5%.Seq2-bown-CNN combined two sequence-convolution layers with a document-level bow3 convolution layer.
- Embedding analysis: CNN can use unseen test n-grams when some constituent words were observed during training, unlike bag-of-n-gram representations.The convolutional embedding operates on constituent words rather than requiring the complete n-gram to have appeared in training.
4 Conclusion
The paper presents CNN as an alternative way to use word order in text categorization by directly embedding small text regions. Parallel CNNs combine complementary embeddings, achieving state-of-the-art performance on sentiment and topic classification.
- CNN directly embeds small text regions to use word order for text categorization, unlike traditional bag-of-n-gram or word-vector CNN approaches.
- Parallel CNNs learn and combine several embedding types that can complement one another for higher accuracy.
- The approach achieved state-of-the-art performance on sentiment classification and topic classification.