Source-linked AI summary
Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings
Rie Johnson, Tong Zhang
TL;DR
Text categorization needs representations that capture word order and concepts across variable-sized regions beyond traditional fixed-region approaches. The paper develops simplified one-hot LSTM region embeddings within a region-embedding-plus-pooling framework and combines them with CNN embeddings trained on unlabeled data. The combined approach obtained the best results and exceeded previous best results on four benchmark datasets.
Problem
Fixed-size CNN regions may not match variable relevant-region sizes, while large regions increase parameters; larger regions in bag-of-word variants lose word order.
Method
The paper jointly trains a linear model with region embedding and pooling, using one-hot LSTMs for variable-sized regions in supervised and semi-supervised settings.
Results
The best results came from combining LSTM and CNN region embeddings trained on unlabeled data, exceeding previous best results on four benchmark datasets.
Takeaways & Limitations
Text-region embeddings can convey higher-level concepts and were more useful than single-word embeddings in isolation for this task.
Takeaways & Limitations
The bag-of-word variation supports larger regions only by losing word order, limiting its use; tv-embedding learning also depends on task-relevant relations between its two views.
Abstract
from arXiv · showhide
One-hot CNN (convolutional neural network) has been shown to be effective for text categorization (Johnson & Zhang, 2015). We view it as a special case of a general framework which jointly trains a linear model with a non-linear feature generator consisting of `text region embedding + pooling'. Under this framework, we explore a more sophisticated region embedding method using Long Short-Term Memory (LSTM). LSTM can embed text regions of variable (and possibly large) sizes, whereas the region size needs to be fixed in a CNN. We seek effective and efficient use of LSTM for this purpose in the supervised and semi-supervised settings. The best results were obtained by combining region embeddings in the form of LSTM and convolution layers trained on unlabeled data. The results indicate that on this task, embeddings of text regions, which can convey complex concepts, are more useful than embeddings of single words in isolation. We report performances exceeding the previous best results on four benchmark datasets.
1. Introduction
Text categorization moved beyond bag-of-word linear models toward non-linear methods that use word order. This work generalizes one-hot CNNs with region embedding and pooling, then explores LSTM-based embeddings for variable-sized regions in supervised and semi-supervised settings.
- Non-linear methods that use word order became more accurate than traditional bag-of-word linear predictors for text categorization.
- One-hot CNN embeds fixed-size text regions with a shared function, but large regions increase parameters and bag-of-word variants lose word order.
- The proposed framework jointly trains a linear model with a feature generator composed of region embedding followed by pooling.
- LSTM processes sequences recurrently, allowing text regions of variable and possibly large sizes while learning dependencies over larger time lags.
- The study simplifies supervised LSTM by removing word embeddings, uses unlabeled data for region embeddings, and combines LSTM and CNN embeddings.
- The combined embeddings produced the best results, and the study reported performance exceeding previous best results on four benchmark datasets.
2. Supervised LSTM for text categorization
The paper develops one-hot LSTM region embeddings within a region-embedding-plus-pooling framework, simplifying the model for accuracy and efficiency. Supervised experiments compare the resulting bidirectional model with prior LSTM, CNN, and SVM methods across four datasets.
- 2.1. Elimination of the word embedding layer: One-hot LSTM removes the word-embedding layer by feeding one-hot word vectors directly into the recurrent model.The paper argues this preserves model behavior while avoiding extra embedding parameters and tuning.
- 2.2. More simplifications: Pooling changes the LSTM objective from representing an entire document with one vector to detecting and embedding relevant text regions.The LSTM emits vectors at each time step, which pooling aggregates into a document vector.
- 2.2. More simplifications: Chopping documents into fixed-length segments enables parallel mini-batch processing because the model no longer needs to process each document sequentially from beginning to end.This follows from targeting text-region embeddings rather than document embeddings.
- 2.2. More simplifications: Removing input and output gates nearly halves training and testing time and memory without typically improving accuracy when pooling follows the LSTM.The simplified formulation fixes the input and output gates to one.
- 2.2. More simplifications: Bidirectional LSTM concatenates forward and backward outputs to improve accuracy, producing the one-hot bidirectional model abbreviated oh-2LSTMp.Earlier simplifications make adding the backward LSTM practical.
- 2.3. Experiments (supervised): On three of four datasets, oh-2LSTMp outperforms SVM and one-hot CNN, but on RCV1 it trails both, where bag-of-words regions outperform strict word order.The paper attributes the RCV1 pattern to topic-indicative expression variability being easier to cover with word bags.
- 2.3. Experiments (supervised): LSTM supports variable and possibly large region sizes, while CNN requires fixed sizes; nevertheless, two-layer one-hot CNNs rival oh-2LSTMp and train faster.The paper suggests insufficient training data may limit the benefit of learning longer word sequences with LSTM.
- Comparison with the previous best results on 20NG: On 20NG, oh-2LSTMp achieves an error rate of 13.32, 2% better than the previous best performance of 15.3 from DL15.The previous result used a pre-trained word-vector LSTM with 1024 units.
3. Semi-supervised LSTM
The paper learns LSTM region embeddings from unlabeled text through two-view prediction and uses them as additional input to supervised models. These embeddings improve semi-supervised performance, and combining LSTM and CNN region embeddings yields complementary gains.
- Learning LSTM tv-embeddings: Two-view feature learning treats a region and its surrounding context as views, learning embeddings useful for predicting the other view and classification.The approach extends earlier nonlinear two-view learning from CNN region embeddings to one-hot LSTM region embeddings.
- Learning LSTM tv-embeddings: LSTM tv-embeddings predict the next k words from preceding words, using forward and backward one-hot LSTMs trained on unlabeled data.The target vocabulary excludes function words to reduce undesirable syntactic relations between the views.
- Using tv-embeddings: The learned tv-embeddings are fixed and supplied as additional input to supervised LSTM models rather than being fine-tuned during labeled training.The framework can naturally add multiple tv-embeddings and uses unlabeled data to provide additional input instead of pre-training.
- Semi-supervised experiments: Semi-supervised one-hot bidirectional LSTM with pooling improved clearly over its supervised counterpart on IMDB, Elec, and RCV1.The model used two LSTM tv-embeddings trained on unlabeled data and supplied them to one-hot LSTMs in both directions.
- Comparisons: Region tv-embeddings outperformed pre-trained word-vector LSTM models, while LSTM tv-embeddings rivaled or exceeded CNN tv-embeddings on IMDB and Elec but underperformed on RCV1.Increasing LSTM tv-embedding dimensionality on RCV1 improved its error rate to 8.62, still above the CNN result of 7.97.
- Combining region embeddings: Combining LSTM and CNN tv-embeddings improved performance, including IMDB error from 6.66 to 5.94 and RCV1 error from 7.71 to 7.15.The results support the paper’s expectation that the two region-embedding formulations have complementary strengths.
- Comparison with previous best results: The best models exceeded previous best results on IMDB, Elec, and RCV1, while the paper reports 5.94 error on IMDB.The previous best IMDB result cited was 6.51, which the new model improved to 5.94.
4. Conclusion
The conclusion presents one-hot LSTM as an effective region-embedding method within the region embedding plus pooling framework. It finds that region embeddings are more useful than isolated word embeddings, and that LSTM and CNN embeddings learned from unlabeled data work best together.
- Conclusion: One-hot LSTM region embeddings rivaled or outperformed state-of-the-art one-hot CNN embeddings and strongly outperformed previous LSTM methods.The paper places these results within a general framework that jointly trains a feature generator and linear model.
- Conclusion: The best results came from combining LSTM and CNN region embeddings trained on unlabeled data, suggesting complementary strengths.The paper reports substantial improvements over previous best results on benchmark datasets.
- Conclusion: Text-region embeddings can convey higher-level concepts and were more useful for this task than embeddings of single words in isolation.Useful region embeddings could be learned directly from one-hot vectors on labeled or unlabeled data.
- Conclusion: The paper identifies new region-embedding methods with complementary benefits as a promising future direction.