Source-linked AI summary
Bag of Tricks for Efficient Text Classification
Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov
TL;DR
Text classification models can perform well but are often too slow for very large datasets. This paper introduces fastText, a simple efficient classifier, and finds it often matches deep-learning methods while training and evaluating much faster.
Problem
Neural text classifiers achieve strong performance but are relatively slow to train and test, limiting their use on very large datasets.
Method
fastText uses a rank-constrained linear model with averaged word representations and a fast loss approximation for large-scale text classification.
Results
Across tag prediction and sentiment analysis, fastText performs on par with deep-learning-inspired methods while achieving speedups of at least 15,000× on larger datasets.
Takeaways & Limitations
fastText provides a simple baseline that can match recently proposed deep-learning methods while being much faster.
Abstract
from arXiv · showhide
This paper explores a simple and efficient baseline for text classification. Our experiments show that our fast text classifier fastText is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation. We can train fastText on more than one billion words in less than ten minutes using a standard multicore~CPU, and classify half a million sentences among~312K classes in less than a minute.
1 Introduction
Text classification is important across NLP applications, but neural models can be slow to train and test. This work scales linear baselines to large corpora and output spaces using fastText, training on a billion words within ten minutes while matching state-of-the-art performance.
- Text classification supports applications including web search, information retrieval, ranking, and document classification, while neural models have become increasingly popular.
- Neural models achieve very good practical performance but tend to be relatively slow at training and testing.
- Linear classifiers are strong text-classification baselines that can achieve state-of-the-art performance with suitable features and scale to very large corpora.
- A billion words can be trained within ten minutes using linear models with a rank constraint and fast loss approximation, while achieving state-of-the-art performance.
2 Model architecture
fastText represents text with averaged word or n-gram features and feeds this representation to a rank-constrained linear classifier. It improves scalability with hierarchical softmax and efficient asynchronous stochastic-gradient training.
- Model architecture: fastText embeds and averages bag-of-words features into a hidden text representation, then applies a linear classifier with a rank constraint.The first weight matrix is a word lookup table, and the averaged representation is passed to the classifier.
- Model architecture: The model uses softmax to compute class probabilities and minimizes the negative log-likelihood over labeled documents.Training uses normalized bag-of-features inputs and weight matrices A and B.
- Hierarchical softmax: O(kh) is the linear-classifier complexity for k classes and representation dimension h; hierarchical softmax reduces training complexity to O(h log2(k)).The hierarchical softmax uses a Huffman coding tree.
- Hierarchical softmax: O(h log2(k)) is also the observed test-time complexity with hierarchical softmax, while computing the T-top targets costs O(log(T)).Depth-first search prunes low-probability branches, and a binary heap extends the approach to T-top targets.
- N-gram features: Bag-of-n-gram features capture partial local word order efficiently, complementing the word-order invariance of bag-of-words representations.The n-gram mapping uses the hashing trick with 10M bins for bigrams and 100M otherwise.
3 Experiments
The experiments evaluate fastText on sentiment classification and large-output-space tag prediction, comparing accuracy, scalability, and training or inference speed against neural and frequency-based baselines. fastText remains competitive in accuracy while offering substantial efficiency advantages, including speedups that increase with dataset size and output-space cardinality.
- Experimental tasks: The experiments cover sentiment analysis and scalable tag prediction, with a tailored implementation observed to be 2-5× faster than an equivalent Vowpal Wabbit implementation.The tag-prediction evaluation targets large output spaces.
- Sentiment analysis: On sentiment datasets, adding bigrams improves performance by 1-4%; fastText is slightly better than char-CNN and char-CRNN but slightly worse than VDCNN.Using trigrams raises Sogou performance to 97.1%.
- Sentiment analysis: 97.1% is the reported Sogou performance when fastText uses trigrams instead of only the evaluated unigram-and-bigram configurations.The paper states that more n-grams can slightly increase accuracy.
- Training time: At least 15,000× speed-up is achieved over neural-network methods as dataset size increases; fastText trains in less than a minute on the sentiment datasets.Convolutional methods are several orders of magnitude slower, while GRNNs take around 12 hours per CPU epoch.
- Tag prediction: The YFCC100M tag-prediction dataset contains almost 100M images, and its training split has 91,188,648 examples and 1.5B tokens after filtering.Predictions use titles and captions, not images.
- Tag prediction: More than 300K classes make Tagspace relatively slow at inference, whereas fastText provides more than an order-of-magnitude speed-up with better quality; adding bigrams substantially boosts accuracy.The comparison includes a frequency-based baseline and linear Tagspace variants with hidden layers of 50 and 200.
4 Discussion and conclusion
The paper proposes fastText as a simple baseline whose word features can be averaged into effective sentence representations. Across several tasks, it matches recently proposed deep-learning-inspired methods while being much faster, and the authors plan to publish the code for reuse.
- Method and implications: fastText uses word features that can be averaged to form good sentence representations.This distinguishes the method from unsupervised word vectors trained with word2vec.
- Method and implications: On several tasks, fastText achieves performance on par with recently proposed deep-learning-inspired methods while being much faster.The discussion presents this as evidence for a simple and efficient baseline for text classification.
- Method and implications: The paper questions whether simple text-classification problems such as sentiment analysis are appropriate tests for deep neural networks.Although deep neural networks have theoretically greater representational power than shallow models, their suitability for these problems is unclear.
- Method and implications: The authors will publish their code so the research community can build on the work.The stated goal is to make the methods easier for researchers to reuse and extend.