Source-linked AI summary
FastText.zip: Compressing text classification models
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, Tomas Mikolov
TL;DR
Text classification models must fit within limited memory without losing substantial accuracy. The paper combines product quantization with pruning, hashing, quantization, and re-training to compress fastText models, typically using much less memory while retaining similar accuracy. Experiments on multiple benchmarks show strong memory–accuracy trade-offs and comparisons with other classifiers.
Problem
Text classification requires compact models that fit within limited memory while preserving classification accuracy.
Method
The approach combines product quantization of embeddings with discriminative pruning, hashing, quantization, and re-training to reduce fastText model size.
Results
Models often use less than 100kB, with compression reaching ×1,000 with little performance drop and no additional test-time overhead.
Takeaways & Limitations
The compressed models reduce memory complexity by several orders of magnitude without sacrificing accuracy or speed, and can compare favorably with recent convolutional neural networks.
Takeaways & Limitations
Aggressive feature pruning can exclude all retained features from some documents, causing a significant performance drop unless the retained features cover the training set.
Abstract
from arXiv · showhide
We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutions inspired by the hashing literature, we propose a method built upon product quantization to store word embeddings. While the original technique leads to a loss in accuracy, we adapt this method to circumvent quantization artefacts. Our experiments carried out on several benchmarks show that our approach typically requires two orders of magnitude less memory than fastText while being only slightly inferior with respect to accuracy. As a result, it outperforms the state of the art by a good margin in terms of the compromise between memory usage and accuracy.
1 INTRODUCTION
Text classification is important across NLP applications, but neural networks do not consistently surpass prior methods and can be costly for memory-limited deployment. The paper targets this accuracy–model-size trade-off with compact fastText-based models.
- Text classification supports applications including spam filtering, search, ranking, and document categorization.
- Neural networks achieve state-of-the-art results in several NLP applications but do not convincingly beat prior techniques on some text classification problems.
- Compact models matter for applications running on limited-memory systems such as smartphones.
- Feature pruning, quantization, hashing, and re-training produce models often smaller than 100kB without noticeably sacrificing accuracy or speed.
- The paper extends fastText and plans to release reproducible code and scripts for evaluating the model-size–accuracy compromise.
2 RELATED WORK
The related work spans scalable linear text classifiers, language-model compression, vector hashing and quantization, softmax approximation, and neural-network compression. The paper focuses on reducing fastText model size while preserving comparable classification efficiency.
- Linear classifiers can achieve state-of-the-art text-classification performance and scalability but often require memory-intensive word and n-gram embeddings.
- Language-model compression commonly combines feature pruning, quantization, and efficient storage of n-grams using structures such as trees or hashes.
- Locality-sensitive hashing binarizes vectors to approximate cosine similarity through Hamming distance, while later methods improve the binarization process.
- Product Quantization provides memory–distance trade-offs by comparing quantized vector approximations and bounds Euclidean-distance error through quantization error.
- Softmax approximation requires approximating maximum inner products rather than only Euclidean distances or cosine similarities.
- Neural-network compression methods include vector quantization and binarization, motivated partly by evidence that classification models are overparameterized.
- Because fastText is already efficient, this work prioritizes reducing model size while maintaining comparable classification efficiency.
3 PROPOSED APPROACH
The approach reduces text-classification memory through product quantization, retraining, vocabulary reduction, and hashing while preserving accuracy. It also addresses quantization artefacts caused by widely varying vector norms.
- 3.1 TEXT CLASSIFICATION: Large matrices A and B can require gigabytes of memory when the vocabulary and output space are large.The classifier represents documents with bags of one-hot vectors and labels, making memory reduction necessary.
- 3.2 BOTTOM-UP PRODUCT QUANTIZATION: Product quantization splits vectors into subvectors and quantizes each using a distinct subquantizer, enabling compact reconstruction from quantization codes.PQ uses orthogonal subspaces, while OPQ additionally learns a rotation to reduce dependence on the original coordinate system.
- 3.2 BOTTOM-UP PRODUCT QUANTIZATION: A 1000 ratio between maximum and minimum vector norms makes ordinary k-means perform poorly, so the method quantizes vector norm and angle separately.This avoids the tendency to map low-norm vectors to zero, with an extra b bits required per vector.
- 3.2 BOTTOM-UP PRODUCT QUANTIZATION: Retraining layers after quantization lets the model readjust to quantization, motivating a bottom-up strategy that freezes the quantized input before retraining and quantizing the output.The experiments report that this strategy is worth adopting.
- 3.2 BOTTOM-UP PRODUCT QUANTIZATION: A 10× compression factor is achieved with no noticeable performance loss using bottom-up PQ, whereas omitting retraining reduces accuracy by 0.1%–0.5%.The accuracy drop depends on the dataset and setting.
- 3.3 FURTHER TEXT SPECIFIC TRICKS: Vocabulary reduction can shrink memory by up to ×1000 when combined with PQ, while retaining features that cover the training set avoids severe performance drops.The selection problem is related to NP-hard set covering, and the implementation uses an online parallelizable greedy approach.
4 EXPERIMENTS
Experiments evaluate the compression pipeline across multiple text-classification benchmarks, comparing quantization, pruning, hashing, and output encoding against compact and full-model baselines. The results show substantial memory reductions with limited performance loss, including under extreme compression and on FlickrTag.
- Evaluation protocol: The evaluation uses eight Zhang et al. (2015) datasets plus FlickrTag, comparing LSH, PQ, OPQ, NPQ, pruning strategies, and full fastText.The standard pipeline trains fastText with default settings and evaluates compact representations under controlled embedding dimensions.
- Compression techniques: Normalized PQ and OPQ are almost lossless with k = 4 subquantizers per embedding, while normalized PQ is retained because its difference from OPQ is limited and it trains faster.The study generally uses k = d/2, corresponding to half of the embedding components.
- Pruning: ×10 compression from NPQ increases to as much as ×1,000 with aggressive pruning, with little performance loss and faster test-time inference from the smaller dictionary.NPQ is reported as already matching CNN memory usage despite the larger default fastText model.
- Extreme compression: At 64KiB and 32KiB, performance drops only around 0.8% and 1.7%, respectively, despite compression rates of ×1,000−4,000.These models combine k = 1 quantization, hashing, and extreme pruning.
- Output encoding: Quantizing both input matrix A and output matrix B does not degrade performance compared to the full model, achieving a compression factor of 8 with d = 256 and k = d/2.Without retraining the output matrix, performance remains only 0.2% away from the full model.
- Pruning: On FlickrTag, max-coverage pruning avoids test-set coverage loss, whereas naive pruning with 1M features misses about 30−40% of the test set.The full model itself misses 11.6% because of missing words, and overly aggressive pruning substantially reduces coverage.
5 FUTURE WORK
Future work proposes adapting representation size to feature frequency and norm, combining pruning criteria, and decomposing nondiscriminative words into character trigrams. These directions target further compression and reduced coverage problems, especially for short inputs.
- Adaptive vector sizes: Future compression could condition vector sizes for input features and labels on frequency and norm, particularly for rare labels.The paper notes that rare FlickrTag labels may not warrant full 256-dimensional vectors.
- Combined pruning: Combining entropy and norm pruning could retain features that are both frequent and discriminative, reducing the observed coverage problem.The proposal replaces selection based solely on frequency or norm with a joint criterion.
- Feature decomposition: Nondiscriminative words could be decomposed into character trigrams instead of pruned, helping when training and test examples are very short.The example splits each such word into a sequence of character trigrams.
6 CONCLUSION
The paper reduces the memory complexity of text classifiers by several orders of magnitude while preserving accuracy and speed, using pruning, quantization, and hashing. Compared with recent convolutional models, fastText.zip is often more accurate, trains much faster on common CPUs, and uses less memory.
- Several orders of magnitude reduction in memory complexity is achieved without sacrificing accuracy or speed.The approach applies discriminative pruning to retain important features, quantizes weight matrices, and hashes the dictionary.
- Discriminative pruning keeps important trained features, while quantization compresses weight matrices and hashing compresses the dictionary.
- Compared with recent convolutional neural networks, fastText.zip is often more accurate while requiring much less training time and memory.The comparison concerns common CPUs and memory complexity at test time.
APPENDIX
The appendix reports additional experiments covering quantization, pruning, comparisons with CNNs, hashing versus Bloom filters, and retraining on a large dataset. These results are organized across Tables 5, 6, 7, 8, and 9.
- The appendix compares LSH, PQ, and OPQ across 8 different datasets.
- Table 5 compares standard quantization methods for an original model with dimensionality 8 and 2M buckets, excluding the dictionary.
- Table 6 compares different quantization methods and pruning levels, using “co” as the pruning cut-off parameter.
- Table 7 compares CNNs with fastText models with and without quantization, reporting model size under stated storage assumptions.CNN sizes assume float32 storage, while fastText(+PQ) memory is measured in RAM at test time.