Source-linked AI summary
Online Embedding Compression for Text Classification using Low Rank Matrix Factorization
Anish Acharya, Rahul Goel, Angeliki Metallinou, Inderjit Dhillon
TL;DR
Deep NLP models face deployment memory constraints, and existing compression methods can lose accuracy or add latency. This paper compresses word embeddings through low-rank factorization followed by retraining, reporting up to 90% model-size compression with minimal accuracy degradation on examined text-classification tasks.
Problem
Word embeddings often account for most NLP model size, creating a memory burden for deployment; existing compression approaches can be lossy or introduce latency.
Source: §1 · p. 1Method
Results
Up to 90% model-size compression was achieved for LSTM and DAN models, with 1%-2% relative accuracy degradation versus uncompressed models, while outperforming fixed-point quantization and offline embedding compression on examined classification problems.
Source: §6 · p. 7Takeaways & Limitations
The compression proportion can be selected to suit downstream accuracy-memory trade-offs, and the paper analyzes conditions under which inference can be faster than quantization.
Source: §6 · p. 7Fewer FLOPs alone do not guarantee faster inference than quantization; the speed advantage depends on the per-FLOP time condition and stated large-dimension assumptions.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank matrix factorization during training,to compress the word embedding layer which represents the size bottleneck for most NLP models. Our models are trained, compressed and then further re-trained on the downstream task to recover accuracy while maintaining the reduced size. Empirically, we show that the proposed method can achieve 90% compression with minimal impact in accuracy for sentence classification tasks, and outperforms alternative methods like fixed-point quantization or offline word embedding compression. We also analyze the inference time and storage space for our method through FLOP calculations, showing that we can compress DNN models by a configurable ratio and regain accuracy loss without introducing additional latency compared to fixed point quantization. Finally, we introduce a novel learning rate schedule, the Cyclically Annealed Learning Rate (CALR), which we empirically demonstrate to outperform other popular adaptive learning rate algorithms on a sentence classification benchmark.
1 Introduction
Deep NLP models face deployment memory constraints, and word embeddings often account for most of their size. The paper compresses embeddings through low-rank factorization and retraining, and introduces CALR for sentence classification.
1 Introduction
- Deep models pose memory and computing constraints for deployment, including on mobile devices and servers handling millions of requests daily.The passage cites mobile CPU speed, memory, and battery life as constraints and notes compression can save server memory and inference costs.Source: §1 · p. 1
- Word embeddings can dominate NLP model storage: 300-dimensional vectors for 100K words require 60M parameters, and embeddings account for 98.8% of a simple sentiment model.The passage notes practical vocabularies can reach one million tokens, corresponding to up to 2Gbs of memory.Source: §1 · p. 1
- The proposed method compresses word embeddings through low-rank matrix factorization and further training, with configurable compression and reported advantages over quantization and offline embedding compression.The paper also analyzes inference time and claims no additional latency compared with quantization methods.Source: §1 · pp. 1–2
- CALR is introduced as a learning-rate schedule that outperforms popular adaptive algorithms on the SST2 sentence-classification benchmark for DAN and LSTM models.The paper says its uncompressed-model experiments also exceed reported state-of-the-art accuracy for both models.Source: §1 · p. 2
2 Related Work
Prior neural-model compression includes low-precision computation, quantization, pruning, and sparse encoding, while low-rank factorization has seen comparatively limited use in deep neural models. For NLP, the paper focuses on embedding matrices because they account for much of model size.
2 Related Work
- Prior compression approaches include low-precision computation, quantization, pruning, and sparse encoding; the passage notes potential quantization loss and tuning latency for some methods.It describes quantization loss as difficult to recover through further training and says some compression-aware approaches add latency through slow training and tuning.Source: §2 · p. 2
- Low-rank factorization has had relatively limited use in deep neural models, and reducing internal layer matrices may not shrink NLP models much because embeddings contain most parameters.The passage contrasts neural speech-model work with the size distribution of typical NLP models.Source: §2 · p. 2
- The paper's rationale is that overcomplete input embeddings need not use a unique basis combination, allowing a lower-dimensional parameter space with minimal information loss.This motivates applying low-rank factorization to the word-embedding matrix as a size bottleneck.Source: §2 · p. 2
3 Methodology
The method compresses neural matrices by replacing them with lower-rank factors, and applies this decomposition online to word embeddings so training can fine-tune the reduced representation. It also introduces CALR, a cyclic learning-rate policy, and compares against fixed-point quantization and offline embedding compression.
3.1 Low-rank Matrix Factorization (LMF) for Compressing Neural Models
- Choosing the k largest singular values gives a lower-rank factorization, replacing one neural-network matrix with two factors and allowing the compression fraction to be selected.The factors reduce the parameter count from m × n to k × (m + n).Source: §3.1 · pp. 2–3, Figure 1 · p. 3
Compressing the Word Embedding Matrix
- The embedding matrix is decomposed during training, with one factor becoming the embedding layer and the other the next layer; backpropagation fine-tunes the reduced space and can regain accuracy within a few epochs.Source: Compressing the Word Embedding Matrix · p. 3
3.2 Cyclically Annealed Learning Rate
- CALR varies the learning rate in triangular windows, exponentially decays the upper bound, and restores it when the rate reaches the lower bound.The passage motivates the restarts as a way to explore nearby regions of the optimization landscape.Source: Figure 2b · p. 3, §3.2 · p. 3
3.3 Baselines
- The experiments compare the proposed approach with post-training fixed-point quantization and offline embedding compression using SVD as a preprocessing step.Source: §3.3 · p. 3, Fixed Point Weight Quantization… · p. 4, Offline Embedding Compression… · p. 4
4 Analysis
The analysis compares low-rank factorization with lower-precision quantization for storage and inference time. It derives conditions under which factorization can offer faster inference, while emphasizing that fewer FLOPs alone do not guarantee lower latency.
4 Analysis
- The comparison models a one-layer network with input X and weight matrix W, then represents the factorized approach as two successive matrix multiplications using Wa and Wb.The factor matrices have dimensions m×k and k×n, respectively.Source: §4 · p. 4
4.1 Space Complexity Analysis
- Quantization reduces storage by using BQ rather than BS bits per weight, whereas factorization stores two reduced matrices, with its space reduction depending on k.The factorized matrices Wa and Wb replace W; the analysis compares their storage with lower-precision storage without restructuring the network.Source: §4.1 · p. 4, §4 · p. 4
4.2 Time Complexity Analysis
- Fewer FLOPs do not guarantee faster inference: quantized operations may take less time per FLOP, so latency superiority requires a separate time-weighted condition.The analysis expresses this comparison using FLOP counts and the time per FLOP for each method.Source: §4.2 · p. 4
- Under the assumed relationship between time per FLOP and stored weight bits, p < BQ/BS ensures the derived condition for faster inference; practical time savings are usually sublinear to space savings.The paper states that this condition makes its derived inequality hold and that a separate condition is necessary and sufficient for faster inference than fixed-point quantization.Source: §4.2 · p. 4
5 Experiments and Results
Experiments compare embedding compression and learning-rate schedules on SST2 and a proprietary Books Intent dataset, using LSTM and DAN-RAND classifiers. The reported results cover accuracy, model size, retraining behavior, and inference time.
5.1 Datasets:
- Experiments use SST2 and a proprietary Books Intent dataset; the latter contains about 60,000 utterances labeled across 20 intents, split into 50,000 training, 5,000 test, and 5,000 development examples.SST2 is a movie-review sentiment benchmark with standard train, development, and test splits.Source: §5.1 · p. 5
5.2 Models:
- The comparison uses LSTM models initialized with 300-dimensional GloVe embeddings and DAN-RAND models with randomly initialized 300-dimensional embeddings.For both model types, the embedding matrix is the compressed input matrix.Source: §5.2 · p. 5
5.3 Experimental Setup
- Experiments use SGD with CALR, a 0.001 upper-bound initial learning rate, 0.4 dropout, and L2 regularization weight 0.005; outcomes include accuracy and model size.Model size is calculated from the TensorFlow .data, .index, and .meta files.Source: §5.3 · p. 5
5.4 Results on Uncompressed Models
- On SST2, CALR reaches 86.03% LSTM accuracy versus 84.9% prior reported and 84.61% DAN-RAND accuracy versus 83.2% prior reported; it also outperforms CLR.CLR achieves performance similar to previously reported state-of-the-art results for both models.Source: §5.4 · p. 5, Table 1 · p. 5
5.5 Results on SST2
- At 90% compression on SST2, the proposed method reaches 85.11 LSTM accuracy (1% relative degradation) and 83.11 DAN-RAND accuracy (1.7% relative degradation), outperforming the reported compression baselines.For LSTM at 90%, offline embedding compression reaches 82.54 accuracy; at 50%, the proposed method reaches 85.67 versus 84.24 for offline compression and 85.08 for 16-bit quantization.Source: §5.5 · pp. 5–6, Table 2 · p. 6
5.6 Results on Books Intent Dataset
- On Books Intent, the proposed method has better accuracy than offline embedding compression and quantization across compression percentages for both models; at 90%, LSTM accuracy is 90.94%.At 90% compression, offline embedding compression reaches 89.83 LSTM accuracy; retraining plots show similar trends to SST2, though the plots are omitted.Source: Table 3 · p. 6, §5.6 · pp. 6–7
5.5 Results on SST2
- After SVD compression, retraining raises dev accuracy from below the uncompressed and quantized benchmarks to above quantization and comparable to the uncompressed model.The authors attribute the recovery to fine-tuning embedding-layer factors Wa and Wb toward the target task.Source: §5.5 · p. 6
5.7 Results on Inference Time
- At similar compression rates, the proposed method and fixed-point quantization have similar inference times, without significant latency while regaining accuracy.The compared rates are 16-bit quantization versus 50% compression and 8-bit quantization versus 75% compression.Source: Table 4 · p. 7, §5.7 · p. 7
6 Conclusions and Future Work
The authors report substantial compression with minimal accuracy degradation for the classification problems examined, alongside outperformance of common compression baselines. They also report conditional inference-time gains and better CALR performance on SST2.
6 Conclusions and Future Work
- Up to 90% model-size compression with 1%-2% relative accuracy degradation was achieved for LSTM and DAN, outperforming fixed-point quantization and offline word embedding compression.The comparison is against uncompressed models and is reported for the text-classification tasks examined.Source: §6 · p. 7
- Inference can be faster than model quantization under specified conditions, although the conclusion does not claim faster inference in every setting.The paper analyzes the conditions under which its method can achieve this inference-time advantage.Source: §6 · p. 7
- CALR leads to better performance than other popular adaptive learning-rate algorithms on the SST2 benchmark.Source: §6 · p. 7
- Future evaluation is planned on a broader range of NLP tasks, including sequence tagging and sequence-to-sequence modeling.Source: §6 · p. 7