Compression of Deep Learning Models for Text: A Survey

Manish Gupta; Puneet Agrawal

Source-linked AI summary

Compression of Deep Learning Models for Text: A Survey

Manish Gupta, Puneet Agrawal

arXiv:2008.05221v4Aug 12, 2020cs.CLcs.AIcs.CVcs.LG

TL;DR

Large NLP models create model-size, latency, and computational constraints for real-world applications. This survey organizes compression research into six method families and reports compression with bounded performance loss, while noting important method limitations.

Problem
Large NLP models create practical demands for smaller model size, lower response times, and lower computational power.
Method
The survey organizes NLP model compression methods into pruning, quantization, knowledge distillation, parameter sharing, tensor decomposition, and sub-quadratic Transformers.
Results
Up to 13x compression retained performance drops within 2.3% for MNLI and SQuAD and 1.1% for SST-2 and CoNLL-03.
Takeaways & Limitations
The survey presents compression methods as approaches for reducing model size and Transformer complexity while considering accuracy, latency, and deployment requirements.
Takeaways & Limitations
Binary quantization of recurrent weights in RNNs and LSTMs never worked, and practical implementation of sparse data structures is difficult.

Abstract

from arXiv · show

In recent years, the fields of natural language processing (NLP) and information retrieval (IR) have made tremendous progress thanksto deep learning models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs)networks, and Transformer [120] based models like Bidirectional Encoder Representations from Transformers (BERT) [24], GenerativePre-training Transformer (GPT-2) [94], Multi-task Deep Neural Network (MT-DNN) [73], Extra-Long Network (XLNet) [134], Text-to-text transfer transformer (T5) [95], T-NLG [98] and GShard [63]. But these models are humongous in size. On the other hand,real world applications demand small model size, low response times and low computational power wattage. In this survey, wediscuss six different types of methods (Pruning, Quantization, Knowledge Distillation, Parameter Sharing, Tensor Decomposition, andSub-quadratic Transformer based methods) for compression of such models to enable their deployment in real industry NLP projects.Given the critical need of building applications with efficient and small models, and the large amount of recently published work inthis area, we believe that this survey organizes the plethora of work done by the 'deep learning for NLP' community in the past fewyears and presents it as a coherent story.

1 MODEL COMPRESSION METHODS: OVERVIEW

The survey organizes NLP model compression around six method families that reduce model size, computational demands, or Transformer complexity. It compares design choices across pruning, quantization, distillation, parameter sharing, tensor decomposition, and sub-quadratic Transformers.

1 MODEL COMPRESSION METHODS: OVERVIEW: The survey covers pruning, quantization, knowledge distillation, parameter sharing, tensor decomposition, and sub-quadratic Transformer methods.These families are presented as the main approaches to compressing deep learning models for NLP.
1 MODEL COMPRESSION METHODS: OVERVIEW: Pruning reduces model size by removing weights or structures such as neurons, blocks, attention heads, or layers.The survey distinguishes unstructured pruning from structured pruning and discusses choices including pruning criteria, schedules, and regularization.
1 MODEL COMPRESSION METHODS: OVERVIEW: Quantization compresses models by representing weights with fewer bits, using choices such as binary, ternary, or multi-bit values.The survey considers uniform versus non-uniform, deterministic versus stochastic, loss-aware versus unaware, and trained versus tuned quantization schemes.
1 MODEL COMPRESSION METHODS: OVERVIEW: Knowledge distillation trains a shallow student to mimic a larger pretrained teacher, after which the student is deployed without the teacher.Design choices include the teacher, loss, training data, mimicked outputs or features, and whether learning uses assistants or fellow students.
1 MODEL COMPRESSION METHODS: OVERVIEW: Parameter sharing reduces overall weights by reusing parameters across multiple weight structures, while tensor decomposition factorizes large tensors into smaller components.The survey organizes these methods by what is shared, how sharing is performed, factorization type, factorized matrices, and exploited matrix properties.
1 MODEL COMPRESSION METHODS: OVERVIEW: Sub-quadratic Transformers address latency and RAM costs that grow quadratically with input length by computing token representations from a fixed small set of other tokens.The surveyed methods may have super-linear or linear complexity and differ in how they define the tokens used for each representation.

2 PRUNING

Pruning compresses NLP models by removing weights or larger structures, with methods varying in granularity, saliency criterion, schedule, and regularization. Reported results show substantial compression with limited accuracy loss, but practical gains depend on hardware support and model architecture.

Pruning removes individual weights or structured components such as neurons, matrix blocks, attention heads, and layers.
Unstructured pruning creates sparse matrices requiring specialized inference-time manipulation, whereas structured pruning avoids this handling but is less flexible.
Large-sparse models consistently outperform small-dense models while achieving up to 10x fewer non-zero parameters with minimal accuracy loss.
Unstructured Weight Pruning: Pruning criteria include Hessian-based saliency, magnitude, iterative magnitude pruning, and iterative pruning with densification.
Hessian-based Methods: Hessian-based methods can be computationally prohibitive because second-derivative calculations are expensive, while diagonal-Hessian computation matches gradient complexity.
Magnitude Pruning Methods: Class-blind magnitude pruning outperforms class-uniform and class-distribution schemes, with higher layers, attention, and softmax weights generally more important in LSTM NMT.
Iterative Magnitude Pruning Methods: Gradual pruning can reduce RNN/LSTM model size by 90% and provide roughly 2x to 7x speed-up without accuracy deterioration.
Iterative Magnitude Pruning Methods: Iterative pruning improves accuracy in some settings, but it performs poorly for Transformer models such as BERT and can interact problematically with regularization.

2.2 Pruning Neurons

Neuron pruning removes entire rows or columns of weight matrices to avoid sparse-matrix handling, but this sacrifices flexibility compared with individual-weight pruning. Neurons can be ranked using output or input weight norms, entropy, or related importance scores.

2.2 Pruning Neurons: Neuron pruning removes rows or columns from weight matrices, avoiding sparse-matrix handling but requiring entire structures to be deleted.
2.2 Pruning Neurons: Neuron pruning is less flexible than weight pruning because only complete rows or columns can be removed.
2.2.1 Neuron Importance: Neurons can be ranked by entropy, output-weight norm, or input-weight norm, then low-scoring nodes are removed.
2.2.1 Neuron Importance: Output-weight norm is reported as the best-performing importance function among the listed neuron-ranking criteria.
2.2.2 Removing Redundant Neurons: When neurons have equal weight sets, one redundant neuron can be removed by transferring its coefficient to the remaining neuron.

2.3 Pruning Blocks

Block and structured pruning reduce model size while targeting hardware-friendly structures, but sparsity granularity and accuracy must be balanced. Attention-head and layer pruning can remove substantial redundancy with limited accuracy impact.

Block pruning: Block pruning stores contiguous blocks to reduce irregular memory accesses, and group Lasso guides which blocks are removed during iterative pruning.Zeroed blocks remain zero after pruning, producing a sparse model at the end of training.
Block pruning: BBS achieves nearly the same accuracy as unstructured sparsity and significantly outperforms block sparsity at equal sparsity, while supporting FPGA acceleration.BBS balances each matrix row into equal-sized banks and prunes each bank independently.
Layer pruning: LayerDrop’s Every Other strategy works well across many tasks, whereas validation search and data-driven pruning provide only marginal gains.Sensitivity-based structured pruning can identify subnetworks comparable to the full model, while similarly sized random subnetworks perform worse.

2.5 Summary

Across tasks, pruning methods achieve substantial compression with task- and architecture-dependent accuracy tradeoffs. Results favor gradual or structured approaches selected for the target model and hardware.

Architecture comparisons: For comparable parameter counts, pruned Transformers outperform pruned LSTMs on WMT, while RPP generally outperforms iterative magnitude pruning on Transformer tasks.RPP is reported as stronger for Transformer models, whereas iterative magnitude pruning with regularization works well for RNNs and LSTMs.
Task comparisons: LayerDrop halves summarization models without noticeable accuracy change, while on SST-2 it improves over RPP using less than half the model size.On GLUE, RPP produces smaller and more accurate models than iterative magnitude pruning for CoLA, and LayerDrop improves further on SST-2.
Overall observations: Pruning can recover or improve baseline performance at high sparsity: NMT reaches 20.91 BLEU at 80% pruning versus 20.48 BLEU baseline.At 90% pruning, performance falls modestly to 20.13 BLEU.
Block pruning: Block-sparse RNNs support 80%–90% sparsity with small accuracy loss, but sparsity beyond 90% for 4×4 blocks causes at least 30% relative accuracy loss.Increasing block size worsens the accuracy tradeoff, and output-near layers tolerate less sparsity than initial layers.
Structured pruning: Attention-head pruning often removes substantial redundancy, but dropping heads alone provides limited model compression; combining head and layer removal is more effective for Transformers.Only 8 of 96 NMT heads significantly affect performance when removed individually, and half of those removals improve BLEU.
Practical considerations: Pruning effectiveness depends on implementation constraints: unstructured sparsity can prune extensively theoretically, but sparse data structures are difficult to implement efficiently.The survey also notes that pruning and regularization must be coordinated carefully.

3 QUANTIZATION

Quantization compresses neural models by reducing the bit precision used to store weights. The survey covers binary, ternary, and general quantization spanning roughly 3–32 bits.

Quantization motivation: Quantization reduces the number of bits needed to store weights, replacing the common 32-bit representation with lower-precision alternatives.The survey motivates average weight storage of roughly 4–7 bits from biological precision estimates.
Quantization types: The survey covers 1-bit binary quantization, ternary quantization, and general quantization across the spectrum from 3 to 32 bits.Figure 3 distinguishes binary, ternary, and general quantized weight distributions by weight value and frequency.

3.1 Binarized Networks

Binarized networks represent weights with one bit to reduce memory and accelerate computation, but naïve binarization can substantially harm accuracy. More flexible schemes learn scaling or quantization parameters to improve the approximation.

Binary quantization: 1-bit quantization provides 32× compression and can accelerate inference, but it causes significant accuracy loss across many tasks unless applied selectively.Hybrid schemes can use binary quantization for selected layers, while bitwise operations improve power efficiency.
Binarization methods: Deterministic and stochastic binarization use different training and inference procedures, with the Straight Through Estimator enabling quantization within back-propagation.Stochastic training binarizes weights during propagation but retains higher-precision weights for parameter updates.
Accuracy and size tradeoffs: Using original full-precision weights at test time preserves accuracy but does not reduce model size, limiting the compression benefit.Naïve static binarization instead reduces storage but can lead to large accuracy loss.
Binary Weight Networks: Binary weight networks approximate W with a scaled binary vector, W ≈ αB, and learn a scaling parameter alongside the binarized matrix.The binary vector uses values +1 and −1, with signs assigned according to the original weights.
Loss-aware binarization: Loss-aware binarization formulates weight quantization as an optimization problem that identifies suitable scaling factors and binary vectors for the network loss.The formulation uses layer-specific weight counts and can be solved with a proximal Newton algorithm.

3.2 Ternarized Networks

Ternary quantization replaces weights with three values, avoiding the severe gradient problems of binary recurrent-weight quantization while substantially reducing memory.

Binary quantization fails for recurrent weights because mapping near-zero values to -1 or 1 increases weight magnitudes and worsens vanishing or exploding gradients.Ternary quantization is presented as an alternative for RNNs and LSTMs.
Ternary quantization can provide 16x compression, or up to 32x when hardware avoids storing zeros.
Ternary connect assigns weights to three values and eliminates all multiplications in the forward pass.Variants include stochastic, Bernoulli, Gaussian-based, learned-boundary, and loss-aware ternary quantization.
Loss-aware ternary quantization greedily selects layerwise step sizes by measuring output error after quantizing the network progressively.The initial step size can be determined with the Lloyd-Max algorithm.
Hybrid ternary quantization saves 1.4x memory for weights during training and up to 16x when both weights and activations are quantized.

3.3 General Quantized Networks

General quantized networks use uniform, non-uniform, loss-aware, and Transformer-specific methods to reduce precision while addressing distributional and architectural constraints.

General quantization methods include uniform equal-width binning, non-uniform equal-frequency methods, loss-aware schemes, and Transformer-specific approaches.
Uniform quantization is easy to implement but far from optimum, and recurrent architectures require extra handling for multiplication bit-widths and differing gate ranges.
Non-uniform methods such as histogram equalization, exponential quantization, and KMeans adapt quantization to weight distributions or shared cluster values.Exponential quantization stores weights at integer powers of two and can eliminate multiplications.
KMeans-based methods share centroids among connections, while product and residual quantization partition vectors or recursively quantize residuals.Residual quantization may require substantial memory for codebooks at each iteration.
Transformer quantization uses mixed precision across layers, attention heads, or neuron groups; 8-bit embeddings with 2–4-bit weight matrices can match full-precision BERT results.Quantization during training performs better than post-training quantization in the cited comparison.

3.4 Summary

Across recurrent, convolutional, embedding, and Transformer models, quantization generally outperforms binarization and preserves accuracy with substantial compression, especially through loss-aware and mixed-precision designs.

Quantization generally yields substantial size reduction without accuracy loss across tasks, whereas binary quantization alone performs poorly for text-based neural models.
On PTB, 3-bit Alternating LAQ provides approximately 10.5x memory savings and 3x real inference acceleration, with a 2.1-point perplexity-per-word gap at 2/3 bits.
Up to 13x weight compression keeps BERT performance drops within 2.3% on MNLI and SQuAD and 1.1% on SST-2 and CoNLL-03 using mixed precision.The cited method uses 2/3-bit weights with 8-bit embeddings and adds 5MB of storage.
Transformer models can match full-precision performance with 4-bit representations, while mixed-precision Q-BERT outperforms QBERT across several NLP tasks.Attention distributions in 4-bit Transformer models remain highly similar to the original model.
Non-uniform, loss-aware, and mixed-precision methods outperform simpler alternatives, while 3-bit execution may require casting to 4 or 8 bits in hardware.

4 KNOWLEDGE DISTILLATION (KD)

Knowledge distillation compresses a deep teacher into a shallow deployable student by transferring logits, soft outputs, or other teacher-derived information through specialized losses.

Knowledge distillation trains a shallow student to mimic a pretrained deep teacher, after which the student model is deployed.
Distillation can transfer teacher logits, softmax outputs, noisy logits, or information from ensembles, teacher assistants, and fellow students.
Logit-based students can be trained against teacher outputs without original labels, using an L2 objective across training instances.
Distilled students are more accurate than equally shallow students trained directly on labels because teacher outputs can filter noisy labels and convey uncertainty.
Hinton-style distillation combines soft loss between teacher and student outputs with hard loss against the original labels, usually weighting soft loss more heavily.Temperature scaling makes teacher outputs less peaked so they convey more information.

T LSTM (S)

Knowledge distillation compresses NLP models by training smaller students to reproduce information from larger teachers, with variants transferring outputs, intermediate representations, derivatives, or layer-to-layer feature flow. Across tasks, distilled models often achieve substantial reductions in size and latency while retaining competitive performance.

4 KNOWLEDGE DISTILLATION: Knowledge distillation trains a shallow student to mimic a pretrained deep teacher, then deploys the student instead of the teacher.The transferred information can include logits, softmax outputs, intermediate representations, derivatives, or feature flow between layers.
4.2 Collaborative Learning: Collaborative learning shows that simple students can learn from one another without a powerful static teacher and can outperform teacher-based distillation.The survey reports little to no adverse effect on final model quality from co-distillation.
4.5 Summary: 30x smaller and 57x faster models retain over 95% of BERT-base’s slot-filling F1, while remaining task-agnostic.This result comes from mixed-vocabulary training on a 6-layer model without quantization.
4.5 Summary: 7.5x smaller and 9.4x faster 4-layer TinyBERT maintains competitive performance against BERT-base, while 6-layer TinyBERT achieves comparable results.TinyBERT also outperforms several 4-layer and 6-layer baselines, including PKD, DistilBERT, and MiniLM.
4.5 Summary: 2.0x faster MiniLM retains more than 99% performance across tasks, while distilled BiLSTM uses 349 times fewer parameters and is 434 times faster than BERT-large.Other reported systems include MobileBERT, which is 4.3x smaller and 5.5x faster than BERT-base with competitive benchmark scores.
4.5 Summary: Knowledge distillation is especially effective when abundant unlabeled data can be pseudo-labeled by a teacher.The survey also reports gains from combining distillation with quantization and pruning.

5 PARAMETER SHARING

Parameter sharing reduces model size by reusing weight values or structured factors across connections, embeddings, layers, or Transformer components. Character-based representations and shared-parameter architectures achieve large compression ratios while often preserving or improving task performance.

5 PARAMETER SHARING: Parameter sharing compresses models by finding weight blocks that reuse the same parameters, including shared embeddings and Transformer-layer structures.The section covers character-based language models, hashing, Toeplitz-like matrices, low-rank factors, and other sharing mechanisms.
5.1 Character-aware Language Models: Character-composed word representations require one vector per character type and a fixed compositional model instead of independent vectors for every word.C2W representations are much smaller than word-embedding matrices while achieving comparable results across multiple text tasks.
5.4 Summary: 18x fewer parameters and 1.7x faster training are achieved by an ALBERT configuration similar to BERT-large.Parameter sharing in Transformers is also reported to support strong GLUE performance and regularization.
5.4 Summary: 40–100x smaller LightRNN models reduce training time by 2x and achieve lower perplexity than word-LSTMs on the One-Billion-Word benchmark.Slim embeddings can preserve word-LSTM perplexity using 1% or 0.2% of trainable parameters on PTB and the 44M GigaWord corpus.
5.4 Summary: C2W consistently outperforms word lookup tables for POS tagging, while Universal Transformers are one-sixth the size and outperform standard Transformers on several tasks.These results illustrate that shared or compositional representations can improve performance rather than merely reduce storage.

6 TENSOR DECOMPOSITION

Tensor decomposition compresses NLP models by factorizing large matrices and tensors into smaller components, often combined with pruning, quantization, or structured embeddings. The surveyed methods achieve substantial reductions in model size with small or sometimes improved losses across language modeling, translation, embeddings, speech, and recommendation tasks.

6 TENSOR DECOMPOSITION: Tensor decomposition replaces large matrices or tensors with smaller factors, including SVD, Tensor-Train, CP, Tucker, sparse coding, and block-term decompositions.Tensor-Train decomposition reconstructs high-dimensional tensors from multiple 2D and 3D cores governed by TT-ranks.
6.4 Summary: 4193x embedding compression is achieved with TT-embeddings at TT-rank=2, producing a 0.53M model with test loss 0.4530 for CTR prediction.The survey also reports compression ratios up to 2011 times with slight test-loss improvement in related embedding experiments.
6.4 Summary: An 8M FLOP model reaches 1.13 on Enwiki8 versus 1.14 for gradual pruning, while FLOP also outperforms other pruning methods on WikiText-103.On WikiText-103, the FLOP-pruned model reaches 25.3 perplexity at 50M parameters, compared with 25.7 for gradual unstructured pruning and 26.7 for neuron pruning.
6.4 Summary: BTD uses half the Transformer-XL-Base size with approximately 10% lower perplexity on PTB and reduces Transformer size by more than half with improved BLEU on WMT16.For language modeling, BTD also produces a model with 20% of the Transformer-XL large size.
6.4 Summary: 98% compression without performance loss is achieved by compositional codes on IMDB, while TT-embeddings yield better accuracy with 441x embedding compression at TT-rank=16.Compositional codes also reach loss-free compression rates of 94% on IWSLT14 and 99% on ASPEC.
6.3 Decomposition for RNNs: 11x compression of a 3-layer LSTM is achieved without performance loss by combining WEST matrix decomposition with quantization.WEST alone produces a 3x smaller model with slightly reduced word error rate on live traffic utterances.

7 TRANSFORMERS WITH SUB-QUADRATIC COMPLEXITY

Transformer compression methods reduce quadratic attention costs through sparse connectivity, hashing, low-rank projections, and structured attention patterns. These approaches can enable longer contexts while preserving or improving performance across language, understanding, and speech tasks.

Motivation: O(n^2) time and activation memory growth limits Transformer input lengths and prevents semantic representation of long documents.The quadratic cost arises because each position attends to every other position in every layer.
Sparse attention: O(n√n) sparse Transformers reduce computation by evaluating attention only on selected query-key pairs using strided or fixed sparse factorizations.The sparse patterns reduce the dense attention matrix to a sparse version.
Hashing-based attention: Reformer changes attention complexity from O(n^2) to O(n log n) by using locality-sensitive hashing to restrict attention within hash buckets.Queries focus on nearby keys after positions are clustered into groups.
Linear-complexity methods: O(n) attention variants use different mechanisms, including low-rank projections in Linformer, differentiable sorting in Sinkhorn attention, and sparse global attention in Longformer.Linformer reduces self-attention complexity in both time and space, while SortCut reaches O(nk).
Empirical findings: Longformer consistently outperforms RoBERTa on downstream long-document tasks, with larger gains on tasks requiring long context.Its improvement is more modest when local context is usually sufficient, such as TriviaQA.
Empirical findings: Linear Transformers achieve similar speech-recognition performance to vanilla Transformers while running up to 4000x faster on very long autoregressive sequences.They also outperform LSTM and Reformer and train faster than Reformer.

8 SUMMARY AND FUTURE DIRECTIONS

The survey compares six model-compression families, finding that they can reduce model size, latency, and power consumption across text tasks. It identifies strong task- and architecture-dependent trade-offs while highlighting unresolved challenges in real-world compression.

Summary: The survey covers pruning, quantization, knowledge distillation, parameter sharing, tensor decomposition, and sub-quadratic Transformer methods.These methods are reviewed across model-size, accuracy, latency, and application settings.
Summary: Compression methods reduce model size while also lowering prediction latency and power consumption through reduced computation.The survey emphasizes deployment-oriented efficiency rather than size reduction alone.
Method comparisons: Knowledge distillation is the most popular Transformer-compression method, while parameter sharing often needs to be combined with other techniques.Quantization is effective when mixed precision is balanced with hardware that supports efficient low-bit computation.
Cross-method comparison: Distillation combined with quantization provides better GLUE accuracy-versus-size trade-offs than other methods, although the top three initial models report results on only four of nine tasks.The survey therefore recomputes the comparison using methods evaluated on at least five tasks.
Task-specific comparisons: For PTB language modeling, balanced sparsity works best with LSTMs, whereas BTD performs best with Transformers and reaches 0.16B parameters with perplexity 19.5 on the 1B Word Benchmark.The strongest method depends on the model family and task.
Future directions: Future work must address complex real-world settings, method selection across model structures, interpretability, task-specific compression, and limited evidence for combining methods.Latency measurements also vary with GPU architecture.