PaperlayerView exact revision on arXiv

Source-linked AI summary

Compression of Generative Pre-trained Language Models via Quantization

Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, Ngai Wong

arXiv:2203.10705v2cs.CLcs.CV

TL;DR

Generative PLMs are costly, and their compression remains less explored than BERT compression, with the underlying difficulty unclear. The paper quantizes these models using token-level contrastive distillation and module-wise dynamic scaling. Across language modeling, next utterance prediction, and summarization, its quantized models show comparable performance to full precision at 8/4-bit weights and outperform earlier GPT compression methods on language modeling.

  • Problem

    Generative PLMs are costly in computation and memory, while compression research has focused mostly on BERT and the difficulty of compressing generative models remains unclear.

    Source: §1 · p. 1

  • Method

    The method combines token-level contrastive distillation for distinguishable token embeddings with module-wise dynamic scaling that adapts quantizers to different modules.

    Source: §1 · pp. 1–2

  • Results

    Across language modeling, next utterance prediction, and summarization, QuantGPT and QuantBART are comparable to full precision at 8/4-bit weights, and QuantGPT outperforms prior GPT compression methods on language modeling.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The experiments demonstrate the efficacy of quantization for compressing generative PLMs across language modeling, next utterance prediction, and abstractive summarization.

    Source: §7 · p. 10

    The study does not empirically verify real speedups from low-bit operators across hardware platforms.

    Source: §8 · p. 10

Abstract

from arXiv · show

The increasing size of generative Pre-trained Language Models (PLMs) has greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rates on GPT-2 and BART, respectively.

1 Introduction

Generative PLMs are costly to run, while prior compression work has focused mainly on BERT and the difficulty of compressing generative models remains unclear. This paper studies quantization and reports comparable full-precision performance at 8/4-bit weights alongside gains over earlier GPT compression methods.

1 Introduction

  • Generative PLMs are costly in computation and memory, and prior compression methods have focused mostly on BERT while GPT-2 compression achieves much smaller ratios.The underlying difficulty of compressing generative PLMs remains unclear.Source: §1 · p. 1
  • Quantization makes generative PLMs difficult to compress because reduced capacity yields homogeneous word embeddings and weight distributions vary across modules and layers.Sequential left-to-right prediction further magnifies quantization error as it accumulates over time.Source: §1 · p. 1
  • QuantGPT and QuantBART achieve comparable performance to full-precision models at 8/4-bit weights, with only a slight drop at 2-bit and over 13× smaller models.QuantGPT also clearly outperforms previous GPT compression methods on language modeling.Source: §1 · pp. 1–2
  • The method combines token-level contrastive distillation to make embeddings distinguishable with module-wise dynamic scaling to adapt quantizers to different modules.The paper reports efficacy across various tasks.Source: §1 · pp. 1–2

2 Difficulty of Qunatizing Generative Pre-trained Language Models

Conventional quantization methods lose performance on generative PLMs as weight bit-width decreases, while learned embeddings become less distinguishable and weight distributions complicate clipping. The section introduces quantization-aware training and analyzes these challenges.

2.1 Network Quantization

  • Quantization-aware training clips full-precision weights before mapping them to low-bit values, then uses the quantized weights for loss computation and a straight-through estimator for updates.The setup uses layer-wise weight quantization and rowwise embedding quantization, while leaving layer normalization, skip connections, and biases unquantized.Source: §2.1 · p. 2

2.2 Difficulty Analysis

  • Performance drops sharply as weight bit-width decreases, especially for PACT and LSQ when applied to GPT-2.The comparison covers LAQ, PACT, and LSQ.Source: §2.2 · p. 2
  • In visualizations of the 500 most frequent embeddings, PACT, LSQ, and LAQ produce more homogeneous, less distinguishable embeddings than the full-precision model.Greater embedding homogeneity is associated with fewer token dependencies being preserved.Source: §2.2 · pp. 2–3, Figure 2 · p. 3
  • Full-precision GPT-2 weight distributions are highly skewed with outliers, making clipping-factor estimation difficult for heuristic methods and PACT.PACT’s approximate clipping-factor gradient ignores weights inside the clipping range, and poor initialization can make the clipping factor too large.Source: §2.2 · p. 3, Figure 4 · p. 3
  • The proposed module-wise dynamic scaling adapts clipping factors across Transformer modules and considers weights both inside and outside the clipping range when estimating their gradients.The resulting clipping factor gives finer resolution to the majority of weights.Source: §2.2 · pp. 3–4

3 Proposed Method

The method pairs token-level contrastive distillation, which makes quantized representations more distinguishable, with module-dependent scaling to adapt clipping to each module’s weight statistics.

3.1 Token-level Contrastive Distillation

  • Token-level contrastive distillation uses each student token’s teacher counterpart as a positive and other in-sequence tokens as negatives, with memory banks for momentum representations.It also averages student-to-teacher and teacher-to-student contrastive losses and adds a logits distillation loss.Source: §3.1 · pp. 4–5
  • Token-level contrastive distillation outperforms its sequence-level counterpart in the paper’s experiments.The authors conjecture that in-sequence negatives are harder than negatives from other sequences, supporting more efficient representation learning.Source: §3.1 · p. 5

3.2 Module-dependent Dynamic Scaling

  • Module-dependent scaling learns γ and sets clipping α from each module’s average weight magnitude rather than learning α directly.The scaling is initialized to 1, keeping initial clipping near the full-precision weights despite differences in weight distributions.Source: §3.2 · p. 5
  • The γ gradient sums contributions from weights inside and outside the clipping range, unlike PACT’s estimate, which uses only weights at or beyond α.Considering both regions reflects that α affects quantization error for both clipped and unclipped weights and makes the scaling less sensitive to varied weight distributions.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate quantized GPT-2 on language modeling and next-utterance prediction, and quantized BART on abstractive summarization. Results report competitive language-modeling and summarization performance, while next-utterance prediction remains challenging for all quantization methods.

4.1 Setup

  • The evaluation covers GPT-2 language modeling and next-utterance prediction, plus BART abstractive summarization, using the proposed method to produce QuantGPT and QuantBART.The study compares against self-implemented PACT, LSQ, and LAQ quantization baselines.Source: §4.1 · pp. 5–6

4.2 Language Modeling

  • At 2-bit weights, the method has an average 2 PPL-point drop and 14.4× model-size reduction; 8-bit results are comparable to full precision on PTB and WikiText103.The 8-bit result drops slightly on WikiText2; reducing from 8 to 4 bits costs around 1 PPL point on WikiText2 and WikiText103 and less than 0.1 on PTB.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Next Utterance Prediction

  • All quantization methods lose performance against full precision on next-utterance prediction, with PACT and LAQ degrading more than the proposed method as bit-width falls.At 2-bit weights, LSQ diverges and reaches 5% accuracy, no better than random guessing across 20 classes.Source: Table 1 · p. 6, §4.3 · p. 7

4.2 Language Modeling

  • The method outperforms PACT, LSQ, and LAQ across language-modeling bit-widths and tasks, and other GPT-2 compression methods in model size and performance, including at 2 bits.LSQ’s average PPL increases by over 5 times from 8 to 4 bits, and LSQ and PACT fail on all datasets at 2 bits.Source: Table 2 · p. 6, §4.2 · pp. 6–7

4.4 Abstractive Summarization

  • On XSum abstractive summarization, the method consistently outperforms other methods by a clear margin, and QuantBART summaries are logical and terse compared with PACT’s repeated text.The task uses BART and evaluates summaries with ROUGE 1, 2, and L.Source: Table 3 · p. 7, §4.4 · p. 7

5 Discussion

The ablations examine negative selection, sample count, training cost, representation choice, and dynamic scaling. They favor full-precision token negatives, moving-average representations, decoder-last states, and module-wise scaling, while noting modest training overhead.

5.1.1 Choices of Negative Sampling

  • Full-precision token negatives outperform fp+quan. and quan.-only variants, while global and sequence-level in-batch negatives perform worse.The authors suggest same-sequence negatives are more informative than random tokens, and token-level variants better suit token-wise generation.Source: §5.1.1 · pp. 7–8, Figure 6 · p. 7, Table 4 · p. 8

5.1.2 Number of Negative Samples

  • As negative-sample count increases, 2-bit QuantGPT’s PTB perplexity improves and gradually converges; moving-average negatives perform better than immediate representations.The authors attribute the moving-average advantage to smoother, more consistent token representations; results are means and standard deviations over five runs.Source: Figure 7 · p. 8, §5.1.2 · p. 8

5.1.3 Training Cost of the Contrastive Loss

  • Token-level contrastive loss improves PTB performance with only slightly slower training and greater memory consumption.The comparison trains GPT-2 on PTB with and without the loss.Source: Table 5 · p. 8, §5.1.3 · p. 8

5.1.4 Representations for the Contrastive Loss

  • Decoder-last representations outperform decoder-first, and also yield better ROUGE-1, ROUGE-2, and ROUGE-L than the other tested counterparts.The contrastive loss is applied to hidden states after a linear transformation; encoder ablations use source articles rather than target summaries.Source: §5.1.4 · p. 8, Table 6 · p. 9

5.2 Ablation on Dynamic Scaling

  • PACT gradient estimation drops performance significantly, especially on WikiText103; the ablation attributes QuantGPT’s main gain to module-wise dynamic scaling.Removing contrastive learning makes performance slightly worse.Source: Table 7 · p. 9, §5.2 · p. 9

6 Related Work

Prior work compresses generative PLMs through decomposition, distillation, or sparsity, while quantization research has largely focused on BERT. This paper distinguishes its work through analysis of generative-task difficulties, coverage of decoder and encoder-decoder models, and token-level contrastive distillation.

Compression of Generative Pre-trained Language Models

Quantization of Pre-trained Language Models

  • Prior PLM quantization work largely targets BERT, and attempts on generative PLMs are scarce, with their underlying difficulty unclear.BERT methods had reported negligible drops on natural-language-understanding tasks, with some reducing weights to 1-bit.Source: §6 · p. 9

Contrastive Learning

  • Unlike Wav2vec 2.0 (Baevski et al., 2020), this method quantizes parameters and activations, contrasts full-precision and quantized views, and computes loss autoregressively over all tokens.The paper also distinguishes its scope by studying both decoder and encoder-decoder generative models.Source: §6 · pp. 9–10

7 Conclusion

The paper studies low-bit quantization of generative PLMs and identifies homogeneous word embeddings and varied weight distributions as key difficulties. It addresses them with token-level contrastive learning and module-dependent dynamic scaling, and reports efficacy across three generative task types.

7 Conclusion

  • The paper identifies homogeneous word embeddings and varied weight distributions as difficulties in low-bit quantization of generative PLMs.The conclusion frames its subject as low-bit quantization of generative PLMs.Source: §7 · p. 10
  • Token-level contrastive learning targets more distinguishable token embeddings, while module-dependent dynamic scaling targets more accurate quantization.Source: §7 · p. 10
  • Experiments on language modeling, next utterance prediction, and abstractive summarization demonstrate the proposed method’s efficacy.The authors hope the work informs future exploration of generative-PLM compression.Source: §7 · p. 10

8 Limitations

The authors identify training-memory costs, sensitivity in the contrastive loss, and unverified hardware speedups as areas for improvement.

8 Limitations

  • Module-dependent dynamic scaling requires storing Q(u) during backpropagation to compute the scale gradient.The paper says this slightly increases training memory consumption and reduces experimental batch size.Source: §8 · p. 10
  • Contrastive-loss robustness remains limited: performance degrades when negatives are poorly chosen or the trade-off factor λ is large.Source: §8 · p. 10
  • Real speedups from low-bit operators remain unverified across hardware platforms.Source: §8 · p. 10

A Derivation of Gradient of Dynamic Scaling

This appendix derives the gradient of the dynamic scaling parameter γ, using the chain rule and a straight-through estimate for the uniform quantizer.

A Derivation of Gradient of Dynamic Scaling

  • The gradient of scaling γ with respect to the training loss is derived by applying the chain rule to the quantization process.The forward process uses uniform quantization, and the quantizer gradient is estimated with a straight-through estimator.Source: A Derivation of Gradient of Dynamic… · p. 13
  • Combining the stated equations yields a gradient that accounts for weights both inside and outside the clipping value and is proportional to ∥w∥1.Source: A Derivation of Gradient of Dynamic… · p. 13

B.1 Datasets

The appendix specifies dataset splits, model configurations, training hyperparameters, comparison methods, and task-specific framework changes.

B.1 Datasets

B.2 Model Architectures

  • Experiments use 12-layer GPT-2-small and BART-base, with 24-layer GPT-2-base and BART-large evaluated in Appendix C.1.The smaller models have hidden dimensions of 768; the larger models have dimensions of 1024.Source: §B.2 · p. 13

B.3 Hyperparameters

  • Training settings vary by task, including sequence lengths, batch sizes, learning rates, negative-sample counts, and epoch totals.Language-modeling runs use batch size 32; next-utterance prediction uses 16 for two epochs, and summarization uses 128 for eight epochs.Source: §B.3 · pp. 13–14

B.4 Description of the Compared Methods

  • Compared methods include PACT, LSQ, and LAQ; alternate language-modeling and attention-distillation losses worsened performance for all three self-implemented methods.PACT learns clipping factors, LSQ learns quantizer step sizes, and LAQ treats quantization as a loss-aware optimization problem.Source: §B.4 · p. 14

B.5 Frameworks of Double-head GPT-2 and BART

  • For next-utterance prediction and abstractive summarization, the frameworks are modified for the tasks, and the double-head GPT-2 output layer is also quantized.Source: §B.5 · p. 14

C More Experimental Results

Additional experiments report successful quantization on larger models, provide example summaries, and show further token-representation visualizations.

C.1 Performance of Larger Models

  • QuantGPT training converges across bit-widths on 24-layer GPT-2 and BART, outperforms PACT—especially at 2-bit weights—and performs better than the 12-layer models.Table 9 reports the ablation study on these larger models.Source: §C.1 · p. 14

C.2 Examples of Summarizations

  • Generated 2-bit BART summaries are described as more logical and terse than those from PACT, LSQ, and LAQ, whose embeddings show some homogeneity.Table 10 contains example summaries on XSum.Source: §C.2 · p. 14, Table 10 · p. 15
  • Figures 9 and 10 illustrate task-specific training workflows for double-head GPT-2 next-utterance prediction and BART abstractive summarization, omitting the full-precision teacher and distillation loss for simplicity.Source: Figure 9 · p. 16, Figure 10 · p. 16

C.3 More Visualizations for the Token Representations

  • Figure 11 presents cosine-similarity matrices for token representations from full-precision and 2-bit GPT-2 models on additional PTB samples, with observations similar to Section 2.Source: §C.3 · p. 16, Figure 11 · p. 17

Related papers

Loading 2203.10705v2…