DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization

• Large pre-trained models demand substantial computation and memory at inference, challenging deployment in real-time or resource-constrained settings.Quantization can reduce model footprints using lower-bit weights without changing the model architecture.Source: §1 · p. 1
• Efficient quantization of encoder-decoder transformers remains insufficiently studied, and low-bit seq2seq quantization has faced accumulated quantization errors.Prior seq2seq quantization was not aimed at large-scale pre-trained models or broader NLP tasks beyond machine translation.Source: §1 · p. 1
• DQ-BART transfers knowledge from a full-precision teacher to a student with fewer layers and ultra-low-bit parameters through joint distillation and quantization.The authors report 16.5x footprint compression with competitive performance across multiple language-generation benchmarks, and explore compression up to 27.7x.Source: §1 · p. 2

2.1 Quantization

• Quantization maps high-precision values to low-precision counterparts to reduce model footprint, using an approximation-based method with linear quantization.The method quantizes weights in hidden layers and most embeddings, leaves positional embeddings unquantized, and uses 8-bit activations.Source: §2.1 · p. 2
• Weight quantization replaces full-precision weights wt with a scaling factor αt times low-precision values bt.For more than 2 bits, the method uses symmetric linear quantization; at 2 bits, it uses the approximation-based TWN method.Source: §2.1 · p. 2

2.2 Distillation

• Distillation trains a smaller student to mimic the teacher’s output logits, attentions, and hidden states, initializing it from maximally spaced teacher layers.Unlike Shleifer and Rush (2020), the method distills both encoder and decoder; it then fine-tunes with task and distillation losses.Source: §2.2 · pp. 2–3

2 Distilling and Quantizing BART

• Combining quantization and distillation increases compression without significant performance sacrifice in the reported BART benchmark settings.The table covers CNN/DailyMail and XSUM summarization and ELI5 long-form question answering, using ROUGE metrics.Source: Table 1 · p. 3

2.3 Distillation-aware quantization

• Distillation-aware quantization trains a low-precision student against a full-precision teacher while updating a full-precision copy of the student at each iteration.Each step quantizes that copy, computes task and distillation losses with the low-precision model, then uses those losses to update the full-precision copy.Source: §2.3 · p. 3

3.1 Experimental Setup

• Experiments covered CNN/DailyMail, XSUM, ELI5, and WMT English-Romanian; base-sized BART served summarization and QA, while translation used mBART-large.For ELI5, a dense retriever supplied 10 Wikipedia documents per question; models were selected by development-set rouge-L.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 DQ-BART Results and Discussions

• On CNN/DailyMail, performance degradation stayed minimal as compression increased, especially before 20x; Figure 1 contrasts quantization-only with distillation plus quantization.Halving the decoder layers yielded less than 0.5 rouge-L drop across tasks and a further 2.9x compression.Source: Figure 1 · p. 4, §3.2 · p. 4
• Distillation-aware 8-bit models matched or exceeded full-precision models across tasks, while direct quantization cut rouge-L 50-75% relatively.At 2-bit quantization, compression reached 13.6x with a slight drop for summarization and no drop for long-form QA.Source: §3.2 · p. 4
• At 27.7x compression, rouge-L fell 5.67 on CNN/DailyMail (42.09 → 36.42), 12.24 on XSUM (35.71 → 23.47), and 1.06 on ELI5 (15.36 → 14.30).The authors conclude that suitable compression ratios should be chosen task by task, balancing performance and efficiency.Source: §3.2 · p. 4

3.3 DQ-mBART for Translation

• On WMT English-Romanian, mBART-large distillation-aware quantization performed reasonably, but 2-bit quantization with distillation caused substantial drops, possibly from accumulated errors.The authors suggest accumulation may be more significant for deeper models and challenging machine translation.Source: Table 2 · p. 4, §3.3 · p. 4

3.4 Distillation and Quantization v.s. Distillation Only

• At similar sizes, joint distillation and quantization outperformed distillation-only models across all tasks.Table 3 compares the two approaches; the authors report an advantage for the joint approach.Source: Table 3 · p. 5, §3.4 · p. 5

4 Conclusion

• DQ-BART achieved 16.5x model-footprint compression with little performance drop on three generative benchmarks, and explored trade-offs up to 27.7x.The conclusion also reports mBART translation experiments and identifies low-bit quantization of deeper models as challenging.Source: §4 · p. 5
• The work applies joint quantization and distillation to pretrained language models and targets sequence-to-sequence generation.The authors describe it as the first work to effectively distill and quantize pretrained seq2seq models for language generation.Source: §4 · p. 5
• Latency-improvement measurements across settings and additional methods, including attention-head pruning and sequence-level distillation, remain future work.Source: §4 · p. 5

A Details of TWN Quantization

• TWN quantization approximates each weight vector with a positive scaling factor and ternary values, then uses an approximate threshold to obtain the quantized weights.The threshold is set to 0.7 times the weight vector’s L1 norm divided by its dimension.Source: A Details of TWN Quantization · p. 7

B Details of Distillation Losses

• The distillation-loss notation uses layer-index mappings to specify which teacher layers student layers emulate.The mapping functions are defined for encoder and decoder layers.Source: B Details of Distillation Losses · pp. 7–8
• The logits distillation loss is the mean squared error between student and teacher output logits.Source: B Details of Distillation Losses · p. 7
• Attention distillation sums losses for encoder attention, decoder attention, and cross-attention.Source: B Details of Distillation Losses · pp. 7–8
• The hidden-state distillation loss covers both encoder and decoder hidden states between student and teacher layers.The loss is expressed as the sum of encoder-hidden-state and decoder-hidden-state losses.Source: B Details of Distillation Losses · p. 8

C Dataset Statistics

• Table 4 presents dataset statistics.Source: Table 4 · p. 8

D ELI5 Additional Details

• The appendix section provides additional details for the ELI5 dataset.Source: D ELI5 Additional Details · p. 8

D.1 Dense Retriever

• The authors could not find a public version of ELI5’s supporting documents and followed the author’s implementation to train a dense retriever.Source: §D.1 · p. 8
• Retriever recall was 0.3273 for the reproduced system versus 0.3247 for the reported implementation.Source: §D.1 · p. 8
• The trained dense retriever retrieves supporting documents from Wikipedia.Source: §D.1 · p. 8

D.2 Evaluating ELI5 Results

• The paper reports ELI5 results with the ROUGE-SCORE package, which differs from the ROUGE implementation used in the ELI5 and BART papers.For consistency, the authors also compared their trained baseline with the public one using the ELI5/BART implementation.Source: §D.2 · p. 8
• The authors’ base-size ELI5 baseline performs close to the large-size result reported by Lewis et al. (2020).Source: §D.2 · p. 8, Table 5 · p. 8

E Visualizations of Experimental Results on XSUM and ELI5 datasets

• Figure 2 visualizes performance against model-footprint compression ratio on XSUM using Table 1 results.Source: Figure 2 · p. 8
• The two figures present this performance-compression comparison for separate datasets: XSUM and ELI5.Source: Figure 2 · p. 8, Figure 3 · p. 9
• Figure 3 visualizes performance against model-footprint compression ratio on ELI5 using Table 1 results.Source: Figure 3 · p. 9

F Comparisons on “Shrink and Finetune”

• The comparison used three randomly selected models with the “Shrink and Finetune” schema and the paper’s hyperparameter settings.Source: F Comparisons on “Shrink and Finetune” · p. 9
• Distillation losses performed slightly better than “Shrink and Finetune” under the paper’s setting.The authors say this signals that guidance in weighting is important for the quantized and distilled model to learn well.Source: F Comparisons on “Shrink and Finetune” · p. 9, Table 6 · p. 9
• Table 6 compares the paper’s loss with the “Shrink and Finetune” loss.Source: Table 6 · p. 9