Source-linked AI summary
DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition
Hang Shao, Bei Liu, Wei Wang, Xun Gong, Yanmin Qian
TL;DR
Smaller Whisper variants lag larger ones in multilingual ASR, while larger models’ compute and storage needs challenge deployment on restricted devices. DQ-Whisper combines dynamic matching distillation with quantization-aware distillation; reported results include up to 5.18x model-size reduction with only a slight performance drop.
Problem
Smaller Whisper variants show a substantial ASR performance gap versus larger variants, especially in low- and mid-resource languages, while larger models’ computational and storage requirements hinder restricted-device deployment.
Source: §1 · p. 1Method
DQ-Whisper combines dynamic matching distillation, which selects teacher layers to guide student learning, with quantization-aware distillation.
Source: §1 · p. 1, §2.2 · p. 3Results
Up to 5.18x reduction in model size is achieved with only a slight performance drop, and distillation enhances small Whisper models’ multilingual capabilities without increasing computational costs.
Source: §1 · p. 2Takeaways & Limitations
Integrating quantization with distillation can result in greater compression while maintaining performance.
Source: §1 · p. 2
Abstract
from arXiv · showhide
As a popular multilingual and multitask pre-trained speech model, Whisper has the problem of curse of multilinguality. To enhance multilingual capabilities in small Whisper models, we propose DQ-Whisper, a novel joint distillation and quantization framework to compress Whisper for efficient inference. Firstly, we propose a novel dynamic matching distillation strategy. Then, a quantization-aware distillation framework is introduced to integrate quantization with distillation. Experimental results on various multilingual datasets show that our suggested distillation approach can effectively enhance the multilingual capabilities of small Whisper models without increasing computational costs. Up to 5.18x reduction in model size is achieved with marginal performance degradation. In addition, quantization is compatible with distillation, which can result in a higher compression rate.
1. INTRODUCTION
Smaller Whisper variants have a multilingual ASR performance gap, while larger models’ resource demands hinder deployment. DQ-Whisper combines distillation and quantization to compress Whisper while preserving multilingual capabilities.
1. INTRODUCTION
- Smaller Whisper variants show a substantial ASR performance gap versus larger variants across languages, especially low- and mid-resource languages.Larger models’ computational and storage requirements also make deployment on resource-restricted devices challenging.Source: §1 · p. 1
- Earlier Whisper compression work [20] required massive training time, while a language-specific module incurred extra computational expenses.Other cited compression approaches mainly targeted encoder-only architectures, which the passage says are unsuitable for Whisper.Source: §1 · p. 1
- DQ-Whisper combines dynamic matching distillation, which selects teacher layers to guide student learning, with quantization-aware distillation.The framework is proposed to compress Whisper for efficient inference while maintaining multilingual capabilities.Source: §1 · p. 1
- Up to 5.18x reduction in model size is achieved with only a slight performance drop; combining distillation with quantization can yield greater compression while maintaining performance.The reported multilingual results also show that the distillation approach enhances small Whisper models without increasing computational costs.Source: §1 · p. 2
2. METHODS
DQ-Whisper distills both prediction outputs and hidden layers, dynamically matching student layers to teacher layers. It integrates uniform quantization into distillation, using quantization loss to guide layer selection and jointly training the losses.
2.1. Dynamic Matching Distillation
- Dynamic matching distills prediction outputs and hidden layers, selecting teacher layers to guide the student rather than relying only on logits or fixed layer choices.The authors describe fixed layer selection as ad hoc and potentially suboptimal.Source: §2.1 · p. 2
- DM selects each student layer’s teacher match by minimum MSE; RDM constrains that mapping to increase monotonically, with f(i) ≥ i.The hidden-layer objective weights each layer’s distillation by λ_i.Source: §2.1 · p. 2
2.2. Joint Distillation and Quantization
- Quantization-aware distillation uses uniform 8-bit quantization and quantization loss to guide selection of teacher layers suited to quantization.The student is uniformly quantized before the layer-level quantization loss is used for matching.Source: §2.2 · p. 3
- The framework jointly trains prediction and hidden-layer distillation loss, quantization loss, and cross-entropy loss.The quantized student’s layer loss compares weights before and after quantization.Source: §2.2 · p. 3
3. EXPERIMENT
The experiments use whisper-small as teacher and whisper-base and whisper-tiny as students, with Chinese and four-language COMMONVOICE data. Training uses 8-bit quantization and freezes the encoder under GPU-memory constraints.
3.1. Experimental Setup
- Whisper-small is the teacher, while whisper-base and whisper-tiny are the student models; multilingual evaluation uses four COMMONVOICE languages totaling 1,393 hours.The Chinese training set contains 450 hours; IHC1 and IHC2 test sets contain 10 and 15 hours, respectively.Source: §3.1 · p. 3
3.2. Implementation Details
- Training uses 80-channel log-magnitude Mel inputs, 8-bit quantization, and a frozen encoder, with only the decoder trained.The spectrogram uses a 25 ms window computed every 10 ms; SpecAugment and Adam are also used.Source: §3.2 · p. 3
- The experiments use an in-house speech-recognition toolkit with ESPnet, training models until convergence.The implementation uses a byte-level BPE tokenizer for multilingual models.Source: §3.2 · p. 3
4. RESULTS AND ANALYSIS
Across multilingual COMMONVOICE tasks, joint distillation and quantization reduces model size with negligible WER degradation. Dynamic matching distillation improves small-model CER, while joint training outperforms separate compression at the same compression rate.
4.2. Evaluation on Joint Distillation and Quantization
- 5.18x and 10.48x compression ratios are reported for whisper-base and whisper-tiny with joint distillation and quantization, with performance nearly identical to the best distilled model.All student decoder layers are quantized to 8-bit integers; the reported model sizes fall from 139MB to 89MB for base and 72MB to 44MB for tiny.Source: Table 1 · p. 4, §4.2 · p. 4
4.1. Evaluation on Dynamic Matching Distillation
- 29.9% CER for whisper-base nearly matches whisper-small, with a model 3.32x smaller; whisper-tiny reaches 33.6% CER with 6.4x fewer parameters.Whisper-base CER moves from 34.1% with no distillation to 31.5% with logit distillation and then 29.9% with hidden-layer distillation; the initial out-of-domain gap reaches 11% CER.Source: §4.1 · pp. 3–4, Table 1 · p. 4
- Restrained dynamic matching performs best, and a monotonically increasing layer order is most effective for Whisper.The authors attribute the gains from combining hidden-layer and logit distillation to information in both intermediate and prediction layers.Source: §4.1 · p. 4
4.3. Ablation Study
- 9.2% and 10.7% CER relative improvements for whisper-base and whisper-tiny, respectively, are reported for joint versus separate compression at the same compression rate.Separate compression uses two training steps and requires more computational resources and time.Source: Table 3 · p. 4, §4.3 · p. 5
4.4. Evaluation on Multilingual Corpus
- 139MB to 89MB, joint distillation and quantization raises compression from 3.32x to 5.18x with negligible WER degradation versus direct whisper-base decoding.Similar conclusions are reported for whisper-tiny.Source: Table 4 · p. 5, §4.4 · p. 5
5. CONCLUSION
DQ-Whisper combines distillation and quantization to compress Whisper while preserving multilingual capabilities. Across multilingual evaluations, it improves smaller models without additional computational burden and achieves up to 5.18x model-size reduction with a slight performance drop.
5. CONCLUSION
- Up to 5.18x model-size reduction is achieved with a negligible performance decline, while combining distillation and quantization yields a higher compression ratio while maintaining performance.The evaluation on COMMONVOICE also confirms that DQ-Whisper compresses Whisper while preserving multilingual speech recognition.Source: §5 · p. 5
- DQ-Whisper combines distillation and quantization to compress Whisper for inference while preserving multilingual capabilities.Source: §5 · p. 5
- Dynamic matching distillation selects specific teacher-network layers to guide the student network’s learning.Source: §5 · p. 5
- Quantization-aware distillation merges quantization with distillation.Source: §5 · p. 5
- Distillation substantially enhances smaller Whisper models’ multilingual abilities without additional computational burden.Source: §5 · p. 5