Source-linked AI summary
Efficient Speech Translation through Model Compression and Knowledge Distillation
Yasmin Moslem
TL;DR
Large audio-language models are computationally intensive, making efficient speech-translation deployment challenging. The paper combines iterative layer pruning, QLoRA, and knowledge distillation in Qwen2-Audio-7B-Instruct systems for German and Chinese. The pruned models reduce size while recovering most teacher-model translation quality.
Problem
Large audio-language models are computationally intensive, challenging their deployment for speech translation in resource-limited settings.
Source: §1 · p. 1Method
The systems combine iterative layer pruning, QLoRA, and knowledge distillation, with pruning followed by fine-tuning to restore translation quality.
Source: §1 · p. 1, §3.3 · p. 4Results
Up to 50% compression in model parameters and storage accompanies recovery of 97% of Chinese and 100% of German teacher-model translation quality.
Source: §3.3 · p. 4Takeaways & Limitations
Combining compression techniques yields substantial model-size reduction with minimal impact on speech-translation performance.
Source: §6 · p. 8Generalization remains to be evaluated across more diverse datasets, including authentic and synthetic training data.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Efficient deployment of large audio-language models for speech translation remains challenging due to their significant computational requirements. In this paper, we address this challenge through our system submissions to the "Model Compression" track at the International Conference on Spoken Language Translation (IWSLT 2025). We experiment with a combination of approaches including iterative layer pruning based on layer importance evaluation, low-rank adaptation with 4-bit quantization (QLoRA), and knowledge distillation. In our experiments, we use Qwen2-Audio-7B-Instruct for speech translation into German and Chinese. Our pruned (student) models achieve up to a 50% reduction in both model parameters and storage footprint, while retaining 97-100% of the translation quality of the in-domain (teacher) models.
1 Introduction
Large audio-language models are computationally intensive, motivating compression methods that retain performance. The paper presents two Qwen2-Audio-7B-Instruct submissions combining fine-tuning, quantization, pruning, and knowledge distillation for German and Chinese speech translation.
1 Introduction
- Qwen2-Audio accepts varied audio inputs and performs audio analysis or responds directly to speech instructions; the IWSLT 2025 track requires submissions derived from it.The task's official language pairs are EN-DE and EN-ZH.Source: §1 · p. 1
- The experiments combine QLoRA with 4-bit quantization, iterative layer pruning based on layer-importance evaluation, and sequence-level knowledge distillation.The work is based mainly on Qwen2-Audio-7B-Instruct.Source: §1 · p. 1
- 50% compression in parameters and storage is achieved by pruning the decoder to 24 layers while retaining all 32 encoder layers, then fine-tuning with knowledge-distilled data.The data includes ACL 60/60 and a portion of CoVoST2.Source: §1 · p. 1
- More than 40% model-size reduction in parameters and storage accompanies the best translation performance for both EN-DE and EN-ZH with QLoRA, 4-bit quantization, and knowledge distillation.Table 1 evaluates this approach.Source: §1 · p. 1, Table 1 · p. 2
2 Data
The experiments use ACL 60/60 as in-domain data and supplement it with CoVoST2 after pruning to help restore translation quality.
2.1 In-Domain Data
- The ACL 60/60 dataset provides multilingual translations of ACL 2022 technical presentations into 10 target languages; its dev and eval splits total 884 utterances.The two splits were merged, with 100 utterances sampled for testing and 784 left for training; the dataset was required for constrained IWSLT Model Compression submissions.Source: §2.1 · p. 2
2.2 Out-of-Domain Data
- After pruning, a portion of CoVoST2 was added alongside teacher-model distillation data to provide more training data for restoring translation quality.CoVoST2 is a multilingual speech-to-text translation corpus covering English-to-15-language translation, including German and Chinese.Source: §2.2 · p. 2
3 Methodologies
The section develops compressed Qwen2-Audio-7B-Instruct translation models through QLoRA, knowledge distillation, and iterative layer pruning. The pruned model retains 97–100% of teacher translation quality at up to 50% compression.
3.1 Full-Parameter Fine-tuning
- The fully fine-tuned Qwen2-Audio-7B-Instruct model served as the foundation and teacher for both compression setups.On ACL 60/60, it outperformed the baseline by an average of 48 COMET points across German and Chinese translation.Source: §3.1 · p. 2
3.2 QLoRA with 4-bit Quantization
- QLoRA with 4-bit quantization and sequence-level distillation achieved the highest translation performance across metrics for German and Chinese while reducing model parameters and storage by more than 40% versus baseline.The teacher translated ACL 60/60 training data; duplicate translations were removed before fine-tuning students on the augmented data.Source: §3.2 · p. 3
3.3 Iterative Layer Pruning
- Layer-importance selection using chrF/chrF++ produced stronger pruned models than selection using COMET, and English-to-German translation was more affected by pruning than English-to-Chinese.The study also reports that iterative pruning outperformed fixed middle-layer pruning, while decoder-only pruning performed better than pruning both encoder and decoder layers.Source: §3.3 · p. 4
- Iterative pruning removes one decoder layer at a time based on the least negative impact on ACL 60/60 test chrF/chrF++ scores, then fine-tunes the pruned model with in-domain and teacher-generated data.The final QLoRA stage also used 100,000 CoVoST2 utterances and teacher-distillation data oversampled by a factor of 10.Source: §3.3 · p. 4
- Up to 50% compression retained 97% of Chinese and 100% of German teacher translation quality after pruning and QLoRA fine-tuning.The pruning process reduced model size by about 20% before 4-bit quantization and QLoRA; Table 2 reports the full pipeline comparison.Source: Table 2 · p. 3, §3.3 · p. 4
4 Ablation Study
The ablations favor iterative, decoder-only pruning and show that compression depth, recovery timing, and added out-of-domain data affect translation quality. Their comparisons also identify trade-offs between storage, inference speed, and computational cost.
4.1 Qwen2-Audio Base vs Instruct
- The Qwen2-Audio-7B-Instruct model outperformed the base model for both English-to-German and English-to-Chinese before fine-tuning.The authors therefore used the Instruct model throughout their experiments.Source: Table 4 · p. 5, §4.1 · p. 5
4.2 Encoder-Decoder Layer Pruning
- Decoder-only pruning from 32 to 24 layers achieved 4+ additional BLEU and chrF++ points and 18+ COMET points over pruning both encoder and decoder.The decoder-only model's parameters and storage footprint were only 1% larger than the model with both encoder and decoder pruned.Source: §4.2 · p. 5
4.3 Iterative vs Middle Layer Pruning
- BLEU 42.52, chrF 36.92, and COMET 39.42 followed iterative Chinese pruning, versus BLEU 1.3, chrF 3.8, and COMET -94.05 after middle-layer pruning.Using chrF/chrF++ to guide iterative pruning performed better than using COMET; the authors say the ACL 60/60 dataset's scientific nature may explain this and call for exploration on other datasets.Source: §4.3 · p. 6, Table 5 · p. 6
4.4 Immediate Recovery
- Immediate recovery changed which German decoder layers were selected, but did not outperform fine-tuning once after pruning and required substantially more computation.The authors suggest repeated fine-tuning on the small in-domain dataset may cause overfitting.Source: §4.4 · p. 7, Table 7 · p. 7
4.5 Pruning more layers
- Pruning up to 12 layers kept quality close to pruning 8, while pruning 16 caused considerable degradation; pruning 8 and 16 layers accelerated inference by approximately 20% and 40%.These quality results cover pruning and initial fine-tuning, not the full process with distillation and QLoRA; QLoRA reduces storage at the cost of inference speed.Source: §4.5 · pp. 7–8
4.6 Fine-tuning before/after pruning
- For German, valid translations after pruning one quarter of the decoder layers required further training, and fine-tuning the teacher on in-domain data before pruning is recommended.Source: §4.6 · p. 8
4.7 Out-of-Domain Data Size
- For English-to-German, adding 50k or 80k CoVoST2 segments improved performance over 10k, while 100k improved COMET but not BLEU or chrF++.Source: Table 8 · p. 8, §4.7 · p. 8
5 Inference and Evaluation
Evaluation used greedy generation with a batch size of one and a maximum length of 1024 tokens. Translation quality was assessed with BLEU, chrF++ for German, raw chrF for Chinese, and COMET.
5 Inference and Evaluation
- Greedy generation used batch size 1 and a maximum generation length of 1024 tokens.Source: §5 · p. 8
- Evaluation used BLEU and COMET for both language directions, with chrF++ for German and raw chrF for Chinese.Raw chrF was used for Chinese following the metric author's note that the concept of Chinese words is generally unclear.Source: §5 · p. 8
- The evaluation also included ablations of the baseline, pruning approach, number of pruned layers, and out-of-domain data size.Source: §5 · p. 8
6 Conclusions and Future Work
The paper combines compression techniques to reduce model size while minimally affecting speech translation performance. Its future work proposes adaptive compression and broader dataset evaluation, among other directions.
6 Conclusions and Future Work
- Combining compression techniques substantially reduced model size with minimal impact on speech translation performance.QLoRA with knowledge distillation achieved better translation quality than layer pruning alone, while the combined iterative pruning, quantization, distillation, and multi-stage fine-tuning approach targeted higher compression while preserving quality.Source: §6 · p. 8
- Future work will explore adaptive pruning and quantization for deployment constraints and evaluate generalization across more diverse authentic and synthetic training data.Source: §6 · p. 8
- The code for the experiments is publicly available.Source: §6 · p. 8