Source-linked AI summary
Investigating Model Compression for Neural Machine Translation in the Biomedical Domain
Maria Zafar, Souhail Bakkali, Rejwanul Haque
TL;DR
Low-resource biomedical translation has limited specialized parallel data, complicating knowledge transfer and model compression. This paper combines knowledge distillation, fine-tuning, and quantization for French-to-English translation. The best student is reported to increase inference speed by 98.21% and reduce CO2 emissions by 98.46% over the teacher, with an insignificant translation-quality drop.
Problem
Method
The study combines knowledge distillation with fine-tuning strategies and quantizes student models, using quantized, domain-fine-tuned teachers to generate supervision.
Source: §1 · p. 2, §5 · p. 10Results
The best student model increased inference speed by 98.21% and reduced CO2 emissions by 98.46% over the teacher, with an insignificant drop in translation quality.
Source: §5 · p. 10Takeaways & Limitations
The authors conclude that their distillation setup yields a smaller student with competitive translation quality, faster inference, and lower emissions for online machine-translation services.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Large-scale pretrained transformer models have achieved state-of-the-art performance across diverse machine translation tasks, including multilingual settings. Knowledge distillation has emerged as a sustainable approach for model compression, transferring knowledge from large teacher models to smaller, more efficient student models. Similarly, quantization, which reduces the numerical precision of model weights and activations (e.g., from 32-bit to 8-bit representations) is widely used to accelerate inference, enabling models to run several times faster during deployment. However, both techniques face limitations when applied to specialized domain data, particularly under low-resource conditions. In knowledge distillation, the effectiveness of transfer is often constrained by the scarcity of domain-specific parallel data, while quantization can lead to performance degradation as bit precision decreases. In this work, we investigate the combined application of knowledge distillation and quantization for French-to-English biomedical translation, a domain characterized by specialized terminology and limited parallel resources. We develop and compare multiple fine-tuning strategies to adapt compressed student models to this challenging setting. Our experiments demonstrate that a collaboratively distilled and quantized student model achieves a 69% reduction in size, a 98.21% increase in inference speed, and a 98.46% reduction in CO2 emissions compared to the original baseline all without sacrificing translation quality. These results indicate that jointly optimized compression techniques can yield efficient, high-performance models suitable for translation service providers operating under resource constraints.
1 Introduction
Knowledge distillation and quantization aim to make machine translation models more compact and efficient, but specialized, low-resource domains pose challenges for both. This work combines distillation, fine-tuning strategies, and quantization for French-to-English biomedical translation, reporting student models that perform comparably to or outperform standard distillation baselines.
1 Introduction
- Knowledge distillation transfers knowledge from a larger teacher to a smaller student, while quantization lowers numerical precision to reduce memory use and accelerate inference.The combined goal is compact, deployable machine-translation models balancing quality and speed.Source: §1 · pp. 1–2
- Scarce specialized parallel data can hinder distillation, shallow students may mishandle polysemous terms, and low-bit quantization can degrade translation quality in low-resource domains.Students may overfit small datasets and fail to generalize the teacher’s broader knowledge.Source: §1 · p. 2
- The quantized student models performed comparably to or significantly better than standard knowledge-distillation baselines on French-to-English biomedical translation.Experiments used the ELRC-EMEA OPUS corpus; the baselines followed standard KD training.Source: §1 · p. 2
- The study targets translation service providers that need compact, fast, competitive online systems despite constrained budgets and limited domain training data.Source: §1 · p. 2
2 Related Work
Related work describes distillation and quantization as routes to smaller, more efficient translation models, while noting challenges from limited parallel data and noisy teacher outputs. This paper applies knowledge distillation with different fine-tuning strategies and quantizes student models for machine translation in a specialized domain.
2 Related Work
- Sequence-level knowledge distillation trains smaller students on teacher-generated pseudo-target sequences, with a pipeline of distilled-data generation followed by student training and fine-tuning.Prior work also explored prompting and rationale approaches [29] [10], but studies often did not address fixed teacher budgets or noisy annotations [16] [3] [17].Source: §2 · pp. 2–3
- Earlier domain-aware and multi-domain distillation approaches sought performance near expert models or BLEU gains without increasing translation time or memory,.The domain-classifier approach was less parameter-efficient than a target-token model.Source: §2 · p. 3
- Low-resource translation is constrained by scarce high-quality parallel data, and noisy teacher outputs can propagate incorrect patterns into distilled students.Smaller models offer lower computational and energy demands but often lag larger models in translation quality, especially for low-resource languages.Source: §2 · p. 3
- Prior quantization work studies lower-precision deployment, benchmarking tools, and combinations with pruning or distillation; reported strategies include 4-bit QLoRA and confidence-filtered teacher outputs [19],,,.The cited speech-translation study reported up to 50% fewer parameters and storage while retaining 97–100% of original translation quality for English-to-German and English-to-Chinese tasks.Source: §2 · p. 4
- The paper aims to provide translation service providers a cost-effective machine-translation compression approach combining distillation, quantization, and different fine-tuning strategies.Source: §2 · p. 4
3 Experiments
The experiments use MarianMT teacher and student models, test quantized and non-quantized settings, and apply standard student–teacher training with teacher translations as pseudo-targets. Evaluation uses French–English biomedical parallel data from ELRC-EMEA OPUS, with separate validation and test samples.
3.1 Experimental Setups
- The experiments compare MarianMT teacher and student checkpoints, referred to as the big transformer (BT) and small transformer (ST), respectively.Training used batch size 8, evaluation batch size 8, learning rate 1e-5, 20 epochs, and maximum sequence length 256.Source: §3.1 · p. 5
- Models were tested in quantized and non-quantized settings, with FP32-to-FP16 conversion for inference and quantized teacher models generating student distillation data.All models used the same configurations and an A100 GPU; the student was fine-tuned before distillation, using teacher English translations as pseudo-targets.Source: §3.1 · p. 5
3.2 Dataset
- The dataset is French–English biomedical parallel text from ELRC-EMEA OPUS, with 1,000 sentence pairs sampled for validation and 2,000 for testing.Training duplicates were removed, and Table 1 presents the data statistics.Source: §3.2 · p. 5, Table 1 · p. 6
4 Results and Discussions
The results compare teacher and student variants across translation quality and inference efficiency, finding that the best-performing student stays close to the teacher while reducing size, inference time, and emissions versus the baseline teacher.
4.1 Evaluation Metrics
- Evaluation reports BLEU, chrF++, COMET, BERTScore, and BLEURT alongside inference time, model size, and CO2 emissions.Inference time was measured in seconds, model size in MB, and emissions in kilograms during inference.Source: §4.1 · p. 6, Table 2 · p. 6
4.2 Teacher Models
- Teacher fine-tuning improved all seven reported quality metrics, and quantization added further gains across those metrics.QLoRA fine-tuning did not improve teacher test-set scores over the baseline models.Source: §4.2 · p. 7
4.3 Student Models
- Fine-tuning improved the ST model; quantization further improved the distilled student, while fine-tuning on augmented original and distilled data did not work.Quantization added up to 1.15 BLEU and 1.5 COMET points over the non-quantized student.Source: §4.3 · p. 8
4.4 Efficiency and Sustainability Analysis
- 69% size decrease and 98.21% inference-speed increase versus baseline BT; CO2 emissions fell by up to 98.46%, while student quality was close to teacher.These comparisons concern the best-performing models; the emissions reduction is reported versus baseline BT.Source: Table 5 · p. 9, §4.4 · p. 9, §4.3 · p. 9
- The paper positions compact, fast, competitive models as useful for online translation providers operating with limited budgets.The reported efficiency comparisons include model size, inference speed, and inference emissions.Source: §4.4 · pp. 9–10
5 Conclusion
The study combines teacher fine-tuning and quantized supervision in a knowledge-distillation setup for domain adaptation. The best student was faster and emitted less CO2 than the teacher, with an insignificant drop in translation quality.
5 Conclusion
- Fine-tuning the teacher for the domain and using its quantized model for supervision sped up soft-label generation.The authors report that this approach improved translation quality or provided more reliable KD supervision, while facilitating faster execution and lower resource consumption.Source: §5 · p. 10
- The strategy could benefit translation service providers seeking domain-adapted teachers while reducing computational costs and maintaining translation quality.Source: §5 · p. 10
- 98.21% increase in inference speed and 98.46% reduction in CO2 emissions over the teacher came with an insignificant drop in translation quality.The quantized, fine-tuned student achieved quality close to, and sometimes surpassing, the teacher.Source: §5 · p. 10