Explainable Multi-Loss Distillation Framework for Efficient and Interpretable Shrimp Disease Text Classification

• SALT unifies teacher-student learning with explainability-driven evaluation for Vietnamese domain-specific text understanding while comparing distribution-based and metric-learning distillation losses.Source: Highlights · p. 2
• The framework combines LIME, aggregated SHAP feature attribution, and keyword deletion to assess local explanations, global patterns, and explanation faithfulness.Source: Highlights · p. 2
• SALT is evaluated against multiple distillation objectives and strong baselines using classification performance, computational efficiency, and explanation analysis.Source: Highlights · p. 2

• Global shrimp production exceeds 3 million tons, with Vietnam among the top three exporters, but disease outbreaks severely reduce farming productivity.Source: §1 · p. 4
• Existing shrimp-disease studies primarily use CNNs, transfer learning, or limited NLP methods, emphasizing accuracy while underaddressing compactness, interpretability, and deployment conditions [2, 3, 4, 5, 6].Source: §1 · p. 4
• Vietnamese PLMs such as BERT and PhoBERT capture contextual semantics, but their use for symptom-based shrimp diagnosis remains limited and computationally demanding.Source: §1 · pp. 4–5
• Noisy, domain-dependent symptom descriptions and black-box PLMs can reduce reliability and produce untrustworthy predictions, making interpretability and robustness essential for aquaculture deployment.Source: §1 · p. 5
• Knowledge distillation compresses large teacher models into smaller students, but existing approaches mainly target performance or reduction and provide limited transparency during knowledge transfer.Source: §1 · pp. 5–6
• SALT unifies teacher–student learning with LIME, aggregated SHAP, keyword deletion, and systematic comparisons of distribution-based and metric-learning losses, evaluating performance, efficiency, and explanation faithfulness.Source: §1 · pp. 6–7

2.1. Data collection and preprocessing

• ShrimpCap initially contains 1,000 Vietnamese symptom descriptions spanning 10 common shrimp diseases in an even class distribution.The descriptions were collected from aquaculture guidelines and online forums, with disease labels manually assigned and verified through aquaculture-expert collaboration.Source: §2.1 · p. 7
• The dataset was curated from Vietnamese aquaculture resources, manually categorized, and expert-reviewed to verify symptom-label alignment across disease classes.Source: §2.1 · p. 7
• The preprocessing pipeline standardizes text by removing HTML and nonsemantic special characters, normalizing Unicode and Vietnamese diacritics, lowercasing, and deleting excess whitespace.Figure 1 presents the data preprocessing pipeline.Source: §2.1 · p. 9, Figure 1 · p. 10
• Semantic redundancy is reduced by filtering sentence-embedding similarities above 0.95, followed by manual inspection and removal or revision of similar descriptions.Source: §2.1 · p. 9
• Vietnamese word segmentation is performed with ViTokenizer because Vietnamese contains many meaningful compound words.Figure 2 illustrates the transformation of a raw description through the preprocessing pipeline.Source: §2.1 · pp. 9–10, Figure 2 · p. 10
• Sentence-embedding analysis reveals high semantic similarity among several disease pairs, indicating that diseases can share common symptom descriptions.Figure 3 shows the inter-class matrix using average pairwise cosine similarity.Source: §2.1 · p. 10, Figure 3 · p. 11
• After preprocessing, the experimental ShrimpCap dataset contains 979 symptom descriptions split into training, validation, and test sets at a 6 : 2 : 2 ratio.The resulting data statistics are summarized in Table 2.Source: §2.1 · p. 11, Table 2 · p. 12

2.2. Augmentation for baseline evaluation.

• The augmented training data were constructed to test whether conventional linguistic augmentation improves student-model performance.The resulting augmented ShrimpCap dataset was used exclusively for the linguistic augmentation baseline.Source: §2.2 · p. 12
• Four strategies diversify training examples: synonym replacement, PhoBERT-based contextual augmentation, VietAI/vit52base paraphrasing, and symptom mix-up within disease classes.The methods respectively improve lexical accuracy, contextual diversity, grammatical variation, and learning of disease-group characteristics.Source: §2.2 · pp. 12–13
• A GPT-5.5 [33] prompt-guided framework generates diagnostically consistent descriptions through goal, key, and translation controls spanning wording, structure, style, abstraction, and expansion.The <KEY> component preserves disease characteristics and prevents contradictions or fabricated symptoms, while <TRANSLATE> controls stylistic transformations.Source: §2.2 · p. 13
• Experts manually reviewed generated descriptions for semantic consistency, disease-label preservation, linguistic clarity, and symptom validity before inclusion, then the expanded training set was paired with unchanged validation and test sets.Noncompliant descriptions were revised or discarded, and the resulting data distribution is summarized in Table 3.Source: §2.2 · p. 13, Table 3 · p. 16

2.3. Method

• The framework selects and fine-tunes a teacher, compares distilled and supervised lightweight students, then evaluates their decision-making with explainability methods.Teacher selection uses validation monitoring and test-set evaluation, while explainability supports selecting a lightweight, reliable classifier.Source: §2.3 · pp. 16–17

2.3.1. Teacher selection

• Teacher candidates include Vietnamese-specific PhoBERT and multilingual mBART, with Focal Loss emphasizing difficult samples during fine-tuning.PhoBERT is selected for Vietnamese contextual modeling, while mBART provides a multilingual comparative baseline; Focal Loss uses class balancing and a focusing parameter.Source: §2.3.1 · p. 18

2.3.2. Student model architecture

• The student uses a truncated PhoBERT encoder with gated fusion of [CLS] and attention-pooled representations, reducing layers while adapting to varied pathological contexts.The architecture applies the first four Transformer layers, followed by a projection head with ReLU and dropout, and predicts 10 disease classes.Source: §2.3.2 · pp. 18–20

2.3.3. Knowledge distillation

• Distillation combines temperature-smoothed soft-target loss with hard-label cross-entropy, while comparing KL [36], MSE [37], Cosine [38], and JSD losses.Soft targets help students learn relationships among closely related classes and semantic knowledge from teachers, while hard-label cross-entropy uses ground-truth labels to keep predictions aligned with the original target.Source: §2.3.3 · p. 20

2.3.4. Explainability analysis

• Explainability combines instance-level LIME [30] and globally aggregated SHAP [31]: LIME compares shared top-five word attribution weights, SHAP reports mean-absolute values across test samples, and keyword deletion measures confidence degradation.SHAP uses 200 samples from the training set as background, and keyword deletion removes top-k keywords (k ∈{1, 2, 3, 5}) while comparing confidence degradation with random-word removal.Source: §2.3.4 · pp. 21, 23

2.4.1. Experimental configuration

• All models share identical optimization settings, while distillation, metric-learning, and explainability hyperparameters remain fixed across experiments for fair and reproducible comparisons.General training hyperparameters are summarized in Table 4, and method-specific settings are summarized in Table 5.Source: §2.4.1 · pp. 24–25, Table 4 · p. 24, Table 5 · p. 25
• Accuracy, precision, recall, and F1 are reported, with teacher selection based on F1 and model selection based exclusively on validation macro-F1 before test evaluation.All experiments are repeated five times with different random seeds, and results are reported as mean ± standard deviation.Source: §2.4.1 · pp. 25–26
• Efficiency is measured by parameter count, model size, GFLOPs, inference latency, and throughput under batch size 1 and maximum sequence length 384.Latency is averaged over 100 warmed-up iterations on specified GPU and CPU hardware.Source: §2.4.1 · p. 26

2.4.2. Baselines and comparative setups

• Comparative experiments isolate SALT components through larger-teacher, no-gated-fusion, and linguistic-augmentation setups, alongside internal and external baselines.The no-gated-fusion variant replaces gated fusion with the original [CLS] embedding while keeping other configurations unchanged.Source: §2.4.2 · pp. 26–27
• Baselines include supervised fine-tuning, a reproduced best TF-IDF+DNN configuration, and frozen F2LLM-v2-80M embeddings [43] with a trained classifier.The supervised student uses the same data splits and optimization settings, while F2LLM-v2-80M uses Adam [44], early stopping, and the highest validation macro-F1 checkpoint.Source: §2.4.2 · pp. 27–28

3.1. Evaluation of teacher models

• PhoBERT outperformed mBART as teacher, reaching 89.08% accuracy and 0.8899 F1-score versus 86.43% and 0.8647, with approximately 26× lower inference latency.PhoBERT also used fewer parameters and a smaller model size, while showing better class discrimination for several challenging diseases.Source: §3.1 · pp. 28–30
• Across models and objectives, errors concentrated among diseases with overlapping symptom descriptions, while conventional objectives generally retained stronger diagonal dominance and class separation.PhoBERT, Cosine, and JSD showed cleaner confusion patterns than weaker alternatives, whereas SupCon and augmented models produced more dispersed off-diagonal predictions.Source: §3.1 · pp. 28–29, Figure 8 · p. 29, §3.2 · p. 31

3.2. Evaluation of student models

• 89.18% accuracy and 0.8935 F1-score made Cosine distillation the strongest student configuration, surpassing TF-IDF + DNN at 88.98% accuracy and 0.8896 F1-score while requiring only 3.75–3.83 ms inference.Knowledge distillation also outperformed supervised fine-tuning, which achieved 86.53% accuracy and 0.8666 F1-score; TF-IDF + DNN required 76.46 ms per sample.Source: §3.2 · pp. 30–31, Table 7 · p. 31, §3.3 · pp. 33–34
• Conventional distillation losses performed comparably, whereas metric-learning objectives were generally weaker; Cosine preserved angular similarity and produced the best overall student metrics.SupCon was lowest at 86.02% accuracy and 0.8601 F1-score, while Triplet achieved the lowest latency at 3.75 ms per sample.Source: §3.2 · pp. 30–31, Figure 10 · p. 32

3.3. Comparison with baseline methods

• The distilled students outperformed supervised and external baselines, while F2LLM-v2-80M was weakest at 82.86% accuracy and 0.8287 F1-score and had weaker class separability.Baseline confusion was greatest for clinically similar diseases, whereas distillation reduced off-diagonal errors and learned more discriminative semantic representations.Source: §3.3 · pp. 33–36, Figure 11 · p. 34

3.4.1. Effectiveness of teacher model scale

• 91.94±0.94% accuracy and 0.9183±0.0097 F1-score were achieved by fine-tuned PhoBERT-large, but its distilled students did not consistently improve over PhoBERT-base students.Center was best for the PhoBERT-large students at 88.16% accuracy and 0.8822 F1-score, indicating that greater teacher capacity did not ensure more effective transfer.Source: §3.4.1 · pp. 36–37, Table 9 · p. 38

3.4.2. Effectiveness of gated fusion

• Removing gated fusion did not consistently reduce performance, with JSD and Center showing modest improvements and other objectives remaining comparable or slightly lower.Decision patterns and class-level discrimination remained largely stable, indicating that gated fusion contributed differently depending on the distillation objective.Source: §3.4.2 · pp. 37, 39, 41, Table 10 · p. 41

3.4.3. Effectiveness of linguistic augmentation

• Linguistic augmentation reduced teacher and student performance rather than improving generalization; the best augmented Cosine student reached 85.71% accuracy and 0.8569 F1-score.Its F1-score declined from 0.8935 on the original dataset, likely because generated samples were semantically similar and added limited discriminative information.Source: §3.4.3 · pp. 41, 42, 44, 45, Table 11 · p. 44

3.5.1. Qualitative explainability

• Cosine and SupCon produce only one LIME-unmatched sample each, indicating the strongest teacher–student explanation consistency among the evaluated objectives.Unmatched explanations remain limited to a few classes, with most concentrated in White Feces Disease for KL, Triplet, and JSD.Source: §3.5.1 · pp. 45, 47
• Across all ten disease classes, Cosine KD highlights clinically meaningful symptoms and disease-specific terminology relevant to each diagnosis.Examples include “chấm” and “trắng” for White Spot Disease, “mang” for Black Gill Disease, and “đuôi” with “đỏ” for Taura Syndrome.Source: §3.5.1 · pp. 48–49, Figure 25 · p. 50

3.5.2. Global explanation using SHAP

• 0.0212 average SHAP makes JSD the strongest attribution objective, whereas Cosine and Triplet yield lower averages of 0.0097 and 0.0093.JSD also has the largest SHAP standard deviation, 0.0457, while Triplet has the smallest; JSD’s distribution is broadest, whereas Cosine and Triplet are narrower and lower.Source: §3.5.2 · p. 49, Table 12 · p. 49, Figure 26 · p. 51
• 0.8935 F1-score makes Cosine the top-performing objective despite its low average SHAP value, showing that attribution strength and predictive performance are complementary.SupCon combines relatively low attribution with the lowest F1-score, while JSD has stronger attribution but lower F1 than KL, MSE, and Cosine.Source: §3.5.2 · pp. 49, 51, Figure 27 · p. 52

3.5.3. Keyword deletion faithfulness

• Removing LIME-selected Top-k keywords reduces classification accuracy substantially more than random deletion across all distillation objectives, supporting explanation faithfulness.The gap widens as more keywords are removed, and explanation-guided deletion also causes larger predicted-class confidence reductions; JSD has the largest confidence drop.Source: §3.5.3 · pp. 51–52, Figure 28 · p. 53

3.6. Computational efficiency

• The distilled student reduces computational cost from 43.49 to 14.50 GFLOPs and GPU latency from 13.34 to 3.83 ms while maintaining competitive predictive performance.CPU latency falls from 518.01 to 175.67 ms, throughput rises from 1.93 to 5.69 samples/s, and model size decreases from 515.01 to 304.73 MB; shared architecture with SFT attributes gains to knowledge distillation rather than increased capacity.Source: §3.6 · p. 53
• 11.45M parameters and 996.21 samples/s make TF–IDF + DNN the cheapest and fastest baseline, but its efficiency comes with lower predictive performance.F2LLM-v2-80M has comparable size to the student but requires 32.21 versus 14.50 GFLOPs and 5.53 versus 3.83 ms GPU latency while performing worse.Source: §3.6 · pp. 53–54, Table 13 · p. 53
• The proposed student offers a favorable predictive-performance and efficiency balance among evaluated transformer models, although mobile and resource-constrained deployment remains untested.The reported efficiency gains were measured on desktop hardware.Source: §3.6 · p. 54

Overall distillation findings

• The 79.88M-parameter distilled student achieves performance comparable to PhoBERT-base while substantially reducing computational cost, and PhoBERT-base transfers knowledge more effectively than mBART.Effective teachers require both predictive strength and semantic representations that compact students can efficiently learn.Source: §4 · p. 54

Distillation objectives

• Cosine loss delivers the best overall classification performance and highest LIME explanation consistency, whereas SupCon, Center, and Triplet generally perform worse.Preserving angular similarity appears more effective than imposing explicit metric constraints for this task.Source: §4 · p. 54

Limitations and future directions

• Larger teachers do not consistently improve transfer, gated fusion depends on the loss, and linguistic augmentation reduces performance; limitations include small Vietnamese-only data and untested mobile deployment.Future work should address multilingual datasets and multimodal diagnosis combining text with shrimp images.Source: §4 · pp. 55–56

Baseline comparisons

• Supervised fine-tuning underperforms the distilled student, while TF–IDF + DNN and F2LLM-v2-80M offer weaker efficiency or predictive-performance trade-offs.These comparisons attribute the gains to knowledge transfer rather than increased architecture capacity.Source: §4 · p. 55

Explainability analysis

• LIME shows teacher-like explanations and clinically meaningful keywords, SHAP reveals objective-specific attributions, and keyword deletion causes greater degradation than random deletion.JSD achieves the highest average SHAP score, but Cosine remains best for classification, showing that attribution strength alone does not determine predictive quality.Source: §4 · pp. 55–56

Overall findings

• SALT consistently outperforms supervised fine-tuning and competitive baselines while substantially reducing computational cost relative to the teacher model.PhoBERT-base transfers knowledge more effectively than mBART, while larger teacher capacity does not consistently improve the distilled student.Source: §5 · p. 56

Dataset contribution

• ShrimpCap is the first Vietnamese dataset dedicated to shrimp disease symptom descriptions, comprising 979 manually validated samples across ten major diseases.Source: §5 · p. 56

Distillation objectives

• Among seven distillation objectives, Cosine loss achieves the best overall classification performance while maintaining high teacher–student explanation consistency.JSD achieves the highest average SHAP score despite not producing the best classification performance.Source: §5 · pp. 56–57

Explainability analysis

• LIME shows that the student generally preserves the teacher’s explanation patterns, while qualitative visualizations identify clinically meaningful disease-specific keywords.Source: §5 · p. 56
• SHAP attribution patterns differ across distillation objectives, and keyword deletion experiments provide additional evidence that the generated explanations are faithful.Source: §5 · pp. 56–57

6. Future work

• Future research will diversify data across farming areas, disease stages, and expression styles to expand model applicability.The proposed expansion addresses limitations in dataset coverage and linguistic diversity.Source: §6 · p. 57
• The student model should be deployed and evaluated in real-world, limited-resource environments to verify speed, memory consumption, and energy efficiency.This evaluation is intended to confirm practical advantages beyond experimental settings.Source: §6 · p. 57
• Semi-supervised and unsupervised knowledge distillation will be explored to reduce labeled-data requirements, costs, and implementation time.These techniques are proposed as alternatives to the current distillation approach.Source: §6 · p. 57
• Intrinsic explainability research will aim to reduce reliance on post-hoc LIME and SHAP while improving model transparency and reliability.The goal is to make interpretability part of the model rather than depending primarily on external explanation methods.Source: §6 · p. 57

• Anh Nguyen Quynh contributed to conceptualization, data curation, methodology, software, visualization, and writing the original draft.Source: Declarations · p. 57
• Nguyen Quoc contributed to methodology, supervision, validation, and writing review and editing.Source: Declarations · p. 58
• Luyl-Da Quach contributed to project administration, supervision, validation, and writing review and editing.Source: Declarations · p. 58