MI-PEFT: Mixture-of-Experts Integrated Parameter-Efficient Fine-Tuning Protein Language Models Improves Acidophilic Proteins Classification

1 Introduction

• Identifying acidophilic proteins has relied on labor-intensive, time-consuming, and costly wet-lab experiments, despite their value in industrial biocatalysis and acid-related bioprocessing.Acidophilic proteins maintain structural integrity and catalytic function under extremely low pH conditions.Source: §1 · pp. 1–2
• Handcrafted sequence summaries may miss context-dependent signals relevant to acidophilic adaptation, motivating richer protein-language-model representations.Prior work reported that pretrained protein-language-model embeddings support downstream acidophilic-protein classification, including with lightweight classifiers.Source: §1 · p. 2
• MI-PEFT adapts ESM C using LoRA-based PEFT methods and replaces a linear classification head with a DeepSeekMoE router that activates only a small subset of experts per sample.The framework also applies post-hoc temperature scaling to validation-set predictions and evaluates calibrated logits on the test fold.Source: §1 · p. 2
• C3A achieved the best performance among the six approaches, which outperformed basic machine-learning methods, deep-learning methods, and the previous SOTA baseline.The experiments used 10-fold cross-validation on the training set and a held-out independent test set, evaluating six standard metrics.Source: §1 · p. 2

2.1 Dataset

• A previous study collected a large set of candidate acidophilic proteins by searching for proteins annotated as acidophilic, naturally producing far more positives (237,109) than negatives (7,394).Source: §2.1 · p. 3
• The non-acidophilic contrast set used characterized thermophilic, halophilic, and alkaliphilic proteins rather than treating every unannotated protein as negative.The passage notes that unannotated proteins may have partial acid tolerance or belong to mixed categories such as thermo-acidophilic proteins.Source: §2.1 · p. 3
• Curation removed fragments shorter than 100 residues, then used MMseqs2 with UniqueProt to reduce homologous overlap, limiting sequence identity within the test set and between training and test sets to 20%.The procedure aimed to preserve diversity in the training data.Source: §2.1 · p. 3
• The final curated dataset contained 4,089 acidophilic proteins and 1,654 non-acidophilic proteins, then split into training and testing subsets in an 80/20 ratio.The independent test set contained 818 positive and 331 negative samples.Source: §2.1 · p. 3

2.2 Protein Language Model

• ESM C-600M serves as the study’s pretrained model for generating protein representations used in classification.The passage describes ESM C as an upgraded ESM-family model trained at larger dataset and computational scales than ESM2.Source: §2.2 · p. 3
• ESM C embeddings capture higher-order contextual and evolutionary patterns, including motif–context interactions, non-local residue couplings, and family-level signals.These signals are described as emerging from shared evolutionary constraints.Source: §2.2 · p. 3
• The model provides pretrained attention and feed-forward parameters that encode general protein regularities, rather than learning representations from scratch on the labeled acidophilic dataset.The passage presents these pretrained representations as an initialization for supervised classification.Source: §2.2 · pp. 3–4

2.3 Parameter-Efficient Fine-Tuning

• PEFT adapts the model by training a small set of additional parameters while keeping most pretrained weights frozen, addressing the cost of end-to-end updates for large backbones and long sequences.The study controlled the trainable-parameter proportion within 3% for all six methods.Source: §2.3 · p. 4
• The study benchmarked six PEFT methods: LoRA, MiSS, C3A, OFT, RandLoRA, and SHiRA.LoRA parameterizes its weight increment as ΔW = BA, with rank much smaller than the minimum of input and output dimensions, and optimizes A and B.Source: §2.3 · p. 4
• PEFT modules replaced selected high-dimensional linear layers, including query, key, value, attention-output, and major feed-forward transformations, while backbone parameters remained largely frozen.Table 1 compares PEFT methods by their forward mappings and initialization schemes.Source: §2.3 · p. 4, Table 1 · p. 4

2.4 DeepSeekMoE

• The dense classifier’s 2304 × 4608 mapping contributes about 1.06 × 10^7 parameters, comparable to LoRA’s roughly 1.2 × 10^7 trainable parameters.Thus, the head can dominate the trainable budget even when the backbone is adapted efficiently.Source: §2.4 · p. 5
• Replacing the dense classifier with a DeepSeekMoE head uses gated routing to activate only the top-k experts for each pooled representation.The implementation uses three routed experts, selects two per input, and models each as a lightweight two-layer MLP.Source: §2.4 · p. 5
• A shared expert branch remains active regardless of routing, while an auxiliary load-balancing objective discourages usage concentrating on only one or two experts.The passages identify this balance as particularly useful in imbalanced classification, where unstable routing may weaken the expert structure’s benefit.Source: §2.4 · p. 5
• The design significantly decreases trainable parameters, allocated GPU memory, and training time, with Table 2 reporting these resource measures for six MI-PEFT methods.The shared branch provides stable capacity independent of routing while routed experts enable conditional computation.Source: Table 2 · p. 5, §2.4 · p. 5

2.5 Temperature Scaling

• Temperature Scaling adjusts softmax probabilities with a single positive temperature to soften or sharpen the output distribution.The paper motivates this step because softmax probabilities may not accurately reflect true likelihood.Source: §2.5 · p. 6
• The optimal temperature is learned by minimizing validation-set negative log likelihood while all other model parameters remain frozen.The optimization uses L-BFGS.Source: §2.5 · p. 6
• Within each fold, the learned temperature calibrates independent-test predictions before final metric computation and probability-based analyses.The temperature is fitted using validation logits and their true labels.Source: §2.5 · p. 6

2.6 Performance Evaluation

• The evaluation uses 10-fold cross-validation and a fully sequestered independent test set to estimate generalization to unseen data and mitigate overfitting risk.Source: §2.6 · p. 6
• Six metrics are calculated from confusion matrices: Accuracy, Sensitivity, Specificity, Matthews Correlation Coefficient, F1-Score, and AUC.The confusion matrix also exposes true and false positives and negatives, allowing inspection of class-specific prediction bias.Source: §2.6 · p. 6
• UMAP projects final-hidden-layer embeddings into two dimensions to assess class separation and ambiguous regions in feature space.Source: §2.6 · p. 7
• Prediction-confidence distributions, defined by the predicted class’s softmax probability, are examined to assess calibration and confidence reliability.Source: §2.6 · p. 7

2.7 Experimental Setup

• Models are trained and evaluated in Python with PyTorch and Hugging Face Transformers on a single NVIDIA GeForce RTX 5090 GPU.Training uses mixed-precision computation via PyTorch Automatic Mixed Precision.Source: §2.7 · p. 7
• Optimization uses AdamW, a maximum sequence length of 512 tokens, and a base learning rate of 5 × 10^-5, or 5 × 10^-4 for C3A.Weight decay is 0.01, and the learning rate warms up linearly over the first 10% of optimization steps before decaying linearly to zero.Source: §2.7 · p. 7
• The effective batch size is 32, achieved through gradient accumulation to fit within GPU memory constraints.Source: §2.7 · p. 7
• Training runs for at most 10 epochs per cross-validation fold, with patience-three early stopping on validation cross-entropy loss and restoration of the lowest-loss epoch’s weights.The key hyperparameter configurations for each PEFT method are listed in Table 3.Source: §2.7 · pp. 7–8, Table 3 · p. 8

2.8 Baseline Models

• The baselines assess how commonly used classifiers capture discriminative signals without task-specific fine-tuning of large pretrained models.Source: §2.8 · p. 8
• Six classical machine-learning methods and three deep-learning baselines were compared with the proposed method under consistent data splits and evaluation criteria.The classical methods include logistic regression, SVM, random forest, XGBoost, and LightGBM; the deep-learning models are DNN, LSTM, and BiLSTM.Source: §2.8 · p. 8

3.1 The dataset distribution demonstrates class imbalance

• Acidophilic proteins outnumber negative samples by roughly three to one, and naive optimization may favor the majority class at specificity’s expense.Most proteins are shorter than 200 amino acids; positive samples dominate that length range, while class composition profiles overlap substantially despite modest residue-frequency differences.Source: §3.1 · pp. 8–9

3.3 C3A performs the best on both ten-fold cross-validation and an independent test set

• Among six PEFT strategies, C3A stood out in specificity and MCC, using a relatively small proportion of trainable parameters and low GPU memory but taking more training time.Source: §3.3 · p. 11, Figure 2 · p. 11
• On the independent test set, MI-PEFT(C3A) showed consistent advantages over classical and deep-learning baselines, especially in specificity, while the classical models showed deficiencies in specificity and MCC.The comparisons include an ACE model fine-tuned on the same ESM C backbone and evaluated on the same independent set.Source: §3.3 · p. 12
• C3A achieved the most favorable overall performance, particularly in specificity and MCC, and MI-PEFT(C3A) scored best or near-best across six metrics on the independent test set.Source: §3.3 · p. 12, Figure 3 · p. 13

3.5 Ablation studies of MI-PEFT show the effectiveness of components

• DeepSeekMoE scored highest across all six metrics, while MLP and Vanilla MoE lagged especially in sensitivity and MCC.The authors describe a shared expert pathway as providing a representation trained by all samples, with an auxiliary load-balancing loss also used in DeepSeekMoE.Source: §3.5 · pp. 13–14
• Under MI-PEFT(C3A), ESM C-600M performed strongest across metrics, especially MCC; ESM C-300M ranked second, while ESM2-3B did not outperform ESM2-650M or ESM2-150M.The results suggest parameter scale alone is insufficient for downstream performance in this PEFT setting.Source: §3.5 · p. 14
• MI-PEFT(C3A) produced more compact, class-consistent UMAP clusters and broader, flatter confidence distributions than ACE, which the authors say can guide experimental candidate selection.Source: §3.5 · p. 15

• MI-PEFT combines pretrained representations, constrained adaptation, and expert routing, balancing predictive performance with computational efficiency.Source: Conclusion · p. 16
• C3A achieves the most favorable overall performance among six PEFT methods, particularly in specificity and MCC for this class-imbalanced task.Source: Conclusion · p. 16
• DeepSeekMoE is reported to enhance the model’s capacity to capture heterogeneous sequence patterns.Source: Conclusion · p. 16
• Temperature scaling improves the reliability of prediction confidence.Source: Conclusion · p. 16
• The framework is presented as a practical acidophilic-protein identification tool and a paradigm for PEFT adaptation in limited, imbalanced-data settings.Source: Conclusion · p. 16