PaperlayerView exact revision on arXiv

Source-linked AI summary

Rigorous Evaluation of Large Language Models for Malaria Drug Discovery: Trade-offs in Performance, Scale, and Resource Utility

Marvellous O. Ajala, Zainab Ashimiyu-Abdusalam, Comfort Adesina

arXiv:2608.20418v1q-bio.QMcs.AIcs.LG

TL;DR

Reliable antimalarial bioactivity prediction requires evidence beyond generic few-shot prompting, motivating a curated dataset and rigorous structurally dissimilar benchmark. The study evaluates fine-tuned and few-shot models against classical and proprietary baselines, finding metric-dependent advantages among fine-tuned open-source LLMs and scope boundaries requiring experimental validation.

  • Problem

    Reliable LLM-based antimalarial virtual screening requires evaluation under structurally novel conditions and clarification of fine-tuning’s value relative to few-shot prompting and other baselines.

  • Method

    The study constructs Malaria-Instruct from ChEMBL Legacy Malaria and evaluates five open-source LLMs, classical models, and proprietary models using Lo-Hi dissimilarity-based splitting.

  • Results

    Fine-tuned open-source LLMs led the benchmark overall: TxGemma 9B achieved ROC-AUC 0.7315 ± 0.0053, while LlaSMol-Mistral achieved EF1% 4.9865 ± 0.0048.

  • Takeaways & Limitations

    TxGemma 9B is favored for global discrimination, whereas LlaSMol-Mistral is favored for prospective screening focused on the top-ranked fraction.

  • Takeaways & Limitations

    The dataset covers historically screened antimalarial chemotypes and SMILES representations, while prospective utility still requires wet-lab confirmation.

Abstract

from arXiv · show

We introduce Malaria-Instruct, a curated instruction-following dataset derived from the ChEMBL Legacy Malaria corpus for Malaria virtual screening, and conduct a systematic evaluation of five open-source LLMs; Gemma-2 2B/9B, TxGemma-2B/9B, and LlaSMol-Mistral-7B, on a rigorous out-of-distribution data split. Performance was benchmarked against classical ML models (Random Forest, XGBoost) and frontier proprietary models (Gemini 2.5, OpenAI o3) under few-shot conditions. Fine-tuned LLMs substantially outperformed all baselines: TxGemma-9B achieved the highest ROC-AUC ($0.731 \pm 0.005$) and LlaSMol-Mistral-7B the best enrichment factor (EF@1\% $\approx$ 4.99). Domain-specific fine-tuning proved categorically indispensable with TxGemma-9B collapsing from ROC-AUC 0.731 to 0.499, under its best few-shot condition, and neither Gemini 2.5 (ROC-AUC $\approx$ 0.53) nor o3 (ROC-AUC $\approx$ 0.59) achieved reliable discrimination without fine-tuning. Biomedical pretraining conferred a measurable advantage at equivalent scale, while chemistry-aware pretraining yielded superior prospective enrichment. Fine-tuned open-source LLMs represent a compelling, resource-efficient paradigm for antimalarial VS, outperforming both classical pipelines and proprietary reasoning models under structurally challenging conditions.

1 Introduction

Malaria’s substantial global burden and emerging artemisinin resistance motivate structurally novel chemotype discovery through virtual screening. The study frames domain-specific fine-tuning and dissimilarity-based evaluation as central requirements for reliable molecular prediction.

  • 263 million cases and 597,000 deaths were recorded globally in 2023, with 94% of cases and 95% of deaths in the WHO African Region.
  • Partial artemisinin resistance motivates identifying structurally novel chemotypes through ligand-based virtual screening.Virtual screening ranks molecules by predicted bioactivity, with EF@1% emphasized for experimental hit selection.
  • SMILES-based language modelling has produced chemistry-aware instruction-following models including LlaSMol, TxGemma, and Tx-LLM.
  • Fine-Tuning: Parameter-efficient fine-tuning internalises structure–activity relationships that few-shot prompting may not recover from a handful of SMILES-label pairs.
  • Evaluation Framework: Lo-Hi splitting enforces strong train–test and train–validation dissimilarity so reported metrics reflect out-of-distribution generalisation.
  • Research Contribution: The benchmark compares fine-tuned open-source LLMs with frontier proprietary models and classical cheminformatics baselines under rigorous evaluation conditions.

2 Related Works

The study addresses curation and evaluation gaps in molecular instruction datasets by constructing Malaria-Instruct specifically for antimalarial virtual screening. Its design combines assay-aware curation with structurally dissimilar splitting and builds on prior chemistry language models.

  • Dataset curation addresses duplicate bioactivity records, assay heterogeneity, and class imbalance through assay-level deduplication, readout harmonisation, and negative augmentation.
  • Malaria-Instruct is specifically constructed for antimalarial virtual screening rather than general molecular instruction tasks.It provides assay-level contextualisation for Plasmodium strain, assay duration, and mechanistic context.
  • LlaSMol models were fine-tuned on SMolInstruct, with the Mistral variant previously identified as a strong chemistry LLM.
  • Random splitting can inflate molecular-model performance because structurally similar compounds appear across training and test sets.

3 Methodology

The methodology constructs and standardises Malaria-Instruct, harmonises assay records, enforces structurally dissimilar partitions, and evaluates classical models, open-source LLMs, and proprietary models through fine-tuning or few-shot learning.

  • Dataset Curation: Malaria-Instruct is derived from the ChEMBL Legacy Malaria corpus as an instruction-following dataset for molecular bioactivity prediction.
  • Dataset Curation: Potency and IC50 assays were retained, while replicate molecule–assay records were aggregated using concordance-based activity labels.
  • Dataset Curation: 24-, 48-, and 96-hour assay readings were harmonised, with conflicting time-point readings discarded.
  • Dataset Curation: TxGemma prompts incorporated assay type, Plasmodium strain, and experimental context, unlike the LlaSMol prompt format.
  • Dataset Curation: Negative augmentation used chemically proximate discarded-assay molecules to restore the broader corpus’s positive-to-negative ratio.
  • Dataset Curation: Molecules were standardised through charge neutralisation, parent-fragment extraction, valence and functional-group normalisation, and canonical tautomer selection.
  • Data Splitting: Lo-Hi splitting constrained maximum train–test Tanimoto similarity to 0.4 and train–validation similarity to 0.55.
  • Model Evaluation: Evaluation covered fine-tuning and few-shot learning across classical baselines, open-source LLMs, and frontier closed-source models.

4 Results

Fine-tuned models led the benchmark’s principal metrics, while few-shot performance was generally weaker for open-source models and proprietary models showed limited discrimination.

  • TxGemma 9B achieved the best ROC-AUC at 0.7315 ± 0.0053.
  • LlaSMol-Mistral achieved the best EF1% at 4.9865 ± 0.0048 and the best MCC at 0.56415 ± 0.0257.
  • XGBoost achieved the fastest inference time at 0.00004 seconds per sample.
  • Except for LlaSMol, open-source few-shot accuracies remained below 0.3 for Gemma 9B, 0.08 for TxGemma 2B, and 0.008 for TxGemma 9B.
  • OpenAI o3 reached the highest closed-source few-shot ROC-AUC at 0.5907, while Gemini 2.5 reached the highest EF1% at 3.788.

5 Discussion

The discussion identifies fine-tuning, domain specialization, and metric choice as decisive for antimalarial virtual screening, while highlighting trade-offs between predictive performance, enrichment, and computational cost. It also emphasizes that structurally rigorous evaluation exposes limitations of both few-shot models and dataset coverage.

  • Fine-Tuning is Non-Negotiable: Few-shot in-context learning failed across model classes, including frontier proprietary models, whereas domain-adapted fine-tuning produced strong discrimination.TxGemma-9B fell to ROC-AUC ≈0.499 under 4-shot prompting despite achieving the strongest fine-tuned ROC-AUC.
  • Biomedical Pretraining: 0.731 ± 0.005 ROC-AUC made TxGemma-9B the strongest fine-tuned model, ahead of Random Forest at 0.683 and Gemma-2-9B at 0.638.The equivalent-scale comparison supports a measurable advantage for biomedical pretraining at 9B parameters.
  • Chemistry-Aware Pretraining and Enrichment: EF@1% ≈4.99 ± 0.005 made LlaSMol-Mistral the strongest enrichment model, exceeding TxGemma-9B at approximately 4.23, Random Forest at 2.58, and XGBoost at 1.79.Its ROC-AUC was 0.702, showing that global discrimination and extreme-tail enrichment can favor different models.
  • Metric Choice for Deployment: EF@1% is argued to be the operationally primary metric for virtual screening because only the top-ranked compounds typically proceed to experimental assay.This makes LlaSMol-Mistral preferable for prospective screening despite TxGemma-9B's ROC-AUC lead.
  • Evaluation Design: The benchmark's Lo-Hi splitting and independent resplitting distinguish genuine out-of-distribution evaluation from interpolation between structurally similar molecules.Random splitting would likely reduce the gap between classical and LLM-based models by permitting structural interpolation.
  • Resource Utility: Gemma-2-2B and TxGemma-2B occupy an efficiency frontier, combining approximately 4–7 ms per molecule with ROC-AUC > 0.695.Fine-tuned models required 16GB or 24GB GPUs for training and at least 24GB for inference, while LlaSMol-Mistral reached 0.080 seconds per molecule.
  • Limitations: The study's conclusions are bounded by Malaria-Instruct's exclusive derivation from the ChEMBL Legacy Malaria corpus and require wet-lab validation of top-ranked predictions.The corpus leaves previously unexplored chemical space unrepresented, and prospective experimental confirmation remains necessary.

6 Conclusion and Future Directions

Domain-specific fine-tuning is presented as essential for reliable antimalarial bioactivity prediction, while model advantages depend on the evaluation metric and model scale.

  • Domain-specific fine-tuning is a categorical prerequisite for reliable antimalarial bioactivity prediction with LLMs.
  • TxGemma-9B achieves the highest global discriminative performance, with ROC-AUC 0.731 ± 0.005.
  • LlaSMol-Mistral achieves the highest enrichment factor, with EF@1% 4.99 ± 0.005, and the highest MCC, 0.564 ± 0.026.
  • Biomedical and chemistry-aware pretraining confer measurable but metric-dependent advantages among fine-tuned models.
  • Classical machine learning baselines remain competitive reference points.

A. ChEMBL Dataset

ChEMBL is a large, open bioactivity resource assembled from medicinal chemistry literature, depositor submissions, and partner databases, with a Legacy Malaria dataset for antimalarial screening data.

  • ChEMBL is a manually curated, large-scale, open-access database integrating compound–target interaction data from multiple sources.
  • ChEMBL contains bioactivity data for over two million distinct compounds.
  • The ChEMBL Legacy Malaria dataset aggregates decades of antimalarial screening data.

B. Instruction Tuning and Dataset Curation for Scientific LLMs

Scientific LLMs use instruction tuning and curated molecular datasets to improve responses to specialized queries, with Tx-LLM and TxGemma extending biomedical task coverage.

  • B. Instruction Tuning and Dataset Curation for Scientific LLMs: Instruction tuning fine-tunes language models on curated instruction–response pairs to improve instruction following and task generalisation.
  • B. Instruction Tuning and Dataset Curation for Scientific LLMs: Scientific LLMs apply instruction tuning to answer molecular queries expressed in natural language.
  • C. Biomedical Domain Specific Pretrained Models: Tx-LLM was fine-tuned from PaLM-2 on 709 datasets spanning 66 drug-discovery tasks.
  • C. Biomedical Domain Specific Pretrained Models: Tx-LLM encoded knowledge about small molecules, proteins, nucleic acids, cell lines, and diseases in one set of model weights.
  • C. Biomedical Domain Specific Pretrained Models: Tx-LLM achieved near-state-of-the-art performance on 43 of 66 Therapeutics Data Commons tasks.
  • C. Biomedical Domain Specific Pretrained Models: TxGemma-9B-Predict improved over Tx-LLM on 45 of 66 TDC tasks.
  • C. Biomedical Domain Specific Pretrained Models: TxGemma-9B-Predict matched or exceeded best-in-class specialist model performance on 50 tasks.

D. Evaluation considerations

Evaluation metrics and baseline selection shape conclusions in virtual-screening benchmarks, especially under class imbalance and realistic molecular splits.

  • Accuracy can exceed 90% for a majority-class predictor while providing no practical virtual-screening utility.
  • ROC-AUC measures global discriminative capacity while remaining insensitive to class imbalance.
  • Fingerprint-based classical baselines can remain competitive with complex deep-learning approaches under realistic data-splitting conditions.

F. Training Parameters

All LLMs were fine-tuned with QLoRA using 4-bit quantisation, low-rank adaptation, and one training epoch over the full training partition.

  • QLoRA fine-tuning used 4-bit quantisation and LoRA rank 8 across attention and feed-forward projection modules.Target modules were q proj, o proj, k proj, v proj, gate proj, up proj, and down proj.
  • Training ran for one epoch over the full training partition.
  • Optimisation used paged AdamW 8-bit with learning rate 2 × 10^-4, weight decay 0.001, and two warmup steps.
  • The batch configuration used train batch size 2 with gradient accumulation of 2.

G. Inference Consequence of Few-shots resplitting

Independent resplitting across few-shot conditions makes near-random in-context performance robust to both shot count and partition changes, while deployment accessibility may be overstated.

  • Near-random ICL performance persisted despite independently changing the partition for each shot-count condition.This design prevents the result from being attributed to a fixed-partition artefact.
  • Performance degradation from 3-shot to 5-shot cannot be attributed to a fixed structural partition.
  • The observed failure is interpreted as intrinsic to the task-model mismatch rather than an accident of a particular structural partition.
  • Figures that are not reproducible in standard deployment contexts may substantially overstate the practical accessibility of LLM-based virtual screening.

H. Closed Source Few-Shot Variance

Closed-source few-shot performance was highly sensitive to example identity, while standard PyTorch inference introduces architectural inefficiency for short activity-label outputs.

  • Gemini 2.5 EF@1% ranged from 2.34 to 3.79 across duplicate 4-shot evaluations, with ±2.499 reported.Shot count and partition were held constant within duplicate pairs; only the few-shot example identities varied.
  • Sensitivity to individual few-shot examples is presented as evidence of unstable in-context bioactivity prediction rather than methodological noise.
  • Standard PyTorch allocates the key-value cache per sequence without batching or memory reuse across requests.This produces GPU idle time between token-generation steps, even for short outputs.
  • Activity-label prediction requires only one or two generated tokens per molecule, making cache initialisation a central inference consideration.
Loading 2608.20418v1…