Source-linked AI summary
Analysis of Prompt Engineering for Drug Toxicity Prediction
Mia MacGregor, Aakash Welgamage Don, Mark Bartlett
TL;DR
Drug toxicity prediction matters because clinical trials are costly, yet LLM outputs can vary with prompt phrasing and the effect of that variation is understudied. The paper tests prompt, role, model, and stochastic variation by using LLM-identified features to build toxicity datasets, then finds that chemoinformatics extraction outperforms LLM-generated values while prompt fine-tuning has limited effect.
Problem
LLM responses can vary substantially with prompt phrasing, but the effect of prompt engineering on drug-toxicity prediction is often not considered.
Method
The study prompts multiple LLMs to identify toxicity-related chemical features, constructs labelled datasets from SMILES, and evaluates machine-learning toxicity models across prompt and extraction conditions.
Results
Chemoinformatics feature extraction outperformed LLM-generated feature values, while natural LLM response variation outweighed fine-tuning of prompts.
Takeaways & Limitations
For this toxicity-prediction setup, fine-tuning prompts may not improve prediction quality as productively as using chemoinformatics code for feature extraction.
Takeaways & Limitations
The study’s scope is bounded by its use of SMILES representations and the toxicity classifications determined by the training data.
Abstract
from arXiv · showhide
Clinical trials in the UK can cost up to £1.3 million, with approximately 90% drug failure rate. Toxicity is a major contributing factor in drug failure. Testing is time and cost intensive. In recent years, the use of artificial intelligence has been increasingly explored to aid in the prediction of drug toxicity, with extensive use of large language models (LLMs). However, LLMs can show considerable variation when minor changes are made to prompts, which raises concerns about their sensitivity to prompt engineering. Prompt engineering is used to optimise a prompt given to an LLM to generate the desired output. This paper proposes a method to analyse prompt engineering for drug toxicity prediction. The aim of the paper is to investigate the importance of prompt phrasing for drug toxicity prediction. LLMs were prompted to identify chemical properties of significance when predicting drug toxicity. Prompts were constructed to investigate; job role, prompt structuring, and rule interpretation. LLMs were then used to generate datasets, using the identified features from initial prompting, which were then passed to machine learning algorithms. The experiments show that the natural variance which occurs in LLMs outweighs any fine-tuning of prompts. There were, however, substantial improvements in model performance when using chemoinformatic code to extract features instead of using LLM-generated values. The proposed analysis methodology is applicable to a wide range of prompt types across different areas of bioinformatics.
1 Introduction
Drug toxicity contributes to costly clinical-trial failures, while LLM-based toxicity prediction can vary substantially with prompt phrasing. The paper therefore examines how prompt engineering affects LLM outputs and downstream toxicity prediction.
- Motivation: UK clinical trials can cost up to £1.3 million, and toxicity is a major reason drugs fail in clinical trials.Wet-lab toxicity prediction also carries considerable time and cost burdens.
- Motivation: AI methods, including LLMs, are increasingly used to predict drug toxicity before clinical trials.
- Research gap: LLM responses can differ substantially with prompt changes, but prompt phrasing is often not considered in toxicity-prediction studies.
- Research aim: The paper proposes analysing prompt engineering and its effects on LLM output variation and subsequent toxicity-prediction accuracy.Prompt engineering is defined as optimising an LLM output.
- Scope: Toxicity categories depend on the training data, so different datasets allow the approach to examine specific toxicity types.
2 Methods and Results
The study prompts multiple LLMs to identify chemical features, uses those outputs to construct toxicity datasets, and evaluates machine-learning models across prompt, role, model, and randomness conditions. Results show substantial feature variation and better AUC when chemoinformatics code extracts features instead of LLMs.
- Methods: The pipeline asks an LLM for 15 toxicity-related features, generates feature values for PubChem SMILES, appends toxicity labels, and trains machine-learning models.The described workflow uses 80% of the data for training, though the supplied passage truncates the remaining split description.
- Prompt design: Prompts vary job role, prompt structure, and rule interpretation, while Table 1 reports 20 prompt variants across four job roles.
- Prompt design: Five LLMs were selected to investigate output variance under identical prompts rather than to produce a state-of-the-art result.
- Feature generation: LogP appeared in every feature list, while only 7 features occurred in more than 25% of lists, indicating substantial variation in LLM-identified features.
- Dataset construction: 18 datasets were generated from 2,294 SMILES containing 789 toxic and 1,505 non-toxic compounds to study LLM, role, prompt, and randomness effects.
- Evaluation and results: Chemoinformatics-extracted features outperformed LLM-generated features by AUC, while randomness changed performance, LLM choice had little impact, and no job variant consistently performed best.The evaluation also used permutation feature importance for LogP and molecular weight.
3 Conclusions
The study concludes that fine-tuning prompts has limited effect on drug-toxicity prediction because natural LLM response variation is comparable to changes in prompts or LLMs. Chemoinformatics feature extraction performs better than LLM extraction, while future work should test broader representations and feature-engineering methods.
- Conclusions: Simple prompt fine-tuning did not have large effects on LLM output, and random response variation was similar to variation from changing prompts or LLMs.
- Conclusions: Chemoinformatic feature extraction produced better toxicity-prediction performance than LLM-based feature extraction.
- Future work: Future work should examine representations other than SMILES, more complex feature engineering such as retrieval-augmented generation, and significance testing.