Large Language Models Streamline Automated Machine Learning for Clinical Studies

Introduction

• Complex ML development, implementation, and validation remain inaccessible to many clinicians and medical researchers, while existing AutoML tools do not use natural-language commands converted to Python code.The paper positions ChatGPT ADA as an accessible, conversational tool that can analyze data and write and execute code.Source: Introduction · p. 2
• The study asks whether ChatGPT ADA can autonomously develop and implement ML methods reliably, testing four large clinical-trial datasets from varied medical specialties.The authors hypothesized that it could do so efficiently and accurately without prior ML training, resources, or specific guidance.Source: Introduction · p. 2

Results

• For metastatic disease, ChatGPT ADA selected gradient boosting, whose AUROC was similar to the optimized AdaBoost re-implementation (0.949 vs. 0.951; P=0.464).The original published AdaBoost model had outperformed clinical experts in its study.Source: Results · p. 5
• In hereditary hearing loss, ChatGPT ADA's random forest had higher AUROC than the original model (0.773 vs. 0.751), but lower accuracy (0.767 vs. 0.812) and F1-score (0.845 vs. 0.861).Its AUROC was not significantly different from the validatory re-implementation (P=0.624).Source: Hereditary Hearing Loss… · p. 9
• Across datasets, ChatGPT ADA-selected models performed on par with or better than hand-crafted models, with no significant differences from optimized re-implementations across the reported metrics.The evaluation used datasets from original studies to assess predictions on previously unseen data.Source: Table 3 · p. 8, Discussion · p. 12

Discussion

• The authors used SHAP to assess feature contributions; across trials, the identified predictors included clinical measurements, cytologic features, gene variants, and medical history.Examples included age and laboratory values, atypical glandular cells, the c.235delC variant, and prior cardiomyopathy.Source: Cardiac Amyloidosis [Cardiology] –… · p. 10, Discussion · p. 14
• The findings are limited by uncertainty about performance on less-curated datasets and sensitivity to prompting; the authors call for validation across medical domains.They also note possible training-data overlap and inherent limits to comparability between model implementations.Source: Discussion · p. 14
• The authors describe ADA as a way to simplify complex ML and broaden access, while stressing that it should enhance rather than replace specialized training, resources, and guidance.They also identify privacy, security, interpretability, reliability, and ethical concerns in clinical use.Source: Discussion · pp. 12–14

• The retrospective study was approved by the Medical Faculty of RWTH Aachen University ethics committee (Reference No. EK 028/19).Source: Ethics Statement · p. 15

Patient Cohorts

• Across four studies, public-repository datasets covered metastatic disease, oesophageal cancer screening, hereditary hearing loss, and cardiac amyloidosis.Source: Patient Cohorts · p. 15

Experimental Design

• Separate chat sessions used the original training and test data; ADA received study context and was asked to select a model and generate test-set predictions without ML-specific guidance.Each trial began in a new session to avoid memory-retention bias, and each dataset took less than five minutes to input and analyze.Source: Experimental Design · p. 16

Data Pre-Processing and ML Model Development

• A data scientist re-implemented and optimized each original study’s best-performing model; the team also checked ADA’s intermediate code and verified its SHAP outputs.The re-implementation enabled head-to-head comparisons because original patient-level predictions were unavailable.Source: Data Pre-Processing and ML Model… · pp. 16, 18, 19

Reproducibility Analysis

• Across three consecutive days, separate sessions using the same data, instructions, and prompts produced the same responses and qualitative and quantitative findings.Source: Reproducibility Analysis · p. 19

Statistical Analysis and Performance Evaluation

• Performance metrics were evaluated with paired bootstrapping on test sets, 1,000 redraws, false-discovery-rate adjustment, and a familywise alpha threshold of 0.05.F1-score, sensitivity, and specificity thresholds were chosen using Youden’s criterion.Source: Statistical Analysis and Performance… · p. 19

Data Availability

• The clinical-trial datasets were obtained from public repositories, including Zenodo, GitHub, Mendeley, and the original cardiac amyloidosis study.Source: Data Availability · p. 20

Hardware

• Experiments used an eight-core Intel CPU with 16 GB RAM and no GPU.Source: Hardware · p. 20

• STA received RACOON and BMBF support; SN and DT received DFG grants, and DT also received federal and Horizon Europe support.Source: Funding Sources · p. 20

Competing Interests

• JNK disclosed consulting, shares, and lecture honoraria, DT disclosed shares, and the other authors declared no competing interests.JNK’s consulting and honoraria involved named companies; both JNK and DT held shares in StratifAI GmbH.Source: Competing Interests · p. 21

Supplementary Information

• For metastatic-disease prediction, the user requested training on internal-validation patients and testing on external-validation patients, with labels and probabilities returned for independent evaluation.The test set was unlabeled to ChatGPT ADA; the user planned to calculate AUC, accuracy, F1 score, sensitivity, and specificity using ground-truth labels.Source: Supplementary Information · p. 25
• The proposed workflow separated IT and EV patients, handled missing values, processed numerical and categorical features, trained a binary classifier, and exported test predictions.The user described the data as a cross-sectional PMT-trial cohort from Germany, Poland, and the Netherlands; ADA proposed loading, filtering, preprocessing, model selection, training, prediction, and export.Source: Supplementary Information · p. 25
• The training data contained 21 missing values for primary-tumor spherical volume, and ADA proposed median imputation for numerical features.It also identified one missing value each for age at diagnosis, plasma NMN, and plasma MTY, and six for plasma MN; it proposed standardizing numerical features and encoding categorical features.Source: Supplementary Information · p. 26
• ADA saved test predictions with associated probabilities and asked the user to compute evaluation metrics from ground-truth labels.When asked for the trained model’s parameters, it reported an issue accessing them and subsequently described the settings as defaults.Source: Supplementary Information · pp. 26–28
• ADA initially used default Gradient Boosting Classifier parameters without grid search or an explicitly reserved validation set.It described validation data as typically useful for hyperparameter tuning and reported defaults including max_depth 3, 100 trees, learning_rate 0.1, and loss 'deviance'.Source: Supplementary Information · pp. 27–28

1. Maximum Depth of Individual Trees (max_depth): 3

• Maximum tree depth controls how deep each tree can grow during a boosting round.The section title specifies max_depth as 3.Source: §1 · p. 28

2. Number of Trees (n_estimators): 100

• The number of trees specifies how many boosting stages will run.The section title specifies n_estimators as 100.Source: §2 · p. 28

3. Minimum Samples per Leaf (min_samples_leaf): 1

• The setting specifies the minimum number of samples required at a leaf node.Source: §3 · p. 28

4. Minimum Samples per Split (min_samples_split): 2

• The setting specifies the minimum number of samples required to split an internal node.Source: §4 · p. 28

5. Learning Rate (learning_rate): 0.1

• The 7,899-row training set had 150 columns, no missing values, and severe class imbalance: 97.46% without high-grade lesions versus 2.54% with them.The model was intended to predict whether high-grade lesions were present.Source: §5 · pp. 29–30
• ADA chose Gradient Boosting Classifier for modeling; the test dataset had 6,698 entries and 172 columns, more than training, while CheckID was object versus float64 in training.Source: §5 · pp. 31–32
• The classifier achieved approximately 99.48% training accuracy, while the training-data threshold of approximately 0.1 yielded about 98.59% accuracy.The threshold was chosen to maximize sensitivity plus specificity while maintaining good accuracy.Source: §5 · pp. 32–33
• No separate validation set was used: ADA trained on the entire training dataset and optimized the threshold on that same data.ADA noted that setting aside a validation portion would help evaluate performance on unseen data and reduce overfitting risk.Source: §5 · p. 35
• The default learning rate was 0.1, shrinking each tree’s contribution as regularization; smaller values may reduce overfitting but typically require more trees.The model used 100 trees, and no grid search was performed.Source: §5 · pp. 34, 36

• The Gradient Boosting configuration used subsample=1.0, max_depth=3, min_samples_leaf=1, min_samples_split=2, deviance/log loss, and no tuned alpha or lambda.Source: §5 · p. 36
• ADA used the Discovery Set to train a model and returned validation-set predictions in a CSV with predicted labels and associated probabilities.Source: §5 · pp. 38–39
• ADA handled whitespace or nonnumeric Discovery Set cells by filling them with zeros, assuming those entries indicated absent gene variants.Source: §5 · p. 38
• The Random Forest achieved approximately 74.53% average accuracy on the Discovery Set in 5-fold cross-validation.Source: §5 · p. 38
• For the Random Forest, learning rate and loss were not applicable; ADA used default parameters, did not perform grid search, and validated with cross-validation on the Discovery Set.Source: §5 · p. 39

1. Model Type: Random Forest Classifier

• Parameter descriptions list both 100 and 1,000 trees; another response says the model used scikit-learn defaults and did not use grid search.The 1,000-tree response attributes the absence of grid search to time and computational costs in a high-dimensional feature space.Source: §1 · pp. 40, 44
• One passage describes 5-fold cross-validation on the Discovery Set, while another says no separate validation set was used for this instance.The latter response says hyperparameter tuning and overfitting prevention would typically use a validation set or cross-validation.Source: §1 · pp. 40, 44
• For missing binary genetic-variant data, ADA justified zero imputation as matching variant absence, while noting that other data types may call for other strategies.The dialogue cautions that median imputation could yield 0 or 1 and might introduce bias in this setting.Source: §1 · pp. 40–41
• With 1,712 training samples and 430 test samples, each having 1,874 features, ADA switched to Random Forest after LightGBM was unavailable and thresholded probabilities at 0.5.The initial plan was to use LightGBM; probabilities at or above 0.5 yielded label 1, and lower probabilities yielded label 0.Source: §1 · p. 43
• The Random Forest produced a CSV of test-patient IDs, predicted labels, and associated probabilities.The user planned to calculate evaluation metrics using the test-set ground truth.Source: §1 · pp. 42–43
• ADA said feature scaling was unnecessary for Random Forest and described averaging decorrelated trees as inherent regularization, with no additional regularization applied.The response contrasts tree-based methods with algorithms sensitive to feature scales.Source: §1 · pp. 40, 44