Source-linked AI summary
Matching Patients to Clinical Trials with Large Language Models
Qiao Jin, Zifeng Wang, Charalampos S. Floudas, Fangyuan Chen, Changlin Gong, Dara Bracken-Clarke, Elisabetta Xue, Yifan Yang, Jimeng Sun, Zhiyong Lu
TL;DR
Clinical trial recruitment requires matching patients with suitable trials despite challenging patient records and trial criteria. TrialGPT uses LLMs for end-to-end retrieval, criterion-level matching, and trial-level ranking; evaluations report strong recall, expert-aligned matching and ranking, and reduced screening time.
Problem
Matching patients to clinical trials is challenging because patient records and trial criteria are heterogeneous and ambiguous.
Method
TrialGPT is an end-to-end LLM framework that retrieves candidate trials, predicts criterion-level eligibility, and aggregates results into trial-level rankings.
Results
TrialGPT effectively recalls relevant trials, produces expert-aligned matching and ranking results, and outperforms other linearly aggregated scores for ranking and excluding candidate trials.
Takeaways & Limitations
TrialGPT shows promising opportunities for patient-to-trial matching and can significantly reduce screening time by 42.6% on average.
Takeaways & Limitations
The evaluation is limited by label ambiguities and the need for further validation with larger samples and real-world applicability.
Abstract
from arXiv · showhide
Patient recruitment is challenging for clinical trials. We introduce TrialGPT, an end-to-end framework for zero-shot patient-to-trial matching with large language models. TrialGPT comprises three modules: it first performs large-scale filtering to retrieve candidate trials (TrialGPT-Retrieval); then predicts criterion-level patient eligibility (TrialGPT-Matching); and finally generates trial-level scores (TrialGPT-Ranking). We evaluate TrialGPT on three cohorts of 183 synthetic patients with over 75,000 trial annotations. TrialGPT-Retrieval can recall over 90% of relevant trials using less than 6% of the initial collection. Manual evaluations on 1,015 patient-criterion pairs show that TrialGPT-Matching achieves an accuracy of 87.3% with faithful explanations, close to the expert performance. The TrialGPT-Ranking scores are highly correlated with human judgments and outperform the best-competing models by 43.8% in ranking and excluding trials. Furthermore, our user study reveals that TrialGPT can reduce the screening time by 42.6% in patient recruitment. Overall, these results have demonstrated promising opportunities for patient-to-trial matching with TrialGPT.
Introduction
Patient-to-trial matching aims to help patients and referral offices explore potentially eligible trials, but heterogeneous records and ambiguous criteria make matching difficult. TrialGPT addresses this challenge with an end-to-end LLM framework for retrieval, criterion-level matching, and trial-level ranking.
- Motivation: Patient-to-trial matching connects one patient with candidate clinical trials and can help patients and referral offices explore eligibility.This patient-centric direction contrasts with trial-to-patient matching, which connects one trial with candidate patients.
- Challenges: Heterogeneous patient records and ambiguous trial criteria create significant challenges for AI-based matching.Manual matching is labor-intensive, time-consuming, and prone to human errors.
- Challenges: Dense neural retrieval requires large paired patient-criterion datasets and provides limited explainability for debugging and expert trust.These constraints can make training infeasible and create skepticism when models encounter new criteria and patient groups.
- Approach: TrialGPT provides an end-to-end solution spanning trial retrieval, criterion-level patient eligibility matching, and trial-level ranking.Retrieval generates keywords and uses hybrid-fusion search; Matching produces explanations, relevant patient-note sentences, and eligibility classifications; Ranking aggregates these results into a trial list.
- Results: TrialGPT-Retrieval recalled over 90% of relevant trials using less than 6% of the candidate collection.Its generated keywords also outperformed keywords produced by four human clinicians.
Results
TrialGPT combines keyword-based retrieval, criterion-level eligibility prediction, and trial-level score aggregation to support patient–trial matching. Across retrieval, matching, and ranking evaluations, it retained relevant trials efficiently, approached expert-level criterion predictions, and produced scores useful for ranking or excluding trials.
- TrialGPT-Retrieval: Hybrid retrieval outperforms semantic MedCPT and lexical BM25, while GPT-generated keywords outperform raw patient notes across all three cohorts.Clinician-generated keywords fall between LLM-generated keywords and raw notes in the SIGIR cohort.
- TrialGPT-Matching: 90.1% precision, 87.9% recall, and 88.6% F1 show that TrialGPT accurately locates relevant sentences in patient notes.These predictions are close to human-expert performance ranging from 86.9% to 91.5%, supporting faithful explanations for oversight.
- TrialGPT-Ranking: TrialGPT-Ranking scores are highly correlated with patient–trial eligibility and can be used to rank or exclude clinical trials.Eligible pairs have more met inclusion and unmet exclusion criteria, whereas ineligible pairs have more met exclusion criteria; the scores also correlate significantly with manual labels.
Discussion
TrialGPT uses large language models across retrieval, criterion-level matching, and trial-level score aggregation to support patient-to-trial matching. The study reports effective retrieval, criterion-level explanations, ranking performance, and reduced screening time, while identifying important limitations around data scope, model access, and human oversight.
- Framework: TrialGPT applies LLMs to retrieval, criterion-level matching, and aggregation of criterion-level predictions into trial-level scores.The framework uses natural-language patient summaries and trial criteria without requiring fixed criterion formats.
- Framework: TrialGPT-Retrieval filters most irrelevant trials from large collections, while TrialGPT-Matching predicts criterion-level eligibility with explanations.The retrieval stage addresses collections containing tens of thousands of trials, and matching provides faithful explanations according to the evaluation summary.
- Evaluation: Trial-level scores from TrialGPT-Ranking outperform linearly aggregated scores for ranking and excluding candidate clinical trials.Ranking aggregates criterion-level predictions into scores used to rank or exclude candidate trials.
- Scope: The evaluations use three publicly available patient-trial matching datasets with patients represented by free-text clinical summaries.The study does not evaluate longitudinal notes, laboratory values, structured data, or multimodal inputs, although these may be needed in more comprehensive matching.
- Scope: The SIGIR and TREC datasets focus on inclusion and exclusion criteria, so TrialGPT still needs to ensure identified trials are appropriate using factors outside that semantic scope.Geolocation and recruitment status are excluded from those datasets because they can be addressed through structured queries.
- Implications: TrialGPT is intended to assist rather than replace human recruiters, and its pilot user study reported a 42.6% average reduction in screening time.The authors call for larger prospective evaluations and note limitations in the pilot study’s sample size.
Methods
The study evaluates TrialGPT on three publicly available cohorts using synthetic patient notes and pooled candidate clinical trials. Its three-module pipeline retrieves candidates, predicts criterion-level eligibility with explanations, and aggregates predictions into trial-level scores.
- Data: Three publicly available cohorts—the SIGIR 2016, TREC 2021, and TREC 2022 cohorts—provide synthetic patient case descriptions for evaluation.The TREC 2021 and 2022 cohorts contain 75 and 50 patients, respectively.
- System setup: TrialGPT is LLM-agnostic and can use different backbone language models; the study mainly uses GPT-4 and GPT-3.5 through Azure OpenAI services.Inference uses temperature 0 for deterministic outputs.
- TrialGPT-Retrieval: TrialGPT-Retrieval generates ranked patient keywords, combines BM25 lexical and MedCPT semantic retrieval with reciprocal rank fusion, and selects top-ranked candidate trials.Keyword importance is ranked by the language model, and scores across keywords use a decaying weight.
- TrialGPT-Matching: TrialGPT-Matching analyzes inclusion and exclusion criteria separately, producing relevance explanations, supporting sentence IDs, and criterion-level eligibility predictions.The model first generates explanations to ground subsequent sentence-ID and eligibility predictions, then returns criterion-level predictions in JSON format.
- TrialGPT-Ranking: TrialGPT-Ranking aggregates criterion-level predictions into trial-level scores for ranking and excluding clinical trials using linear, LLM-based, and combined features.The aggregation includes general relevance and eligibility features, with intentionally simple linear strategies used for probing.
Data availability
The study makes its cohorts, annotations, supplementary user-study materials, and generated data publicly available through named repositories and websites.
- Public cohorts: The TREC Clinical Trial 2021 and 2022 cohorts and the SIGIR cohort are available from their respective TREC and CSIRO webpages.The passages provide download links for all three cohorts.
- Annotations: Criterion-level annotations generated in the study are deposited in the Hugging Face database.
- Supplementary and code resources: User-study clinical vignettes and study data files are provided through the Supplementary Materials and the TrialGPT GitHub repository.The repository passage identifies the TrialGPT accession code.