Source-linked AI summary
Automated Paper Screening for Clinical Reviews Using Large Language Models
Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Mike Paget, Christopher Naugler
TL;DR
Clinical review screening requires comprehensive, duplicate assessment, but duplicate screening increases resource demands and disagreement resolution. This paper evaluates an OpenAI GPT API workflow against human-reviewed datasets, finding high overall agreement but lower sensitivity for identifying included papers. The authors conclude that GPT may streamline reviews as a complement to, rather than replacement for, human reviewers.
Problem
Duplicate screening supports comprehensive and reliable clinical reviews but increases financial and human-resource demands and can involve substantial inter-rater variability.
Method
The study used plain-language prompts and a Python script to have the OpenAI GPT API classify titles and abstracts across six clinical review datasets, comparing decisions with two independent human reviewers.
Results
0.91 overall accuracy, 0.76 sensitivity for included papers, and 0.91 sensitivity for excluded papers were obtained against human-reviewed papers.
Takeaways & Limitations
The GPT API could streamline clinical review screening and save researchers’ and clinicians’ time when used as an aid alongside human expertise.
Takeaways & Limitations
The lower 0.76 sensitivity for included papers may omit important studies, so the GPT API should not replace human expertise in final decisions.
Abstract
from arXiv · showhide
Objective: To assess the performance of the OpenAI GPT API in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review datasets and compare its performance against ground truth labelling by two independent human reviewers. Methods: We introduce a novel workflow using the OpenAI GPT API for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the GPT API with the screening criteria in natural language and a corpus of title and abstract datasets that have been filtered by a minimum of two human reviewers. We compared the performance of our model against human-reviewed papers across six review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. On a randomly selected subset of papers, the GPT API demonstrated the ability to provide reasoning for its decisions and corrected its initial decision upon being asked to explain its reasoning for a subset of incorrect classifications. Conclusion: The GPT API has the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, the GPT API can enhance efficiency and lead to more accurate and reliable conclusions in medical research.
1 BACKGROUND AND SIGNIFICANCE
Clinical reviews depend on comprehensive searches and duplicate screening, but duplicate review is costly and can involve disagreement. The paper proposes using plain-language prompts to the OpenAI GPT API to screen titles and abstracts across real-world clinical review datasets.
- Systematic and scoping reviews synthesize literature to identify knowledge gaps and inform healthcare guidelines, protocols, and decisions.
- Overlooking relevant articles can introduce bias, while including discrepant studies can produce misleading conclusions and greater heterogeneity.
- Duplicate screening is recommended by clinical-review guidelines but doubles financial and human-resource requirements compared with single screening.
- Artificial-intelligence and machine-learning tools have enabled semi-automated or fully automated steps in systematic-review workflows.
- The proposed workflow uses plain-language prompts to the publicly available OpenAI GPT API to screen titles and abstracts across six clinical review datasets containing over 24,000 records.
2 METHODS
The study used six expert-labeled clinical-review screening datasets and a Python application that sent consistent natural-language criteria and article text to the OpenAI GPT API. Performance was evaluated against human decisions using agreement, sensitivity, classification, and reasoning analyses.
- A Python script sent each paper and its inclusion/exclusion criteria to the OpenAI GPT API using a consistent instruction prompt.
- Evaluation computed absolute agreement, sensitivity for decision tags, a scikit-learn classification report, and Cohen’s kappa against human-reviewed papers.
- The corpus comprised six title-and-abstract datasets from systematic and scoping reviews spanning different medical topics, sizes, methodologies, and screening complexities.
- Expert reviewers supplied inclusion/exclusion decisions and the screening criteria used for each title-and-abstract entry.
- The API was instructed to exclude an article if any exclusion criterion was met or not all inclusion criteria were met, and otherwise include it.
- A subset of results was additionally analyzed by asking GPT to explain decisions and reflect on incorrect answers.
3 Results
GPT screening was evaluated against human-reviewed ground truth across the included datasets, with confusion matrices summarizing included and excluded decisions. Overall accuracy was 0.91, while sensitivity was higher for excluded than included papers.
- 0.91 overall accuracy was achieved by GPT when screening titles and abstracts against human-reviewed decisions.
- 0.76 sensitivity for included papers indicates weaker identification of relevant papers than exclusion of irrelevant papers.
- 0.91 sensitivity for excluded papers shows stronger identification of papers labeled for exclusion.
- The NOA dataset contained n=14,771 records, including 354 included abstracts, and required 643m 50.8s to run at an approximate cost of 25 USD.
- Figure 2 presents confusion matrices for included and excluded papers for datasets A–F and their overall performance in panel G.
4 DISCUSSION
The GPT API may improve the efficiency and consistency of clinical-review screening, particularly as a first-pass aid when resources are limited. However, its lower sensitivity for included papers means human expertise remains necessary to protect review comprehensiveness.
- Performance: 0.91 overall accuracy and 0.91 sensitivity for excluded papers indicate strong agreement and effective exclusion of irrelevant studies.Sensitivity for included papers was lower at 0.76.
- Performance: The API can provide reasoning for inclusion or exclusion decisions and corrected some initial errors after being asked to explain its reasoning.The reasoning assessment used a subset of papers and incorrect classifications.
- Implications: Automating aspects of screening may improve consistency, reduce human error and bias, and help produce more objective and reliable evidence selection.The authors describe these as potential benefits of using the API as an aid to researchers and clinicians.
- Implications: As a first-pass decision aid, the API could reduce screening workload and let human reviewers focus on a smaller, more relevant subset of papers.This use is presented as particularly valuable when paper volumes are large or human resources are scarce.
- Limitations: The disparity between 0.91 sensitivity for excluded papers and 0.76 sensitivity for included papers may cause important studies to be omitted.The authors link this risk to reduced comprehensiveness and quality of the final review.
- Limitations: The API should complement rather than replace human expertise, with human screeners retaining final decision-making responsibility.Human involvement is emphasized particularly where the API may insufficiently identify relevant papers.
5 CONCLUSION
The conclusion presents the GPT API as a potentially valuable tool for improving the efficiency and consistency of clinical-review inclusion and exclusion decisions. It also identifies performance refinement and broader application as future priorities.
- Conclusion: The GPT API shows potential for improving the efficiency and consistency of clinical-review inclusion and exclusion decisions.The conclusion characterizes the API as promising but acknowledges implementation limitations and challenges.
- Conclusion: Future work should refine performance, expand applications, and examine the API in other aspects of clinical research.The conclusion frames these as research and development priorities.
- Conclusion: The potential broader impact concerns clinical-review writing and medical research.This is stated as a possible extension of the study’s implications.
FUNDING
The study reports no funding.
- Funding: The study was not funded.
- Funding: No funding source is reported for the study.
- Funding: The funding statement contains no external-support attribution.
CREDIT AUTHOR STATEMENT
The author statement assigns contributions across conceptualization, methodology, software, analysis, investigation, writing, visualization, supervision, project administration, and data curation.
- Author contributions: Eddie Guo contributed across conceptualization, methodology, software, analysis, investigation, writing, visualization, supervision, and project administration.
- Author contributions: Mehul Gupta contributed to conceptualization, methodology, investigation, writing, supervision, and project administration.
- Author contributions: Jiawen Deng contributed to methodology, software, analysis, investigation, data curation, writing, and visualization.