PaperlayerView exact revision on arXiv

Source-linked AI summary

CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study

Zihan Guan, Zihao Wu, Zhengliang Liu, Dufan Wu, Hui Ren, Quanzheng Li, Xiang Li, Ninghao Liu

arXiv:2307.11346v1cs.CLcs.AI

TL;DR

Participant recruitment from unstructured medical text is difficult, while general-purpose LLMs lack specialized clinical knowledge for this classification task. CohortGPT combines a disease knowledge graph with reinforcement-learning-enhanced CoT sample selection in few-shot prompts. The framework achieves satisfactory performance compared with fine-tuning strategies using much more labeled data and can extend to other healthcare applications.

  • Problem

    Recruiting participants from unstructured clinical notes and radiology reports is difficult, and LLMs can underperform on knowledge-intensive medical text classification without domain-specific context.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    CohortGPT embeds a clinical knowledge graph into prompts and uses reinforcement-learning-enhanced CoT selection to choose report-specific examples.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    CohortGPT achieves satisfactory few-shot performance compared with fine-tuning strategies using much more labeled data.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The framework can be implemented with open-source LLMs and extended to healthcare applications including diagnosis, prognosis, and treatment optimization.

    Source: §6 · p. 9

Abstract

from arXiv · show

Participant recruitment based on unstructured medical texts such as clinical notes and radiology reports has been a challenging yet important task for the cohort establishment in clinical research. Recently, Large Language Models (LLMs) such as ChatGPT have achieved tremendous success in various downstream tasks thanks to their promising performance in language understanding, inference, and generation. It is then natural to test their feasibility in solving the cohort recruitment task, which involves the classification of a given paragraph of medical text into disease label(s). However, when applied to knowledge-intensive problem settings such as medical text classification, where the LLMs are expected to understand the decision made by human experts and accurately identify the implied disease labels, the LLMs show a mediocre performance. A possible explanation is that, by only using the medical text, the LLMs neglect to use the rich context of additional information that languages afford. To this end, we propose to use a knowledge graph as auxiliary information to guide the LLMs in making predictions. Moreover, to further boost the LLMs adapt to the problem setting, we apply a chain-of-thought (CoT) sample selection strategy enhanced by reinforcement learning, which selects a set of CoT samples given each individual medical report. Experimental results and various ablation studies show that our few-shot learning method achieves satisfactory performance compared with fine-tuning strategies and gains superb advantages when the available data is limited. The code and sample dataset of the proposed CohortGPT model is available at: https://anonymous.4open.science/r/CohortGPT-4872/

1 Introduction

Participant recruitment from unstructured clinical text is difficult because medical documentation is complex and enrollment criteria may be incompletely recorded. CohortGPT addresses this challenge by combining LLM prompting with clinical knowledge graphs and reinforcement-learning-enhanced CoT selection, achieving competitive performance with much less labeled data.

  • Recruitment from clinical notes and radiology reports is difficult because abbreviations, jargon, errors, inconsistent formatting, and nonstandard terminology complicate enrollment-criteria identification.Source: §1 · p. 2
  • Rule-based and BERT-based recruitment methods either require expert-crafted rules or substantial labeled data for fine-tuning.Source: §1 · p. 2
  • LLMs provide strong general language understanding and in-context learning but perform worse in healthcare because open-domain pretraining lacks domain-specific vocabulary and knowledge.Source: §1 · p. 2
  • CohortGPT integrates ChatGPT and GPT-4 with specialized prompting for radiology-report participant recruitment.Source: §1 · p. 2
  • The framework embeds clinical knowledge through a knowledge graph and uses CoT prompting to guide step-by-step reasoning, achieving competitive results with much less labeled data and extending to other medical NLP tasks.Source: §1 · p. 3

2 Preliminary

The paper formulates medical-report diagnosis as multi-label classification: an LLM must identify all diseases implied by each report. It defines the dataset split, fixed-parameter LLM input, CoT prompting, and k-shot prompting used in this setting.

2.1 Problem Statement

  • Medical report classification is defined as multi-label prediction of all potential diseases implied by a patient’s report.Source: §2.1 · p. 3

2.2 Symbol Notation

  • The task uses training and test subsets, with a fixed-parameter LLM receiving each test report as a text sequence.Source: §2.2 · p. 3
  • CoT prompting prepends question-answer examples that detail how answers are derived before the target report is submitted to the LLM.Source: §2.2 · p. 3
  • In this paper, k-shot learning means using k training samples as part of the prompt.Source: §2.2 · p. 3

3 Methodology

CohortGPT combines hierarchical clinical knowledge with dynamically selected CoT examples in prompts for medical-report classification. Its policy-gradient selector chooses samples from a candidate pool using report and CoT representations, while rewards reflect classification performance.

3 Methodology

  • The overall pipeline trains a policy model to select CoT samples, converts disease-label knowledge into executable rules, and queries the LLM with reports plus the resulting prompt.Source: Figure 1 · p. 3, Figure 2 · p. 4

3.1 Static Prompting From Explicit Knowledge

  • The clinical knowledge graph links disease labels to organs or tissues, with labels sharing a tissue or organ forming clusters.Source: §3.1 · p. 4
  • KG-as-Tree preserves hierarchy, KG-as-Relation expresses triplets, and KG-as-Rule converts each label cluster into human-readable rules.Source: §3.1 · p. 4

3.2 Dynamic Prompting via Policy Gradient

  • CoT examples provide task-specific logical chains through detailed input-output demonstrations, supporting reasoning for medical-report diagnosis.Source: §3.2 · p. 4
  • Policy-gradient optimization addresses unstable CoT selection in black-box LLMs using feed-forward propagation and a reward based on classification performance.Source: §3.2 · pp. 4–5
  • The selector chooses K CoT samples from a candidate pool and combines them with a knowledge-graph prompt to form each report-specific prompt.Source: §3.2 · p. 4
  • The policy network uses fixed BioGPT embeddings with trainable projection parameters to capture contextual similarity between reports and candidate CoT samples.Source: §3.2 · p. 5

4 Experiment

Experiments evaluate CohortGPT on IU-RR and MIMIC-CXR using fine-tuning and few-shot baselines. Results favor the proposed method in limited-data settings, while ablations examine knowledge-graph prompting, sample sizes, and CoT selection.

4.1 Experimental Settings

  • Experiments use MIMIC-CXR, compare fine-tuned Bio-Bert and BioGPT models with few-shot Alpaca, BloomZ, ChatGPT, and GPT-4 baselines, and report five multi-label metrics.The default configuration uses a 25-sample CoT candidate pool, 160 training samples, KG-as-Rule prompting, and 5-shot examples.Source: §4.1 · pp. 5–6

4.2 RQ1: Medical Report Classification Performance

  • With 185 IU-RR training samples, ChatGPT and GPT-4 achieve F1 scores of 0.69 and 0.81, exceeding BioBERT’s 0.44 and BioGPT’s 0.25.Fine-tuning eventually becomes advantageous as more training data becomes available, whereas the proposed method is advantageous in few-shot settings on both datasets.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 RQ2: Ablation Study

  • KG-as-Rule achieves the best performance across all reported metrics among the evaluated KG-to-prompt strategies.The authors suggest that command-style or rule-style inputs are easier for ChatGPT to handle.Source: §4.3 · p. 6, Table 1 · p. 6
  • As training samples increase, precision, recall, F1, and Exact Match tend to rise, while Hamming Loss tends to decrease.The authors attribute this trend to policy-model training on a more generalized data distribution, improving test-set generalization.Source: §4.3 · p. 6, Figure 4 · p. 7
  • Performance strengthens as the candidate-sample pool grows, which the authors associate with a larger search space that helps the policy model escape saddle points.This ablation concerns the number of candidate samples used for dynamic CoT selection.Source: §4.3 · pp. 6–7, Figure 5 · p. 7
  • Dynamic CoT selection outperforms random, manual, and most-similar selection, while performance peaks at k = 5 or k = 8 before declining with higher variance.The dynamic strategy considers both sample similarity and classification performance through a crafted reward function; excessive CoT samples may introduce confusing information.Source: §4.3 · pp. 7–8, Table 2 · p. 8, Figure 6 · p. 8

4.4 RQ3: Case Studies

  • Case studies show that selected CoT examples support answers based on multiple reasoning steps rather than only superficial similarity.The examples concern prompts and corresponding ChatGPT-generated answers.Source: §4.4 · p. 8

5 Related Work

The related work frames LLMs as adaptable tools for healthcare language tasks and presents chain-of-thought reasoning as a way to improve complex inference through stepwise processing.

5.1 LLMs in healthcare

  • LLMs can adapt to contextual variations with minimal post-training and support few-shot or zero-shot learning across healthcare language tasks.Healthcare applications include biomedical classification, clinical information extraction, radiology summarization, and depression or suicidality detection.Source: §5.1 · p. 8
  • RLHF and instruction fine-tuning improve LLM alignment with individualized human values and communication nuances relevant to healthcare interactions.The passage emphasizes healthcare’s demands for personal interaction, empathy, and mutual understanding.Source: §5.1 · p. 8

5.2 Chain-of-thought reasoning with LLMs

  • Chain-of-thought reasoning decomposes complex problems into smaller sequential steps, making them easier to solve.This approach is presented as analogous to human problem solving through simpler sub-problems.Source: §5.2 · pp. 8–9
  • Step-by-step prompting improves LLM reasoning, while few-shot examples can further improve performance through in-context learning.Active-Prompt extends this direction with uncertainty-based exemplar selection and reports 83.4% accuracy on GSM8K arithmetic reasoning.Source: §5.2 · p. 9

6 Conclusion

The conclusion presents CohortGPT as a knowledge- and reinforcement-learning-enhanced approach for medical LLM inference. It reports few-shot performance comparable to fine-tuning with more labels and identifies broader deployment and healthcare applications.

  • CohortGPT combines a domain knowledge graph with reinforcement-learning-enhanced chain-of-thought sample selection to improve medical LLM inference.Source: §6 · p. 9
  • The framework achieves satisfactory few-shot performance compared with fine-tuning strategies that use substantially more labeled samples.Source: §6 · p. 9
  • CohortGPT can use open-source LLMs for local deployment and extend beyond classification to diagnosis, prognosis, and treatment optimization.Source: §6 · p. 9

A More Details About Reward Function

The reward-function implementation fixes the training coefficients to a combination observed to produce satisfactory performance.

B More Details About Datasets

The dataset details describe how radiology-report fields are combined to form the final diagnosis report used by the method.

C More Details About Rules

Figure 8 presents the rules constructed by the KG-as-Rule Method for the IU-RR dataset.

C More Details About Rules

D Case Studies

Figures 9 and 10 present two IU-RR test samples together with the CoT samples selected for each by the dynamic algorithm.

D Case Studies

Related papers

Loading 2307.11346v1…