Source-linked AI summary
Does Synthetic Data Generation of LLMs Help Clinical Text Mining?
Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, Xia Hu
TL;DR
The paper asks whether ChatGPT can support clinical text mining despite limited evidence for healthcare-task performance and privacy constraints on patient-data uploads. It generates labeled synthetic clinical text with ChatGPT and uses it to fine-tune a local model. The framework improves downstream NER and RE performance while mitigating privacy concerns and reducing data-collection and labeling effort.
Problem
The study addresses uncertain ChatGPT effectiveness for healthcare text mining, alongside poor direct-task performance and privacy concerns about uploading patient information to APIs.
Method
The framework uses a small set of human-labeled seeds and prompts ChatGPT to generate filtered labeled synthetic data for fine-tuning a local offline model.
Results
The synthetic-data approach significantly improves downstream NER and RE performance compared with zero-shot ChatGPT and can approach models fine-tuned on original training data.
Takeaways & Limitations
The framework presents a promising way to apply LLM capabilities to clinical text mining while reducing data-collection effort and mitigating patient-data privacy concerns.
Takeaways & Limitations
Synthetic and original data show distinct distributions, and future work is needed to produce synthetic data with a more similar distribution to the original data.
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) have led to the development of highly potent models like OpenAI's ChatGPT. These models have exhibited exceptional performance in a variety of tasks, such as question answering, essay composition, and code generation. However, their effectiveness in the healthcare sector remains uncertain. In this study, we seek to investigate the potential of ChatGPT to aid in clinical text mining by examining its ability to extract structured information from unstructured healthcare texts, with a focus on biological named entity recognition and relation extraction. However, our preliminary results indicate that employing ChatGPT directly for these tasks resulted in poor performance and raised privacy concerns associated with uploading patients' information to the ChatGPT API. To overcome these limitations, we propose a new training paradigm that involves generating a vast quantity of high-quality synthetic data with labels utilizing ChatGPT and fine-tuning a local model for the downstream task. Our method has resulted in significant improvements in the performance of downstream tasks, improving the F1-score from 23.37% to 63.99% for the named entity recognition task and from 75.86% to 83.59% for the relation extraction task. Furthermore, generating data using ChatGPT can significantly reduce the time and effort required for data collection and labeling, as well as mitigate data privacy concerns. In summary, the proposed framework presents a promising solution to enhance the applicability of LLM models to clinical text mining.
1 Introduction
The paper examines whether ChatGPT can support clinical text mining, but direct use is limited by poor healthcare-task performance and patient-data privacy concerns. It proposes generating labeled synthetic data with ChatGPT and fine-tuning a local model instead.
- ChatGPT is investigated for extracting biological entities and relations from unstructured healthcare texts.
- Direct ChatGPT use performs poorly on healthcare tasks because the model was not specifically trained for this domain.
- Uploading patient information to LLM APIs creates privacy concerns for healthcare providers.
- The proposed paradigm uses human-labeled examples as seeds, prompts ChatGPT to generate varied labeled data, removes low-quality or duplicated samples, and fine-tunes a local model.
- Experiments on four representative datasets report improved local-model performance and reduced reliance on uploading patient data to LLM APIs.
2 Preliminaries
The preliminaries define biomedical named entity recognition, relation extraction, zero-shot learning, and the datasets used to evaluate these clinical text-mining tasks.
- Biomedical NER identifies and categorizes medical entities using IOB tags marking entity beginnings, interiors, and non-entity words.
- Biomedical RE predicts the relation between two medical entities from their sentence context as a classification task.
- Zero-shot learning enables LLMs to perform tasks not explicitly trained through prompts describing the task and possible outputs.
- The study evaluates NER on NCBI and BC5CDR and relation extraction on GAD and EU-ADR datasets.
- GAD and EU-ADR contain noisy labels, so three annotators manually labeled 200 test samples from each original test dataset for evaluation.
3 Benchmarking LLM on Biomedical NER and RE Tasks
The benchmark compares prompted ChatGPT with state-of-the-art models fine-tuned for biomedical tasks. ChatGPT performs below specialized models, with a much larger gap on named entity recognition than on relation extraction.
- The benchmark uses ChatGPT prompts for biological NER and RE and compares its results with state-of-the-art models trained on the datasets.
- 35.93% average disease-recognition F1-score is achieved by ChatGPT, compared with 88.60% for the SOTA model.
- 78.35% average relation-extraction F1-score is achieved by ChatGPT, compared with 84.35% for the SOTA model.
- ChatGPT performs slightly worse than SOTA models on relation extraction but has a substantial performance gap on biological named entity recognition.
4 Exploring Synthetic Data Generation of ChatGPT for Clinical Text Mining
The paper proposes using ChatGPT to generate labeled synthetic clinical-text data rather than applying it directly, with prompts, seed examples, and post-processing guiding data quality.
- Training paradigm: ChatGPT is used to generate labeled synthetic data instead of performing healthcare extraction directly in a zero-shot setting.The approach is designed to address both performance limitations and privacy concerns associated with direct API use.
- Generated data: Table 4 contains generated NER texts and labels with target entities highlighted, while Table 5 labels whether target gene–disease pairs are related.
- Prompt engineering: Prompt engineering iteratively selects and improves data-generation prompts, reaching an optimal prompt after three testing rounds.
- Data generation: Synthetic sentences are seeded with entity names for NER and labeled positive or negative examples for RE to promote varied, task-relevant outputs.ChatGPT is also instructed to mimic PubMed writing style, while post-processing removes low-quality or duplicated samples.
- Prompt specification: Table 6 presents the prompts used for the named entity recognition and relation extraction tasks.
5 Named Entity Recognition
For biomedical NER, models fine-tuned on ChatGPT-generated synthetic data substantially outperform zero-shot ChatGPT and can approach models trained on original labeled data.
- Experimental setup: The NER experiments generate N sentences with annotations for each of M seed entities, setting N = 30, then fine-tune three pretrained language models.
- Main results: Fine-tuning on synthetic data improved average BERT performance over ChatGPT by more than 35% in Precision, 34% in Recall, and 36% in F1.
- Main results: 81.96% Recall on the BC5CDR Chemical dataset with synthetic-data fine-tuning rose from 11.69% zero-shot and approached 88.66% using the original training set.
- Data-volume analysis: Increasing synthetic sentences improves performance up to a point, after which gains become marginal; changing the synthetic-to-real entity ratio especially helps under-represented entities.
6 Relation Extraction
For biomedical relation extraction, synthetic-data fine-tuning improves all reported metrics over zero-shot ChatGPT and achieves performance comparable to original-data fine-tuning.
- Experimental setup: The experiments compare zero-shot ChatGPT, synthetic-data fine-tuned models, and original-training-set fine-tuned models using BERT, RoBERTa, and BioBERT backbones.
- Main results: Synthetic-data fine-tuning improved average Precision, Recall, and F1 over ChatGPT by more than 6%, 10%, and 8%, respectively.
- Main results: Models fine-tuned on synthetic data achieved results comparable to models fine-tuned on the original training set, and slightly exceeded them on GAD.
- Data-volume analysis: Increasing synthetic examples improves performance until gains become marginal; 3500 synthetic sentences and 80 seed examples were sufficient for optimal results in the reported RE experiments.
7 Analysis of Generated Texts
The analysis compares original and synthetic sentence embeddings to assess whether ChatGPT reproduced the source data, finding distinct distributions rather than simple memorization.
- Embedding analysis: Sentence-transformer embeddings for original and synthetic data were projected with T-SNE to compare their distributions.
- Findings: Distinct distribution patterns indicated that ChatGPT did not simply memorize and reproduce the original dataset.
- Implications: The distribution shift may explain the performance gap between models fine-tuned on synthetic versus original data, motivating future work on distribution matching.
- Embedding analysis: Figure 4 compares the sentence embeddings of the original data and synthetic data.
8 Related Work
Related work positions LLMs as increasingly capable tools across domains and highlights established biomedical NLP applications. Prior biomedical work includes clinical-record analysis, drug discovery, medical chatbots, and NER/RE tasks.
- LLMs have attracted attention for their natural-language understanding and high performance across education, healthcare, and content creation.
- Biomedical NLP is applied to electronic health records, drug discovery, and medical chatbots.
- Previous biomedical research has explored named entity recognition and relation extraction.
9 Conclusion
The study investigates ChatGPT for clinical text mining but finds direct use unsatisfactory and privacy-sensitive. It instead generates synthetic data with ChatGPT and fine-tunes a local offline model, improving downstream performance while reducing labeling effort and addressing privacy concerns.
- The framework generates high-quality synthetic data with ChatGPT and fine-tunes a local offline model for clinical text-mining tasks.
- Direct ChatGPT use for named entity recognition and relation extraction yielded unsatisfactory results and raised privacy concerns.
- Synthetic data significantly improved downstream-task performance while reducing data-collection and labeling effort and addressing privacy concerns.
- The framework is intended for further refinement, improved data quality, and extension to other clinical tasks.