Source-linked AI summary
Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization
Bharath Chintagunta, Namit Katariya, Xavier Amatriain, Anitha Kannan
TL;DR
Medical dialogue summarization requires coherent summaries that retain medically relevant information, but large expert-labeled datasets are difficult to obtain. The paper uses medically aware GPT-3 ensembling to generate synthetic training data from 210 human examples. The resulting models match performance associated with roughly 6400 human-labeled examples, and mixing generated with human-labeled data outperforms either source alone.
Problem
Medical dialogue summarization needs coherent, medically complete summaries, but expert-labeled datasets are costly and difficult to obtain at scale.
Method
GPT-3-ENS combines medical knowledge with multiple GPT-3 summaries to generate synthetic training data for an in-house summarization model.
Results
Using 210 human-labeled examples, the approach produces models with performance comparable to training on roughly 6400 human-labeled examples, while combining synthetic and human data performs better than either source alone.
Takeaways & Limitations
Medically aware synthetic labels can support high-quality medical dialogue summarization with far fewer human-labeled examples and can complement human-labeled data.
Takeaways & Limitations
The paper leaves the trade-off between human and GPT-3-ENS-labeled data for future work and identifies improving the ensembling function as an open direction.
Abstract
from arXiv · showhide
In medical dialogue summarization, summaries must be coherent and must capture all the medically relevant information in the dialogue. However, learning effective models for summarization require large amounts of labeled data which is especially hard to obtain. We present an algorithm to create synthetic training data with an explicit focus on capturing medically relevant information. We utilize GPT-3 as the backbone of our algorithm and scale 210 human labeled examples to yield results comparable to using 6400 human labeled examples (~30x) leveraging low-shot learning and an ensemble method. In detailed experiments, we show that this approach produces high quality training data that can further be combined with human labeled data to get summaries that are strongly preferable to those produced by models trained on human data alone both in terms of medical accuracy and coherency.
1. Introduction
Medical dialogue summarization needs coherent, medically complete summaries, but expert annotation is costly, slow, and constrained by privacy. The paper proposes GPT-3-ENS to generate training data and shows that combining synthetic and human-labeled data improves performance over either source alone.
- Motivation: Expert annotation for medical dialogue summarization is costly, slow, and difficult to share because of privacy and HIPAA constraints.These constraints limit the availability of large-scale annotated datasets.
- Deployment considerations: The approach is designed to support practitioner feedback, allowing manually corrected summaries to become additional training data.This feedback loop addresses the long tail and diversity of medical dialogue distributions.
- Approach: GPT-3-ENS combines medical knowledge with an ensemble of GPT-3 outputs to label medical dialogue summarization data.The approach targets medically relevant information in generated summaries.
- Results: With 210 expert-curated summaries, GPT-3-ENS can mimic the performance of models trained on 6400 expert-curated summaries.The paper uses GPT-3 as a labeled-data simulator to scale a small human-labeled set.
- Results: Combining GPT-3-ENS-generated and human-labeled datasets yields better performance than training on either data source alone.Figure 1 presents this mixed-data training strategy.
2. Generalizable Insights about Machine Learning in the Context of Healthcare
Healthcare deep learning is limited by scarce expert annotations, while large pretrained models offer low-shot performance but can be noisy in medical settings. The paper uses GPT-3 to generate medically informed synthetic labels for an in-house model, with fixed training data helping address privacy and deployment constraints.
- Problem: Healthcare deep learning lacks large annotated datasets because expert labeling is costly and time-consuming.The paper motivates low-shot methods that can perform well with few labeled examples.
- Problem: Large pretrained models such as GPT-3 show good low-shot performance but can be noisy in the medical domain.The paper frames medical knowledge infusion as a way to mitigate this noise while retaining pretrained-model strengths.
- Approach: The proposed approach uses GPT-3 as a data generator to create a large training set for an in-house medical dialogue summarization model.The external model sees a fixed, small set of examples, while the in-house model handles inference.
- Implications: Using an in-house model avoids practical inference-time constraints associated with protecting patient privacy when accessing GPT-3.The approach also supports practitioner-in-the-loop deployment.
3. Related work
Prior summarization work has advanced extractive, abstractive, and hybrid models, while medical dialogue research incorporates medical knowledge into modeling. This paper instead incorporates medical knowledge during data labeling and uses pretrained models to collect training data.
- General summarization: Recent summarization research includes extractive, abstractive, and hybrid sequence-to-sequence models that generate coherent and reasonably factual summaries.These approaches build on sequence-to-sequence architectures and attention mechanisms.
- Medical dialogue summarization: Medical dialogue summarization work incorporates medical knowledge from a modeling perspective.The cited research addresses medical knowledge within the summarization model.
- Positioning: This paper incorporates medical knowledge from a data-labeling perspective using pretrained language models and low-shot learning.The resulting labeled data is used for medical dialogue summarization.
- Positioning: The generated data improves performance over models trained solely on existing human-labeled data.This is the paper’s stated distinction from training exclusively on human labels.
4. Background: Can GPT-3 serve as a medical summarizer?
The paper first tests GPT-3 as a medical dialogue summarizer using low-shot priming. GPT-3 can produce summaries, but its outputs may omit medical information, mishandle affirmations, or repeat redundant content; sensitivity to priming motivates the later ensemble approach.
- The initial experiment deliberately ignores privacy and practitioner-in-the-loop considerations while assessing GPT-3's standalone summarization ability.
- GPT-3 uses a priming context containing a task description and demonstrations to summarize previously unseen dialogue snippets.The priming context defines dialogue snippet summarization through a few examples.
- GPT-3 summaries can omit important medical concepts, mishandle affirmations, and repeat redundant information from doctor queries.These issues make some generated summaries unusable for medical summarization.
- GPT-3 is sensitive to the priming context because it follows the supplied examples rather than learning a stable task behavior.The paper exploits this output variability through ensembling and medical-knowledge infusion.
5. Infusing Medical Knowledge in GPT-3 for use as a Data Generator
The paper proposes GPT-3-ENS, which combines medical knowledge, multiple GPT-3 priming contexts, and ensemble selection to generate synthetic labels for an in-house summarizer. The selected summary maximizes medical-concept recall, while the resulting model avoids sending raw patient conversations to GPT-3 at inference time.
- Approach: GPT-3-ENS infuses medical knowledge into GPT-3 and uses an ensemble within an inner loop for medical dialogue summarization.The method is designed to use GPT-3 as a medically aware data labeler.
- Data generation: GPT-3-ENS generates a large labeled training set for an in-house summarization model rather than requiring GPT-3 access during inference.This design addresses practical constraints involving external access to raw patient conversations.
- Practical considerations: The in-house model can support practitioner feedback and avoid inference-time privacy constraints associated with sending conversations to an external GPT-3 service.Training data is fixed and can be privacy protected, unlike raw inference-time conversations.
- Motivation: The method addresses GPT-3's sensitivity to priming context by using multiple contexts rather than seeking one maximally informative context.Finding a single ideal context would itself require the labeled examples that the approach aims to obtain.
- Approach: The ensemble samples multiple priming contexts, generates K summaries, and selects the summary with the highest recall of medical concepts in the dialogue.A medical entity extractor identifies concepts in both the input dialogue and candidate summaries.
- Results: GPT-3-ENS summaries capture medical concepts more comprehensively and have better grammatical structure than GPT-3 summaries in the qualitative comparison.Doctor evaluation also found GPT-3-ENS significantly better at summarization than GPT-3.
6. Datasets
The dataset construction splits telemedicine dialogues into local snippets, obtains doctor-written clinical summaries, and uses the resulting human-labeled pool for testing, priming, and synthetic-label generation.
- Human-labeled data: Medical dialogues are split into local snippets using the turns between two successive physician questions.The snippets follow the dialogue's chronological structure for later summary stitching.
- Human-labeled data: Doctors summarize eligible snippets as clinical notes containing all relevant history-taking information, excluding snippets without such information.Greetings and other non-history-taking content can therefore be omitted from annotation.
- Splits: The human-labeled dataset contains 6,900 snippet-summary pairs, including a randomly sampled test set of 500 examples.The remaining labeled examples support priming and training-data generation.
- Synthetic data: GPT-3-ENS synthetic datasets are generated from randomly sampled human-labeled examples used as priming data across ensembling trials.Sampling is performed without replacement, and a shared random seed enables comparisons across K values.
7. Evaluation Metrics
Evaluation combines automated measures of lexical overlap, medical-concept coverage, and negation correctness with blinded doctor judgments of factual medical relevance and usability.
- Automated metrics: The evaluation reports standard ROUGE metrics alongside medical-concept and negation measures.
- Automated metrics: Medical-concept coverage compares concepts in the model summary with concepts in the reference summary using precision, recall, and F1.An in-house medical entity extractor identifies the concepts used for evaluation.
- Automated metrics: Negation correctness evaluates whether the decoded summary accurately identifies the negated status of medical concepts using NegEx.
- Human evaluation: Doctors grade summaries as All, Most, Some, or None according to how much factually correct and medically relevant information they capture from a snippet.The categories correspond to 100%, at least 75%, at least one but less than 75%, and 0% of concepts, respectively.
- Human evaluation: Doctors also compare summaries from different models by choosing the best one for replacing the dialogue snippet.Usability requires correct concepts and accurate negations.
- Evaluation protocol: Model identities are hidden from doctors and comparison summaries are presented in randomized order to reduce evaluation bias.
8. Experiments and Results
Experiments compare PEGASUS and DRSUM trained with human-labeled data, GPT-3-ENS-generated data, and mixtures of both. Using 210 human examples, GPT-3-ENS achieved performance comparable to or better than training with 6400 human examples, while mixed datasets improved automated and doctor evaluations.
- Experimental setup: GPT-3-ENS generates training data for both PEGASUS and DRSUM using only 210 human-labeled examples.The experiments compare these models with datasets based on human labels and GPT-3-ENS synthesis.
- Automated results: 210 human-labeled examples enabled PEGASUS and DRSUM to achieve comparable or better performance than models trained on 6400 human-labeled examples.The 210-example setup represents an approximately 30-fold reduction in human-labeled data.
- Automated results: PEGASUS performance improved drastically with GPT-3-ENS data, whereas DRSUM showed no performance gain from generated data but still performed on par.The authors attribute this difference to GPT-3's generative outputs being more compatible with abstractive than hybrid summarization models.
- Automated results: Increasing GPT-3-ENS data from 6400 to 12800 or 25600 examples produced little or no further improvement in concept and negation F1, while ROUGE-L F1 improved.The ROUGE-L change reflected improved summary coherency.
- Data mixing: Mixing human-labeled and GPT-3-ENS data consistently improved almost all automated metrics for both PEGASUS and DRSUM across tested mixture ratios.The metric lift was lower for DRSUM, again indicating greater compatibility of generated data with PEGASUS.
- Human evaluation: Doctors preferred mixture-trained summaries over summaries trained on either source alone for medical-information coverage and overall quality.For PEGASUS, doctors also preferred the model using GPT-3-ENS data generated from 210 human examples over the model trained on 6400 human examples.
- Caveat: The experiments do not establish that increasing the mixture ratio α improves metrics; the human-to-generated-data trade-off remains future work.The supported conclusion is specifically that mixing the two data sources improves metrics over human-only training.
9. Conclusion
The paper introduces GPT-3-ENS, a medically aware ensemble that generates quality training data for medical dialogue summarization. Using 210 human-labeled examples, it produces more medically correct and higher-quality summaries than models trained with roughly thirty times as many human-labeled examples, while identifying several directions for improving the ensemble.
- Contribution: GPT-3-ENS combines medical awareness with ensembling multiple GPT-3 summaries for medical conversation summarization.The approach uses a medically aware ensembling criterion to generate training data.
- Results: 210 human-labeled examples yielded more medically correct and higher-quality summaries than roughly thirty times as many human-labeled examples.The comparison was demonstrated with two different summarization models.
- Limitations and Future Work: The current ensembling function retains all medical information but does not yet incorporate priors such as affirmations or information importance.The authors identify improved medical priors and alternative ensembling methods as future work.
- Limitations and Future Work: Future work will examine the trade-off between human-labeled and GPT-3-ENS-labeled data and ways to improve GPT-3 domain coverage.The authors also propose applying synthetic labeling with large pretrained models to other healthcare tasks.
Appendix A. GPT-3 Prompt
The appendix describes a simple GPT-3 prompt that concatenates example snippets and summaries with special markers. Conversational turns are separated with [SEP], and 21 examples are used to prime GPT-3 in the experiments.
- Prompt Construction: Each prompt example concatenates snippet_text and summary_text with [SUMMARIZED] and [STOP] markers.The resulting transformation is used to construct GPT-3 training prompts.
- Prompt Construction: Conversational turns in snippet_text are separated with the [SEP] token.This token marks boundaries between turns within each dialogue snippet.
- Prompt Construction: 21 examples are used to generate the GPT-3 prompt in the experiments.The appendix notes that the prompt is primed using these examples.