PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization

Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerova, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, John Pauly, Akshay S. Chaudhari

arXiv:2309.07430v5cs.CL

TL;DR

Clinical summarization burdens clinicians, and evidence for LLM performance across diverse clinical tasks remains limited. The paper adapts eight LLMs across four tasks, evaluates them quantitatively and with ten physicians, and finds that adapted LLM summaries are often preferred to medical expert summaries, while noting important scope and evaluation limitations.

  • Problem

    Clinical documentation requires substantial clinician time, while general NLP benchmarks do not establish LLM effectiveness across diverse clinical summarization tasks.

  • Method

    The study adapts eight LLMs across four clinical summarization tasks and evaluates them with NLP metrics, a ten-physician reader study, and safety analyses.

  • Results

    LLM summaries are often preferred over medical expert summaries for completeness, correctness, and conciseness across the evaluated tasks.

  • Takeaways & Limitations

    The findings suggest that incorporating LLM-generated candidate summaries into clinical workflows could reduce documentation load.

  • Takeaways & Limitations

    The study covers only selected clinical document types, including ICU progress notes from a single medical center, so extrapolation is tentative.

Abstract

from arXiv · show

Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown promise in natural language processing (NLP), their effectiveness on a diverse range of clinical summarization tasks remains unproven. In this study, we apply adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Quantitative assessments with syntactic, semantic, and conceptual NLP metrics reveal trade-offs between models and adaptation methods. A clinical reader study with ten physicians evaluates summary completeness, correctness, and conciseness; in a majority of cases, summaries from our best adapted LLMs are either equivalent (45%) or superior (36%) compared to summaries from medical experts. The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information. Our research provides evidence of LLMs outperforming medical experts in clinical text summarization across multiple tasks. This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.

1 Introduction

Clinical documentation consumes substantial clinician time, while the effectiveness of LLMs across diverse clinical summarization tasks and their non-inferiority to medical experts remain insufficiently established. This study evaluates adapted LLMs across multiple tasks and reports that they can outperform medical experts.

  • Clinicians spend substantial time summarizing diagnostic reports, progress notes, and treatment histories, creating opportunities for error in precision-critical healthcare.
  • Up to two hours of physician documentation accompany each hour of patient interaction, while nurses may spend up to 60% of their time on documentation.
  • Existing general NLP benchmarks do not evaluate clinical tasks, motivating LLM assessment for clinical text summarization.
  • Adaptation across diverse clinical summarization tasks and non-inferiority to medical experts had not been thoroughly established.
  • The study evaluates eight LLMs across four summarization tasks and six datasets, then compares the best adapted model with medical expert summaries using ten physicians.
  • Adapted LLM summaries can outperform medical expert summaries across the diverse clinical documents evaluated.

2 Related Work

Prior work adapted LLMs for medical language tasks and specific summarization settings, but often evaluated limited model-method combinations using standard NLP metrics alone. This study broadens evaluation across tasks and includes clinical readers, finding summaries comparable to or better than expert outputs.

  • Clinical LLM adaptation has used new-model training, fine-tuning, and domain-specific examples supplied in prompts.
  • Prior clinical summarization work addressed radiology reports, progress-note problem lists, and doctor-patient conversations.
  • Earlier studies often covered only a small subset of possible models and approaches or relied predominantly on standard NLP metrics.
  • Because medical tasks require clinical readiness, the paper extends evaluation beyond NLP metrics to a multi-task clinical reader study.
  • Across many tasks, LLM summaries are reported as comparable to—and often better than—summaries created by human experts.

3 Approach

The approach evaluates diverse sequence-to-sequence and autoregressive LLMs, adapts them through in-context learning or QLoRA, and tests four clinical summarization tasks across six datasets. The design spans models, datasets, and task-specific adaptation choices.

  • Large language models: The study evaluates eight sequence-to-sequence and autoregressive models, including open-source and proprietary systems with varied parameter counts and context lengths.
  • Large language models: The model set includes FLAN-T5 and FLAN-UL2, Llama-family models, and GPT-3.5 and GPT-4.
  • Adaptation methods: Zero-shot prompting serves as the baseline for adaptation experiments.
  • Adaptation methods: In-context learning adds relevant nearest-neighbor examples to prompts without changing model weights, with the number constrained by context length.
  • Adaptation methods: QLoRA fine-tunes inserted low-rank matrices while keeping the original model weights frozen.
  • Data: The data comprise four summarization tasks and six open-source datasets varying in sample counts, token lengths, and lexical variance.
  • Data: The tasks summarize radiology findings, patient questions, hospitalization progress notes, and doctor-patient dialogues.

4 Experiments

The experiments vary prompts, temperature, models, and adaptation configurations, evaluate summaries with multiple NLP metrics, and compare the best model against medical experts in a blinded physician reader study. A safety analysis examines potential harm and fabricated information.

  • Model prompts and temperature: Prompts use task-specific instructions, desired output lengths, and tested expertise settings to guide concise clinical summaries.
  • Model prompts and temperature: Lower temperature generally improves performance, and temperature 0.1 is selected for all models.
  • Quantitative evaluation: Quantitative evaluation uses BLEU, ROUGE-L, BERTScore, and MEDCON to assess syntactic, semantic, and conceptual summary quality.
  • Clinical reader study: The best model and adaptation method are evaluated against medical expert summaries in reader studies covering radiology reports, patient questions, and progress notes.
  • Clinical reader study: Ten physicians make blinded, randomized comparisons of completeness, correctness, and conciseness using five-point Likert scales.
  • Safety analysis: Safety analysis connects summary disparities to potential medical harm and categorizes fabricated information.
  • Connecting evaluations: NLP metrics measure similarity, whereas reader scores measure which summary is preferred, so the study reports negative correlations for their inverse relationship.
  • Model comparisons: Most Alpaca-versus-Med-Alpaca data points fall below equivalence, indicating worse clinical summarization performance for Med-Alpaca.

5 Results and Discussion

Results show that task adaptation produces substantial trade-offs across models and methods: domain adaptation alone may not improve summarization, while in-context learning and QLoRA vary by model, context budget, dataset, and metric.

  • 5.1.1 Impact of domain-specific fine-tuning: Domain adaptation alone can underperform task adaptation: Med-Alpaca performs worse than Alpaca on the evaluated clinical summarization tasks.The authors distinguish adaptation to medical text from adaptation to the summarization task.
  • 5.1.2 Comparison of adaptation strategies: FLAN-T5 is the best QLoRA model on Open-i radiology reports, although it uses almost 8× fewer parameters than FLAN-UL2.With sufficient in-context examples, most models surpass the best QLoRA fine-tuned model on the evaluated comparisons.
  • 5.1.2 Comparison of adaptation strategies: One in-context example typically trails QLoRA for stronger models, but sufficient in-context examples can reverse this relationship and surpass FLAN-T5.The allowable number of examples varies across models and datasets.
  • 5.1.3 Effect of context length for in-context learning: One example considerably improves MEDCON over zero-shot prompting in almost all cases, while GPT-3.5 and GPT-4 outperform other models given sufficient examples.The comparison uses MEDCON scores across models and datasets.
  • 5.1.4 Head-to-head model comparison: GPT-4 generally achieves the best model win rate, whereas seq2seq models perform better on syntactic metrics but worse on semantic or conceptual matching.Across open-source models, seq2seq models outperform autoregressive models on radiology reports but underperform them on patient questions and progress notes.

5.2 Clinical reader study

Across the clinical reader study, the best adapted model generally produced summaries that physicians rated as more complete, correct, and concise than medical expert summaries, though performance varied by task and ambiguity.

  • Completeness: The best model was more complete than medical experts across all three summarization tasks, with p < 0.001.Summary lengths were comparable across datasets, so the completeness advantage was not explained by longer outputs.
  • Correctness: The best model generated significantly fewer errors overall and on two of three tasks, with p < 0.001.On radiology reports, it avoided a common medical-expert laterality error; progress-note examples also showed model hallucinations.
  • Ambiguity: Both model and expert summaries struggled with ambiguous patient questions, with the model tending toward literal reformulation while experts sometimes inferred broader meanings.Readers sometimes favored the model’s literalness and sometimes did not.
  • Correctness: The model committed misinterpretations, factual inaccuracies, and hallucinations on 6%, 2%, and 5% of samples, respectively, versus 9%, 4%, and 12% for medical experts.The study defined these as distinct categories of fabricated information.
  • Conciseness: The best model was significantly more concise overall and on two tasks, with p < 0.001, while also being more complete.Radiology reports were the only task where physicians did not prefer the model summaries; explicit length instructions may have helped other tasks.

5.3 Safety Analysis

The safety analysis compared potential harm and metric–reader-score relationships, finding lower estimated harm for the best model but emphasizing that downstream human evaluation remains necessary.

  • 14% likelihood and 22% extent of possible harm were estimated for medical-expert summaries, versus 12% and 16% for the best model.These percentages were computed over all samples, treating similar summary pairs as contributing no harm.
  • Metric correlations: BERTScore and MEDCON correlated most strongly with correctness, whereas BLEU and ROUGE-L correlated most with completeness.The metrics therefore aligned differently with clinical reader attributes.
  • Evaluation limitations: NLP metrics are limited because they rely on reference summaries that may contain errors, making human evaluation essential for assessing clinical feasibility.If human evaluation is unavailable, syntactic metrics are suggested for completeness and semantic or conceptual metrics for correctness.

5.5 Limitations

The study’s limitations constrain how broadly its findings can be interpreted, including model selection, clinical document coverage, context-specific preferences, data leakage, and potential bias.

  • Prompt and decoding choices: Temperature and prompt phrasing were explored only narrowly, leaving potential performance gains from broader tuning unmeasured.Only three temperature values and 1–2 prompt variants on a small dataset were tried.
  • Model selection: Larger Llama-2 models provide only marginal improvement over the 7B model in the analysis.The quantitative analysis focuses on selected models, including the 7B-parameter open-source tier.
  • Scope of clinical documents: The study does not cover all clinical document types, so extrapolation beyond its evaluated settings remains tentative.Progress notes come from a single medical center’s ICU notes, while longer or mixed-document tasks may require additional context length.
  • Context-specific preferences: Summary preferences may vary by specialty and clinician, but the study does not examine this granular context-specific adaptation.Examples include differing preferences among gastroenterologists, radiologists, oncologists, and subspecialty radiologists.
  • Data leakage: The study cannot verify whether proprietary or open-source models saw particular datasets during training, so data leakage remains possible.The authors highlight validating results on internal data when possible.
  • Bias and group membership: The study notes potential LLM bias, while its datasets do not contain demographic information.

6 Conclusion

The study evaluates adaptation methods for eight LLMs across diverse clinical summarization tasks and finds benefits from task- and domain-specific adaptation. Clinical readers often preferred adapted LLM summaries, while safety analysis identified limitations in both LLM and expert summaries.

  • Adaptation methods for eight LLMs were evaluated across a diverse set of clinical summarization tasks.
  • Task- and domain-specific adaptation improved quantitative clinical summarization results, although models and methods involved trade-offs.
  • LLM summaries were often preferred over medical expert summaries for completeness, correctness, and conciseness.
  • Safety analysis identified potential medical harm and fabricated information in summaries from both LLMs and medical experts.
  • The findings suggest that candidate LLM summaries could reduce documentation load, motivating prospective clinical studies.

8 Data and Code Availability

The project provides selected preprocessed datasets, experiment code, and links to open-source models, while access and redistribution remain governed by source-dataset requirements.

  • Preprocessed versions are provided for Open-i, MeQSum, and ACI-Bench, which do not require PhysioNet access.
  • The repository contains experiment code and links to open-source models hosted by HuggingFace.
  • Original datasets can be accessed through the provided references, subject to their creators’ data-sharing terms.

9 Author contributions

The contributions describe primary responsibility for data, experiments, analysis, figures, and manuscript preparation, alongside collaborators’ technical, qualitative, infrastructure, metric, fine-tuning, and reader-study roles.

  • DVV collected data, developed code, ran experiments, designed studies, analyzed results, created figures, and wrote the manuscript.
  • All authors reviewed the manuscript and provided revisions and feedback.
  • CVU, LB, and JBD contributed technical advice and additional qualitative-analysis, infrastructure, and MEDCON implementation work.
  • AA assisted with model fine-tuning, while CB, AP, MP, EPR, and AS participated in the reader study as radiologists.

A Appendix

The appendix compares in-context learning, QLoRA, model sizes, reader evaluations, and dataset-specific baselines across clinical summarization experiments. It also documents reader-study design and task-specific evaluation figures.

  • Adaptation and in-context learning: Figure A1 plots metric scores against the number of in-context examples and includes the best FLAN-T5 model fine-tuned with QLoRA as a reference line.The allowable number of in-context examples varies substantially by model and dataset.
  • Adaptation and in-context learning: QLoRA typically outperforms one in-context example with stronger open-source models, but sufficient in-context examples can reverse this relationship.The comparison is reported for patient health questions, with similar results for the Open-i radiology report dataset.
  • Model size: The larger Llama-2 model with 13B parameters delivers only marginal improvement over the 7B model for clinical summarization tasks.Each plotted data point averages scores from 250 samples for a dataset and in-context-example configuration.
  • Reader study: The reader study evaluates completeness, correctness, and conciseness across individual and pooled readers on a score range from -10 to 10.Positive scores indicate preference for the best model over the medical expert; pooled-row asterisks indicate p < 0.001 significance by a one-sided Wilcoxon signed-rank test.
  • Reader study: Reader-study annotations illustrate patient health questions and progress notes, while doctor-patient dialogue is excluded because parsing many lengthy transcribed conversations is unwieldy.Figures A4 and A5 provide example and task-average reader scores; Figure A6 presents example dialogue results.
  • Baselines and evaluation: The comparison with dataset-specific baselines emphasizes that the study does not target state-of-the-art quantitative results because NLP metrics and reader-study scores can disagree.The table compares GPT-4 with in-context learning against baselines tailored to individual datasets.
Loading 2309.07430v5…