PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?

Nathan Brake, Thomas Schaaf

arXiv:2404.06503v1cs.CL

TL;DR

Clinical-note generation must organize encounter information into SOAP sections using appropriate clinical terminology, and the paper asks how note-generation designs compare in consistency. It compares independent section generation with joint generation and reports similar ROUGE and Factuality results, alongside improved consistency for GENMOD and Llama2–human agreement on selected consistency categories.

  • Problem

    ROUGE may not reliably reflect human preference, and consistency conflicts across clinical-note sections are not clearly captured by existing automatic metrics.

    Source: §1 · p. 2

  • Method

    The study compares a single PEGASUS-X model that generates a whole note with three separately fine-tuned models that generate individual sections, and evaluates notes using automatic metrics, human review, and Llama2.

    Source: §1 · p. 2, §4.1 · p. 3

  • Results

    GENMOD reduces ROUGE by less than 1% versus SPECMOD with no Factuality difference, while human review finds improved age, gender, and body-part consistency by 4.5%, 2.5%, and 8.5%, respectively.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Llama2 can assess selected note-consistency criteria with agreement to human reviewers, offering an additional quality measure for criteria not captured clearly by automatic metrics.

    Source: §1 · p. 2, §6 · p. 8

    The human and LLM review covers specific categories and only 40 encounters, and Llama2 performance may change with specialties or physicians unlike those used to develop its prompts.

    Source: §7 · p. 9

Abstract

from arXiv · show

Following an interaction with a patient, physicians are responsible for the submission of clinical documentation, often organized as a SOAP note. A clinical note is not simply a summary of the conversation but requires the use of appropriate medical terminology. The relevant information can then be extracted and organized according to the structure of the SOAP note. In this paper we analyze two different approaches to generate the different sections of a SOAP note based on the audio recording of the conversation, and specifically examine them in terms of note consistency. The first approach generates the sections independently, while the second method generates them all together. In this work we make use of PEGASUS-X Transformer models and observe that both methods lead to similar ROUGE values (less than 1% difference) and have no difference in terms of the Factuality metric. We perform a human evaluation to measure aspects of consistency and demonstrate that LLMs like Llama2 can be used to perform the same tasks with roughly the same agreement as the human annotators. Between the Llama2 analysis and the human reviewers we observe a Cohen Kappa inter-rater reliability of 0.79, 1.00, and 0.32 for consistency of age, gender, and body part injury, respectively. With this we demonstrate the usefulness of leveraging an LLM to measure quality indicators that can be identified by humans but are not currently captured by automatic metrics. This allows scaling evaluation to larger data sets, and we find that clinical note consistency improves by generating each new section conditioned on the output of all previously generated sections.

1 Introduction

The paper compares two PEGASUS-X designs for generating clinical-note sections from doctor–patient conversations and examines whether independent versus joint generation affects consistency. It focuses on CC, HPI, and combined A&P, omitting Objective because physical-examination information is often not verbalized in the conversations.

1 Introduction

  • Clinical-note generation uses a doctor–patient conversation to produce structured documentation with clinical terminology, commonly organized as a SOAP note.A common workflow first uses ASR to transcribe encounter audio.Source: §1 · pp. 1–2
  • The experiment covers CC and HPI in Subjective and combines Assessment with Plan as A&P.Objective is omitted because it commonly contains physical-examination information not directly verbalized in the dataset’s conversations; generating it is left for future work.Source: §1 · pp. 1–2
  • GENMOD generates the whole note with one model, whereas SPECMOD uses three separately fine-tuned models, one per section.The paper also proposes Llama2 review for quality criteria that automatic metrics may not capture clearly.Source: §1 · p. 2
  • ROUGE is a common summarization metric but may not reliably reflect human preference, motivating an additional Llama2-based quality measure.The authors expect independently generated sections to be more prone to conflicts, such as inconsistent patient gender between HPI and A&P.Source: §1 · p. 2

2 Related Work

Related work covers clinical-note generation, LLM-based evaluation, and comparisons of full-note versus section-wise generation. The paper positions its study around evaluating consistency in a larger, longer-conversation setting than prior ACI-Bench work.

2 Related Work

  • Prior clinical-note work includes section-specific generation and a modified single-model approach that generates one section at a time using section tokens and cross-attention.The latter generates from extracted conversation segments, with the section selected by a prepended special token.Source: §2 · p. 2
  • Prior studies report LLM evaluation aligned with human judgments on cohesion or preferences, while G-Eval and Prometheus provide other evaluator approaches.The paper describes its work as the first to assess LLM review of clinical notes against specific criteria.Source: §2 · p. 2
  • ACI-Bench (Yim et al., 2023) found division-based note generation (SPECMOD) outperformed full-note generation (GENMOD); this study tests whether that comparison varies with dataset size and full-note consistency.ACI-Bench used 207 encounters averaging 1,300 tokens; this study uses a larger proprietary dataset with more average tokens per conversation.Source: §2 · p. 3

3 Data

The data come from asynchronous scribing encounters, with physicians aware that audio is provided to a human scribe after the visit. The authors note that ACI-Bench differs in size, conversation length, and clinical setting.

3 Data

  • The experiment’s conversations come from an asynchronous scribing configuration, and physicians may dictate instructions to the scribe after the encounter.The physicians span 20 specialties identified using the National Provider Identification Registry.Source: §3 · p. 3
  • Training, validation, and test splits have similar length distributions for input conversations and target notes.Source: §3 · p. 3
  • ACI-Bench is not a suitable substitute training dataset because it is small, has shorter conversations, and partly uses a different ambient clinical intelligence setting.It contains 207 encounters averaging 1,300 tokens, and 112 encounters come from the ambient clinical intelligence setting.Source: §3 · p. 3

4 Methodology

The study compares a single PEGASUS-X model that generates the full CC, HPI, and A&P note with three section-specific models. It evaluates generated notes using automatic metrics and human and Llama2 judgments of consistency.

4.1 Training

  • GENMOD generates the full note with one PEGASUS-X model, whereas SPECMOD uses one model per section and combines their outputs.The generated sections are CC, HPI, and A&P.Source: §4.1 · p. 3

4.2 Evaluation

  • Automatic evaluation uses ROUGE and Factuality; human and Llama2 reviews assess four consistency criteria on 40 generated test-set notes.Reviewers and the LLM receive neither the reference note nor the conversation transcript.Source: §4.2 · p. 4

4.3 Human Evaluation

  • Five reviewers assessed notes from 40 encounters, seeing GENMOD and SPECMOD outputs one at a time in randomized order.Without reference notes, reviewers judged consistency with each note rather than correctness against a reference.Source: §4.3 · p. 5

4.4 LLM Evaluation

  • Llama2 Chat was tested with 7B, 13B, and 70B models and zero- through three-shot prompts; results report the 70B full-precision model.Each prompt evaluates one criterion for an individual note; prompts were developed using evaluation-set notes, while reported results use test-set notes.Source: §4.4 · pp. 5–6

5 Results and Discussion

The automatic-metric comparison varies with NRNS, while human and Llama2 reviews assess note consistency and reveal differences in how reviewers interpret coherence.

5.1 Automatic Metrics

  • At NRNS 12, the gap between SPECMOD and GENMOD is less than 0.5% ROUGE, and both reach the same Factuality score.At NRNS 5, SPECMOD generally outperforms GENMOD across sections by less than 3%.Source: §5.1 · p. 6

5.2 Human and LLM Review

  • All human reviewers preferred GENMOD for age, gender, and body-part consistency, supporting fewer contradictions within its notes for those categories.The review covered 80 notes: 40 from each model.Source: §5.2 · p. 6
  • For coherence, medical experts preferred SPECMOD while non-medical experts preferred GENMOD, reflecting different criteria for omissions versus contradictions.The two medical experts who treated missing A&P content as incoherent preferred SPECMOD; the three reviewers focused on contradictions preferred GENMOD.Source: §5.2 · p. 6
  • For Llama2, SPECMOD was preferred only by the 0-shot model for coherence; in 1-shot prompting, a single positive example biased age judgments toward matching that exact scenario.The 2- and 3-shot Llama CK scores were 0 because the model always selected “consistent”; Llama2 results were correlated to human preferences.Source: §5.2 · pp. 6, 8
  • Reviewer agreement was moderate for body-part consistency and minimal for coherence; reviewers differed on whether “left foot pain” and “left big toe pain” were consistent.The paper attributes low coherence agreement to different interpretations of the criterion.Source: §5.2 · p. 8
  • Even with shared interpretations of age and gender criteria, human agreement was imperfect; majority decisions were correct, but at least one reviewer erred on some notes.The randomly selected reviewer’s Cohen Kappa for age matched the 2-shot Llama model, while its gender score was lower.Source: §5.2 · p. 8

6 Conclusion

The study compares two PEGASUS-X clinical-note designs and finds similar automatic scores, while GENMOD improves the measured consistency categories. Llama2 shows agreement with human reviewers on some categories, though coherence is difficult to assess reliably.

6 Conclusion

  • Less than 1%: GENMOD reduces ROUGE compared with SPECMOD, with no difference in Factuality.The conclusion reports similar automatic scores for the two designs.Source: §6 · p. 8
  • 4.5%, 2.5%, and 8.5%: GENMOD improves measured age, gender, and body-part consistency, respectively, compared with SPECMOD.Source: §6 · p. 8
  • Cohen Kappa is 0.79 for age, 1.00 for gender, and 0.32 for body-part injury consistency between Llama2 and human reviewers.The paper presents Llama2 as a potential evaluator for criteria that are impractical to review comprehensively by humans at scale.Source: §6 · p. 8
  • Coherence is difficult to measure because its criteria may not be clearly specified for reliable interpretation by humans and LLMs.The authors identify developing a robust LLM-based clinical-note evaluation suite across more criteria as future work.Source: §6 · p. 8

7 Limitations

The study's relative comparison uses the same ASR-derived transcripts for both designs, but ASR errors may affect absolute note quality. Human and LLM evaluation is also limited in reviewer, encounter, category, and specialty scope.

7 Limitations

  • ASR errors may decrease generated-note quality, so absolute performance may depend on transcript quality even though both designs use the same dataset.Source: §7 · p. 9
  • Human evaluation covers only 5 reviewers and 40 DoPaCos, so ratings on a few encounters can have an outsized impact.Source: §7 · p. 9
  • Human and LLM review covers specific consistency categories, and Llama2 performance may change for specialties or physicians absent from the prompt-development dataset.Source: §7 · p. 9

A Effect of Llama2 Model Size and Quantization on Quality

The appendix compares Llama2 configurations and reports human and LLM review scores, but the supplied passages do not provide the underlying values.

A Effect of Llama2 Model Size and Quantization on Quality

B System Prompts

The system prompts ask the evaluator to judge four criteria in a generated note and return a binary consistency decision.

B System Prompts

  • Age is consistent if it does not change between CC and HPI; an age mentioned only once counts as consistent.The prompt says to consider only explicitly stated ages, not infer age from dates.Source: B System Prompts · p. 12
  • Gender is inconsistent when the patient's pronoun or gender differs between HPI and A&P.Source: B System Prompts · p. 12
  • Body-part consistency requires the injury location in CC to match HPI; when no relevant injury is mentioned, the prompt marks the criterion not applicable and TRUE.Source: B System Prompts · p. 12
  • Coherence asks whether A&P is reasonable given CC and HPI, addressing their conditions without contradiction.The prompt allows additional A&P content but marks a condition left unaddressed as FALSE.Source: B System Prompts · p. 12
Loading 2404.06503v1…