Source-linked AI summary
Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques
Kundan Krishna, Sopan Khosla, Jeffrey P. Bigham, Zachary C. Lipton
TL;DR
Creating SOAP notes from clinical conversations is burdensome, motivating automated generation of structured notes. The paper proposes extractive-abstractive pipelines, including CLUSTER2SENT, and reports stronger results than fully abstractive generation, with similar benefits on AMI. The approach also exposes supporting utterances for verification, although generated factual errors require medical-practitioner supervision.
Problem
SOAP-note creation is time-consuming and contributes to physician burnout, motivating systems that generate notes from doctor-patient conversations.
Method
The paper develops extractive-abstractive pipelines that use SOAP subsection structure and utterances linked as evidence for summary sentences.
Results
The proposed pipelines outperform fully abstractive and standard extractive-abstractive approaches, with similar trends reproduced on the AMI corpus.
Takeaways & Limitations
Section structure and annotated evidence support summaries whose sentences can be checked against localized conversation utterances.
Takeaways & Limitations
Generated SOAP notes can contain factual errors, so the methods require medical-practitioner supervision for verification and editing.
Abstract
from arXiv · showhide
Following each patient visit, physicians draft long semi-structured clinical summaries called SOAP notes. While invaluable to clinicians and researchers, creating digital SOAP notes is burdensome, contributing to physician burnout. In this paper, we introduce the first complete pipelines to leverage deep summarization models to generate these notes based on transcripts of conversations between physicians and patients. After exploring a spectrum of methods across the extractive-abstractive spectrum, we propose Cluster2Sent, an algorithm that (i) extracts important utterances relevant to each summary section; (ii) clusters together related utterances; and then (iii) generates one summary sentence per cluster. Cluster2Sent outperforms its purely abstractive counterpart by 8 ROUGE-1 points, and produces significantly more factual and coherent sentences as assessed by expert human evaluators. For reproducibility, we demonstrate similar benefits on the publicly available AMI dataset. Our results speak to the benefits of structuring summaries into sections and annotating supporting evidence when constructing summarization corpora.
1 Introduction
The paper addresses the burden of creating SOAP notes by introducing end-to-end pipelines that summarize doctor-patient conversations into sectioned clinical notes. Its best approach combines extraction, clustering, and abstractive generation, with strong automatic and human-evaluation results.
- Motivation: SOAP notes document subjective information, objective observations, assessments, and care plans, but populating them can contribute substantially to physician workload and burnout.The notes may also be subdivided into 15 subsections.
- Contribution: The paper introduces the first end-to-end methods for generating whole SOAP notes from clinical conversations.The work uses a corpus of recorded clinical conversations paired with SOAP notes written according to documentation standards.
- Approach: The proposed methods span a spectrum from fully abstractive summarization to pipelines that increasingly shift work toward extraction.CONV2NOTE performs no extraction, while later approaches select noteworthy utterances, organize them by subsection, or cluster utterances supporting the same sentence.
- Results: Experiments on AMI show similar trends, supporting the broader usefulness of methods that exploit section structure and annotated evidence utterances.The paper reports quantitative evaluation, human assessment, and qualitative error analysis.
- Approach: CLUSTER2SENT extracts and clusters noteworthy utterances before abstractively summarizing each cluster, forming the paper’s best-performing workflow.The workflow is illustrated in Figure 1 and requires clustering utterances that support the same SOAP sentence.
- Results: The predicted-utterance and predicted-cluster pipeline achieves the highest ROUGE scores and the most factual, coherent, and non-repetitive sentences in human evaluation.Its shorter sequences also make large transformer-based abstractive models such as T5 more feasible, while localizing evidence for physician verification.
2 Related Work
The related work spans extractive and abstractive summarization, conversation summarization, and medical-conversation processing. This paper differs by generating full-length, highly structured SOAP notes whose sentences may synthesize evidence from multiple dialogue turns.
- Summarization: Summarization research has progressed from extracting important content to generating abstractive paraphrases, including neural sequence-to-sequence approaches.Prior work covers news, meetings, customer service, and tourist-information conversations.
- Extractive-abstractive methods: Two-step extractive-abstractive systems typically extract or paraphrase important source sentences before generating summaries.The cited prior approaches generally assume each summary sentence is supported by exactly one source sentence.
- Research gap: Medical conversations often require summary sentences to synthesize content distributed across multiple dialogue turns, such as sequences of questions and answers.This evidence structure distinguishes the paper’s setting from one-source-sentence assumptions.
- Medical conversation summarization: Prior medical-conversation work focused on selected patient-nurse subtasks, whereas this paper targets full-length SOAP notes containing up to 15 subsections.The paper proposes methods specifically for this broader structured-generation challenge.
- Dataset: The medical resource contains recorded doctor-patient conversations paired with annotated SOAP notes, supporting research on structured clinical summarization.The corpus includes real-life visits, de-identified transcripts, speaker and timestamp annotations, and professionally created notes.
- Dataset: SOAP notes exhibit substantial structural and linguistic variation: subsection styles differ, average notes contain 27.47 sentences, and 85.22% of trigrams are novel relative to conversations.The Assessment subsection averages 5.16 sentences, while the Medications subsection commonly uses bulleted medicine names and dosages.
4 Methods
The methods decompose conversation summarization into extraction and abstraction with progressively greater responsibility assigned to extraction. The most extraction-heavy pipeline predicts subsection evidence, clusters related utterances, and generates one sentence per cluster.
- Pipeline spectrum: The paper compares four decompositions ordered from abstraction-heavy to extraction-heavy: CONV2NOTE, EXT2NOTE, EXT2SEC, and CLUSTER2SENT.These pipelines differ in whether they extract noteworthy utterances, predict their relevant sections, and cluster evidence.
- Pipeline spectrum: CONV2NOTE generates the entire SOAP note directly from the conversation, while EXT2NOTE first extracts noteworthy utterances before generating the note.EXT2NOTE does not condition extraction on the associated section.
- Pipeline spectrum: EXT2SEC extracts noteworthy utterances and predicts their relevant sections, then generates each SOAP section using only its predicted evidence.This decomposition uses section-specific extraction without the clustering stage.
- CLUSTER2SENT: CLUSTER2SENT groups noteworthy utterances likely to support the same sentence and generates a sentence from each resulting cluster.Its extraction module operates per subsection before clustering.
- Implementation: The pipelines leave model choices open, using pointer-generator or conditioned pointer-generator networks and fine-tuned T5 models according to input and section-conditioning requirements.T5 could not be used for the longest-input CONV2NOTE and EXT2NOTE settings because of GPU memory requirements.
- Clustering: The clustering heuristic merges utterances separated by at most τ utterances, iterating until every pair of close utterances shares cluster membership.τ is tuned on the validation set, and extractor thresholds are adjusted to produce approximately stable cluster counts across sections.
5 Model Architectures
The model architectures combine sequence-to-sequence abstractive generators with section conditioning and neural utterance extractors. Architecture choices reflect whether the pipeline must generate from long conversations, subsection evidence, or clustered utterances.
- Abstractive models: Pointer-generator networks encode conversations with bidirectional LSTMs and attention, while combining vocabulary generation with direct copying from the input.The model learns timestep-specific weights governing copying versus generation.
- Abstractive models: Section-conditioned pointer-generator networks inject the target subsection embedding into both encoder word representations and decoder LSTM inputs.This architecture is used for EXT2SEC and CLUSTER2SENT.
- Abstractive models: T5 supplies a pretrained encoder-decoder transformer that is fine-tuned for summarization and receives the target section name prepended to its input.Section conditioning is therefore implemented through the input text for T5.
- Extractive models: The hierarchical LSTM classifier encodes each utterance, incorporates neighboring utterance context, and predicts section-specific noteworthiness probabilities.It uses a bidirectional LSTM at both the token and utterance-sequence levels.
- Extractive models: The BERT-LSTM classifier replaces the utterance encoder with BERT while retaining the subsequent bidirectional-LSTM and linear prediction architecture.BERT-LSTM is fine-tuned end to end.
6 Experiments
Experiments compare increasingly extraction-heavy summarization pipelines, oracle evidence settings, predicted evidence, and performance across medical and AMI datasets. CLUSTER2SENT improves over the purely abstractive baseline, while context-aware extraction and dataset conditions affect results.
- RANDOMNOTE and ORACLEEXT have similar performance, and both are outperformed by CONV2NOTE.
- All algorithms using oracle noteworthy utterances outperform CONV2NOTE, with ROUGE scores rising significantly toward extraction-heavy methods.
- BERT-LSTM and hierarchical LSTM extractors outperform independent utterance predictors, with BERT-LSTM performing best.
- With predicted evidence, CLUSTER2SENT outperforms CONV2NOTE by about 8 ROUGE-1 points, despite lower scores than oracle settings.
- AMI experiments show similar ROUGE trends, but the clustering heuristic performs less well on that dataset.
- The best method still outperforms CONV2NOTE by about 11 ROUGE-1 points.
7 Qualitative Analysis
CLUSTER2SENT can generate section-specific summaries from shared supporting utterances, but its output remains vulnerable to irrelevant evidence and unsupported inferences. Qualitative examples show both its ability to localize information and several sources of factual error.
- CLUSTER2SENT generates different section-specific summaries from the same supporting utterances, placing “low blood pressure” in review of systems and “lisinopril” in medications.
- Irrelevant utterance clusters can prompt fabricated section-specific information, such as non-smoking status or flu-shot history.
- Pronouns without explicit antecedents can cause T5 models to substitute arbitrary entities, including “daughter,” “haemoglobin,” or “lisinopril.”
- The abstractive module can infer unstated medical facts, such as heart disease from digoxin or high cholesterol from pravastatin, and add unsupported actions such as “check valves.”
- Independent generation can create contradictions when repeated questions refer to different time periods, such as simultaneously denying and confirming chest pain.
8 Human evaluation
Human evaluation compared generated SOAP notes using sentence-level quality labels and aggregate measures. CLUSTER2SENT methods achieved higher post-editing yield than the fully abstractive baseline, with T5-base performing best within CLUSTER2SENT.
- Evaluation design: The evaluation labeled each sentence for factual correctness, repetition, clinical irrelevance, incoherence, and section appropriateness.Annotators also rated comprehensiveness and transcript copying on aggregate scales.
- Post-editing yield: CLUSTER2SENT methods produced higher yield than the CONV2NOTE baseline, with p< 0.02.Yield retained factually correct sentences that were neither repetitive nor incoherent.
- Post-editing yield: T5-base produced significantly more yield than the conditioned pointer-generator within CLUSTER2SENT.T5-base also produced fewer incoherent sentences.
- AMI replication: AMI evaluation showed a similar yield trend, although CONV2NOTE exceeded 0.5 redundant sentences and pointer-generator CLUSTER2SENT had many incoherent sentences.These corpus-specific error patterns qualify the general evaluation trend.
9 Conclusion
The paper presents an end-to-end approach for generating full-length SOAP notes from doctor-patient conversations using section structure and linked supporting utterances. It reports better performance than fully abstractive and standard extractive-abstractive approaches, while retaining a need for medical supervision because factual errors can occur.
- Contribution: The paper is presented as the first attempt to generate full-length SOAP notes by summarizing doctor-patient conversation transcripts.Its methods exploit SOAP-note section structure and utterance-to-sentence evidence links.
- Findings: The proposed methods outperform fully abstractive and standard extractive-abstractive approaches that do not use the available annotations.Similar results on the public AMI corpus support wider applicability to similarly structured, annotated data.
- Limitation: Generated SOAP notes can contain factual errors, so the methods are intended for supervision from a medical practitioner who can check and edit them.The paper describes section-frequency and pronoun-replacement errors requiring vigilance.
- Privacy: The study reports that conversations were manually de-identified and need not be stored long-term after SOAP-note generation.The authors therefore do not anticipate additional privacy risks from using the proposed methods.
Appendix
Appendix analyses examine decoder choices, extraction variants, copy mechanisms, and pretraining. They show benefits from section-specific extraction and conditioning, copy mechanisms, and pretrained T5 parameters.
- Additional results: Appendix results include ROUGE scores for CLUSTER2SENT using oracle noteworthy utterances with oracle and predicted clusters.These analyses isolate the effects of cluster quality and decoder configuration.
- Ablations: Both EXT2SEC ablations mostly perform worse than EXT2SEC, demonstrating benefits from section-specific extraction and section-conditioning in the abstractive decoder.ALLEXT2SEC removes per-section extraction, while EXT2SECNOCOND removes section-conditioning.
- Copy mechanism: Removing the copy mechanism reduces pointer-generator performance in the CLUSTER2SENT setting with oracle noteworthy utterances and clusters.Copy mechanisms were therefore used in all pointer-generator models.
- Pretraining: A randomly initialized T5-Base model achieved ROUGE-1 around 40, over 16 points below the pretrained T5 model.The randomly initialized model was even worse than CONV2NOTE.
Sample generated SOAP notes
The appendix documents sample outputs, decoding and training details, ASR-error simulation, and decoder comparisons. It also provides a figure describing conversation length and evidence-utterance counts per summary sentence.
- Examples: The appendix includes an obfuscated test conversation, its SOAP note generated by the best method, and the ground-truth SOAP note.Sensitive medicines, diseases, and dosages were modified because the original conversations cannot be published.
- Dataset statistics: Figure A1 summarizes conversation word counts and the number of evidence utterances supporting each summary sentence.It characterizes input length and evidence density in the medical dataset.
- Decoder comparison: The appendix compares ROUGE scores for different abstractive decoders using oracle noteworthy utterances.The referenced table reports decoder results under oracle extraction conditions.
- Decoding: Beam search uses beam size 4, with modified decoding to generate SOAP sections in proper order for vanilla pointer-generator models.The procedure starts from the first section header and checks predicted headers against the required section sequence.
- ASR simulation: ASR errors are simulated by replacing randomly selected conversation words with randomly chosen phonetically similar candidates at a specified error rate.Candidates are filtered using autocorrect suggestions and Refined SoundEx distance 1.
More Experimental Details
The experiments used multiple Nvidia GPUs and evaluated extraction, summary quality, human-rated characteristics, and section-level performance. Supplementary materials also document a sample conversation with a generated SOAP note and ground truth.
- Implementation: Models were trained on Nvidia Quadro RTX 8000, RTX 2080Ti, and V100 GPUs.
- Evaluation: Extractive modules used standard scikit-learn classification metrics, while summaries were evaluated with ROUGE calculated using pyrouge and ROUGE-1.5.5.
- Human evaluation: Table A4 reports human-rater judgments of sentence characteristics across CONV2NOTE and two Cluster2Sent variants.The variants use pointer-generator and T5-base generators, with BERT-LSTM for the medical dataset and hierarchical-LSTM for AMI.
- Section-level results: Table A5 reports average ROUGE scores for Cluster2Sent T5Base+BLSTM across SOAP sections.The table also gives the number of populated test datapoints and the average ground-truth word count for each section.
- Extraction: Table A6 reports BERT-LSTM performance for extracting noteworthy utterances across SOAP sections.
- Qualitative example: Figure A2 presents an obfuscated sample conversation alongside a SOAP note generated by the best method and the ground truth.