Source-linked AI summary
Speech-to-SOAP: End-to-End Summarization of Medical Dialogues: KIT@BeTraC 2026
Enes Yavuz Ugan, Fabian Retkowski, Yuka Ko, Thai-Binh Nguyen, Maike Züfle, Jan Niehues, Alexander Waibel
TL;DR
The paper addresses end-to-end generation of medical SOAP notes from speech by unifying heterogeneous dialogue data and adapting a speech-language model. Its pipeline combines synthetic speech, generated SOAP supervision, and multiple adaptation strategies; joint audio-text training improves Concept-F1 from 0.4780 to 0.4902, while checkpoint averaging produces the strongest submitted system across official test sets.
Problem
The work targets direct speech-to-SOAP generation to reduce reliance on intermediate transcripts while preserving speech information and reducing clinical documentation burden.
Method
The authors unify heterogeneous medical dialogue datasets through synthetic speech generation and automatically generated SOAP supervision to adapt Qwen2.5-Omni.
Results
Joint audio-text training increases Concept-F1 from 0.4780 to 0.4902 with nearly identical ROUGE scores, while intermediate adaptation and checkpoint averaging yield strong submitted systems.
Takeaways & Limitations
The final merged system achieved the best performance among the submitted systems across official test sets, indicating improved robustness under domain shift.
Abstract
from arXiv · showhide
With the advent of Large Language Models and its instruction following capabilities a promising application is the task of summarization. Within this domain of task the extractive sub-task of clinical protocolling has emerged as a topic of particular interest as it can significantly reduce the downtime and protocolling burden of health-care workers thus enabling them to focus on their core work helping humans. A further step towards automation is the direct generation of clinical notes from speech without intermediate transcripts, reducing processing time while preserving information such as coughing or other paralinguistic cues that may be lost in transcript-based systems. To this end, we present KIT's submission to this years BeTraC challenge in the lightweight track. Our main contribution is a scalable data augmentation pipeline that unifies heterogeneous medical dialogue datasets through synthetic speech generation and automatically generated SOAP supervision, enabling robust adaptation of a speech foundation model for end-to-end speech-to-SOAP generation.
I. EXPERIMENTAL SETUP
The system adapts Qwen2.5-Omni-3B for medical dialogue understanding and SOAP generation, using heterogeneous datasets unified through synthetic speech and generated supervision.
- Qwen2.5-Omni-3B is adapted for medical dialogue understanding and SOAP note generation using LoRA in LLaMAFactory.
- The training collection combines five medical dialogue datasets with varied synthetic, role-played, simulated, recorded, and text-only data sources.
- Datasets without audio receive Kokoro-82M synthetic speech, and applicable examples are converted into unified Audio→SOAP, Transcript→SOAP, and Audio→Diarized Transcript formats.
- GPT-3.5-27B generates SOAP supervision for datasets lacking SOAP-style notes, while template and concept-statistics prompts normalize note styles and terminology.
A. Experiments
The experiments examine prompting, data cleaning and duration, intermediate adaptation, modality, and reasoning strategies for end-to-end medical SOAP generation.
- Prompts: Three system prompts are compared, with two more complex prompts also tested in the model’s user instruction.
- Duration and Cleaning: Alignment-based cleaning removes non-aligned segments caused by severe TTS hallucinations, while duration filtering tests 15-, 21-, and 25-minute thresholds.
- Multi-stage Methods: Intermediate adaptation tests whether Transcript→SOAP and Audio→ASR improve later Audio→SOAP generation, alongside explicit speaker diarization.
- Multi-stage Methods: Audio→ASR adaptation yields approximately 3% speaker-attributed WER, so subsequent experiments do not pursue explicit speaker diarization.
- Training on Speech versus Text and Speech: Joint Audio→SOAP and Transcript→SOAP training is compared with audio-only training to test whether transcript supervision improves speech-to-SOAP generation.
- Chain-of-Thought Generation: Chain-of-thought supervision introduces intermediate targets for medical concepts, entities, and terminology, comparing explicit traces with natural-language reasoning prompts.
B. Training Details
Training uses parameter-efficient LoRA adaptation with frozen multimodal projection components and resource-conscious optimization settings.
- LoRA with rank r = 32 is applied to all target modules while the multimodal projector remains frozen.
- Models use AdamW, a learning rate of 1 × 10−4, cosine decay, 10% warmup, and an effective batch size of 4.
- Checkpoint selection uses the model with the lowest development-set perplexity.
C. Evaluation
Evaluation follows the official BeTraC protocol and emphasizes lexical overlap while recognizing that lexical metrics may not fully represent clinical correctness.
- The official BeTraC evaluation protocol is used for medical SOAP note generation.
- ROUGE and related lexical-overlap metrics quantify similarity between generated and reference SOAP notes.
- Lexical metrics may miss clinical correctness when semantically equivalent notes use substantially different wording.
II. RESULTS
The results examine prompt placement, multimodal supervision, intermediate adaptation, audio cleaning, and reasoning strategies for medical SOAP generation. Instruction-position prompts, joint audio-text training, and intermediate adaptation provide the clearest supported benefits, while chain-of-thought supervision does not improve direct generation.
- Prompt Ablation: Detailed prompts substantially improve summarization quality when placed as instructions, achieving the best ROUGE-2 and ROUGE-3 scores.The same prompts consistently degrade performance when used as system prompts.
- Training on Speech versus Text and Speech: Joint audio-text training increases Concept-F1 from 0.4780 to 0.4902 while producing nearly identical ROUGE scores.The authors consequently adopt joint Audio+Text→SOAP training for subsequent experiments.
- Multi-Stage Adaptation: Intermediate adaptation strategies substantially outperform the Audio→ASR baseline, with Audio→ASR followed by Audio→SOAP achieving the strongest ROUGE-2 and ROUGE-3 scores.CoT supervision achieves the highest Concept-F1, so no single strategy dominates every metric.
- Duration and Cleaning Ablation: Cleaning synthetic audio does not improve performance, while the best results use uncleaned data filtered to conversations of at most 21 minutes.Thresholds above 21 minutes slightly degrade performance.
- Chain-of-Thought Generation: None of the explored chain-of-thought variants improves over direct end-to-end SOAP generation, so all final systems use direct SOAP generation.Natural-language reasoning performs better than explicit <think> tags, but neither surpasses direct generation.
- Final Systems: Checkpoint averaging rows 13, 16, and 17 achieves the best development-set performance and is selected for the final submission.The final systems are trained using all available datasets and generated supervision.
III. DISCUSSION
The official evaluation compares representative and merged systems across in-domain, mock, and realistic dialogue test sets. The merged model performs best under domain shift, while few-shot SOAP supervision is promising but remains outside the full training pipeline.
- Official Evaluation: The official evaluation comprises the in-domain DoPaCo, Mock Dialogue, and realistic dialogue recording test sets.Table III reports representative final systems, the merged development-set submission model, and official submission results.
- Official Evaluation: The primary merged model consistently outperforms the contrastive submission across all official test sets, with the largest gains under increasing domain shift.The authors interpret this pattern as improved robustness and reduced overfitting to synthetic TTS data.
- Official Evaluation: The final merged system achieves the best performance among the submitted systems across the official test sets.The paper associates this outcome with averaging diversely trained checkpoints.
- SOAP Data Generation Prompt Variants: The few-shot SOAP examples prompt obtains the best development scores among the evaluated SOAP-generation prompt variants.The full downstream training pipeline instead uses the SOAP template plus concept statistics prompt.
IV. CONCLUSION
The paper presents a scalable pipeline for end-to-end speech-to-SOAP generation and reports that its final merged system performed best across official test sets, indicating improved robustness under domain shift.
- The submission targets end-to-end speech-to-SOAP generation in the BeTraC 2026 Lightweight Track.
- The main contribution is a scalable pipeline that unifies heterogeneous medical dialogue datasets through synthetic speech and automatically generated SOAP supervision.
- The final merged system achieved the best performance among the submitted systems across official test sets, indicating improved robustness under domain shift.