PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Modality Gap in Long-Form Clinical Audio: A Comparative Study of Lightweight and Heavyweight End-to-End SOAP Generation

Ziyu Zhang, Mingchen Shao, Wenjie Tian, Tianlun Zuo, Longhao Li, Lei Xie

arXiv:2609.14467v1cs.SDcs.AIeess.AS

TL;DR

Long-form audio reasoning remains challenging for clinical documentation. The ASLP team trains an end-to-end multimodal system to generate SOAP notes directly from audio, and reports that it outperforms cascaded ASR+LLM baselines.

  • Problem

    Reasoning over long-form audio exceeding five minutes remains challenging, including for automated clinical documentation from doctor-patient conversations.

    Source: §I · p. 1

  • Method

    The ASLP team proposes an end-to-end multimodal clinical SOAP generation pipeline, using multi-task instruction tuning and clinical reward-based GRPO.

    Source: §I · p. 1, §VI · p. 3

  • Results

    The best Lightweight model exceeded Whisper-Large-v3 and Qwen3-ASR pipelines by over 0.122 Concept F1, while the Heavyweight model widened the margin to 0.230.

    Source: B. Main Results and Comparative Analysis · p. 3

  • Takeaways & Limitations

    The authors conclude that scaling improves factual accuracy and reduces hallucinations, while lightweight models remain limited by representation capacity.

    Source: §VI · p. 3

Abstract

from arXiv · show

Automating clinical documentation from long-form doctor-patient conversations remains challenging for modern audio-language models. While cascaded ASR systems perform well, end-to-end (E2E) models often struggle with information loss and hallucinations on extended audio. For the BeTraC 2026 challenge, the ASLP team presents a fully E2E multimodal system that generates structured SOAP notes directly from audio, bypassing intermediate transcripts. We constructed a 1.41-million-sample multi-task corpus and applied a multi-stage pipeline: domain pre-training, supervised fine-tuning, and reward optimization. Evaluating the architecture under both Lightweight (3B) and Heavyweight (30B) constraints reveals that each training stage progressively enhances performance. Furthermore, scaling to 30B parameters substantially boosts concept extraction and summarization quality. Ultimately, our E2E systems consistently outperform representative cascaded ASR+LLM baselines, proving the efficacy of direct multimodal optimization for clinical documentation.

I. INTRODUCTION

Long-form audio reasoning remains challenging, including for clinical documentation from extended conversations. The ASLP team proposes an end-to-end multimodal system and a four-stage training approach, with comparisons across model scales.

I. INTRODUCTION

  • Audio reasoning beyond five minutes remains challenging, including generating structured SOAP notes directly from long doctor-patient conversations.The BeTraC 2026 challenge requires note generation from raw medical audio without an intermediate transcript.Source: §I · p. 1
  • The proposed end-to-end clinical generation pipeline uses a four-stage training approach to address long-audio reasoning.The stages are full-parameter pre-training, supervised fine-tuning, GRPO, and Online Preference Distillation (OPD) [18].Source: §I · p. 1
  • A specialized multitask strategy and four-dimensional GRPO reward function are designed to boost medical concept recall and reduce hallucinations.The rewards target clinical accuracy, ROUGE, formatting, and length control.Source: §I · p. 1
  • The study compares Lightweight and Heavyweight models to examine parameter capacity and fact retention in long-form end-to-end speech understanding.The two tracks provide a comparative study of model scale for long-audio hallucination mitigation.Source: §I · p. 1

II. DATASET CONSTRUCTION

The dataset combines clinical SOAP conversations with long-audio, medical-subtask, and biomedical NLP data to support multi-task training.

A. Dataset Composition

  • The unified corpus aggregates over 1.41 million valid samples and approximately 8,650 hours of audio from ten sources.The dataset summary is identified as Table I.Source: A. Dataset Composition · p. 1
  • The core SOAP data includes 7,200 long-form synthetic conversations paired with complete notes, and core SOAP tasks account for roughly 70% of training weight.The conversations average 9 minutes and reach a maximum of 47 minutes; real-world clinical audio dialogues and TTS-concatenated medical dialogue data supplement them.Source: A. Dataset Composition · p. 1
  • The corpus also includes 8,098 temporal-positioning and dense-audio-captioning samples plus 2,679 long-audio ASR samples from the MLC-SLM challenge.The long-audio ASR samples are 9–13 minutes each.Source: A. Dataset Composition · p. 1
  • Medical additions include real medical ASR, Chief Complaint extraction, medical QA, and seven biomedical NLP datasets converted to instruction-tuning format.The biomedical NLP data is included to bolster medical terminology comprehension.Source: A. Dataset Composition · pp. 1–2

B. Multi-Task Formulation

The system unifies diverse audio and text tasks in one instruction-response format, while directly generating SOAP notes from audio without an intermediate transcript.

B. Multi-Task Formulation

  • Nine instruction templates cover full SOAP generation, section extraction, ASR, medical QA, keyword extraction, and dense audio description.Samples are standardized in the ms-swift messages format.Source: B. Multi-Task Formulation · p. 2
  • The unified format maps audio plus a text instruction to a response, with attention allocated according to the specified clinical or temporal task constraints.Source: B. Multi-Task Formulation · p. 2
  • To meet the no-intermediate-transcription requirement, the system uses an end-to-end multimodal architecture rather than a cascaded ASR+LLM pipeline.Lightweight and Heavyweight configurations are evaluated within the same framework.Source: §III · p. 2
  • Patient-clinician audio is mapped directly to structured SOAP notes in a single forward pass, without generating an intermediate transcript.Source: A. End-to-End Multimodal Architecture · p. 2
  • A frozen audio encoder extracts acoustic representations, which a frozen audio-to-LLM aligner projects into the LLM embedding space.Source: A. End-to-End Multimodal Architecture · p. 2
  • A fully fine-tuned dense LLM decoder attends jointly to acoustic features and the textual instruction to autoregressively produce the SOAP note.Source: A. End-to-End Multimodal Architecture · p. 2

B. Lightweight and Heavyweight Configurations

The Lightweight and Heavyweight systems retain the same end-to-end architecture, while inference uses a fixed SOAP prompt and deterministic decoding.

B. Lightweight and Heavyweight Configurations

C. Inference Pipeline

  • At inference, audio is concatenated with the fixed instruction prompt defining the SOAP structure, which is also used in training.This permits zero-shot generation without additional system prompts or in-context demonstrations.Source: C. Inference Pipeline · p. 2
  • Greedy decoding uses T = 0, p = 1.0, a 1.15 repetition penalty, and a 2048-token maximum; post-processing extracts only dialog_id and preserves model output verbatim.These settings are used to improve stability on long recordings and ensure evaluations reflect native generation quality.Source: Figure 1 · p. 2

IV. TRAINING METHODOLOGY

The training pipeline uses supervised fine-tuning (SFT) to establish audio-to-SOAP alignment, then applies GRPO and Online Preference Distillation (OPD) to refine generation.

IV. TRAINING METHODOLOGY

  • SFT follows domain pre-training and teaches the model to generate structured SOAP notes directly from audio, establishing audio-to-text alignment and clinical formatting.Source: §IV · p. 2
  • GRPO optimizes a weighted reward spanning Concept F1, ROUGE-2, SOAP-format compliance, and a length penalty.The components reward concept extraction while penalizing unsupported medical claims, encourage lexical overlap, enforce structure, and suppress excessive or repetitive output.Source: §IV · pp. 2–3
  • OPD distills preferred chain-of-thought outputs into standard single-pass generation by training on prompts with and without reasoning trajectories.The authors describe this as a regularizer against repetition loops on ultra-long audio while preserving factual accuracy under the no-intermediate-transcription rule.Source: §IV · p. 3

A. Experimental Setup

The Lightweight and Heavyweight pipelines use different model and compute configurations, while sharing deterministic inference settings.

A. Experimental Setup

  • The Heavyweight Qwen3-Omni-30B pipeline trained on 32 NVIDIA H20 GPUs across four nodes, with global batch size 16 and learning rate 5e−6.Source: A. Experimental Setup · p. 3
  • The Lightweight Qwen2.5-Omni-3B pipeline used eight GPUs, effective batch size 16, and learning rate 1e−6.Source: A. Experimental Setup · p. 3
  • Both tracks used greedy decoding with T = 0 and p = 1.0, plus a 1.15 repetition penalty to mitigate structural loops on ultra-long audio.Source: A. Experimental Setup · p. 3

B. Main Results and Comparative Analysis

Both end-to-end tracks outperform the cascaded ASR+LLM baselines, and the 30B track achieves higher Concept F1 than the 3B track. Training-stage results show progressive gains, with GRPO contributing further improvements.

B. Main Results and Comparative Analysis

  • Lightweight Concept F1 increased from 0.2604 zero-shot to 0.3872 after SFT and 0.4082 after GRPO, with upward trends also reported for ROUGE-2 and ROUGE-3.Source: B. Main Results and Comparative Analysis · p. 3
  • The Heavyweight model's Concept F1 rose from 0.1879 at zero-shot to 0.5129 after domain pre-training, 0.5136 after SFT, and 0.5167 after GRPO.GRPO produced the best overall Concept F1 and ROUGE scores, although its final gain was relatively modest for the large model.Source: B. Main Results and Comparative Analysis · p. 3
  • Scaling from 3B to 30B raised Concept F1 from 0.4082 to 0.5167, an absolute margin of 0.108 under the same architecture and training pipeline.Source: B. Main Results and Comparative Analysis · p. 3
  • The Lightweight and Heavyweight end-to-end systems exceeded cascaded ASR+LLM baselines by Concept F1 margins of over 0.122 and 0.230, respectively.The authors report that direct end-to-end optimization is more effective than relying on intermediate speech transcription for long-form clinical summarization.Source: B. Main Results and Comparative Analysis · p. 3

VI. CONCLUSION

The conclusion presents end-to-end multimodal SOAP generation for the BeTraC 2026 challenge and identifies model scaling and representation capacity as key findings.

VI. CONCLUSION

  • The ASLP systems generate structured SOAP notes directly from long-form audio using multi-task instruction tuning and clinical reward-based GRPO.Source: §VI · p. 3
  • The conclusion reports that scaling improves factual accuracy and reduces hallucinations, while lightweight models remain limited by representation capacity.Source: §VI · p. 3
  • Future work will explore memory mechanisms and acoustic token compression for more efficient clinical reasoning.Source: §VI · p. 3
Loading 2609.14467v1…