PaperlayerView exact revision on arXiv

Source-linked AI summary

MedSonor: Learning to Use Clinical Audio in Medical LLMs

Geyang Yu, Junying Chen, Benyou Wang

arXiv:2610.22405v1cs.SD

TL;DR

Medical audio responses require interpreting acoustic evidence alongside clinical requests and dialogue context, but existing supervision and feedback provide limited support for that task. MedSonor combines collected medical audio, synthesized spoken dialogue, and audio-integrated requests with medical fine-tuning and two rubric-guided reinforcement-learning stages. It outperforms the evaluated general-purpose open-weight audio-language baselines across seven medical benchmarks and ranks highest in aggregate physician preference evaluation.

  • Problem

    Existing medical audio datasets do not fully reflect real-world clinical tasks, and task-specific feedback for open-ended medical audio dialogue remains limited.

    Source: §1 · p. 1

  • Method

    MedSonor combines collected medical audio, synthesized spoken dialogues, and authentic audio paired with clinical requests for medical fine-tuning, followed by two stages of rubric-guided reinforcement learning.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference on 30 MedVoice-AID cases.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support adapting medical voice assistants through supervision that connects audio evidence to clinical requests, with broadly retained speech capabilities.

    Source: §6 · p. 9

    The datasets are English-only, and MedSonor is for research use only rather than clinical diagnosis without rigorous prospective validation and appropriate clinical oversight.

    Source: §6 · p. 9

Abstract

from arXiv · show

Clinical audio conveys medical information that transcripts may not preserve, but using it in medical responses requires interpreting it alongside clinical requests and dialogue context. Existing medical audio supervision largely targets classification or recognition, while general audio-language models lack specialized medical knowledge and direct feedback for open-ended responses. We present MedSonor, a medical dialogue model integrating medical knowledge, spoken dialogue, and clinical audio understanding. We pair authentic medical audio with clinical requests and combine these audio-integrated dialogues with collected medical audio and synthesized spoken dialogues for supervised fine-tuning, followed by two stages of rubric-guided reinforcement learning. We introduce MedVoice-Dial and MedVoice?AID to evaluate responses within supplied multi-turn medical contexts and to requests involving clinical audio, respectively. Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference evaluation, while broadly retaining general speech capabilities.

1 Introduction

Medical audio responses must integrate acoustic cues with clinical requests and dialogue context, yet existing datasets and feedback offer limited support for that task. MedSonor combines medical audio, spoken dialogue, and rubric-guided reinforcement learning, and leads the evaluated open-weight baselines across seven medical benchmarks.

1 Introduction

  • Existing medical audio datasets do not fully reflect real-world clinical tasks, and improving responses remains difficult without task-specific feedback for medical audio dialogue.Medical voice assistants must interpret acoustic cues alongside clinical requests and dialogue context.Source: §1 · p. 1
  • MedSonor combines standardized medical audio, synthesized spoken dialogues, and authentic audio paired with clinical requests, then uses instance-specific rubrics to guide two reinforcement-learning stages.Severity-based reward caps limit rewards when judges detect major or critical violations.Source: §1 · pp. 1–2
  • MedSonor is evaluated on MedVoice-Dial for responses within supplied multi-turn medical contexts and MedVoice-AID for clinical requests containing medical audio.The evaluation also includes five public or speech-adapted medical benchmarks.Source: §1 · p. 1
  • Across seven medical benchmarks, MedSonor leads the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference on 30 MedVoice-AID cases.General speech capabilities are broadly retained.Source: §1 · p. 2

2 Background

Medical audio can convey both spoken information and acoustic evidence, but interpreting a sound alone does not determine an appropriate response. Responses depend on the recording, clinical request, and any supplied dialogue history.

2.1 Clinical Role of Medical Audio

  • Medical audio spans physiological sounds, spoken information, and vocal cues; acoustic findings require clinical context and do not establish diagnoses alone.Respiratory and cardiac sounds support auscultation, while voice characteristics can inform research on laryngeal function, neurological status, and emotional state.Source: Table 1 · p. 2, §2.1 · p. 2

2.2 From Sound Interpretation to Clinical Responses

  • Representative models cover different aspects of medical audio and interaction, and their reported scope describes task coverage rather than equivalent performance or clinical validation.HeAR focuses on health-acoustic representations, AMIE on text-based consultation, and Qwen2.5-Omni on general audio understanding and dialogue.Source: §2.2 · p. 2, Table 2 · p. 3
  • A sound may call for description, an explanation of its relevance, or guidance on what information is needed next, depending on the request and dialogue history.A fluent medical answer may still misinterpret the sound or draw conclusions beyond the evidence.Source: §2.2 · p. 2
  • Sound-centered datasets provide limited supervision for connecting acoustic evidence to clinical responses, while text-synthesized speech dialogues do not teach interpretation of authentic recordings.A sound description can miss the clinical question, and a medically plausible answer can contradict the recording or dialogue history.Source: §2.2 · p. 3

3 Method

MedSonor uses a Qwen2.5-Omni backbone and combines three complementary medical data sources with two sequential rubric-guided reinforcement-learning stages. The data and training design connects authentic audio with clinical requests while also supporting medical content and contextual dialogue.

3.1 Overview

  • MedSonor receives user audio and, when available, dialogue history and transcript context; its Qwen2.5-Omni backbone generates text through the Thinker and speech through the Talker.Evaluation uses the Thinker’s text outputs for consistent response-content assessment.Source: §3.1 · p. 3

3.2 Data Construction

  • Data construction combines collected medical audio, synthesized spoken multi-turn dialogues, and single-turn dialogues pairing authentic medical audio with clinical requests.Collected audio includes physiological sounds and clinically relevant speech; synthesized dialogue covers 25 specialties and 15 scenarios, while audio-integrated requests pair clips with scenarios.Source: §3.2 · p. 3, Figure 1 · p. 4
  • Instance-specific rubrics specify required content, contextual constraints, and prohibited errors to provide item-level feedback for reinforcement learning.For spoken dialogues, rubrics use preceding user and reference-assistant turns; for audio-integrated requests, they use the clip description.Source: §3.2 · p. 3

3.3 Training

  • Training proceeds from medical supervised fine-tuning to medical dialogue reinforcement learning and then audio-integrated dialogue reinforcement learning, with both RL stages using severity-guarded rubric rewards.SFT uses collected medical audio, synthesized spoken dialogues, and audio-integrated dialogues.Source: §3.3 · p. 4, Figure 2 · p. 4
  • Separate local judges evaluate dialogue and audio-integrated responses against positive requirements, prohibited errors, and history-derived constraints; the reward also penalizes length and truncation.Major violations cap the reward at zero and critical violations at −0.5, before clipping to [−1, 1].Source: §3.3 · p. 5
  • Medical dialogue RL trains on multi-turn instances with group-relative advantages and KL regularization, while audio-integrated dialogue RL applies the same optimization to spoken requests paired with clinical recordings.The audio-integrated stage uses a separate judge.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate medical dialogue and clinical-audio responses, compare MedSonor with open-weight baselines, and test training components and noise robustness. MedSonor ranks first across the reported medical benchmarks and physician preference study, while retaining broad general speech capabilities.

4.1 Evaluation Protocol

  • Medical-response benchmarks include spoken latest turns, text histories, and hidden references; general-speech evaluations compare the backbone, SFT, and full model under fixed inputs and decoding.The medical evaluations use GPT-5.4 signed instance-rubric scoring; general speech tests include recognition, instruction following, and dialogue tasks.Source: §4.1 · p. 5

4.2 Results

  • Physicians rank MedSonor above all six open-weight baselines across 30 MedVoice-AID cases.The study included two attending and four resident physicians; pairwise reviewer agreement across shared comparisons was 59.6% (95% item-bootstrap interval: 46.0–73.8%).Source: §4.2 · p. 5, Figure 3 · p. 6
  • MedSonor ranks first among evaluated open-weight systems across seven medical response evaluations spanning medical dialogue, clinical audio, and adapted spoken-response tasks.The comparisons cover LiveMedBench Clean-TTS, MMedFD-Lite, MedVoice-AID, MedVoice-Dial, ACI-Bench, HealthDial, and ThReadMed-QA.Source: §4.2 · pp. 5–6, Table 3 · p. 6, Table 4 · p. 7
  • MedSonor broadly retains the backbone’s speech capabilities, with improvements on several dialogue and instruction tasks but declines on English ASR and OpenHermes-Audio.Table 5 summarizes speech recognition, spoken dialogue, and open-ended instruction following.Source: §4.2 · pp. 6–7, Table 5 · p. 7

4.3 Ablation Study

  • SFT improves over the backbone on four of five tasks, and the full pipeline improves over SFT on all five; RL-only beats the backbone only on MedVoice-Dial.RL-only falls below the full pipeline on every task, supporting medical SFT before reinforcement learning.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Removing spoken dialogue most affects conversation-context tasks, while removing collected medical audio reduces performance across all evaluated tasks.The ablations support combining both sources in medical SFT.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Removing audio-integrated dialogue produces the largest performance decline on MedVoice-AID, supporting explicit training to connect medical audio with clinical requests.Together with the spoken-dialogue ablation, this suggests complementary supervision formats contribute to handling audio and dialogue context.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Robustness to Background Noise

  • MedSonor achieves the highest medical-response scores at both tested noise levels, although all evaluated models score lower under strong than moderate noise.The test overlays babble noise on complete input audio and evaluates responses with the same signed-rubric metric.Source: §4.4 · p. 8, Table 6 · p. 9

5 Related Work

Prior work addresses medical knowledge, dialogue, and health-related acoustics, while rubric-based learning provides feedback for open-ended responses. This paper focuses on connecting acoustic evidence with responses across clinical requests and interaction contexts.

Medical models

  • Medical language models and dialogue systems advance medical knowledge and interaction, but their reported tasks do not directly address clinical acoustic evidence in spoken consultation.Examples include medical QA, literature pretraining, dialogue supervision, and simulated consultations.Source: §5 · p. 9

Medical audio

  • Medical-audio work spans general audio understanding, health and respiratory acoustics, spoken consultation, and cardiopulmonary tasks; this work emphasizes diverse sounds in request and consultation context.The cited approaches include Qwen2-Audio, Qwen2.5-Omni, HeAR, OPERA, SpeechMedAssist, and StethoLM.Source: §5 · p. 9

Rubric-based learning

  • Rubric-based methods provide criteria for medical response evaluation and open-ended learning; local item-level judges assess acoustic interpretation and clinical appropriateness, with severity-based caps for critical errors.The passage describes HealthBench, MedDialogRubrics, Rubrics as Rewards, and AudioRubrics as related approaches.Source: §5 · p. 9

6 Conclusion

MedSonor combines multimodal medical SFT with two-stage rubric-guided RL and leads the evaluated open-weight audio-language baselines across seven medical benchmarks. The conclusion also bounds its use to research pending prospective validation and clinical oversight.

6 Conclusion

  • Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference on 30 MedVoice-AID cases.Evaluations also show broadly retained speech capabilities and relative stability under babble noise.Source: §6 · p. 9

Ethics and Safety Considerations

  • Deployment requires qualified healthcare oversight, audio-quality checks, output review, and ongoing monitoring, with uncertain or high-risk cases escalated to professionals.Healthcare professionals retain responsibility for clinical decisions.Source: Ethics and Safety Considerations · p. 9

AI Use Statement

  • LLMs assisted with language editing, data synthesis, and code implementation; generated code was verified and tested.The authors take responsibility for the final content, including AI-assisted artifacts.Source: AI Use Statement · p. 10

A.1 Department–Scenario Design

  • The synthesized dialogue design crosses 25 medical departments with 15 clinical interaction scenarios to yield 375 combinations spanning specialties, roles, environments, and urgency levels.The design specifies topics, user roles, and objectives for each setting.Source: §A.1 · p. 14

B Data Statistics and Filtering

This section describes filtering, severity-based reward handling, and the sequential training setup for medical SFT and two RL stages.

B Data Statistics and Filtering

C Reward and Severity Details

  • When explicit severity labels are absent, negative items are classified using weight magnitude and safety-sensitive keywords, with thresholds of 8 for critical and 5 for major items.Other negative items are minor; explicit severity labels are used directly when available.Source: C Reward and Severity Details · p. 15
  • The length penalty rises linearly from zero at the free threshold to pmax at the hard threshold, while nonempty responses lacking EOS incur a truncation penalty.Medical dialogue RL and audio-integrated dialogue RL use pmax = pEOS = 0.15 and a 256-token completion budget; thresholds depend on reference length or default to 120/200 tokens.Source: C Reward and Severity Details · p. 15

D Training Configuration

  • Training starts from Qwen2.5-Omni-7B and proceeds through medical SFT, medical dialogue RL, and audio-integrated dialogue RL.Table 12 summarizes stage data sizes and hyperparameters.Source: D Training Configuration · p. 15, Table 12 · p. 15
  • SFT freezes the visual encoder and trains the language model and aligner; RL uses DAPO, KL regularization, and constant learning rates after warmup.All stages use fused AdamW, BF16, gradient checkpointing, and seed 42; the visual branch receives no image or video inputs during RL.Source: D Training Configuration · p. 15

D.1 Local judge training and validation

This section details the local judges, evaluation inputs, and an ACI-Bench adaptation that scores dialogue responses rather than visit notes.

D.1 Local judge training and validation

  • Both local judges fine-tune Qwen3-4B-Instruct for one epoch with LoRA and GPT-5.4 binary labels on response–criterion pairs.Table 13 reports data sizes and agreement with held-out GPT-5.4 labels.Source: §D.1 · p. 15, Table 13 · p. 16
  • Judge training includes responses from GPT-4.1, GPT-4o-mini, and GPT-4.1-mini, with disjoint training and validation conversations.Audio splits are disjoint by source-audio family; replay uses only dialogue training data, and judges receive audio evidence as reference text.Source: §D.1 · p. 15
  • Baselines include seven general-purpose open-weight models and closed-source references Gemini-2.5-Flash, Gemini-3.7-Flash, and GPT-Audio.The open-weight set includes Qwen2.5-Omni-7B, MiniCPM-o-2.6, Baichuan-Omni-1.5, GLM-4-Voice-9B, Step-Audio-2-mini, LLaMA-3.1-8B-Omni, and SpeechGPT-2.0-preview.Source: §D.1 · p. 15
  • The ACI-Bench adaptation converts a patient utterance to speech, supplies earlier turns as text, and scores the held-out doctor utterance rather than generating a visit note.Its rubric scores measure dialogue-response quality under the adaptation, not original ACI-Bench note-generation scores.Source: §D.1 · p. 15
  • Evaluation settings provide matched task content in each model’s input format, while current-target references and scoring rubrics remain judge-only.MedVoice-Dial supplies user speech with its original TTS source text and reference-assistant history.Source: Table 14 · p. 16, §D.1 · p. 16

E Audio Data

This section describes the final SFT manifest, collected-audio source coverage, and a limitation involving reconstructed Bridge2AI-Voice waveforms.

E Audio Data

  • The final SFT manifest includes 6,000 audio-integrated examples from two 3,000-example sources, while 1,400 validation examples are counted separately.Table 15 describes the selected and assembled training-example composition.Source: E Audio Data · p. 16, Table 15 · p. 16
  • The collected medical-audio stream contains 7,569 selected training examples partitioned by source domain, with provenance and source references listed separately.Table 16 gives the source-domain breakdown; Table 17 lists selected counts and dataset references, not full source-dataset sizes.Source: E Audio Data · p. 16, Table 16 · p. 17, Table 17 · p. 17
  • The 349 Bridge2AI-Voice training examples use waveforms reconstructed from spectrograms, not original clinical recordings, and may have altered clinically relevant acoustic details.They should not be treated as evidence of diagnostic performance on original Bridge2AI-Voice recordings.Source: E Audio Data · p. 17

F Human Evaluation Details

The physician evaluation compares MedSonor with six open-weight baselines across 30 cases, using blinded, randomized comparisons. Shared-item physician agreement was 59.6%, with a 95% item-bootstrap interval of 46.0–73.8%.

Cohort and assignment

  • The blinded physician evaluation compared MedSonor with six open-weight audio-language baselines across 30 cases.Each physician completed 120 comparisons, including 18 shared by all reviewers and 102 unique assignments.Source: F Human Evaluation Details · pp. 17–18
  • Six physicians each completed 120 comparisons, totaling 720 judgments; after 12 judgments were unjudgeable, the remaining 708 covered 621 comparisons with at least one valid vote.Source: F Human Evaluation Details · p. 18

Preference aggregation and uncertainty

  • Preference credits average wins, ties, and losses as 1, 0.5, and 0, respectively, before reviewer averaging and Bradley–Terry fitting.Uncertainty intervals came from 2,000 source-case-cluster bootstrap resamples that retained all comparisons from each sampled case.Source: F Human Evaluation Details · p. 18

Shared-item agreement

  • 59.6% agreement (95% item-bootstrap interval: 46.0–73.8%) was observed across comparable reviewer pairs on 18 shared comparisons.The shared items had 105 valid judgments; four comparisons received the same choice from all six physicians.Source: F Human Evaluation Details · p. 18

G Prompt Templates and Evaluation Protocols

The appendix documents prompt templates and protocols for data synthesis, training, and evaluation. It describes how dialogue and audio cases are constructed, rubric feedback is used, and benchmark-specific evaluation is conducted.

G Prompt Templates and Evaluation Protocols

G.1 Data synthesis prompts

  • Dialogue synthesis uses a controlled department–scenario pair and requires role-consistent, natural spoken English in structured JSON output.The dialogue template specifies four user turns and four assistant turns that progress logically.Source: §G.1 · p. 18
  • Audio-case synthesis separates observable audio evidence from hidden metadata and produces a spoken request, reference response, construction instructions, and rubric.The template requires claims grounded in supplied audio and prohibits inferring diagnoses from a recording alone.Source: §G.1 · p. 19

G.2 Training input and reward prompts

  • The canonical policy message is shared by medical dialogue SFT and RL, while the current target response, rubric, and reward are withheld during rollout.Preceding reference-assistant turns remain available as dialogue history.Source: §G.2 · p. 19
  • Instance-specific rubrics contain 8–12 binary criteria, with positive items in the majority and signed weights reflecting clinical importance.A second review prompt removes redundant or logically inverted items before training.Source: §G.2 · p. 19
  • Local Qwen3-4B-Instruct judges assess one rubric criterion per response, returning a binary label for satisfaction or prohibited behavior.The judge uses dialogue and judge-only evidence, treating gold audio evidence as authoritative rather than inferring findings from an audio placeholder.Source: §G.2 · p. 20

G.3 Medical benchmark judge prompts

  • Medical benchmark evaluation uses separate external scorers, while signed-rubric scoring receives preceding dialogue and historical constraints unavailable to simpler query-and-response judging.Arena comparisons use a separate blinded pairwise prompt with model identities and training stages hidden.Source: §G.3 · p. 20

G.4 General speech evaluation prompts

  • General speech evaluations use fixed examples and greedy decoding, with policy prompts varying by benchmark while judge references remain outside policy input.ASR uses WER/CER after fixed normalization, and paired intervals use 20,000 bootstrap replicates.Source: §G.4 · p. 21

H Complete Ablation Results

The complete ablation results are reported in Table 19, whose caption defines the compared training configurations and scoring protocol. The supplied passages do not give the table’s numerical outcomes.

H Complete Ablation Results

  • Table 19 provides the complete numerical results for Figure 5.Source: H Complete Ablation Results · p. 22
  • Table 19 reports GPT-5.4 signed instance-rubric scores for ablations of multimodal medical SFT and its training sources.All variants use the same evaluation protocol.Source: Table 19 · p. 22
  • The ablation contrasts SFT without RL, RL-only from Qwen2.5-Omni-7B, source-removal variants, and the full pipeline.The full pipeline applies SFT followed by medical dialogue RL and then audio-integrated dialogue RL.Source: Table 19 · p. 22

Related papers

Loading 2610.22405v1…