Source-linked AI summary
Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization
Yanis Labrak, David Grünert, Séverin Baroudi, Jiyun Chun, Pawel Cyrta, Sergio Burdisso, Ahmed Hassoon, David Liu, Adam Rothschild, Reed Van Deusen, Petr Motlicek, Andrew Perrault, Ricard Marxer, Thomas Schaaf
TL;DR
Long-context audio reasoning remains underserved by short-context benchmarks, and open-ended tasks such as summarization are difficult to evaluate automatically. The paper builds an open-weight pipeline for synthetic doctor-patient conversations, audio, and reference SOAP notes, releasing 8,800 conversations with 1,329 hours of audio. Cascaded systems substantially outperform end-to-end models, while the authors identify reasoning over long, noisy conversations as the primary challenge.
Problem
Long-context audio reasoning is understudied because benchmarks focus mainly on short-context tasks, and open-ended summaries and notes are difficult to evaluate with surface metrics.
Source: §1 · p. 1Method
The paper builds an open-weight pipeline for persona-driven doctor-patient dialogue, multi-speaker audio synthesis, and reference SOAP note production.
Source: §1 · p. 1Results
Cascaded systems substantially outperform end-to-end models, which show hallucination rates around 60% versus 20% for cascaded approaches.
Source: §5 · p. 5Takeaways & Limitations
The primary challenge is reasoning over long, noisy conversations to produce faithful clinical documentation, rather than transcription.
Source: §5 · p. 5The synthetic data still leaves a gap to real clinical audio, which future work aims to address.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Long-context audio reasoning is underserved in both training data and evaluation. Existing benchmarks target short-context tasks, and the open-ended generation tasks most relevant to long-context reasoning pose well-known challenges for automatic evaluation. We propose a synthetic data generation pipeline designed to serve both as a training resource and as a controlled evaluation environment, and instantiate it for first-visit doctor-patient conversations with SOAP note generation as the task. The pipeline has three stages, persona-driven dialogue generation, multi-speaker audio synthesis with overlap/pause modeling, room acoustics, and sound events, and LLM-based reference SOAP note production, built entirely on open-weight models. We release 8,800 synthetic conversations with 1.3k hours of corresponding audio and reference notes. Evaluating current open-weight systems, we find that cascaded approaches still substantially outperform end-to-end models.
1. Introduction
Long-context audio reasoning remains difficult to assess: existing benchmarks emphasize short tasks, while open-ended summaries and notes resist reliable automatic evaluation. The paper proposes an open-weight synthetic pipeline for training and controlled evaluation, releases 8,800 doctor-patient conversations, and reports that cascaded systems substantially outperform end-to-end approaches.
1. Introduction
- Long-context audio reasoning remains understudied because benchmarks mainly cover short-context tasks and capable long-context LALMs emerged only in mid-2025.Constructing meaningful benchmarks for these tasks is also difficult.Source: §1 · p. 1
- Open-ended summarization and notetaking are difficult to evaluate automatically because valid outputs vary, surface metrics align weakly with human judgments, and fluent outputs can hallucinate.The proposed pipeline is intended as both training signal and a controlled evaluation environment.Source: §1 · p. 1
- Cascaded systems substantially outperform end-to-end approaches on audio-to-SOAP-note generation, with near-ceiling ASR performance in the cascaded pipeline.The authors also report that all data and code will be released to support future long-context audio training and real-world validation.Source: §1 · p. 1
- The paper contributes an open-weight synthetic pipeline and a dataset of 8,800 doctor-patient conversations with audio and reference SOAP notes.The pipeline and dataset are developed in consultation with medical doctors.Source: §1 · p. 1
2. Related Work
Prior long-audio work finds that processing capacity does not guarantee reasoning ability, while clinical audio documentation faces scarce distributable conversational data. This work extends text-only synthetic medical dialogue toward multi-speaker audio and uses LLM judging for open-ended clinical summaries.
2. Related Work
- Long-context audio models can ingest lengthy recordings, but evaluations report degraded long-sequence multi-hop reasoning and end-to-end systems still trail cascaded ASR-to-text systems on open-ended generation.Reported issues include representational drift and lexical dominance when processing speech compared with text.Source: §2 · p. 1
- Public clinical conversation resources are limited: PriMock57 has 57 brief actor-performed consultations, while ACI-Bench offers 187 encounters as text without audio.Privacy restrictions limit distribution of proprietary datasets containing thousands of hours of real clinical audio.Source: §2 · p. 2
- Earlier synthetic medical dialogue methods generate text only, whereas this pipeline adds multi-speaker audio, persona-conditioned voices, and acoustic simulation.The evaluation uses an LLM-as-a-judge framework because surface metrics are inadequate for clinical summarization.Source: §2 · p. 2
3. Transcript and Audio Generation
The data-generation pipeline has three stages, from persona and context sampling through dialogue generation to audio synthesis with acoustic simulation, using Gemma3-27B-IT and SDialog.
3. Transcript and Audio Generation
- The pipeline comprises persona and context sampling, persona-conditioned text dialogue generation, and audio synthesis with acoustic simulation.Gemma3-27B-IT is the LLM, selected based on early dialogue-generation tests, and SDialog orchestrates the pipeline.Source: §3 · p. 2
- Gemma3-27B-IT supplies the language model for the pipeline, while SDialog handles orchestration.The model was selected for performance in early dialogue-generation tests.Source: §3 · p. 2
- Audio synthesis with acoustic simulation is the final of the three stated pipeline stages.The other stages are persona and context sampling and persona-conditioned text dialogue generation.Source: §3 · p. 2
3.1. Sampling Personas
Structured persona sampling is used to increase dialogue diversity after direct prompting produced only three unique doctor-patient pairs among 2,800 dialogues. Personas combine shared doctor-patient attributes with role-specific traits and sampled chief complaints.
3.1. Sampling Personas
- Direct prompting yielded only three unique doctor-patient pairs among 2,800 dialogues, motivating structured persona sampling to increase diversity.Most directly prompted dialogues shared the same visit reason and interpersonal dynamics.Source: §3.1 · p. 2
- Doctor and patient personas share demographic and interaction attributes, while patient-only traits include English fluency, occupation, insurance, and visit reason.Doctor-only attributes include years of experience; attributes are sampled uniformly from predefined lists except the chief complaints.Source: §3.1 · p. 2
- The 724 chief complaints were compiled by a clinically expert co-author using expert elicitation and span typical and atypical primary-care clinical or administrative presentations.The complaints contain no personally identifiable information and are to be released with the code.Source: §3.1 · p. 2
3.2. Personas →Text Dialogue
The authors use turn-by-turn persona-conditioned generation to make conversations less smooth and more interpersonally challenging. Generated doctors also show higher language complexity and longer turns than either reference set.
3.2. Personas →Text Dialogue
- Multi-turn generation raised interpersonal-challenge ratings from 2.01 to 2.50 and persona-adherence ratings for doctors from 3.10 to 3.30 and patients from 3.60 to 3.90.The ratings were collected on a 2,800-dialogue subset; the authors hypothesize turn-by-turn generation helps the model focus on the active role.Source: §3.2 · p. 2
- Generated doctors had a fog index of 10.4 versus 6.6–7.0 in the reference sets and longer turns, possibly because Gemma 3 was trained primarily on written text.Direct prompting for conversational style had a moderate effect; few-shot Mocks were excluded after causing personality and topic leakage without meaningfully reducing complexity.Source: §3.2 · p. 2
- Multiple rounds of physician feedback informed iterative refinement of the dialogue personas and prompts.The reference set included three mock clinical encounter recordings with an actor and real physician in a realistic setting.Source: §3.2 · p. 2
3.3. Text Dialogue →Conversation Audio
The pipeline maps persona-based dialogues into audio using voice synthesis, conversational timing, sound events, and room acoustics. Signal augmentation adds compression artifacts and balances speaker amplitude; perceptual fidelity scores are close to the stated reference recordings.
3.3. Text Dialogue →Conversation Audio
- 8,800 dialogues were synthesized from 200 patient and 100 doctor personas, with conversational artifacts and acoustic triggers included in the text.This design creates a cross-register task: mapping informal, spontaneous, acoustically noisy speech into formal clinical documentation.Source: §3.3 · p. 3
3.3.1. Persona-Conditioned Voice Synthesis
- Speaker voices were assigned once using persona gender and kept disjoint across training, development, and test; age-based voice estimates were rejected as often inaccurate.Random age assignment was retained, particularly after age estimates for elderly voices proved inaccurate in listening tests.Source: §3.3.1 · p. 3
3.3.2. TTS Engine Selection and Subjective Assessment
- Qwen3-TTS-1.7B was selected from nine TTS configurations for linguistic accuracy, expressiveness, and permissive licensing, with dry WER below 2%.XTTS was excluded because of licensing restrictions.Source: §3.3.2 · p. 3
3.3.3. Acoustic Simulation and Flow Enrichment
- Audio enrichment combines voice matching and neural synthesis with overlap and pause insertion, sound events, scene composition, and room-acoustic simulation.The pipeline adds a room impulse response for an 8 m^2 examination room and background noise; 66 sound-event classes are placed using forced alignment and dialogue stage directions.Source: §3.3.3 · p. 3
3.3.4. Signal Augmentation
- 1.27 UTMOS was achieved, close to the in-person Mocks baseline (1.28) and DISPLACE-M reference data (1.29).The audio uses Opus at 16 kbps, approximately 14:1 compression, and scales patient amplitude to one-quarter of the doctor’s.Source: §3.3.4 · p. 3
3.4. Summary of Dataset
Synth-DoPaCo6 provides synthetic doctor-patient dialogues with long-form audio and non-speech events. The wet audio is intelligible yet acoustically challenging, and the reported dataset production involved substantial GPU use.
3.4. Summary of Dataset
- 8,800 dialogues total 1,329 hours of audio, averaging 28 turns, 1,500 words, 37 non-speech events, and 9 minutes per dialogue.Dialogue lengths range from 2 to 47 minutes; the dataset is split into train, test, and development sets, with statistics summarized in Table 2.Source: §3.4 · p. 3, Table 2 · p. 3
- Whisper Large V3 achieves 2–3% WER on wet audio, while Qwen3-ASR shows 10–14%, indicating intelligible but acoustically challenging audio.The Qwen3-ASR gap is primarily driven by substitutions and deletions, exacerbated by Opus compression.Source: §3.4 · p. 3
- TTS synthesis used approximately 2.5k GPU-hours, and SOAP-note generation used approximately 300 GPU-hours on two NVIDIA A100 GPUs.TTS ran on one NVIDIA A100 40 GB GPU; Qwen3-32B SOAP-note generation ran on two NVIDIA A100 40 GB GPUs.Source: §3.4 · p. 3
4. SOAP Note Generation and Evaluation
This section describes reference SOAP-note generation from speaker-attributed transcripts and evaluates generated notes with LLM-based and reference-grounded approaches. The reported comparison finds a large performance gap between cascaded and end-to-end systems, while the dataset and evaluation have stated scope limitations.
4.1. Reference SOAP Note Generation
- The reference-note pipeline first extracts transcript-grounded facts into structured JSON with supporting quotes and turn indices, then generates notes using only that fact table.Terminology may be rewritten into standard clinical language, but adding clinical content beyond extracted facts is prohibited; clinical feedback also shaped note organization and excluded administrative content.Source: §4.1 · p. 4
4.2. SOAP Note Evaluation with LLM-as-a-Judge
- The LLM-as-a-judge pipeline extracts atomic note claims and labels their transcript support, scoring 12 dimensions across ratings, counts, and two rates.The four 1–5 ratings are faithfulness, structure, coverage, and conciseness; the two rates are unsupported claims and contradictions.Source: §4.2 · p. 4
4.3. Comparison of Open-Weight LALMs
- Four baselines include two ASR-to-Qwen3-32B cascades and two Qwen3-Omni models that generate notes directly from audio.The cascades use Qwen3-ASR or Whisper Large V3 without speaker attributions; the Omni models are 32B parameters.Source: §4.3 · p. 4
- End-to-end Qwen3 systems show near-minimal faithfulness and coverage, with hallucination rates of 0.99–1.00 versus 0.21–0.23 for cascaded systems and 0.01 for reference notes.The comparison covers three judged outcomes; Table 3 presents a subset of judge dimensions.Source: §4.3 · p. 4, Table 3 · p. 4
4.4. Reference-Grounded SOAP Note Evaluation
- Reference-grounded ROUGE F1 and Open Medical Concept F1 also show a large gap between cascaded and end-to-end systems.The concept metric extracts medical concepts from reference and hypothesis notes and computes F1 over their overlap; Table 4 reports dev-set F1 scores.Source: §4.4 · p. 4, Table 4 · p. 4
- Reference notes were LLM-generated, and the dialogues are limited to English, two-speaker primary-care first visits; acoustic sim-to-real comparison remains future work.Whisper’s low wet-audio WER (<3%) suggests the acoustic simulation may not fully capture the difficulty of real clinical recordings.Source: §4.4 · p. 4
5. Conclusion
The synthetic pipeline yields 8,800 doctor-patient conversations and 1,329 hours of audio; cascaded systems substantially outperform end-to-end models, whose hallucination rates are higher. The reported results place the main challenge in reasoning over long, noisy conversations to produce faithful clinical documentation.
5. Conclusion
- The fully synthetic, open-weight pipeline produces 8,800 conversations and 1,329 hours of audio, with reference SOAP notes.Source: §5 · pp. 4–5
- End-to-end systems hallucinate around 60% of the time versus 20% for cascaded approaches, which substantially outperform them.Source: §5 · p. 5
- With Whisper Large V3 at 2–3% WER on wet audio, the primary challenge is reasoning over long, noisy conversations, not transcription.Source: §5 · p. 5
- Future work targets the synthetic-to-real clinical audio gap through multi-party scenarios, accent-conditioned voices, and more natural conversations, and proposes extension to other professional domains.Source: §5 · p. 5
8. Generative AI Use Disclosure
Generative AI was used both as part of the research methodology and to support manuscript preparation and experimental coding. The authors reviewed the manuscript-related assistance and distinguish development-only evaluation from the released pipeline.
8. Generative AI Use Disclosure
- LLMs assisted with proofreading, concision, and LaTeX table formatting; authors directed and reviewed this assistance, and AI produced no significant manuscript portions without subsequent human revision.Source: §8 · p. 6
- AI assistance also supported experimental code and HPC scripts; Claude Sonnet 4 evaluated dialogue generation during development only, not as part of the released dataset or final system.Source: §8 · p. 6
- Gemma3-27B-IT generated dialogues, Qwen3-TTS synthesized speech, and Kimi K2 Thinking generated reference SOAP notes and served as the LLM-as-a-judge evaluator.Source: §8 · p. 6