Source-linked AI summary
Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models
Kyudan Jung, Jihwan Kim, Soyoon Kim, Jeonghoon Kim, Jaegul Choo, Cheonbok Park
TL;DR
Full-duplex SLMs lack sufficient high-quality multi-speaker conversational data, especially for overlapping dialogue. Sommelier introduces an open, scalable preprocessing pipeline that preserves conversational structure while improving diarization and transcription quality. The pipeline is validated by fine-tuning Moshi on Sommelier-processed speech.
Problem
High-quality multi-speaker conversational data suitable for full-duplex SLM training remains scarce, while existing resources provide limited supervision for overlapping speech.
Method
Sommelier combines modular conversational-audio processing with diarization, overlap separation, and ensemble-based ASR while preserving overlaps and backchannels.
Results
Sommelier is validated by fine-tuning Moshi on processed speech, with improvements reported across overlap scenarios and full-duplex interaction behaviors.
Takeaways & Limitations
The released pipeline supports reproducible industrial research and aims to accelerate progress toward natural, real-time human–AI interaction.
Takeaways & Limitations
The pipeline focuses exclusively on speech and produces slightly lower audio fidelity than datasets recorded with isolated channels.
Abstract
from arXiv · showhide
As the paradigm of AI shifts from text-based LLMs to Speech Language Models (SLMs), there is a growing demand for full-duplex systems capable of real-time, natural human-computer interaction. However, the development of such models is constrained by the scarcity of high-quality, multi-speaker conversational data, as existing large-scale resources are predominantly single-speaker or limited in volume. Addressing the complex dynamics of natural dialogue, such as overlapping and back-channeling remains a challenge, with standard processing pipelines suffering from diarization errors and ASR hallucinations. To bridge this gap, we present a robust and scalable open-source data processing pipeline designed for full-duplex model.
1 Introduction
Full-duplex SLM development is constrained by scarce, high-quality multi-speaker conversational data and difficult overlap, backchanneling, and transcription conditions. Sommelier addresses these challenges with an open, scalable preprocessing pipeline and validates it for full-duplex training.
- Full-duplex SLMs require conversational data that captures overlapping speech, backchanneling, interruptions, and multi-turn interaction.
- Large-scale audio resources are predominantly single-stream, while high-fidelity conversational corpora remain limited in scale and diversity for supervised fine-tuning.
- In-the-wild recordings amplify diarization and transcription errors through overlaps, backchannels, acoustic clutter, and irrelevant audio regions.
- Sommelier proposes an open, robust, and scalable preprocessing pipeline for full-duplex SLMs.
- The pipeline provides overlap processing with diarization analysis, parallel model ensembling, and n-gram filtering to reduce ASR hallucinations.
- The authors validate the pipeline by fine-tuning Moshi on Sommelier-processed speech and analyzing data requirements for stable full-duplex training.
2 Method
Sommelier transforms raw conversational audio into structured training data while preserving overlaps and backchannels. Its modular pipeline combines standardization, diarization, overlap separation, music handling, and ensemble ASR for scalable processing.
- Sommelier uses a modular framework that preserves overlaps, backchannels, interruptions, and simultaneous speech while allowing configurable trade-offs between purity and authenticity.
- Audio standardization: Audio is standardized, segmented at silence intervals, and partitioned by VAD into chunks shorter than five minutes to avoid downstream memory issues.
- Speaker diarization: Sortformer replaces pyannote speaker-diarization-3.1 because it more robustly captures very short utterances such as backchannelings.
- Overlap handling: The pipeline selects overlap Case 4, retaining overlap in both speaker segments and applying two-speaker separation to overlapped intervals.
- Overlap handling: Speaker identities are assigned by comparing separated candidates with embeddings from non-overlapping speech longer than 2 seconds, then concatenating the resulting segments.
- Audio cleanup: PANNs detects background music, while Demucs extracts vocals using full two-minute contexts; SAM-Audio was excluded because its A100 inference RTF was 0.73.
- Ensemble-based ASR: ROVER combines three ASR models through word-level majority voting, and RepetitionFilter removes samples with at least 5 repetitions of 15-grams.
3 Experiments
The experiments evaluate Sommelier’s full-duplex data-processing components, including diarization, overlap separation, ASR ensembling, scalability, and downstream Moshi fine-tuning. Across these evaluations, the pipeline improves conversational robustness and transcript quality while exposing latency and training-data constraints.
- Effectiveness of Sommelier-Processed Data for Full-Duplex Models: Fine-tuning Moshi on Sommelier-processed speech improved Backchanneling, Smooth Turn-Taking, and User Interruption handling, while Pause Handling remained comparable to base Moshi.The authors associate the increased latency after fine-tuning with active processing of user input for backchannels and interruptions.
- Diarization Model Choice: Sortformer consistently outperformed Pyannote 3.1 on global diarization metrics and was more robust to short utterances and rapid turn-taking.The evaluation used DER, JER, short-duration DER, and turn-taking-region DER on VoxConverse recordings with at most four speakers.
- Speech Quality of Overlap Separation: As overlap ratio increased, the baseline degraded significantly, whereas Sep consistently outperformed Original and approached Oracle perceptual quality.The overlap evaluation used 900 two-speaker mixtures across nine SIR and overlap-ratio conditions.
- ASR Ensemble Performance: 37%: the three-model ASR ensemble reduced WER from 6.26% to 3.92% versus the single Whisper-large-v3 baseline.The improvement was particularly evident in noisy segments containing low volume or background music, and the method corrected repetitive Whisper hallucinations.
- ASR Ensemble Performance: 3×: the ASR ensemble required approximately three times the baseline inference time.The extra latency was primarily attributed to Canary, with concurrent loading and inference adding slight overhead.
- Latency: 0.0443 RTF per GPU: three concurrent processes on one A100 reduced processing cost, enabling 10,000 hours of audio to be processed with eight A100 GPUs in approximately 55 hours.A single process achieved 0.1746 RTF, or 0.133 without optional FlowSE denoising; peak memory usage was 23GB.
4 Conclusion
Sommelier is presented as the first scalable, open-source pipeline for full-duplex SLMs. It combines diarization, overlap handling, and ensemble ASR, and is released to support reproducible research on natural, real-time human–AI interaction.
- Conclusion: Sommelier combines rigorous diarization, overlap handling, and ensemble-based ASR to improve transcript quality for full-duplex SLM data.The pipeline was validated by fine-tuning Moshi on Sommelier-processed speech.
- Conclusion: The released pipeline is intended to support reproducible industrial research and accelerate progress toward natural, real-time human–AI interaction.
Limitations
The pipeline is designed for conversational speech and does not explicitly cover non-speech acoustic events or general sound scenes. Its separated audio can also have slightly lower fidelity than recordings with isolated speaker channels.
- The pipeline exclusively processes speech data, limiting coverage of non-speech acoustic events and general sound scenes.
- Separated audio fidelity is slightly inferior to datasets recorded with distinct, isolated channels because artificial separation may introduce minor acoustic artifacts.
Ethical Considerations
The paper emphasizes open-source licensing and responsible use of high-fidelity speech-processing tools. It notes risks of non-consensual voice cloning or deepfake generation and calls for consent and privacy safeguards.
- The pipeline’s software, libraries, and pretrained models use commercially permissive licenses, primarily MIT and Creative Commons.
- Demonstration podcast samples were selected from sources explicitly released under Creative Commons licenses, with usage terms verified.
- The authors acknowledge that high-quality conversational datasets may be misused for non-consensual voice cloning or deepfake generation.
- They recommend processing private data only with appropriate consent and privacy safeguards.
A.2 Large-Scale Speech Datasets
Existing speech datasets and processing tools remain poorly suited to full-duplex modeling: many lack spontaneous multi-speaker interaction, while others face scale, quality, transparency, diarization, and transcription limitations.
- LibriSpeech and GigaSpeech are dominated by scripted read speech or solitary monologues rather than spontaneous interactive dialogue.
- Fisher and Switchboard provide multi-speaker interaction but have 8kHz telephony quality, narrow bandwidth, and relatively small scale.
- Speech data-curation recipes remain largely proprietary, despite frequent open release of Speech Language Model weights.
- Single-stream processing tools are ill-suited to concurrent conversational dynamics required for full-duplex communication.
- Standard diarization tools can misinterpret overlaps and rapid turn-taking as segmentation errors or noise, degrading conversational structure.
- Whisper-like ASR systems may generate repetitive loops or nonsensical text during silence or non-speech intervals.
B.1 Analysis of Results
The overlap-separation module provides its largest benefits under severe overlap and for the subordinate speaker, while baseline comparisons at lower overlap can be artificially favorable because they include substantial clean audio.
- Speaker 2 WER improves from 0.444 to 0.138 at 0 dB SIR and 1.0 overlap with separation, while Speaker 1 improves less.
- At 1.0 overlap, separation restores UTMOS from baseline values around 1.7 to approximately 3.0.
- STOI remains above 0.9 under full overlap when separation is applied.
- Baseline SI-SDR can appear competitive at ρ ∈{0.2, 0.5} because direct extraction leaves most segments non-overlapping and clean.
- The overlap cases distinguish backchanneling, where one segment is contained within another, from partial overlap without containment.
- Figure 3 compares WER by method, SIR, and overlap ratio for both speakers, with standard-deviation error bars.
D.1 Dataset Statistics
This section reports statistics for the data used in fine-tuning and indicates that the training data spans diverse conversational domains.
- The statistics describe the data fine-tuned in Section 3.1.
- Figure 6 shows that the training data comes from a wide range of conversational domains.
- The section connects the reported dataset statistics to the diversity of conversational-domain sources.
D.2 Full-Duplex-Bench 1.0: Metric Definitions
Full-Duplex-Bench 1.0 evaluates spoken dialogue models across pause handling, backchanneling, smooth turn-taking, and user interruption. Its metrics quantify turn-taking behavior, timing, and response relevance under full-duplex interaction.
- Full-Duplex-Bench 1.0 evaluates pause handling, backchanneling, smooth turn-taking, and user interruption handling.
- Latency is defined as ∆i = tstart,i − tend,i, measuring response onset relative to the end of the relevant user event.Here, tstart denotes response onset and tend denotes the end of the relevant user event.
- Pause Handling: Pause Handling uses Synthetic/Candor TOR (↓) to measure how often the model starts speaking during mid-utterance pauses.A failure requires output during a pause lasting ≥1 second or containing > 3 words.
- Backchanneling: Backchanneling measures full-turn takeover with Backchannel TOR (↓), normalized frequency with Frequency (↑), and timing alignment with JSD (↓).
- Smooth Turn Taking: Smooth Turn Taking reports Candor TOR (↑) for successful responses and Latency (↓) for instances where the model takes the turn.
- User Interruption: User Interruption measures response rate with Interruption TOR (↑), response timing with Latency (↓), and semantic relevance with a GPT-4o score (↑).The GPT-4o relevance score uses a 0–5 scale.
D.3 Results on Full-Duplex-Bench 1.5
The paper evaluates Sommelier-processed data with Full-Duplex-Bench 1.5 and describes fine-tuned Moshi as improving audio quality and conversational responsiveness across overlap scenarios. The section also motivates rich, context-aware captioning for speech metadata.
- Sommelier-processed data is evaluated with Full-Duplex-Bench 1.5, the successor to v1.0.
- Fine-tuned Moshi significantly outperforms the base model across all evaluated overlap scenarios.
- SI-SDR in the ‘Background Speech’ scenario improves from 5.43 dB to 20.76 dB after fine-tuning.
- Fine-tuned Moshi achieves sub-second average stop and response latencies in most overlap cases.
- Captioning: Qwen3-Omni-Captioner generates metadata covering emotion, gender, age group, and situation descriptions for audio segments.
- Captioning: Context-aware captioning conditions each segment’s caption on the preceding two audio segments to preserve conversational context.For consecutive segments a1, a2, and a3, the method uses P(C3|I, a1, a2) to generate C3.
F Example
The example shows Sommelier processing a two-minute audio recording into speaker-attributed segments with multiple ASR transcripts, separation flags, timestamps, and processing metadata. The displayed excerpts include short acknowledgements, overlapping dialogue, and transcript variation across models.
- Figure 7 visualizes preprocessing results for a 1-minute audio clip using a mel-spectrogram.
- The example processes a 120.0-second recording, equivalent to 2.0 minutes of audio.
- Longer excerpts show transcription differences in names and wording, including “Ira Glass,” “Iraq Glass,” and “Tyra Glass.”
- Each displayed segment can include transcripts from Whisper, Parakeet, and Canary together with a speaker label.
- The example marks segments as separated while indicating whether SepReformer separation was used.
- The JSON records segment start and end timestamps alongside processing-time and real-time-factor metadata.
- Short acknowledgements such as “That’s right.”, “Yep.”, and “That’s fair” appear with differing ASR outputs.